Automation Test Engineer (AI & Chatbot Testing)

Automation Test Engineer (AI & Chatbot Testing)Location: UAEJob SummaryWe are looking for an experienced Automation Test Engineer to test and improve AI-powered customer support and troubleshooting solutions. The role focuses on test automation, API testing, AI quality validation, and performance testing to ensure reliable and accurate customer experiences.Key ResponsibilitiesAI & Chatbot TestingDevelop automated tests for AI chatbots and RAG (Retrieval-Augmented Generation) systems.Validate response accuracy, relevance, and AI performance.Measure AI quality metrics such as hallucination rates and answer relevance.Perform regression testing when AI models or knowledge bases are updated.Automation & Functional TestingAutomate testing of troubleshooting workflows and decision trees.Validate end-to-end customer support scenarios.Simulate customer interactions and verify expected outcomes.API & Integration TestingBuild and execute automated API tests.Validate integrations between AI platforms and backend systems.Create mock services and test data where required.Performance & Security TestingConduct load and performance testing of conversational AI systems.Test AI guardrails, prompt injection risks, and unsafe responses.Identify performance and reliability issues.CI/CD & Quality AssuranceIntegrate automated tests into Azure DevOps pipelines.Track quality metrics and generate test reports.Work closely with developers, AI engineers, and product teams.Requirements4+ years of experience in QA Automation or SDET roles.1+ year of experience testing AI, Chatbots, LLMs, or Agentic AI solutions.Strong programming skills in Python, TypeScript, or C#.Experience with REST APIs, GraphQL, WebSockets, Postman, and WireMock.Experience with Azure DevOps and CI/CD pipelines.Knowledge of RAG, semantic search, and vector databases.Experience with MQTT or IoT telemetry is a plus.Exposure to embedded systems or consumer electronics is preferred.