Senior QA Engineer — AI Platform & Agentic Solutions

About the roleWe are looking for a Senior QA Engineer with proven expertise in testing agentic systemsto establish and coordinate quality assurance across the platform and all AgenticSolutions published on it.Based in Egypt, you will work with developers, engineering leads, the Proxy Product Ownerand business stakeholders across the entire project. You will combine hands-on testingand automation with the development of consistent quality standards, evaluationmethods and release criteria.Your scope will cover both the reliability of the underlying platform and the ability ofindividual Agentic Solutions to complete business tasks accurately, securely and withintheir authorised boundaries.Key responsibilities1. QA strategy and practice

• Define and implement a QA strategy covering functional, integration, end-to-end,regression, performance and AI evaluation testing.
• Establish consistent testing standards across teams and workstreams.
• Prioritise coverage according to business impact and risk, automating critical journeysfirst.
• Work with developers to embed testing throughout delivery and improve the testabilityof new features.
• Maintain a consolidated view of quality, coverage gaps and release risks.2. platform testing
• Design and execute tests for platform APIs, user interfaces, tenant provisioning,authentication, roles and permissions.
• Validate knowledge-base management, agent and crew orchestration, tracing,configuration and billing capabilities.
• Test integrations between platform components and external services.
• Verify tenant isolation, access boundaries and correct handling of sensitive data,working with security specialists.
• Test failure handling and recovery, including timeouts, retries, unavailabledependencies and interrupted workflows.
• Ensure platform changes do not introduce regressions in existing capabilities orpublished Agentic Solutions.3. Agentic Solution validation
• Define and execute a repeatable validation process for every Agentic Solution beforepublication and following significant changes.
• Test complete business journeys against agreed requirements and expectedoutcomes.
• Validate tool selection, tool inputs, execution sequences, API interactions, knowledgeretrieval, memory and human handoffs where applicable.
• Inspect execution traces to assess intermediate actions and identify the source offailures.
• Verify that agents handle ambiguous requests, missing information, unexpectedinputs and dependency failures appropriately.
• Confirm that agents respect authorisation boundaries, approval requirements andlimits on their actions.
• Test safeguards against prompt injection, sensitive-data disclosure, unauthorisedactions and unintended or duplicate execution.
• Use controlled test environments to validate actions without affecting live customersor production data.4. AI evaluation and regression
• Build and maintain versioned evaluation datasets covering representative businessscenarios, edge cases and adversarial inputs.
• Define measurable acceptance thresholds with the Proxy Product Owner and relevantbusiness owners.
• Evaluate task completion, factual accuracy, grounding, tool-use correctness andcompliance with solution-specific requirements.
• Combine automated scoring with human review where appropriate, accounting forvariability in AI outputs.
• Run regression evaluations following changes to models, prompts, tools, knowledgesources or platform components.
• Maintain traceability between evaluation results and the versions of the solution,model and configuration tested.5. Test automation and CI/CD
• Build and maintain automated API, integration, browser-based and end-to-end testsuites.
• Integrate tests and AI evaluations into GitHub-based CI/CD workflows.
• Implement agreed quality gates for pull requests, releases and solution publication.
• Maintain reliable test data, fixtures and environments.
• Investigate unstable tests and keep automated results actionable and trustworthy.6. Performance and reliability
• Test platform and solution behaviour under expected load and concurrent usage.
• Measure latency, throughput, failure rates and recovery behaviour.
• Assess response times and cost per completed agentic journey against agreedtargets.
• Identify bottlenecks and work with engineers to verify improvements.7. Defect management and release readiness
• Record defects in GitHub with reproducible steps, expected and actual results, severity,evidence and affected versions.
• Work with the Proxy Product Owner and engineering leads to assess business impactand prioritise fixes.
• Track defects through resolution, retesting and closure.
• Provide evidence-based release and publication recommendations.
• Document outstanding defects, untested areas and residual risks so designatedrelease owners can make informed decisions.
• Translate production incidents and recurring defects into regression tests and processimprovements.8. Reporting and collaboration
• Provide a formal weekly quality update to the Global Head of Platform, Proxy ProductOwner and engineering leads.
• Report test results, critical defects, coverage gaps, AI evaluation trends and releasereadiness.
• Escalate significant quality risks promptly, with their impact and recommendedactions.
• Participate in refinement and planning to challenge unclear requirements andimprove acceptance criteria.
• Present relevant findings and demonstrations during team reviews and Show & Tellsessions.Required experience and skills
• At least five years of experience in software QA or test engineering, includingsubstantial hands-on automation experience.
• Demonstrated experience establishing or improving QA practices across softwaredelivery teams.
• Proven hands-on expertise testing tool-using AI agents and multi-step agenticworkflows, including intermediate actions, failure recovery and business-outcomecompletion.
• Practical experience building AI evaluation datasets, automated evaluation pipelinesand regression suites.
• Strong Python skills for test automation, using pytest or equivalent tooling.
• Strong API testing skills and experience with browser automation frameworks such asPlaywright, Cypress or equivalent.
• Experience integrating automated tests into CI/CD pipelines, preferably GitHubActions.
• Experience with performance and load testing of API-based services.
• Working understanding of RAG, agent orchestration, tool calling, memory andexecution tracing.
• Familiarity with security and privacy testing relevant to enterprise AI systems.
• SQL and data-validation skills.
• Excellent written and spoken English, with the ability to explain quality risks clearly totechnical and business stakeholders.
• A proactive approach: you identify gaps, investigate issues independently and bringpractical recommendations.Nice to have
• Experience with AI evaluation tools such as promptfoo, Ragas, DeepEval or equivalent.
• Experience testing enterprise platforms with multiple tenants.
• Background in conversational AI, contact-centre technology or customer experience.
• Experience testing voice agents, including interruptions, speech recognition, latencyand human transfer.
• Exposure to GCP, containerised environments and cloud-native applications.
• Experience with production monitoring and AI observability.
• ISTQB, ISTQB AI Testing or equivalent certification.