AI QA & Evaluation Engineer

Role purposeEstablish and run the quality assurance and evaluation practices needed to release AI systems safely and reliably. Test complete workflows as well as individual models, covering accuracy, retrieval quality, agent actions, security, robustness and operational behaviour. Provide clear, repeatable evidence of whether each AI release meets its intended use and agreed acceptance criteria.Key responsibilitiesEvaluation design and test dataDefine risk-based test strategies for generative AI, agent workflows, predictive models and automation integrations, matching coverage to intended use and potential impact.Create representative, versioned evaluation datasets with business experts, including normal cases, ambiguous requests, rare exceptions and sensitive brokerage scenarios.Measure grounded accuracy, retrieval relevance, citation correctness, structured-output validity, task completion and appropriate escalation to a human.Develop scoring rubrics and reviewed reference answers; calibrate automated or LLM-based judges against human assessments rather than relying on a single model score.Adversarial and end-to-end testingTest prompt injection, indirect malicious instructions, data leakage, cross-client access, excessive agent permissions and attempts to bypass approval steps.Validate end-to-end outcomes across CRM, support and other connected systems, including duplicate actions, partial completion, timeouts, retries and recovery.Test brokerage-specific risks such as inaccurate product terms, outdated policies, unsupported trading advice, inappropriate client messaging and incorrect account information.Evaluate latency, concurrency, provider outages, degraded retrieval and model changes; confirm that fallback and human handoff behave as designed.Release assurance and continuous evaluationIntegrate regression evaluations into CI/CD and compare candidate releases against approved baselines with consistent datasets and documented thresholds.Independently reproduce key quantitative test results with the Quantitative Lead, checking dataset separation, leakage controls and stability across relevant segments.Maintain defect triage, release evidence and exception records; challenge release readiness and escalate unresolved failures through the agreed governance process.Use redacted production samples, incidents and user feedback to extend test coverage while keeping development data separate from held-out evaluation sets.Essential experience and qualifications5+ years in software QA, test automation or quality engineering, including hands-on evaluation of AI, ML or data-driven systems.Strong Python, API testing, SQL and automated test framework skills, with experience integrating tests into delivery pipelines.Understanding of LLM variability, retrieval systems, agent tool execution and the limitations of exact-match testing for generated outputs.Ability to design representative test data, evaluation rubrics and meaningful thresholds with business and technical stakeholders.Working knowledge of statistical metrics, sampling, class imbalance and separation of training, validation and test data.Experience testing access controls, sensitive data handling, integration failures and recoverability in production-oriented systems.Bachelor’s degree in Computer Science, Engineering, Mathematics or a related field, or equivalent relevant experience.Preferred experienceFinancial services QA experience, especially brokerage platforms, payment processes or regulated client communications.Experience with AI red teaming, human annotation and multilingual evaluations.Experience assessing predictive models, drift monitoring and model documentation.