QA AI Evaluator

About the roleThe QA AI Evaluator will be responsible for assuring the quality, reliability, governance, and business effectiveness of Agentic AI solutions. The role focuses on evaluating AI agents, prompts, LLM outputs, and end-to-end workflows to ensure accurate, consistent, and trusted outcomes.

• Define AI evaluation frameworks, standards, KPIs, and success criteria.
• Validate agent accuracy, reasoning, reliability, workflow quality, and business outcomes.
• Assess prompts, LLM responses, hallucination risks, and AI-generated content quality.
• Benchmark models, tools, prompts, and agent configurations to improve performance.
• Ensure Responsible AI, security, privacy, governance, and compliance requirements are met.
• Provide clear insights, risks, recommendations, and scorecards to leadership and stakeholders.What you’ll be doing
• Strong experience in AI Testing, Quality Engineering, AI/LLM Evaluation, or Agentic AI Assurance.
• Deep understanding of LLMs, Prompt Engineering, RAG, AI Agents, and Multi-Agent Architectures.
• Expertise in defining evaluation frameworks, quality metrics, KPIs, scorecards, and benchmarking approaches.
• Experience validating AI accuracy, reasoning, reliability, hallucinations, and business outcomes.
• Knowledge of Responsible AI, Security, Privacy, Governance, Risk, and Compliance practices.
• Strong analytical and problem-solving skills with experience in root cause analysis and continuous improvement.
• Familiarity with AI evaluation tools, test automation, and data-driven quality assessment techniques.
• Ability to collaborate effectively with Product, Engineering, Architecture, QA, and Business stakeholders.
• Excellent communication and stakeholder management skills, including presenting insights to leadership.
• Experience driving AI adoption, change management, and business transformation initiatives.Essential Skills / Experience
• 4-6 Years of experience with strong experience in AI testing, quality engineering, AI/LLM evaluation, or Agentic AI assurance.
• Good understanding of LLMs, prompt engineering, RAG, AI agents, and multi-agent workflows.
• Ability to define evaluation frameworks, quality metrics, KPIs, scorecards, and success criteria.
• Hands-on experience validating AI accuracy, reasoning, reliability, hallucination risk, relevance, completeness, and business outcomes.
• Strong analytical, problem-solving, stakeholder management, and communication skills.Desirable Skills / Experience
• Minimum 3-4 years of experience with AI evaluation tools, automation frameworks, and data-driven quality assessment techniques.
• Exposure to benchmarking models, prompts, tools, and agent configurations to improve performance.
• Experience supporting AI adoption, user acceptance, change management, or wider business transformation initiatives.About usBT is the UK’s leading communications group and the holding company behind some of the country’s most recognised brands – including BT, EE, Openreach and Plusnet. Our purpose is as simple as it is ambitious: we connect for good. Our customers include consumers, small, medium and large businesses, public sector organisations and other communications providers.Having come through the most capital-intensive phase of our fibre investment, our focus now is on what comes next – simplifying how we operate, using technology and AI to work smarter, and organising ourselves to serve customers better and grow sustainably.We have a singular culture that unites all our people: we are customer-first challengers, who are committed, clear and connected. These behaviours unite us as one team to deliver for our colleagues, our customers, our stakeholders and the country. Joining BT means working at the heart of a business that matters to the UK, with the opportunity to shape decisions, influence outcomes and help set the future course of one of the country’s most important companies.