QA & AI Evaluation Engineer

We’re looking for a QA & AI Evaluation Engineer to join a team building and improving a large-scale internal AI platform.This isn’t a traditional QA role.You’ll be working at the intersection of software quality and AI, helping define how Generative AI applications are tested, evaluated and monitored in real-world production environments.As LLM-based applications aren’t always deterministic, you’ll help answer a much more interesting question than simply “does it work?”:Is the AI accurate, consistent, safe and genuinely useful?What you’ll be doingDefine measurable AI quality and evaluation criteria based on business requirements.Build and maintain golden datasets covering real-world scenarios, edge cases and failure scenarios.Design deterministic and non-deterministic evaluations using techniques such as:LLM-as-a-judgeSemantic similarityGroundedness and relevanceSchema and format validationSource/citation validationTool-call validationEvaluate RAG applications for retrieval quality, groundedness and source accuracy.Test AI agents, including tool selection, parameters, multi-turn behaviour and error handling.Carry out adversarial testing and AI red teaming, including prompt injection, jailbreaks, data leakage and misleading outputs.Test and validate guardrails and content filters.Monitor regression following model upgrades and prompt changes.Integrate automated AI evaluations into CI/CD and delivery pipelines.Monitor AI quality in production using observability and tracing.Support UAT sessions and turn production feedback into repeatable test cases.What we’re looking forExperience working as a QA Engineer within an Agile software or IT environment.Hands-on experience testing or evaluating LLM / Generative AI applications, or strong knowledge of AI evaluation methodologies.Good understanding of the challenges involved in testing non-deterministic AI systems.Strong analytical and problem-solving skills.A naturally critical mindset with an interest in finding edge cases, unexpected behaviours and potential failure scenarios.Excellent Dutch and good English. French is a plus.Comfortable working across both technical teams and business stakeholders.Technical experienceYou don’t need to tick every box, but experience with some of the following would be highly relevant:Python / pytestLangSmith, Azure AI Foundry Evaluations, promptfoo, DeepEval, Ragas or LangfuseRAG & embeddingsLLM evaluationPrompt engineeringAI agents & tool callingAPI testing: REST, Postman, OpenAPIAzure DevOps & CI/CDSQL / JSONOpenTelemetry / Application InsightsScrum / KanbanWhy consider it?This is a good opportunity for someone who already has a strong QA/testing foundation but wants to specialise further in AI quality and evaluation.You’ll get hands-on exposure to LLMs, RAG, AI agents, automated evaluation, red teaming and production monitoring, while helping establish the standards used to determine whether AI applications are genuinely ready for production.If your background sits somewhere between QA Engineering, Test Automation and AI/LLM evaluation, I’d be keen to speak.