About DataSpectaDataSpecta is a Data & AI consulting and engineering company helping organizations transform data into measurable business value through advanced analytics, modern data platforms and enterprise AI solutions.We design and deliver solutions across both on-premise and cloud environments, working closely with enterprise customers from discovery through production.Alongside customer projects, we build our own AI products such as CogniSpecta (enterprise knowledge & cognitive search) and CodeSpecta (AI-powered software engineering).We're looking for a Senior AI QA / Evaluation Engineer based in Baku, Azerbaijan, with strong test automation expertise and hands-on experience evaluating LLM and ML systems.You will own quality assurance for production AI systems in a regulated enterprise environment, acting as the quality gate for every production release.Location & Work Model
Location: Baku, AzerbaijanWork model: [Hybrid / On-site]What You Will Work OnEvaluation harnesses and golden datasets for enterprise AI agentsEvaluation of RAG assistants and report generation agentsAdversarial and security testing for LLM applicationsAudit evidence and quality governanceResponsibilitiesBuild versioned golden datasets and evaluation harnessesGate production releases on evaluation baselines in CIRevalidate agents at each model version changeRun quality control on recurring reportsRun adversarial tests for prompt injection, data leakage and unsafe tool useTriage support and agent output incidentsMaintain the audit evidence packRequired QualificationsBased in Baku or willing to relocate4+ years in QA or test engineering, including test automation ownershipProven experience evaluating LLM or ML outputsStrong Python skills for evaluation toolingExperience with formal defect managementFluent EnglishTechnical SkillsEvaluation frameworks: Ragas, DeepEval, promptfoo or equivalentMetrics: Groundedness, faithfulness, context precision/recall, answer correctness, tool call accuracyStatistics: Repeated sampling, bootstrapped confidence intervalsAutomation: Python, pytest, API testing, Playwright, CIAdversarial testing: OWASP Top 10 for LLMs, garak, PyRITObservability: Langfuse, Arize Phoenix or equivalentMultilingual: Azerbaijani and mixed-language evaluationPreferred Qualifications (a plus)Audit evidence experience in a regulated sectorModel risk management frameworksWhat We ValueCuriosity about solving business problems with AIComfort across business, data and technologyEnd-to-end ownershipContinuous learningFollow us:LinkedIn: https://www.linkedin.com/company/innovance-consultancyLinkedIn: https://www.linkedin.com/company/dataspectaInstagram: https://www.instagram.com/innovanceconsultancyInstagram: https://www.instagram.com/dataspecta