AI Agent Evaluation Engineer

Blossom AI Labs is building agentic AI systems for real-world decision making.We are looking for an AI Agent Evaluation Engineer to build realistic tasks, find where agents fail, and test whether their results can be trusted.An agent can complete a task once and still be unreliable in a real workflow. The hard question is whether it keeps making good decisions with messy data, incomplete context, ambiguous instructions, unreliable tools, and changing conditions. A higher score is useful only when the evaluation measures what matters.This role helps answer that question. You will build the evaluation tools, datasets, rubrics, and feedback loops that turn agent behavior into evidence we can inspect and act on. The work connects engineering with research: making failures reproducible, distinguishing meaningful improvements from measurement artifacts, and turning findings into better systems.We are looking for someone who can move between code, model behavior, human judgment, and product usefulness. Someone who can look at an agent run and ask: did it solve the task, use the right evidence, handle tool failures, and know when to stop or ask for help? Then build the tests that help us catch the failure next time.What you will doBuild Python evaluation harnesses for agentic workflows, retrieval, tool use, and multi-step decision making.Design realistic tasks, datasets, and rubrics with clear success criteria and reproducible execution.Evaluate behavior across models and agent configurations, including interactions between agents.Inspect traces, compare runs, and develop failure taxonomies for reasoning, retrieval, tool use, unsupported claims, escalation, and workflow mismatch.Audit evaluations for answer leakage, shortcuts, ambiguous rubrics, and unreliable judging.Turn failures and edge cases into regression tests, then track quality, reliability, latency, and cost over time.Build tools for human review and use structured feedback to improve datasets and evaluation coverage.Work with the team to translate research findings and product requirements into practical quality checks.Write clear reports that explain what changed, what the evidence supports, and where uncertainty remains.What we are looking forStrong engineering ability, with experience building reliable software, APIs, data pipelines, or internal tools.Experience or deep interest in LLMs, agents, retrieval, tool use, evaluations, observability, or human feedback systems.Ability to turn qualitative judgments into clear rubrics, datasets, metrics, and software.Careful experimental judgment: you can design fair comparisons, control variables, and distinguish real improvements from noise or evaluation flaws.Ability to debug across model outputs, prompts, data quality, retrieval, orchestration, and product assumptions.Comfort owning an ambiguous problem from investigation through implementation and documentation.Clear writing and communication. You should be able to explain why an agent failed, what to change, and how to verify the fix.Curiosity about how to make AI systems dependable in real-world decisions.