Member of Technical Staff, AI Evaluation

Role: Member of Technical Staff - AI Evaluation Engineer

Location: San Francisco Bay AreaWe are working with a well-funded AI company, recently backed by a significant investment from a tier-one VC, who are building an AI engineer agent for the physical world.The role:Study how engineers and customers use the agent in the field, identify where it succeeds or fails from an engineering perspective, and turn those observations into actionable evaluationsRecreate real-world engineering tasks and failure modes in reusable environments for the development teamsBuild and refine evaluation methodologies, including automated judges and human evaluation processesAssess whether automated judges align with expert human judgment, and improve them when they don’tWork across engineering, product, customers, SMEs, and external partners to ensure evaluations reflect real engineering workflows rather than abstract benchmarksUse evaluation results to show where the agent needs to improve and which problems matter most to usersWhat they’re looking for:Technical background in a physical engineering domain such as power systems, power electronics, mechanical engineering, aerospace, robotics, or a closely related fieldStrong enough understanding of agentic AI systems to design and reason about evaluationsExperience with AI evaluations, including creating evaluation tasks, designing judges, and comparing automated evaluation against human judgmentComfort working directly with customers, engineers, SMEs, and internal technical teams to extract what matters from ambiguous workflowsStrong product judgment, with the ability to turn observations about how a system is used or fails into concrete evaluation workMotivation to make AI systems genuinely capable in the physical worldWhat is on offer:Competitive salary + meaningful equityThe chance to define how a frontier engineering agent is evaluated and improvedIf interested, please apply or reach out to me directly