Agentic AI Evaluator – Singapore Languages

About the CompanyJob OverviewWe are looking for experienced professionals to evaluate the performance, quality, and effectiveness of an AI agent (AgentX) across real-world, multi-step tasks.This role involves agentic AI evaluation, where you will assess how an AI agent plans and executes tasks, uses tools, interacts with data environments, follows instructions, and ultimately completes the assigned task. This is more than simple chatbot response evaluation and requires strong analytical and critical-thinking skills.About the RoleThis role involves agentic AI evaluation, where you will assess how an AI agent plans and executes tasks, uses tools, interacts with data environments, follows instructions, and ultimately completes the assigned task. This is more than simple chatbot response evaluation and requires strong analytical and critical-thinking skills.ResponsibilitiesReview agentic traces, including tool calls, actions, and interactions with data environments.Analyze the complete sequence of actions taken by the AI agent to complete a task.Compare two candidate responses and determine which response better fulfills the task requirements.Evaluate AI-generated responses based on accuracy, helpfulness, safety, relevance, and adherence to instructions.Assess whether the AI agent has correctly understood and completed multi-step tasks.Identify and flag unclear, incomplete, inconsistent, or ambiguous cases based on defined evaluation guidelines.Review the agent's planning, reasoning process, tool usage, and final output.Maintain consistency and accuracy while applying project-specific evaluation criteria.Document evaluation outcomes and provide appropriate feedback where required.Follow all project guidelines, quality standards, and confidentiality requirements.QualificationsMaster’s degree or above in any relevant field, ORCandidates with relevant professional experience and strong analytical/language skills may also be considered.Required SkillsProficiency in one of the specified Singapore languages.Good proficiency in English, with strong reading and comprehension skills.Strong analytical and critical-thinking abilities.Excellent attention to detail.Ability to understand and evaluate complex, multi-step tasks.Ability to compare outputs objectively and make evidence-based decisions.Strong ability to follow detailed guidelines and evaluation criteria.Good written communication skills.Preferred SkillsCandidates with experience in any of the following areas will be preferred:AI/LLM EvaluationGenerative AIAI Data AnnotationData Quality / Content EvaluationNLP / LinguisticsSearch or Content Quality EvaluationAI Agent TestingMachine Learning or AI-related projectsPay range and compensation packageNot specified.Equal Opportunity StatementNot specified.Additional Role RequirementsAbility to work independently and consistently follow project guidelines.Ability to handle repetitive evaluation tasks while maintaining high accuracy.Strong understanding of language, context, intent, and cultural nuances.Ability to identify subtle errors, inconsistencies, and quality issues in AI-generated outputs.Languages RequiredProficiency in one of the following Singapore languages is mandatory:Mandarin Chinese – SingaporeMalay – SingaporeTamil – SingaporeHindi – SingaporeBengali – Singapore