AI Conversation Research & Evaluation Specialist

About GramianGramian Consultancy is a boutique consultancy specializing in IT professional services and engineering talent solutions. With a strong background in software engineering and leadership, we help companies build high-performing teams by matching them with professionals who truly fit their needs.About The RoleWe are seeking experienced Subject Matter Experts (SMEs)in psychology, social work, counseling, qualitative research, human-computer interaction (HCI), communication, conflict resolution, coaching, and related advisory fields to support AI conversation evaluation research. You will review conversations between people and AI models, assess whether sufficient context was gathered before recommendations were made, identify missed opportunities for clarification, and provide structured feedback to improve conversational quality. The ideal candidate has at least 3-5 years of relevant professional experience, strong analytical skills, and the ability to distinguish information that could materially change a recommendation from details that are less relevant.Key ResponsibilitiesReview AI conversations involving real-world interpersonal situations and everyday decision-makingIdentify missing context and recommend relevant follow-up questions that could improve the quality of AI responsesEvaluate whether gathering additional information would materially change an AI-generated recommendationDistinguish between load-bearing information that affects a decision and non-essential or merely descriptive detailsConduct qualitative reviews and document findings using structured evaluation guidelinesAssess whether conversations adequately consider relevant personal, psychological, social, financial, practical, and organizational factorsIdentify missed opportunities for clarification, deeper understanding, and information gatheringReview conversations involving relationships, work, education, family, wellbeing, planning, and resource constraintsCollaborate with research teams to ensure consistent, evidence-based assessmentsMaintain high standards of accuracy, clarity, and consistency while meeting project deadlinesRequirementsCandidates must have at least 3-5 years of relevant direct professional experience in one or more of the following domains:2. Counseling, Therapy, or Psychology3. Qualitative Research, HCI, or Communication4. Mediation, Coaching, Student Services, Organizational Consulting, or Related Advisory Fields Licensed Social WorkMaster's degree in Social Work (MSW) or an equivalent graduate-level qualificationLCSW, LICSW, LMSW, or an equivalent applicable social work license or credentialExperience in client assessment, case review, interviewing, counseling, or related social-work practiceMaster's degree in a relevant discipline with an LMFT, LPC, LCPC, or equivalent counseling or therapy license or credential; orPhD or PsyD in Clinical, Counseling, Social, or Industrial-Organizational PsychologyRelevant experience in counseling, therapy, psychological assessment, interviewing, case review, or other client-facing professional workMA, PhD, or equivalent advanced expertise in Human-Computer Interaction, Communication, Anthropology, Sociology, Qualitative Research, or a closely related disciplineExperience conducting qualitative research, interviews, user research, conversation analysis, thematic analysis, or qualitative codingAbility to analyze conversations systematically and identify missing context, communication patterns, and opportunities for clarificationRelevant professional qualification, credential, or demonstrated subject-matter expertiseAt least 5 years of direct advisory or client-facing experience for candidates qualifying through this pathwayExperience in coaching, mediation, student advising, organizational consulting, or comparable professional workDemonstrated ability to gather context, understand individual or organizational needs, and provide informed recommendations