LLM Evaluation Engineer

Overall – 8+ | AI Exp – 3+Hands-on proficiency with libraries like Ragas, DeepEval, TruLens, Promptfoo, or OpenAI Evals to measure: Context Precision & Context Recall (retrieval quality from vector stores).Faithfulness / Groundedness (checking against hallucination in bounded context).Answer Relevance and Toxicity / Guardrail Compliance.Analyzing evaluation datasets, confusion matrices, similarity score distributions, and benchmark trends.Computing classification metrics (Precision, Recall, F1, Cosine Similarity thresholds) for retrieval systems.Azure OpenAI SDK / OpenAI API and LangChain / LlamaIndex for building automated LLM-as-a-judge scoring scripts and synthetic test dataset generation.PyTest / Unittest: Writing automated unit/regression tests for prompt templates and model version upgrades.Writing complex SQL queries to extract, curate, and version golden test datasets from relational databases (e.g., PostgreSQL, Azure SQL).Querying vector stores and telemetry databases (e.g., pgvector, Splunk/log tables) to evaluate real-world candidate/user query retrieval accuracy and fallback rates.Handling structured benchmark datasets and prompt configurations using JSON / JSONL, YAML, and CSV.