Prompt Engineer
BayOne Solutions · Mountain View, CA
Apply & track with Apply EdgeIn this role, you will bridge the gap between complex Large Language Model (LLM) capabilities and business applications. You will be responsible for designing, optimizing, and maintaining system prompts, complex context pipelines, and prompt architectures (such as Retrieval-Augmented Generation / RAG) to ensure our AI models deliver accurate, instruction-aligned, and safety-compliant outputs.
Key Responsibilities
Prompt Engineering & System Instruction: Design, test, and refine system prompts, instruction templates, and context windows across diverse LLM architectures (e.g., OpenAI GPT, Anthropic Claude, open-source models).Context Architecture & RAG: Optimize context retrieval and injection strategies, ensuring models receive relevant, dense, and structured data for downstream tasks.Evaluations & Benchmarking: Establish evaluation metrics (evals), benchmark suites, and testing pipelines to measure prompt performance, latency, accuracy, hallucination rates, and instruction adherence.Safety & Alignment: Implement guardrails, red-teaming protocols, and edge-case handling to mitigate risks such as prompt injections, jailbreaks, data leakage, and toxic outputs.Cross-Functional Collaboration: Partner with Data Engineers, Machine Learning Engineers, and Product Managers to integrate optimized prompts into core software APIs and product workflows.Model Optimization & Fine-Tuning: Collaborate with ML engineers to format and curate high-quality datasets for fine-tuning or few-shot learning setups.Qualifications & SkillsMust-HaveLLM Expertise: Deep operational understanding of LLMs, attention mechanisms, context limits, tokenization, and model behavior variations.Prompting Mastery: Proficiency in advanced techniques including Chain-of-Thought (CoT), Few-Shot learning, ReAct, System Message steering, and Structured Output generation (JSON/Pydantic schemas).
Technical Skills
Proficiency in Python and familiarity with AI frameworks/libraries (e.g., LangChain, LlamaIndex, LiteLLM, Guidance, or instructor).Data & Evals Experience: Experience building automated eval frameworks using tooling like Ragas, DeepEval, Braintrust, or custom scoring scripts.Structured Thinking: Exceptional written English and natural language clarity; ability to express complex logic precisely in text.Preferred / Nice-to-HaveExperience working with Vector Databases (e.g., Pinecone, Weaviate, Qdrant, Milvus) and hybrid search strategies.Background in Computational Linguistics, Natural Language Processing (NLP), Computer Science, or Cognitive Science.Understanding of API integration, RESTful services, and asynchronous programming in Python.BayOne is an Equal Opportunity Employer and does not discriminate against any employee or applicant for employment because of race, color, sex, age, religion, sexual orientation, gender identity, status as a veteran, and basis of disability or any federal, state, or local protected class.This job posting represents the general duties and requirements necessary to perform this position and is not an exhaustive statement of all responsibilities, duties, and skills required. Management reserves the right to revise or alter this job description.