Principal Software Engineer

Note: If you previously applied for a similar role with us, please do not reapply. We will review prior applications internally and reach out if your background aligns with this opportunity. This role is specifically targeted at the Principal Engineer level and requires a minimum of 8–10 years of relevant industry experience.Principal Software Engineer AI & Distributed SystemsWe are looking for engineers who have operated real production systems at scale — not just built prototypes or proof-of-concepts.You have been on-call. You have debugged live production incidents under pressure. You understand the difference between code that looks clean in a pull request and code that remains stable during peak production traffic, downstream failures, and operational stress.This is not a research-only role or a greenfield experimentation position. This is a principal-level engineering role focused on building, operating, and evolving critical backend and AI-powered systems in production environments.You will be expected to take ownership, drive engineering quality, and make sound technical decisions when systems fail — because eventually, they do.What You’ll DoOwn and maintain production-grade Python services end-to-end, including reliability, scalability, observability, and operational support.Diagnose and resolve live production issues using distributed tracing, log analytics, profilers, Azure Monitor, Application Insights, and related observability tooling.Design, build, and operate Azure-native distributed systems using Azure Functions, Durable Functions, Event Grid, Service Bus, and supporting cloud infrastructure.Build and maintain production-ready AI systems, including RAG pipelines, agentic workflows, and LLM orchestration frameworks such as LangChain, AutoGen, and Semantic Kernel.Develop systems that are observable, resilient, recoverable, and designed with operational realities in mind.Establish and uphold strong engineering standards around code reviews, CI/CD, testing strategy, deployment safety, and incident post-mortem practices.Contribute to secondary technology stacks, including C#, when needed. Deep expertise is not required, but practical working proficiency and transparency around strengths and limitations are expected.What We’re Looking ForNon-Negotiable Qualifications8–10+ years of professional backend or full-stack engineering experience, including significant hands-on ownership of systems running in production.Strong Python engineering expertise, including performance optimization, concurrency, debugging, and production reliability considerations.Real-world Azure production experience — beyond certifications or lab environments. You have worked through issues such as cold-start latency, distributed failure modes, event ordering, scaling bottlenecks, and operational recovery.Hands-on experience using monitoring, observability, and debugging tooling as part of regular engineering workflows.Ability to clearly explain a past production incident, including:root causeinvestigation approachremediation stepsvalidation strategylong-term prevention measuresStrong Additional SignalsExperience shipping and maintaining production LLM or RAG systems beyond initial launch, including handling prompt drift, latency optimization, evaluation degradation, and cost management.Experience with high-concurrency or distributed data systems, including queue-backed architectures, streaming systems, or large-scale SQL/NoSQL workloads.Ownership of CI/CD systems and deployment pipelines end-to-end, including rollback strategies, deployment safety, and operational automation.Integrity & ValidationExperience VerificationAll candidates will undergo comprehensive background and experience verification.Authenticity Over BuzzwordsWe strongly value candidates who are honest about both their strengths and limitations. We are looking for genuine engineering depth, operational judgment, and production experience — not keyword-heavy résumés.Additional InformationNo visa sponsorship is available for this position. Applicants must have valid authorization to work in the United States.Final compensation will be determined based on relevant skills, experience, and qualifications.Equal Opportunity StatementInnoveIQ is an Equal Opportunity Employer and complies with all applicable federal, state, and local employment laws. We do not discriminate on the basis of any protected characteristic. Information regarding benefits, where applicable, will be provided during the interview process. We are committed to providing reasonable accommodations for qualified individuals with disabilities.Consent to Communication & Use of AI TechnologyBy submitting your application and providing your contact information, you consent to receive text (SMS), email, and/or voice communications — whether automated, AI-assisted, pre-recorded, or individually initiated — from InnoveIQ, its representatives, partners, or affiliates using the contact information provided.These communications may relate to job opportunities and related recruiting information. Message and data rates may apply. Frequency may vary.You represent and warrant that the contact information you provide belongs to you and that you are authorized to receive communications at those contacts. To opt out of communications or for related questions, please contact us directly.