Lead Data Engineer

Our client, a technology-driven organization in the healthcare industry, is seeking an experienced Lead Data Engineer / Principal Data Engineer to lead the design, development, and optimization of production-grade data pipelines and cloud data platforms. This is a hands-on technical leadership (player/coach) opportunity focused on Azure, Databricks, real-time streaming, IoT telemetry, and scalable data services supporting operational applications, analytics, and ML/AI initiatives. The ideal candidate will combine strong data engineering expertise with architectural leadership, team management, and stakeholder collaboration. This individual must be comfortable working directly with technology while mentoring engineers, establishing best practices, and translating evolving business needs into scalable solutions.ResponsibilitiesDesign, develop, optimize, and maintain production-grade batch and real-time streaming data pipelines using Azure and Databricks.Build scalable ETL/ELT workflows to ingest, transform, validate, and process structured, semi-structured, and streaming data.Develop reliable data products, APIs, microservices, and data services supporting operational applications, analytics, and ML/AI initiatives.Utilize Python, PySpark, Spark SQL, SQL, Delta Lake, and Databricks Workflows to build maintainable and scalable data solutions.Lead real-time streaming and IoT telemetry initiatives using Azure Event Hubs, Stream Analytics, and related cloud technologies.Establish Databricks lakehouse architecture standards, including Medallion architecture, data modeling, performance tuning, and cost optimization.Lead, mentor, and manage data engineers while providing hands-on technical guidance, architectural direction, and code reviews.Implement engineering best practices for DevOps, CI/CD, automated testing, deployment, monitoring, and documentation.Collaborate with business, product, software engineering, analytics, and ML/AI stakeholders to translate requirements into scalable technical solutions.Support ML/AI initiatives through reliable data pipelines, feature engineering, model-ready datasets, and data integration.Establish data quality, security, governance, observability, and production reliability standards.Troubleshoot complex production issues, conduct root cause analysis, and drive continuous platform improvements.RequirementsBachelor's degree in Computer Science, Engineering, Information Technology, or a related technical field.10+ years of experience in data engineering, software engineering, data platforms, distributed systems, or related technical disciplines.7+ years of hands-on experience designing, building, and supporting production data pipelines, platforms, or services.5+ years of experience with modern cloud data platforms, including Databricks, Apache Spark, Delta Lake, SQL, and Python/PySpark.3+ years of experience leading, managing, or mentoring data engineering teams.Strong hands-on expertise with Microsoft Azure cloud services, including data integration, streaming, storage, and production operations.Experience designing and operating real-time streaming, event-driven, or near-real-time data pipelines.Strong understanding of Databricks lakehouse architecture, ETL/ELT development, data modeling, and pipeline optimization.Experience implementing DevOps, CI/CD, automated testing, version control, and deployment practices.Experience developing data services, APIs, microservices, or reusable data consumption frameworks.Knowledge of data quality, observability, monitoring, security, governance, and cloud cost optimization.Strong architectural judgment, stakeholder management, communication, and problem-solving skills.Ability to operate as a hands-on technical leader in a fast-paced environment with evolving requirements.Preferred Qualifications12+ years of relevant professional experience, including principal, staff, lead, architect, or engineering management responsibilities.Experience directly managing engineers, establishing priorities, and driving team performance and accountability.Background in commercial software, SaaS, digital products, healthcare technology, fintech, IoT, or other product-driven organizations.Experience working in startup, scale-up, or rapid product development environments.Experience with Azure Event Hubs, Azure Stream Analytics, Azure Service Bus, Azure Data Factory, Azure Functions, ADLS, Azure Cosmos DB, or Event Grid.Experience processing connected-device telemetry, IoT data, remote patient monitoring data, or other high-volume event streams.Familiarity with healthcare data, HIPAA requirements, regulated environments, or safety-critical applications.Experience building data platforms supporting APIs, microservices, ML/AI applications, GenAI workflows, RAG, and executive analytics.Knowledge of data contracts, semantic layers, feature stores, schema evolution, change data capture (CDC), and data lineage.Experience with performance tuning, FinOps, cloud cost optimization, and production incident management.Demonstrated success developing MVPs, validating technical approaches, and scaling prototypes into enterprise production solutions.