Lead Data & BI Engineer

Role Title: Senior Data Engineer(Azure | Databricks | End-to-End Delivery)Experience: 8–12 years (with strong Databricks depth)Location: Client location - DubaiEmployment Type: Contract - Full timeWe are looking for a lead, hands-on Databricks Lead / Lakehouse Architect who can drive project kickoff, define the implementation strategy, and lead end-to-end delivery of an Azure Databricks Lakehouse along with Power BI semantic modeling and reporting for hospitality analytics. This role will own architecture, technical decisions, delivery planning, stakeholder alignment, and initial build-out, and then help ramp the broader team for implementation and rollout.Key ResponsibilitiesProject Kickoff & StrategyLead discovery workshops, clarify business objectives, define scope, milestones, and delivery approach.Lead discovery workshops with hospitality business/tech stakeholders (Revenue Management, Operations, Finance, Loyalty, Digital).Define target-state lakehouse architecture and delivery roadmap (phased onboarding of domains: reservations, stays, POS, loyalty, digital).Create end-to-end implementation strategy: architecture, design standards, data onboarding plan, governance, and operating model.Define best practices, coding standards, branching strategy, CI/CD approach, and environments (dev/test/prod).Architecture & Platform BuildDesign and implement Azure Databricks Lakehouse architecture using Delta Lake and Medallion (Bronze/Silver/Gold).Design data ingestion patterns for batch (and streaming if needed) from varied sources (APIs, DBs, files, event streams).Build scalable transformation frameworks using PySpark / Spark SQL with performance and cost optimization.Data Engineering DeliveryImplement and review complex pipelines: incremental loads, CDC patterns, SCD Type 1/2, data quality rules, and reconciliation.Own performance tuning: partitioning, file sizing, Z-ORDER, caching, adaptive query execution, cluster sizing, autoscaling.Implement monitoring/alerting, job orchestration, dependency management, and restart/recovery strategies.Security, Governance & OperationsDefine security model: RBAC/ACLs, service principals, Key Vault integration, secrets management.Implement Unity Catalog strategy (if applicable): catalogs, schemas, lineage, permissions, data sharing.Establish operational readiness: runbooks, SLAs, logging, audit, cost governance.Collaboration & LeadershipAct as primary technical point of contact for client stakeholders.Produce design documentation (HLD/LLD), implementation roadmap, and knowledge transfer plan.Mentor engineers, conduct code reviews, and ensure delivery quality.Must-Have SkillsDeep hands-on Databricks experience in end-to-end implementations (not only development).Strong Spark expertise: PySpark + Spark SQL, distributed processing concepts, optimization.Strong experience with Delta Lake (MERGE, OPTIMIZE, VACUUM, schema evolution, time travel).Experience designing lakehouse architecture (medallion layers, curated zones, semantic datasets).Strong Azure data platform exposure: ADLS Gen2, Azure Data Factory (or alternative orchestration), Key Vault, Azure SQL/Synapse (any).Power BI development experience is mandatory.Strong understanding of:Incremental ingestion, CDC, SCD, data modeling for analyticsData quality frameworks, reconciliation, and observabilityNice-to-Have (Strong Plus)Unity Catalog implementation experience (access control, lineage, governance).CI/CD for Databricks: Azure DevOps/GitHub, Databricks Repos, asset bundles, Terraform.Streaming patterns (Kafka/Event Hubs), Auto Loader, DLT.Exposure to BI semantic layers (Power BI)Databricks certifications (Data Engineer Associate/Professional).