Senior Databricks Engineer

General DescriptionJoin a leading enterprise data transformation initiative supporting the design and delivery of modern cloud-based data platforms for a large-scale organization in Abu Dhabi. This engagement focuses on building scalable, high-performance data engineering solutions using the Databricks Lakehouse Platform, enabling advanced analytics, AI, and data-driven decision-making across the business.We are looking for an experienced Senior Databricks Engineer to lead the development of enterprise-grade data pipelines, Lakehouse architectures, and cloud-native data platforms. You will work closely with Data Architects, Data Engineers, Data Scientists, DevOps Engineers, and business stakeholders to build secure, scalable, and high-performing data solutions that support analytics, machine learning, and operational workloads.This role is ideal for a senior data engineering professional with deep expertise in Databricks, PySpark, Spark SQL, Delta Lake, and modern cloud platforms, who is passionate about delivering robust data solutions while driving engineering best practices.This is a 6-month contract / freelance engagement with a strong possibility of transitioning into a permanent position, based onsite in Abu Dhabi.Key ResponsibilitiesDesign, develop, and optimize scalable ETL/ELT pipelines using Databricks, PySpark, Spark SQL, and Delta Lake.Build and maintain enterprise Lakehouse architectures capable of supporting large-scale analytics, reporting, and AI workloads.Integrate data from multiple enterprise systems, cloud services, APIs, databases, and streaming platforms into centralized data platforms.Optimize Spark workloads, cluster configurations, storage performance, and query execution to maximize efficiency and reduce processing costs.Implement and manage data governance, metadata management, and access controls using Unity Catalog and enterprise security best practices.Collaborate with Data Architects and business stakeholders to translate business requirements into scalable technical solutions.Design and implement streaming data pipelines using Kafka and Spark Structured Streaming where applicable.Develop automated CI/CD pipelines for data engineering workloads and Infrastructure as Code (IaC) using Terraform and modern DevOps practices.Ensure data quality, reliability, monitoring, lineage, and operational excellence across production data platforms.Participate in Agile ceremonies including sprint planning, backlog refinement, technical design reviews, and retrospectives while mentoring junior engineers and promoting engineering best practices.Required Skills & Experience5+ years of professional experience in Data Engineering, Big Data Engineering, or Cloud Data Platform development.3+ years of hands-on experience delivering enterprise solutions on the Databricks Lakehouse Platform.Strong expertise in Databricks, PySpark, Spark SQL, and Delta Lake.Proven experience designing and implementing scalable ETL/ELT pipelines and modern Lakehouse architectures.Hands-on experience with at least one major cloud platform (Microsoft Azure, AWS, or Google Cloud Platform).Strong understanding of distributed data processing, Spark optimization, partitioning, caching, and performance tuning.Experience implementing Unity Catalog, enterprise data governance, data security, and metadata management.Experience with CI/CD, DevOps methodologies, Git, and Infrastructure as Code using Terraform.Strong knowledge of data modelling, data warehousing, and enterprise data architecture principles.Experience with streaming technologies including Apache Kafka and Spark Structured Streaming.Experience implementing data quality frameworks, monitoring, and automated validation processes.Excellent analytical, problem-solving, communication, and stakeholder management skills.Experience working within Agile or Scrum delivery environments.Preferred ExperienceDatabricks Certified Data Engineer Professional certification.Apache Spark certifications or equivalent Big Data certifications.Experience supporting enterprise AI, machine learning, or advanced analytics platforms.Experience with Delta Live Tables, MLflow, Databricks Workflows, or Databricks Asset Bundles.Experience working within highly regulated industries such as Banking, Financial Services, Insurance, Government, or Telecommunications.Knowledge of enterprise security, compliance, and cloud governance frameworks.Experience mentoring engineering teams and establishing data engineering best practices.Engagement DetailsEngagement Type: Contract / FreelanceStart Date: (Immediate Preferred)Work Setup: Full Onsite (5 Days per Week)Location: Abu Dhabi, UAE