Databricks Data Engineer

Key Responsibilities

· Data Pipeline Development:Design, build, and maintain scalable data pipelines and architectures using Databricks Lakehouse Platform (Delta Lake, Spark, Spark SQL, Auto Loader, and MLflow)Implement complex ETL/ELT solutions across bronze, silver, and gold layers.Build reusable frameworks for ingestion, transformation, and monitoring.Databricks Platform Engineering:Develop and optimize Databricks notebooks, workflows, and job clusters.Implement best practices for Delta Lake: optimization, compaction, schema enforcement, time travel.Troubleshoot performance issues and fine-tune Spark jobs for efficiency.Cloud Integration (Azure / AWS):Integrate Databricks with cloud storage (ADLS/S3).Implement security, secrets, and networking configurations.Data Quality, Observability & Governance:Build data validation frameworks and error-handling mechanisms.Work with Unity Catalog for cataloging, governance, and access control.Ensure data compliance, security standards, and audit processes.DevOps & CI/CD:Maintain Git-based development workflows.Develop CI/CD pipelines for Databricks workflows using Git, and orchestration tools like Apache Airflow or Databricks Workflows.Collaboration, Leadership & Guidance:Collaborate with architects, analysts, and business stakeholders to refine requirements.Mentor junior engineers and conduct code reviews.Participate in solution design and architecture discussions.Skills and Qualifications: · Bachelor’s / Master’s degree in computer science, computer engineering, or relevant field.· Overall 3+ years of experience, at least 2 years of experience in Databricks Data Engineering.· Strong, hands-on experience with the Databricks workspace, Delta Lake, Delta Live Tables (DLT), and Unity Catalog.· Proficiency in Apache Spark, PySpark, and distributed computing principles.· Advanced proficiency in Python and SQL.· Hands-on experience with cloud environments such as AWS, Azure.· Experience with Git, CI/CD pipelines, and infrastructure-as-code tools (Terraform, CloudFormation, or Databricks Asset Bundles).· Experience with ETL processes: Extracting, transforming, and loading data.· Strong analytical and problem-solving skills.· Good communication and collaboration skills.· Experience with data security and governance principles.· Understanding of data visualization techniques.