Data Engineer – Azure Databricks & Azure Data Factory

We are looking for a skilled and hands-on Data Engineer to join the Data & Analytics team and contribute to building, enhancing, and supporting scalable enterprise data solutions on Microsoft Azure.In this role, you will be responsible for developing and maintaining data pipelines, integrating data from SAP and non-SAP source systems, and implementing data transformations using Azure Databricks.You will work extensively with Azure Data Factory, Databricks, PySpark, Python, and SQL to deliver reliable, high-quality, and well-structured datasets that support enterprise reporting, business intelligence, and advanced analytics.The ideal candidate combines strong technical expertise with practical experience in enterprise-scale data engineering, data integration, performance optimization, and production support.Key Responsibilities1. Data Ingestion & IntegrationDesign, develop, and maintain scalable data ingestion pipelines using Azure Data Factory (ADF).Integrate data from multiple sources, including SAP systems, relational databases, REST APIs, flat files, and cloud applications.Implement incremental data loading and Change Data Capture (CDC) mechanisms.Monitor, troubleshoot, and resolve data integration and pipeline-related issues.2. Data Transformation & ProcessingDevelop and maintain data transformation workflows using PySpark and SQL within Azure Databricks.Implement and manage the Medallion Architecture (Bronze, Silver, and Gold layers).Perform data cleansing, validation, enrichment, and transformation to meet business requirements.Optimize data processing jobs for performance, scalability, and cost efficiency.Work with Delta Lake to manage reliable and efficient data processing.3. Data Warehousing & ModelingSupport the implementation of enterprise data models and Common Data Models (CDM).Develop curated and structured datasets for reporting and analytics.Apply data warehousing and dimensional modeling principles.Ensure data consistency, accuracy, and reliability across multiple business domains.4. Analytics EnablementDeliver trusted and optimized datasets for Power BI reporting and self-service analytics.Collaborate with Business Analysts, BI Developers, and Data Architects to understand business and technical requirements.Support improvements in reporting and dashboard performance.5. Data Governance & QualityImplement data quality checks, validation rules, and monitoring frameworks.Ensure compliance with data governance standards and security policies.Document data flows, business rules, and technical processes.6. Production Support & OperationsMonitor scheduled jobs, workflows, and data pipelines.Investigate and resolve production incidents and data processing failures.Perform root cause analysis and implement corrective actions.Participate in deployment, release management, and CI/CD activities.Required Technical Skills & ExperienceStrong hands-on experience with Azure Databricks (4+ years).Practical experience with Azure Data Factory (2+ years).Strong programming experience in PySpark (2+ years) and Python (2+ years).Strong SQL development skills, including complex queries and data transformations (3+ years).Good understanding of Azure Data Lake Storage (ADLS) and Delta Lake.Solid understanding of ETL/ELT processes and enterprise data warehousing.Experience implementing Medallion Architecture (Bronze, Silver, Gold).Knowledge of incremental data loading, data modeling, and data quality management.Experience building and maintaining enterprise-scale cloud data pipelines.Familiarity with Git, Azure DevOps, and CI/CD concepts.Understanding of Power BI datasets and reporting requirements.Preferred QualificationsExperience integrating data from SAP ECC or SAP S/4HANA.Experience with REST API integrations.Familiarity with event-driven architectures.Knowledge of Unity Catalog and Microsoft Fabric.Experience in the Retail or Automotive industry.Experience supporting enterprise analytics and BI environments.EducationBachelor's degree in Computer Science, Information Technology, Engineering, or a related technical field.What Success Looks LikeBuilding reliable, scalable, and efficient data pipelines.Delivering high-quality, trusted datasets for business reporting and analytics.Reducing production incidents and improving issue resolution.Improving the performance and cost efficiency of data processing workloads.Collaborating effectively with business, analytics, and technology stakeholders.