Sr. Data Engineer - Databricks

Description:We are seeking a highly skilled and experienced Sr. Data Engineer to join our team. Theideal candidate will have a strong background in data engineering, with a focus on working withDatabricks, PySpark, Scala-Spark, and advanced SQL. This role requires hands-on experience inimplementing or migrating projects to Unity Catalog, optimizing performance on DatabricksSpark, and orchestrating workflows using various tools.

Key Responsibilities

• Data engineering and analytics project delivery experience – Min. 4+ years
• Minimum one project done in past of Databricks Migration (Ex. Hadoop to Databricks, Teradata to Databricks, Oracle to Databricks, Talend to Databricks etc)
• Hands on with Advanced SQL and Pyspark and/or Scala Spark
• Design, develop, and optimize data pipelines and ETL processes using Databricks and ApacheSpark.
• Implement and optimize performance on Databricks Spark, ensuring efficient data processingand management.
• Develop and validate data formulation and data delivery for Big Data projects.
• Collaborate with cross-functional teams to define, design, and implement data solutions thatmeet business requirements.
• Conduct performance tuning and optimization of complex queries and data models.
• Manage and orchestrate data workflows using tools such as Databricks Workflow, Azure DataFactory (ADF), Apache Airflow, and/or AWS Glue.
• Maintain and ensure data security, quality, and governance throughout the data lifecycle

Technical Skills

• Extensive experience with PySpark and Scala-Spark.
• Advanced SQL skills for complex data manipulation and querying.
• Proven experience in performance optimization on Databricks Spark across at least threeprojects.
• Hands-on experience with data formulation and data delivery validation in Big Data projects.
• Experience in data orchestration using at least two of the following: Databricks Workflow,Azure Data Factory (ADF), Apache Airflow, AWS Glue.

Preferred Qualifications

• Experience with cloud platforms such as AWS, Azure, or Google Cloud Platform.
• Familiarity with data governance and data security best practices.
• Experience with other Big Data technologies and frameworks is a plus