Senior Data Engineer

About the Role

We are seeking a highly skilled Senior Data Engineer with hands-on experience in Databricks, Python, and Apache Spark to join our data engineering team. You will be responsible for building scalable data pipelines, optimizing big data processing workflows, and enabling advanced analytics and machine learning initiatives.

Key Responsibilities

Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark.

Write efficient, reusable, and modular Python code for data processing and transformation.

Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver high-quality solutions.

Optimize Spark jobs for performance and cost-efficiency in cloud environments.

Implement data quality checks, monitoring, and alerting mechanisms.

Work with structured and unstructured data from various sources including APIs, databases, and cloud storage.

Ensure data security, governance, and compliance with organizational standards.

Document data workflows, architecture, and best practices.

Required Qualifications

Bachelors or master's degree in computer science, Engineering, or related field.

6+ years of experience in data engineering with strong expertise in Python and Apache Spark.

3+ years of hands-on experience with Databricks in a production environment.

Proficiency in building and optimizing ETL/ELT pipelines.

Experience with cloud platforms (e.g., AWS, Azure, GCP) and cloud-native data tools.

Strong understanding of distributed computing and big data processing.

Excellent problem-solving and communication skills.