Pyspark Developer

About Birlasoft:

Birlasoft is a global technology company enabling “next-generation” digital transformation through expertise in Cloud, AI, Data, and enterprise solutions. Combining industry proficiency with advanced digital capabilities, it helps businesses accelerate change with speed, scale, and purpose, delivering “future-ready” solutions that enhance agility, resilience, and customer experience. Part of the CKA Birla Group and led by Chairman Mrs. Amita Birla, Birlasoft’s nearly 12,000 professionals drive innovation while building a diverse, inclusive, and learning-oriented culture. With a strong focus on sustainability and long-term value creation, Birlasoft transforms enterprises and communities, earning its reputation as a trusted partner and one of the best places to work

About the Job :

Title:

Job Location:

Educational Background:

Key Responsibilities:

• Develop and manage data pipelines using PySpark on Databricks.
• Implement ETL/ELT processes to process structured and unstructured data at scale.
• Optimize data pipelines for performance, scalability, and cost-efficiency in Databricks.
• Experience in Perform Design, Development & Deployment using Azure Services (Data Factory, Databricks, PySpark, SQL)
• Develop and maintain scalable data pipelines and build new Data Source integrations to support increasing data volume and complexity.
• Leverage the Databricks Lakehouse architecture for advanced analytics and machine learning workflows.
• Manage Delta Lake for ACID transactions and data versioning.
• Develop notebooks and workflows for end-to-end data solutions.
• Deploy and manage Databricks on Azure (e.g., Azure Databricks).
• Use Databricks Jobs, Clusters, and Workflows to orchestrate data pipelines.
• Optimize resource utilization and troubleshoot performance issues on the Databricks platform.
• Deploy and manage Databricks on Azure (e.g., Azure Databricks).
• Use Databricks Jobs, Clusters, and Workflows to orchestrate data pipelines.
• Optimize resource utilization and troubleshoot performance issues on the Databricks platform.
• Work closely with data scientists, data analysts, and IT teams to deliver robust data solutions.
• Document Databricks workflows, configurations, and best practices for internal use.

Skills Required:

• 4+ years of experience in data engineering or distributed systems development.
• Strong programming skills in Python and PySpark.
• Hands-on experience with Databricks and its ecosystem, including Delta Lake and Databricks SQL.
• Knowledge of big data frameworks like Hadoop, Spark, and Kafka.
• 4+ years of experience in data engineering or distributed systems development.
• Strong programming skills in Python and PySpark.
• Hands-on experience with Databricks and its ecosystem, including Delta Lake and Databricks SQL.
• Knowledge of big data frameworks like Hadoop, Spark, and Kafka.
• Expertise in deploying Databricks solutions Azure (e.g., Data Lake, Synapse).
• Knowledge of Kubernetes for managing containerized workloads is advantageous
• Experience with both SQL (e.g., PostgreSQL, SQL Server) and NoSQL databases (e.g., MongoDB, Cosmos DB).