You are excited about creating change because your skills can help the greater good of every customer, industry and community We are hiring a talented
Databricks developer
>to join our team
If youre excited to be part of a winning team, CirrusLabs is a great place to grow your career
Experience - 3-8 yearsLocation - Hyderabad
ETL/ELT pipelines using Databricks and Apache Spark, PySpark , Advanced SQL Skillset, Automation & DevOps Integration
A Pyspark and Databricks Developer with a good understanding of the entire ETL/Azure lifecycle with a background of data projects
Work youll do
1
Design and Build Data Pipelines
Develop robust, scalable ETL/ELT pipelines using Databricks and Apache Spark
Integrate data from on-premises databases, cloud storage, APIs, and third-party sources into the enterprise data lake
Automate both batch and real-time data ingestion to support enterprise reporting and advanced analytics (with specific experience in SAP HANA integration being highly valuable)
Enhance pipeline designs by leveraging Azure Data Factory (ADF), diverse connectors, and core cloud services to create seamless, end-to-end data flows
2
Collaboration & Migration
Collaborate with onshore teams, architects, fellow developers, and business analysts to align new data flows with overall enterprise data warehouse (EDW) migration strategies
Support the migration of legacy and on-premises systems (especially SAP HANA) to cloud-native services, including the use of Delta Lake on Databricks
Optimize data modeling for analytical workloads, including the development of star/snowflake schemas and dimensional models
3
Data Transformation & Preparation
Cleanse, transform, and join diverse datasets for analytics and machine learning use cases
Use PySpark (mandatory) and, optionally, Scala within Databricks notebooks and jobs for efficient data processing
Ensure high data quality through comprehensive validation, exception handling, and logging practices
4
Performance Tuning & Cost Optimization
Optimize Spark jobs for efficiency, cost, and scalability (utilizing partitioning, caching, and dynamic resource allocation)
Monitor Databricks workloads and resolve bottlenecks affecting compute, storage, or data skew
5
Data Governance & Security
Assist in enforcing security controls such as access management, data masking, and encryption
Support enterprise-wide data governance policies, focusing on metadata management, data lineage, and auditability
6
Advanced SQL Skillset
Build, optimize, and maintain SQL queries for ETL processes within Databricks
Create and manage data models and tables with Databricks SQL or Spark SQL
Write and automate high-performance SQL scripts for joining, filtering, aggregating, and analyzing large datasets
Document SQL logic and data flows; convert and optimize SAP HANA SQL code to Databricks-compatible SQL as part of migration
Collaborate with analysts and end-users to deliver accurate reporting and dashboarding
7
Automation & DevOps Integration
Develop reusable components and orchestrate workflows; integrate pipelines with DevOps and CI/CD practices
Utilize ADO (Azure DevOps) for version control and automated deployment of data assets and Databricks notebooks
8
Documentation & Best Practices
Document all pipeline logic, ETL job definitions, and key system processes with clarity and thoroughness
Champion coding standards, reusable code design, and ongoing knowledge sharing within the team
9
SAFe Agile Teamwork
Work closely with business analysts, product owners, and data stewards to understand evolving requirements and build jobs accordingly
Provide transparent updates and proactively escalate technical challenges in alignment with SAFe Agile processes
Qualifications
Required:
Computer Science University degree or equivalent work experience
At least 1
5-2 5 years for Aanalyst and 4-5 years of total exp
Experience in Databricks, Pyspark and ADF etc
Experience in Databricks end-to-end implementation
Expert in developing solutions and delivering the high-quality projects using Pyspark and Databricks
Proficiency with Azure Data Factory, Azure DevOps, and cloud-native data services
Experience integrating and migrating data from SAP HANA to cloud platforms (Good to have)
Advanced SQL skills for data modeling and migration
Experience implementing data security and governance policies
Familiarity with DevOps, CI/CD, and infrastructure-as-code concepts
Expert on data integration tools, such as Data Stage is good to have
Knowledge of basic Python programming - Preferable
Knowledge of Cloud based ETL services
Excellent organizational and communication skills
Must have strong interpersonal skills, and the ability to effectively work with others in teams
Strong problem-solving skills in identifying issues and resolution
Adeptness at learning new technology and tools
The ability to effectively manage multiple assignments and responsibilities in a fast-paced environment
Strong commitment to professional client service excellence
Should be a strong individual contributor
