Responsibilities:

Design, develop, and deploy products and services produced by the team.
Continually improve technologies used by the team to deliver products and services to our customers. Implement all application requirements, including functional, security, integration, performance, quality, and operations requirements. Work closely with the Product Owners and Architects to develop Azure Data Platforms Work closely with Team and provide guidance and assistance on technical issues Knowledge of handling un-structured, semi structured and structured data Serverless and cloud technologies: Azure Functions, AWS Lambda. Strong programming knowledge. Asses and understand data from a variety of corporate data sources and perform required transformations. Data modelling capabilities including designing effective BI data models in line with BUS-Matrix.

Required Skills:

Minimum 3+ years and Max of 6 years of experience in managing, designing, and maintaining large scale data solutions
Data Engineering with cloud or on-premise using Python, Spark, Scala Experience in working with Hadoop eco system. Experience with Pipeline design and development (Data Ingestion, Transformation, Schematization and Orchestration) Hands on experience on ELT/ ETL pipelines Experience working with cloud Storage Excellent in data ingestion – Batch and real time processing Exposure to DevOps CI/CD pipeline development and deployment process Good knowledge of scripting languages for custom API and data transformation as needed Good programming knowledge on Python and Spark Experience in writing complex queries using SQL Good knowledge in writing Unix Shell scripts

Knowledge on Below is a plus:

Azure Data Factory, Azure SQL DW/Synapse, Databricks, PySpark
Hands on experience in Azure Databricks pipeline development Hands on knowledge in AWS Cloud and EMR

Job Features