Lead Engineer - Data Engg & AI

You will be responsible for designing, developing, and maintaining data pipelines and ETL workflows for processing large-scale structured/unstructured data. Your expertise in AWS Data Services (S3, Workflows, Databricks, SQL), big data processing, real-time analytics, and cloud data integration is crucial for this role. Additionally, your experience in Team Leading will be valuable. Your key responsibilities will include redesigning optimized and scalable ETL using Spark, Python, SQL, UDF, implementing ETL/ELT Databricks workflows for structured data processing, and creating Data quality checks using Unity Catalog. You will also be expected to drive daily status calls, sprint planning meetings, and ensure the security, quality, and compliance of data pipelines. Collaborating with data architects and analysts to meet business requirements is also a key aspect of this role. To qualify for this position, you should have at least 8 years of experience in data engineering, with a minimum of 2 years working on AWS services. Hands-on experience with tools like S3, Databricks, or Workflows is essential. Knowledge in Adverity, experience with ticketing tools like Asana or JIRA, and data analyzing skills are considered advantageous. Strong SQL and data processing skills (e.g., PySpark, Python) are required, along with experience in data cataloging, lineage, and governance frameworks. Your contribution to CI/CD integration, observability, and documentation, as well as your ability to quickly analyze issues and drive collaboration within the team, will be instrumental in achieving the goals of the organization.,