Lead Data Engineer

As an experienced Lead Data Engineer, you will be responsible for designing, building, and optimizing cloud-native data platforms on AWS. Your main focus will be leading a team of Data Engineers to develop scalable data pipelines, modern data architectures, and enterprise-grade data solutions for BFSI customers. **Key Responsibilities:** - Lead, mentor, and manage a team of Data Engineers to deliver high-quality outcomes. - Own project planning, task delegation, sprint delivery, and quality assurance. - Collaborate with cross-functional teams including data scientists, analysts, architects, and product teams. **Data Engineering & Architecture:** - Architect, design, and implement scalable data pipelines using AWS, PySpark, and Airflow. - Build and maintain ETL/ELT processes for ingestion, transformation, and publishing of structured/unstructured data. - Develop scalable data models and cloud-native architectures aligned with BFSI requirements. - Optimize data processing (Spark), data storage (S3, Iceberg/Hudi/Delta), and query performance. **Cloud & Platform Engineering:** - Work across AWS services such as S3, Glue, EMR, Lambda, Redshift, Kinesis, RDS, CloudWatch, IAM. - Ensure cost-efficiency, reliability, and scalability of the cloud data environment. - Implement automation for CI/CD, data deployments, and pipeline orchestration. **Governance, Security & Compliance:** - Enforce data governance standards, lineage tracking, and data quality frameworks. - Implement data security, encryption, RBAC, and compliance aligned with BFSI/regulatory norms. - Ensure high availability, monitoring, and proactive issue resolution. **Required Skills & Experience:** - 6+ years of hands-on experience in data engineering. - Strong expertise in PySpark for distributed data processing. - Experience building and orchestrating pipelines using Airflow. - Strong AWS experience with focus on data services. - Advanced SQL skills and experience in OLTP, OLAP, Data Lakes, and Data Warehouses. - Experience with Hudi / Iceberg / Delta Lake. - Proficiency in Python for ETL, utilities, and API integrations. - Understanding of big data technologies (Spark, Hadoop ecosystem). **Preferred Skills:** - Experience in BFSI data ecosystems. - Exposure to Docker, Kubernetes, and containerized data applications. - API development experience (preferably Python/FastAPI/Flask). - Experience working with data scientists and BI teams. **Behavioral & Leadership Skills:** - Strong ownership mindset and accountability. - Excellent problem solving and debugging skills. - Ability to mentor junior team members and lead by example. - Strong communication and stakeholder management. - Ability to work in a fast-paced, entrepreneurial environment. **Education:** - Bachelors or Masters degree in Computer Science, Engineering, or related field.,