D-TechWorks - Data Engineering Lead - ETL

Data Engineer Lead - Pune

Job Title :

Experience :

Location :

Responsibilities

• Lead and mentor a team of data engineers, providing technical guidance, setting best practices, and overseeing task execution for the migration project.
• Design, develop, and architect scalable ETL processes to extract, transform, and load petabytes of data from on-premises SQL
  

• Oversee the comprehensive analysis of existing SQL Server schemas, data types, stored procedures, and complex data models, defining strategies for their optimal conversion and refactoring for PostgreSQL.
• Establish and enforce rigorous data validation, quality, and integrity frameworks throughout the migration lifecycle, ensuring accuracy and consistency.
• Collaborate strategically with Database Administrators, application architects, business stakeholders, and security teams to define migration scope, requirements, and cutover plans.
• Lead the development and maintenance of advanced scripts (primarily Python) for automating large-scale migration tasks, complex data transformations, and reconciliation processes.
• Proactively identify, troubleshoot, and lead the resolution of complex data discrepancies, performance bottlenecks, and technical
  

• Define and maintain comprehensive documentation standards for migration strategies, data mapping, transformation rules, and
  

• Ensure data governance, security, and compliance standards are meticulously applied throughout the migration process, including data encryption and access controls within GCP.
• Implement Schema conversion or custom schema mapping strategy for SQL Server to PostgreSQL shift
• Refactor and translate complex stored procedures and T-SQL logic to PostgreSQL-compatible constructs while preserving functional equivalence.
• Develop and execute comprehensive data reconciliation strategies to ensure consistency and parity between legacy and migrated datasets post-cutover.
• Design fallback procedures and lead post-migration verification and support to ensure business continuity.
• Ensuring metadata cataloging and data lineage tracking using GCP-native or integrated tools.