Data Engineer

As a Data Engineer II in our team, you will be responsible for managing the deprecation of migrated workflows and ensuring the seamless migration of workflows into new systems. Your expertise in building and maintaining scalable data pipelines, both on-premises and on the cloud, will be crucial. You should have a deep understanding of input and output data sources, upstream downstream dependencies, and data quality assurance. Proficiency in tools like Git, Apache Airflow, Apache Spark, SQL, data migration, and data validation is essential for this role. Your key responsibilities will include: Workflow Deprecation: - Evaluate current workflows" dependencies and consumption for deprecation. - Identify, mark, and communicate deprecated workflows using tools and best practices. Data Migration: - Plan and execute data migration tasks ensuring accuracy and completeness. - Implement strategies for accelerating data migration pace and ensuring data readiness. Data Validation: - Define and implement data validation rules for accuracy and reliability. - Monitor data quality using validation solutions and anomaly detection methods. Workflow Management: - Schedule, monitor, and automate data workflows using Apache Airflow. - Develop and manage Directed Acyclic Graphs (DAGs) in Airflow for complex data processing tasks. Data Processing: - Develop and maintain data processing scripts using SQL and Apache Spark. - Optimize data processing for performance and efficiency. Version Control: - Collaborate using Git for version control and manage the codebase effectively. - Ensure code quality and repository management best practices. Continuous Improvement: - Stay updated with the latest data engineering technologies. - Enhance performance and reliability by improving and refactoring data pipelines and processes. Skills and Qualifications: - Bachelor's degree in Computer Science, Engineering, or a related field. - Proficient in Git, SQL, and database technologies. - Experience in Apache Airflow and Apache Spark for data processing. - Knowledge of data migration, validation techniques, governance, and security. - Strong problem-solving skills and ability to work independently and in a team. - Excellent communication skills to collaborate with a global team in a high-performing environment.,