Data Engineer

The Core Data Engineer – DQ Framework & Validation Expert will focus on building and managing robust data quality (DQ) engines across staging and curated data layers. This role requires strong expertise in data validation frameworks, CI/CD automation, and integration with AI-driven QA validators to ensure high-quality datasets. The engineer will play a critical role in defining validation KPIs, developing Deequ/Great Expectations-based rules, and supporting data lifecycle and logging strategies.

Responsibilities

• Understand the values and vision of the organization.
• Protect the Intellectual Property.
• Adhere to all the policies and procedures.
• Build and manage Deequ/Great Expectations-based data validation rule sets for tables and schemas.
• Define and track validation KPIs (e.g., null percentages, threshold checks, data type conformity).
• Integrate data quality checks with AWS Glue pipelines and monitor QA outcomes.
• Automate DQ checks and notifications using AWS Step Functions.
• Feed validation metrics to the Gen AI QA Validator and enable summary generation.
• Develop QA audit table generation logic to monitor dataset health.
• Support intermediate dataset lifecycle management and logging strategies.
• Collaborate with cross-functional teams to ensure PII, KPI, and derived field-level tagging logic.
• Implement Bitbucket CI/CD pipelines for automated DQ rule validation.

Job

• Strong hands-on expertise in PySpark and AWS Glue 3.0+.
• Experience with AWS Glue Catalog & Tags for data classification.
• Proficiency in AWS Step Functions for DQ workflows.
• Expertise in Deequ or Great Expectations for building validation rules.
• Proficiency in Bitbucket CI/CD pipelines for data validation automation.
• Strong SQL skills with experience in Athena for validation dashboards.

Personal

• Excellent problem-solving abilities and attention to detail.
• Strong interpersonal and collaboration skills with cross-functional teams.
• A proactive mindset, capable of driving initiatives independently.
• Ability to manage priorities in fast-paced and evolving environments.

Preferred Skills

Job

• Prior experience with Gen AI-driven QA validation pipelines.
• Knowledge of data lifecycle logging strategies.
• Hands-on experience in creating PII and KPI tagging mechanisms.
• Experience with data quality dashboards and audit reporting.

Personal

• Demonstrate proactive thinking
• Strong communication and collaboration skills.
• Should have strong interpersonal relations, expert business acumen, and mentoring skills
• Strong problem-solving skills with attention to detail.
• Have the ability to work under stringent deadlines and demanding client conditions.
• Strong analytical and problem-solving skills.
• Ability to work independently and as part of a team.

Other Relevant Information

• Bachelor’s degree in Computer Science, Information Technology, or a related field.
• 4-6 years of experience in data engineering and data quality frameworks

  .

• This role offers the flexibility of working remotely.

Apply by sharing your resume/CV at shahrozk@leewayhertz.com

LeewayHertz is an equal opportunity employer and does not discriminate based on race, color, religion, sex, age, disability, national origin, sexual orientation, gender identity, or any other protected status. We encourage a diverse range of applicants.