Big Data Test Engineer

Role:

• Support Development, and maintain automated test frameworks, tools, and test cases for Data Engineering and Data Warehouse applications.
• Collaborate with cross-functional teams, including software developers, data engineers, and data analysts, to ensure comprehensive testing coverage and adherence to quality standards.
• Conduct thorough testing of data pipelines, ETL processes, and data transformations using Big Data technologies.
• Apply your knowledge of Data Warehouse/Data Lake methodologies and best practices to validate the accuracy, completeness, and performance of our data storage and retrieval systems.
• Identify, document, and track software defects, working closely with the development team to ensure timely resolution.
• Participate in code reviews, design discussions, and quality assurance meetings to provide valuable insights and contribute to the overall improvement of our software products.

Base Skill Requirements

• Bachelors or masters degree in computer science, Engineering, or a related field.
  5-9 years of experience in software testing and development, with a focus on data-intensive applications.
• Proven experience in testing data pipelines and ETL processes - Test planning, Test Environment planning, End to End testing, Performance testing.
• Solid programming skills in Python - proven automation effort to bring efficiency in the test cycles.
• Solid understanding of Data models and SQL.
• Must have experience with ETL (Extract, Transform, Load) processes and tools (Scheduling and Orchestration tools, ETL Design understanding)
• Good understanding of Big Data technologies like Spark, Hive, and Impala.
• Understanding of Data Warehouse methodologies, applications, and processes.
• Experience working in an Agile/Scrum environment, with a solid understanding of user stories, acceptance criteria, and sprint cycles.
  

Optional Technical:

• Experience with scripting languages like Bash or Shell.
• Experience working with large-scale datasets and distributed data processing frameworks (e.g., Hadoop, Spark).
• Familiarity with data integration tools like Apache NiFi is a plus.
• Excellent problem-solving and debugging skills, with a keen eye for detail.
  Strong communication and collaboration skills to work effectively in a team-oriented environment.
• Eagerness to learn and contribute to a growing team.