Automation Engineer - Databricks

Job Title

Job Type

Location

About Us:

Our mission at micro1 is to match the most talented people in the world with their dream jobs. If you are looking to be at the forefront of AI innovation and work with some of the fastest-growing companies in Silicon Valley, we invite you to apply for a role. By joining the micro1 community, your resume will become visible to top industry leaders, unlocking access to the best career opportunities on the market.

Job Summary:

We are seeking a detail-oriented and innovative Automation Engineer- Databricks to join our customer's team. In this critical role, you will design, develop, and execute automated tests to ensure the quality, reliability, and integrity of data within Databricks environments. If you are passionate about data quality, thrive in collaborative environments, and excel at both written and verbal communication, we'd love to meet you.

Key Responsibilities:

• Design, develop, and maintain robust automated test scripts using Python, Selenium, and SQL to validate data integrity within Databricks environments.
• Execute comprehensive data validation and verification activities to ensure accuracy and consistency across multiple systems, data warehouses, and data lakes.
• Create detailed and effective test plans and test cases based on technical requirements and business specifications.
• Integrate automated tests with CI/CD pipelines to facilitate seamless and efficient testing and deployment processes.
• Work collaboratively with data engineers, developers, and other stakeholders to gather data requirements and achieve comprehensive test coverage.
• Document test cases, results, and identified defects; communicate findings clearly to the team.
• Conduct performance testing to ensure data processing and retrieval meet established benchmarks.
• Provide mentorship and guidance to junior team members, promoting best practices in test automation and data validation.

Required Skills and Qualifications:

• Strong proficiency in Python, Selenium, and SQL for developing test automation solutions.
• Hands-on experience with Databricks, data warehouse, and data lake architectures.
• Proven expertise in automated testing of data pipelines, preferably with tools such as Apache Airflow, dbt Test, or similar.
• Proficient in integrating automated tests within CI/CD pipelines on cloud platforms (AWS, Azure preferred).
• Excellent written and verbal communication skills with the ability to translate technical concepts to diverse audiences.
• Bachelor’s degree in Computer Science, Information Technology, or a related discipline.
• Demonstrated problem-solving skills and a collaborative approach to teamwork.

Preferred Qualifications:

• Experience with implementing security and data protection measures in data-driven applications.
• Ability to integrate user-facing elements with server-side logic for seamless data experiences.
• Demonstrated passion for continuous improvement in test automation processes, tools, and methodologies.