Prismberry Technology - Databricks Engineer

Job Summary

Responsibilities

• Design, develop, and manage scalable and reliable data pipelines using Databricks, PySpark, and SQL.
• Collaborate with data architects, analysts, and business stakeholders to understand data requirements and deliver efficient solutions.
• Implement CI/CD pipelines for data workflows, ensuring smooth deployment and integration
  

• Optimize and troubleshoot performance issues in existing pipelines.
• Develop and maintain unit test cases to ensure data accuracy and reliability.
• Work with MongoDB, leveraging its aggregation framework for efficient data querying and Document data flows, system architecture, and best practices for development and Skills and Qualifications :
• 5 - 8 years of professional experience in Data Engineering or related roles.
• Hands-on experience with the Databricks platform (including clusters, notebooks, jobs, and data pipelines).
• Strong programming skills in Python and PySpark.
• Proficiency in SQL for data querying, transformation, and validation.
• Experience with CI/CD practices and tools (e.g., Git, Azure DevOps, Jenkins).
• Solid experience in implementing and maintaining unit test cases for data pipelines.
• Good understanding and practical experience with MongoDB, including aggregation queries.
• Strong problem-solving skills and the ability to work independently in a fast-paced