Azure Data Engineer - ETL/Data Pipeline

As a Senior Software Engineer (SSE) specializing in Kafka, Python, and Azure Databricks, you will play a crucial role in leading and contributing to healthcare data engineering projects. Your primary responsibility will involve building scalable real-time data pipelines and processing large healthcare datasets within a secure and compliant cloud environment. Your key responsibilities will include designing and developing scalable real-time data streaming solutions using Apache Kafka and Python. You will also be tasked with architecting and implementing ETL/ELT pipelines utilizing Azure Databricks for both structured and unstructured healthcare data. Additionally, you will optimize and maintain Kafka applications, Python scripts, and Databricks workflows to ensure high performance and reliability. Ensuring data integrity, security, and compliance with healthcare standards such as HIPAA and HITRUST will be a critical aspect of your role. Collaboration with data scientists, analysts, and business stakeholders to gather requirements and translate them into robust data solutions is also expected. You will be responsible for mentoring junior engineers, conducting code reviews, and advocating for engineering best practices. Remaining updated with the latest technologies in cloud computing, big data, and healthcare data standards will be essential. Contribution to the development of CI/CD pipelines and containerized environments, including Docker and Kubernetes, is part of the role's expectations. To excel in this position, you should possess at least 4 years of hands-on experience in data engineering roles. Proficiency in Kafka, Python, Azure Databricks, SQL, NoSQL databases, and data modeling for large-scale systems is required. Exposure to containerization tools like Docker, orchestration using Kubernetes, and CI/CD pipelines for data applications is advantageous. Prior experience with healthcare datasets such as EHR, HL7, FHIR, and claims data is highly desirable. In addition to technical skills, you should have excellent problem-solving abilities, a proactive mindset, and strong communication and interpersonal skills to collaborate effectively in cross-functional teams.,