Azure Data Engineer

As a Senior Software Engineer at our organization, you will play a crucial role in leading and contributing to our healthcare data engineering initiatives. Your primary focus will be on building scalable, real-time data pipelines and processing large-scale healthcare datasets in a secure and compliant cloud environment. You will design and develop scalable real-time data streaming solutions using Apache Kafka and Python. Additionally, you will architect and implement ETL/ELT pipelines using Azure Databricks for both structured and unstructured healthcare data. It will be your responsibility to optimize and maintain Kafka applications, Python scripts, and Databricks workflows to ensure performance and reliability. Ensuring data integrity, security, and compliance with healthcare standards such as HIPAA and HITRUST will be a key aspect of your role. You will collaborate with data scientists, analysts, and business stakeholders to gather requirements and translate them into robust data solutions. Furthermore, you will mentor junior engineers, perform code reviews, and promote engineering best practices. To excel in this role, you should have 4+ years of hands-on experience in data engineering roles. Proficiency in Kafka (including Kafka Streams, Kafka Connect, Schema Registry) and Python for data processing and automation is essential. Experience with Azure Databricks (or readiness to ramp up quickly) and a solid understanding of cloud platforms, with a preference for Azure (AWS/GCP is a plus) are required. You should also possess strong knowledge of SQL and NoSQL databases, data modeling for large-scale systems, familiarity with containerization tools like Docker, and orchestration using Kubernetes. Exposure to CI/CD pipelines for data applications and prior experience with healthcare datasets (EHR, HL7, FHIR, claims data) is highly desirable. In addition to technical skills, excellent problem-solving abilities, a proactive mindset, and strong communication and interpersonal skills to work in cross-functional teams will be valuable assets in this role. Stay current with evolving technologies in cloud, big data, and healthcare data standards to contribute effectively to our projects. If you are passionate about data engineering, possess the required skills and qualifications, and are eager to make a significant impact in the healthcare industry, we look forward to receiving your application.,