Lead Azure Data Engineer

As a Data Engineer, your primary responsibility will be to build and manage systems that collect, store, clean, and deliver data for various teams within the company, such as analysts, data scientists, and business users. Working with large datasets, you will focus on improving data quality, optimizing performance, and ensuring others can access the necessary data reliably and securely. Your key responsibilities will include building and maintaining data pipelines to pull data from diverse sources, processing it, and storing it for easy access. You will assemble large, complex datasets that align with both business and technical requirements, automate and optimize processes to enhance efficiency, and create tools to assist data teams in uncovering valuable insights like customer behavior and performance metrics. Collaboration with different teams, including product, design, and executive teams, will be essential to address data-related challenges, while ensuring the security and protection of sensitive data. Supporting data scientists by developing tools and systems to streamline their work processes is also part of your role, along with continuously seeking improvements in how data is utilized within the business. To excel in this role, you must possess expertise in data architecture and modeling to design scalable systems for easy analysis, proficiency in building and managing data pipelines, and ensuring data cleanliness for usability. Strong collaboration skills, the ability to document processes clearly, and basic knowledge of machine learning concepts are crucial. Proficiency in technical tools such as Python for writing clean, reusable code, Spark (PySpark and SparkSQL), SQL for data querying and manipulation, as well as familiarity with Git, GitHub, and JIRA for version control and task tracking is essential. Effective communication skills are necessary to convey data concepts to non-technical individuals. Additionally, having experience with Azure Cloud tools like Databricks or Synapse Analytics, familiarity with Azure Data Management tools such as Azure SQL Data Warehouse or Cosmos DB, and hands-on experience in deploying machine learning models in real-world applications are considered advantageous for this role.,