Sr. Data Engineer

Your Role and Impact

Your Contribution

• Analyze the existing Hive Catalog metadata, schema, and security configurations.
• Design and execute a robust migration plan to Unity Catalog with minimal disruption and data integrity assurance.
• Collaborate with Data Governance, Security, and Cloud Infrastructure teams to implement access controls and policies leveraging Azure Active Directory (AAD).
• Develop automation scripts and tools to support migration, validation, and ongoing management.
• Troubleshoot migration challenges and provide post-migration support.
• Document migration processes and train stakeholders on Unity Catalog capabilities.
• Integrate Unity Catalog with Azure native services such as Azure Data Lake Storage Gen2, Azure Key Vault, and Azure Active Directory for security and identity management.
• Optimize Azure resource utilization during migration and production workloads.
• Keep current with Azure Databricks Unity Catalog enhancements and Azure cloud best practices.
• trong knowledge of metadata management, data governance frameworks, and data cataloging.
• Proficient in SQL, Python, and scripting for automation.
• Hands-on experience with Azure Databricks, Apache Spark, and Azure cloud services including Azure Data Lake Storage Gen2, Azure Key Vault, and Azure Active Directory.
• In-depth understanding of Azure cloud infrastructure: compute (VMs, Azure Databricks clusters), storage, networking, and security components.
• Experience integrating data catalog solutions with Azure identity and access management (Azure AD, RBAC).
• Strong grasp of data security, IAM policies, and access control in Azure environments.
• Excellent analytical, problem-solving, and communication skills.