Lead Data Engineer

Date of Posting - May 12th, 2025

Position Summary

We are looking for a Lead Data Engineer who has a strong background in designing, developing, and maintaining complex data systems. The ideal candidate will have experience working with large datasets, real-time data processing, and cloud technologies. You will work closely with data scientists, analysts, and software engineers to develop end-to-end data solutions that support business intelligence, analytics, and reporting. We are looking for advance data lake implementation capabilities like MS fabrics and AWS Lakehouse.

Key Responsibilities

• Data Pipeline Development: Design, build, and maintain scalable data pipelines that efficiently ingest, process, and store data from various sources, ensuring high data quality and accessibility.
• Data Architecture & Modeling: Develop and implement data architecture strategies to support data warehousing, data lakes, and other big data initiatives. Design and optimize database structures and data models.
• ETL/ELT Processes: Lead the development of ETL/ELT workflows to transform raw data into actionable insights. Optimize existing workflows for performance and reliability.
• Collaboration with Cross-Functional Teams: Work closely with Data Scientists, Analysts, and Product Teams to understand business requirements and provide solutions that enable data-driven decision-making.
• Cloud Technologies: Implement and manage data solutions in cloud platforms (AWS, Azure, GCP), including leveraging services like S3, Redshift, BigQuery, and others for scalable data storage and processing.
• Data Quality & Governance: Ensure that data pipelines are compliant with data governance and quality standards. Implement monitoring and alerting systems to track data accuracy and availability.
• Optimization & Performance Tuning: Regularly optimize database performance, queries, and data pipelines for speed and cost-effectiveness. Troubleshoot performance issues and suggest improvements.
• Documentation & Reporting: Document data engineering processes, tools, and methodologies to maintain a clear and organized framework for team collaboration and knowledge sharing.
• Mentorship & Leadership: Mentor and provide technical guidance to junior engineers and other team members. Lead code reviews and help shape best practices for the engineering team.

Skills and Qualifications

• Experience: 8+ years of experience in data engineering, data pipeline development, and data management in large-scale environments.
• Programming Languages: Expertise in Python, SQL, and familiarity with other programming languages like Java, Scala, or Go.
• Data Technologies: Strong experience with big data technologies such as Hadoop, Spark, Kafka, and related frameworks.
• Cloud Platforms: Proficient in cloud data services such as AWS, Google Cloud, or Azure (preferably AWS).
• Database Management: Solid experience with relational and NoSQL databases (e.g., PostgreSQL, MySQL, MongoDB, Cassandra).
• Data Warehousing & BI: Knowledge of data warehousing concepts and tools like Redshift, Snowflake, or BigQuery, and experience working with BI tools like Tableau, Looker, or Power BI.
• Data Modeling & Architecture: Strong understanding of data modeling, schema design, and architectural best practices for scalable data solutions.
• Version Control & CI/CD: Proficiency in version control tools (Git) and experience with CI/CD pipelines to ensure efficient and reliable deployments.
• Problem-Solving: Excellent troubleshooting skills and the ability to solve complex data engineering challenges.
• Communication: Strong verbal and written communication skills to interact with both technical and non-technical stakeholders.

Technical Expertise

• Programming Languages: Proficiency in Python, Java, or Scala.
• Big Data Technologies: Hands-on experience with tools like Hadoop, Spark, and Kafka.
• Cloud Platforms: Proficiency with AWS (e.g., S3, Redshift, Glue), Azure (e.g., Synapse, Data Factory), or GCP (e.g., BigQuery, Dataflow).
• Databases: Expertise in SQL Server, MySQL, PostgreSQL.
• Reporting: Expertise in Power BI, SSRS.
• Tools: Familiarity with orchestration tools like Apache Airflow and dbt.
• Technology – Understanding the latest trends in Data technology and updated on offerings like Databricks, MS Fabrics, AWS Sagemaker Lakehouse

Soft Skills

• Strong problem-solving and analytical abilities.
• Excellent communication and collaboration skills.
• Ability to mentor junior engineers and lead data engineering initiatives.

Send us your resume at:

careers@hexaviewtech.com