Lead Data Engineer

You will be part of a team responsible for developing a next-generation Data Analytics Engine that converts raw market and historical data into actionable insights for the electronics supply chain industry. This platform processes high-volume data from suppliers, parts, and trends to provide real-time insights and ML-driven applications. We are seeking an experienced Lead or Staff Data Engineer to assist in shaping and expanding our core data infrastructure. The ideal candidate should have a strong background in designing and implementing scalable ETL pipelines and real-time data systems in AWS and open-source environments such as Airflow, Spark, and Kafka. This role involves taking technical ownership, providing leadership, improving our architecture, enforcing best practices, and mentoring junior engineers. Your responsibilities will include designing, implementing, and optimizing scalable ETL pipelines using AWS-native tools, migrating existing pipelines to open-source orchestration tools, leading data lake and data warehouse architecture design, managing CI/CD workflows, implementing data validation and quality checks, contributing to Infrastructure as Code, and offering technical mentorship and guidance on architectural decisions. To qualify for this role, you should have at least 8 years of experience as a Data Engineer or similar role with production ownership, expertise in AWS tools, deep knowledge of open-source data stack, strong Python programming skills, expert-level SQL proficiency, experience with CI/CD tools, familiarity with Infrastructure as Code, and the ability to mentor engineers and drive architectural decisions. Preferred qualifications include a background in ML/AI pipelines, experience with serverless technologies and containerized deployments, and familiarity with data observability tools and alerting systems. A Bachelor's or Master's degree in Computer Science, Data Engineering, Information Systems, or a related field is preferred. In return, you will have the opportunity to work on impactful supply chain intelligence problems, receive mentorship from experienced engineers and AI product leads, work in a flexible and startup-friendly environment, and enjoy competitive compensation with opportunities for career growth.,