Lead Data Engineer

Role Overview: As the Team Lead of Data Engineering, you will be responsible for driving the design, development, and optimization of data engineering pipelines and services with a strong software engineering mindset. Your role will also involve fostering growth and collaboration within the team, mentoring junior colleagues, and refining processes to maintain excellence. If you are eager to have a lasting impact both technically and as a team leader, we are excited to meet you. Key Responsibilities: - Utilize very strong software engineering principles to write maintainable, scalable code, primarily in Python (experience with Rust or C# is a plus). - Perform data processing and manipulation using Polars and Pandas, scaled via Apache Ray. - Make efficient use of various open-source data storage technologies such as S3, Apache Iceberg, DuckDB, Clickhouse, PostgreSQL, and others. - Implement containerization and orchestration tools like Docker and Kubernetes, with automated deployments in cloud environments (AWS, Azure, GCP). - Manage complex workflow orchestration using open-source tools such as Apache Airflow. - Work with big data and distributed processing technologies, including Hadoop, Spark, and Kafka. Qualifications Required: - Strong educational background in Mathematics, Computer Science, Engineering, Physics, or a related field. - 8+ years of proven experience in Data Engineering, including data warehousing, ETL, automation, cloud technologies, or Software Engineering in data-related areas. - Experience leading and mentoring junior team members, fostering skill development, collaboration, and process improvements. - Cross-functional collaboration experience, working with software engineers, data scientists, and ML engineers to deliver high-impact solutions.,