Data Engineer

About us: We are building a modern, scalable, fully automated on-premise data platform , designed to handle complex data workflows, including data ingestion, ETL processes, physics-based calculations and machine learning predictions. Orchestrated using Dagster , our platform integrates with multiple data sources, edge devices, and storage systems. A core principle of our architecture is self-service : granting data scientists, analysts, and engineers granular control over the entire journey of their data assets as well empowering teams to modify and extend their data pipelines with minimal friction. We're looking for a hands-on Data Engineer to help develop, maintain, and optimize this platform. Role & responsibilities: - Design, develop, and maintain robust data pipelines using Dagster for orchestration - Build and manage ETL pipelines with python and SQL - Optimize performance and reliability of the platform within on-premise infrastructure constraints - Develop solutions for processing and aggregating data on edge devices , including data filtering, compression, and secure transmission - Maintain metadata, data lineage, ensure data quality, consistency, and compliance with governance and security policies - Implement CI/CD workflows of the platform on a local Kubernetes cluster - Architect the platform with a self-service mindset , including clear abstractions, reusable components, and documentation - Develop in collaboration with data scientists, analysts, and frontend developers to understand evolving data needs - Define and maintain clear contracts/interfaces with source systems , ensuring resilience to upstream changes Preferred candidate profile: -5-7 years of experience in database-driven projects or related fields. -1-2 years of experience with data platforms, orchestration, and big data management. -Proven experience as a Data Engineer or similar role, with focus on backend data processing and infrastructure -Hands-on experience with Dagster or similar data orchestration tools (e.g., Airflow, Prefect, Luigi, Databricks) - Proficiency with SQL and Python - Strong understanding of data modeling , ETL/ELT best practices, and batch/stream processing - Familiarity with on-premises deployments and challenges (e.g., network latency, storage constraints, resource management) - Experience with version control (Git) and CI/CD practices for data workflows - Understanding of data governance , access control , and data cataloging