Senior Data Engineer

Responsibilities

• Design, build, and maintain our end-to-end data infrastructure on AWS and GCP cloud platforms.
• Develop and optimize ETL/ELT pipelines to process large volumes of data from multiple sources.
• Build and support data pipelines for reporting, analytics, and machine learning applications.
• Implement and manage streaming data solutions using Kafka and other technologies.
• Design and optimize database schemas and data models in ClickHouse and other databases.
• Develop and maintain data workflows using Apache Airflow and similar orchestration tools.
• Write efficient, maintainable, and scalable code using PySpark and other data processing frameworks.
• Collaborate with data scientists to implement ML infrastructure for model training and deployment.
• Ensure data quality, reliability, and security across all data platforms.
• Monitor data pipelines and implement proactive alerting systems.
• Troubleshoot and resolve data infrastructure issues.
• Document data flows, architectures, and processes.
• Stay current with industry trends and emerging technologies in data engineering.
  

Requirements

• Bachelor's degree in Computer Science, Engineering, or related technical field (Master's preferred).
• 5+ years of experience in data engineering roles.
• Strong expertise in AWS and/or GCP cloud platforms and services.
• Proficiency in building data pipelines using modern ETL/ELT tools and frameworks.
• Experience with stream processing technologies such as Kafka.
• Hands-on experience with ClickHouse or similar analytical databases.
• Strong programming skills in Python and experience with PySpark.
• Experience with workflow orchestration tools like Apache Airflow.
• Solid understanding of data modeling, data warehousing concepts, and dimensional modeling.
• Knowledge of SQL and NoSQL databases.
• Strong problem-solving skills and attention to detail.
• Excellent communication skills and ability to work in cross-functional teams.
• Experience in D2C, e-commerce, or retail industries.
• Knowledge of data visualization tools (Tableau, Looker, Power BI).
• Experience with real-time analytics solutions.
• Familiarity with CI/CD practices for data pipelines.
• Experience with containerization technologies (Docker, Kubernetes).
• Understanding of data governance and compliance requirements.
• Experience with MLOps or ML engineering Technologies.
• Cloud Platforms: AWS (S3 Redshift, EMR, Lambda), GCP (BigQuery, Dataflow, Dataproc).
• Data Processing: Apache Spark, PySpark, Python, SQL.
• Streaming: Apache Kafka, Kinesis.
• Data Storage: ClickHouse, S, 3 BigQuery, PostgreSQL, MongoDB.
• Orchestration: Apache Airflow.
• Version Control: Git.
• Containerization: Docker, Kubernetes (optional).