Senior Data Engineer

Job Summary:

We are seeking a highly motivated and experienced Senior Data Engineer to join our team. This role requires a deep curiosity about our business and a passion for technology and innovation. You will be responsible for designing and developing robust, scalable data engineering solutions that drive our business intelligence and data-driven decision-making processes. If you thrive in a dynamic environment and have a strong desire to deliver top-notch data solutions, we want to hear from you.

Key Responsibilities:

• Collaborate with agile teams to design and develop cutting-edge data engineering solutions.
• Build and maintain distributed, low-latency, and reliable data pipelines ensuring high availability and timely delivery of data.
• Design and implement optimized data engineering solutions for Big Data workloads to handle increasing data volumes and complexities.
• Develop high-performance real-time data ingestion solutions for streaming workloads.
• Adhere to best practices and established design patterns across all data engineering initiatives.
• Ensure code quality through elegant design, efficient coding, and performance optimization.
• Focus on data quality and consistency by implementing monitoring processes and systems.
• Produce detailed design and test documentation, including Data Flow Diagrams, Technical Design Specs, and Source to Target Mapping documents.
• Perform data analysis to troubleshoot and resolve data-related issues.
• Automate data engineering pipelines and data validation processes to eliminate manual interventions.
• Implement data security and privacy measures, including access controls, key management, and encryption techniques.
• Stay updated on technology trends, experimenting with new tools, and educating team members.
• Collaborate with analytics and business teams to improve data models and enhance data accessibility.
• Communicate effectively with both technical and non-technical stakeholders.

Qualifications:

• Education:

   Bachelors degree in Computer Science, Computer Engineering, or a related field.

• Experience:

   Minimum of 5+ years in architecting, designing, and building data engineering solutions and data platforms.
• Proven experience in building Lakehouse or Data Warehouses on platforms like Databricks or Snowflake.
• Expertise in designing and building highly optimized batch/streaming data pipelines using Databricks.
• Proficiency with data acquisition and transformation tools such as Fivetran and DBT.
• Strong experience in building efficient data engineering pipelines using Python and PySpark.
• Experience with distributed data processing frameworks such as Apache Hadoop, Apache Spark, or Flink.
• Familiarity with real-time data stream processing using tools like Apache Kafka, Kinesis, or Spark Structured Streaming.
• Experience with various AWS services, including S3, EC2, EMR, Lambda, RDS, DynamoDB, Redshift, and Glue Catalog.
• Expertise in advanced SQL programming and performance tuning.

Key Skills:

• Strong problem-solving abilities and perseverance in the face of ambiguity.
• Excellent emotional intelligence and interpersonal skills.
• Ability to build and maintain productive relationships with internal and external stakeholders.
• A self-starter mentality with a focus on growth and quick learning.
• Passion for operational products and creating outstanding employee experiences.