Senior Engineer(DT_330)

4 - 12 Years

• Job Description – Senior Data Engineer

We at Pine Labs are looking for those who share our core belief - “Every Day is Game day”. We bring our best selves to work each day to realize our mission of enriching the world through the power of digital commerce and financial services.

Role Purpose

We are looking for skilled Senior Data Engineers with 4-12 years of experience to join our growing team. You will design, build, and optimize real-time and batch data pipelines, leveraging AWS cloud technologies and Apache Pinot to enable high-performance analytics for our business. This role is ideal for engineers who are passionate about working with large-scale data and real-time processing.

Responsibilities we entrust you with

Data Pipeline Development:

• Build and maintain robust ETL/ELT pipelines for batch and streaming data using tools like Apache Spark, Apache Flink, or AWS Glue.
• Develop real-time ingestion pipelines into Apache Pinot using streaming platforms like Kafka or Kinesis.

Real-Time Analytics:

• Configure and optimize Apache Pinot clusters for sub-second query performance and high availability.
• Design indexing strategies and schema structures to support real-time and historical data use cases.

Cloud Infrastructure Management:

• Work extensively with AWS services such as S3, Redshift, Kinesis, Lambda, DynamoDB, and CloudFormation to create scalable, cost-effective solutions.
• Implement infrastructure as code (IaC) using tools like Terraform or AWS CDK.

Performance Optimization:

• Optimize data pipelines and queries to handle high throughput and large-scale data efficiently.
• Monitor and tune Apache Pinot and AWS components to achieve peak performance.

Data Governance & Security:

• Ensure data integrity, security, and compliance with organizational and regulatory standards (e.g., GDPR, SOC2).
• Implement data lineage, access controls, and auditing mechanisms.

Collaboration:

• Work closely with data scientists, analysts, and other engineers to translate business requirements into technical solutions.
• Collaborate in an Agile environment, participating in sprints, standups, and retrospectives.

Relevant work experience

• 4-12 years of hands-on experience in data engineering or related roles.
• Proven expertise with AWS services and real-time analytics platforms like Apache Pinot or similar technologies (e.g., Druid, ClickHouse).
• Proficiency in Python, Java, or Scala for data processing and pipeline development.
• Strong SQL skills and experience with both relational and NoSQL databases.
• Hands-on experience with streaming platforms such as Apache Kafka or AWS Kinesis.
• Familiarity with big data tools like Apache Spark, Flink, or Airflow.
• Strong problem-solving skills and a proactive approach to challenges.
• Excellent communication and collaboration abilities in cross-functional teams.

Preferred Qualifications:

• Experience with data lakehouse architectures (e.g., Delta Lake, Iceberg).
• Knowledge of containerization and orchestration tools (e.g., Docker, Kubernetes).
• Exposure to monitoring tools like Prometheus, Grafana, or CloudWatch.
• Familiarity with data visualization tools like Tableau or Superset.

What We Offer:

• Competitive compensation based on experience.
• Flexible work environment with opportunities for growth.
• Work on cutting-edge technologies and projects in data engineering and analytics.

What we Value in Our people

• You take the shot: You Decide Fast and You Deliver Right
• You are the CEO of what you do: you show ownership and make things happen
• You own tomorrow: by building solutions for the merchants and doing the right thing
• You sign your work like an artist: You seek to learn and take pride in the work you do