Manager

We are looking for an experienced Data Engineer to design, build, and maintain scalable data pipelines for processing clickstream data in Google Cloud Platform (GCP). The ideal candidate will have 5+ years of experience working with GCP, particularly in eCommerce, and possess a deep understanding of BigQuery and data pipeline automation. This role will involve building robust data workflows using Airflow , handling large data volumes, and ensuring smooth integration with Google Analytics BigQuery data exports. Key Responsibilities Pipeline Development: Design, implement, and maintain automated data pipelines using Airflow to process clickstream data in GCP, ensuring efficiency and reliability. BigQuery Expertise: Leverage BigQuery for data storage, querying, and optimizing the performance of large data sets, ensuring fast query performance on 100B+ row tables. Data Integration: Work closely with the team to integrate clickstream data from various sources, particularly focusing on Google Analytics BigQuery exports, into the data pipeline. Automation & Monitoring: Automate data processing workflows and establish robust monitoring processes to ensure seamless data flow and timely delivery of data to end-users. Data Quality & Optimization: Ensure high-quality data with proper transformations and aggregations. Optimize large data queries to reduce latency and improve processing efficiency. Collaboration: Work closely with cross-functional teams (Data Science, Analytics, Product, and Business teams) to understand their data requirements and deliver solutions. Documentation & Best Practices: Document processes, workflows, and pipeline architecture. Promote best practices for data pipeline development and GCP services usage. Scalability: Design and implement scalable systems that can handle growing volumes of clickstream data in eCommerce applications. Skills & Qualifications Experience: 5+ years of experience in data engineering with a focus on GCP, particularly in eCommerce or digital analytics environments. GCP Expertise: Extensive experience with Google Cloud Platform (GCP) services such as BigQuery , Cloud Storage , Cloud Functions , Pub/Sub , and Dataflow . BigQuery Mastery: Deep understanding of BigQuery for large-scale data processing and optimization, including partitioning, clustering, and query optimization for massive datasets. Data Pipelines: Hands-on experience automating, scheduling, and monitoring data pipelines using Airflow . Handling Large Data Volumes: Experience working with very large datasets (e.g., 100B+ rows) and optimizing data storage and processing for high performance. Clickstream Data: Familiarity with working with clickstream data and integrating it into data pipelines. Google Analytics BigQuery Export: Ideally, experience working with Google Analytics BigQuery data exports and integrating analytics data into a centralized data warehouse. Programming: Strong proficiency in Python for data processing, pipeline orchestration, and automation. SQL Skills: Proficient in writing complex SQL queries for data extraction, transformation, and analysis. Problem Solving: Strong analytical and problem-solving skills, with an ability to troubleshoot and resolve data pipeline and performance issues. Collaboration & Communication: Excellent communication skills to work across teams and explain complex technical concepts to non-technical stakeholders. Preferred Qualifications Experience with eCommerce Analytics: Previous experience working with eCommerce data, particularly clickstream and transactional data. Monitoring & Alerts: Familiarity with setting up monitoring, logging, and alerts to ensure data pipelines are running smoothly and issues are flagged promptly. Cloud Security: Knowledge of data security and access control best practices in the cloud environment (IAM, VPC, etc.). Show more Show less