GCP Data Engineer

Senior GCP Data Engineer

Senior GCP Data Engineer

Responsibilities:

• Architect, build, and maintain end-to-end data processing pipelines using the full suite of GCP products, including

  BigQuery, Data Fusion, Dataproc, Dataflow, Cloud Composer, and Pub/Sub.

• Design and develop scalable and reliable data solutions for both batch and streaming data.
• Orchestrate complex data workflows and automate jobs using

  Apache Airflow

  and/or

  GCP Cloud Composer

  .
• Write clean, efficient, and well-documented code in

  Python and/or PySpark

  for data transformation and processing tasks.
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver high-quality data products.
• Implement and manage infrastructure as code using

  Terraform

  to provision and configure GCP resources.
• Leverage

  Docker

  for containerization to ensure consistency across development, testing, and production environments.
• Utilize

  Git/GitHub

  for version control and collaborative development.
• Monitor data pipeline performance, troubleshoot issues, and optimize queries and processes to ensure data quality, reliability, and cost-effectiveness.

Essential Skills and Experience:

• 8-10 years of overall experience with 5+ years of relevant experience on cloud platforms (

  GCP

  ).
• Deep, hands-on experience with core GCP data services such as

  BigQuery, Data Fusion, Dataproc, Cloud Composer, Dataflow, and Pub/Sub.

• Expertise in a programming language, with a strong preference for

  Python

  and experience with

  Spark/PySpark

  .
• In-depth knowledge of data orchestration tools, specifically

  Apache Airflow

  or

  GCP Cloud Composer

  .
• Proven experience with

  Docker

  and

  Terraform

  .
• Proficiency with

  Git/GitHub

  for source control.

Desirable Qualifications:

• Knowledge of CI/CD pipelines using

  Cloud Build

  or

  Azure DevOps

  .

• Google Cloud Certified Professional Data Engineer

  certification is a significant plus.