Jobs
Interviews
UST logo

GCP & PySpark with ETL - Lead

5 - 9 years

0 Lacs

kochi kerala

Posted:1 week ago| Platform: Shine logo

Apply

Skills Required

gcp etl pyspark

Work Mode

On-site

Job Type

Full Time

Job Description

You will be responsible for designing, developing, and optimizing ETL pipelines using PySpark on Google Cloud Platform (GCP). Your role will involve working with BigQuery, Cloud Dataflow, Cloud Composer (Apache Airflow), and Cloud Storage for data transformation and orchestration. You will need to develop and optimize Spark-based ETL processes for large-scale data processing. It will be essential to implement best practices for data governance, security, and monitoring in a cloud environment. Collaboration with data engineers, analysts, and business stakeholders to understand data requirements is a key aspect of your role. Troubleshooting performance bottlenecks and optimizing Spark jobs for efficient execution will be part of your daily tasks. Automation of data workflows using Apache Airflow or Cloud Composer is crucial. You will also be responsible for ensuring data quality, validation, and consistency across pipelines. To excel in this role, you should have at least 5 years of experience in ETL development with a focus on PySpark. Strong hands-on experience with Google Cloud Platform (GCP) services such as BigQuery, Cloud Dataflow/Apache Beam, Cloud Composer (Apache Airflow), and Cloud Storage is required. Proficiency in Python and PySpark for big data processing is a must. Experience with data lake architectures, data warehousing concepts, and SQL for data querying and transformation is essential. Experience with CI/CD pipelines for data pipeline automation, strong debugging and problem-solving skills, and familiarity with Kafka or Pub/Sub for real-time data processing are desirable. Knowledge of Terraform for infrastructure automation on GCP, experience with containerization (Docker, Kubernetes), and familiarity with DevOps and monitoring tools like Prometheus, Stackdriver, or Datadog will be beneficial for this role. Your skills in GCP, PySpark, and ETL will be put to the test as you contribute to the development and optimization of data pipelines in a cloud environment.,

More Jobs at UST

Architect II - Technical SAP ABAP

Bengaluru

12 - 15 yrs
|

INR 0 - 0 Lacs

Lead I - SAP FICO with PS (Project Systems) and RAR

Bengaluru

5 - 7 yrs
|

INR 0 - 0 Lacs

Lead II - SAP ABAP

Bengaluru

7 - 9 yrs
|

INR 0 - 0 Lacs

Procurement

Bengaluru, Karnataka

Experience: Not specified

|

Salary: Not disclosed

Manual & Python Automation

Bengaluru, Karnataka

Experience: Not specified

|

Salary: Not disclosed

Mock Interview

Practice Video Interview with JobPe AI

Start PySpark Interview
cta

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.

Job Application AI Bot

Job Application AI Bot

Apply to 20+ Portals in one click

Download Now

Download the Mobile App

Instantly access job listings, apply easily, and track applications.

Google Play App Store
coding practice

Enhance Your Skills

Practice coding challenges to boost your skills

Start Practicing Now
UST logo
UST

IT Services and IT Consulting

Aliso Viejo CA

RecommendedJobs for You

GCP & PySpark with ETL - Lead

UST

kochi, kerala

UST logo

GCP & PySpark with ETL - Lead

UST

Kochi, Kerala, India

GCP & PySpark with ETL - Lead

UST

kochi, kerala

UST logo

GCP & PySpark with ETL - Lead

UST

Kochi, Kerala, India