1338 Data Pipeline Jobs

Salary: 15 to 25 LPA Exp: 4 to 7 years Location: Gurgaon/Pune/Bengalore Notice: Immediate to 30 days..!! Job Profile: Experienced Data Engineer with a strong foundation in designing, building, and maintaining scalable data pipelines and architectures. Skilled in transforming raw data into clean, structured formats for analytics and business intelligence. Proficient in modern data tools and technologies such as SQL, T-SQL, Python, Databricks, and cloud platforms (Azure). Adept at data wrangling, modeling, ETL/ELT development, and ensuring data quality, integrity, and security. Collaborative team player with a track record of enabling data-driven decision-making across business units. As a Dat...

We are looking for a self-starter to join our Data Engineering team. You will work in a fast-paced environment where you will get an opportunity to build and contribute to the full lifecycle development and maintenance of the data engineering platform. With the Data Engineering team you will get an opportunity to - Design and implement data engineering solutions that is scalable, reliable and secure on the Cloud environment Understand and translate business needs into data engineering solutions Build large scale data pipelines that can handle big data sets using distributed data processing techniques that supports the efforts of the data science and data application teams Partner with cross-...

Skill: GCP Data Engineer Experience : 4 to 13 Years Location : AIA - Gurgaon Mandatory Skills : GCP, DBT, Google Dataform, BigQuery, PySpark/Spark SQL, Advanced SQL, Agile Methodologies . Key Responsibilities : Design, build, and optimize scalable data pipelines using BigQuery, and PySpark. Leverage GCP-native services like Cloud Storage, Pub/Sub, Dataproc, Cloud Functions, and Composer for ETL/ELT workflows. Collaborate with cross-functional teams including Infra, Security, and DataOps for reliable, secure, and high-quality data delivery. Lead code reviews, mentor junior engineers, and enforce best practices in data engineering. Participate in Agile sprints, backlog grooming, and Jira-based...

We are seeking a skilled and experienced Data Engineer to join our team, for designing, implementing, and maintaining our data infrastructure, pipelines, and systems.

Job Description Q2 is seeking a Senior Data Engineer to join our Data Insights organization . As part of the Data Engineering team, you will build and maintain the Cloud infrastructure for developing and deploying data analytic products and core data transformation jobs. Our team collaborates closely with data scientists and other product engineering teams. Our goal is to enable Data Scientists to focus on creating data products through discovery and development of transformation jobs and machine-learning models, while ensuring their quick deployment to production. Our team supports the data and machine-learning aspects for multiple Q2 products including Online Banking, PrecisionLender and F...

Design, build a BI&analytics solution Implement batch & near real time data movement design patterns define best practices in DE Design &develop optimal cloud data solutions (lakes, warehouses ,marts, analytics) by collaborating with diverse IT teams Required Candidate profile Exp in designing, implementing, managing large-scale & ETL solutions, BI data modelling, DE, ETL, multi-dimensional DW, cubes, with expertise in relevant languages and frameworks like SQL, Python

6+ years of experience in data engineering, specifically in cloud environments like AWS. Proficiency in Python and PySpark for data processing and transformation tasks. Solid experience with AWS Glue for ETL jobs and managing data workflows. Hands-on experience with AWS Data Pipeline (DPL) for workflow orchestration. Strong experience with AWS services such as S3, Lambda, Redshift, RDS, and EC2. Technical Skills : Deep understanding of ETL concepts and best practices.. Strong knowledge of SQL for querying and manipulating relational and semi-structured data. Experience with Data Warehousing and Big Data technologies, specifically within AWS. Additional Skills: Experience with AWS Lambda for ...

6+ years of experience in data engineering, specifically in cloud environments like AWS. Proficiency in Python and PySpark for data processing and transformation tasks. Solid experience with AWS Glue for ETL jobs and managing data workflows. Hands-on experience with AWS Data Pipeline (DPL) for workflow orchestration. Strong experience with AWS services such as S3, Lambda, Redshift, RDS, and EC2. Technical Skills : Deep understanding of ETL concepts and best practices.. Strong knowledge of SQL for querying and manipulating relational and semi-structured data. Experience with Data Warehousing and Big Data technologies, specifically within AWS. Additional Skills: Experience with AWS Lambda for ...

We are looking for a highly skilled and experienced Software Engineer-Data Engineer to join our team at CGI Information Systems and Management Consultants. The ideal candidate will have a strong background in software engineering and data analysis, with excellent problem-solving skills. Roles and Responsibility Design, develop, and test software applications for data processing and analysis. Collaborate with cross-functional teams to identify business requirements and develop solutions. Develop and maintain large-scale databases and data systems. Analyze complex data sets to identify trends and patterns. Implement data quality checks and validation processes. Optimize database performance an...

We are looking for skilled ML Engineer with strong expertise in Python, data engineering, model development, and API deployment. should be able to work from data extraction cleaning to model building, MLOps, production deployment using FastAPI.

Role & responsibilities Proven experience as a Senior Data Engineer, with a focus on Google Cloud Platform (GCP). Strong proficiency in GCP services such as BigQuery, Dataproc, Dataflow, Pub/Sub, Cloud Composer, and Cloud Storage. Expertise in SQL and database management. Experience with ETL processes and tools. Proficiency in Python programming language. Knowledge of data modeling, data warehousing, and data lakes. Excellent problem-solving skills and attention to detail. Strong communication and collaboration skills. Preferred Skills: Experience with machine learning and AI frameworks. Knowledge of containerization and orchestration tools. Experience with data visualization tools (Tableau,...

We are seeking a Senior Data Analytics Engineer who can take ownership of analytics ecosystems end-to-end ( full-stack) from raw data ingestion to business-ready dashboards. The ideal candidate has a blend of data engineering skills and an eye for storytelling through data. You"ll work with Python , Databricks , ADF , Snowflake , MS Fabric and SQL to build robust data pipelines, and use Power BI, DAX, and Power Query to deliver actionable insights. You will play a critical role in designing, developing, and maintaining end-to-end data solutions, from building robust data pipelines and models to delivering insightful dashboards and visualizations that empower informed decision-making across t...

Role & responsibilities: Architect and maintain structured databases for tick-level and MFT data. Develop efficient ingestion & ETL pipelines using Python/C++, handling large volumes in near-real time. Write optimised SQL queries and manage schemas for time-series and reference data. Implement automated data validation, reconciliation, and version control. Work with quants to expose clean datasets for research and live trading systems. Ensure data integrity across time zones, instruments, and asset classes. Manage historical data archives and set up policies for retention, compression, and retrieval. Collaborate with the execution team to align live feeds with stored data. Requirements : Str...

Greetings!!! TCS is hiring... Role: AWS Data Engineer EXP: 5- 8 YEARS LOCATION: Hyderabad/ Kolkata **** Notice Period- 30 Day s *** Desired Skills -Must-Have AWS Python , AWS Glue, Lambda, Athena, AWS Pyspark Good hands-on experience in Python programming and Pyspark/Scala Data Engineering experience using AWS core services (Lambda, Glue, EMR, S3 and RedShift) Good knowledge on any of the ETL tools and SQLs. Inviting applications with updated resume.

Greetings!!! TCS is hiring... Role: Snowflake Developer EXP: 6 - 10 YEARS LOCATION: Hyderabad/ Bangalore / Chennai/ Pune **** Notice Period- Immediate to Max 30 Day s *** Desired Skills -Must-Have SQL, Snowflake, Snowflake modeling roles, databases, schemas, ETL , python Migration to AWS Snowflake platform and Knowledge in S3 buckets Knowledge of SQL language and cloud-based technologies Data warehousing concepts, data modelling, metadata management Data lakes, multi-dimensional models, data dictionaries Migration to AWS Snowflake platform and Knowledge in S3 buckets Performance tuning and setting up resource monitors Snowflake modelling – roles, databases, schemas (Hands on must) SQL perfor...

Job Title: Data Engineer Location: Kolkata, India Experience: Minimum 5 years in Data Engineering Joining: Immediate Job Description: We are looking for a skilled Data Engineer to design, build, and maintain scalable data architectures and pipelines. The ideal candidate will have hands-on experience in data transformation, data lake solutions, and analytics enablement, along with strong problem-solving and collaboration skills. Key Responsibilities: Develop & Maintain: Design, build, and maintain scalable data architectures and pipelines to support analytics and business requirements. Ensure proper documentation of solutions. Monitor & Troubleshoot: Support stakeholders with data issues, mon...

Lead the development and implementation of data engineering solutions using AWS. Collaborate with cross-functional teams to design and implement scalable data pipelines. Develop and maintain large-scale data architectures and systems. Ensure high-quality data processing and analysis across the organization. Mentor junior engineers and provide technical guidance. Stay up-to-date with industry trends and emerging technologies in data engineering. Job Requirements Strong understanding of data engineering principles and practices. Experience working with large datasets and distributed computing environments. Excellent leadership and communication skills. Ability to work collaboratively with cros...

Design, develop, and implement data pipelines and architectures. Collaborate with cross-functional teams to identify and prioritize project requirements. Develop and maintain large-scale data systems and databases. Ensure data quality and integrity through data validation and testing procedures. Optimize data processing workflows for improved performance and efficiency. Troubleshoot and resolve technical issues related to data engineering. Job Requirements Strong understanding of data engineering principles and practices. Experience with data modeling, database design, and data warehousing concepts. Proficiency in programming languages such as Python or Java. Excellent problem-solving skills...

Job Description_ Senior Data Engineer, Secure ETL & Ingestion Platform/Data Architect Why would you like to join us? TransOrg Analytics specializes in Data Science, Data Engineering and Generative AI, providing advanced analytics solutions to industry leaders and Fortune 500 companies across India, US, APAC and the Middle East. We leverage data science to streamline, optimize, and accelerate our clients' businesses. Visit at www.transorg.com to know more about us. Function: Data Engineering / Data Platform Role Overview - We are looking for a Senior Data Engineer to design and build our next-generation ETL / ingestion pipeline with a strong focus on security, performance, reliability, and go...

We are looking for a skilled Data Engineer with expertise in PySpark, AWS, and SQL to support data processing and analytical initiatives. This role involves working closely with data engineering and data science teams to build, maintain, and optimize large-scale data pipelines and integrations on AWS. The ideal candidate will be proficient in ETL processes using PySpark and SQL, with a deep understanding of cloud data infrastructure, specifically within the AWS ecosystem. Key Responsibilities : Data Pipeline Development : - Design, build, and optimize ETL pipelines using PySpark for data ingestion, transformation, and storage on AWS. - Collaborate with stakeholders to understand data require...

Skill: AWS Glue Experience: 9 to 12 years Location: AIA Chennai , AIA Bangalore Job Summary Join our dynamic team as an Architect where you will leverage your expertise in Spark Optimization AWS DevOps and Python to design and implement cutting-edge solutions. With a focus on AWS Glue Studio AWS Glue ETL and Amazon S3 you will drive innovation and efficiency in our hybrid work model. This role requires a minimum of 7 years of experience offering an exciting opportunity to make a significant impact. Responsibilities Design and implement scalable data architectures using Spark Optimization to enhance performance and efficiency. Collaborate with cross-functional teams to integrate AWS DevOps pr...

AIA - Face to Face Drive - Chennai Skill: GCP, BigQuery & Python Experience: 4 to 9 Years Location: Chennai Drive Date: 06 Dec 2025 Job Summary Data Engineer with a focus on Google Cloud Platform (GCP). Hands-on experience with GCP services: Dataflow Pub/Sub Cloud Storage Cloud Composer. Expertise in workflow orchestration using Apache Airflow. Experience with cloud SQL or other relational databases (PostgreSQL preferred). Solid understanding of data modeling ETL processes and streaming/batch data pipelines. Familiarity with DevOps practices CI/CD and containerization (Docker Kubernetes). Responsibilities Key Responsibilities and Skills Build Data Pipelines on GCP: They need hands-on experie...

Retail Media is transforming how advertisers connect with consumers through personalized and targeted campaigns across retailers' digital and physical touchpoints. Retail Media Measurement plays a pivotal role in ensuring the effectiveness of these campaigns, driving value for advertisers, retailers, and consumers alike. This role focuses on designing, building, and scaling solutions that enable the accurate measurement of retail media campaigns across various channels. By providing actionable insights, it empowers stakeholders to optimize media investments, improve ROI, and enhance the overall customer experience. Job Title: Senior Data Engineer Job Summary We are looking for a talented and...

Role: Data Engineer Location: Noida JD: We are seeking a skilled and motivated Data Engineer to join our team. The ideal candidate will have a strong background in data engineering, data architecture, and data integration. This role involves building and maintaining scalable data pipelines, ensuring data quality and integrity, and supporting data analytics initiatives. You will work closely with internal customers, product owners, core team members, and other stakeholders to ensure that quality data is available, reliable, and easily accessible in a cost effective efficient, scalable way. Key Responsibilities: Data Pipeline Development: Design, build, and maintain robust, scalable, and effic...

5+ years of QA experience with a focus on automation and cloud-native environments. Hands-on expertise in automation tools such as Selenium, Cypress, or Playwright. Experience with ETL/data pipeline testing, including validating source-to-target data mapping, transformations, and aggregation logic. Proficient in writing and executing complex SQL queries for data validation. Working knowledge of GCP services such as BigQuery, Pub/Sub, Cloud Functions, Cloud Storage, etc. Familiarity with API testing using Postman, RestAssured, or similar tools. Scripting knowledge in Python, JavaScript, or Java for test automation. Experience working in CI/CD pipelines and version control (Git). Roles and Res...