20 Pipeline Orchestration Jobs

Job Description: As a Senior Data Engineer, you will be responsible for building and optimizing data systems that drive batch processing, real-time streaming, pipeline orchestration, data lake management, and data cataloging. Your role will involve leveraging your expertise to tackle large-scale data challenges, utilizing design thinking, coding, and analytical skills to create data pipelines to enhance marketing data products. We are seeking passionate Data Engineers who are eager to develop innovative data-driven solutions using cutting-edge Big Data technology.,

What is Hunch Hunch is a dating app that helps you land a date without swiping like a junkie. Designed for people tired of mindless swiping and commodified matchmaking, Hunch leverages a powerful AI-engine to help users find meaningful connections by focusing on personality over just looks. With 2M+ downloads and a 4.4-star rating , Hunch is going viral in the US by challenging the swipe-left/right norm of traditional apps. Hunch is a Series A funded ($23 Million) startup building the future of social discovery in a post-AI world. Link to our fundraising announcement Key Offerings Of Hunch Swipe Less, Vibe More: Curated profiles, cutting the clutter of endless swiping. Personality Matters: O...

As a Sr. Data Engineer Data Science at NTT DATA in Bangalore, Karnataka, India, your role will involve driving data engineering tasks from demand management to design to implementation. You will act as a trusted advisor to the team, proactively communicating operational status, issues, and risks to management. Additionally, you will lead internal consulting engagements, manage stakeholder expectations, monitor deliverables" quality, and align with Mercedes-Benz IT project management tools and methodologies. **Key Responsibilities:** - Drive data engineering tasks from demand management to implementation - Act as a trusted advisor to the team and communicate operational status, issues, and ri...

As a Data Science Specialist, your role will involve collaborating with cross-functional teams to translate business problems into data science use cases. You will design, develop, and deploy data science models, including classification, regression, and survival analysis techniques. Your responsibilities will also include building and productionizing data products that deliver measurable business impact, performing exploratory data analysis, feature engineering, model validation, and performance tuning. Additionally, you will apply statistical methods to uncover trends, anomalies, and actionable insights, while staying up to date with advancements in NLP and Generative AI to evaluate their ...

Req ID: 342869 NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Sr. Data Engineer Data Science to join our team in Bangalore, Karn?taka (IN-KA), India (IN). Description Responsibilities: Drive data engineering and tasks from demand management to design to implementation. Act as a trusted advisor to our team and proactively communicate operational status, issues & risks in the projects to management. Lead internal consulting engagements, managing stakeholder expectations and monitor time to time and drive quality of de...

Role Overview We are looking for a Senior Data Engineer to lead the design and development of a next-generation data platform that powers real-time and batch applications at scale. This platform supports business logic, personalisation, user cohorting, CRM, product analytics, and Generative AI workloads. You'll work cross-functionally with analysts, engineers, and product teams to build high-throughput data pipelines, ensure reliability and cost-efficiency, and empower data-driven decisions across all touchpoints web, mobile (Android/iOS) and server. Key Responsibilities Data Platform & Pipeline Development Build and scale real-time and batch data pipelines capable of supporting high-volume ...

At INDI, we're passionate about empowering individuals and businesses worldwide. Our cutting-edge recruiters connect leading companies with top talent, fostering a dynamic environment where innovation thrives. Join us in shaping the future of work. Overview of the role: The Solutions Engineer will play a key role in driving enterprise adoption of an advanced data and AI platform. This position involves architecting and implementing end-to-end solutions, engaging in technical discussions with clients, and ensuring seamless integration of AI technologies to accelerate customer success. Key responsibilities: - Collaborating with customers, partners, and prospects to design tailored data and AI ...

Role Overview: As a Senior Data Engineer, you will be responsible for building and optimizing data systems that support batch processing, real-time streaming, pipeline orchestration, data lake management, and data cataloging. You will leverage your expertise in solving big data problems and apply design thinking, coding, and analytical skills to develop data pipelines that facilitate marketing data products. Key Responsibilities: - Build and optimize data systems for batch processing and real-time streaming. - Manage pipeline orchestration to ensure smooth data flow. - Oversee data lake management and data cataloging processes. - Develop data pipelines that support marketing data products. Q...

Job Description JOB DESCRIPTION We are seeking an experienced SAP Consultant with a strong focus on Data Integration, SAP Datasphere, and SAP Business Technology Platform (BTP) to join our team. This role will drive the design and implementation of scalable integration frameworks that support SAP Commissions (CallidusCloud) and related Sales Performance Management (SPM) solutions. The successful candidate will bring deep expertise in data modeling, pipeline orchestration, and analytics integration , along with the ability to lead technical teams and collaborate with business stakeholders. This position emphasizes data strategy, performance optimization, and end-to-end integration delivery , ...

About the Role: You will be working on ensuring scaling of ML products and credit risk policy (incl. models) within pre-approved/ pre-qualified campaigns and live digital journeys. It will entail ensuring APIs operate smoothly at scale, identify ways to ensure simplified and low risk customer journey. We are looking for someone with entrepreneurial mindset, cloud-based tools/ packages for automation and encryption and APIs. About the team: You will be part of ML Engineering Team within Digital Banking Unit of the bank. You will work in a fast-paced environment where new ideas are encouraged. This will involve liaising with teams within Digital and those across the bank e.g. data engineers, d...

Description The Data Engineering Lead is a senior-level position within our Data Engineering team, responsible for spearheading the design, implementation, and optimization of advanced data architectures using Databricks. This role involves leading a team of data engineers, collaborating with business and data analysts, and business stakeholders, and delivering scalable, secure data solutions tailored to the needs of commercial analytics in the pharmaceutical industry. Responsibilities 1. Architecture Design Develop and maintain scalable data architectures using Databricks, including data lakes, data warehouses, and real-time processing systems. Create detailed blueprints for data processes ...

About the Role TheData Platform teamis part of the Platform org which is based in Bangalore and is responsible for building Uber's Data Lake for the rest of the company. The team owns end to end charters for multiple batch data areas around Pipeline Orchestration and Authoring, Metadata Platform, Lineage, Data Quality, Data Observability, Data Efficiency and Data Engineering. We have an eclectic team with a diverse set of backgrounds from companies such as Google, Cloudera, Amazon, Microsoft, LinkedIn, Twitter, other startups and college grads from the top schools. The team is proud to be a part of the open source community in innovating and shaping such exciting technologies as we move forw...

As a Senior Technical Lead Associate Architect specializing in Microsoft Fabric, your role will involve the following key responsibilities: - **Architect & Design Solutions**: Lead the end-to-end architecture of data platforms using Microsoft Fabric, ensuring alignment with business goals and technical standards. - **Data Engineering Leadership**: Guide the data engineering team in building robust, scalable, and efficient data pipelines and semantic models. - **Migration Strategy**: Define and implement validation rules, reconciliation mechanisms, and ensure zero data loss during migration from legacy systems to Fabric. - **Reporting & Visualization**: Architect and optimize reporting layers...

As a BI Architect in Gen AI, your role involves architecting a scalable and performant BI platform that serves as a single source of truth for key business metrics across functions such as Sales, Marketing, Supply Chain, and Finance. You will be advising the BI Lead by aligning architecture decisions with long-term strategy and business priorities. Additionally, you will design a GenAI-powered natural language interface on top of BI dashboards to enable business users to query KPIs, trends, and anomalies conversationally. Your responsibilities also include selecting and implementing appropriate architectures like RAG using vector stores and Knowledge Graphs to support intelligent, context-aw...

In this role at American Express, as a Senior Data Engineer on the Infrastructure Data & Analytics team, you will play a crucial part in unifying FinOps, Data Science, and Business Intelligence to enhance Technology cost transparency, optimize infrastructure performance, and improve commercial efficiency through high-quality data and predictive analytics. Your responsibility will be to lead the design, development, and enhancement of the data model and architecture, ensuring data quality meets the highest standards for users. Key Responsibilities: - **Data Architecture**: Perform technical aspects of data architecture and database management, develop data pipelines, new database structures, ...

Job Description At American Express, our culture is built on a 175-year history of innovation, shared and Leadership Behaviors, and an unwavering commitment to back our customers, communities, and colleagues. As part of Team Amex, you'll experience this powerful backing with comprehensive support for your holistic well-being and many opportunities to learn new skills, develop as a leader, and grow your career. Here, your voice and ideas matter, your work makes an impact, and together, you will help us define the future of American Express. As part of our diverse tech team, you can architect, code and ship software that makes us an essential part of our customers digital lives.Here, you can w...

Hiring GCP Data Engineers (3–5 yrs) & Senior GCP Data Engineers (5–8 yrs). Skills: Python, SQL, BigQuery, Dataflow, Pub/Sub, Composer. Location: Remote/Hybrid/Onsite.

You are a Data Science Specialist with 4-5 years of experience in designing and implementing advanced analytical solutions. You should have a strong foundation in statistics and expertise in solving real-world business problems using Machine Learning and Data Science. Your track record should include building and deploying data products, and any exposure to NLP and Generative AI will be considered an advantage. Your key responsibilities will include collaborating with cross-functional teams to translate business problems into data science use cases, designing, developing, and deploying data science models, building and productionizing data products for measurable business impact, performing ...

Key Skills: Data engineering, Apache Airflow, GCP, BigQuery, GCS, SQL, ETL/ELT, Docker, Kubernetes, data governance, Agile, CI/CD, DevOps, pipeline orchestration, technical leadership. Roles & Responsibilities: Evaluate and provide scalable technical solutions to address complex and interdependent data processes. Ensure data quality and accuracy by implementing data quality checks, data contracts, and governance processes. Collaborate with software development teams and business analysts to understand data requirements and deliver fit-for-purpose data solutions. Lead the team in delivering end-to-end data engineering solutions. Design, develop, and maintain complex applications to support da...

Key responsibilities include: Data Architecture: Perform all technical aspects of data architecture and database management for ID&A, including developing data pipelines, new database structures and APIs as applicable Data Design: Translate logical data architectures into physical data designs, ensuring alignment with data modeling best practices and standards Data Process and Monitoring: Ensure proper data ingestion, validation, testing, and monitoring for ID&A data lake Support database and data platform administration for initiatives building, enhancing, or maintaining databases, data warehouses and data pipelines Data Migration: Design and support migration to a data model that is indepe...