688 Apache Spark Jobs

About the role: Data Management roles are skills and expertise in the secure storage and management of data in compliance with guidelines, standards and best practice They enable efficient and high quality data analysis by designing and implementing approaches to clean and transform data and store data in the most appropriate way Purpose of Position: We are seeking a highly skilled and experienced Data Engineering Lead to spearhead our data architecture strategy, oversee robust data engineering initiatives, and manage data lake operations The ideal candidate will be a hands-on leader with deep expertise in building and optimizing data platforms that support advanced analytics and business in...

Role Overview: At PwC, your focus in managed services will involve providing outsourced solutions and supporting clients across various functions. You will assist organisations in streamlining operations, reducing costs, and enhancing efficiency by managing key processes and functions on their behalf. Your expertise in project management, technology, and process optimization will enable you to deliver high-quality services to clients. As part of the managed service management and strategy team at PwC, you will be responsible for transitioning and running services, managing delivery teams, programmes, commercials, performance, and delivery risk. Your role will revolve around continuous improv...

Role Overview: As a Spark (Scala) Developer at Viraaj HR Solutions, your primary responsibility is to design, develop, and implement data processing applications using Apache Spark and Scala. You will work closely with cross-functional teams to gather requirements, define project specifications, and build scalable ETL pipelines for efficient data processing. Your role also involves optimizing Spark jobs for performance, utilizing SQL for data querying and transformation, and developing data models to support analytics initiatives. Additionally, you will be engaged in data analysis, integration of big data tools and frameworks, code reviews, troubleshooting application issues, and ensuring hi...

You will be joining GlobalLogic to work on a significant software project for a world-class company specializing in M2M / IoT 4G/5G modules for industries like automotive, healthcare, and logistics. Your role will involve contributing to the development of end-user modules" firmware, implementing new features, ensuring compatibility with the latest telecommunication and industry standards, and analyzing customer requirements to provide estimations. - Design end-to-end AI/ML architecture and solutions in alignment with business objectives - Lead the development and deployment of ML models using frameworks like TensorFlow, PyTorch, or Scikit-learn - Architect scalable data pipelines for traini...

Role Overview: Do you aspire to drive the next level of application architecture in the Compliance IT space Are you passionate about applying Agile methodologies at scale and developing a cutting-edge Data Ingestion, Aggregation & Calculation Platform on Cloud Would you like to contribute to shaping the future of Compliance technology alongside a dynamic cross-functional team If so, we have an exciting opportunity for you as a seasoned full stack developer. In this role, you will be responsible for implementing and delivering high-quality software solutions for our state-of-the-art Data Ingestion, Aggregation & Calculation platform. Key Responsibilities: - Implement and deliver high-quality ...

**Role Overview:** As a Data Engineer, your primary responsibility will involve designing, developing, and maintaining ETL processes to facilitate the smooth integration of raw data from diverse sources into data lakes or warehouses. You will be leveraging tools such as Python, PySpark, SQL, and various AWS services like Lambda, Glue, Redshift, and S3 to efficiently process, analyze, and store large-scale datasets. Additionally, you will be tasked with developing and optimizing data pipelines using AWS Glue, PySpark, and Python, while also utilizing your expertise in Apache Spark/Databricks for advanced ETL workflows. Writing and managing SQL queries for data retrieval, transformation, and s...

Job Description: As a Principal Software Engineer, you will provide technical leadership within development teams, ensuring architectural coherence, collaboration, and best practices. Your role involves bridging the gap between data engineering, data science, and software engineering to help build scalable, maintainable data solutions. You will design and implement solutions, mentor developers, influence technical direction, and drive best practices. Additionally, you will have line management responsibilities, focusing on the growth and development of team members. The role will involve working in an Azure and Databricks environment, leveraging cloud-native technologies and modern data plat...

As the Director of Data and Analytics at KASH Tech, your role will involve driving the strategy, execution, and delivery of data and analytics solutions across modern cloud platforms, AI/ML, and big data ecosystems. Your responsibilities will include defining and executing data modernization strategies, leading enterprise cloud data transformations, developing best-fit methodologies for client solutions, identifying opportunities for AI, automation, and data-driven decision-making, and architecting and implementing scalable data platforms using various tools like Databricks, Azure Synapse, and Microsoft Fabric. Key Responsibilities: - Define and execute data modernization strategies aligned ...

Role Overview: As a member of our team at Quantiphi, you will be responsible for creating and maintaining LookML code to define data models, dimensions, measures, and relationships within Looker. Your role will involve developing reusable LookML components to ensure efficiency in report and dashboard creation, building and customizing dashboards with data visualizations, writing complex SQL queries, optimizing Looker performance, configuring user roles and permissions, and providing training and support to business users. Key Responsibilities: - Create and maintain LookML code to define data models, dimensions, measures, and relationships within Looker. - Develop reusable LookML components f...

As a Lead Data AI/ML Engineer - Vice President at Barclays, you will spearhead the evolution of the Corporate Data Services function. Your role will involve effective stakeholder management, leadership, and decision-making to support business strategy and risk management. **Key Responsibilities:** - Participating in daily stand-up meetings to discuss progress, blockers, and plans for the day. - Working on deploying AI/ML models into production environments using AWS services like Glue, Lambda, and Step Functions. - Building and maintaining data pipelines using Spark and Python to ensure seamless data flow and integration. - Collaborating with data scientists, software engineers, and other st...

Role Overview: The Distributed Data Systems Team at Dataflare is seeking talented engineers who are passionate about designing and deploying distributed data systems at PB cloud-scale. As a Team Lead, you will play a crucial role in understanding business requirements, mentoring a team of engineers, designing high-performance storage algorithms, and contributing to the foundational programming model of the data platform. Key Responsibilities: - Understand and craft solutions based on business requirements - Mentor and develop a team of engineers - Design and implement performant, parallelized, and multi-threaded storage algorithms - Contribute to the scheduling and programming model for all ...

As a Data Services Engineering Lead joining our innovative development team, you will have the opportunity to lead the design and implementation of cutting-edge data solutions that drive business intelligence and analytics. Your responsibilities will include: - Leading the design, development, and deployment of scalable Java applications integrated with Databricks for big data processing - Architecting and optimizing distributed computing solutions using Apache Spark within the Databricks environment to handle large-scalable datasets efficiently - Implementing and managing data lake house architecture using Databricks, ensuring data integrity, security, and accessibility - Developing and mai...

As a Senior Data Engineer at Enable Data Incorporated, you will be a key player in designing, developing, and maintaining scalable and efficient data solutions in the cloud. Leveraging your expertise in cloud technologies, Spark, and Databricks, you will collaborate with cross-functional teams to gather data requirements, design and implement data processing pipelines, and ensure data quality and integrity. Your role will involve optimizing data infrastructure for clients and providing advanced analytics insights. If you are passionate about using cutting-edge technologies to solve complex data challenges, this is the perfect opportunity for you. **Key Responsibilities:** - Design, develop, ...

Role & responsibilities Around 6+ years of experience in data engineering or cloud computing data development Design, develop, and optimize ETL/ELT data pipelines using Apache Spark (PySpark or Scala), AWS Glue, and Azure Data Factory Work with structured and unstructured data to build scalable ingestion and transformation workflows across cloud platforms. Build data lake and data warehouse solutions using AWS S3, Azure Data Lake Collaborate with data scientists, analysts, and application developers to support advanced analytics, reporting, and ML workflows Implement job orchestration, monitoring, and error handling for reliable pipeline execution. Maintenance and support of Data Pipeline. D...

Experience in developing and delivering scalable big data pipelines using Apache Spark and Databricks on AWS. Position Requirements : Must Have : Build and maintain scalable data pipelines using Databricks and Apache Spark. Develop and optimize ETL/ELT processes for structured and unstructured data. Knowledge of Lakehouse architecture for efficient data storage, processing, and analytics. Orchestrating ETL/ELT Pipelines: Design and manage data workflows using Databricks Workflows, Jobs API. Worked with AWS Data Services (S3, Lambda, CloudWatch) for seamless integration. Performance Optimization: Optimize queries using pushdown capabilities and indexing strategies. Worked on data governance w...

Role Overview: You will be responsible for integrating data from various siloed systems into CDF using ETL tools such as Azure Data Factory or Informatica PowerCenter. Additionally, you will transform and contextualize data within CDF using programming languages like Python and tools like Apache Spark and Databricks for data analysis and visualization. Your understanding of industrial data and processes will be crucial for successful data integration. Key Responsibilities: - Utilize ETL tools like Azure Data Factory or Informatica PowerCenter to integrate data from different systems into CDF - Write scripts in Python, especially for using the Cognite Python SDK, to transform and contextualiz...

As an experienced Azure Data Engineer with over 8 years of experience, you will play a key role in designing and implementing scalable data pipelines and ETL processes within the Azure ecosystem. Your expertise in data warehousing, data pipelines, and cloud-based data processing frameworks will be crucial for this role. Key Responsibilities: - Design and implement scalable data pipelines and ETL processes using Azure Data Factory (ADF), PySpark, and Databricks. - Manage and optimize Azure Data Lake and integrate with Azure Synapse Analytics for large-scale data storage and analytics. - Collaborate with cross-functional teams to gather requirements, design data solutions, and deliver actionab...

As a Senior Data Engineer, you will be responsible for: - Designing and implementing solutions for problems arising out of large-scale data processing - Attending/driving various architectural, design, and status calls with multiple stakeholders - Ensuring end-to-end ownership of all tasks aligned, including development, testing, deployment, and support - Designing, building, and maintaining efficient, reusable, and reliable code - Testing implementation, troubleshooting, and correcting problems - Working both as an individual contributor and within a team - Ensuring high-quality software development with complete documentation and traceability - Fulfilling organizational responsibilities by...

As a Data Engineer with 4-5 years of experience, you will be responsible for designing and developing scalable data infrastructure and systems to support critical business functions. Your role will involve building robust ETL pipelines, managing large-scale datasets, and optimizing data workflows in a fast-paced environment. Key Responsibilities: - Design, develop, and maintain scalable and reliable ETL/ELT pipelines for processing large volumes of data. - Model and structure data for performance, scalability, and usability. - Work with cloud infrastructure, preferably Azure, to optimize data workflows. - Build and manage data lake/lakehouse architectures following best practices. - Optimize...

Role Overview: As a software engineer in Corporate Planning and Management (CPM) Engineering, you will be involved in creating new financial and spend management workflows and developing intelligent reporting systems to drive commercial benefits for the firm. You will have the opportunity to work in small, agile teams and contribute to various aspects of corporate planning and management in alignment with the firm's strategic objectives. Key Responsibilities: - Be energetic, self-directed, and self-motivated, capable of building and maintaining long-term relationships with clients and colleagues - Approach problems with an open mind, working collaboratively within a team - Utilize exceptiona...

As a member of the Data Core Apps Team at Agoda, you will be part of a skilled, driven, and diverse team dedicated to making an impact in the travel industry. You will have the opportunity to work on cutting-edge technology and contribute to the development of first-class big data tools. Here is what you can expect in this role: Role Overview: - Work with a technology stack including Scala, Spark, Python, Hadoop, Kubeflow, SQL, S3 - Enhance scalability, stability, accuracy, speed, and efficiency of existing data systems - Design, build, test, and deploy new libraries, frameworks, or full systems while maintaining high standards of testing and code quality - Collaborate with experienced engin...

As a Machine Learning Engineer at Tecblic, you will be part of a forward-thinking team that specializes in delivering AI-driven solutions to empower businesses in the digital age. Your primary responsibilities will include: - Research and Development: Design, develop, and fine-tune machine learning models across LLM, computer vision, and Agentic AI use cases. - Model Optimization: Fine-tune and optimize pre-trained models for improved performance, scalability, and minimal latency. - Computer Vision: Build and deploy vision models for tasks such as object detection, classification, OCR, and segmentation. - Integration: Collaborate closely with software and product teams to integrate models in...

As a Machine Learning Scientist I at Wayfair in Bangalore, India, your role will involve designing, building, deploying, and refining large-scale machine learning/AI models and algorithmic decision-making systems to solve real-world problems for customers. You will work cross-functionally with commercial stakeholders, engineers, analysts, and product managers to develop ML solutions that drive real-world impact. Collaboration with various infrastructure and ML platform teams is essential to develop production-grade services following best practices. Additionally, you will be responsible for building cutting-edge robust ML models that directly impact customers every day and identifying new op...

You will capture user requirements and translate them into business and digitally enabled solutions across a range of industries. Your responsibilities will include: - Designing, developing, optimizing, and maintaining data pipelines that adhere to ETL principles and business goals - Solving complex data problems to deliver insights that help our business achieve its goals - Sourcing data (structured and unstructured) from various touchpoints, formatting, and organizing them into an analyzable format - Creating data products for analytics team members to improve productivity - Calling AI services like vision, translation, etc., to generate an outcome that can be used in further steps along t...

As a Software Engineering Manager at SAP, you will lead the App2App Integration team within the SAP Business Data Cloud. Your role will involve driving the design, development, and delivery of integration services that facilitate real-time, secure, and scalable data exchange across SAP's enterprise applications and unified data fabric. **Key Responsibilities:** - Own end-to-end delivery of App2App integration components and services using Java, Apache Kafka, RESTful APIs, and modern data pipelines. - Accelerate the App2App integration roadmap by identifying reusable patterns, driving platform automation, and establishing best practices. - Drive the evolution of ETL pipelines and real-time da...