61 Spark Programming Jobs

We are seeking a highly skilled and experienced Senior Data Engineer with strong expertise in Streaming Data and Big Data Technologies , AWS & Scala Mandatory . The ideal candidate will have hands-on experience in architecting and implementing scalable data solutions using modern cloud-native tools and frameworks. Key Responsibilities: Architect and implement scalable data pipelines for real-time, batch, structured, and unstructured data . Design and develop solutions using AWS services such as EMR, Kinesis, Glue, Athena, S3, CloudFormation, API Gateway . Work extensively with streaming platforms like Kafka, Flink, Spark Streaming . Develop and optimize data ingestion workflows using Apache ...

As a Data Engineer at IBM, you'll play a vital role in the development, design of application, provide regular support/guidance to project teams on complex coding, issue resolution and execution. Your primary responsibilities include: Lead the design and construction of new solutions using the latest technologies, always looking to add business value and meet user requirements. Strive for continuous improvements by testing the build solution and working under an agile framework. Discover and implement the latest technologies trends to maximize and build creative solutions Required education Bachelor's Degree Preferred education Master's Degree Required technical and professional expertise Ex...

We are seeking a highly skilled and experienced Senior Data Engineer with strong expertise in Streaming Data and Big Data Technologies , AWS & Scala Mandatory . The ideal candidate will have hands-on experience in architecting and implementing scalable data solutions using modern cloud-native tools and frameworks. Key Responsibilities: Architect and implement scalable data pipelines for real-time, batch, structured, and unstructured data . Design and develop solutions using AWS services such as EMR, Kinesis, Glue, Athena, S3, CloudFormation, API Gateway . Work extensively with streaming platforms like Kafka, Flink, Spark Streaming . Develop and optimize data ingestion workflows using Apache ...

About The Role Project Role : Data Engineer Project Role Description : Design, develop and maintain data solutions for data generation, collection, and processing. Create data pipelines, ensure data quality, and implement ETL (extract, transform and load) processes to migrate and deploy data across systems. Must have skills : PySpark Good to have skills : NA Minimum 5 year(s) of experience is required Educational Qualification : 15 years full time education Summary :As a Data Engineer, you will design, develop, and maintain data solutions that facilitate data generation, collection, and processing. Your typical day will involve creating data pipelines, ensuring data quality, and implementing...

About The Role Project Role : Data Engineer Project Role Description : Design, develop and maintain data solutions for data generation, collection, and processing. Create data pipelines, ensure data quality, and implement ETL (extract, transform and load) processes to migrate and deploy data across systems. Must have skills : PySpark Good to have skills : NA Minimum 5 year(s) of experience is required Educational Qualification : 15 years full time education Summary :As a Data Engineer, you will design, develop, and maintain data solutions that facilitate data generation, collection, and processing. Your typical day will involve creating data pipelines, ensuring data quality, and implementing...

Key Responsibilities Design, develop, and maintain robust ETL/ELT pipelines using SQL, dbt, and orchestration tools. Architect and implement data warehouse and lakehouse solutions leveraging Snowflake, Databricks, and cloud-native services. Work on data modeling, schema design, and query optimization for performance and scalability. Migrate and integrate legacy/on-prem data platforms to cloud environments (Azure or AWS). Implement data governance, quality, and security practices to ensure reliability and compliance. Collaborate with data scientists, BI developers, and business stakeholders to deliver analytics-ready datasets. Optimize pipelines and storage for cost efficiency and cloud-nativ...

Knowledge of setting up operating & maintaining wire EDM machine to produce parts & components. Set up & operating of EDM equipment to ensure proper making precision parts. Hand of experience in operating wire EDM machine, programming knowledge Required Candidate profile Shall be responsible for executing the job as per drawing, GD & T knowledge, Knowledge of 5S, safety & autonomous maintenance. Hands on experience in wire-cut, EDM & Spark Machine.

Minimum 6+ years of experience in the fundamental technology work of architecting / designing / building / managing / Optimizing the core frameworks for our cloud-based system platforms and end-to-end solutions. Responsible for optimization of cloud infrastructure, including Databricks , ADLS, Blob, VMs etc., Conduct Proof-of-Concept activities in support of advanced use cases. Must have architectural focus for ensuring availability, reliability and scalability are necessary Responsible for documenting the technical solutions Administer and manage Azure Databricks workspaces, clusters, and jobs with a focus on maximizing efficiency and performance. Implement and manage Unity Catalog to provi...

Streaming data Technical skills requirements :- Experience- 5+ Years Solid hands-on and Solution Architecting experience in Big-Data Technologies (AWS preferred) - Hands on experience in: AWS Dynamo DB, EKS, Kafka, Kinesis, Glue, EMR - Hands-on experience of programming language like Scala with Spark. - Good command and working experience on Hadoop Map Reduce, HDFS, Hive, HBase, and/or No-SQL Databases - Hands on working experience on any of the data engineering analytics platform (Hortonworks Cloudera MapR AWS), AWS preferred - Hands-on experience on Data Ingestion Apache Nifi, Apache Airflow, Sqoop, and Oozie - Hands on working experience of data processing at scale with event driven syste...

Minimum 6+ years of experience in the fundamental technology work of architecting / designing / building / managing / Optimizing the core frameworks for our cloud-based system platforms and end-to-end solutions. Responsible for optimization of cloud infrastructure, including Databricks , ADLS, Blob, VMs etc., Conduct Proof-of-Concept activities in support of advanced use cases. Must have architectural focus for ensuring availability, reliability and scalability are necessary Responsible for documenting the technical solutions Administer and manage Azure Databricks workspaces, clusters, and jobs with a focus on maximizing efficiency and performance. Implement and manage Unity Catalog to provi...

About The Team/Role As a Senior Staff SWE of Data Lake House Engineering, this technical leader will be responsible for the design and implementation of the Data Lake house platform (supporting both Data Engineering and Data Lake house applications), Data Engineering pipeline productionalization (E2E data pipelines, model development, deployment, monitoring, refresh, etc), Data Lake house, and related technology, systems, and tools. This technologist will drive the technology development and architecture in these areas to ensure the platforms, systems, tools, models, and services generated from this team are technically sound with strong security, quality, reliability, usability, scalability...

As an Applications Development Senior Programmer Analyst, your role involves participating in the establishment and implementation of new or revised application systems and programs in coordination with the Technology team. You will contribute to applications systems analysis and programming activities. Key Responsibilities: - Conduct feasibility studies, time and cost estimates, IT planning, risk technology, applications development, and establish new or revised applications systems and programs to meet specific business needs - Monitor and control all phases of the development process including analysis, design, construction, testing, and implementation, providing user and operational supp...

Role Overview: As an Applications Development Senior Programmer Analyst, you will be an intermediate level individual contributor responsible for participating in the establishment and implementation of new or revised application systems and programs. Your main objective will be to contribute to applications systems analysis and programming activities, focusing on hands-on application development in Java technologies. Your expertise in applications development will be utilized to analyze complex problems, evaluate business processes, and adhere to industry standards. Key Responsibilities: - Participate in hands-on application development and application management activities in Java and rela...

Streaming data Technical skills requirements :- Experience- 5+ Years Solid hands-on and Solution Architecting experience in Big-Data Technologies (AWS preferred) - Hands on experience in: AWS Dynamo DB, EKS, Kafka, Kinesis, Glue, EMR - Hands-on experience of programming language like Scala with Spark. - Good command and working experience on Hadoop Map Reduce, HDFS, Hive, HBase, and/or No-SQL Databases - Hands on working experience on any of the data engineering analytics platform (Hortonworks Cloudera MapR AWS), AWS preferred - Hands-on experience on Data Ingestion Apache Nifi, Apache Airflow, Sqoop, and Oozie - Hands on working experience of data processing at scale with event driven syste...

Job Title: Principal Data Engineer, Logistics Employment Type: Full Time Experience: 12+ Years About the Role: We are looking for a Principal Data Engineer to lead the design and delivery of scalable data solutions using Azure Data Factory and Azure Databricks. This is a consulting-focused role that requires strong technical expertise, stakeholder engagement, and architectural thinking. You will work closely with business, functional, and technical teams to define data strategies, design robust pipelines, and ensure smooth delivery in an Agile environment. Responsibilities Collaborate with business and technology stakeholders to gather and understand data needs Translate functional requireme...

Ready to shape the future of work At Genpact, we don&rsquot just adapt to change&mdashwe drive it. AI and digital innovation are redefining industries, and we&rsquore leading the charge. Genpact&rsquos , our industry-first accelerator, is an example of how we&rsquore scaling advanced technology solutions to help global enterprises work smarter, grow faster, and transform at scale. From large-scale models to , our breakthrough solutions tackle companies most complex challenges. If you thrive in a fast-moving, tech-driven environment, love solving real-world problems, and want to be part of a team that&rsquos shaping the future, this is your moment. Genpact (NYSE: G) is an advanced technology ...

Responsibilities * Design, develop, test, and deploy Spark Streaming applications using Scala/Spark Core. * Collaborate with cross-functional teams to gather requirements and deliver high-quality solutions. * Troubleshoot issues in Spark Streaming applications and optimize performance for improved efficiency. * Stay up-to-date with the latest developments in Spark Streaming and contribute to its evolution. Job Requirements * 3-10 years of experience in developing Spark Streaming applications using Scala/Spark Core. * Strong understanding of Spark programming concepts, including RDDs, Dataframes, Datasets, etc. * Experience with big data processing technologies such as Hadoop/Hive/Pig is a pl...

As a DevOps Pipeline Optimization Specialist (ETL) in our team, you will be responsible for optimizing and enhancing ETL pipelines within a DataLake application environment. Your role will involve identifying and resolving bottlenecks in managing multiple ETL instances, optimizing runtime and resource consumption, and ensuring seamless integration and regression testing. You will analyze the existing DevOps pipeline to identify areas for performance improvement, optimize ETL pipeline execution, and refactor and enhance existing code to improve efficiency. Collaboration with ETL teams for regression testing and validation, addressing data validation and optimization issues, preparing technica...

You will be responsible for designing and implementing highly performant algorithms to process, transform, and analyze large volumes of data. You will apply advanced DSA concepts like Trees, Graphs, Tries, Heaps, and Hashing for data indexing, filtering, and routing. Additionally, you will develop and optimize data pipelines, stream processors, or caching systems, and architect scalable systems for data ingestion, storage, and retrieval (structured/unstructured). Collaboration with cross-functional teams to integrate and deploy performant services will be a key part of your role. You will also perform profiling, tuning, and memory optimization to ensure low-latency operations. Writing clean,...

Long Description Experienceand Expertise inany of the followingLanguagesat least 1 of them : Java, Scala, Python Experienceand expertise in SPARKArchitecture Experience in the range of 6-10 yrs plus Good Problem SolvingandAnalytical Skills Ability to Comprehend the Business requirementand translate to the Technical requirements Good communicationand collaborative skills with fellow teamandacross Vendors Familiar with development of life cycle includingCI/CD pipelines. Proven experienceand interested in supportingexistingstrategicapplications Familiarity workingwithagile methodology Mandatory Skills: Scala programming.: Experience: 5-8 Years.

Project Role : Data Platform Engineer Project Role Description : Assists with the data platform blueprint and design, encompassing the relevant data platform components. Collaborates with the Integration Architects and Data Architects to ensure cohesive integration between systems and data models. Must have skills : Databricks Unified Data Analytics Platform Good to have skills : NAMinimum 5 year(s) of experience is required Educational Qualification : A Engineering graduate preferably Computer Science graduate 15 years of full time education Summary :Overall 7+ years of experience In Industry including 4 Years of experience As Developer using Big Data Technologies like Databricks/Spark and ...

As an Azure API Developer in a senior developer delivery role based in Mumbai, Pune, or Bangalore, you will be responsible for implementing and managing Azure API solutions. Your key skill set should include proficiency in Azure API management, Azure functions, Azure storage, security, infrastructure as code, clusters, jobs, containers, workspaces, Python programming, RESTful API design, OAuth, Swagger, CI/CD engineering with Azure DevOps, Docker, Kubernetes, Spark programming (PySpark), SQL, and Git. Additionally, strong presentation and communication skills when interacting with stakeholders are essential. It would be beneficial to have experience with Databricks Workspace, notebooks, log ...

As a Senior Staff Software Engineer in Data Lake House Engineering, you will play a crucial role in designing and implementing the Data Lake house platform, supporting both Data Engineering and Data Lake house applications. Your responsibilities will include overseeing Data Engineering pipeline productionalization, end-to-end data pipelines, model development, deployment, monitoring, refresh, etc. Additionally, you will be involved in driving technology development and architecture to ensure the platforms, systems, tools, models, and services meet the technical standards for security, quality, reliability, usability, scalability, performance, efficiency, and operability to meet the evolving ...

Expertise in Java/Python, Scala & SPARK Architecture Ability to Comprehend the Business requirement & translate to the Technical requirements Familiar with development of life cycle including CI/CD pipelines Familiarity working with agile methodology Required Candidate profile Experience with big-data technologies Spark/Databricks and Hadoop/ADLS is a must Experience in any one of the cloud platform Azure (Preferred), AWS or Google

The Applications Development Senior Programmer Analyst position entails participating in the establishment and implementation of new or revised application systems and programs in collaboration with the Technology team. Your main objective in this role is to contribute to applications systems analysis and programming activities. Responsibilities include conducting tasks related to feasibility studies, time and cost estimates, IT planning, risk technology, applications development, model development, and establishing and implementing new or revised applications systems and programs to meet specific business needs or user areas. You will be responsible for monitoring and controlling all phases...