1199 Apache Spark Jobs

We are seeking an experienced Databricks Developer / Data Engineer to design, develop, and optimize data pipelines, ETL workflows, and big data solutions using Databricks. The ideal candidate should have expertise in Apache Spark, PySpark, SQL, and cloud-based data platforms (Azure, AWS, GCP). This role involves working with large-scale datasets, data lakes, and data warehouses to drive business intelligence and analytics. Key Responsibilities : Design, build, and optimize ETL and ELT pipelines using Databricks and Apache Spark. Work with big data processing frameworks (PySpark, Scala, SQL) for data transformation and analytics. Implement Delta Lake architecture for data reliability, ACID tr...

As an Azure Data Architect at EY, you'll play a crucial role in leveraging your deep knowledge of data management in the cloud and big data analytics to build innovative solutions. Your responsibilities will include: - Building solutions and capabilities, and managing large teams with proven thought leadership - Hands-on experience with various Big Data analytics platforms and cloud data engineering stack - Designing and implementing robust data solutions using cloud-based technologies and expertise in data-related tools and languages - Knowledge of Cloud computing platforms such as Azure, Databricks, and Snowflake, along with Apache Spark, Apache Kafka, Splunk, and Datadog - Experience in d...

You have an exciting opportunity to join a dynamic team at a leading digital banking solutions company, Finacle. **Role Overview:** As a Senior Engineer, you will leverage your 7+ years of engineering experience to develop solutions using technologies such as SQL, ETL processes, and more. Your role will involve identifying and optimizing database queries, demonstrating strong understanding of algorithms and data structures, and applying software development methodologies like Agile and Scrum. Your ability to quickly grasp requirements, ask insightful questions, and lead a team of developers will be key to success. **Key Responsibilities:** - Develop solutions using SQL, ETL processes, and si...

As a Data Analyst at Allime Tech Solutions, you will play a crucial role in developing and managing data infrastructure within an Azure environment. Your expertise in working with Data Bricks for data processing and analysis will be essential for implementing and optimizing data solutions. Your responsibilities will include: - Developing data pipelines and managing data infrastructure within Azure - Working on end-to-end Data Product deployment - Collaborating with cross-functional teams for reporting, analysis, and MIS - Utilizing strong analytical and problem-solving skills to design effective data solutions - Utilizing data querying and processing skills using SQL, Oracle, Apache Spark, e...

As a Data Engineer with expertise in Generative AI, your role will involve designing, developing, and maintaining robust data pipelines and workflows. You will be responsible for integrating Generative AI models into existing data systems to enhance functionality. Collaborating with cross-functional teams to understand business needs and translating them into scalable data and AI solutions will be a key part of your role. Additionally, you will optimize data storage, processing, and retrieval systems for performance and scalability, ensuring data security, quality, and governance across all processes. It will also be essential for you to stay updated with the latest advancements in Generativ...

Role Overview: You are a highly motivated AI/ML Engineer specializing in RAN software optimization, joining the dynamic and international AI/ML team at Rakuten Symphony. Your primary responsibility is to develop, deploy, and maintain advanced machine learning models that enhance the efficiency, capacity, and reliability of next-generation mobile networks. Your role involves collaborating with RAN engineers, software developers, and product managers to integrate AI/ML logic directly into RAN products, contributing to the continuous improvement of Rakuten Symphony's Open RAN solutions. Key Responsibilities: - Collaborate with RAN engineering and software development teams to identify AI/ML use...

As a Snowflake Engineer - Digital Solution Consultant Sr. Analyst at NTT DATA in Hyderabad, Telangana (IN-TG), India (IN), you will be part of an inclusive, adaptable, and forward-thinking organization. Your role will involve the following responsibilities: - Experience with other cloud data warehousing solutions. - Knowledge of big data technologies (e.g., Apache Spark, Hadoop). - Experience with CI/CD pipelines and DevOps practices. - Familiarity with data visualization tools. NTT DATA is a $30 billion trusted global innovator of business and technology services. They serve 75% of the Fortune Global 100 and are committed to helping clients innovate, optimize, and transform for long-term su...

As an experienced engineer in the field, you will be responsible for various tasks and activities. Your role will involve the following key responsibilities: - Having 7+ years of relevant engineering experience - Demonstrating development expertise with at least 2 years of experience in SQL, ETL processes, and other similar technologies - Identifying and optimizing database queries - Possessing a strong understanding of Algorithms and Data Structures used in data applications - Having a solid understanding of software development methodologies such as Agile and Scrum - Quickly grasping requirements and designs, asking insightful questions to ensure a clear understanding of project goals - Ex...

Key Skills: C++, Linux, C Programming, Apache Spark Roles and Responsibilities: Collaborate with cross-functional teams to deliver innovative algorithms and techniques for our GPU-powered, parallel, and distributed execution engine. Develop enterprise-grade, reliable code and manage end-to-end delivery of solutions, ensuring smooth production rollouts. Design, implement, test, and maintain enhancements across various components of the engine, ensuring modularity and clean integration. Define and document algorithms, mechanisms, procedures, and policies, securing necessary approvals from technical leads. Continuously analyze and identify areas for improvement in the execution engine. Identify...

Develop, test, and deploy data processing applications using Apache Spark and Scala. Optimize and tune Spark applications for better performance on large-scale data sets. Work with the Cloudera Hadoop ecosystem (e.g., HDFS, Hive, Impala, HBase, Kafka) to build data pipelines and storage solutions. Collaborate with data scientists, business analysts, and other developers to understand data requirements and deliver solutions. Design and implement high-performance data processing and analytics solutions. Ensure data integrity, accuracy, and security across all processing tasks. Troubleshoot and resolve performance issues in Spark, Cloudera, and related technologies. Implement version control an...

As Data Engineer, you will develop, maintain, evaluate and test big data solutions. You will be involved in the development of data solutions using Spark Framework with Python or Scala on Hadoop and Azure Cloud Data Platform Responsibilities: Experienced in building data pipelines to Ingest, process, and transform data from files, streams and databases. Process the data with Spark, Python, PySpark and Hive, Hbase or other NoSQL databases on Azure Cloud Data Platform or HDFS Experienced in develop efficient software code for multiple use cases leveraging Spark Framework / using Python or Scala and Big Data technologies for various use cases built on the platform Experience in developing strea...

About The Role Project Role : Custom Software Engineer Project Role Description : Design, build and configure applications to meet business process and application requirements. Must have skills : PySpark Good to have skills : Apache Spark Minimum 3 year(s) of experience is required Educational Qualification : 15 years full time education Summary :As an Application Developer, you will engage in the design, construction, and configuration of applications tailored to fulfill specific business processes and application requirements. Your typical day will involve collaborating with team members to understand project needs, developing innovative solutions, and ensuring that applications are optim...

Key Responsibilities: Develop, test, and deploy data processing applications using Apache Spark and Scala. Optimize and tune Spark applications for better performance on large-scale data sets. Work with the Cloudera Hadoop ecosystem (e.g., HDFS, Hive, Impala, HBase, Kafka) to build data pipelines and storage solutions. Collaborate with data scientists, business analysts, and other developers to understand data requirements and deliver solutions. Design and implement high-performance data processing and analytics solutions. Ensure data integrity, accuracy, and security across all processing tasks. Troubleshoot and resolve performance issues in Spark, Cloudera, and related technologies. Implem...

We are seeking a highly skilled and motivated Big Data Engineer to join our data engineering team The ideal candidate will have hands-on experience with Hadoop ecosystem, Apache Spark, and programming expertise in Python (PySpark), Scala, and Java You will be responsible for designing, developing, and optimizing scalable data pipelines and big data solutions to support analytics and business intelligence initiatives

Experience: 7+ Years Location: Hyderabad Employment Type: Full-time Notice Period: Immediate to 30 Days (preferred) About the Role We are seeking a highly skilled Application Developer with strong expertise in building and optimizing large-scale Ratings Data Ingestion pipelines. The ideal candidate will have hands-on experience with AWS cloud technologies, big data frameworks, and performance tuningespecially around Java garbage collection. You will work closely with cross-functional and geographically distributed teams to design, develop, and deliver high-quality data engineering solutions. Key Responsibilities Design and develop Ratings Data Ingestion pipelines using AWS and big data techn...

Role Description Role Description:As a Technical Lead - Cloud Data Platform (AWS) at Incedo, you will be responsible for designing, deploying and maintaining cloud-based data platforms on the AWS platform. You will work with data engineers, data scientists and business analysts to understand business requirements and design scalable, reliable and cost-effective solutions that meet those requirements. Roles & Responsibilities: Designing, developing and deploying cloud-based data platforms using Amazon Web Services (AWS) Integrating and processing large amounts of structured and unstructured data from various sources Implementing and optimizing ETL processes and data pipelines Developing and m...

ABOUT THE ROLE Role Description: The role is responsible for designing, building, maintaining , analyzing, and interpreting data to provide actionable insights that drive business decisions. This role involves working with large datasets, developing reports, supporting and executing data governance initiatives and visualizing data to ensure data is accessible, reliable, and efficiently managed. The ideal candidate has strong technical skills, experience with big data technologies, and a deep understanding of data architecture and ETL processes Roles & Responsibilities: Design and maintain upstream data pipelines that deliver analytics-ready datasets optimized for Tableau reporting. Develop a...

Department Name: Enterprise Engineering and Analytics Role GCF: 4 Hiring Manager Name: Kedarnath Dixit Role Description: We are looking for highly motivated expert Data Engineer who can own the design & development of complex data pipelines, solutions and frameworks. The ideal candidate will be responsible to design, develop, and optimize data pipelines, data integration frameworks, and metadata-driven architectures that enable seamless data access and analytics. This role prefers deep expertise in big data processing, distributed computing, data modeling, and governance frameworks to support self-service analytics, AI-driven insights, and enterprise-wide data management. Roles & Responsibil...

As a Big Data Engineer, you will play a crucial role in developing and managing the Big Data solutions for our company. Your responsibilities will include designing and implementing Big Data tools and frameworks, implementing ELT processes, collaborating with development teams, building cloud platforms, and maintaining the production system. Key Responsibilities: - Meet with managers to determine the companys Big Data needs. - Develop Big Data solutions on AWS using tools such as Apache Spark, Databricks, Delta Tables, EMR, Athena, Glue, and Hadoop. - Load disparate data sets and conduct pre-processing services using Athena, Glue, Spark, etc. - Collaborate with software research and developm...

As a Software Engineer III at JPMorgan Chase within the Consumer and community banking technology team, you will be part of an agile team designing and delivering trusted market-leading technology products in a secure, stable, and scalable manner. Your role involves critical technology solutions across multiple technical areas to support the firm's business objectives. **Key Responsibilities:** - Develop and maintain applications using Apache Spark and Java - Work with different aspects of the Spark ecosystem including Datasets, Spark SQL, and Scalability support - Collaborate with business analysts and lead engineers to understand requirements and design software solutions - Implement perfo...

As a Mid/Senior Software Engineer at this company, your role will involve developing and maintaining the Data Management Platform within the Big Data Team. You will work alongside a cross-functional international Agile team comprising skilled engineers, product managers, and leaders across multiple global locations. Your primary programming language will be Java/Scala, with a focus on tasks within the Hadoop ecosystem. Key Responsibilities: - Autonomously develop well-defined tasks, test, and deploy code changes and services - Collaborate with and learn from other teammates - Follow Adform and team coding and testing practices - Actively participate in Scrum ceremonies, raise questions, and ...

As a Senior Data Engineer at our company, you will play a crucial role in designing, building, and optimizing scalable data pipelines on AWS. Your responsibilities will include implementing data ingestion, transformation, and integration solutions using Databricks, Glue, and SQL. You will also be tasked with managing and optimizing cloud storage and compute environments, as well as ensuring robust, automated deployments with Terraform and Jenkins. Collaborating with cross-functional teams to deliver high-quality data products will be a key part of your role. Key Responsibilities: - Design, build, and optimize scalable data pipelines on AWS - Implement data ingestion, transformation, and inte...

You will be responsible for designing, developing, and supporting scalable ETL processes using open source tools and data frameworks such as AWS Glue, AWS Athena, Redshift, Apache Kafka, Apache Spark, Apache Airflow, and Pentaho Data Integration (PDI). Additionally, you will design, create, and maintain data lakes and data warehouse on the AWS cloud. Your role will involve maintaining and optimizing the data pipeline architecture, formulating complex SQL queries for big data processing, collaborating with product and engineering teams for designing a platform for data modeling and machine learning operations, implementing various data structures and algorithms, ensuring data privacy and comp...

As a Cloud Architect with DevOps expertise, your role will be crucial in driving the cloud migration journey and ensuring cloud security at scale. You will be responsible for implementing automation and working with ETL DataStage. **Key Responsibilities:** - Drive cloud migration journey from ETL DataStage - Implement automation for efficient operations - Ensure cloud security at scale - Work with Cloud Architecture on AWS, Azure, and GCP - Utilize Apache Spark and Snowflake for data processing - Implement CI/CD and Infrastructure as Code practices - Follow DevOps best practices for efficient development - Utilize your experience in ETL Migration, with a preference for DataStage **Qualificat...

As a Java developer at the company, your responsibilities will include: - Working with client developers & off-shore developers to deliver project tasks - Analyzing system domain and client requirements - Collaborating with the team to create and maintain a robust system supporting business with a focus on usability - Ensuring productivity and teamwork within the constraints of a planned project to complete necessary milestones - Solving challenging real-world problems using OOP & functional programming concepts - Writing production-grade backend code, SP, and upholding high code quality standards You must have the following qualifications: - Very strong knowledge in OOP and Core Java - Inhe...