124 Spark Sql Jobs

We are looking for a proactive and detail-oriented Data Support Engineer to join our growing team in Visakhapatnam . The ideal candidate should possess strong analytical and debugging skills, with hands-on experience in resolving production issues related to ADF, Snowflake, and Databricks (Spark SQL) . You will be responsible for monitoring, troubleshooting, and resolving daily production bugs/issues, while coordinating with cross-functional and US-based teams. Key Responsibilities: Identify, analyze, and resolve production issues across ADF, Snowflake, and Databricks environments. Perform SQL debugging and data validation to ensure smooth data flow across pipelines. Collaborate with interna...

Role Overview: As a seasoned Spark Developer with 5+ years of experience, you have a deep understanding of developing, testing, deploying, and debugging Spark Jobs using Scala on the Hadoop Platform. Your expertise includes proficiency in Spark Core and working with RDDs and Spark SQL, knowledge of Spark Optimization Techniques and Best practices, familiarity with Scala Functional Programming concepts like Try, Option, Future, and Collections, understanding of Scala OOPS principles including Classes, Traits, Objects (Singleton and Companion), and Case Classes, strong grasp of Scala Language Features such as Type System and Implicit/Givens, hands-on experience in Hadoop Environment (HDFS/Hive...

As a Sr. Data Engineer at NTT DATA in Bangalore, Karnataka (IN-KA), India, your primary responsibility is to build and implement PySpark-based data pipelines in Azure Synapse to transform and load data into ADLS in Delta format. You will design and implement dimensional (star/snowflake) and 3NF data models optimized for access using Power BI. Additionally, you will perform unit testing of data pipelines and transformations, design metadata-driven data pipelines using PySpark in Azure Synapse, and analyze and optimize Spark SQL queries. Collaborating with cross-functional teams to ensure data models align with business needs and maintaining version control and CI/CD pipelines in Git and Azure...

As a Senior Developer at TransUnion, you will play a crucial role in designing, developing, and implementing cutting-edge solutions using a variety of technologies such as Java, Big Data frameworks, AI/ML concepts, and cloud platforms. Your contributions will be vital to the architecture, development, and deployment of data-driven applications. Key Responsibilities: - Design, develop, and maintain high-performance applications utilizing Java and related technologies. - Work with Big Data technologies like Hadoop, HDFS, Spark, and Scala to process and analyze large datasets. - Develop data pipelines for ingestion, transformation, and storage. - Write, optimize, and debug SQL queries across pl...

Notice Period-Immediate Joiner or Max 10 days (Do not share long notice period profile.) Permanent Payroll - Anlage Infotech Client - Orion Innovation Location- Kochi, Chennai, Coimbatore, Pune - HYBRID (2/3 days in a week) Role & responsibilities Work Location: Kochi, Chennai, Coimbatore, Pune Hybrid Mode Mandate skills - Pyspark + Azure DataBricks + Azure Data Factory + MS Fabric Experience- 5+ years We are seeking a skilled and experienced Senior Data Engineer with a strong background in Azure Data Services, particularly Azure Databricks, Azure Data Factory, PySpark and MS Fabric. The ideal candidate will be responsible for designing, developing, and optimizing scalable data pipelines and...

Notice Period-Immediate Joiner or Max 10 days (Do not share long notice period profile.) Permanent Payroll - Anlage Infotech Client - Orion Innovation Location- Kochi - Chennai - Coimbatore - HYBRID (2/3 days in a week) Role & responsibilities Bachelors or Masters degree in Computer Science, Engineering, Information Systems, or a related field. 5+ years of hands-on experience in data engineering roles. Strong expertise in: Azure Databricks Azure Data Factory PySpark Experience working with Azure Data Lake Storage (ADLS) and Delta Lake. Strong SQL skills for data extraction and transformation. Experience with version control tools like Git and CI/CD practices. Familiarity with orchestration, ...

As a Graph Database Developer, you will be responsible for designing, developing, and maintaining graph database schemas using Neo4j. You will optimize Neo4j queries for performance and efficiency. Utilizing Python, PySpark, or Spark SQL, you will conduct data transformation and analysis. Your role will also involve conducting User Acceptance Testing (UAT) to ensure data accuracy and overall system functionality. Additionally, you will develop and manage scalable data pipelines using Databricks and Azure Data Factory (ADF). Working with Azure cloud services and being familiar with Azure data engineering components will be a crucial part of your responsibilities. Key Responsibilities: - Desig...

As a Machine Learning Scientist I at Wayfair in Bangalore, India, your role will involve designing, building, deploying, and refining large-scale machine learning/AI models and algorithmic decision-making systems to solve real-world problems for customers. You will work cross-functionally with commercial stakeholders, engineers, analysts, and product managers to develop ML solutions that drive real-world impact. Collaboration with various infrastructure and ML platform teams is essential to develop production-grade services following best practices. Additionally, you will be responsible for building cutting-edge robust ML models that directly impact customers every day and identifying new op...

As a Senior Engineer at Impetus Technologies, you will be responsible for designing, developing, and deploying scalable data processing applications using Java and Big Data technologies. Your role will involve collaborating with cross-functional teams to gather requirements, develop high-quality code, and optimize data processing workflows. Additionally, you will mentor junior engineers and contribute to architectural decisions to enhance system performance and scalability. Key Responsibilities: - Design, develop, and maintain high-performance applications using Java and Big Data technologies. - Implement data ingestion and processing workflows utilizing frameworks like Hadoop and Spark. - C...

You will be responsible for architecting, building, and optimizing scalable data pipelines and platforms to drive critical business decisions. Your expertise in PySpark, ETL frameworks, SQL Stored Procedures, and Data Modeling will be crucial for performance tuning across large datasets. - Lead the design and development of scalable data pipelines and ETL workflows using PySpark and SQL - Develop and maintain Stored Procedures for batch processes and data transformations - Design and implement robust data models (Star, Snowflake, and normalized models) for data warehouse and lake environments - Collaborate with BI, Analytics, and Product teams to understand data requirements and ensure data ...

Job Description: As a Spark Expert, you will be expected to demonstrate expert proficiency in Spark and possess the ability to design and implement efficient data processing workflows. Your experience with Spark SQL and DataFrames will be crucial, as well as your good exposure to Big Data architectures and a solid understanding of the Big Data ecosystem. Additionally, your experience with framework building on Hadoop and strong DB knowledge with SQL tuning experience will be valuable. It would be beneficial if you have experience with Python APIs and exposure to Kafka. Key Responsibilities: - Demonstrate expert proficiency in Spark - Design and implement efficient data processing workflows -...

Role Overview: At Derevo, we empower companies and individuals by unlocking the value of data within organizations. With over 15 years of experience, we design end-to-end data and AI solutions - from integration into modern architectures to implementing intelligent models in key business processes. We are currently seeking a talented Senior Data Engineer specialized in Databricks to join our team. Key Responsibilities: - Design and implement modern data architectures with high quality, driving analytical solutions based on Big Data technologies. - Maintain and optimize parallel processing systems, applying best practices for storage and management in data warehouses, data lakes, and lakehous...

As a Data Engineer at Data Bricks in Melbourne, your role will involve working with a tech stack that includes Data Lake, Azure Data Bricks, and other Big Data Technologies. You will be responsible for designing and deploying Databricks platforms on AWS, Azure, or Google Cloud. Your key responsibilities will include building and deploying data engineering pipelines into production, following automation best practices for CI/CD. Additionally, you will guide clients in implementing transformational big data projects and developing industry-leading big data and AI applications. You will also streamline the customer machine learning lifecycle and integrate it with the data platform. Key Responsi...

Role Overview: You will be joining the ICG TTS Operations Technology (OpsTech) Group to assist in the implementation of the next generation Digital Automation Platform and Imaging Workflow Technologies. Your role as a Lead Spark Scala Engineer will involve managing development teams in the distributed systems Eco-System and collaborating effectively as a team player. Your primary objective will be to contribute to applications, systems analysis, and programming activities. Key Responsibilities: - Develop, test, and deploy production-grade Spark applications in Scala, ensuring optimal performance, scalability, and resource utilization. - Provide technical leadership and guidance to a team of ...

Role Overview: As a Senior Azure Data Engineer, you will utilize Azure technologies to facilitate data transformation, analytics, and machine learning. Your primary responsibility will involve designing scalable Databricks data pipelines using PySpark to convert raw data into actionable insights. Additionally, you will be tasked with constructing, deploying, and managing machine learning models using MLlib or TensorFlow, all while optimizing cloud data integration from various Azure sources such as Blob Storage, Data Lake, and SQL/NoSQL databases. Your role will also encompass executing large-scale data processing using Spark Pools and fine-tuning configurations for efficiency. Key Responsib...

Role Overview: You will be working as a Lead Spark Scala Engineer under ICG TTS Operations Technology (OpsTech) Group to help in the implementation of the next generation Digital Automation Platform and Imaging Workflow Technologies. Your role will involve managing development teams in the distributed systems Eco-System and being a strong team player. It is expected that you will have superior technical knowledge of current programming languages, technologies, and other leading-edge development tools to contribute to applications, systems analysis, and programming activities. Key Responsibilities: - Develop, test, and deploy production-grade Spark applications in Scala, ensuring optimal perf...

Role Overview: You will be working as a Solutions Architect at Koantek, collaborating with customers to design scalable data architectures using Databricks technology and services. Your role will involve building secure, highly scalable big data solutions, focusing on achieving tangible, data-driven outcomes while ensuring simplicity and operational effectiveness. With your technical expertise and business acumen, you will engage in complex technology discussions, demonstrating the value of the Databricks platform throughout the sales process. Collaborating with Account Executives, you will interact with customers" technical leaders to become a trusted advisor who delivers concrete results. ...

As a skilled Senior Engineer at Impetus Technologies, you will utilize your expertise in Java and Big Data technologies to design, develop, and deploy scalable data processing applications. Your responsibilities will include collaborating with cross-functional teams, developing high-quality code, and optimizing data processing workflows. Additionally, you will mentor junior engineers and contribute to architectural decisions to enhance system performance and scalability. Key Responsibilities: - Design, develop, and maintain high-performance applications using Java and Big Data technologies. - Implement data ingestion and processing workflows with frameworks like Hadoop and Spark. - Collabora...

As a Pre-Sales Solution Engineer in India, your role will involve engaging with customers in the evaluation process, conducting product demonstrations, and supporting marketing activities. Your main focus will be on pre-sales experience with software or analytics products, demonstrating expertise in various technical skills and tools such as: - OLAP - Microsoft Analysis services - Data engineering - ETL - Hadoop ecosystem - AWS, Azure, GCP Cluster - Tableau, Microstrategy, BI tools - Hive QL, Spark SQL, PLSQL, TSQL, SQL programming - MDX queries - Linux, Python, Java, and JavaScript You will act as the primary point of contact for customer interactions, handle product demonstrations for pote...

You're ready to gain the skills and experience needed to grow within your role and advance your career - and we have the perfect software engineering opportunity for you. As a Software Engineer III at JPMorgan Chase within the Consumer & Community Banking Team, you are part of an agile team that works to enhance, design, and deliver the software components of the firm's state-of-the-art technology products in a secure, stable, and scalable way. As an emerging member of a software engineering team, you execute software solutions through the design, development, and technical troubleshooting of multiple components within a technical product, application, or system, while gaining the skills and...

Role: Senior Member of Technical Staff Experience: 4 to 9 years Location: Manyata Tech Park, Bangalore Work mode: Hybrid Skills: Design Data Analytics, Data Engineering, Data Warehouse, Java, SQL, Spark & Spark-SQL, Autonomous Data Warehouse, Object Store, Data Flow/DIS (or similar tech stack), exposure to cloud Short description : Looking for a Software Developer (Data Engineer) who are willing to perform Data Engineering tasks with skills sets like SQL, Spark, Big Data Development, Java & Cloud experience. Job description: Design, develop, troubleshoot and debug software programs for databases, applications etc. Strong knowledge of SQL is required. You should be having programming experien...

As a PySpark Developer at our company, you will be responsible for developing and maintaining data processing pipelines using PySpark, Apache Spark's Python API. You will collaborate with data engineers, data scientists, and other stakeholders to design and implement scalable and efficient data processing solutions. Key Responsibilities: - Designing, developing, and maintaining PySpark data processing pipelines for processing structured and unstructured data - Collaborating with data engineers and data scientists to understand data requirements and design efficient data models and transformations - Optimizing and tuning PySpark jobs for performance, scalability, and reliability - Implementin...

As a Data Engineer specializing in supply chain applications at NovintiX in Coimbatore, India, you will play a crucial role in enhancing our Supply Chain Analytics team. Your primary focus will be on developing intelligent data solutions that drive real-world logistics, procurement, and demand planning. Your responsibilities will include: - Creating and optimizing scalable data pipelines for inventory, shipping, and procurement data - Integrating data from ERP, PLM, and external sources through the development of APIs - Designing, building, and maintaining enterprise-grade data warehouses and data lakes while ensuring data quality, integrity, and security - Collaborating with stakeholders to...

As an AWS Data Engineer, you will be responsible for designing, building, and optimizing data pipelines and scalable data assets. Your key responsibilities will include: - Designing, building, and optimizing data pipelines and scalable data assets. - Developing and maintaining high-performance code using PySpark/Python with best practices. - Performing code optimization using Spark SQL and PySpark for efficiency. - Implementing code refactoring to modernize legacy codebases for readability and maintainability. - Writing and maintaining unit tests (TDD) to ensure code reliability and reduce bugs. - Debugging and resolving complex issues including performance, concurrency, and logic flaws. - W...

- Role Overview: You have a minimum of 5 years of experience in Bengaluru or Gurugram. Your expertise should include Python, Airflow (Orchestration), GCP (Cloud), Spark SQL, PySpark, CI/CD, Git, and GitHub. Your responsibilities will involve designing and constructing data models, developing and managing data ingestion and processing systems, implementing data storage solutions, ensuring data consistency and accuracy, and collaborating with cross-functional teams to address data-related issues. Proficiency in Python programming and experience with GCP and Airflow are necessary. You should also be familiar with security and governance aspects, such as role-based access control and data lineag...