126 Spark Sql Jobs - Page 3

As a Data Engineer at our company, you will play a crucial role in our data team's growth and success. With 6 to 10 years of experience in data engineering, proficiency in SQL, Databricks, Spark SQL, PySpark, and BI tools like Power BI or Tableau, you are well-equipped to take on the responsibilities that come with this position. Your primary responsibilities will include designing and developing scalable data pipelines using PySpark and Databricks, crafting efficient SQL and Spark SQL queries for data transformation and analysis, and collaborating closely with BI teams to facilitate reporting through Power BI or Tableau. Additionally, you will be tasked with optimizing the performance of bi...

About the Role: Looking for a Principal Data Engineer to lead the design and delivery of scalable data solutions using Azure Data Factory and Azure Data bricks. This is a consulting- focused role that requires strong technical expertise, stakeholder engagement, and architectural thinking. You will work closely with business, functional, and technical teams to define data strategies, Design robust pipelines, and ensure smooth delivery in an Agile environment. Responsibilities Collaborate with business and technology stakeholders to gather and understand data needs Translate functional requirements into scalable and maintainable data architecture Design and implement robust data pipelines Lead...

You have an exciting opportunity to join a dynamic and high-impact firm as a Python, PySpark, and SQL Developer with 8-12 years of relevant experience. In this role, you will be responsible for working on development activities, collaborating with cross-functional teams, and designing scalable data pipelines using Python and PySpark. You will also be involved in implementing ETL processes, developing Power BI reports and dashboards, and optimizing data pipelines for performance and reliability. The ideal candidate should have 8+ years of experience in Spark, Scala, and PySpark for big data processing. Proficiency in Python programming for data manipulation and analysis is essential, along wi...

We have an exciting and rewarding opportunity for you to take your software engineering career to the next level. As a Software Engineer III at JPMorgan Chase within the Corporate Technology Consumer and Community Banking Risk Technology team, you serve as a seasoned member of an agile team to design and deliver trusted market-leading technology products in a secure, stable, and scalable way. You are responsible for carrying out critical technology solutions across multiple technical areas within various business functions in support of the firm's business objectives. You will be responsible for executing software solutions, design, development, and technical troubleshooting with the ability...

You will be responsible for designing, developing, and maintaining graph database schemas using Neo4j. Your role will involve optimizing Neo4j queries to enhance performance and efficiency. Additionally, you will utilize Python, PySpark, or Spark SQL for data transformation and analysis. Conducting User Acceptance Testing (UAT) to ensure data accuracy and overall system functionality will be a part of your duties. You will also be involved in developing and managing scalable data pipelines using Databricks and Azure Data Factory (ADF) for efficient data pipeline management. Working with Azure cloud services and having familiarity with Azure data engineering components will be essential for c...

NTT DATA is looking for a Sr. Data Modeler to join their team in Bangalore, Karntaka (IN-KA), India. As a Sr. Data Modeler, your primary responsibility will be to design and implement dimensional (star/snowflake) and 3NF data models that are optimized for analytical and reporting needs in Azure Synapse and Power BI. You will also be required to perform STM (source to target mapping) from the data source to multiple layers in the data lake. Additionally, you will analyze and optimize Spark SQL queries and collaborate with cross-functional teams to ensure that the data models align with business requirements. The ideal candidate should have at least 7 years of experience in SQL and PySpark. Ha...

You are an experienced PySpark ETL Lead responsible for driving data integration efforts in analytics and data warehousing projects. Your role includes developing and managing PySpark scripts, creating ETL workflows, and ensuring efficient data processing and integration across systems. You should be strong in PySpark and Python for ETL and data processing, with experience in ETL pipeline design, data integration, and data warehousing. Proficiency in SQL and working with large datasets is required, along with familiarity with workflow schedulers such as Airflow and Cron. Hands-on experience with Big Data tools like Hive, HDFS, and Spark SQL is essential. It would be beneficial if you have ex...

As a Senior Azure Data Engineer, your responsibilities will include: Building scalable data pipelines using Databricks and PySpark Transforming raw data into usable business insights Integrating Azure services like Blob Storage, Data Lake, and Synapse Analytics Deploying and maintaining machine learning models using MLlib or TensorFlow Executing large-scale Spark jobs with performance tuning on Spark Pools Leveraging Databricks Notebooks and managing workflows with MLflow Qualifications: Bachelors/Masters in Computer Science, Data Science, or equivalent 7+ years in Data Engineering, with 3+ years in Azure Databricks Strong hands-on in: PySpark, Spark SQL, RDDs, Pandas, NumPy, Delta Lake Azur...

We are seeking an experienced PySpark Developer / Data Engineer to design, develop, and optimize big data processing pipelines using Apache Spark and Python (PySpark). The ideal candidate should have expertise in distributed computing, ETL workflows, data lake architectures, and cloud-based big data solutions. Key Responsibilities: Develop and optimize ETL/ELT data pipelines using PySpark on distributed computing platforms (Hadoop, Databricks, EMR, HDInsight). Work with structured and unstructured data to perform data transformation, cleansing, and aggregation. Implement data lake and data warehouse solutions on AWS (S3, Glue, Redshift), Azure (ADLS, Synapse), or GCP (BigQuery, Dataflow). Op...

About the Roe: Grade Leve (for interna use): 11 The Team: Our team is responsibe for the design, architecture, and deveopment of our cient facing appications using a variety of toos that are reguary updated as new technoogies emerge. You wi have the opportunity every day to work with peope from a wide variety of backgrounds and wi be abe to deveop a cose team dynamic with coworkers from around the gobe. The Impact: The work you do wi be used every singe day, its the essentia code you write that provides the data and anaytics required for crucia, daiy decisions in the capita and commodities markets. Whats in it for you: Buid a career with a goba company. Work on code that fues the goba financ...

You are a PySpark Developer with over 7 years of experience and expertise in Reltio MDM. As a part of the Data Engineering team, your primary responsibility will be to design, build, and optimize scalable data pipelines, ensuring seamless integration with Reltio's cloud-native MDM platform. This is an immediate requirement, and we are looking for candidates who can join us promptly. Your key responsibilities will include designing, developing, and maintaining scalable data pipelines using PySpark in distributed computing environments like AWS EMR and Databricks. You will also be responsible for integrating and synchronizing data between enterprise systems and the Reltio MDM platform, impleme...

Role & responsibilities: Key Responsibilities: Design, develop, and optimize Java-based backend services (Spring Boot / Microservices) for API integrations. Develop and maintain Spark SQL queries and data processing pipelines for large-scale data ingestion. Build Spark batch and streaming jobs to land raw data from multiple vendor APIs into data lakes or warehouses. Implement robust error handling, logging, and monitoring for data pipelines. Collaborate with cross-functional teams across geographies to define integration requirements and deliverables. Troubleshoot and optimize Spark SQL for performance and cost efficiency. Participate in Agile ceremonies, daily standups, and client discussio...

Required Skills: Design and develop data pipelines using Azure Data Factory, Spark SQL, PySpark, and Python Implement ETL/ELT processes for both structured and unstructured data Optimize queries and data models in Azure Synapse Analytics and OneLake Build and maintain Lakehouse and Datawarehouse architectures using Microsoft Fabric Develop interactive dashboards and reports in Power BI (DirectLake mode) Integrate Microsoft Purview for data governance and compliance Work with DevOps teams to automate deployment of Fabric assets via Azure DevOps and CI/CD pipelines Ensure data security and performance tuning for large-scale datasets Role & responsibilities

As a software developer, you will be working in a constantly evolving environment driven by technological advances and the strategic direction of the organization you are employed by. Your primary responsibilities will include creating, maintaining, auditing, and enhancing systems to meet specific needs, often based on recommendations from systems analysts or architects. You will be tasked with testing both hardware and software systems to identify and resolve system faults. Additionally, you will be involved in writing diagnostic programs and designing and developing code for operating systems and software to ensure optimal efficiency. In situations where necessary, you will also provide re...

As a candidate for the role, you are expected to have a thorough understanding of collaboration in software development teams and shared software repositories and their concepts. Additionally, you should possess experience with Jira for effective project management. You should also have practical experience with at least one of the following tools for reporting and analysis: SSIS, SSAS, SSRS, or Power BI. Your expertise in using these tools will be crucial for generating insights and reports. In terms of Azure Data Factory and Synapse pipelines development, you are required to have a comprehensive understanding of how Data Factory pipelines function and their integration with other Azure ser...

As a Principal Architect - Data & Cloud at Quantiphi, you will bring your 14-20 years of experience in Technical, Solutioning, and Analytical roles to lead the way in architecting, designing, and implementing end-to-end data pipelines and data integration solutions for structured and unstructured data sources and targets. With a focus on Cloud platforms such as GCP, AWS, and Azure, you will be responsible for building and managing Data Lakes, Data Warehouse, Data Integration, Data Migration, and Business Intelligence/Artificial Intelligence solutions. Your role will involve understanding business requirements and translating them into functional and non-functional areas, defining boundaries ...

About the Company At Tide, we are dedicated to creating a business management platform that aims to streamline operations for small businesses, enabling them to save valuable time and resources. Our services include offering business accounts, banking solutions, as well as a range of integrated administrative tools spanning from invoicing to accounting. Established in 2017, Tide has garnered a user base of over 1 million small businesses globally, catering to SMEs in the UK, India, and Germany. Headquartered in central London, we also have offices in Sofia, Hyderabad, Delhi, Berlin, and Belgrade, with a team of more than 2,000 employees. Tide is on a trajectory of rapid growth, continuously ...

As a Spark and Scala Developer at Infosys, you will play a crucial role in facilitating digital transformation for our clients within a global delivery model. Your responsibilities will include conducting independent research on technologies, providing recommendations for suitable solutions, and contributing to technology-specific best practices and standards. It will be essential for you to effectively interact with key stakeholders and utilize your technical expertise across various stages of the Software Development Life Cycle. As part of our learning culture, teamwork and collaboration are highly encouraged, excellence is acknowledged, and diversity is both respected and valued. Required...

As a Senior AWS Data Engineer Cloud Data Platform at Teamware Solutions, a division of Quantum Leap Consulting Pvt. Ltd, located in Bangalore, you will be responsible for end-to-end implementation of Cloud data engineering solutions like Enterprise Data lake and Data hub in AWS. Working onsite in an office environment for 5 days a week, you will collaborate with the Offshore Manager and Onsite Business Analyst to understand the requirements and deliver scalable, distributed, cloud-based enterprise data solutions. You should have a strong background in AWS cloud technology, with 4-8 years of hands-on experience. Proficiency in architecting and delivering highly scalable solutions is a must, a...

As a Senior Data Engineer, you will be an integral part of our team, contributing to the design, implementation, and maintenance of scalable data pipelines utilizing cutting-edge tools like Airflow, PySpark, and Databricks. Your responsibilities will extend to supporting machine learning models in a production environment, crafting data products from vast datasets, and engaging with diverse teams to grasp data requirements effectively. Your primary duties will include designing and deploying robust data pipelines by leveraging Airflow, PySpark, and Databricks. You will also be tasked with ensuring the reliability and performance of machine learning models in production, creating data product...

Join us as a Principal Engineer in a challenging role where you will be responsible for designing and engineering software with a primary focus on customer or user experience. You will actively contribute to our architecture, design, and engineering center of excellence, collaborating to enhance the bank's overall software engineering capability. This role offers valuable stakeholder exposure, allowing you to build and leverage relationships while honing your technical talents. The position is available at the vice president level. As a Principal Engineer, your primary responsibilities will include creating exceptional customer outcomes through innovative engineering solutions to both existi...

NTT DATA is looking for a Databricks Engineer to join their team in Bangalore, Karnataka, India. As a Databricks Engineer, you will be responsible for various tasks related to data extraction, ETL pipeline modernization, job design, development, automation, metadata management, documentation, testing, collaboration, performance tuning, security, governance, and compliance. Your primary job duties will include extracting and analyzing data from SQL Server and Teradata sources, translating legacy SQL/DataStage transformations into Databricks-native code, building and orchestrating jobs within Databricks using tools like Databricks Workflows, Delta Lake, and Auto Loader, generating and maintain...

Join us as a Principal Engineer. This challenging role will involve designing and engineering software with a primary focus on customer or user experience. You will actively contribute to our architecture, design, and engineering center of excellence, collaborating to enhance the bank's software engineering capability. This role, offered at the vice president level, provides valuable stakeholder exposure, the opportunity to build and leverage relationships, and a chance to refine your technical skills. As a Principal Engineer, your responsibilities will include creating exceptional customer outcomes through innovative engineering solutions for both existing and new challenges. You will work ...

As a Senior Data Engineer, you will be responsible for designing and developing scalable data pipelines and notebooks using Microsoft Fabric or Synapse Analytics. You should have a research-oriented mindset in Data Projects, thinking outside of the box and focusing on future needs. Your role will involve building and managing Lakehouses and Data Warehouses using Fabrics OneLake architecture, integrating data from diverse sources into Fabric, and collaborating with BI developers for seamless integration with Power BI and other reporting tools. Additionally, you will be required to implement data governance, security, and compliance within the Fabric ecosystem, optimize data storage and proces...

Are you intellectually curious and passionate about promoting solutions across organizational boundaries Join the Consumer & Community Banking (CCB) Stress Testing Transformation team for a dynamic opportunity to design and build creative solutions for the future of stress testing and annual CCAR exercises. As a Senior Associate in the Stress Testing Transformation Solution team, you will be a strategic thinker who is passionate about designing and building creative solutions for the future of Stress Testing. You will spend your time solving complex problems, demonstrating strategic thought leadership, and designing the way our stakeholders operate. By leveraging a deep understanding of CCB ...