126 Spark Sql Jobs - Page 2

As an AWS Data Engineer, you will be responsible for designing, building, and optimizing data pipelines and scalable data assets. Your key responsibilities will include: - Designing, building, and optimizing data pipelines and scalable data assets. - Developing and maintaining high-performance code using PySpark/Python with best practices. - Performing code optimization using Spark SQL and PySpark for efficiency. - Implementing code refactoring to modernize legacy codebases for readability and maintainability. - Writing and maintaining unit tests (TDD) to ensure code reliability and reduce bugs. - Debugging and resolving complex issues including performance, concurrency, and logic flaws. - W...

- Role Overview: You have a minimum of 5 years of experience in Bengaluru or Gurugram. Your expertise should include Python, Airflow (Orchestration), GCP (Cloud), Spark SQL, PySpark, CI/CD, Git, and GitHub. Your responsibilities will involve designing and constructing data models, developing and managing data ingestion and processing systems, implementing data storage solutions, ensuring data consistency and accuracy, and collaborating with cross-functional teams to address data-related issues. Proficiency in Python programming and experience with GCP and Airflow are necessary. You should also be familiar with security and governance aspects, such as role-based access control and data lineag...

Role Overview: Join the Consumer & Community Banking division at Chase, a leading U.S. financial services firm as an Analytical Solutions Manager within the Consumer and Community Banking (CCB) Finance Data & Insights Team. You will be part of an agile product team responsible for developing, producing, and transforming financial data and reporting across the CCB division. Your role will involve transforming raw data into actionable insights, creating data visualizations, and intelligence solutions used by top leaders to achieve strategic imperatives. Additionally, you will play a significant role in modernizing the data environment to a cloud platform. Key Responsibilities: - Lead the Finan...

As a Senior Architect - Data & Cloud at our company, you will be responsible for architecting, designing, and implementing end-to-end data pipelines and data integration solutions for varied structured and unstructured data sources and targets. You will need to have more than 15 years of experience in Technical, Solutioning, and Analytical roles, with 5+ years specifically in building and managing Data Lakes, Data Warehouse, Data Integration, Data Migration, and Business Intelligence/Artificial Intelligence solutions on Cloud platforms like GCP, AWS, or Azure. Key Responsibilities: - Translate business requirements into functional and non-functional areas, defining boundaries in terms of Ava...

Role Overview: As a Principal Architect - Data & Cloud at Quantiphi, you will be responsible for leveraging your extensive experience in technical, solutioning, and analytical roles to architect and design end-to-end data pipelines and data integration solutions for structured and unstructured data sources and targets. You will play a crucial role in building and managing data lakes, data warehouses, data integration, and business intelligence/artificial intelligence solutions on Cloud platforms like GCP, AWS, and Azure. Your expertise will be instrumental in designing scalable data warehouse solutions on Big Query or Redshift and working with various data integration, storage, and pipeline ...

About Koantek: Koantek is a Databricks Pure-Play Elite Partner, helping enterprises modernize faster and unlock the full power of Data and AI. Backed by Databricks Ventures and honored as a six- As time Databricks Partner of the Year, we enable global enterprises to modernize at speed, operationalize AI, and realize the full value of their data. Our deep expertise spans industries such as healthcare, financial services, retail, and SaaS, delivering end-to-end solutions from rapid prototyping to production-scale AI deployments. We deliver tailored solutions that enable businesses to leverage data for growth and innovation. Our team of experts utilizes deep industry knowledge combined with cut...

What this job involves: JLL, an international real estate management company, is seeking an Data Engineer to join our JLL Technologies Team. We are seeking candidates that are self-starters to work in a diverse and fast-paced environment that can join our Enterprise Data team. We are looking for a candidate that is responsible for designing and developing of data solutions that are strategic for the business using the latest technologies Azure Databricks, Python, PySpark, SparkSQL, Azure functions, Delta Lake, Azure DevOps CI/CD. Responsibilities Design, Architect, and Develop solutions leveraging cloud big data technology to ingest, process and analyze large, disparate data sets to exceed b...

What this job involves: JLL, an international real estate management company, is seeking an Data Engineer to join our JLL Technologies Team. We are seeking candidates that are self-starters to work in a diverse and fast-paced environment that can join our Enterprise Data team. We are looking for a candidate that is responsible for designing and developing of data solutions that are strategic for the business using the latest technologies Azure Databricks, Python, PySpark, SparkSQL, Azure functions, Delta Lake, Azure DevOps CI/CD. Responsibilities Design, Architect, and Develop solutions leveraging cloud big data technology to ingest, process and analyze large, disparate data sets to exceed b...

As a Senior Python Developer specializing in Python and Spark programming, you will be responsible for leveraging your thorough and hands-on knowledge to write efficient, reusable, and reliable Python code. You should possess a minimum of 5+ years of experience in Python programming and have a strong proficiency in both Python and Spark programming. Your key responsibilities will include implementing Spark Core, Spark SQL, and Spark Streaming, working with Spark within the Hadoop Ecosystem, and designing and implementing low-latency, high-availability, and performance applications. Additionally, you will lead and guide a team of junior Python developers, collaborate with other team members a...

At Capgemini Invent, we believe that difference drives change. As inventive transformation consultants, we blend our strategic, creative, and scientific capabilities to collaborate closely with clients in delivering cutting-edge solutions tailored to address the challenges of today and tomorrow. Our approach is informed and validated by science and data, superpowered by creativity and design, and all underpinned by purpose-driven technology. Your role will involve proficiency in various technologies such as MS Fabric, Azure Data Factory, Azure Synapse Analytics, Azure Databricks Lakehouses, OneLake, Data Pipelines, Real-Time Analytics, Power BI Integration, and Semantic Model. You will be re...

About this role: Wells Fargo is seeking a Analytics Consultant. In this role, you will: Consult with business line and enterprise functions on less complex research Use functional knowledge to assist in non-model quantitative tools that support strategic decision making Perform analysis of findings and trends using statistical analysis and document process Present recommendations to increase revenue, reduce expense, maximize operational efficiency, quality, and compliance Identify and define business requirements and translate data and business needs into research and recommendations to improve efficiency Participate in all group technology efforts including design and implementation of data...

You are a highly skilled and experienced Data Modeler who will be joining our data engineering team. Your expertise lies in designing scalable and efficient data models for cloud platforms, with a focus on Oracle Data Warehousing and Databricks Lakehouse architecture. Your role will be crucial in our strategic transition from an on-prem Oracle data warehouse to a modern cloud-based Databricks platform. Your responsibilities will include designing and implementing conceptual, logical, and physical data models to meet business requirements and analytics needs. You will lead the migration of data models from Oracle Data Warehouse to Databricks on AWS or Azure cloud, reverse-engineer complex Ora...

As a Data Engineer at Capgemini, you will have the opportunity to design, build, and optimize modern data solutions using your expertise in SQL, Spark SQL, Databricks, Unity Catalog, and PySpark. With a focus on data warehousing concepts and best practices, you will contribute to the development of scalable and high-performance data platforms. You will bring 4-10 years of experience in Data Engineering/ETL Development, showcasing your strong skills in SQL, Spark SQL, and Databricks on Azure or AWS. Your proficiency in PySpark for data processing and knowledge of Unity Catalog for data governance will be key in ensuring the success of data solutions. Additionally, your understanding of data w...

Unlock your potential with Dassault Systmes, a global leader in Scientific Software Engineering as a Big Data Engineer in Pune, Maharashtra! Role Description & Responsibilities: - Data Pipeline Development: Design, develop, and maintain robust ETL pipelines for batch and real-time data ingestion, processing, and transformation using Spark, Kafka, and Python. - Data Architecture: Build and optimize scalable data architectures, including data lakes, data marts, and data warehouses, to support business intelligence, reporting, and machine learning. - Data Governance: Ensure data reliability, integrity, and governance by enabling accurate, consistent, and trustworthy data for decision-making. - ...

This position falls under the ICG TTS Operations Technology (OpsTech) Group, focusing on assisting in the implementation of a next-generation Digital Automation Platform and Imaging Workflow Technologies. The ideal Candidate should have relevant experience in managing development teams within the distributed systems Eco-System and must exhibit strong teamwork skills. The candidate is expected to possess superior technical knowledge of current programming languages, technologies, and leading-edge development tools. The primary objective of this role is to contribute to applications, systems analysis, and programming activities. As a Lead Spark Scala Engineer, the candidate should have hands-o...

As a Data Engineer, you will be responsible for building scalable data pipelines using PySpark. Your role will involve implementing complex business logic using Spark SQL, DataFrame, and RDD APIs. You should have strong programming skills in Python, with a solid understanding of data structures, algorithms, and software engineering principles. Your expertise in designing, developing, and maintaining batch and streaming data pipelines will be crucial. You should be familiar with ETL/ELT processes and best practices for data transformation, data quality, and performance optimization. Knowledge of the modern data engineering ecosystem, including distributed data processing, storage systems, and...

As a Data Engineer II at Media.net, you will be responsible for designing, executing, and managing large and complex distributed data systems. Your role will involve monitoring performance, optimizing existing projects, and researching and integrating Big Data tools and frameworks as required to meet business and data requirements. You will play a key part in implementing scalable solutions, creating reusable components and data tools, and collaborating with teams across the company to integrate with the data platform efficiently. The team you will be a part of ensures that every web page view is seamlessly processed through high-scale services, handling a large volume of requests across 5 m...

As a Databricks AWS/Azure/GCP Architect at Koantek based in Mumbai, you will play a crucial role in building secure and highly scalable big data solutions that drive tangible, data-driven outcomes while emphasizing simplicity and operational efficiency. Collaborating with teammates, product teams, and cross-functional project teams, you will lead the adoption and integration of the Databricks Lakehouse Platform into the enterprise ecosystem and AWS/Azure/GCP architecture. Your responsibilities will include implementing securely architected big data solutions that are operationally reliable, performant, and aligned with strategic initiatives. Your expertise should include an expert-level know...

As a Senior Databricks Developer at Newscape Consulting, you will play a crucial role in our data engineering team, focusing on building scalable and efficient data pipelines using Databricks, Apache Spark, Delta Lake, and cloud-native services (Azure/AWS/GCP). Your responsibilities will include collaborating closely with data architects, data scientists, and business stakeholders to deliver high-performance, production-grade solutions that enhance user experience and productivity in the healthcare industry. Your key skills should include a strong hands-on experience with Databricks including Workspace, Jobs, DLT, Repos, and Unity Catalog. Proficiency in PySpark, Spark SQL, and optionally Sc...

We are searching for a proficient Python Developer to become a valuable member of our product development team. Your primary focus will involve automating data ingestion, processing, and validation workflows utilizing PySpark to create robust, scalable, and efficient data pipelines. In this role, you will collaborate closely with data engineers, analysts, and stakeholders to provide impactful data solutions within a dynamic work environment. Your responsibilities will include collaborating with data engineers and stakeholders to outline requirements and implement automated solutions. You will design, develop, and upkeep scalable and efficient PySpark-based automation for data ingestion, proc...

Key Responsibilities: Develop and optimize data pipelines in Databricks for transforming and processing data from various sources. Integrate data using Unity Catalog and external data sources (data lakes, APIs, etc.). Write Spark SQL and PySpark scripts for data transformations, optimizations, and creating views/procedures. Perform data analysis to identify quality issues, optimize pipelines, and enhance data processing for analytics. Collaborate on report generation and dashboard creation with front-end teams. Use GitLab for version control, CI/CD automation, and task management (Jira). Required Skills and Qualifications: Masters or Bachelor’s degree in Data Engineering or related field. 3+...

As a skilled professional with over 7 years of experience, you will be responsible for reviewing and understanding business requirements to ensure timely completion of development tasks with rigorous testing to minimize defects. Collaborating with a software development team is crucial to implement best practices and enhance the performance of Data applications, meeting client needs effectively. In this role, you will collaborate with various teams within the company and engage with customers to comprehend, translate, define, and design innovative solutions for their business challenges. Your tasks will also involve researching new Big Data technologies to evaluate their maturity and alignme...

The ideal candidate will be responsible for designing and implementing streaming data pipelines that integrate Kafka with Databricks using Structured Streaming. You will also be tasked with architecting and maintaining the Medallion Architecture, which consists of well-defined Bronze, Silver, and Gold layers. Additionally, you will need to implement efficient data ingestion processes using Databricks Autoloader for high-throughput data loads. You will work with large volumes of structured and unstructured data to ensure high availability and performance, applying performance tuning techniques like partitioning, caching, and cluster resource optimization. Collaboration with cross-functional t...

As a Data Engineer at MS Fabric, your primary responsibility will be designing, implementing, and managing scalable data pipelines using the MS Fabric Azure Tech stack, including ADLS Gen2, ADF, and Azure SQL. You should have a strong background in data integration techniques, ETL processes, and data pipeline architectures. Additionally, you will need to be well-versed in data quality rules, principles, and implementation. Your key result areas and activities will include: 1. Data Pipeline Development & Optimization: - Design and implement data pipelines using MS Fabric. - Manage and optimize ETL processes for data extraction, transformation, and loading. - Conduct performance tuning for dat...

Intellectt Inc. is seeking AI Product Engineers to join our team and contribute to our expanding AI-focused product portfolio. As an AI Product Engineer, you will play a crucial role in bridging the gap between product development and advanced AI technologies. This position requires a deep understanding of AI models, APIs, and data pipelines, along with a strong focus on product design, user experience, and deployment strategies. Key Responsibilities - Collaborate with product managers and AI teams to design and develop AI-driven products leveraging LLMs such as GPT-3.5, GPT-4, and Gemini. - Translate business requirements into AI product features and develop functional prototypes. - Impleme...