170 Spark Sql Jobs

As a Senior Architect - Data & Cloud at our company, you will play a crucial role in architecting, designing, and implementing end-to-end data pipelines and data integration solutions for various structured and unstructured data sources and targets. Your responsibilities will include the following: - More than 15 years of experience in Technical, Solutioning, and Analytical roles. - 5+ years of experience in building and managing Data Lakes, Data Warehouse, Data Integration, Data Migration, and Business Intelligence/Artificial Intelligence solutions on Cloud GCP/AWS/Azure. - Ability to understand business requirements and translate them into functional and non-functional areas, defining non-...

Role Overview: You will be working as a Principal Architect in the Data & Cloud domain with Quantiphi. With more than 15 years of experience in Technical, Solutioning, and Analytical roles, you will be responsible for architecting, designing, and implementing end-to-end data pipelines and data integration solutions for various structured and unstructured data sources and targets. You will also play a key role in designing scalable data warehouse solutions on cloud platforms like Big Query or Redshift. Key Responsibilities: - Build and manage Data Lakes, Data Warehouse, Data Integration, Data Migration, and Business Intelligence/Artificial Intelligence solutions on Cloud platforms such as GCP...

Job Summary At PwC, our Commercial Technology & Innovation (CT&I) leaders are shaping the firms transformation—driving how we build and scale AI-enabled solutions that create value for clients and the business. Directors in CT&I guide multi-year strategies, investments, and delivery outcomes that embed AI, automation, and digital platforms into the fabric of how Advisory operates. Years of Experience: 4-9 year(s) Responsibilities Design and build scalable batch and streaming pipelines using Python, PySpark, Spark SQL, and Databricks/Synapse. Develop reusable ETL/ELT components and ingestion flows for common data formats (JSON, Parquet, CSV). Integrate data from relational databases (PostgreS...

As an AWS Data Engineer, you will play a crucial role in designing, developing, and maintaining data pipelines and assets on AWS. Your responsibilities will include optimizing and refactoring legacy PySpark/Spark SQL code for enhanced performance and maintainability. You will be implementing unit testing/Test-Driven Development (TDD) to ensure the delivery of robust, bug-free code. Additionally, you will be debugging and resolving complex performance, concurrency, and logic issues while managing code versioning and repositories using tools like Git and JFrog Artifactory. Leveraging AWS services such as S3, EC2, Lambda, Redshift, and CloudFormation will be essential for creating scalable data...

As a Software Engineer III at JPMorgan Chase within the Consumer and community banking technology team, you will be part of an agile team designing and delivering trusted market-leading technology products in a secure, stable, and scalable manner. Your role involves critical technology solutions across multiple technical areas to support the firm's business objectives. **Key Responsibilities:** - Develop and maintain applications using Apache Spark and Java - Work with different aspects of the Spark ecosystem including Datasets, Spark SQL, and Scalability support - Collaborate with business analysts and lead engineers to understand requirements and design software solutions - Implement perfo...

You will be working as a Developer/Lead Developer with a minimum of 10 years of work experience. Your responsibilities will include: - Having sound knowledge on SQL and analytical queries. - Demonstrating proficiency in any reporting tools, preferably Tableau/Power BI. - Possessing beginner to intermediate level skills in Python Development. - Exhibiting good knowledge on APIs and data architecture. - Having familiarity with Atlassian tools like Jira and Confluence. - Having working experience on Atlassian Analytics/Spark SQL will be an added advantage. - Demonstrating a good understanding of various databases. - Leading offshore teams effectively. - Being open to learning new technologies a...

**Job Description:** As a Data Engineer at our company, you will play a crucial role in simplifying our organization by developing innovative data-driven solutions through data pipelines, modeling, and ETL design. Your primary focus will be on creating effortless, digital-first customer experiences while ensuring the safety and security of our customers" data. **Key Responsibilities:** - Build advanced automation of data engineering pipelines by eliminating manual stages - Incorporate new data techniques into the business through role modeling, training, and overseeing experiment design - Provide a clear understanding of data platform costs to meet cost-saving and income targets - Source new...

You're ready to gain the skills and experience needed to grow within your role and advance your career and JPMorgan Chase has the perfect software engineering opportunity for you. As a Software Engineer III at JPMorgan Chase within the Consumer & Community Banking Team, you are part of an agile team that works to enhance, design, and deliver the software components of the firm's state-of-the-art technology products in a secure, stable, and scalable way. As an emerging member of a software engineering team, you execute software solutions through the design, development, and technical troubleshooting of multiple components within a technical product, application, or system, while gaining the s...

Role Overview: You would be joining Quantiphi as a Principal Architect - Data & Cloud with a focus on technical, solutioning, and analytical roles. Your role will involve architecting, designing, and implementing end-to-end data pipelines and data integration solutions for structured and unstructured data sources and targets. You will work on various data integration, ETL technologies on Cloud like Spark, Pyspark/Scala, Dataflow, DataProc, EMR, etc. You will also be responsible for architecting scalable data warehouse solutions on cloud platforms like Big Query or Redshift. Key Responsibilities: - More than 15 years of experience in Technical, Solutioning, and Analytical roles. - Experience ...

Role Overview: At Capgemini Invent, you will be working as an inventive transformation consultant, blending strategic, creative, and scientific capabilities to deliver cutting-edge solutions tailored to client's challenges. Your role will involve proficiency in various technologies such as MS Fabric, Azure Data Factory, Azure Synapse Analytics, and more. You will integrate Fabric capabilities for seamless data flow, governance, and collaboration across teams while utilizing strong programming skills in Python, PySpark, Scala, or Spark SQL/TSQL for data transformations. Key Responsibilities: - Proficiency in MS Fabric, Azure Data Factory, Azure Synapse Analytics, Azure Databricks Lakehouses, ...

As an AWS Data Engineer, your primary role will involve designing, building, and optimizing data pipelines and scalable data assets. You will be responsible for developing and maintaining high-performance code using PySpark/Python with best practices. Additionally, you will perform code optimization using Spark SQL and PySpark to ensure efficiency. Code refactoring to modernize legacy codebases for readability and maintainability will also be a key aspect of your responsibilities. Ensuring code reliability and reducing bugs through writing and maintaining unit tests (TDD) will be crucial. Furthermore, you will need to debug and resolve complex issues including performance, concurrency, and l...

As a Senior Python Data Engineer with 4+ years of experience, you will play a crucial role in designing, building, and optimizing reliable data platforms and analytics solutions by partnering with product, analytics, and business teams. Your expertise in Azure Databricks, PySpark, Spark SQL, and Power BI will be essential in ensuring the success of our projects in a fast-paced and collaborative environment. **Key Responsibilities:** - Design and implement scalable ETL/ELT pipelines on Azure Databricks using PySpark/Spark SQL. - Model data for semantic layers to power BI and analytics use cases. - Build and maintain Power BI datasets, reports, and dashboards, utilizing DAX, data modeling, and...

As a candidate for the position at Standard Chartered, you will be responsible for leading the design and development of advanced quantitative and AI-driven models for market abuse detection across multiple asset classes and trading venues. You will drive the solutioning and delivery of large-scale surveillance systems in a global investment banking environment, ensuring alignment of technical solutions with business objectives and regulatory requirements. Your key responsibilities will include: - Collaborating with consultants, MAR monitoring teams, and technology stakeholders to gather requirements and co-create innovative solutions - Translating regulatory and business requirements into a...

Role Overview: You will be responsible for designing and implementing data solutions using technologies such as PySpark, Spark SQL, and Power BI. As a Lead Python Data Engineer, you will play a key role in building and optimizing scalable data pipelines while ensuring data governance, security, and compliance in cloud environments. Key Responsibilities: - Utilize PySpark to build and optimize scalable data pipelines, including Spark SQL, DataFrames, and Datasets. - Work with Azure Databricks (Unity Catalog) to manage Delta tables, workflows, and script in Python. - Design and model data for Data Warehousing and Visualization projects through Semantic Layers. - Write complex SQL queries and b...

Role Overview: You will be responsible for designing and deploying predictive models, building ML pipelines, optimizing model performance, implementing Delta Lake, enabling CI/CD for ML pipelines, and troubleshooting issues in Spark Jobs and Databricks Environment. As a Senior Programmer Analyst in AI-ML Engineering, you are expected to have strong technical expertise and experience in Python, PySpark, MLflow, and predictive modeling techniques. Key Responsibilities: - Design and deploy predictive models (e.g., forecasting, churn analysis, fraud detection) using Python/SQL, Spark MLlib, and Databricks ML. - Build end-to-end ML pipelines (data ingestion, feature engineering, model training, d...

Position Summary The Senior BI Data Analyst will collaborate with business stakeholders and team members to define BI solutions, data requirements, and project scope. They will recommend innovative BI solution approaches that meet or exceed business needs and are responsible for managing BI development projects from inception to completion. The role also includes providing ongoing support and enhancements, ensuring that solutions remain effective and aligned with organizational goals. Additionally, the analyst will contribute to the BI Center of Excellence by sharing knowledge, training users, and staying abreast of emerging BI trends to maintain a competitive edge. Key Responsibilities BI R...

As a highly skilled and experienced Data Engineer with 6 to 10 years of experience in data engineering, you will be joining our growing data team. Your primary responsibility will be to build scalable data pipelines, optimize data processing workflows, and enable insightful reporting and analytics across the organization. Your expertise in SQL, Databricks, Spark SQL, PySpark, and BI tools like Power BI or Tableau will be crucial for success in this role. Key Responsibilities: - Design and develop robust, scalable data pipelines using PySpark and Databricks. - Write efficient SQL and Spark SQL queries for data transformation and analysis. - Collaborate closely with BI teams to enable reportin...

As an AI Product Engineer at Intellectt Inc., you will play a crucial role in bridging the gap between product development and advanced AI technologies. You will work closely with AI models, APIs, and data pipelines to design and develop AI-driven products that leverage cutting-edge technologies. Your work will contribute to product design, user experience, and deployment strategies, ultimately impacting business outcomes. **Key Responsibilities:** - Collaborate with product managers and AI teams to design and develop AI-driven products using LLMs such as GPT-3.5, GPT-4, and Gemini. - Translate business requirements into AI product features and build functional prototypes to showcase the cap...

As a Data Engineer, your role will involve designing and implementing streaming data pipelines that integrate Kafka with Databricks using Structured Streaming. You will be responsible for architecting and maintaining the Medallion Architecture with well-defined Bronze, Silver, and Gold layers. Your tasks will include implementing efficient ingestion using Databricks Autoloader for high-throughput data loads and working with large volumes of structured and unstructured data to ensure high availability and performance. Additionally, you will apply performance tuning techniques such as partitioning, caching, and cluster resource optimization. Collaboration with cross-functional teams including ...

As a Sr. Lead Data at our company, you will play a crucial role in driving the execution of enterprise data migration from diverse ERP systems into the Microsoft Fabric platform. Your focus will be on delivering high-quality, scalable data solutions leveraging Fabrics Lakehouse architecture, Delta Lake, and modern data engineering tools. Key Responsibilities: - Lead data migration efforts from diverse ERP systems into Microsoft Fabric, leveraging Lakehouse / OneLake architecture to support fabric-based reporting. - Design, build, and maintain robust data pipelines and ETL/ELT workflows using Fabric tools such as Dataflows Gen2, Pipelines (formerly ADF), notebooks, and Delta Lake. - Implement...

Role Overview: Join the digital revolution at NatWest Digital X as a Data Engineer, where you will play a key role in simplifying the organization by developing innovative data-driven solutions, ensuring customer value, and maintaining data security. As a Data Engineer, you will be responsible for building effortless, digital-first customer experiences and driving customer value through the application of appropriate data solutions. Key Responsibilities: - Develop innovative data-driven solutions through data pipelines, modelling, and ETL design - Drive customer value by understanding complex business problems and requirements - Build advanced automation of data engineering pipelines - Embed...

As a Software Developer at our organization, you will work in a constantly evolving environment, due to technological advances and the strategic direction of the organization. Your main responsibilities will include creating, maintaining, auditing, and improving systems to meet specific needs, often as advised by a systems analyst or architect. You will be testing both hardware and software systems to diagnose and resolve system faults. Additionally, you will be writing diagnostic programs, designing, and writing code for operating systems and software to ensure efficiency. When required, you will also make recommendations for future developments. Key Responsibilities: - Develop, test, deplo...

As a Data Engineer at our company, you will have the following responsibilities: - Review and understand business requirements to ensure development tasks are completed within the provided timeline and are thoroughly tested with minimal defects. - Collaborate with a software development team to implement best practices and optimize the performance of Data applications. - Engage with various teams across the company and interact with customers to define, design, and showcase new concepts and solutions. - Conduct research on new Big Data technologies, evaluating their maturity and alignment with business and technology strategies. - Work closely with other developers to meet client needs effic...

Job Description: Concrew is the name of a collection of individuals who believe in excellence. We specialize in hiring people who have the drive to succeed and the will to implement the discipline required to succeed. We focus on nurturing our team and providing our team an environment that is conducive to creative thought. Key Responsibilities: - Good knowledge in spark architecture - Develop spark applications using RDD, Spark SQL, Spark Streaming with Kafka, scalaBasics Qualification Required: - Any Engineering or Arts Graduate (Note: Additional details about the company or any benefits were not provided in the job description.),

As a Senior Data Modeler at NTT DATA, your role will involve designing and implementing dimensional (star/snowflake) and 3NF data models optimized for analytical and reporting needs in Azure Synapse and Power BI. You will be responsible for performing Source-to-Target Mapping (STM) from source to multiple layers in the data lake, analyzing and optimizing Spark SQL queries, and collaborating with cross-functional teams to ensure data models align with business needs. Moreover, you will maintain version control and CI/CD pipelines in Git and Azure DevOps, as well as integrate Azure Purview to enable access controls and implement row-level security. Key Responsibilities: - Design and implement ...