12 Spark Architecture Jobs

As a Data Engineering Manager at EY, you will be responsible for leading large scale solution architecture design and optimization to provide streamlined insights to partners throughout the business. You will lead a team of Mid and Senior data engineers to partner with visualization on data quality and troubleshooting needs. Your key responsibilities will include: - Implementing data processes for the data warehouse and internal systems - Leading a team of Junior and Senior Data Engineers in executing data processes and providing quality, timely data management - Managing data architecture and designing ETL processes - Cleaning, aggregating, and organizing data from disparate sources and tra...

This job is with Standard Chartered Bank, an inclusive employer and a member of myGwork the largest global platform for the LGBTQ+ business community. Please do not contact the recruiter directly. Job Summary 5+ years of experience as a developer with experience in developing distributed systems Demonstrated ability to program in Python (pandas, numpy, pytest, pyspark modules and data manipulation techniques) Working knowledge/familiarity with Flask framework, ORM (Object Relational Mapper) and Object Oriented Programming Knowledge of data visualization software like Airflow Good understanding of API, microservices architecture. Design with Rest or GraphQL. Solid understanding of the develop...

Design, develop, and optimize scalable, high performance Spark applications using Scala. Work on mission critical projects, ensuring high availability, reliability, and performance. Analyze and optimize Spark jobs for efficient data processing and resource utilization. Collaborate with cross functional teams to deliver robust, production ready solutions. Troubleshoot and resolve complex issues related to Spark applications and data pipelines. Integrate Spark applications with Kafka for real time data streaming and MongoDB for data storage and retrieval. Follow best practices in coding, testing, and deployment to ensure high quality deliverables. Mentor junior team members and provide technic...

Job Description Job Summary Sr. ETL Developer (Data Analyst) Exp - (5-10 yrs). Location: Bangalore Technical Skills: Very Good experience in ETL, Informatica, Snowflake, SQL Development experience in Spark (Mandatory): Spark Architecture, AQE, PySpark Coding, Coalesce and Repartition functions Development experience in Hadoop (Mandatory): Hadoop/Hive, Big Data Fundamentals Development experience in SQL Queries and Hive (Most Important and mandatory): Window functions and its variations, Complex Query Execution, SQL joins Experience in using Snowflake, SnowSQL, Snowpipe and PySpark Data Integration & ETL Development Design, develop and maintain ETL processes using Informatica PowerCenter Deve...

Job Description: Concrew is the name of a collection of individuals who believe in excellence. We specialize in hiring people who have the drive to succeed and the will to implement the discipline required to succeed. We focus on nurturing our team and providing our team an environment that is conducive to creative thought. Key Responsibilities: - Good knowledge in spark architecture - Develop spark applications using RDD, Spark SQL, Spark Streaming with Kafka, scalaBasics Qualification Required: - Any Engineering or Arts Graduate (Note: Additional details about the company or any benefits were not provided in the job description.),

Job Summary 5+ years of experience as a developer with experience in developing distributed systems Demonstrated ability to program in Python (pandas, numpy, pytest, pyspark modules and data manipulation techniques) Working knowledge/familiarity with Flask framework, ORM (Object Relational Mapper) and Object Oriented Programming Knowledge of data visualization software like Airflow Good understanding of API, microservices architecture. Design with Rest or GraphQL. Solid understanding of the development process pipeline, effective code version control & continuous Integration, Deployment. Experience on Kubernetes, Helm deployments. Writing reusable, testable, and efficient code and excellent ...

**Job Description:** **Role Overview:** You are an experienced developer with over 5 years of experience in developing distributed systems. You have a demonstrated ability to program in Python, utilizing modules such as pandas, numpy, pytest, and pyspark for data manipulation techniques. Additionally, you have a working knowledge of the Flask framework, ORM, and Object Oriented Programming. Your expertise extends to data visualization software like Airflow, API, microservices architecture, and design with Rest or GraphQL. You possess a solid understanding of the development process pipeline, effective code version control & continuous Integration, Deployment. Experience with Kubernetes, Helm...

We are hiring for one of our client company based in bangalore - Position : Senior Apache Spark Expert Experience : 6+ Years Client location: Bangalore Mode of work : Work from Office Mode of Interview : Virtual Qualification: Bachelors degree in computer science, Engineering, or a related field (preferred) Immediate joiners only Job Description Strong knowledge of Spark architecture, able to set up, manage and admin Spark cluster with Kubernetes, Strong on Scala, build efficient ETL and data pipelines. Experience in building end-to-end flows, orchestrate and optimize. Strong experience database RDBMS, SQL, Snowflake.

Hey Jobseekers, we are looking for AWS Data Engineer with glue & Pyspark experience Experience : 4 to 10 Years Location : Hyderabad / Chennai / Noida / Pune / Bangalore kindly reply with below details [HIDDEN TEXT] CTC :- ECTC :- Notice Period :- Preferred Location :- Hands on experience in Dimensional, Data vault and relation modelling Expertise in Data warehousing and ETL Design and implementation Hands on experience with Programming language like Python/Scala Good understanding of Spark architecture along with internals Hand on experience using AWS services like Glue(Pyspark), Lambda, S3, Athena, RDS, IAM, Lake formation Experience on Mulesoft is good to have Hands on experience on implem...

Role Overview: You will be responsible for ETL design, development, and business system designs. Additionally, you will need to have a good understanding of AWS networking concepts, proficiency in IAM concepts, hands-on experience in various AWS services, and the ability to write clean, modular, testable, and maintainable code. Your role will also involve working with programming languages such as Python, Scala, or Java, willingness to learn Python/PySpark if not already experienced, understanding Spark architecture, execution plans, and performance tuning, strong SQL skills, experience with Redshift and/or Athena, writing and maintaining Terraform scripts, familiarity with Git, GitHub workf...

Role Overview: You will be responsible for leveraging your expertise in any of the following Languages such as Scala, Python, or Java with a minimum of 5-10 years of experience. Your role will involve working on SPARK Architecture, demonstrating good problem-solving and analytical skills, and translating business requirements into technical specifications. Additionally, you will collaborate effectively with team members and vendors, possess familiarity with CI/CD pipelines, and have experience supporting existing strategic applications in an agile environment. Key Responsibilities: - Possess expertise in at least one of the following Languages: Scala/Python/Java - Demonstrate proficiency in ...

Role: Data Engineer Exp: 5 to 8 Years Location: Bangalore, Noida, and Hyderabad (Hybrid, weekly 2 Days office must) NP: Immediate to 15 Days (Try to find only immediate joiners) Note: Candidate must have experience in Python, Kafka Streams, Pyspark, and Azure Databricks. Not looking for candidates who have only Exp in Pyspark and not in Python. Job Title: SSE Kafka, Python, and Azure Databricks (Healthcare Data Project) Experience: 5 to 8 years Role Overview: We are looking for a highly skilled with expertise in Kafka, Python, and Azure Databricks (preferred) to drive our healthcare data engineering projects. The ideal candidate will have deep experience in real-time data streaming, cloud-ba...