88 Apache Spark Jobs - Page 3

Role & responsibilities: Technical Skills: 1. Core Databricks Platform: Databricks workspace, clusters, jobs, notebooks, Unity Catalog 2. Big Data Technologies: Apache Spark (PySpark/Scala), Delta Lake, Apache Kafka 3. Programming Languages: Python (advanced), SQL (advanced), Scala (preferred) 4. Cloud Platforms: Azure (preferred) or AWS with Databricks integration 5. Data Pipeline Tools: Apache Airflow, Azure Data Factory, or similar orchestration tools 6. Version Control & CI/CD: Git, Azure DevOps, Jenkins, or GitHub Actions 7. Data Formats & Storage: Parquet, JSON, Avro, Azure Data Lake, S3 8. Monitoring & Observability: Databricks monitoring, custom metrics, alerting systems Leadership & Soft Skills: 1. Strong leadership and people management capabilities 2. Excellent communication skills with ability to explain complex technical concepts 3. Experience with Agile/Scrum methodologies 4. Problem-solving mindset with attention to detail 5. Ability to work in fast-paced, dynamic environments 6. 8+ years of overall experience in data engineering, software engineering, or related technical roles 7. 4+ years of hands-on experience with Databricks/big data platform and Apache Spark 8. 2+ years of team leadership or technical mentoring experience Preferred Qualifications: 1. Databricks certifications (Certified Data Engineer Associate/Professional) 2. Experience with MLOps and machine learning pipeline deployment 3. Knowledge of data mesh or data fabric architectures 4. Experience with streaming data processing using Spark Structured Streaming 5. Background in financial services, healthcare, or retail domains

Responsibilities Lead the design, development, and implementation of big data solutions using Apache Spark and Databricks. Architect and optimize data pipelines and workflows to process large volumes of data efficiently. Utilize Databricks features such as Delta Lake, Databricks SQL, and Databricks Workflows to enhance data processing and analytics capabilities. Collaborate with data engineers, data scientists, and business stakeholders to understand data requirements and deliver high-quality data solutions. Implement best practices for data engineering, including data quality, data governance, and data security. Monitor and troubleshoot performance issues in Spark jobs and Databricks clusters. Mentor and guide junior engineers in the team, promoting a culture of continuous learning and improvement. Stay up-to-date with the latest advancements in Spark and Databricks technologies and incorporate them into the team's practices.

We specialize in delivering high-quality human-curated data and AI-first scaled operations services Based in San Francisco and Hyderabad, we are a fast-moving team on a mission to build AI for Good, driving innovation and societal impact Role Overview: We are looking for a Data Scientist to join and build intelligent, data-driven solutions for our client that enable impactful decisions This role requires contributions across the data science lifecycle from data wrangling and exploratory analysis to building and deploying machine learning models Whether youre just getting started or have years of experience, were looking for individuals who are curious, analytical, and driven to make a difference with data Responsibilities: Design, develop, and deploy machine learning models and analytical solutions Conduct exploratory data analysis and feature engineering Own or contribute to the end-to-end data science pipeline: data cleaning, modeling, validation, and deployment Collaborate with cross-functional teams (engineering, product, business) to define problems and deliver measurable impact Translate business challenges into data science problems and communicate findings clearly Implement A/B tests, statistical tests, and experimentation strategies Support model monitoring, versioning, and continuous improvement in production environments Evaluate new tools, frameworks, and best practices to improve model accuracy and scalability Required Skills: Strong programming skills in Python including libraries such as pandas, NumPy, scikit-learn, matplotlib, seaborn Proficient in SQL, comfortable querying large, complex datasets Sound understanding of statistics, machine learning algorithms, and data modeling Experience building end-to-end ML pipelines Exposure to or hands-on experience with model deployment tools like FastAPI, Flask, MLflow Experience with data visualization and insight communication Familiarity with version control tools (eg, Git) and collaborative workflows Ability to write clean, modular code and document processes clearly Nice to Have: Experience with deep learning frameworks like TensorFlow or PyTorch Familiarity with data engineering tools like Apache Spark, Kafka, Airflow, dbt Exposure to MLOps practices and managing models in production environments Working knowledge of cloud platforms like AWS, GCP, or Azure (e, SageMaker, BigQuery, Vertex AI) Experience designing and interpreting A/B tests or causal inference models Prior experience in high-growth startups or cross-functional leadership roles Educational Qualifications: Bachelors or Masters degree in Computer Science, Data Science, Mathematics, Engineering, or a related field Location : - Mumbai, Delhi / NCR, Bengaluru , Kolkata, Chennai, Hyderabad, Ahmedabad, Pune,India

Role Purpose The purpose of the role is to support process delivery by ensuring daily performance of the Production Specialists, resolve technical escalations and develop technical capability within the Production Specialists. Do Oversee and support process by reviewing daily transactions on performance parameters Review performance dashboard and the scores for the team Support the team in improving performance parameters by providing technical support and process guidance Record, track, and document all queries received, problem-solving steps taken and total successful and unsuccessful resolutions Ensure standard processes and procedures are followed to resolve all client queries Resolve client queries as per the SLAs defined in the contract Develop understanding of process/ product for the team members to facilitate better client interaction and troubleshooting Document and analyze call logs to spot most occurring trends to prevent future problems Identify red flags and escalate serious client issues to Team leader in cases of untimely resolution Ensure all product information and disclosures are given to clients before and after the call/email requests Avoids legal challenges by monitoring compliance with service agreements Handle technical escalations through effective diagnosis and troubleshooting of client queries Manage and resolve technical roadblocks/ escalations as per SLA and quality requirements If unable to resolve the issues, timely escalate the issues to TA & SES Provide product support and resolution to clients by performing a question diagnosis while guiding users through step-by-step solutions Troubleshoot all client queries in a user-friendly, courteous and professional manner Offer alternative solutions to clients (where appropriate) with the objective of retaining customers and clients business Organize ideas and effectively communicate oral messages appropriate to listeners and situations Follow up and make scheduled call backs to customers to record feedback and ensure compliance to contract SLAs Build people capability to ensure operational excellence and maintain superior customer service levels of the existing account/client Mentor and guide Production Specialists on improving technical knowledge Collate trainings to be conducted as triage to bridge the skill gaps identified through interviews with the Production Specialist Develop and conduct trainings (Triages) within products for production specialist as per target Inform client about the triages being conducted Undertake product trainings to stay current with product features, changes and updates Enroll in product specific and any other trainings per client requirements/recommendations Identify and document most common problems and recommend appropriate resolutions to the team Update job knowledge by participating in self learning opportunities and maintaining personal networks Deliver NoPerformance ParameterMeasure1ProcessNo. of cases resolved per day, compliance to process and quality standards, meeting process level SLAs, Pulse score, Customer feedback, NSAT/ ESAT2Team ManagementProductivity, efficiency, absenteeism3Capability developmentTriages completed, Technical Test performance Mandatory Skills: Apache Spark. Experience: 5-8 Years.

Role Description: We are looking for highly motivated expert Data Engineer who can own the design & development of complex data pipelines, solutions and frameworks. The ideal candidate will be responsible to design, develop, and maintain data pipelines, data integration frameworks, and metadata-driven architectures that enable seamless data access and analytics. This role prefers deep expertise in big data processing, distributed computing, data modeling, and governance frameworks to support self-service analytics, AI-driven insights, and enterprise-wide data management. Roles & Responsibilities: Design, develop, and maintain complex ETL/ELT data pipelines in Databricks using PySpark, Scala, and SQL to process large-scale datasets Understand the biotech/pharma or related domains & build highly efficient data pipelines to migrate and deploy complex data across systems Design and Implement solutions to enable unified data access, governance, and interoperability across hybrid cloud environments Ingest and transform structured and unstructured data from databases (PostgreSQL, MySQL, SQL Server, MongoDB etc.), APIs, logs, event streams, images, pdf, and third-party platforms Ensuring data integrity, accuracy, and consistency through rigorous quality checks and monitoring Expert in data quality, data validation and verification frameworks Innovate, explore and implement new tools and technologies to enhance efficient data processing Proactively identify and implement opportunities to automate tasks and develop reusable frameworks Work in an Agile and Scaled Agile (SAFe) environment, collaborating with cross-functional teams, product owners, and Scrum Masters to deliver incremental value Use JIRA, Confluence, and Agile DevOps tools to manage sprints, backlogs, and user stories. Support continuous improvement, test automation, and DevOps practices in the data engineering lifecycle Collaborate and communicate effectively with the product teams, with cross-functional teams to understand business requirements and translate them into technical solutions Must-Have Skills: Hands-on experience in data engineering technologies such as Databricks, PySpark, SparkSQL Apache Spark, AWS, Python, SQL, and Scaled Agile methodologies. Proficiency in workflow orchestration, performance tuning on big data processing. Strong understanding of AWS services Ability to quickly learn, adapt and apply new technologies Strong problem-solving and analytical skills Excellent communication and teamwork skills Experience with Scaled Agile Framework (SAFe), Agile delivery practices, and DevOps practices. Good-to-Have Skills: Data Engineering experience in Biotechnology or pharma industry Experience in writing APIs to make the data available to the consumers Experienced with SQL/NOSQL database, vector database for large language models Experienced with data modeling and performance tuning for both OLAP and OLTP databases Experienced with software engineering best-practices, including but not limited to version control (Git, Subversion, etc.), CI/CD (Jenkins, Maven etc.), automated unit testing, and Dev Ops Education and Professional Certifications Minimum 5 to 8 years of Computer Science, IT or related field experience AWS Certified Data Engineer preferred Databricks Certificate preferred Scaled Agile SAFe certification preferred Soft Skills: Excellent analytical and troubleshooting skills. Strong verbal and written communication skills Ability to work effectively with global, virtual teams High degree of initiative and self-motivation. Ability to manage multiple priorities successfully. Team-oriented, with a focus on achieving team goals. Ability to learn quickly, be organized and detail oriented. Strong presentation and public speaking skills.

Whats in it for you? Pay above market standards The role is going to be contract based with project timelines from 2 12 months, or freelancing Be a part of an Elite Community of professionals who can solve complex AI challenges Work location could be: Remote (Highly likely) Onsite on client location Deccan AIs Office: Hyderabad or Bangalore Responsibilities: Design and architect enterprise-scale data platforms, integrating diverse data sources and tools Develop real-time and batch data pipelines to support analytics and machine learning Define and enforce data governance strategies to ensure security, integrity, and compliance along with optimizing data pipelines for high performance, scalability, and cost efficiency in cloud environments Implement solutions for real-time streaming data (Kafka, AWS Kinesis, Apache Flink) and adopt DevOps/DataOps best practices Required Skills: Strong experience in designing scalable, distributed data systems and programming (Python, Scala, Java) with expertise in Apache Spark, Hadoop, Flink, Kafka, and cloud platforms (AWS, Azure, GCP) Proficient in data modeling, governance, warehousing (Snowflake, Redshift, BigQuery), and security/compliance standards (GDPR, HIPAA) Hands-on experience with CI/CD (Terraform, Cloud Formation, Airflow, Kubernetes) and data infrastructure optimization (Prometheus, Grafana) Nice to Have: Experience with graph databases, machine learning pipeline integration, real-time analytics, and IoT solutions Contributions to open-source data engineering communities What are the next steps? Register on our Soul AI website

Responsibilities: Design and architect enterprise-scale data platforms, integrating diverse data sources and tools Develop real-time and batch data pipelines to support analytics and machine learning Define and enforce data governance strategies to ensure security, integrity, and compliance along with optimizing data pipelines for high performance, scalability, and cost efficiency in cloud environments Implement solutions for real-time streaming data (Kafka, AWS Kinesis, Apache Flink) and adopt DevOps/DataOps best practices Required Skills: Strong experience in designing scalable, distributed data systems and programming (Python, Scala, Java) with expertise in Apache Spark, Hadoop, Flink, Kafka, and cloud platforms (AWS, Azure, GCP) Proficient in data modeling, governance, warehousing (Snowflake, Redshift, Big Query), and security/compliance standards (GDPR, HIPAA) Hands-on experience with CI/CD (Terraform, Cloud Formation, Airflow, Kubernetes) and data infrastructure optimization (Prometheus, Grafana) Nice to Have: Experience with graph databases, machine learning pipeline integration, real-time analytics, and IoT solutions Contributions to open-source data engineering communities

We specialize in delivering high-quality human-curated data and AI-first scaled operations services Based in San Francisco and Hyderabad, we are a fast-moving team on a mission to build AI for Good, driving innovation and societal impact Role Overview: We are seeking a Data Engineer / Data Architect who will be responsible for designing, building, and maintaining scalable data infrastructure and systems for a client Youll play a key role in enabling efficient data flow, storage, transformation, and access across our organization or client ecosystems Whether youre just beginning or already an expert, we value strong technical skills, curiosity, and the ability to translate complex requirements into reliable data pipelines Responsibilities: Design and implement scalable, robust, and secure data pipelines Build ETL/ELT frameworks to collect, clean, and transform structured and unstructured data Collaborate with data scientists, analysts, and backend engineers to enable seamless data access and model integration Maintain data integrity, schema design, lineage, and quality monitoring Optimize performance and ensure reliability of data workflows in production environments Design and manage data warehousing and lakehouse architecture Set up and manage infrastructure using IaC (Infrastructure as Code) when applicable Required Skills: Strong programming skills in Python, SQL, and Shell scripting Hands-on experience with ETL tools and orchestration frameworks (e g, Airflow, Luigi, dbt) Proficiency in relational databases (e g , PostgreSQL, MySQL) and NoSQL databases (e g , MongoDB, Redis) Experience with big data technologies: Apache Spark, Kafka, Hive, Hadoop, etc Deep understanding of data modeling, schema design, and data warehousing concepts Proficient with cloud platforms (AWS/GCP/Azure) and services like Redshift, BigQuery, S3, Dataflow, or Databricks Knowledge of DevOps and CI/CD tools relevant to data infrastructure Nice to Have: Experience working in real-time streaming environments Familiarity with containerization and Kubernetes Exposure to MLOps and collaboration with ML teams Experience with security protocols, data governance, and compliance frameworks Educational Qualifications: Bachelors or Masters in Computer Science, Data Engineering, Information Systems, or a related technical field Location - Mumbai, Delhi / NCR, Bengaluru , Kolkata, Chennai, Hyderabad, Ahmedabad, Pune, India

We are looking for Indias top 1% Computer Vision Engineers for a unique job opportunity to work with the industry leaders Who can be a part of the community? We are looking for top-tier Computer Vision (CV) Engineers with expertise in image/video processing, object detection, and generative AI If you have experience in this field then this is your chance to collaborate with industry leaders Whats in it for you? Pay above market standards The role is going to be contract based with project timelines from 2 12 months, or freelancing Be a part of an Elite Community of professionals who can solve complex AI challenges Work location could be: Remote (Highly likely) Onsite on client location Deccan AIs Office: Hyderabad or Bangalore Responsibilities: Develop and optimize computer vision models for tasks like object detection, image segmentation, and multi-object tracking Lead research on novel techniques using deep learning frameworks (TensorFlow, PyTorch, JAX) Build efficient computer vision pipelines and optimize models for real-time performance Deploy models using microservices (Docker, Kubernetes) and cloud platforms (AWS, GCP, Azure) Lead MLOps practices, including CI/CD pipelines, model versioning, and training optimizations Required Skills: Expert in Python, OpenCV, NumPy, and deep learning architectures (eg, ViTs, YOLO, Mask R-CNN) Strong knowledge in computer vision fundamentals, including feature extraction and multi-view geometry with experience in deploying and optimizing models with TensorRT, Open VINO, and cloud/edge solutions Proficient with MLOps tools (ML flow, DVC), CI/CD, and distributed training frameworks Experience in 3D vision, AR/VR, or LiDAR processing is a plus Nice to Have: Experience with multi-camera vision systems, LiDAR, sensor fusion, and reinforcement learning for vision tasks Exposure to generative AI models (eg, Stable Diffusion, GANs) and large-scale image processing (Apache Spark, Dask) Research publications or patents in computer vision and deep learning Location-Delhi NCR,Bangalore,Chennai,Pune,Kolkata,Ahmedabad,Mumbai,Hyderabad

Lead Analyst/Senior Software Engineer - Data Engineer with Python, Apache Spark, HDFS Job Overview : CGI is looking for a talented and motivated Data Engineer with strong expertise in Python, Apache Spark, HDFS, and MongoDB to build and manage scalable, efficient, and reliable data pipelines and infrastructure Youll play a key role in transforming raw data into actionable insights, working closely with data scientists, analysts, and business teams. Key Responsibilities: Design, develop, and maintain scalable data pipelines using Python and Spark. Ingest, process, and transform large datasets from various sources into usable formats. Manage and optimize data storage using HDFS and MongoDB. Ensure high availability and performance of data infrastructure. Implement data quality checks, validations, and monitoring processes. Collaborate with cross-functional teams to understand data needs and deliver solutions. Write reusable and maintainable code with strong documentation practices. Optimize performance of data workflows and troubleshoot bottlenecks. Maintain data governance, privacy, and security best practices. Required qualifications to be successful in this role: Minimum 6 years of experience as a Data Engineer or similar role. Strong proficiency in Python for data manipulation and pipeline development. Hands-on experience with Apache Spark for large-scale data processing. Experience with HDFS and distributed data storage systems. Strong understanding of data architecture, data modeling, and performance tuning. Familiarity with version control tools like Git. Experience with workflow orchestration tools (e.g., Airflow, Luigi) is a plus. Knowledge of cloud services (AWS, GCP, or Azure) is preferred. Bachelors or Masters degree in Computer Science, Information Systems, or a related field. Preferred Skills: Experience with containerization (Docker, Kubernetes). Knowledge of real-time data streaming tools like Kafka. Familiarity with data visualization tools (e.g., Power BI, Tableau). Exposure to Agile/Scrum methodologies. Skills: Hadoop Hive Python SQL English Note This role will require- 8 weeks of in-office work after joining, after which we will transition to a hybrid working model, with 2 days per week in the office. Mode of interview F2F Time : Registration Window -9am to 12.30 pm. Candidates who are shortlisted will be required to stay throughout the day for subsequent rounds of interviews Notice Period: 0-45 Days

Roles & Responsibilities: Data Engineering Leadership & Strategy: Lead and mentor a team of data engineers, fostering a culture of technical excellence and collaboration. Define and implement data engineering best practices, standards, and processes. Data Pipeline Architecture & Development: Design, build, and maintain scalable, robust, and efficient data pipelines for ingestion, transformation, and loading of data from various sources. Optimize data pipelines for performance, reliability, and cost-effectiveness. Implement data quality checks and monitoring systems to ensure data integrity. Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver high-quality data solutions. Cloud-Based Data Infrastructure: Design, implement, and manage cloud-based data infrastructure using platforms like AWS, Azure, or GCP. Leverage cloud services (e.g., data lakes, data warehouses, serverless computing) to build scalable and cost-effective data solutions. Leverage opensource airbyte , mage ai and similar Ensure data security, governance, and compliance within the cloud environment. Data Modeling & Warehousing: Design and implement data models to support business intelligence, reporting, and analytics. Optimize data warehouse performance for efficient querying and reporting. Collaboration & Communication: Collaborate effectively with cross-functional teams including product managers, software engineers, and business stakeholders. Requirements: Bachelor's or master's degree in computer science, Engineering, or a related field. 8+ years of proven experience in data engineering, with at least 3+ years in a lead role. Expertise in building and maintaining data pipelines using tools such as Apache Spark, Apache Kafka, Apache Beam, or similar. Proficiency in SQL and one or more programming languages like Python, Java, or Scala. Hands-on experience with cloud-based data platforms (AWS, Azure, GCP) and services. Locations : Mumbai, Delhi NCR, Bengaluru , Kolkata, Chennai, Hyderabad, Ahmedabad, Pune, Remote Work Timings: 2.30 pm - 11.30 pm IST

Key Responsibilities: Backend Service Development: Design and implement robust, scalable, and maintainable backend services using Python. Utilize appropriate frameworks and libraries to streamline development and enhance productivity. Integrate AI models and algorithms into backend services, ensuring efficient and reliable communication. AI Model Integration: Collaborate with data scientists and AI engineers to understand AI model requirements and specifications. Develop APIs and interfaces to facilitate seamless integration of AI models into backend services. Cloud Infrastructure Management: Deploy and manage backend services on cloud platforms (e.g., AWS, Azure & GCP). Leverage cloud-native technologies and services to optimize infrastructure costs and performance. Ensure the security and compliance of cloud infrastructure. Collaboration and Mentorship: Work collaboratively with a cross-functional team of engineers, data scientists, and project stakeholders. Provide technical guidance and mentorship to junior engineers. Qualifications and Skills: Bachelor's or Master's degree in Computer Science, Engineering, or a related field. 5+ years of experience in Python programming, with a focus on backend development. Strong understanding of object-oriented programming (OOP) principles and design patterns. Experience with Python web frameworks (e.g., Django, Flask) and RESTful API development. Proficiency in cloud technologies (e.g., AWS, Azure & GCP) and containerization (e.g., Docker & Kubernetes). Familiarity with AI principles, machine learning algorithms, and deep learning frameworks (e.g., TensorFlow, PyTorch). Preferred Qualifications: Experience with large-scale distributed systems and microservices architectures. Knowledge of data engineering principles and big data technologies (e.g., Apache Spark). Experience : 5+Years Job Location : Remote, Delhi NCR, Bangalore, Chennai, Pune, Kolkata, Ahmedabad, Mumbai, Hyderabad Work Timings 2.30 pm -11.30 pm

Working Model : Our flexible work arrangement combines both remote and in-office work, optimizing flexibility and productivity. This position will be part of Sapiens Digital (Data Suite) division, for more information about it, click here: Designation: Senior Developer Must Skills 4-5 Years Experience in Data Bricks, PySpark, SQL, Data warehousing Criterias Job Requirements General Job Description A seasoned, experienced professional with a full understanding of area of specialization; resolves a wide range of issues in creative ways. This job is the fully qualified, career-oriented, journey-level position. Pre - requisites Knowledge & Experience B.E (or equivalent) Extensive hands-on experience in Java development, including strong knowledge of core Java concepts, data structures, and algorithms. In-depth understanding of distributed data processing frameworks like Apache Spark , with specific expertise in Databricks . Proficiency in designing and building data pipelines for data extraction, transformation, and loading (ETL). Familiarity with big data technologies and concepts, including Hadoop, Hive, and HDFS. Proven experience in building scalable and high-performance data solutions for large datasets. Solid understanding of data modelling, database design, and data warehousing concepts. Knowledge of both SQL and NoSQL databases , and ability to choose the right database type based on project requirements. Demonstrated ability to write clean, maintainable, and efficient Java code for data processing and integration tasks. Experience with Java libraries commonly used in data engineering, such as Apache Kafka for streaming data. Extensive hands-on experience with Databricks for big data processing and analytics. Ability to set up and configure Databricks clusters and optimize their performance. Proficiency in Spark Data Frame and Spark SQL for data manipulation and querying. Understanding of data architecture principles and experience in designing data solutions that meet scalability and reliability requirements. Familiarity with cloud-based data platforms like AWS or Azure. Problem-Solving and Analytical Skills: Strong problem-solving skills and the ability to analyse complex data-related issues. Capacity to propose innovative and efficient solutions to data engineering challenges. Excellent communication skills, both verbal and written, with the ability to convey technical concepts to non-technical stakeholders effectively. Experience working collaboratively in cross-functional teams, including Data Scientists, Data Analysts, and business stakeholders. A strong inclination to stay updated with the latest advancements in data engineering, Java, and Databricks technologies. Adaptability to new tools and technologies to support evolving data requirements. Required Product/project Knowledge Ability to work in an agile development environment. Hand on experience in technical design document preparation Proven experience in fine tuning and identifying the potential bottle necks on the applications Required Skills Ability to work on tasks (POCs, Stories, CR's, Defects etc.) without taking much help. Technical ability includes Programming, Debugging and Logical skills. Ability to technically guide juniors in completion of POC, Stories, CR's, Defects etc Common Tasks Come up and follow process for: Technical compliance and documentation Code review Unit & Functional testing Deployment Ensures that the team is also following the process properly. Able to write at least two technical paper or present one tech talk in a year 100% Compliance to Sprint Plan. Required Soft Skills Providing technical leadership and mentoring to junior developers Collaboration and teamwork skills Self-motivated with strong initiative and excellent Communication Skills Abilities of becoming a technical activity leader Proactive and initiative approach Self-motivated, flexible and a team player Have good understanding of the requirements in the area of functionality being developed

We are looking for a Senior GCP Data Engineer / GCP Technical Lead with strong expertise in Google Cloud Platform (GCP), Apache Spark, and Python to join our growing data engineering team. The ideal candidate will have extensive experience working with GCP data services and should be capable of leading technical teams, designing robust data pipelines, and interacting directly with clients to gather requirements and ensure project delivery. Project Duration : 1 year and extendable Role & responsibilities Design, develop, and deploy scalable data pipelines and solutions using GCP services like DataProc and BigQuery. Lead and mentor a team of data engineers to ensure high-quality deliverables. Collaborate with cross-functional teams and client stakeholders to define technical requirements and deliver solutions aligned with business goals. Optimize data processing and transformation workflows for performance and cost-efficiency. Ensure adherence to best practices in cloud data architecture, data security, and governance. Mandatory Skills: Google Cloud Platform (GCP) especially DataProc and BigQuery Apache Spark Python Programming Preferred Skills: Experience in working with large-scale data processing frameworks. Exposure to DevOps/CI-CD practices in a cloud environment. Hands-on experience with other GCP tools like Cloud Composer, Pub/Sub, or Cloud Storage is a plus. Soft Skills: Strong communication and client interaction skills. Ability to work independently and as part of a distributed team. Excellent problem-solving and team management capabilities.

Hands on experience with Snowflake and Python a must. Hands on experience with Apache Spark a must. Hands on experience with DBT preferred. Experience with performance tuning SQL queries, Spark job, and stored procedures. An understanding of E-R data models (conceptual, logical, and physical).

Role & responsibilities Designing and building optimized data pipelines using cutting-edge technologies in a cloud environment to drive analytical insights. Constructing infrastructure for efficient ETL processes from various sources and storage systems. Leading the implementation of algorithms and prototypes to transform raw data into useful information. Architecting, designing, and maintaining database pipeline architectures, ensuring readiness for AI/ML transformations. Creating innovative data validation methods and data analysis tools. Ensuring compliance with data governance and security policies. Interpreting data trends and patterns to establish operational alerts. Developing analytical tools, programs, and reporting mechanisms. Conducting complex data analysis and presenting results effectively. Preparing data for prescriptive and predictive modeling. Continuously exploring opportunities to enhance data quality and reliability. Applying strong programming and problem-solving skills to develop scalable solutions.

Role: Performance Testing Specialist Databricks Pipelines Job Seniority: Advanced (4-6 years) OR Experienced (3-4 years) Location: Magarpatta City,Pune Unit: Amdocs Data and Intelligence Mandatory SKills: All Skills must be in the resume in the roles and responsibilities Strong understanding of Databricks, Apache Spark, and performance tuning techniques for distributed data processing systems. Hands-on experience in Spark (PySpark/Scala) performance profiling, partitioning strategies, and job parallelization. 2+ year s of experience in performance testing and load simulation of data pipelines. Solid skills in SQL, Snowflake, and analyzing performance via query plans and optimization hints. Familiarity with Azure Databricks, Azure Monitor, Log Analytics, or similar observability tools. Proficient in scripting (Python/Shell) for test automation and pipeline instrumentation. Experience with DevOps tools such as Azure DevOps, GitHub Actions, or Jenkins for automated testing. Comfortable working in Unix/Linux environments and writing shell scripts for monitoring and debugging. Notice Period: Only Serving NP candidate who can join in the month of June ( 15 days to Immediate) Excellent Communication SKills This is C2H role. Interested Candidate Share Resume at dipti.bhaisare@in.experis.com

We at Acxiom Technologies are hiring for Pyspark Developer for Mumbai Location Relevant Experience : 1 to 4 Years Location : Mumbai Mode of Work : Work From Office Notice Period : Upto 20 days. Job Description: Proven experience as a Pyspark Developer . Hands-on expertise with AWS Redshift . Strong proficiency in Pyspark , Spark , Python , and Hive . Solid experience with SQL . Excellent communication skills. Benefits of working at Acxiom: - Statutory Benefits - Paid Leaves - Phenomenal Career Growth - Exposure to Banking Domain About Acxiom Technologies: Acxiom Technologies is a leading software solutions services company that provides consulting services to global firms and has established itself as one of the most sought-after consulting organizations in the field of Data Management and Business Intelligence. Also here is our website address https://www.acxtech.co.in/ to give you a detailed overview of our company. Interested Candidates can share their resumes on 7977418669 Thank you.

JD: • Good experience in Apache Iceberg, Apache Spark, Trino • Proficiency in SQL and data modeling • Experience with open Data Lakehouse using Apache Iceberg • Experience with Data Lakehouse architecture with Apache Iceberg and Trino

Job Description About Oracle APAC ISV Business Oracle APAC ISV team is one of the fastest-growing and high-performing business units in APAC. We are a prime team that operates to serve a broad range of customers across the APAC region. ISVs are at the forefront of today's fastest-growing industries. Much of this growth stems from enterprises shifting toward adopting cloud-native ISV SaaS solutions. This transformation drives ISVs to evolve from traditional software vendors to SaaS service providers. Industry analysts predict exponential growth in the ISV market over the coming years, making it a key growth pillar for every hyperscaler. Our Cloud engineering team works on pitch-to-production scenarios of bringing ISVs solutions on the Oracle cloud (#oci) with an aim to provide a cloud platform for running their business which is better performant, more flexible, more secure, compliant to open-source technologies and offers multiple innovation options yet being most cost effective. The team walks along the path with our customers and are being regarded as a trusted techno-business advisors by them. Required Skills/Experience Your versatility and hands-on expertise will be your greatest asset as you deliver on time bound implementation work items and empower our customers to harness the full power of OCI. We also look for: Bachelor's degree in Computer Science, Information Technology, or a related field. Relevant certifications in AI Services on OCI and/or other cloud platforms (AWS, Azure, Google Cloud) 8+ years of professional work experience Proven experience with end-to-end AI solution implementation, from data integration to model deployment and optimization. Experience in design, build, and deployment of end-to-end AI solutions with a focus on LLMs and RAG workflows. Proficiency in frameworks such as TensorFlow, PyTorch, scikit-learn, Keras and programming languages such as Python, R, or SQL.Experience with data wrangling, data pipelines, and data integration tools. Hands-on experience with LLM frameworks and plugins, such as LangChain, LlamaIndex, VectorStores and Retrievers, LLM Cache, LLMOps (MLFlow), LMQL, Guidance, etc. Knowledge of containerization technologies such as Docker and orchestration tools like Kubernetes to scale AI models. Expertise in analytics platforms like Power BI, Tableau, or other business intelligence tools. Experience working with cloud platforms, particularly for AI and analytics workloads. Familiarity with cloud-based AI services like OCI AI, AWS SageMaker etc Experience with building and optimizing data pipelines for large-scale AI/ML applications using tools like Apache Kafka, Apache Spark, Apache Airflow, or similar. Excellent communication skills, with the ability to clearly explain complex AI and analytics concepts to non-technical stakeholders. Proven ability to work with diverse teams and manage client expectations Solid experience managing multiple implementation projects simultaneously while maintaining high-quality standards. Ability to develop and manage project timelines, resources, and budgets. Career Level - IC4 Responsibilities What Youll Do As a solution specialist, you will work closely with our cloud architects and key stakeholders of ISVs to propagate awareness and drive implementation of OCI native as well as open-source cloud-native technologies by ISV customers. Design, implement, and optimize AI and analytics solutions using OCI AI & Analytics Services that enable advanced analytics and AI use cases. Assist clients to architect & deploy AI systems that integrate seamlessly with existing client infrastructure, ensuring scalability, performance, and security. Support the deployment of machine learning models, including model training, testing, and fine-tuning. Ensure scalability, robustness, and performance of AI models in production environments. Design, build, and deploy end-to-end AI solutions with a focus on LLMs and Agentic AI workflows (including Proactive, Reactive, RAG etc.). Help customer migrate from other Cloud vendors AI platform or bring their own AI/ML models leveraging OCI AI services and Data Science platform. Design, propose and implement solution on OCI that helps customers move seamlessly when adopting OCI for their AI requirements Provides direction and specialist knowledge to clients in developing AI chatbots using ODA (Oracle digital Assistance), OIC (Oracle integration cloud) and OCI GenAI services. Configure, integrate, and customize analytics platforms and dashboards on OCI. Implement data pipelines and ensure seamless integration with existing IT infrastructure. Drive discussions on OCI GenAI and AI Platform across the region and accelerate implementation of OCI AI services into Production

Databricks Technical Leadership: Guide and mentor teams in designing and implementing Databricks solutions. Architecture & Design: Develop scalable data pipelines and architectures using Databricks Lakehouse. Data Engineering: Lead the ingestion and transformation of batch and streaming data. Performance Optimization: Ensure efficient resource utilization and troubleshoot performance bottlenecks. Security & Compliance: Implement best practices for data governance, access control, and compliance. Collaboration: Work closely with data engineers, analysts, and business stakeholders. Cloud Integration: Manage Databricks environments on Azure, AWS, or GCP. Monitoring & Automation: Set up monitoring tools and automate workflows for efficiency. Qualifications: 7+ years of experience in Databricks, Apache Spark, and big data processing. Proficiency in Python, Scala, or SQL. Strong knowledge of Delta Lake, Unity Catalog, and MLflow. Experience with ETL processes and cloud platforms. Excellent problem-solving and leadership skills.

RESPONSIBILITIES: Choosing the right technologies for our use cases, deploy and operate. Setting up Data stores structured, semi structured and non-structured. Secure data at rest via encryption Implement tool to access securely multiple data sources Implement solutions to run real-time analytics Use container technologies Required Experience & Skills: Experience in one of the following: Elastic Search, Cassandra, Hadoop, Mongo DB Experience in Spark and Presto/Trino Experience with microservice based architectures Experience on Kubernetes Experience of Unix/Linux environments is plus Experience of Agile/Scrum development methodologies is a plus Cloud knowledge a big plus (AWS/GCP) (Kubernetes/Docker) Be nice, respectful, able to work in a team Willingness to learn

We are looking for a highly skilled and experienced Data Management Lead (Architect) with 4 to 9 years of experience to design, implement, and manage data lake environments. The ideal candidate will have a strong background in data management, architecture, and analytics. ### Roles and Responsibility Design and implement scalable, secure, and high-performing data lake architectures. Select appropriate technologies and platforms for data storage, processing, and analytics. Define and enforce data governance, metadata management, and data quality standards. Collaborate with IT security teams to establish robust security measures. Develop and maintain data ingestion and integration processes from various sources. Provide architectural guidance and support to data scientists and analysts. Monitor the performance of the data lake and recommend improvements. Stay updated on industry trends and advancements in data lake technologies. Liaise with business stakeholders to understand their data needs and translate requirements into technical specifications. Create documentation and architectural diagrams to provide a clear understanding of the data lake structure and processes. Lead the evaluation and selection of third-party tools and services to enhance the data lake's capabilities. Mentor and provide technical leadership to the data engineering team. Manage the full lifecycle of the data lake, including capacity planning, cost management, and decommissioning of legacy systems. ### Job Requirements At least 4 years of hands-on experience in designing, implementing, and managing data lakes or large-scale data warehousing solutions. Proficiency with data lake technologies such as Hadoop, Apache Spark, Apache Hive, or Azure Data Lake Storage. Experience with cloud services like AWS (Amazon Web Services), Microsoft Azure, or Google Cloud Platform, especially with their data storage and analytics offerings. Knowledge of SQL and NoSQL database systems, including relational databases (e.g., MySQL, PostgreSQL) and NoSQL databases (e.g., MongoDB, Cassandra). Expertise in data modeling techniques and tools for both structured and unstructured data. Experience with ETL (Extract, Transform, Load) tools and processes, and understanding of data integration and transformation best practices. Proficiency in programming languages commonly used for data processing and analytics, such as Python, Scala, or Java. Familiarity with data governance frameworks and data quality management practices to ensure the integrity and security of data within the lake. Knowledge of data security principles, including encryption, access controls, and compliance with data protection regulations (e.g., GDPR, HIPAA). Experience with big data processing frameworks and systems, such as Apache Kafka for real-time data streaming and Apache Flink or Apache Storm for stream processing. Familiarity with data pipeline orchestration tools like Apache Airflow, Luigi, or AWS Data Pipeline. Understanding of DevOps practices, including continuous integration/continuous deployment (CI/CD) pipelines, and automation tools like Jenkins or GitLab CI. Skills in monitoring data lake performance, diagnosing issues, and optimizing storage and processing for efficiency and cost-effectiveness. Ability to manage projects, including planning, execution, monitoring, and closing, often using methodologies like Agile or Scrum. Self-starter, independent-thinker, curious and creative person with ambition and passion. Bachelor's Degree: A bachelor's degree in Computer Science, Information Technology, Data Science, or a related field is typically required. This foundational education provides the theoretical knowledge necessary for understanding complex data systems. Master's Degree (optional): A master's degree or higher in a relevant field such as Computer Science, Data Science, or Information Systems can be beneficial. It indicates advanced knowledge and may be preferred for more senior positions. Certifications (optional): Industry-recognized certifications can enhance a candidate's qualifications. Examples include AWS Certified Solutions Architect, Azure Data Engineer Associate, Google Professional Data Engineer, Cloudera Certified Professional (CCP), or certifications in specific technologies like Apache Hadoop or Spark. PowerBI or any other reporting platform experience is a must. Knowledge on Power Automate, Qlik View, or any other reporting platform is an added advantage. ITIL Foundation certification is preferred.

Key Skills: SCALA, Apache Spark, SQL, SparkSQL, Spark, Core Java, Java Roles and Responsibilities: Lead technical initiatives and contribute as a senior team member to achieve project goals and deadlines. Collaborate with team members to design, implement, and optimize software solutions aligned with organizational objectives. Build scalable, efficient, and high-performance pipelines and workflows for processing large amounts of batch and real-time data. Perform multidisciplinary work, supporting real-time streams, ETL pipelines, data warehouses, and reporting services. Recommend and advocate for technology upgrades to company leaders to ensure infrastructure remains robust and competitive. Design and develop microservices and data applications while ensuring seamless integration with other systems. Leverage Big Data technologies like Kafka, AWS S3, EMR, and Spark to handle data ingestion, transformation, and querying. Follow coding best practices, including unit testing, code reviews, code coverage, and maintaining comprehensive documentation. Conduct thorough code reviews to maintain quality, mentor junior team members, and promote continuous learning within the team. Enhance system performance through analysis and capacity planning, ensuring efficient and reliable software releases. Actively bring new and innovative solutions to address challenging software issues that arise throughout the product lifecycle. Implement and promote security protocols and data governance standards across development projects. Actively engage in Agile processes to foster collaboration and innovation within the team. Required job skills: Strong software design capabilities with a deep understanding of design patterns and performance optimizations. Proficiency in writing high-quality, well-structured code in Java and Scala. Expertise in SQL and relational databases, with advanced skills in writing efficient, complex queries and optimizing database performance. Expertise in cloud computing infrastructure, particularly AWS (Aurora MySQL, DynamoDB, EMR, Lambda, etc.). Solid experience with Big Data tools such as Apache Spark and Kafka. Ability to clearly document and communicate technical solutions to diverse audiences. Experience mentoring and conducting constructive code reviews to support team development. Familiarity with Agile methodologies and modern development tools. Skills Required: 10+ years experience in designing and developing enterprise level software solutions 3 years experience developing Scala / Java applications and microservices using Spring Boot 7 years experience with large volume data processing and big data tools such as Apache Spark, SQL, Scala, and Hadoop technologies 5 years experience with SQL and Relational databases 2 year Experience working with the Agile/Scrum methodology Education: Bachelors Degree in related field

Job Description Lead Data Engineer Position: Lead Data Engineer Location: Hyderabad (Work from Office Mandatory) Experience: 10+ years overall | 8+ years relevant in Data Engineering Notice Period: Immediate to 30 days. About the Role We are looking for a strategic and hands-on Lead Data Engineer to architect and lead cutting-edge data platforms that empower business intelligence, analytics, and AI initiatives. This role demands a deep understanding of cloud-based big data ecosystems, excellent leadership skills, and a strong inclination toward driving data quality and governance at scale. You will define the data engineering roadmap, architect scalable data systems, and lead a team responsible for building and optimizing pipelines across structured and unstructured datasets in a secure and compliant environment. Key Responsibilities 1. Technical Strategy & Architecture Define the vision and technical roadmap for enterprise-grade data platforms (Lakehouse, Warehouse, Real-Time Pipelines). Lead evaluation of data platforms and tools, making informed build vs. buy decisions. Design solutions for long-term scalability, cost-efficiency, and performance. 2. Team Leadership Mentor and lead a high-performing data engineering team. Conduct performance reviews, technical coaching, and participate in hiring/onboarding. Instill engineering best practices and a culture of continuous improvement. 3. Platform & Pipeline Engineering Build and maintain data lakes, warehouses, and lakehouses using AWS, Azure, GCP, or Databricks. Architect and optimize data models and schemas tailored for analytics/reporting. Manage large-scale ETL/ELT pipelines for batch and streaming use cases. 4. Data Quality, Governance & Security Enforce data quality controls: automated validation, lineage, anomaly detection. Ensure compliance with data privacy and governance frameworks (GDPR, HIPAA, etc.). Manage metadata and documentation for transparency and discoverability. 5. Cross-Functional Collaboration Partner with Data Scientists, Product Managers, and Business Teams to understand requirements. Translate business needs into scalable data workflows and delivery mechanisms. Support self-service analytics and democratization of data access. 6. Monitoring, Optimization & Troubleshooting Implement monitoring frameworks to ensure data reliability and latency SLAs. Proactively resolve bottlenecks, failures, and optimize system performance. Recommend platform upgrades and automation strategies. 7. Technical Leadership & Community Building Lead code reviews, define development standards, and share reusable components. Promote innovation, experimentation, and cross-team knowledge sharing. Encourage open-source contributions and thought leadership. Required Skills & Experience 10+ years of experience in data engineering or related domains. Expert in PySpark, Python, and SQL . Deep expertise in Apache Spark and other distributed processing frameworks. Hands-on experience with cloud platforms (AWS, Azure, or GCP) and services like S3, EMR, Glue, Databricks, Data Factory . Proficient in data warehouse solutions (e.g., Snowflake, Redshift, BigQuery) and RDBMS like PostgreSQL or SQL Server. Knowledge of orchestration tools (Airflow, Dagster, or cloud-native schedulers). Familiarity with CI/CD tools , Git, and Infrastructure as Code (Terraform, CloudFormation). Strong data modeling and lifecycle management understanding.