65 Spark Sql Jobs

Role & responsibilities: Key Responsibilities: Design, develop, and optimize Java-based backend services (Spring Boot / Microservices) for API integrations. Develop and maintain Spark SQL queries and data processing pipelines for large-scale data ingestion. Build Spark batch and streaming jobs to land raw data from multiple vendor APIs into data lakes or warehouses. Implement robust error handling, logging, and monitoring for data pipelines. Collaborate with cross-functional teams across geographies to define integration requirements and deliverables. Troubleshoot and optimize Spark SQL for performance and cost efficiency. Participate in Agile ceremonies, daily standups, and client discussions. EXPERTISE AND QUALIFICATIONS Required Skills: 4 to 8 years of relevant experience. Core Java (Java 8 or above) with proven API development experience. Apache Spark (Core, SQL, DataFrame APIs) for large-scale data processing. Spark SQL strong ability to write and optimize queries for complex joins, aggregations, and transformations. Experience with API integration (RESTful APIs, authentication, payload handling, and rate limiting). Hands-on with data ingestion frameworks and ETL concepts. Experience with MySQL or other RDBMS for relational data management. Proficiency in Git for version control. Strong debugging, performance tuning, and problem-solving skills. Ability to work with minimal supervision in a short-term, delivery-focused engagement. Nice to Have: Experience with Apache Kafka for real-time streaming integrations. Familiarity with AWS data services (S3, EMR, Glue). Exposure to NoSQL databases like Cassandra or MongoDB.

Required Skills: Design and develop data pipelines using Azure Data Factory, Spark SQL, PySpark, and Python Implement ETL/ELT processes for both structured and unstructured data Optimize queries and data models in Azure Synapse Analytics and OneLake Build and maintain Lakehouse and Datawarehouse architectures using Microsoft Fabric Develop interactive dashboards and reports in Power BI (DirectLake mode) Integrate Microsoft Purview for data governance and compliance Work with DevOps teams to automate deployment of Fabric assets via Azure DevOps and CI/CD pipelines Ensure data security and performance tuning for large-scale datasets Role & responsibilities

As a software developer, you will be working in a constantly evolving environment driven by technological advances and the strategic direction of the organization you are employed by. Your primary responsibilities will include creating, maintaining, auditing, and enhancing systems to meet specific needs, often based on recommendations from systems analysts or architects. You will be tasked with testing both hardware and software systems to identify and resolve system faults. Additionally, you will be involved in writing diagnostic programs and designing and developing code for operating systems and software to ensure optimal efficiency. In situations where necessary, you will also provide recommendations for future developments. Joining us offers numerous benefits, including the opportunity to work on challenging projects and solve complex technical problems. You can expect rapid career growth and the chance to assume leadership roles. Our mentorship program allows you to learn from experienced mentors and industry experts, while our global opportunities enable you to collaborate with clients from around the world and gain international experience. We offer competitive compensation packages and benefits to our employees. If you are passionate about technology and interested in working on innovative projects with a skilled team, pursuing a career as an Infosys Power Programmer could be an excellent choice for you. To be considered for this role, you must possess the following mandatory skills: - Proficiency in AWS Glue, AWS Redshift/Spectrum, S3, API Gateway, Athena, Step, and Lambda functions. - Experience with Extract Transform Load (ETL) and Extract Load & Transform (ELT) data integration patterns. - Expertise in designing and constructing data pipelines. - Development experience in one or more object-oriented programming languages, preferably Python. In terms of job specifications, we are looking for candidates who meet the following criteria: - At least 5 years of hands-on experience in developing, testing, deploying, and debugging Spark Jobs using Scala in the Hadoop Platform. - Profound knowledge of Spark Core and working with RDDs and Spark SQL. - Familiarity with Spark Optimization Techniques and Best Practices. - Strong understanding of Scala Functional Programming concepts like Try, Option, Future, and Collections. - Proficiency in Scala Object-Oriented Programming covering Classes, Traits, Objects (Singleton and Companion), and Case Classes. - Sound knowledge of Scala Language Features including the Type System and Implicit/Givens. - Hands-on experience working in the Hadoop Environment (HDFS/Hive), AWS S3, EMR. - Proficiency in Python programming. - Working experience with Workflow Orchestration tools such as Airflow and Oozie. - Experience with API calls in Scala. - Familiarity and exposure to file formats like Apache AVRO, Parquet, and JSON. - Desirable knowledge of Protocol Buffers and Geospatial data analytics. - Ability to write test cases using frameworks like scalatest. - Good understanding of Build Tools such as Gradle & SBT. - Experience using GIT, resolving conflicts, and working with branches. - Preferred experience in workflow systems like Airflow. - Strong programming skills focusing on data structures and algorithms. - Excellent analytical and communication skills. Candidates applying for this position should have: - 7-10 years of industry experience. - A BE/B.Tech in Computer Science or an equivalent qualification.,

As a candidate for the role, you are expected to have a thorough understanding of collaboration in software development teams and shared software repositories and their concepts. Additionally, you should possess experience with Jira for effective project management. You should also have practical experience with at least one of the following tools for reporting and analysis: SSIS, SSAS, SSRS, or Power BI. Your expertise in using these tools will be crucial for generating insights and reports. In terms of Azure Data Factory and Synapse pipelines development, you are required to have a comprehensive understanding of how Data Factory pipelines function and their integration with other Azure services. Your ability to design and implement pipelines, as well as utilize Databricks and/or Synapse Delta Lake for BI purposes, will be essential. Furthermore, your role will involve Spark development, where you will need practical experience in Spark and understanding of its interaction within Synapse and Databricks Notebooks. Knowledge of Spark design, data structures, mpp flows, and expertise in Spark SQL usage within Databricks or Synapse Notebooks is also expected from you. Having an understanding of Kafka and its role in streaming is essential, along with knowledge of integrating Kafka with Spark and other persistence services. This is a permanent position requiring a minimum of 7 years of experience in Azure cloud and Git. Your expertise in these areas will be crucial for success in this role.,

As a Principal Architect - Data & Cloud at Quantiphi, you will bring your 14-20 years of experience in Technical, Solutioning, and Analytical roles to lead the way in architecting, designing, and implementing end-to-end data pipelines and data integration solutions for structured and unstructured data sources and targets. With a focus on Cloud platforms such as GCP, AWS, and Azure, you will be responsible for building and managing Data Lakes, Data Warehouse, Data Integration, Data Migration, and Business Intelligence/Artificial Intelligence solutions. Your role will involve understanding business requirements and translating them into functional and non-functional areas, defining boundaries in terms of Availability, Scalability, Performance, Security, and Resilience. You will leverage your expertise in various Data Integration and ETL technologies on Cloud, including Spark, Pyspark/Scala, Dataflow, DataProc, and more. Additionally, you will have the opportunity to work with traditional ETL tools like Informatica, DataStage, OWB, Talend, and others. Your deep knowledge of Cloud and On-Premise Databases such as Cloud SQL, Cloud Spanner, Big Table, RDS, Aurora, and others will be instrumental in architecting scalable data warehouse solutions on Cloud platforms like Big Query or Redshift. Moreover, your exposure to No-SQL databases and experience with data integration, storage, and data pipeline tool sets will be crucial in designing optimized data analytics solutions. Being a thought leader in architecture design and development of cloud data analytics solutions, you will collaborate with internal and external stakeholders to present solutions, support sales teams in building proposals, and lead discovery workshops with potential customers globally. Your role will also involve mentoring young talent, contributing to building Assets and Accelerators, and ensuring the successful delivery of projects on parameters of Schedule, Quality, and Customer Satisfaction. The position offers the experience of working in a high-growth startup in the AI, Decision Science, and Big Data Domain, along with the opportunity to be part of a diverse and proactive team that constantly raises the bar in translating data into tangible business value for clients. Additionally, flexible remote working options are available to foster productivity and work-life balance. If you are passionate about innovation, excellence, and growth, and enjoy working with a dynamic team of tech enthusiasts, Quantiphi is the place for you to shape your career in Data & Cloud architecture. Join us on our journey of digital transformation and be a part of creating impactful solutions that drive business success.,

About the Company At Tide, we are dedicated to creating a business management platform that aims to streamline operations for small businesses, enabling them to save valuable time and resources. Our services include offering business accounts, banking solutions, as well as a range of integrated administrative tools spanning from invoicing to accounting. Established in 2017, Tide has garnered a user base of over 1 million small businesses globally, catering to SMEs in the UK, India, and Germany. Headquartered in central London, we also have offices in Sofia, Hyderabad, Delhi, Berlin, and Belgrade, with a team of more than 2,000 employees. Tide is on a trajectory of rapid growth, continuously venturing into new markets and products, and continuously seeking individuals who are enthusiastic and motivated to join us in our mission to empower small businesses by aiding them in saving time and resources. About the Role We are in search of an experienced Senior Data Engineer with exceptional skills in PySpark to join our ML/Data engineering team. This team's responsibilities encompass feature development, data quality assessments, deployment, and integration of ML models with backend services, and enhancing the overall Tide platform. As a Senior Data Engineer, you will play a crucial role in designing, developing, and optimizing our upcoming data pipelines and platforms. Your tasks will involve working with extensive datasets, addressing intricate data challenges, and contributing to the creation of robust, scalable, and efficient data solutions that drive business value. This position presents an exciting opportunity for individuals who are passionate about big data technologies, performance optimization, and constructing resilient data infrastructure. As a Data Engineer, You Will: - Focus on Performance Optimization: Identify and resolve complex performance bottlenecks in PySpark jobs and Spark clusters, utilizing Spark UI, query plans, and advanced optimization techniques. - Lead Design & Development: Spearhead the design and implementation of scalable, fault-tolerant ETL/ELT pipelines using PySpark for batch and real-time data processing. - Collaborate on Data Modeling: Work alongside data scientists, analysts, and product teams to design efficient data models for analytical and operational use cases. - Ensure Data Quality & Governance: Implement strong data quality checks, monitoring, and alerting mechanisms to maintain data accuracy, consistency, and reliability. - Contribute to Architectural Decisions: Aid in shaping the data architecture strategy, assess new technologies, and implement best practices to enhance the data platform's capabilities. - Uphold Best Practices: Promote engineering best practices, participate in code reviews, and mentor junior data engineers. - Foster Collaboration: Work closely with cross-functional teams to deliver impactful data solutions. Qualifications: - Possess 8+ years of professional experience in data engineering, with a minimum of 4+ years focusing on PySpark development in a production environment. - Demonstrate expert-level proficiency in PySpark, including Spark SQL, DataFrames, RDDs, and understanding Spark's architecture. - Showcase hands-on experience in optimizing PySpark performance, debugging slow jobs, and handling common issues in large datasets. - Exhibit strong programming skills in Python, proficiency in SQL, and familiarity with data warehousing concepts. - Prior experience with distributed data storage solutions and version control systems. - Strong problem-solving abilities, attention to detail, and excellent communication skills. - Hold a Bachelor's or Master's degree in Computer Science, Engineering, or a related field. What We Offer: - Competitive salary - Health and life insurance for self and family - OPD benefits - Mental well-being support - Learning and development budget - WFH setup allowance - Generous leave policy - Stock options Tide Ways of Working: At Tide, we embrace a flexible workplace model that accommodates both in-person and remote work to cater to the diverse needs of our teams. While we support remote work, we believe in the importance of face-to-face interactions to foster collaboration and team spirit, making our offices hubs for innovation and community building. Tide is a Place for Everyone: We promote a transparent and inclusive environment where every voice is valued and heard. Your personal data will be handled by Tide for recruitment purposes in accordance with our Recruitment Privacy Notice.,

As a Spark and Scala Developer at Infosys, you will play a crucial role in facilitating digital transformation for our clients within a global delivery model. Your responsibilities will include conducting independent research on technologies, providing recommendations for suitable solutions, and contributing to technology-specific best practices and standards. It will be essential for you to effectively interact with key stakeholders and utilize your technical expertise across various stages of the Software Development Life Cycle. As part of our learning culture, teamwork and collaboration are highly encouraged, excellence is acknowledged, and diversity is both respected and valued. Required Qualifications: - Must be located within commuting distance of Raleigh, NC, Charlotte, NC, or Richardson, TX, or be open to relocating to these areas. - A Bachelor's degree or foreign equivalent from an accredited institution is required. Alternatively, three years of progressive experience in the specialty can be considered in place of each year of education. - All candidates authorized to work in the United States are welcome to apply. - Minimum of 4 years of experience in Information Technology. - Profound understanding of distributed computing principles and big data technologies. - At least 3 years of hands-on experience working with Apache Spark, Scala, Spark SQL, and Starburst. - Knowledge of data serialization formats like Parquet, Avro, or ORC. - Familiarity with data processing and transformation techniques. Preferred Qualifications: - Hands-on experience with data lakes, data warehouses, and ETL processes. - Solid comprehension of Agile software development frameworks. - Previous experience in the Banking domain. - Exceptional communication and analytical skills. - Ability to collaborate in teams within a diverse, multi-stakeholder environment involving Business and Technology teams. - Willingness and experience to work in a global delivery environment. This role may involve prolonged periods of sitting and computer work. Effective communication via telephone, email, or face-to-face interactions is essential. Travel might be necessary based on job requirements. About Us: Infosys is a renowned global leader in next-generation digital services and consulting. We assist clients in over 50 countries in navigating their digital transformation journey. With more than four decades of experience in managing the systems and operations of global enterprises, we expertly guide our clients through their digital evolution. By empowering enterprises with an AI-powered core to prioritize change execution and delivering agile digital solutions at scale, we aim to achieve exceptional levels of performance and customer satisfaction. Our commitment to continuous improvement is driven by an always-on learning agenda, enabling the transfer of digital skills, expertise, and innovative ideas from our thriving innovation ecosystem.,

As a Senior AWS Data Engineer Cloud Data Platform at Teamware Solutions, a division of Quantum Leap Consulting Pvt. Ltd, located in Bangalore, you will be responsible for end-to-end implementation of Cloud data engineering solutions like Enterprise Data lake and Data hub in AWS. Working onsite in an office environment for 5 days a week, you will collaborate with the Offshore Manager and Onsite Business Analyst to understand the requirements and deliver scalable, distributed, cloud-based enterprise data solutions. You should have a strong background in AWS cloud technology, with 4-8 years of hands-on experience. Proficiency in architecting and delivering highly scalable solutions is a must, along with expertise in Cloud data engineering solutions, Lambda or Kappa Architectures, Data Management concepts, and Data Modelling. You should be proficient in AWS services such as EMR, Glue, S3, Redshift, and DynamoDB, as well as have experience in Big Data frameworks like Hadoop and Spark. Additionally, you must have hands-on experience with AWS compute and storage services, AWS Streaming Services, troubleshooting and performance tuning in Spark framework, and knowledge of Application DevOps tools like Git and CI/CD Frameworks. Familiarity with AWS CloudWatch, Cloud Trail, Account Config, Config Rules, security, key management, data migration processes, and analytical skills is required. Good communication and presentation skills are essential for this role. Desired skills include experience in building stream-processing systems, Big Data ML toolkits, Python, Offshore/Onsite Engagements, flow tools like Airflow, Nifi or Luigi, and AWS services like STEP & Lambda. A professional background in BE/B.Tech/MCA/M.Sc/M.E/M.Tech/MBA is preferred, and an AWS certified Data Engineer certification is recommended. If you are interested in this position and meet the qualifications mentioned above, please send your resume to netra.s@twsol.com.,

As a Senior Data Engineer, you will be an integral part of our team, contributing to the design, implementation, and maintenance of scalable data pipelines utilizing cutting-edge tools like Airflow, PySpark, and Databricks. Your responsibilities will extend to supporting machine learning models in a production environment, crafting data products from vast datasets, and engaging with diverse teams to grasp data requirements effectively. Your primary duties will include designing and deploying robust data pipelines by leveraging Airflow, PySpark, and Databricks. You will also be tasked with ensuring the reliability and performance of machine learning models in production, creating data products that align with business objectives, and collaborating closely with cross-functional teams to address their data-related needs. Furthermore, you will play a key role in implementing process monitoring mechanisms to uphold data quality and reliability, including support for ACID transactions. Additionally, you will be expected to provide technical guidance and assistance to your colleagues whenever necessary. To qualify for this role, you should possess a Bachelor's or Master's degree in Computer Science, Engineering, or a related field, along with a minimum of 5 years of experience in a data engineering capacity. Your background should demonstrate a successful history of delivering scalable and dependable data solutions. Proficiency in Databricks, AWS, and Airflow ETL architectures, particularly in managing data streams from real-time consumer applications, is essential. Moreover, you should exhibit a strong command of Python, PySpark, and Spark SQL for data processing and analysis, coupled with a solid understanding of SQL databases and data modeling principles. Familiarity with Databricks Lakeview and other data visualization tools for monitoring and reporting purposes will be advantageous. Your ability to work autonomously and collaboratively within a team environment, combined with exceptional problem-solving and communication skills, will be critical for excelling in this role.,

Join us as a Principal Engineer in a challenging role where you will be responsible for designing and engineering software with a primary focus on customer or user experience. You will actively contribute to our architecture, design, and engineering center of excellence, collaborating to enhance the bank's overall software engineering capability. This role offers valuable stakeholder exposure, allowing you to build and leverage relationships while honing your technical talents. The position is available at the vice president level. As a Principal Engineer, your primary responsibilities will include creating exceptional customer outcomes through innovative engineering solutions to both existing and new challenges. You will design technology solutions that are customer-centric, innovative, high performance, secure, and robust. You will work closely with software engineers to produce and prototype innovative ideas, engage with domain and enterprise architects to validate and incorporate relevant architectures, and lead functional engineering teams. Additionally, you will manage end-to-end product implementations, drive demos, and engage stakeholders across platforms. Your role will involve designing and developing software with a focus on automating build, test, and deployment activities, while promoting the discipline of software engineering throughout the business. In addition to the above responsibilities, you will define, create, and oversee engineering and design solutions with a focus on end-to-end automation, simplification, resilience, security, performance, scalability, and reusability. You will collaborate with software engineers within a platform or feature team to design and engineer complex software, scripts, and tools that enable the delivery of bank platforms, applications, and services. You will also be responsible for defining architecture models and roadmaps for application and software components, driving common usability across products and domains. Furthermore, you will design, test, and implement working code, apply Agile methods to software development, and utilize DevOps techniques. To excel in this role, you should have significant experience in software engineering, software or database design, architecture, and working within a DevOps and Agile framework. You should possess an expert understanding of market trends, technologies, and tools, along with experience in implementing programming best practices related to scalability, automation, virtualization, optimization, availability, and performance. Additionally, hands-on experience with gathering business requirements, translating them into technical user stories, and leading functional solution design within the banking domain and CRM (MS Dynamics) is essential. Proficiency in PowerApps, D365 (including Custom Pages), Power BI (SQL, DAX, Power Query, Data Modeling, RLS, Azure, Lakehouse, Python, Spark SQL), and designing or implementing APIs is required. You should also demonstrate the ability to quickly understand and translate product and business requirements into technical solutions.,

NTT DATA is looking for a Databricks Engineer to join their team in Bangalore, Karnataka, India. As a Databricks Engineer, you will be responsible for various tasks related to data extraction, ETL pipeline modernization, job design, development, automation, metadata management, documentation, testing, collaboration, performance tuning, security, governance, and compliance. Your primary job duties will include extracting and analyzing data from SQL Server and Teradata sources, translating legacy SQL/DataStage transformations into Databricks-native code, building and orchestrating jobs within Databricks using tools like Databricks Workflows, Delta Lake, and Auto Loader, generating and maintaining data flow diagrams and job documentation, designing and executing unit tests and integration tests for data pipelines, optimizing data ingestion and transformation for performance and cost efficiency, ensuring compliance with data governance policies, and implementing access control via Unity Catalog. To be successful in this role, you must have a strong understanding of ETL/ELT principles and data pipelines, proficiency with Databricks platform and PySpark or Spark SQL, advanced SQL skills, familiarity with Teradata and SQL Server environments, ability to read and understand data models, schemas, and ERDs, basic proficiency with Git for code versioning, ability to write and validate unit/integration tests, strong communication skills, and an awareness of security and governance principles. NTT DATA is a global innovator of business and technology services, serving 75% of the Fortune Global 100. They are committed to helping clients innovate, optimize, and transform for long-term success. As a Global Top Employer, NTT DATA has diverse experts in more than 50 countries and a robust partner ecosystem. Their services include business and technology consulting, data and artificial intelligence, industry solutions, as well as the development, implementation, and management of applications, infrastructure, and connectivity. NTT DATA is a leading provider of digital and AI infrastructure and is part of the NTT Group, investing in R&D to help organizations and society move confidently into the digital future. Visit us at us.nttdata.com.,

Join us as a Principal Engineer. This challenging role will involve designing and engineering software with a primary focus on customer or user experience. You will actively contribute to our architecture, design, and engineering center of excellence, collaborating to enhance the bank's software engineering capability. This role, offered at the vice president level, provides valuable stakeholder exposure, the opportunity to build and leverage relationships, and a chance to refine your technical skills. As a Principal Engineer, your responsibilities will include creating exceptional customer outcomes through innovative engineering solutions for both existing and new challenges. You will work with software engineers to produce and prototype innovative ideas, collaborate with domain and enterprise architects to validate and incorporate relevant architectures, and lead functional engineering teams. Your role will involve managing end-to-end product implementations, driving demos and stakeholder engagement across platforms, and focusing on automating build, test, and deployment activities. Additionally, you will play a key part in developing the discipline of software engineering within the organization. You will also be responsible for defining, creating, and overseeing engineering and design solutions with a strong emphasis on end-to-end automation, simplification, resilience, security, performance, scalability, and reusability. Working within a platform or feature team, you will collaborate with software engineers to design and engineer complex software, scripts, and tools that enable the delivery of bank platforms, applications, and services. Your role will involve defining and developing architecture models and roadmaps for application and software components, ensuring they meet business and technical requirements, and driving consistent usability across products and domains. You will design, test, and implement working code while applying Agile methods and DevOps techniques to software development. The skills required for this role include significant experience in software engineering, software or database design and architecture, and working within a DevOps and Agile framework. You should possess an expert understanding of the latest market trends, technologies, and tools, along with demonstrable experience in implementing programming best practices, particularly related to scalability, automation, virtualization, optimization, availability, and performance. Additionally, you should have strong experience in gathering business requirements, translating them into technical user stories, and leading functional solution design, especially within the banking domain and CRM (MS Dynamics). Proficiency in PowerApps, D365 (including Custom Pages), and frontend configuration, as well as familiarity with Power BI (SQL, DAX, Power Query, Data Modeling, RLS, Azure, Lakehouse, Python, Spark SQL) is required. A background in designing or implementing APIs and the ability to quickly understand and translate product and business requirements into technical solutions are also essential for this role.,

As a Senior Data Engineer, you will be responsible for designing and developing scalable data pipelines and notebooks using Microsoft Fabric or Synapse Analytics. You should have a research-oriented mindset in Data Projects, thinking outside of the box and focusing on future needs. Your role will involve building and managing Lakehouses and Data Warehouses using Fabrics OneLake architecture, integrating data from diverse sources into Fabric, and collaborating with BI developers for seamless integration with Power BI and other reporting tools. Additionally, you will be required to implement data governance, security, and compliance within the Fabric ecosystem, optimize data storage and processing for performance and cost-efficiency, monitor and troubleshoot data workflows to ensure high data quality and reliability, and document architecture, data models, and processes. It is essential to have experience in automated functional testing along with development. Key Skills required for this role include Pyspark, Data Modelling, Spark SQL, and proficiency in Microsoft Fabric, including an understanding of Shortcuts, Mirroring, Data flows, and all the features. Familiarity with Data Ingestion Design Patterns is also a desired skill for this position.,

Are you intellectually curious and passionate about promoting solutions across organizational boundaries Join the Consumer & Community Banking (CCB) Stress Testing Transformation team for a dynamic opportunity to design and build creative solutions for the future of stress testing and annual CCAR exercises. As a Senior Associate in the Stress Testing Transformation Solution team, you will be a strategic thinker who is passionate about designing and building creative solutions for the future of Stress Testing. You will spend your time solving complex problems, demonstrating strategic thought leadership, and designing the way our stakeholders operate. By leveraging a deep understanding of CCB Stress Testing processes and extensive Finance domain knowledge, you will build scalable solutions that optimize process efficiencies, use data assets effectively, and advance platform capabilities. Responsibilities: - Collaborate with cross-functional teams to lead the design and implementation of end-to-end solutions for Stress Testing, addressing business problems with various technical solutions. - Provide expertise in process re-engineering and guidance based on the roadmap for large-scale Stress Testing transformation initiatives. - Assess, challenge, and provide solutions for Stress Testing processes, focusing on data sources, with the ability to influence and drive the roadmap. - Evaluate, recommend, and develop solutions and architecture, including integration with APIs, Python, AI/ML technology, and other enterprise applications. - Leverage data and best-in-class tools to improve processes and controls, enable cross-business applications, and embrace a consistent framework. - Simplify complex issues into manageable steps and achievements. - Eliminate manual reporting, reengineer processes, and increase the ability to generate insights faster through an integrated data and platform approach. Required Qualifications: - Bachelor's degree in engineering or a related field. - Experience with business intelligence, analytics, and data wrangling tools such as Alteryx, SAS, or Python. - Experience with relational databases, optimizing SQL to extract and summarize large datasets, report creation, and ad-hoc analyses. - Experience with Hive, Spark SQL, Impala, or other big-data query tools. - Ability to understand the underlying business context beyond raw data and identify business opportunities hidden in data. - Collaborative skills to work with global teams in a fast-paced, results-driven environment. - Strong problem-solving and analytical skills with a transformation mindset. Preferred Qualifications: - Experience with Databricks, SQL, Python, or other data platforms. - 8+ years of experience in Analytics Solution and Data Analytics, preferably related to the financial services domain.,

As a Senior Engineer at Impetus Technologies, you will play a crucial role in designing, developing, and deploying scalable data processing applications using Java and Big Data technologies. Your responsibilities will include collaborating with cross-functional teams, mentoring junior engineers, and contributing to architectural decisions to enhance system performance and scalability. Your key responsibilities will revolve around designing and maintaining high-performance applications, implementing data ingestion and processing workflows using frameworks like Hadoop and Spark, and optimizing existing applications for improved performance and reliability. You will also be actively involved in mentoring junior engineers, participating in code reviews, and staying updated with the latest technology trends in Java and Big Data. To excel in this role, you should possess a strong proficiency in Java programming language, hands-on experience with Big Data technologies such as Apache Hadoop and Apache Spark, and an understanding of distributed computing concepts. Additionally, you should have experience with data processing frameworks and databases, strong problem-solving skills, and excellent communication and teamwork abilities. In this role, you will collaborate with a diverse team of skilled engineers, data scientists, and product managers who are passionate about technology and innovation. The team environment encourages knowledge sharing, continuous learning, and regular technical workshops to enhance your skills and keep you updated with industry trends. Overall, as a Senior Engineer at Impetus Technologies, you will be responsible for designing and developing scalable Java applications for Big Data processing, ensuring code quality and performance, and troubleshooting and optimizing existing systems to enhance performance and scalability. Qualifications: - Strong proficiency in Java programming language - Hands-on experience with Big Data technologies such as Hadoop, Spark, and Kafka - Understanding of distributed computing concepts - Experience with data processing frameworks and databases - Strong problem-solving skills - Knowledge of version control systems and CI/CD pipelines - Excellent communication and teamwork abilities - Bachelor's or master's degree in Computer Science, Engineering, or related field preferred Experience: 7 to 10 years Job Reference Number: 13131,

Join us as a Senior Automation QA at Barclays, where you will be responsible for supporting the successful delivery of location strategy projects to plan, budget, agreed quality and governance standards. Spearhead the evolution of the digital landscape, driving innovation and excellence by harnessing cutting-edge technology to revolutionize digital offerings, ensuring unparalleled customer experiences. To be successful in this role, you should have experience with Spark SQL, Python/Pyspark scripting, and ETL concepts. Additionally, skills in AWS exposure and Jupyter Notebook are highly valued. Your key critical skills relevant for success in this role may be assessed, such as risk and controls, change and transformation, business acumen, strategic thinking, and digital and technology, along with job-specific technical skills. This role is based out of Pune. Purpose of the role: Design, develop, and execute testing strategies to validate functionality, performance, and user experience. Collaborate with cross-functional teams to identify and resolve defects, continuously improve testing processes and methodologies to ensure software quality and reliability. Accountabilities: - Develop and implement comprehensive test plans and strategies to validate software functionality and ensure compliance with established quality standards. - Create and execute automated test scripts, leveraging testing frameworks and tools for early detection of defects. - Collaborate with cross-functional teams to analyze requirements, participate in design discussions, and contribute to the development of acceptance criteria. - Conduct root cause analysis for identified defects, working closely with developers for defect resolution. - Participate in code reviews, promote a culture of code quality and knowledge sharing. - Stay informed of industry technology trends and innovations, contribute to the organization's technology communities. As an Assistant Vice President, you are expected to advise and influence decision-making, contribute to policy development, and ensure operational effectiveness. Lead a team performing complex tasks, set objectives, coach employees, and demonstrate leadership behaviours to create an environment for colleagues to thrive. For an individual contributor, lead collaborative assignments, guide team members, identify new directions for assignments, consult on complex issues, and contribute to risk mitigation and policy development. All colleagues are expected to demonstrate the Barclays Values of Respect, Integrity, Service, Excellence, and Stewardship, along with the Barclays Mindset to Empower, Challenge, and Drive.,

As a Senior Data Engineering Architect at Iris Software, you will play a crucial role in leading enterprise-level data engineering projects on public cloud platforms like AWS, Azure, or GCP. Your responsibilities will include engaging with client managers to understand their business needs, conceptualizing solution options, and finalizing strategies with stakeholders. You will also be involved in team building, delivering Proof of Concepts (PoCs), and enhancing competencies within the organization. Your role will focus on building competencies in Data & Analytics, including Data Engineering, Analytics, Data Science, AI/ML, and Data Governance. Staying updated with the latest tools, best practices, and trends in the Data and Analytics field will be essential to drive innovation and excellence in your work. To excel in this position, you should hold a Bachelor's or Master's degree in a Software discipline and have extensive experience in Data architecture and implementing large-scale Data Lake/Data Warehousing solutions. Your background in Data Engineering should demonstrate leadership in solutioning, architecture, and successful project delivery. Strong communication skills in English, both written and verbal, are essential for effective collaboration with clients and team members. Proficiency in tools such as AWS Glue, Redshift, Azure Data Lake, Databricks, Snowflake, and databases, along with programming skills in Spark, Spark SQL, PySpark, and Python, are mandatory competencies for this role. Joining Iris Software offers a range of perks and benefits designed to support your financial, health, and overall well-being. From comprehensive health insurance and competitive salaries to flexible work arrangements and continuous learning opportunities, we are dedicated to providing a supportive and rewarding work environment where your success and happiness are valued. If you are inspired to grow your career in Data Engineering and thrive in a culture that values talent and personal growth, Iris Software is the place for you. Be part of a dynamic team where you can be valued, inspired, and encouraged to be your best professional and personal self.,

We are looking for a highly skilled and experienced Senior Python & ML Engineer with expertise in PySpark, machine learning, and large language models (LLMs). You will play a key role in designing, developing, and implementing scalable data pipelines, machine learning models, and LLM-powered applications. In this role, you will need to have a solid understanding of Python's ecosystem, distributed computing using PySpark, and practical experience in AI optimization. Your responsibilities will include designing and maintaining robust data pipelines with PySpark, optimizing PySpark jobs for efficiency on large datasets, and ensuring data integrity throughout the pipeline. You will also be involved in developing, training, and deploying machine learning models using key ML libraries such as scikit-learn, TensorFlow, and PyTorch. Tasks will include feature engineering, model selection, hyperparameter tuning, and integrating ML models into production systems for scalability and reliability. Additionally, you will research, experiment with, and integrate state-of-the-art Large Language Models (LLMs) into applications. This will involve developing solutions that leverage LLMs for tasks like natural language understanding, text generation, summarization, and question answering. You will also fine-tune pre-trained LLMs for specific business needs and datasets, and explore techniques for prompt engineering, RAG (Retrieval Augmented Generation), and LLM evaluation. Collaboration is key in this role, as you will work closely with data scientists, engineers, and product managers to understand requirements and translate them into technical solutions. You will mentor junior team members, contribute to best practices for code quality, testing, and deployment, and stay updated on the latest advancements in Python, PySpark, ML, and LLMs. Furthermore, you will be responsible for deploying, monitoring, and maintaining models and applications in production environments using MLOps principles. Troubleshooting and resolving issues related to data pipelines, ML models, and LLM applications will also be part of your responsibilities. To be successful in this role, you should have a Bachelor's or Master's degree in Computer Science, Data Science, Engineering, or a related field. Strong proficiency in Python programming, PySpark, machine learning, and LLMs is essential. Experience with cloud platforms like AWS, Azure, or GCP is preferred, along with strong problem-solving, analytical, communication, and teamwork skills. Nice-to-have skills include familiarity with R and Shiny, streaming data technologies, containerization technologies, MLOps tools, graph databases, and contributions to open-source projects.,

BizViz is a company that offers a comprehensive view of a business's data, catering to various industries and meeting the diverse needs of business executives. With a dedicated team of over 50 professionals working on the BizViz platform for several years, the company aims to develop technological solutions that provide our clients with a competitive advantage. At BizViz, we are committed to the success of our customers, striving to create applications that align with their unique visions and requirements. We steer clear of generic ERP templates, offering businesses a more tailored solution. As a Big Data Engineer at BizViz, you will join a small, agile team of data engineers focused on building an innovative big data platform for enterprises dealing with critical data management and diverse application stakeholders at scale. The platform handles data ingestion, warehousing, and governance, allowing developers to create complex queries efficiently. With features like automatic scaling, elasticity, security, logging, and data provenance, our platform empowers developers to concentrate on algorithms rather than administrative tasks. We are seeking engineers who are eager for technical challenges, to enhance our current platform for existing clients and develop new capabilities for future customers. Key Responsibilities: - Work as a Senior Big Data Engineer within the Data Science Innovation team, collaborating closely with internal and external stakeholders throughout the development process. - Understand the needs of key stakeholders to enhance or create new solutions related to data and analytics. - Collaborate in a cross-functional, matrix organization, even in ambiguous situations. - Contribute to scalable solutions using large datasets alongside other data scientists. - Research innovative data solutions to address real market challenges. - Analyze data to provide fact-based recommendations for innovation projects. - Explore Big Data and other unstructured data sources to uncover new insights. - Partner with cross-functional teams to develop and execute business strategies. - Stay updated on advancements in data analytics, Big Data, predictive analytics, and technology. Qualifications: - BTech/MCA degree or higher. - Minimum 5 years of experience. - Proficiency in Java, Scala, Python. - Familiarity with Apache Spark, Hadoop, Hive, Spark SQL, Spark Streaming, Apache Kafka. - Knowledge of Predictive Algorithms, Mllib, Cassandra, RDMS (MYSQL, MS SQL, etc.), NOSQL, Columnar Databases, Big table. - Deep understanding of search engine technology, including Elasticsearch/Solr. - Experience in Agile development practices such as Scrum. - Strong problem-solving skills for designing algorithms related to data cleaning, mining, clustering, and pattern recognition. - Ability to work effectively in a matrix-driven organization under varying circumstances. - Desirable personal qualities: creativity, tenacity, curiosity, and a passion for technical excellence. Location: Bangalore To apply for this position, interested candidates can send their applications to careers@bdb.ai.,

As a Lead database engine developer, you will play a crucial role in enhancing our database engine to operate at an exabyte scale. Our analytical database engine processes trillions of data points daily, enabling rapid queries with an impressive 60 ms response time at P50. Your technical expertise and leadership will be pivotal in ensuring that our system seamlessly manages exabytes of data on a daily basis. Your responsibilities will include developing and executing innovative technical strategies for our database engine that align with Newrelic's business objectives. You will focus on optimizing scalability and performance to handle exabyte-scale data while maintaining exceptional query performance. Enhancing data ingestion pipelines to support trillions of data points, collaborating with cross-functional teams to fine-tune query execution and response times, and ensuring high reliability, fault tolerance, and disaster recovery capabilities for mission-critical cloud services will also be part of your role. To excel in this position, you should possess at least 7 years of experience in database engine development. You must have exposure to core areas of Database Products, including Query Optimization and Execution, Distributed database & Parallel Query Execution, and Expression optimization & evaluation. Proficiency in C, C++, Unix, Linux, Windows, Data Structures & Algorithms, Database Internals, PostgreSQL, CitusDB, and MySQL is required. Experience with major cloud providers like AWS, Azure, or GCP, as well as extensive experience in a SaaS environment building and operating large scale distributed systems is essential. Your ability to collaborate effectively, influence decisions at an interpersonal level, and communicate clearly both in writing and verbally will be crucial. Domain knowledge in observability, experience with operating containerized services like Kubernetes or Mesos/Marathon, and a solid understanding of databases such as RDS, MySQL, and PostgreSQL are also important. Additionally, familiarity with configuration management tools like Ansible, Terraform, or Puppet, as well as technologies like ElasticSearch/OpenSearch, Apache Iceberg, Apache Spark, Spark SQL, and Cassandra will be beneficial. Experience with data platforms, data lakes, scalability, integration with multiple data sources, benchmarking, large-scale distributed database deployments, data ingestion, query performance optimization, integrations, and migrations is highly desirable. Ideally, you should hold a BS/MS/PhD in CS or an equivalent field to thrive in this challenging and rewarding role.,

As a Data Engineer MS Fabric at our Chennai-Excelencia Office location, you will leverage your 4+ years of experience to design, build, and optimize data pipelines using Microsoft Fabric, Azure Data Factory, and Synapse Analytics. Your primary responsibilities will include developing and maintaining Lakehouses, Notebooks, and data flows within the Microsoft Fabric ecosystem, ensuring efficient data integration, quality, and governance across OneLake and other Fabric components, and implementing real-time analytics pipelines for high-throughput data processing. To excel in this role, you must have proficiency in Microsoft Fabric, Azure Data Factory (ADF), Azure Synapse Analytics, Delta Lake, OneLake, Lakehouses, Python, PySpark, Spark SQL, T-SQL, and ETL/ELT Development. Your work will involve collaborating with cross-functional teams to define and deliver end-to-end data engineering solutions, participating in Agile ceremonies, and utilizing tools like JIRA for project tracking and delivery. Additionally, you will be tasked with performing complex data transformations using various data formats and handling large-scale data warehousing and analytics workloads. Preferred skills for this position include a strong understanding of distributed computing and cloud-native data architecture, experience with DataOps practices and data quality frameworks, familiarity with CI/CD for data pipelines, and proficiency in monitoring tools and job scheduling frameworks to ensure the reliability and performance of data systems. Strong problem-solving and analytical thinking, excellent communication and collaboration skills, as well as a self-motivated and proactive approach with a continuous learning mindset are essential soft skills required for success in this role.,

You should be proficient in Apache Spark and PySpark, with a strong understanding of Spark SQL, DataFrames, and RDD optimization techniques. Your programming skills in Python should be solid, and familiarity with languages like Scala is a plus. Experience with cloud platforms, particularly AWS (e.g., EMR, S3, Lambda), is essential. Additionally, having an understanding of DocumentDB, Aurora postgre, and distributed computing environments will be beneficial. Your key skills for this role should include expertise in Spark, Scala, PySpark, Spark SQL, Python, and AWS.,

NTT DATA is looking for a Sr. Data Engineer to join their team in Bangalore, Karnataka, India. As a Sr. Data Engineer, your primary responsibility will be to build and implement PySpark-based data pipelines in Azure Synapse to transform and load data into ADLS in Delta format. You will also design and implement dimensional (star/snowflake) and 3NF data models optimized for access using Power BI. Unit testing of data pipelines and transformations, as well as designing and building metadata-driven data pipelines using PySpark in Azure Synapse, will be part of your tasks. Analyzing and optimizing Spark SQL queries, optimizing the integration of data lake with Power BI semantic model, and collaborating with cross-functional teams to ensure data models align with business needs are also key responsibilities. Additionally, you will perform Source-to-Target Mapping (STM) from source to multiple layers in the data lake and maintain version control and CI/CD pipelines in Git and Azure DevOps. Integrating Azure Purview to enable access controls and implementing row level security will also be part of your role. The ideal candidate for this position should have at least 7 years of experience in SQL and PySpark. Hands-on experience with Azure Synapse, ADLS, Delta format, and metadata-driven data pipelines is required. Experience in implementing dimensional (star/snowflake) and 3NF data models, as well as expertise in PySpark and Spark SQL, including query optimization and performance tuning, are essential. Strong problem-solving and analytical skills for debugging and optimizing data pipelines in Azure Synapse, familiarity with CI/CD practices in Git and Azure DevOps, and working experience in an Azure DevOps-based development environment are also necessary. NTT DATA is a trusted global innovator of business and technology services, serving 75% of the Fortune Global 100. They are committed to helping clients innovate, optimize, and transform for long-term success. With diverse experts in more than 50 countries and a robust partner ecosystem, NTT DATA offers business and technology consulting, data and artificial intelligence solutions, industry solutions, and the development, implementation, and management of applications, infrastructure, and connectivity. NTT DATA is a leading provider of digital and AI infrastructure and is part of the NTT Group, investing over $3.6 billion each year in R&D to support organizations and society in confidently moving into the digital future.,

Title: Ops Data Engineer Location: Bangalore Key Skills Required: Recent hands-on experience with Flink, Spark SQL, and Kafka Mandatory: Strong expertise in real-time streaming data (candidates with only batch data experience will not be suitable) We are seeking a skilled Ops Data Engineer to maintain robust data infrastructure and pipelines that support our operational analytics and business intelligence needs. Candidates will bridge the gap between data engineering and operations, ensuring reliable, scalable, and efficient data systems that enable data-driven decision making across the organization. Key Responsibilities Maintain ETL/ELT pipelines using modern data engineering tools and frameworks 7 * 24 On-call support data pipeline health, performance, and SLA compliance Document data processes, schemas, and best practices SOP Implement data quality checks, monitoring, and alerting systems to ensure data reliability Optimize data pipeline performance and troubleshoot production issues Education and Experience Bachelors degree in computer science, Engineering, Mathematics, or related field 3+ years of experience in data engineering, software engineering, or related role Proven experience building and maintaining production data pipelines Required Qualifications Strong proficiency in Spark SQL, hands-on experience with realtime Kafka, Flink Databases: Strong knowledge of relational databases (Oracle, MySQL) and NoSQL systems Proficiency with Version Control Git, CI/CD practices and collaborative development workflows Strong operations management and stakeholder communication skills Flexibility to work cross time zone Have cross-cultural communication mindset Experience working in cross-functional teams Continuous learning mindset and adaptability to new technologies

Role & responsibilities We are seeking a skilled Data Engineer to maintain robust data infrastructure and pipelines that support our operational analytics and business intelligence needs. Candidates will bridge the gap between data engineering and operations, ensuring reliable, scalable, and efficient data systems that enable data-driven decision making across the organization. Strong proficiency in Spark SQL, hands-on experience with realtime Kafka, Flink Databases: Strong knowledge of relational databases (Oracle, MySQL) and NoSQL systems Proficiency with Version Control Git, CI/CD practices and collaborative development workflow Strong operations management and stakeholder communication skills Flexibility to work cross time zone Have cross-cultural communication mindset Experience working in cross-functional teams Continuous learning mindset and adaptability to new technologies Preferred candidate profile Bachelor's degree in Computer Science, Engineering, Mathematics, or related field 3+ years of experience in data engineering, software engineering, or related role Proven experience building and maintaining production data pipelines Expertise in Hadoop ecosystem - Spark SQL, Iceberg, Hive etc. Extensive experience with Apache Kafka, Apache Flink, and other relevant streaming technologies. Orchestrating tools - Apache Airflow & UC4, Proficiency in Python, Unix or similar languages Good understanding of SQL, oracle, SQL server, Nosql or similar languages Proficiency with Version Control Git, CI/CD practices and collaborative development workflows Preferrable immeidate joiner to less than 30days np