11 Hadoop Cluster Jobs

Hadoop admin support experience is a key criterion. Every resource will be expected to have at least 2 years of support experience. Visas Hadoop clusters span 1000s of nodes per cluster. Experience of having at least 500+ node cluster for at least 3 5 years Hadoop Admin support experience will be expected for the senior resources. Hadoop administration Automation (Ansible, shell scripting or python scripting) DEVOPS skills (Should be able to code at least in one language preferably python Program/Project Overview Role is part of PRE-Big Data team responsible for managing Hadoop platforms. Resource will work during IND hours, and it is hybrid role. Candidates will focus on improving performance, reliability and improving the efficiency of Big Data platforms. Engagement Deliverable(s) The role involves performing Big Data Administration and Engineering activities on multiple open-source platforms such as Hadoop, Kafka, HBase, and Spark. The successful candidate will possess strong troubleshooting and debugging skills. Other responsibilities include effective root cause analysis of major production incidents and the development of learning documentation. The person will identify and implement high-availability solutions for services with a single point of failure. The role involves planning and performing capacity expansions and upgrades in a timely manner to avoid any scaling issues and bugs. This includes automating repetitive tasks to reduce manual effort and prevent human errors. The successful candidate will tune alerting and set up observability to proactively identify issues and performance problems. They will also work closely with Level 3 teams in reviewing new use cases and cluster hardening techniques to build robust and reliable platforms. The role involves creating standard operating procedure documents and guidelines on effectively managing and utilizing the platforms. The person will leverage DevOps tools, disciplines (Incident, problem, and change management), and standards in day-to-day operations. The individual will ensure that the Hadoop platform can effectively meet performance and service level agreement requirements. They will also perform security remediation, automation, and self-healing as per the requirement. The individual will concentrate on developing automations and reports to minimize manual effort. This can be achieved through various automation tools such as Shell scripting, Ansible, or Python scripting, or by using any other programming language

At EY, you'll have the chance to build a career as unique as you are, with the global scale, support, inclusive culture, and technology to become the best version of you. And we're counting on your unique voice and perspective to help EY become even better, too. Join us and build an exceptional experience for yourself, and a better working world for all. As a Staff - Data Engineer at EY, your responsibilities include designing and developing software components using various tools such as PySpark, Sqoop, Flume, Azure Databricks, and more. You will perform detailed analysis and effectively interact with onshore/offshore team members, ensuring all deliverables conform to the highest quality standards and are executed in a timely manner. This role is deadline-oriented and may require working under the US time schedule. Additionally, you will identify areas of improvement, conduct performance tests, consult with the design team, ensure high performance of applications, and work well with development/product engineering teams. To be successful in this role, you should have 2-4 years of experience in BCM or WAM industry, preferably with exposure to US-based asset management or fund administration firms. You should have a strong understanding of data in BCM/WAM space, including knowledge of KDEs such as Funds, Positions, Transactions, Trail Balance, Securities, Investors, and more. Proficiency in programming languages like Python, hands-on experience with Big Data tools such as PySpark, Sqoop, Hive, and Hadoop Cluster, as well as Cloud technologies like Azure Databricks are essential. Expertise in databases like Oracle, SQL Server, and exposure to Big Data is a plus. Knowledge of Data Visualization tools and the ability to write programs for file/data validations, EDA, and data cleansing are also desired. As an ideal candidate, you should be highly data-driven, capable of writing complex data transformation programs using PySpark and Python, and have experience in data integration and processing using Spark. Hands-on experience in creating real-time data streaming solutions using Spark Streaming and Flume, as well as handling large datasets and writing Spark jobs and hive queries for data analysis, are valuable assets. Experience working in an agile environment will be beneficial for this role. Join EY in building a better working world, where diverse teams across assurance, consulting, law, strategy, tax, and transactions help clients grow, transform, and operate. EY aims to create long-term value for clients, people, and society, while building trust in the capital markets through data and technology-enabled solutions worldwide.,

Piller Soft Technology is looking for Lead Data Engineer to join our dynamic team and embark on a rewarding career journey Designing and developing data pipelines: Lead data engineers are responsible for designing and developing data pipelines that move data from various sources to storage and processing systems. Building and maintaining data infrastructure: Lead data engineers are responsible for building and maintaining data infrastructure, such as data warehouses, data lakes, and data marts. Ensuring data quality and integrity: Lead data engineers are responsible for ensuring data quality and integrity, by setting up data validation processes and implementing data quality checks. Managing data storage and retrieval: Lead data engineers are responsible for managing data storage and retrieval, by designing and implementing data storage systems, such as NoSQL databases or Hadoop clusters. Developing and maintaining data models: Lead data engineers are responsible for developing and maintaining data models, such as data dictionaries and entity-relationship diagrams, to ensure consistency in data architecture. Managing data security and privacy: Lead data engineers are responsible for managing data security and privacy, by implementing security measures, such as access controls and encryption, to protect sensitive data. Leading and managing a team: Lead data engineers may be responsible for leading and managing a team of data engineers, providing guidance and support for their work.

Job Overview: As a Site Reliability Engineer (SRE) specializing in DataPlatform OnPremise, you will play a critical role in deployment, ensuring the reliability, scalability, and performance of our Cloudera Data Platform (CDP) infrastructure. You will collaborate closely with cross-functional teams to design, implement, and maintain robust systems that support our data-driven initiatives. The ideal candidate will have a deep understanding of Data Platform, strong troubleshooting skills, and a proactive mindset towards automation and optimization.You will play a pivotal role in ensuring the smooth functioning, operation, performance and security of large high density Cloudera-based infrastructure. Roles and Responsibilities: Work on tasks related to implementation of Cloudera Data Platform Cloudera Data Platform on-premises and be a part of planning, installation, configuration, and integration with existing systems. Infrastructure ManagementManage and maintain the Cloudera-based infrastructure, ensuring optimal performance, high availability, and scalability. This includes monitoring system health, and performing routine maintenance tasks. Strong troubleshooting skills and operational expertise in areas such as system capacity, bottlenecks, memory, CPU, OS, storage, and networking. Creating Runbooks and automating them using scripting tools like Shell scripting, Python etc. Working knowledge with any of the configuration management tools like Terraform, Ansible or SALT Data Security and ComplianceImplement and enforce security best practices to safeguard data integrity and confidentiality within the Cloudera environment. Ensure compliance with relevant regulations and standards (e.g., GDPR, HIPAA, DPR). Performance OptimizationContinuously optimize the Cloudera infrastructure to enhance performance, efficiency, and cost-effectiveness. Identify and resolve bottlenecks, tune configurations, and implement best practices for resource utilization. Capacity PlanningPlanning and performance tuning of Hadoop clusters, Monitor resource utilization trends and plan for future capacity needs. Proactively identify potential capacity constraints and propose solutions to address them. Collaborate effectively with infrastructure, network, database, application, and business intelligence teams to ensure high data quality and availability. Work closely with teams to optimize the overall performance of the PhonePe Hadoop ecosystem. Backup and Disaster RecoveryImplement robust backup and disaster recovery strategies to ensure data protection and business continuity. Test and maintain backup and recovery procedures regularly. Develop tools and services to enhance debuggability and supportability. Patches & UpgradesRoutinely apply recommended patches and perform rolling upgrades of the platform in accordance with the advisory from Cloudera, InfoSec and Compliance. Documentation and Knowledge SharingCreate comprehensive documentation for configurations, processes, and procedures related to the Cloudera Data Platform. Share knowledge and best practices with team members to foster continuous learning and improvement. Collaboration and CommunicationCollaborate effectively with cross-functional teams including data engineers, developers, and IT operations personnel. Communicate project status, issues, and resolutions clearly and promptly. Skills Required: Bachelor's degree in Computer Science, Engineering, or related field. Proficiency in Linux system administration, shell scripting, and networking concepts including IPtables, and IPsec. Strong understanding of networking, open-source technologies, and tools. 3-5 years of experience in the design, set up, and management of large-scale Hadoop clusters, ensuring high availability, fault tolerance, and performance optimization. Strong understanding of distributed computing principles and experience with Hadoop ecosystem technologies (HDFS, MapReduce, YARN, Hive, Spark, etc.). Experience with Kerberos and LDAP. Strong Knowledge of databases like Mysql,Nosql,Sql server Hands-on experience with configuration management tools (e.g., Salt,Ansible, Puppet, Chef). Strong scripting skills (e.g., PERL,Python, Bash) for automation and troubleshooting. Experience with monitoring and logging solutions (e.g., Prometheus, Grafana, ELK stack). Knowledge of networking principles and protocols (TCP/IP, UDP, DNS, DHCP, etc.). Experience with managing *nix based machines and strong working knowledge of quintessential Unix programs and tools (e.g. Ubuntu, Fedora, Redhat, etc.) Excellent communication skills and the ability to collaborate effectively with cross-functional teams. Excellent analytical, problem-solving, and troubleshooting skills.. Proven ability to work well under pressure and manage multiple priorities simultaneously. Good To Have: Cloudera Certified Administrator (CCA) or Cloudera Certified Professional (CCP) certification preferred. Minimum 2 years of experience in managing and administering medium/large hadoop based environments (>100 machines), including Cloudera Data Platform (CDP) experience is highly desirable. Familiarity with Open Data Lake components such as Ozone, Iceberg, Spark, Flink, etc. Familiarity with containerization and orchestration technologies (e.g. Docker, Kubernetes, OpenShift) is a plus Design,develop and maintain Airflow DAGs and tasks to automate BAU processes,ensuring they are robust,scalable and efficient. PhonePe Full Time Employee Benefits (Not applicable for Intern or Contract Roles) Insurance Benefits - Medical Insurance, Critical Illness Insurance, Accidental Insurance, Life Insurance Wellness Program - Employee Assistance Program, Onsite Medical Center, Emergency Support System Parental Support - Maternity Benefit, Paternity Benefit Program, Adoption Assistance Program, Day-care Support Program Mobility Benefits - Relocation benefits, Transfer Support Policy, Travel Policy Retirement Benefits - Employee PF Contribution, Flexible PF Contribution, Gratuity, NPS, Leave Encashment Other Benefits - Higher Education Assistance, Car Lease, Salary Advance Policy Working at PhonePe is a rewarding experience! Great people, a work environment that thrives on creativity, the opportunity to take on roles beyond a defined job description are just some of the reasons you should work with us. Read more about PhonePe on our blog. Life at PhonePe PhonePe in the news

We are hiring for the position "Hadoop Admin" Skill Set: Hadoop, Cloudera, big data, spark, Hive, HDFS, YARN, HIVE, KAFKA, SPARK, SQL DATABASE, RANGER Experience: 7 years Location: Noida, Sector-135 Work Mode: Work from Office Budget: 14-15 LPA

We are looking for a skilled Hadoop Administrator with 5 to 7 years of experience in Hadoop Engineering, working on Python, Ansible, and DevOps methodologies. The ideal candidate will have extensive experience in CDPHDP Cluster and Server build, including Control nodes, Worker nodes, Edge nodes, and Data copy from cluster to cluster. Roles and Responsibility Design and implement scalable and efficient data processing systems using Hadoop technologies. Develop and maintain automation scripts using Python, Ansible, and other DevOps tools. Collaborate with cross-functional teams to identify and prioritize project requirements. Troubleshoot and resolve complex technical issues related to Hadoop clusters. Ensure high-quality standards for data processing and security. Participate in code reviews and contribute to the improvement of the overall codebase. Job Strong understanding of Hadoop ecosystem, including HDFS, MapReduce, and YARN. Experience with Linux operating system and scripting languages such as Bash or Python. Proficient in Shell scripting and YAML configuration files. Good technical design, problem-solving, and debugging skills. Understanding of CI/CD concepts and familiarity with GitHub, Jenkins, and Ansible. Hands-on development solutions using industry-leading Cloud technologies. Working knowledge of Git Ops and DevSecOps. Agile proficient and knowledgeable in other agile methodologies, ideally certified. Strong communication and networking skills. Ability to work autonomously and take accountability to execute and deliver on goals. Strong commitment to high-quality standards. Good communication skills and sense of ownership to work as an individual contributor.

Experience in SQL and understanding of ETL best practices Should have good hands on in ETL/Big Data development Extensive hands on experience in Scala Should have experience in Spark/Yarn, troubleshooting Spark, Linux, Python Setting up a Hadoop cluster, Backup, recovery, and maintenance.

Strong Linux and Strong AWS experience Strong active directory Manage Hadoop clusters on Linux, Active Directory integration Collaborate with data science team on project delivery using Splunk & Spark Exp. managing BigData clusters in Production

Role & responsibilities Role - Hadoop Admin + Automation Experience 8+ yrs Grade AVP Location - Pune Mandatory Skills : Hadoop Admin, Automation (Shell scripting/ any programming language Java/Python), Cloudera / AWS/Azure/GCP Good to have : DevOps tools Primary focus will be on candidates with Hadoop admin & Automation experience,

We are hiring for the position "Hadoop Admin" Skill Set: Hadoop, Cloudera, big data, spark, Hive, HDFS, YARN, HIVE, KAFKA, SPARK, SQL DATABASE, RANGER Experience: 3 years Location: Noida, Sector-135 Work Mode: Work from Office Budget: 8 LPA

Looking for a Hadoop Administrator to manage, monitor, and optimize Hadoop clusters. Responsibilities include deployment, upgrades, performance tuning, and security. Requires 3+ years of experience with Hadoop ecosystem tools and Linux systems. Required Candidate profile Notice Period : Immediate or 30 days max