Big Data Lead

Designation: Big Data Lead/Architect

Location: Pune

Experience: 8-10 years

NP - immediate joiner/15-30 days notice

Reports To – Product Engineering Head

Job Overview

We are looking to hire a talented big data engineer to develop and manage our company’s Big Data solutions. In this role, you will be required to design and implement Big Data tools and frameworks, implement ELT processes, collaborate with development teams, build cloud platforms, and maintain the production system.

To ensure success as a big data engineer, you should have in-depth knowledge of Hadoop technologies, excellent project management skills, and high-level problem-solving skills. A top-notch Big Data Engineer understands the needs of the company and institutes scalable data solutions for its current and future needs.

Responsibilities:

• Meeting with managers to determine the company’s Big Data needs.
• Developing big data solutions on AWS, using Apache Spark, Databricks, Delta Tables, EMR, Athena, Glue, Hadoop, etc.
• Loading disparate data sets and conducting pre-processing services using Athena, Glue, Spark, etc.
• Collaborating with the software research and development teams.
• Building cloud platforms for the development of company applications.
• Maintaining production systems.

Requirements:

• 8-10 years of experience as a big data engineer.
• Must be proficient with Python & PySpark.
• In-depth knowledge of Hadoop, Apache Spark, Databricks, Delta Tables, AWS data analytics services.
• Must have extensive experience with Delta Tables, JSON, Parquet file format.
• Good to have experience with AWS data analytics services like Athena, Glue, Redshift, EMR.
• Familiarity with Data warehousing will be a plus.
• Must have Knowledge of NoSQL and RDBMS databases.
• Good communication skills.
• Ability to solve complex data processing, transformation related problems