As a Data Engineer, you will be part of a dynamic team responsible for developing and maintaining data pipelines, databases, and analytical systems @Kotak811

You will work closely with other data engineers, data scientists, and business stakeholders to ensure data integrity, availability, and reliability.

Responsibilities

Data Ingestion and Integration
- Collaborate with cross-functional teams to gather data requirements and identify data sources.
- Design, develop, and maintain data ingestion pipelines from various structured and unstructured sources.
- Implement data integration processes to ensure data quality and consistency.
- Monitor and troubleshoot data ingestion issues and optimise performance.

Data Transformation and ETL:
- Translate business requirements into technical specifications (data models)
- Extract, transform, and load data from diverse sources into appropriate data models and structures.
  - Develop and maintain ETL workflows and processes using tools like Apache Spark, Apache Kafka, or other data integration frameworks.
- Cleanse, validate, and transform data to meet business and analytical requirements.
- Strong engineering mindset - build automated monitoring, alerting, self healing (restartability/graceful failures) features while building the consumption pipelines

Database Management:
- Create and maintain scalable and efficient data storage systems, including relational databases, data warehouses, or NoSQL databases.
- Optimise database performance, including indexing, partitioning, and query optimization.
- Implement data security and access controls to ensure data privacy and compliance with industry regulations.
- Monitor database health, performance, and capacity, and take proactive measures to ensure optimal system operation.

Data Governance and Documentation:
- Ensure adherence to data management policies and standards.
- Document data pipelines, ETL processes, and data flows to facilitate knowledge sharing and maintain data lineage.
- Contribute to the development and maintenance of data dictionaries, data catalogues, and metadata repositories.
- Participate in data quality initiatives and data governance reviews to maintain high data integrity.

Collaboration and Communication:
- Collaborate effectively with cross-functional teams, including data scientists, business analysts, and software engineers.
- Communicate technical concepts and solutions to both technical and non-technical stakeholders.
- Participate in team meetings, code reviews, and knowledge-sharing sessions to foster a collaborative work environment.
Stay up-to-date with emerging technologies, industry trends, and best practices in data engineering.

Qualifications

2-3 years’ experience with a Bachelor's Degree in Computer Science, Engineering, Technology or related field required.

Proficiency in one or more programming languages such as Python, Java, or Scala.

Good knowledge of Agile, SDLC/CICD practices and tools

Experience with data integration tools and frameworks such as Apache Spark, Apache Kafka, or similar.

Experience is leveraging Cloud / OSS Data platform to build scalable data models and data pipelines for analytical, data science, systemic consumption use cases

Must have good knowledge of performance tuning/optimising data processing jobs, debugging time consuming jobs.

Good understanding of distributed systems

Experience working extensively in multi-petabyte DW environment

Strong problem-solving skills and attention to detail.

More Jobs at Kotak Mahindra

Relationship Manager-PRIME VISHWAAS-Sales

Ujjain

Experience: Not specified

Salary: Not disclosed

Relationship Manager-Loan Against Property (LAP)-Sales

Indore

3.0 - 3.0 yrs

Salary: Not disclosed

Premier Acquisition Manager-RL SALES-Sales

Rajasthan

1.0 - 2.0 yrs

Salary: Not disclosed

Sr. Sales Executive-COMMERCIAL VEHICLE-Marketing Branches Operations

Sīkar

2.0 - 3.0 yrs

Salary: Not disclosed

Trade Relationship Manager-RL SALES-Retail Sales

Calcutta

1.0 - 3.0 yrs

Salary: Not disclosed

Mock Interview

Practice Video Interview with JobPe AI

Start Python Interview

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.