Data Engineer

This role is for one of the Weekday's clients

Requirements

Key Responsibilities

• Data Pipeline Development and Management
• Design, build, test, and maintain efficient data pipelines and data management systems.
• Develop and manage ETL (Extract, Transform, Load) and ELT (Extract, Load, Transform) processes to integrate data from diverse sources such as databases, APIs, and real-time streams.
• Data Modeling and Architecture
• Design data models and implement schemas for data warehouses and data lakes to support analytics and business operations.
• Optimize data storage, access, and performance for scalability and maintainability.
• Data Quality and Integrity
• Implement validation, cleansing, and monitoring to maintain data accuracy, consistency, and reliability.
• Define and enforce best practices and standards for data governance and quality.
• Infrastructure Management
• Manage and monitor key data infrastructure components including databases, data lakes, and distributed computing environments.
• Apply data security protocols and ensure proper access controls are in place.
• Automation and Optimization
• Automate data workflows and pipelines to improve reliability and performance.
• Continuously monitor and fine-tune systems for operational efficiency.
• Collaboration and Support
• Partner with data scientists, analysts, software engineers, and business stakeholders to gather requirements and provide scalable data solutions.
• Document processes, workflows, and system designs; support cross-functional teams with technical guidance.
• Technology Evaluation
• Stay current with emerging tools and technologies in the data engineering space.
• Evaluate and recommend new solutions to enhance data capabilities and performance.
  

Education And Experience

• Bachelor's or Master's degree in Computer Science, Engineering, Information Systems, Data Science, or a related field.
• 5 to 7 years of experience in data engineering, software development, or a similar domain.
  

Technical & Functional Competencies

Required Skills & Qualifications

Technical Proficiency

• Programming: Strong experience in Python and SQL.
• Databases: Proficient in relational (PostgreSQL, MySQL) and NoSQL (MongoDB, Cassandra) databases.
• Data Warehousing & Lakes: Hands-on experience with platforms like Snowflake, Redshift, BigQuery.
• ETL/ELT Tools: Proficiency with tools like Apache Airflow, AWS Glue, Azure Data Factory, Talend.
• Big Data: Working knowledge of Apache Spark or similar big data technologies.
• Cloud Platforms: Experience with AWS, Azure, or GCP for data engineering workflows.
• Data Modeling: Strong understanding of modeling techniques and best practices.
• API Integration: Ability to build and consume APIs for data integration.
• Version Control: Experience with Git or other version control systems.
  

Soft Skills

• Analytical mindset with a strong problem-solving approach.
• Excellent communication skills for both technical and non-technical audiences.
• Team player with a collaborative work ethic.
• Detail-oriented with a commitment to data quality.
• Adaptability to new technologies and changing project requirements.
  

Key Skills: