Data Engineer

Job Description

What you will be doing: Architect and Implement Data Infrastructure: Design, build, and maintain robust and scalable data pipelines and a data warehouse/lake solution using open-source and cloud-based technologies, optimized for both high-frequency small file and large file data ingestion, and real-time data streams. This includes implementing efficient mechanisms for handling high volumes of data arriving at frequent intervals. Develop and Optimize Data Processes: Create custom tools, primarily using Python, for data validation, processing, analysis, and automation. Continuously improve ETL/ELT processes for efficiency, reliability, and scalability. This includes building processes to bridge gaps between different databases and data sources, ensuring data consistency and accessibility. This also includes processing and integrating data from streaming sources. Lead and Mentor: Collaborate with product, engineering, and business teams to understand data requirements and provide data-driven solutions. Mentor and guide junior data engineers (as the team grows) and foster a culture of data excellence. Data Quality and Governance: Proactively identify and address data quality issues. Implement and maintain robust data quality monitoring, alerting, and measurement systems to ensure the accuracy, completeness, and consistency of our data assets. Implement and enforce data governance and security best practices, taking proactive ownership. Research: Research and adapt newer technologies to suit the requirements. You will thrive in this role if you: Are a Hands-On Technical Leader: You possess deep technical expertise in data engineering and are comfortable leading by example, diving into code, and setting technical direction. Are a Startup-Minded Problem Solver: You thrive in a fast-paced, dynamic environment, are comfortable with ambiguity, and are eager to build from the ground up. You proactively identify and address challenges. Are a Collaborative Communicator: You can effectively communicate complex technical concepts to both technical and non-technical audiences and build strong relationships with stakeholders. Are a Strategic Thinker: You can think ahead and architect long lasting systems.