DE&A - Core - Cloud Data Engineering - Snowflake Data Engineering

The ideal candidate will have expertise in data modeling, analytical thinking, and developing ETL processes using Python. In this role, you will be pivotal in transforming raw data from landing tables into reliable, curated master tables, ensuring accuracy, accessibility, and integrity within our Snowflake data platform.

• Design, Develop, and Maintain ETL Processes:
• Build and maintain scalable ETL pipelines in Python to extract, transform, and load data into Snowflake master tables. Automate data mastering, manage incremental updates, and ensure consistency between landing and master tables.
• Data Modeling:
• Create and optimize logical and physical data models in Snowflake for efficient querying and reporting. Translate business needs into well-structured data models, defining tables, keys, relationships, and constraints.
• Analytical Thinking and Problem Solving:
• Analyze complex datasets, identify trends, and work with analysts and stakeholders to resolve data challenges. Investigate data quality issues and design robust solutions aligned with business goals.
• Data Quality and Governance:
• Implement routines for data validation, cleansing, and error handling to ensure accuracy and reliability in Snowflake. Support the creation and application of data governance standards.
• Automation and Optimization:
• Seek automation opportunities for data engineering tasks, enhance ETL processes for performance, and scale systems as data volumes grow within Snowflake.
• Documentation and Communication:
• Maintain thorough documentation of data flows, models, transformation logic, and pipeline configurations. Clearly communicate technical concepts to all stakeholders.
• Collaboration:
• Work closely with data scientists, analysts, and engineers to deliver integrated data solutions, contributing to cross-functional projects with your data engineering expertise.

• Bachelor’s or Master’s degree in Computer Science, IT, Engineering, Mathematics, or related field
• At least 2 years of experience as a Data Engineer or similar role
• Strong Python skills, including experience developing ETL pipelines and automation scripts
• Solid understanding of relational and dimensional data modeling
• Experience with Snowflake for SQL, schema design, and managing pipelines
• Proficient in SQL for querying and data analysis in Snowflake
• Strong analytical and problem-solving skills
• Familiarity with data warehousing and best practices
• Knowledge of data quality, cleansing, and validation techniques
• Experience with version control systems like Git and collaborative workflows
• Excellent communication, both verbal and written

• In-depth knowledge of Snowflake features like Snowpipe, Streams, Tasks, and Time Travel
• Experience with cloud platforms such as AWS, Azure, or Google Cloud
• Familiarity with workflow orchestration tools like Apache Airflow or Luigi
• Understanding of big data tools like Spark, Hadoop, or distributed databases
• Experience with CI/CD pipelines in data engineering
• Background in streaming data and real-time processing
• Experience deploying data pipelines in production

• Develop automated ETL pipelines in Python to ingest daily CSVs into a Snowflake landing table, validate data, and merge clean records into a master table, handling duplicates and change tracking.
• Design scalable data models in Snowflake to support business intelligence reporting, ensuring both integrity and query performance.
• Collaborate with business analysts to adapt data models and pipelines to evolving needs.
• Monitor pipeline performance and troubleshoot inconsistencies, documenting causes and solutions.

• Technical Skills: Python (including pandas, SQLAlchemy); Snowflake SQL and management; schema design; ETL process development
• Analytical Thinking: Ability to translate business requirements into technical solutions; strong troubleshooting skills
• Collaboration and Communication: Effective team player; clear technical documentation
• Adaptability: Willingness to adopt new technologies and proactively improve processes