Data Engineer

Python, Pyspark, Databricks

5 days work from office (ODC)

Data Engineer

Software Engineer (Data)

What are we looking for?

• Role models

  . Leading by example and setting direction on engineering practices and habits are core to our mission.

• Problem solvers

  . We pursue some of the most challenging cross-cutting problems/issues and treat them as opportunities to align and shape our culture.

• Change agents

  . We acknowledge that we might not always understand a particular domain; however, we will ask probing questions and many times will leverage data to drive decision making or challenge the status quo.

• Agile in mindset

  . We provide incremental value and seek customer feedback to drive future work.

• Vocal

  . We encourage open and honest discussions. You should feel comfortable sharing your opinion as well as having your ideas challenged.

Responsibilities:

• Play key role in establishing and implementing migration patterns for the Data Lake Modernization project.
• Actively migrate use cases from our on premises Data Lake to Databricks on GCP.
• Collaborate with Product Management and business partners to understand use case requirements and reporting.
• Adhere to internal development best practices/lifecycle (e.g. Testing, Code Reviews, CI/CD, Documentation) .
• Document and showcase feature designs/workflows.
• Participate in team meetings and discussions around product development.
• Stay up to date on industry latest industry trends and design patterns.

Basic Qualifications:

• Bachelor's in Computer Science, Computer Engineering or related field
• 4+ yrs. Development experience with Spark (PySpark), Python and SQL.
• Extensive knowledge building data pipelines
• Hands on experience with Databricks Devlopment
• Strong experience with
• Strong experience developing on Linux OS.
• Experience with scheduling and orchestration (e.g. Databricks Workflows,airflow, prefect, control-m).
• Solid understanding of distributed systems, data structures, design principles.
• Comfortable communicating with teams via showcases/demos.
• Agile Development Methodologies (e.g. SAFe, Kanban, Scrum).

Nice to Have:

• 3+ years experience with GIT.
• 3+ years experience with CI/CD (e.g. Azure Pipelines).
• Experience with streaming technologies, such as Kafka, Spark.
• Experience building applications on Docker and Kubernetes.
• Cloud experience (e.g. Azure, Google).