Data Brick Data Architect

Job Description

Data Architecture Design and Implementation: Design and implement cloud-based data architectures leveraging Databricks, AWS, Azure, or GCP. Architect end-to-end data pipelines that integrate with Databricks and other cloud services. Ensure that the data architecture supports both batch and real-time data processing and analytics. Leverage Databricks Delta Lake for data storage, management, and processing. Data Integration: Integrate data from various sources, including operational databases, data lakes, and third-party APIs. Work with teams to ensure seamless ETL/ELT processes within the Databricks environment Collaborate with data engineers to build data ingestion pipelines and automate data workflows. Optimization and Performance Tuning: Optimize Databricks Spark workloads for performance, cost, and scalability Monitor and tune Databricks clusters to ensure optimal performance for analytics and data science workflows Identify and resolve performance bottlenecks in data pipelines, storage, and queries. Collaboration with Data Teams: Implement data governance practices to ensure the integrity, security, and compliance of data within Databricks Ensure that the architecture adheres to industry standards for data security, privacy, and compliance (e.g., GDPR, CCPA). Leverage Databricks' built-in features for role-based access control, data encryption, and auditing Security and Governance: Implement data governance practices to ensure the integrity, security, and compliance of data within Databricks Ensure that the architecture adheres to industry standards for data security, privacy, and compliance (e.g., GDPR, CCPA) Leverage Databricks' built-in features for role-based access control, data encryption, and auditing. Cloud Infrastructure and Automation: Design and manage cloud infrastructure for Databricks, ensuring high availability and fault tolerance. Automate infrastructure provisioning, deployment, and monitoring processes using tools like Terraform or CloudFormation. Continuously evaluate and recommend cloud solutions to improve efficiency and scalability Continuous Improvement and Innovation: Stay up to date with the latest Databricks features, best practices, and industry trends. Propose innovative solutions to enhance data architecture and workflows based on new technologies and methodologies. Mentor junior data engineers and developers on Databricks best practices. Skills and Qualifications: Educational Background: Bachelors or Master’s degree in Data Science, Computer Science, Mathematics, Statistics, or a related field. Certifications: Databricks Certified Professional Data Engineer AWS Certified Solutions Architect or similar certification Google Cloud Professional Data Engineer or Azure Data Engineer certifications are a plus. Soft Skills: Strong written and verbal communication skills. Ability to explain complex technical concepts to non-technical audiences. Detail-oriented with a strong focus on quality and accuracy. Capable to collaborate with cross-functional teams, including customers, partners and business stakeholders. Experience: Experience with machine learning frameworks and pipelines, particularly in Databricks. Familiarity with containerized environments (e.g., Docker, Kubernetes) for deploying data workflows. Should be able to do Model development, deployment and AI/ GEN AI use case implementation using Databricks platform & its services Work Environment: Collaborative and fast-paced work environment. Opportunity to work with state-of-the-art technologies. Supportive and dynamic team culture Preferred candidate profile