Databricks Architect

Role Overview: As a Databricks Architect at NTT DATA, you will lead the design and implementation of scalable, secure, and high-performance data solutions using the Databricks Lakehouse Platform. Your focus will be on leveraging Databricks unified analytics capabilities to build enterprise-grade data platforms that support advanced analytics, machine learning, and real-time data processing. Key Responsibilities: - Define and own the end-to-end architecture of data platforms built on Databricks, including ingestion, transformation, storage, and consumption layers. - Design and implement Lakehouse architectures using Delta Lake, Unity Catalog, and structured streaming. - Architect scalable ETL/ELT pipelines using Apache Spark, PySpark, and Databricks Workflows. - Lead the integration of Databricks with enterprise systems such as data catalogs, data quality frameworks, ML platforms, and BI tools. - Guide teams in implementing CI/CD pipelines, version control, and automated testing for Databricks notebooks and jobs. - Collaborate with data scientists, engineers, and business stakeholders to support ML model lifecycle management using MLflow. - Provide technical leadership in performance tuning, cost optimization, and cluster configuration. - Stay current with Databricks innovations and advocate for adoption of new features and capabilities. Qualifications Required: - Bachelors or Masters degree in Computer Science, Software Engineering, Information Technology, or related field required. - 10+ years of experience in data architecture and engineering, with 5+ years in Databricks and Apache Spark. - Deep expertise in Delta Lake, structured streaming, PySpark, and SQL. - Strong understanding of Lakehouse architecture, data mesh, and modern data stack principles. - Experience with Unity Catalog, Databricks Repos, Jobs API, and Workflows. - Familiarity with cloud platforms (Azure, AWS, GCP) and their integration with Databricks. - Experience with data modeling, dimensional modeling, and temporal data structures. - Experience with CI/CD, DevOps, and infrastructure-as-code tools (Terraform, GitHub Actions, Azure DevOps). - Knowledge of machine learning lifecycle, MLflow, and model deployment strategies. - An understanding of E-R data models (conceptual, logical, and physical). - Strong analytical skills, including a thorough understanding of how to interpret customer business requirements and translate them into technical designs and solutions. - Strong communication skills both verbal and written. Capable of collaborating effectively across a variety of IT and Business groups, across regions, roles and able to interact effectively with all levels. - Strong problem-solving skills. Ability to identify where focus is needed and bring clarity to business objectives, requirements, and priorities.,