Data Engineer II

• Data Pipeline Development: Design, build, and maintain scalable and efficient data pipelines to collect, process, and store structured and unstructured data from various sources.
• Data Integration: Integrate data from multiple sources such as databases, APIs, flat files, and streaming platforms into centralized data repositories.
• Data Modeling: Develop and optimize data models and schemas to support analytical and operational requirements. Implement data transformation and aggregation processes as needed.
• Data Quality Assurance: Implement data validation and quality assurance processes to ensure the accuracy, completeness, and consistency of data throughout its lifecycle.
• Performance Optimization: Monitor and optimize data processing and storage systems for performance, reliability, and cost-effectiveness. Identify and resolve bottlenecks and inefficiencies in data pipelines and leverage Automation and AI to improve overall Operations.
• Infrastructure Management: Manage and configure cloud-based or on-premises infrastructure components such as databases, data warehouses, compute clusters, and data processing frameworks.
• Collaboration: Collaborate with cross-functional teams including data scientists, analysts, software engineers, and business stakeholders to understand data requirements and deliver solutions that meet business objectives .
• Documentation and Best Practices: Document data pipelines, systems architecture, and best practices for data engineering. Share knowledge and provide guidance to colleagues on data engineering principles and techniques.
• Continuous Improvement: Stay updated with the latest technologies, tools, and trends in data engineering and recommend improvements to existing processes and systems.

Qualifications and Skills:

• Bachelor's degree or higher in Computer Science, Engineering, or a related field.
• Proven experience in data engineering or related roles, with a strong understanding of data processing concepts and technologies.
• Mastery of programming languages such as Python, Java, or Scala.
• Knowledge of database systems such as SQL, NoSQL, and data warehousing solutions.
• Knowledge of stream processing technologies such as Kafka or Apache Beam.
• Experience with distributed computing frameworks such as Apache Spark, Hadoop, or Apache Flink .
• Experience deploying pipelines in cloud platforms such as AWS, Azure, or Google Cloud Platform.
• Experience in implementing enterprise systems in production setting for AI, natural language processing. Exposure to self-supervised learning, transfer learning, and reinforcement learning is a plus .
• Have full stack experience to build the best fit solutions leveraging Large Language Models (LLMs) and Generative AI solutions with focus on privacy, security, fairness.
• Have good engineering skills to design the output from the AI with nodes and nested nodes in JSON or array, HTML formats for as-is consumption and display on the dashboards/portals.
• Strong problem-solving skills and attention to detail.
• Excellent communication and teamwork abilities.
• Experience with containerization and orchestration tools such as Docker and Kubernetes.
• Familiarity with data visualization tools such as Tableau or Power BI.