Apache Kafka Admin

Role:

As part of the modernization initiative, this role is responsible for designing, deploying, and administering Apache Kafka clusters that form the backbone of our SaaS product’s event-driven architecture. The role requires a hands-on expert who ensures reliability, scalability, and security of Kafka platforms integrated with modernized applications built on .NET Core and React. As a Senior Kafka Administrator, you will be a key member of the team responsible for modernizing our legacy Microsoft technology platform to a .NET Core and React-based architecture. This role focuses on designing, managing, and optimizing Apache Kafka clusters to support real-time data streaming and processing for our SaaS applications. You will work as an individual contributor, collaborating closely with developers, Azure Architects, data engineers, and infrastructure teams to ensure seamless messaging operations, without team management responsibilities.

Responsibilities:

• Install, configure, upgrade, and manage Apache Kafka clusters in production and non-production environments.
• Partner with architects and developers to define messaging strategies, topics, partitions, and retention policies.
• Monitor, tune, and optimize Kafka for throughput, latency, and scalability.
• Implement security controls including SSL, SASL, authentication, and authorization.
• Manage Kafka Connect, Schema Registry, and Kafka Streams to support integrations.
• Develop backup, recovery, disaster recovery, and high availability strategies for Kafka.
• Troubleshoot and resolve incidents related to brokers, producers, and consumers.
• Automate cluster provisioning, scaling, and configuration via scripts and Infrastructure-as-Code.
• Integrate Kafka monitoring and alerting with observability tools.
• Provide expertise and best practices guidance to engineering teams consuming Kafka services.
• Manage, configure, and maintain Apache Kafka clusters to ensure high availability, scalability, and performance for SaaS applications.
• Optimize Kafka configurations, including topics, partitions, replication, and retention policies, to support real-time data streaming for .NET Core and React applications.
• Implement and maintain backup and recovery strategies for Kafka clusters to ensure data durability and reliability.
• Collaborate with the development team to integrate Kafka with modernized .NET Core and React applications, ensuring seamless data flow between components.
• Design and implement Kafka topics, schemas, and producers/consumers to support new application features and data pipelines.
• Assist in migrating legacy messaging or data workflows (e.g., MSMQ, SQL Server-based queues) to Apache Kafka, ensuring minimal disruption.
• Monitor Kafka cluster performance using tools like Confluent Control Center, Prometheus, Grafana, or custom scripts, proactively identifying and resolving issues.
• Troubleshoot complex Kafka-related issues, including connectivity, message delivery failures, and performance bottlenecks.
• Implement monitoring and alerting systems to ensure 24/7 reliability of Kafka clusters for SaaS customers.
• Ensure Kafka cluster security by implementing best practices for access control (e.g., SASL, ACLs), encryption (e.g., SSL/TLS), and auditing.
• Maintain compliance with industry standards (e.g., GDPR, SOC 2, or other relevant regulations) for data streaming and processing.
• Work closely with .NET Core and React developers, data engineers, and DevOps teams to optimize Kafka integration and ensure efficient data processing.
• Document Kafka configurations, processes, and optimization strategies to support team knowledge sharing.

Required Skills:

• Minimum 7-8+ years of hands-on experience as a Kafka Administrator in a production environment, preferably within a SaaS or product-based organization.
• Proven experience supporting messaging or streaming systems for applications built on Microsoft technologies (e.g., .NET Framework, MSMQ) and transitioning to modern stacks like .NET Core.
• Experience with migrating legacy messaging systems to Apache Kafka is highly desirable.
• Deep expertise in Apache Kafka administration, including cluster setup, configuration, topic management, replication, and performance tuning.
• Strong knowledge of Kafka ecosystem tools (e.g., Kafka Connect, Schema Registry, Kafka Streams, ksqlDB).
• Familiarity with .NET Core and React application architectures and their integration with Kafka for real-time data processing.
• Experience with monitoring tools (e.g., Confluent Control Center, Prometheus, Grafana) and scripting (e.g., Bash, Python) for automation.
• Knowledge of cloud-based Kafka deployments (e.g., Confluent Cloud, AWS MSK, Azure Event Hubs for Kafka) is a plus.
• Bachelor’s degree in Computer Science, Information Technology, or a related field (or equivalent experience).
• Good to Have Kafka certifications (e.g., Confluent Certified Administrator for Apache Kafka) or cloud certifications (e.g., AWS, Azure) are a plus.
• Good to Have Familiarity with DevOps practices, including CI/CD pipelines and infrastructure-as-code (e.g., Terraform, Ansible).
• Good to Have Knowledge of containerization (e.g., Docker, Kubernetes) for Kafka deployments. Exposure to front-end technologies like React for understanding application-Kafka interactions.