Apache Nifi Engineer

Apache Nifi

• 5+ years of hands-on experience with Apache NiFi, including developing, managing, and optimizing complex data flows in production environments.
• Proven experience with Cloudera NiFi (CDP Data Flow) in enterprise environments, including integration with Cloudera Manager.
• Experience migrating NiFi flows across major version upgrades with strong understanding of backward compatibility
• Strong proficiency in Groovy scripting, used for ExecuteScript and InvokeScriptedProcessor processors.
• Solid understanding of SSH and SFTP protocols, including authentication schemes (key-based, password), session negotiation, and file permissions handling in NiFi processors (e.g., ListSFTP, FetchSFTP, PutSFTP).
• Good grasp of data encryption mechanisms, key management, and secure flowfile handling using processors like EncryptContent.
• Experience integrating NiFi with MongoDB, including reading/writing documents via processors like GetMongo, PutMongo, and QueryMongo.
• Experience working with Apache Kafka, including producing and consuming from Kafka topics using NiFi (PublishKafka, ConsumeKafka), and handling schema evolution with Confluent Schema Registry.
• Strong knowledge of Red Hat Enterprise Linux (RHEL) environments, including systemd services, filesystem permissions, log rotation, and resource tuning for JVM-based applications like NiFi.

NiFi-Specific Technical Requirements:

• In-depth knowledge of NiFi flow design principles, including proper use of queues, back pressure, prioritizers, and connection tuning.
• Mastery of controller services, including SSLContextService, DBCPConnectionPool, and RecordReader/RecordWriter services.
• Experience with Record-based processing using Avro, JSON, CSV schemas and Record processors like ConvertRecord, QueryRecord, and LookupRecord.
• Ability to debug and optimize NiFi flows using Data Provenance, bulletins, and log analysis.
• Familiarity with custom processor development in Java/Groovy (optional but preferred).
• Experience setting up secure NiFi clusters, configuring user authentication (LDAP, OIDC), TLS certificates, and access policies.
• Proficiency in parameter contexts, variable registry, and flow versioning using NiFi Registry.
• Understanding of Zero-Master clustering model, node coordination, and site-to-site protocol.
• Experience deploying and monitoring NiFi in high-availability, production-grade environments, including using Prometheus/Grafana or Cloudera Manager for metrics and alerting.

Preferred Qualifications:

• Experience working in regulated or secure environments, with strict data handling and audit requirements.
• Familiarity with DevOps workflows, including version-controlled flow templates (JSON/XML), CI/CD integration for NiFi Registry, and automated deployment strategies.
• Strong written and verbal communication skills, with ability to document flows and onboard other engineers.