I am setting up a 3-node Apache Kafka cluster using KRaft mode (3.9.0) with SCRAM-SHA-256 for inter-broker communication on the SASL_PLAINTEXT listener. The cluster is unable to form a quorum due to persistent SASL authentication failures.
All configuration files and passwords have been synchronized. The exact error points to a handshake failure, but the settings appear correct. *Environment and Node Details* *Kafka Version:* 3.9.0 (KRaft mode) *Security Protocol:* SASL_PLAINTEXT (SCRAM-SHA-256) *JAAS File Location:* /opt/kafka/config/kafka_server_jaas.conf *Nodes:* Node 1: 172.20.24.155 (node.id=1) Node 2: 172.20.24.156 (node.id=2) Node 3: 172.20.24.157 (node.id=3) *Errors* The primary error is a SASL handshake failure, leading the client (connecting broker) to prematurely send an API request. This confirms that the broker is attempting to authenticate but is immediately rejected by the remote server. NODE 1 Logs (Example of persistent failures after configuration changes) Nov 13 13:00:10 NDCPRDKAFKA01 kafka-server-start.sh[3995292]: [2025-11-13 13:00:10,518] INFO [SocketServer listenerType=BROKER, nodeId=1] Failed authentication with /172.20.24.156 (channelId=172.20.24.155:9092-172.20.24.156:51726-3) (Unexpected Kafka request of type METADATA during SASL handshake.) (org.apache.kafka.common.network.Selector) NODE 2 Logs (Similar error, often reporting disconnection issues for itself and other nodes) Nov 13 13:00:19 NDCPRDKAFKA02 kafka-server-start.sh[2649709]: [2025-11-13 13:00:19,258] INFO [SocketServer listenerType=BROKER, nodeId=2] Failed authentication with /172.20.24.157 (channelId=172.20.24.156:9092-172.20.24.157:54096-35) (Unexpected Kafka request of type DESCRIBE_CLUSTER during SASL handshake.) (org.apache.kafka.common.network.Selector) NOTE: Previous error (now fixed) was: java.lang.IllegalArgumentException: Could not find a 'KafkaServer'... entry in the JAAS configuration. System property 'java.security.auth.login.config' is not set *Configuration Files (Current State)* 1. server.properties (All Nodes) The core security settings are configured identically on all nodes, differing only by node.id and local IP address in the listeners block. process.roles=broker,controllernode.id=<1, 2, or 3> [email protected]:9093,[email protected]:9093,[email protected]:9093 # Network Configuration listeners=SASL_PLAINTEXT://172.20.24.<155/156/157>:9092,CONTROLLER://172.20.24.<155/156/157>:9093 advertised.listeners=SASL_PLAINTEXT://172.20.24.<155/156/157>:9092 controller.listener.names=CONTROLLER listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL # SASL Security Configurationinter.broker.listener.name=SASL_PLAINTEXT sasl.enabled.mechanisms=SCRAM-SHA-256 sasl.mechanism.inter.broker.protocol=SCRAM-SHA-256 sasl.mechanism.controller.protocol=SCRAM-SHA-256 *2. kafka_server_jaas.conf (All Nodes)* The JAAS file is an external file, and the credentials are set as follows (based on the user/password history provided): Node,User,Password Node 1,broker1,secret1 Node 2,broker2,secret2 Node 3,broker3,secret3 *Example Content (Node 1):* KafkaServer { org.apache.kafka.common.security.scram.ScramLoginModule required username="broker1" password="secret1"; }; KafkaClient { org.apache.kafka.common.security.scram.ScramLoginModule required username="broker1" password="secret1"; }; *3. kafka.service (Systemd Loader - All Nodes)* The environment variable loading the external JAAS file is now correctly configured, resolving the previous IllegalArgumentException. [Service] User=kafka Group=kafka # ... other properties Environment="KAFKA_OPTS=-Djava.security.manager=allow -Djava.security.auth.login.config=/opt/kafka/config/kafka_server_jaas.conf" ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/kraft/server.properties # ... *4. Users Created in KRaft Metadata* The broker principals were created and synchronized using the following command pattern (running against the working listener): # Example for broker1 /opt/kafka/bin/kafka-configs.sh --bootstrap-server 172.20.24.155:9092 --alter --add-config 'SCRAM-SHA-256=[password=secret1]' --entity-type users --entity-name broker1 # Similar commands were run for broker2 and broker3 with their respective passwords. This confirms the hashed password in the KRaft metadata should match the plaintext password in the JAAS file. Given that the following checks are complete: 1. SASL is enabled in server.properties. 2. JAAS is loaded via systemd (KAFKA_OPTS). 3. Plaintext passwords in the JAAS file are identical to the passwords used to create the SCRAM users in the KRaft metadata. Why is the inter-broker SASL handshake still failing, resulting in the "Unexpected Kafka request of type DESCRIBE_CLUSTER during SASL handshake" error? Is there a configuration detail I am missing, such as a requirement for the CONTROLLER listener security, or an issue related to the SCRAM negotiation when the cluster is initially forming its quorum?
