Hello Kafka Team,
we are currently trying to update our Kafka Cluster (3 Nodes with ZK) from Version 3.2.3 to Version 3.3.1. After starting the broker with the new binaries we get the following log output and the 3 brokers are not able to connect to each other: DEBUG [BrokerToControllerChannelManager broker=1 name=forwarding]: Controller isn't cached, looking for local metadata changes (kafka.server.BrokerToControllerRequestThread) DEBUG [BrokerToControllerChannelManager broker=1 name=forwarding]: No controller defined in metadata cache, retrying after backoff (kafka.server.BrokerToControllerRequestThread) and this warning pops up: "WARN Broker configuration 'ssl.client.auth' is applied only to SSL listeners. Listener-prefixed configuration can be used to enable SSL client authentication for SASL_SSL listeners. In future releases, broker-wide option without listener prefix may be applied to SASL_SSL listeners as well. All configuration options intended for specific listeners should be listener-prefixed. (org.apache.kafka.common.network.ChannelBuilders)" Furthermore, on the selected Controller (broker 3 in this case) we get the following Java Exception: java.io.IOException: Connection to 1 was disconnected before the response was read On the failing broker (broker 1), we can see, that the authentication is successful but that the session max lifetime is 0ms and because of that, directly disconnects after completion: DEBUG Successfully authenticate User=kafkaadm (org.apache.kafka.common.security.oauthbearer.internals.OAuthBearerSaslServer) DEBUG Authentication complete; session max lifetime from broker config=0 ms, credential expiration=Tue Jan 20 12:41:53 CET 1970 (-1682029722224 ms); session expiration = Wed May 10 12:10:35 CEST 2023 (0 ms), sending 0 ms to client (org.apache.kafka.common.security.authenticator.SaslServerAuthenticator) DEBUG [SocketServer listenerType=ZK_BROKER, nodeId=1] Successfully authenticated with /10.49.38.230 (org.apache.kafka.common.network.Selector) DEBUG Disconnecting expired channel: mailto:org.apache.kafka.common.network.KafkaChannel@99aa9bfb id=10.49.37.104:9092-10.49.38.230:46282-0 : RequestHeader(apiKey=UPDATE_METADATA, apiVersion=7, clientId=3, correlationId=2) (kafka.network.Processor) We tried setting the session timeouts on all brokers in the server.properties and zookeeper.properties files but with no effect on the errors. We tried changing the SSL authentication with no effect. We tried changing the Certificates with no effect. We tried upgrading all brokers first, but then all of them had the same errors. We hope someone can help us because we have exhausted our Google skills. Kind regards, Lennart
