Hi Luke,
We are using Kafka 3.7.0 Broker/Client system in our prod environment with
SASL_SSL communication between Kafka Clients and Broker. We are starting the
Kafka process from the shell using the below command.
`nohup $EXEC_KAFKA_CONFIG --zookeeper 127.0.0.1:2181 --entity-type brokers
--entity-name 0 --alter --add-config $zooKeeperConfig >>
$KAFKA_HOME/logs/nohup_z.out 2>&1 &`
`nohup $KAFKA_HOME/bin/kafka-server-start.sh
$KAFKA_HOME/config/server.properties >> $KAFKA_HOME/logs/nohup_b.out 2>&1 &`
Here, we are passing the SSL Keystore and truststore password details in
$zooKeeperConfig as shown below:
zooKeeperConfig="listener.name.sasl_ssl.ssl.truststore.password=$KAFKA_SSL_KEYSTORE_AND_TRUSTSTORE_PWD,listener.name.sasl_ssl.ssl.keystore.password=$KAFKA_SSL_KEYSTORE_AND_TRUSTSTORE_PWD,listener.name.sasl_ssl.ssl.key.password=$KAFKA_SSL_KEYSTORE_AND_TRUSTSTORE_PWD,$KAFKA_SSL_PASSWORD_ENCODER_SECRET_PROP"
Due to the security limitations we are not passing the SSL Keystore and
truststore password in the /config/server.properties file.
Everything runs fine when password does not contain any special characters but
gives below exception in kafka server.log and the Kafka fails to start when
some special characters are provided in the passwords.
----------------------------------------------------------------------------------------------------------------------------------------------------------
Caused by: org.apache.kafka.common.KafkaException:
org.apache.kafka.common.KafkaException: Failed to load SSL keystore
/xx/xx/xx/kafka/client.truststore.jks of type JKS
at
org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:184)
at
org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:192)
at
org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:81)
at
org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:119)
at
org.apache.kafka.clients.ClientUtils.createNetworkClient(ClientUtils.java:223)
... 10 more
Caused by: org.apache.kafka.common.KafkaException: Failed to load SSL keystore
/xx/xx/xx/kafka/client.truststore.jks of type JKS
at
org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.load(DefaultSslEngineFactory.java:382)
at
org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.<init>(DefaultSslEngineFactory.java:354)
at
org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.createTruststore(DefaultSslEngineFactory.java:327)
at
org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.configure(DefaultSslEngineFactory.java:171)
at
org.apache.kafka.common.security.ssl.SslFactory.instantiateSslEngineFactory(SslFactory.java:141)
at
org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:98)
at
org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:180)
... 14 more
Caused by: java.io.IOException: Keystore was tampered with, or password was
incorrect
at
java.base/sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:813)
at
java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:221)
at java.base/java.security.KeyStore.load(KeyStore.java:1473)
at
org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.load(DefaultSslEngineFactory.java:379)
... 20 more
Caused by: java.security.UnrecoverableKeyException: Password verification failed
at
java.base/sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:811)
... 23 more
-------------------------------------------------------------------------------------------------------------------------------------------------------------
We have tested various special characters in passwords, including:
! " # $ % & ' ( ) * + , - . / : ; < = > ? @ [ \ ] ^ _ { | } ~`
Among these, the following characters work fine, and the Kafka service runs
without issues:
! @ # % ^ & * _ - . ? / ~ : ; < > | { } $ +
(We tested these by placing them at the end of the password, e.g.,
abc4!@#%^&*_-.?/~:;<>|{}$+.)
However, we observed that some characters behave differently depending on their
position in the password.
$ and + work if used at the end of the password but cause issues if used at the
beginning.
Certain characters, such as , [ ] ( ) ` = do not work regardless of their
position.
Please note that the same password works successfully when passed in
/config/server.properties file.
We think that this behavior occurs because Kafka is started via a shell script,
and some special characters have predefined meanings in the shell, leading to
unintended interpretation issues. Since the position of a character impacts its
behavior, there could be other combinations where the allowed characters
mentioned above are placed differently within the password, which may still
cause failures.
Since we suspect this issue is related to shell interpretation, we have tried
all publicly available solutions (like passing the password in env variable,
backlashing the special character in the passwords and etc), but the issue
persists.
Please help us to understand the issue and provide any solution.
Regards,
Deepak Jain
Cumulus Systems
