Thanks Maciej for the prompt response. But we tried the Percent encoding and unicode encoding but it did not work either.
Please note that we are able to pass the special character via server.properties but not with the zookeeper config in the shell script. We tried putting environment variable in the server.properties and setting it's value with the decrypted password (containing special characters) from the shell script but it also didn't work. Due to the security limitations, we cannot pass the password in the plaintext in the server.properties file and we found no way to pass encrypted password via it. We uses our internal encryption/decryption mechanism which is running fine but unable to pass the decrypted password with special characters via the shell script. Please help us in this regards. Thanks, Deepak ________________________________ From: Maciej Malecki <emaci...@gmail.com> Sent: Tuesday, March 25, 2025 9:51 PM To: users@kafka.apache.org <users@kafka.apache.org> Subject: Re: Kafka process fails to start when special character is present in Keystore password in SSL encryption and SASL authentication Caution: This email originated from outside of the organization. Please call and confirm with the sender before opening attachments or clicking links inside the email. Hej, Look here: https://en.m.wikipedia.org/wiki/Percent-encoding *__________________* *Regards,* *Maciej Małecki* wt., 25 mar 2025, 11:30 użytkownik Deepak Jain < deepak.j...@cumulus-systems.com> napisał: > Hi Luke, > > > > We are using Kafka 3.7.0 Broker/Client system in our prod environment with > SASL_SSL communication between Kafka Clients and Broker. We are starting > the Kafka process from the shell using the below command. > > > `nohup $EXEC_KAFKA_CONFIG --zookeeper 127.0.0.1:2181 --entity-type > brokers --entity-name 0 --alter --add-config $zooKeeperConfig >> > $KAFKA_HOME/logs/nohup_z.out 2>&1 &` > `nohup $KAFKA_HOME/bin/kafka-server-start.sh > $KAFKA_HOME/config/server.properties >> $KAFKA_HOME/logs/nohup_b.out 2>&1 &` > > > Here, we are passing the SSL Keystore and truststore password details in > $zooKeeperConfig as shown below: > > > > zooKeeperConfig="listener.name.sasl_ssl.ssl.truststore.password=$KAFKA_SSL_KEYSTORE_AND_TRUSTSTORE_PWD,listener.name.sasl_ssl.ssl.keystore.password=$KAFKA_SSL_KEYSTORE_AND_TRUSTSTORE_PWD,listener.name.sasl_ssl.ssl.key.password=$KAFKA_SSL_KEYSTORE_AND_TRUSTSTORE_PWD,$KAFKA_SSL_PASSWORD_ENCODER_SECRET_PROP" > > > Due to the security limitations we are not passing the SSL Keystore and > truststore password in the /config/server.properties file. > > > Everything runs fine when password does not contain any special characters > but gives below exception in kafka server.log and the Kafka fails to start > when some special characters are provided in the passwords. > > > > ---------------------------------------------------------------------------------------------------------------------------------------------------------- > > Caused by: org.apache.kafka.common.KafkaException: > org.apache.kafka.common.KafkaException: Failed to load SSL keystore > /xx/xx/xx/kafka/client.truststore.jks of type JKS > at > org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:184) > at > org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:192) > at > org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:81) > at > org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:119) > at > org.apache.kafka.clients.ClientUtils.createNetworkClient(ClientUtils.java:223) > ... 10 more > Caused by: org.apache.kafka.common.KafkaException: Failed to load SSL > keystore /xx/xx/xx/kafka/client.truststore.jks of type JKS > at > org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.load(DefaultSslEngineFactory.java:382) > at > org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.<init>(DefaultSslEngineFactory.java:354) > at > org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.createTruststore(DefaultSslEngineFactory.java:327) > at > org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.configure(DefaultSslEngineFactory.java:171) > at > org.apache.kafka.common.security.ssl.SslFactory.instantiateSslEngineFactory(SslFactory.java:141) > at > org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:98) > at > org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:180) > ... 14 more > Caused by: java.io.IOException: Keystore was tampered with, or password > was incorrect > at > java.base/sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:813) > at > java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:221) > at java.base/java.security.KeyStore.load(KeyStore.java:1473) > at > org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.load(DefaultSslEngineFactory.java:379) > ... 20 more > Caused by: java.security.UnrecoverableKeyException: Password verification > failed > at > java.base/sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:811) > ... 23 more > > ------------------------------------------------------------------------------------------------------------------------------------------------------------- > > > We have tested various special characters in passwords, including: > > ! " # $ % & ' ( ) * + , - . / : ; < = > ? @ [ \ ] ^ _ { | } ~` > > Among these, the following characters work fine, and the Kafka service > runs without issues: > ! @ # % ^ & * _ - . ? / ~ : ; < > | { } $ + > (We tested these by placing them at the end of the password, e.g., > abc4!@#%^&*_-.?/~:;<>|{}$+.) > > However, we observed that some characters behave differently depending on > their position in the password. > $ and + work if used at the end of the password but cause issues if used > at the beginning. > Certain characters, such as , [ ] ( ) ` = do not work regardless of their > position. > > Please note that the same password works successfully when passed in > /config/server.properties file. > > We think that this behavior occurs because Kafka is started via a shell > script, and some special characters have predefined meanings in the shell, > leading to unintended interpretation issues. Since the position of a > character impacts its behavior, there could be other combinations where the > allowed characters mentioned above are placed differently within the > password, which may still cause failures. > > Since we suspect this issue is related to shell interpretation, we have > tried all publicly available solutions (like passing the password in env > variable, backlashing the special character in the passwords and etc), but > the issue persists. > > > Please help us to understand the issue and provide any solution. > > > > > Regards, > > Deepak Jain > > Cumulus Systems >
