Hey, I want to set up Kafka with SASL_SSL in a docker enviroment kafka should be albe to recives message encrypted over the puplic internet in addition, telegraf grafana and more are used in the backend everything runs flawlessly
only the zookeeper and kafka should be allowed to communicate on the internet kafka should be encrypted with SASL_SSL SCRAM-SHA-512 the connection between kafka and zookeeper should run via DIGEST-MD5 but i can't find a solution because i always get the following errors ERROR SASL authentication failed using login context 'Client'. (org.apache.zookeeper.client.ZooKeeperSaslClient ERROR Error occurred while connecting to Zookeeper server[zookeeper:2181,zookeeper:2182]. Authentication failed. javax.security.sasl.SaslException: Error in authenticating with a Zookeeper Quorum member: the quorum member's saslToken is null. i also add the full log ===> Configuring ... Running in Zookeeper mode... SSL is enabled. SASL is enabled. ===> Running preflight checks ... ===> Check if /var/lib/kafka/data is writable ... ===> Check if Zookeeper is healthy ... [2023-08-14 14:15:47,648] INFO SASL is enabled. java.security.auth.login.config=/etc/kafka/sasl.jaas.config (io.confluent.admin.utils.ClusterStatus) [2023-08-14 14:15:47,677] INFO Client environment:zookeeper.version=3.6.3--6401e4ad2087061bc6b9f80dec2d69f2e3c8660a, built on 04/08/2021 16:35 GMT (org.apache.zookeeper.ZooKeeper) [2023-08-14 14:15:47,677] INFO Client environment:host.name=353d245d9a35 (org.apache.zookeeper.ZooKeeper) [2023-08-14 14:15:47,677] INFO Client environment:java.version=11.0.18 (org.apache.zookeeper.ZooKeeper) [2023-08-14 14:15:47,677] INFO Client environment:java.vendor=Azul Systems, Inc. (org.apache.zookeeper.ZooKeeper) [2023-08-14 14:15:47,678] INFO Client environment:java.home=/usr/lib/jvm/zulu11-ca (org.apache.zookeeper.ZooKeeper) 8-14 14:15:47,678] INFO Client environment:java.class.path=/usr/share/java/cp-base-new/zookeeper-jute-3.6.3.jar:/usr/share/java/cp-base-new/jackson-dataformat-yaml-2.14.2.jar:/usr/share/java/cp-base-new/argparse4j-0.7.0.jar:/usr/share/java/cp-base-new/commons-cli-1.4.jar:/usr/share/java/cp-base-new/metrics-core-2.2.0.jar:/usr/share/java/cp-base-new/kafka-storage-7.4.1-ccs.jar:/usr/share/java/cp-base-new/jackson-databind-2.14.2.jar:/usr/share/java/cp-base-new/jackson-annotations-2.14.2.jar:/usr/share/java/cp-base-new/disk-usage-agent-7.4.1.jar:/usr/share/java/cp-base-new/scala-reflect-2.13.10.jar:/usr/share/java/cp-base-new/kafka-metadata-7.4.1-ccs.jar:/usr/share/java/cp-base-new/lz4-java-1.8.0.jar:/usr/share/java/cp-base-new/json-simple-1.1.1.jar:/usr/share/java/cp-base-new/re2j-1.6.jar:/usr/share/java/cp-base-new/snakeyaml-2.0.jar:/usr/share/java/cp-base-new/metrics-core-4.1.12.1.jar:/usr/share/java/cp-base-new/gson-2.9.0.jar:/usr/share/java/cp-base-new/slf4j-api-1.7.36.jar:/usr/share/java/cp-base-new/scala-collection-compat_2.13-2.10.0.jar:/usr/share/java/cp-base-new/kafka-group-coordinator-7.4.1-ccs.jar:/usr/share/java/cp-base-new/paranamer-2.8.jar:/usr/share/java/cp-base-new/audience-annotations-0.5.0.jar:/usr/share/java/cp-base-new/slf4j-reload4j-1.7.36.jar:/usr/share/java/cp-base-new/zstd-jni-1.5.2-1.jar:/usr/share/java/cp-base-new/jackson-dataformat-csv-2.14.2.jar:/usr/share/java/cp-base-new/jose4j-0.9.3.jar:/usr/share/java/cp-base-new/jmx_prometheus_javaagent-0.18.0.jar:/usr/share/java/cp-base-new/common-utils-7.4.1.jar:/usr/share/java/cp-base-new/kafka_2.13-7.4.1-ccs.jar:/usr/share/java/cp-base-new/kafka-clients-7.4.1-ccs.jar:/usr/share/java/cp-base-new/snappy-java-1.1.10.1.jar:/usr/share/java/cp-base-new/jopt-simple-5.0.4.jar:/usr/share/java/cp-base-new/zookeeper-3.6.3.jar:/usr/share/java/cp-base-new/scala-logging_2.13-3.9.4.jar:/usr/share/java/cp-base-new/scala-java8-compat_2.13-1.0.2.jar:/usr/share/java/cp-base-new/jackson-core-2.14.2.jar:/usr/share/java/cp-base-new/jolokia-jvm-1.7.1.jar:/usr/share/java/cp-base-new/logredactor-1.0.12.jar:/usr/share/java/cp-base-new/kafka-server-common-7.4.1-ccs.jar:/usr/share/java/cp-base-new/kafka-storage-api-7.4.1-ccs.jar:/usr/share/java/cp-base-new/jackson-datatype-jdk8-2.14.2.jar:/usr/share/java/cp-base-new/kafka-raft-7.4.1-ccs.jar:/usr/share/java/cp-base-new/scala-library-2.13.10.jar:/usr/share/java/cp-base-new/jackson-module-scala_2.13-2.14.2.jar:/usr/share/java/cp-base-new/jolokia-core-1.7.1.jar:/usr/share/java/cp-base-new/utility-belt-7.4.1.jar:/usr/share/java/cp-base-new/logredactor-metrics-1.0.12.jar:/usr/share/java/cp-base-new/reload4j-1.2.19.jar:/usr/share/java/cp-base-new/minimal-json-0.9.5.jar (org.apache.zookeeper.ZooKeeper) [2023-08-14 14:15:47,678] INFO Client environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper) [2023-08-14 14:15:47,679] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper) [2023-08-14 14:15:47,679] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper) [2023-08-14 14:15:47,679] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper) [2023-08-14 14:15:47,679] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper) [2023-08-14 14:15:47,680] INFO Client environment:os.version=5.10.0-21-amd64 (org.apache.zookeeper.ZooKeeper) [2023-08-14 14:15:47,680] INFO Client environment:user.name=appuser (org.apache.zookeeper.ZooKeeper) [2023-08-14 14:15:47,680] INFO Client environment:user.home=/home/appuser (org.apache.zookeeper.ZooKeeper) [2023-08-14 14:15:47,680] INFO Client environment:user.dir=/home/appuser (org.apache.zookeeper.ZooKeeper) [2023-08-14 14:15:47,681] INFO Client environment:os.memory.free=55MB (org.apache.zookeeper.ZooKeeper) [2023-08-14 14:15:47,681] INFO Client environment:os.memory.max=984MB (org.apache.zookeeper.ZooKeeper) [2023-08-14 14:15:47,681] INFO Client environment:os.memory.total=62MB (org.apache.zookeeper.ZooKeeper) [2023-08-14 14:15:47,687] INFO Initiating client connection, connectString=zookeeper:2181,zookeeper:2182 sessionTimeout=40000 watcher=io.confluent.admin.utils.ZookeeperConnectionWatcher@221af3c0 (org.apache.zookeeper.ZooKeeper) [2023-08-14 14:15:47,694] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util) [2023-08-14 14:15:47,704] INFO jute.maxbuffer value is 1048575 Bytes (org.apache.zookeeper.ClientCnxnSocket) [2023-08-14 14:15:47,715] INFO zookeeper.request.timeout value is 0. feature enabled=false (org.apache.zookeeper.ClientCnxn) [2023-08-14 14:15:47,837] INFO Client successfully logged in. (org.apache.zookeeper.Login) [2023-08-14 14:15:47,842] INFO Client will use DIGEST-MD5 as SASL mechanism. (org.apache.zookeeper.client.ZooKeeperSaslClient) [2023-08-14 14:15:47,884] INFO Opening socket connection to server zookeeper/192.168.112.2:2182. (org.apache.zookeeper.ClientCnxn) [2023-08-14 14:15:47,884] INFO SASL config status: Will attempt to SASL-authenticate using Login Context section 'Client' (org.apache.zookeeper.ClientCnxn) [2023-08-14 14:15:47,899] INFO Socket connection established, initiating session, client: /192.168.112.3:47736, server: zookeeper/192.168.112.2:2182 (org.apache.zookeeper.ClientCnxn) [2023-08-14 14:15:47,905] WARN Session 0x0 for sever zookeeper/192.168.112.2:2182, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. (org.apache.zookeeper.ClientCnxn) EndOfStreamException: Unable to read additional data from server sessionid 0x0, likely server has closed socket at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:77) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290) [2023-08-14 14:15:48,808] INFO Client successfully logged in. (org.apache.zookeeper.Login) [2023-08-14 14:15:48,809] INFO Client will use DIGEST-MD5 as SASL mechanism. (org.apache.zookeeper.client.ZooKeeperSaslClient) [2023-08-14 14:15:48,810] INFO Opening socket connection to server zookeeper/192.168.112.2:2181. (org.apache.zookeeper.ClientCnxn) [2023-08-14 14:15:48,811] INFO SASL config status: Will attempt to SASL-authenticate using Login Context section 'Client' (org.apache.zookeeper.ClientCnxn) [2023-08-14 14:15:48,815] INFO Socket connection established, initiating session, client: /192.168.112.3:58200, server: zookeeper/192.168.112.2:2181 (org.apache.zookeeper.ClientCnxn) [2023-08-14 14:15:48,823] INFO Session establishment complete on server zookeeper/192.168.112.2:2181, session id = 0x101527ddfc400fd, negotiated timeout = 40000 (org.apache.zookeeper.ClientCnxn) [2023-08-14 14:15:48,842] ERROR SASL authentication failed using login context 'Client'. (org.apache.zookeeper.client.ZooKeeperSaslClient) javax.security.sasl.SaslException: Error in authenticating with a Zookeeper Quorum member: the quorum member's saslToken is null. at org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslToken(ZooKeeperSaslClient.java:310) at org.apache.zookeeper.client.ZooKeeperSaslClient.respondToServer(ZooKeeperSaslClient.java:270) at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:936) at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:98) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290) [2023-08-14 14:15:48,843] ERROR Error occurred while connecting to Zookeeper server[zookeeper:2181,zookeeper:2182]. Authentication failed. (io.confluent.admin.utils.ClusterStatus) [2023-08-14 14:15:48,843] INFO EventThread shut down for session: 0x101527ddfc400fd (org.apache.zookeeper.ClientCnxn) Using log4j config /etc/kafka/log4j.properties Here are my configs server.properties # Broker Basics broker.id=1 listeners=SASL_PLAINTEXT://kafka:9092,SSL://kafka:9093,SASL_SSL://kafka:9094 num.network.threads=3 num.io.threads=8 zookeeper.connect=kafka:2181,kafka:2182 zookeeper.set.acl=true authorizer.class.name=kafka.security.auth.AclAuthorizer auto.create.topics.enable=false num.partitions=1 default.replication.factor=1 socket.request.max.bytes=1000000000 max.request.size=1000000000 security.inter.broker.protocol=SSL ssl.client.auth=required ssl.endpoint.identification.algorithm= sasl.mechanism.inter.broker.protocol=SCRAM-SHA-512 sasl.enabled.mechanisms=SCRAM-SHA-512,DIGEST-MD5 ##CA## ssl.truststore.location=/etc/kafka/truststore/kafka.truststore.jks ssl.truststore.password=password ssl.keystore.location=/etc/kafka/keystore/kafka.keystore.jks ssl.keystore.password=password ssl.key.password=password sasl.enabled.mechanisms=PLAIN Kafka Jaas KafkaServer { org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="secret"; org.apache.zookeeper.server.auth.DigestLoginModule required username="admin" password="secret"; }; Client { org.apache.kafka.common.security.scram.ScramLoginModule required username="admin" password="secret"; }; Btw tryed mamy combinnation like just Server{ } as name and more zookeeper.properties dataDir=/var/lib/zookeeper/data dataLogDir=/var/lib/zookeeper/log clientPort=2181 tickTime=2000 maxClientCnxns=100 autopurge.snapRetainCount=12 autopurge.purgeInterval=168 initLimit=10 syncLimit=5 secureClientPort=2182 ssl.keyStore.location=/etc/kafka/keystore/zookeeper.keystore.jks ssl.keyStore.password=password ssl.trustStore.location=/etc/kafka/truststore/zookeeper.truststore.jks ssl.trustStore.password=password serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory authProvider=org.apache.zookeeper.server.auth.SASLAuthenticationProvider admin.enableServer=false Zookeeper.jaas Server { org.apache.zookeeper.server.auth.DigestLoginModule required user_admin="secret"; }; and thats my docker Compose: version: '3.6' services: zookeeper: image: 'confluentinc/cp-zookeeper:latest' container_name: zookeeper restart: always ports: - '2181:2181' - '2182:2182' environment: ZOOKEEPER_CLIENT_PORT: 2181 ZOOKEEPER_SECURE_CLIENT_PORT: 2182 serverCnxnFactory: org.apache.zookeeper.server.NettyServerCnxnFactory ZOOKEEPER_OPTS: -Djava.security.auth.login.config=/etc/kafka/zookeeper.jaas ZOOKEEPER_SERVER_ID: 1 ZOOKEEPER_TICK_TIME: 2000 ZOOKEEPER_INIT_LIMIT: 10 ZOOKEEPER_SYNC_LIMIT: 5 ZOOKEEPER_DATADIR_AUTOCREATE: "false" ZOOKEEPER_MAX_CLIENT_CNXNS: 100 ZOOKEEPER_AUTOPURGE_SNAP_RETAIN_COUNT: 12 ZOOKEEPER_AUTOPURGE_PURGE_INTERVAL: 168 ZOOKEEPER_ADMIN_ENABLE_SERVER: "false" ZOOKEEPER_AUTH_PROVIDER: org.apache.zookeeper.server.auth.SASLAuthenticationProvider ZOOKEEPER_REQUIRE_CLIENT_AUTH_SCHEME: sasl ZOOKEEPER_JAAS_LOGIN_RENEW: 3600000 ZOOKEEPER_AUTH_PROVIDER_X509: org.apache.zookeeper.server.auth.X509AuthenticationProvider ZOOKEEPER_SERVER_CNXN_FACTORY: org.apache.zookeeper.server.NettyServerCnxnFactory ZOOKEEPER_SSL_PROTOCOL: TLSv1.2 ZOOKEEPER_SSL_TRUSTSTORE_LOCATION: /etc/kafka/truststore/zookeeper.truststore.jks ZOOKEEPER_SSL_KEYSTORE_LOCATION: /etc/kafka/keystore/zookeeper.keystore.jks ZOOKEEPER_SSL_KEYSTORE_PASSWORD: password ZOOKEEPER_SSL_TRUSTSTORE_PASSWORD: password ZOOKEEPER_DIGEST_AUTHENTICATION_PROVIDER_SUPERDIGEST: admin:sha1hashpassword KAFKA_LOG4J_ROOTLOGLEVEL: DEBUG volumes: - /data/zookeeper/zookeeper.properties:/etc/kafka/zookeeper.properties - /data/zookeeper/zookeeper.jaas:/etc/kafka/zookeeper.jaas - /data/zookeeper/truststore:/etc/kafka/truststore - /data/zookeeper/keystore:/etc/kafka/keystore networks: - kafka_network kafka: image: 'confluentinc/cp-kafka:latest' container_name: kafka restart: always ports: - '9093:9093' - '9094:9094' depends_on: - zookeeper environment: KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181,zookeeper:2182 KAFKA_LISTENERS: SSL://kafka:9093,SASL_SSL://kakfa:9094 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,SASL_SSL://kafka:9093 KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 KAFKA_SSL_KEYSTORE_FILENAME: kafka.keystore.jks KAFKA_SSL_KEY_CREDENTIALS: password.key KAFKA_SSL_KEYSTORE_CREDENTIALS: password.key KAFKA_OPTS: "-Djava.security.auth.login.config=/etc/kafka/sasl.jaas.config" volumes: - /data/kafka/server.properties:/etc/kafka/server.properties - /data/kafka/keystore/kafka.keystore.jks:/etc/kafka/secrets/kafka.keystore.jks - /data/kafka/truststore/kafka.truststore.jks:/etc/kafka/secrets/truststore.keystore.jks - /data/kafka/password.key:/etc/kafka/secrets/password.key - /data/kafka/sasl.jaas.config/etc/kafka/sasl.jaas.config networks: - kafka_network networks: kafka_network: if you need any more details feel free to ask i can provide more it would help me a lot if someone know a good dokumentation to build this in a docker or if someone know what i do wrong best regrads Malte Haas