Hey,
I want to set up Kafka with SASL_SSL in a docker enviroment kafka should be
albe to recives message encrypted over the puplic internet in addition,
telegraf grafana and more are used in the backend everything runs flawlessly
only the zookeeper and kafka should be allowed to communicate on the internet
kafka should be encrypted with SASL_SSL SCRAM-SHA-512
the connection between kafka and zookeeper should run via DIGEST-MD5
but i can't find a solution because i always get the following errors
ERROR SASL authentication failed using login context 'Client'.
(org.apache.zookeeper.client.ZooKeeperSaslClient
ERROR Error occurred while connecting to Zookeeper
server[zookeeper:2181,zookeeper:2182]. Authentication failed.
javax.security.sasl.SaslException: Error in authenticating with a Zookeeper
Quorum member: the quorum member's saslToken is null.
i also add the full log
===> Configuring ...
Running in Zookeeper mode...
SSL is enabled.
SASL is enabled.
===> Running preflight checks ...
===> Check if /var/lib/kafka/data is writable ...
===> Check if Zookeeper is healthy ...
[2023-08-14 14:15:47,648] INFO SASL is enabled.
java.security.auth.login.config=/etc/kafka/sasl.jaas.config
(io.confluent.admin.utils.ClusterStatus)
[2023-08-14 14:15:47,677] INFO Client
environment:zookeeper.version=3.6.3--6401e4ad2087061bc6b9f80dec2d69f2e3c8660a,
built on 04/08/2021 16:35 GMT (org.apache.zookeeper.ZooKeeper)
[2023-08-14 14:15:47,677] INFO Client environment:host.name=353d245d9a35
(org.apache.zookeeper.ZooKeeper)
[2023-08-14 14:15:47,677] INFO Client environment:java.version=11.0.18
(org.apache.zookeeper.ZooKeeper)
[2023-08-14 14:15:47,677] INFO Client environment:java.vendor=Azul Systems,
Inc. (org.apache.zookeeper.ZooKeeper)
[2023-08-14 14:15:47,678] INFO Client
environment:java.home=/usr/lib/jvm/zulu11-ca (org.apache.zookeeper.ZooKeeper)
8-14 14:15:47,678] INFO Client
environment:java.class.path=/usr/share/java/cp-base-new/zookeeper-jute-3.6.3.jar:/usr/share/java/cp-base-new/jackson-dataformat-yaml-2.14.2.jar:/usr/share/java/cp-base-new/argparse4j-0.7.0.jar:/usr/share/java/cp-base-new/commons-cli-1.4.jar:/usr/share/java/cp-base-new/metrics-core-2.2.0.jar:/usr/share/java/cp-base-new/kafka-storage-7.4.1-ccs.jar:/usr/share/java/cp-base-new/jackson-databind-2.14.2.jar:/usr/share/java/cp-base-new/jackson-annotations-2.14.2.jar:/usr/share/java/cp-base-new/disk-usage-agent-7.4.1.jar:/usr/share/java/cp-base-new/scala-reflect-2.13.10.jar:/usr/share/java/cp-base-new/kafka-metadata-7.4.1-ccs.jar:/usr/share/java/cp-base-new/lz4-java-1.8.0.jar:/usr/share/java/cp-base-new/json-simple-1.1.1.jar:/usr/share/java/cp-base-new/re2j-1.6.jar:/usr/share/java/cp-base-new/snakeyaml-2.0.jar:/usr/share/java/cp-base-new/metrics-core-4.1.12.1.jar:/usr/share/java/cp-base-new/gson-2.9.0.jar:/usr/share/java/cp-base-new/slf4j-api-1.7.36.jar:/usr/share/java/cp-base-new/scala-collection-compat_2.13-2.10.0.jar:/usr/share/java/cp-base-new/kafka-group-coordinator-7.4.1-ccs.jar:/usr/share/java/cp-base-new/paranamer-2.8.jar:/usr/share/java/cp-base-new/audience-annotations-0.5.0.jar:/usr/share/java/cp-base-new/slf4j-reload4j-1.7.36.jar:/usr/share/java/cp-base-new/zstd-jni-1.5.2-1.jar:/usr/share/java/cp-base-new/jackson-dataformat-csv-2.14.2.jar:/usr/share/java/cp-base-new/jose4j-0.9.3.jar:/usr/share/java/cp-base-new/jmx_prometheus_javaagent-0.18.0.jar:/usr/share/java/cp-base-new/common-utils-7.4.1.jar:/usr/share/java/cp-base-new/kafka_2.13-7.4.1-ccs.jar:/usr/share/java/cp-base-new/kafka-clients-7.4.1-ccs.jar:/usr/share/java/cp-base-new/snappy-java-1.1.10.1.jar:/usr/share/java/cp-base-new/jopt-simple-5.0.4.jar:/usr/share/java/cp-base-new/zookeeper-3.6.3.jar:/usr/share/java/cp-base-new/scala-logging_2.13-3.9.4.jar:/usr/share/java/cp-base-new/scala-java8-compat_2.13-1.0.2.jar:/usr/share/java/cp-base-new/jackson-core-2.14.2.jar:/usr/share/java/cp-base-new/jolokia-jvm-1.7.1.jar:/usr/share/java/cp-base-new/logredactor-1.0.12.jar:/usr/share/java/cp-base-new/kafka-server-common-7.4.1-ccs.jar:/usr/share/java/cp-base-new/kafka-storage-api-7.4.1-ccs.jar:/usr/share/java/cp-base-new/jackson-datatype-jdk8-2.14.2.jar:/usr/share/java/cp-base-new/kafka-raft-7.4.1-ccs.jar:/usr/share/java/cp-base-new/scala-library-2.13.10.jar:/usr/share/java/cp-base-new/jackson-module-scala_2.13-2.14.2.jar:/usr/share/java/cp-base-new/jolokia-core-1.7.1.jar:/usr/share/java/cp-base-new/utility-belt-7.4.1.jar:/usr/share/java/cp-base-new/logredactor-metrics-1.0.12.jar:/usr/share/java/cp-base-new/reload4j-1.2.19.jar:/usr/share/java/cp-base-new/minimal-json-0.9.5.jar
(org.apache.zookeeper.ZooKeeper)
[2023-08-14 14:15:47,678] INFO Client
environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib
(org.apache.zookeeper.ZooKeeper)
[2023-08-14 14:15:47,679] INFO Client environment:java.io.tmpdir=/tmp
(org.apache.zookeeper.ZooKeeper)
[2023-08-14 14:15:47,679] INFO Client environment:java.compiler=<NA>
(org.apache.zookeeper.ZooKeeper)
[2023-08-14 14:15:47,679] INFO Client environment:os.name=Linux
(org.apache.zookeeper.ZooKeeper)
[2023-08-14 14:15:47,679] INFO Client environment:os.arch=amd64
(org.apache.zookeeper.ZooKeeper)
[2023-08-14 14:15:47,680] INFO Client environment:os.version=5.10.0-21-amd64
(org.apache.zookeeper.ZooKeeper)
[2023-08-14 14:15:47,680] INFO Client environment:user.name=appuser
(org.apache.zookeeper.ZooKeeper)
[2023-08-14 14:15:47,680] INFO Client environment:user.home=/home/appuser
(org.apache.zookeeper.ZooKeeper)
[2023-08-14 14:15:47,680] INFO Client environment:user.dir=/home/appuser
(org.apache.zookeeper.ZooKeeper)
[2023-08-14 14:15:47,681] INFO Client environment:os.memory.free=55MB
(org.apache.zookeeper.ZooKeeper)
[2023-08-14 14:15:47,681] INFO Client environment:os.memory.max=984MB
(org.apache.zookeeper.ZooKeeper)
[2023-08-14 14:15:47,681] INFO Client environment:os.memory.total=62MB
(org.apache.zookeeper.ZooKeeper)
[2023-08-14 14:15:47,687] INFO Initiating client connection,
connectString=zookeeper:2181,zookeeper:2182 sessionTimeout=40000
watcher=io.confluent.admin.utils.ZookeeperConnectionWatcher@221af3c0
(org.apache.zookeeper.ZooKeeper)
[2023-08-14 14:15:47,694] INFO Setting -D
jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS
renegotiation (org.apache.zookeeper.common.X509Util)
[2023-08-14 14:15:47,704] INFO jute.maxbuffer value is 1048575 Bytes
(org.apache.zookeeper.ClientCnxnSocket)
[2023-08-14 14:15:47,715] INFO zookeeper.request.timeout value is 0. feature
enabled=false (org.apache.zookeeper.ClientCnxn)
[2023-08-14 14:15:47,837] INFO Client successfully logged in.
(org.apache.zookeeper.Login)
[2023-08-14 14:15:47,842] INFO Client will use DIGEST-MD5 as SASL mechanism.
(org.apache.zookeeper.client.ZooKeeperSaslClient)
[2023-08-14 14:15:47,884] INFO Opening socket connection to server
zookeeper/192.168.112.2:2182. (org.apache.zookeeper.ClientCnxn)
[2023-08-14 14:15:47,884] INFO SASL config status: Will attempt to
SASL-authenticate using Login Context section 'Client'
(org.apache.zookeeper.ClientCnxn)
[2023-08-14 14:15:47,899] INFO Socket connection established, initiating
session, client: /192.168.112.3:47736, server: zookeeper/192.168.112.2:2182
(org.apache.zookeeper.ClientCnxn)
[2023-08-14 14:15:47,905] WARN Session 0x0 for sever
zookeeper/192.168.112.2:2182, Closing socket connection. Attempting reconnect
except it is a SessionExpiredException. (org.apache.zookeeper.ClientCnxn)
EndOfStreamException: Unable to read additional data from server sessionid 0x0,
likely server has closed socket
at
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:77)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290)
[2023-08-14 14:15:48,808] INFO Client successfully logged in.
(org.apache.zookeeper.Login)
[2023-08-14 14:15:48,809] INFO Client will use DIGEST-MD5 as SASL mechanism.
(org.apache.zookeeper.client.ZooKeeperSaslClient)
[2023-08-14 14:15:48,810] INFO Opening socket connection to server
zookeeper/192.168.112.2:2181. (org.apache.zookeeper.ClientCnxn)
[2023-08-14 14:15:48,811] INFO SASL config status: Will attempt to
SASL-authenticate using Login Context section 'Client'
(org.apache.zookeeper.ClientCnxn)
[2023-08-14 14:15:48,815] INFO Socket connection established, initiating
session, client: /192.168.112.3:58200, server: zookeeper/192.168.112.2:2181
(org.apache.zookeeper.ClientCnxn)
[2023-08-14 14:15:48,823] INFO Session establishment complete on server
zookeeper/192.168.112.2:2181, session id = 0x101527ddfc400fd, negotiated
timeout = 40000 (org.apache.zookeeper.ClientCnxn)
[2023-08-14 14:15:48,842] ERROR SASL authentication failed using login context
'Client'. (org.apache.zookeeper.client.ZooKeeperSaslClient)
javax.security.sasl.SaslException: Error in authenticating with a Zookeeper
Quorum member: the quorum member's saslToken is null.
at
org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslToken(ZooKeeperSaslClient.java:310)
at
org.apache.zookeeper.client.ZooKeeperSaslClient.respondToServer(ZooKeeperSaslClient.java:270)
at
org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:936)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:98)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290)
[2023-08-14 14:15:48,843] ERROR Error occurred while connecting to Zookeeper
server[zookeeper:2181,zookeeper:2182]. Authentication failed.
(io.confluent.admin.utils.ClusterStatus)
[2023-08-14 14:15:48,843] INFO EventThread shut down for session:
0x101527ddfc400fd (org.apache.zookeeper.ClientCnxn)
Using log4j config /etc/kafka/log4j.properties
Here are my configs
server.properties
# Broker Basics
broker.id=1
listeners=SASL_PLAINTEXT://kafka:9092,SSL://kafka:9093,SASL_SSL://kafka:9094
num.network.threads=3
num.io.threads=8
zookeeper.connect=kafka:2181,kafka:2182
zookeeper.set.acl=true
authorizer.class.name=kafka.security.auth.AclAuthorizer
auto.create.topics.enable=false
num.partitions=1
default.replication.factor=1
socket.request.max.bytes=1000000000
max.request.size=1000000000
security.inter.broker.protocol=SSL
ssl.client.auth=required
ssl.endpoint.identification.algorithm=
sasl.mechanism.inter.broker.protocol=SCRAM-SHA-512
sasl.enabled.mechanisms=SCRAM-SHA-512,DIGEST-MD5
##CA##
ssl.truststore.location=/etc/kafka/truststore/kafka.truststore.jks
ssl.truststore.password=password
ssl.keystore.location=/etc/kafka/keystore/kafka.keystore.jks
ssl.keystore.password=password
ssl.key.password=password
sasl.enabled.mechanisms=PLAIN
Kafka Jaas
KafkaServer {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="admin"
password="secret";
org.apache.zookeeper.server.auth.DigestLoginModule required
username="admin"
password="secret";
};
Client {
org.apache.kafka.common.security.scram.ScramLoginModule required
username="admin"
password="secret";
};
Btw tryed mamy combinnation like just Server{ } as name and more
zookeeper.properties
dataDir=/var/lib/zookeeper/data
dataLogDir=/var/lib/zookeeper/log
clientPort=2181
tickTime=2000
maxClientCnxns=100
autopurge.snapRetainCount=12
autopurge.purgeInterval=168
initLimit=10
syncLimit=5
secureClientPort=2182
ssl.keyStore.location=/etc/kafka/keystore/zookeeper.keystore.jks
ssl.keyStore.password=password
ssl.trustStore.location=/etc/kafka/truststore/zookeeper.truststore.jks
ssl.trustStore.password=password
serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
authProvider=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
admin.enableServer=false
Zookeeper.jaas
Server {
org.apache.zookeeper.server.auth.DigestLoginModule required
user_admin="secret";
};
and thats my docker Compose:
version: '3.6'
services:
zookeeper:
image: 'confluentinc/cp-zookeeper:latest'
container_name: zookeeper
restart: always
ports:
- '2181:2181'
- '2182:2182'
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_SECURE_CLIENT_PORT: 2182
serverCnxnFactory: org.apache.zookeeper.server.NettyServerCnxnFactory
ZOOKEEPER_OPTS:
-Djava.security.auth.login.config=/etc/kafka/zookeeper.jaas
ZOOKEEPER_SERVER_ID: 1
ZOOKEEPER_TICK_TIME: 2000
ZOOKEEPER_INIT_LIMIT: 10
ZOOKEEPER_SYNC_LIMIT: 5
ZOOKEEPER_DATADIR_AUTOCREATE: "false"
ZOOKEEPER_MAX_CLIENT_CNXNS: 100
ZOOKEEPER_AUTOPURGE_SNAP_RETAIN_COUNT: 12
ZOOKEEPER_AUTOPURGE_PURGE_INTERVAL: 168
ZOOKEEPER_ADMIN_ENABLE_SERVER: "false"
ZOOKEEPER_AUTH_PROVIDER:
org.apache.zookeeper.server.auth.SASLAuthenticationProvider
ZOOKEEPER_REQUIRE_CLIENT_AUTH_SCHEME: sasl
ZOOKEEPER_JAAS_LOGIN_RENEW: 3600000
ZOOKEEPER_AUTH_PROVIDER_X509:
org.apache.zookeeper.server.auth.X509AuthenticationProvider
ZOOKEEPER_SERVER_CNXN_FACTORY:
org.apache.zookeeper.server.NettyServerCnxnFactory
ZOOKEEPER_SSL_PROTOCOL: TLSv1.2
ZOOKEEPER_SSL_TRUSTSTORE_LOCATION:
/etc/kafka/truststore/zookeeper.truststore.jks
ZOOKEEPER_SSL_KEYSTORE_LOCATION:
/etc/kafka/keystore/zookeeper.keystore.jks
ZOOKEEPER_SSL_KEYSTORE_PASSWORD: password
ZOOKEEPER_SSL_TRUSTSTORE_PASSWORD: password
ZOOKEEPER_DIGEST_AUTHENTICATION_PROVIDER_SUPERDIGEST:
admin:sha1hashpassword
KAFKA_LOG4J_ROOTLOGLEVEL: DEBUG
volumes:
- /data/zookeeper/zookeeper.properties:/etc/kafka/zookeeper.properties
- /data/zookeeper/zookeeper.jaas:/etc/kafka/zookeeper.jaas
- /data/zookeeper/truststore:/etc/kafka/truststore
- /data/zookeeper/keystore:/etc/kafka/keystore
networks:
- kafka_network
kafka:
image: 'confluentinc/cp-kafka:latest'
container_name: kafka
restart: always
ports:
- '9093:9093'
- '9094:9094'
depends_on:
- zookeeper
environment:
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181,zookeeper:2182
KAFKA_LISTENERS: SSL://kafka:9093,SASL_SSL://kakfa:9094
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,SASL_SSL://kafka:9093
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_SSL_KEYSTORE_FILENAME: kafka.keystore.jks
KAFKA_SSL_KEY_CREDENTIALS: password.key
KAFKA_SSL_KEYSTORE_CREDENTIALS: password.key
KAFKA_OPTS:
"-Djava.security.auth.login.config=/etc/kafka/sasl.jaas.config"
volumes:
- /data/kafka/server.properties:/etc/kafka/server.properties
-
/data/kafka/keystore/kafka.keystore.jks:/etc/kafka/secrets/kafka.keystore.jks
-
/data/kafka/truststore/kafka.truststore.jks:/etc/kafka/secrets/truststore.keystore.jks
- /data/kafka/password.key:/etc/kafka/secrets/password.key
- /data/kafka/sasl.jaas.config/etc/kafka/sasl.jaas.config
networks:
- kafka_network
networks:
kafka_network:
if you need any more details feel free to ask i can provide more
it would help me a lot if someone know a good dokumentation to build this in a
docker or if someone know what i do wrong
best regrads
Malte Haas
