NO SCRAM-SASL AUTHENTICATION BETWEEN KRAFT CONTROL NODES
Hi there.
I successfully build a 5 node Kafka Raft cluster:
3) controller only role node
kraft-ct-1,kraft-ct-2,kraft-ct-3
2) broker only node
kraft-bro-1,kraft-bro-2
i setup ssl authentication :
controller -> controller
broker -> controller
And all works as expected
I enabled SASL_SCRAM, on brokers nodes
i precreate a “scram user” before starting broker
bin/kafka-storage.sh format -t jkZVRnZCRBqx1V6UUFDm5w -c
/opt/kafka/config/server.properties --add-scram
'SCRAM-SHA-256=[name=admin,password=admin-secret]'
and i can l connect from a producer/consumer with that user
But, when i try to enable SCRAM/SASL on controller -> controller
i get this error
[2023-11-03 09:05:02,134] INFO [SocketServer listenerType=CONTROLLER,
nodeId=1] Failed authentication with /192.168.1.34
(channelId=192.168.1.33:9093-192.168.1.34:56006-1304) (Authentication
failed during authentication due to invalid credentials with SASL mechanism
SCRAM-SHA-256) (org.apache.kafka.common.network.Selector)
NON WORKING KAFKA INTER CONTROLLER SCRAM CONFIGURATIONS
“kafka controller only node”
process.roles=controller
node.id=1
controller.quorum.voters=1@kraft-ct-1:9093,2@kraft-ct-2:9093,3@kraft-ct-3
:9093
listeners=CONTROLLER://kraft-ct-1:9093
controller.listener.names=CONTROLLER
listener.security.protocol.map=CONTROLLER:SASL_SSL,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL
sasl.enabled.mechanisms=SASL_SCRAM,SASL_PLAINTEXT,SCRAM-SHA-256,SCRAM-SHA-512
sasl.mechanism.controller.protocol=SCRAM-SHA-256
listener.name.controller.sasl.enabled.mechanisms=SCRAM-SHA-256
listener.name.controller.ssl.client.auth=required
ssl.truststore.location=/etc/keystore/server.truststore.jks
ssl.truststore.password=123456
ssl.keystore.location=/etc/keystore/kraft-kafka.keystore.jks
ssl.keystore.password=123456
ssl.key.password=123456
“controller jaas file config”
KafkaServer {
org.apache.kafka.common.security.scram.ScramLoginModule required
username="admin"
password="admin-secret";
};
Controller number 1 start correctly
Controleer number 2 start but won't join the cluster with error
[2023-11-03 10:02:55,622] INFO [RaftManager id=2] Failed authentication
with kraft-ct-1/192.168.1.33 (channelId=1) (Authentication failed during
authentication due to invalid credentials with SASL mechanism
SCRAM-SHA-256) (org.apache.kafka.common.network.Selector)
[2023-11-03 10:02:55,626] INFO [RaftManager id=2] Node 1 disconnected.
(org.apache.kafka.clients.NetworkClient)
[2023-11-03 10:02:55,627] ERROR [RaftManager id=2] Connection to node 1
(kraft-ct-1/192.168.1.33:9093) failed authentication due to: Authentication
failed during authentication due to invalid credentials with SASL mechanism
SCRAM-SHA-256 (org.apache.kafka.clients.NetworkClient)
[2023-11-03 10:02:55,628] ERROR [kafka-2-raft-outbound-request-thread]:
Failed to send the following request due to authentication error:
ClientRequest(expectResponse=true,
callback=kafka.raft.KafkaNetworkChannel$$Lambda$585/0x0000000840405440@4c3fa10e,
destination=1, correlationId=0, clientId=raft-client-2,
createdTimeMs=1699005774792,
requestBuilder=VoteRequestData(clusterId='jkZVRnZCRBqx1V6UUFDm5w',
topics=[TopicData(topicName='__cluster_metadata',
partitions=[PartitionData(partitionIndex=0, candidateEpoch=2706,
candidateId=2, lastOffsetEpoch=2598, lastOffset=10913)])]))
(kafka.raft.RaftSendThread)
[2023-11-03 10:02:55,633] ERROR Request OutboundRequest(correlationId=0,
data=VoteRequestData(clusterId='jkZVRnZCRBqx1V6UUFDm5w',
topics=[TopicData(topicName='__cluster_metadata',
partitions=[PartitionData(partitionIndex=0, candidateEpoch=2706,
candidateId=2, lastOffsetEpoch=2598, lastOffset=10913)])]),
createdTimeMs=1699005774792, destinationId=1) failed due to authentication
error (kafka.raft.KafkaNetworkChannel)
org.apache.kafka.common.errors.SaslAuthenticationException: Authentication
failed during authentication due to invalid credentials with SASL mechanism
SCRAM-SHA-256
[2023-11-03 10:02:55,640] ERROR [RaftManager id=2] Unexpected error
NETWORK_EXCEPTION in VOTE response: InboundResponse(correlationId=0,
data=VoteResponseData(errorCode=13, topics=[]), sourceId=1)
(org.apache.kafka.raft.KafkaRaftClient)
[2023-11-03 10:02:55,667] ERROR [kafka-2-raft-outbound-request-thread]:
Failed to send the following request due to authentication error:
ClientRequest(expectResponse=true,
callback=kafka.raft.KafkaNetworkChannel$$Lambda$585/0x0000000840405440@44e72a57,
destination=1, correlationId=9, clientId=raft-client-2,
createdTimeMs=1699005775659,
requestBuilder=VoteRequestData(clusterId='jkZVRnZCRBqx1V6UUFDm5w',
topics=[TopicData(topicName='__cluster_metadata',
partitions=[PartitionData(partitionIndex=0, candidateEpoch=2706,
candidateId=2, lastOffsetEpoch=2598, lastOffset=10913)])]))
(kafka.raft.RaftSendThread)
[2023-11-03 10:02:55,667] ERROR Request OutboundRequest(correlationId=9,
data=VoteRequestData(clusterId='jkZVRnZCRBqx1V6UUFDm5w',
topics=[TopicData(topicName='__cluster_metadata',
partitions=[PartitionData(partitionIndex=0, candidateEpoch=2706,
candidateId=2, lastOffsetEpoch=2598, lastOffset=10913)])]),
createdTimeMs=1699005775659, destinationId=1) failed due to authentication
error (kafka.raft.KafkaNetworkChannel)
org.apache.kafka.common.errors.SaslAuthenticationException: Authentication
failed during authentication due to invalid credentials with SASL mechanism
SCRAM-SHA-256
[2023-11-03 10:02:55,668] ERROR [RaftManager id=2] Unexpected error
NETWORK_EXCEPTION in VOTE response: InboundResponse(correlationId=9,
data=VoteResponseData(errorCode=13, topics=[]), sourceId=1)
(org.apache.kafka.raft.KafkaRaftClient)
Did i miss something or it’s not supported SASL/SCRAM controller to
controller?
WORKING CONFIGURATIONS
“kafka controller only node”
process.roles=controller
node.id=1
controller.quorum.voters=1@kraft-ct-1:9093,2@kraft-ct-2:9093,3@kraft-ct-3
:9093
listeners=CONTROLLER://kraft-ct-1:9093
controller.listener.names=CONTROLLER
listener.security.protocol.map=CONTROLLER:SSL,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL
listener.name.controller.ssl.client.auth=required
ssl.truststore.location=/etc/keystore/server.truststore.jks
ssl.truststore.password=123456
ssl.keystore.location=/etc/keystore/kraft-kafka.keystore.jks
ssl.keystore.password=123456
ssl.key.password=123456
“kafka broker only node”
process.roles=broker
node.id=4
controller.quorum.voters=1@kraft-ct-1:9093,2@kraft-ct-2:9093,3@kraft-ct-3
:9093
listeners=BROKER://kraft-bro-1:9092
inter.broker.listener.name=BROKER
advertised.listeners=BROKER://kraft-bro-1:9092
controller.listener.names=CONTROLLER
listener.security.protocol.map=CONTROLLER:SSL,BROKER:SASL_SSL,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL
sasl.enabled.mechanisms=SASL_PLAINTEXT,SCRAM-SHA-256,SCRAM-SHA-512
sasl.mechanism.inter.broker.protocol=SCRAM-SHA-256
ssl.client.auth=required
ssl.truststore.location=/etc/keystore/server.truststore.jks
ssl.truststore.password=123456
ssl.keystore.location=/etc/keystore//kraft-kafka.keystore.jks
ssl.keystore.password=123456
ssl.key.password=123456
“broker jaas_file config”
//opt/kafka/config/kafka_server_jaas.conf
KafkaServer {
org.apache.kafka.common.security.scram.ScramLoginModule required
username="admin"
password="admin-secret";
};
“client jaas confie file”
opt/kafka/config/client-ssl.properties
security.protocol=SASL_SSL
ssl.truststore.location=/etc/aruba-certificate/keystore/kraft-client.truststore.jks
ssl.truststore.password=123456
ssl.keystore.location=/etc/aruba-certificate/keystore/kraft-client.keystore.jks
ssl.keystore.password=123456
ssl.key.password=123456
sasl.mechanism=SCRAM-SHA-256
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule
required \
username="admin" \
password="admin-secret";
with above configurations all work perfectly
the quorum is formed
the broker connect to controller
the producer/consumer connect to broker
/opt/kafka/bin/kafka-metadata-quorum.sh --bootstrap-server
kraft-bro-1:9092 --command-config /opt/kafka/config/client-ssl.properties
describe --status
ClusterId: jkZVRnZCRBqx1V6UUFDm5w
LeaderId: 1
LeaderEpoch: 2598
HighWatermark: 9895
MaxFollowerLag: 0
MaxFollowerLagTimeMs: 0
CurrentVoters: [1,2,3]
CurrentObservers: [4,5]
/opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server
kraft-bro-1:9092 --consumer.config /opt/kafka/config/client-ssl.properties
--topic pippo --from-beginning
111,2222
222,2333
2323123,2312321
312312,12312312
Thanks
