Michal Medvecky created KAFKA-17521:
---------------------------------------
Summary: bootstrap-controller option buggy behavior
Key: KAFKA-17521
URL: https://issues.apache.org/jira/browse/KAFKA-17521
Project: Kafka
Issue Type: Bug
Components: admin
Affects Versions: 3.7.1
Reporter: Michal Medvecky
Once running kafka admin tools with --bootstrap-controller, I am experiencing
weird behavior. Let me show examples.
{code:java}
[appuser@e4bbc669d343 ~]$ kafka-configs --describe --bootstrap-controller
kafka1:9093 --command-config /tmp/kafka-client.properties --entity-type brokers
--entity-name 1
Dynamic configs for broker 1 are: {code}
That's "sort of" fine, but:
* my set up consists of 3 controller nodes (1,2,3) and 3 broker nodes (4,5,6).
* entity-type must be "brokers", even though I am connecting to a controller
(9093/tcp is a controller listener)
* node 1 is not a broker, but a controller instead ("for broker 1 are ...")
When trying to describe config for node 2:
{code:java}
[appuser@e4bbc669d343 ~]$ kafka-configs --describe --bootstrap-controller
kafka1:9093 --command-config /tmp/kafka-client.properties --entity-type brokers
--entity-name 2
Dynamic configs for broker 2 are:
Error while executing config command with args '--describe
--bootstrap-controller kafka1:9093 --command-config
/tmp/kafka-client.properties --entity-type brokers --entity-name 2'
java.util.concurrent.ExecutionException:
org.apache.kafka.common.errors.InvalidRequestException: Unexpected broker id,
expected 1 or empty string, but received 2
at
java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at
java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2096)
at
org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:180)
at kafka.admin.ConfigCommand$.getResourceConfig(ConfigCommand.scala:610)
at
kafka.admin.ConfigCommand$.$anonfun$describeResourceConfig$5(ConfigCommand.scala:568)
at
kafka.admin.ConfigCommand$.$anonfun$describeResourceConfig$5$adapted(ConfigCommand.scala:560)
at scala.collection.immutable.List.foreach(List.scala:333)
at
kafka.admin.ConfigCommand$.describeResourceConfig(ConfigCommand.scala:560)
at kafka.admin.ConfigCommand$.describeConfig(ConfigCommand.scala:538)
at kafka.admin.ConfigCommand$.processCommand(ConfigCommand.scala:343)
at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:97)
at kafka.admin.ConfigCommand.main(ConfigCommand.scala)
Caused by: org.apache.kafka.common.errors.InvalidRequestException: Unexpected
broker id, expected 1 or empty string, but received 2 {code}
Ehm, what? Expected 1? I need to describe configs for node 2, not 1. The same
thing happens, once connecting to node 2 instead of node 1:
{code:java}
[appuser@e4bbc669d343 ~]$ kafka-configs --describe --bootstrap-controller
kafka2:9093 --command-config /tmp/kafka-client.properties --entity-type brokers
--entity-name 2
Dynamic configs for broker 2 are:
Error while executing config command with args '--describe
--bootstrap-controller kafka2:9093 --command-config
/tmp/kafka-client.properties --entity-type brokers --entity-name 2'
java.util.concurrent.ExecutionException:
org.apache.kafka.common.errors.InvalidRequestException: Unexpected broker id,
expected 1 or empty string, but received 2
at
java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at
java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2096)
at
org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:180)
at kafka.admin.ConfigCommand$.getResourceConfig(ConfigCommand.scala:610)
at
kafka.admin.ConfigCommand$.$anonfun$describeResourceConfig$5(ConfigCommand.scala:568)
at
kafka.admin.ConfigCommand$.$anonfun$describeResourceConfig$5$adapted(ConfigCommand.scala:560)
at scala.collection.immutable.List.foreach(List.scala:333)
at
kafka.admin.ConfigCommand$.describeResourceConfig(ConfigCommand.scala:560)
at kafka.admin.ConfigCommand$.describeConfig(ConfigCommand.scala:538)
at kafka.admin.ConfigCommand$.processCommand(ConfigCommand.scala:343)
at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:97)
at kafka.admin.ConfigCommand.main(ConfigCommand.scala)
Caused by: org.apache.kafka.common.errors.InvalidRequestException: Unexpected
broker id, expected 1 or empty string, but received 2 {code}
If i specify {{--all}} instead of {{{}entity-name{}}}, what I see is:
{code:java}
[appuser@e4bbc669d343 ~]$ kafka-configs --describe --bootstrap-controller
kafka2:9093 --command-config /tmp/kafka-client.properties --entity-type brokers
--all
All configs for broker 1 are:
advertised.listeners=null sensitive=false synonyms={}
<redacted>
zookeeper.ssl.truststore.type=null sensitive=false synonyms={}
All configs for broker 2 are:
Error while executing config command with args '--describe
--bootstrap-controller kafka2:9093 --command-config
/tmp/kafka-client.properties --entity-type brokers --all'
java.util.concurrent.ExecutionException:
org.apache.kafka.common.errors.InvalidRequestException: Unexpected broker id,
expected 1 or empty string, but received 2
at
java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at
java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2096)
at
org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:180)
at kafka.admin.ConfigCommand$.getResourceConfig(ConfigCommand.scala:610)
at
kafka.admin.ConfigCommand$.$anonfun$describeResourceConfig$5(ConfigCommand.scala:568)
at
kafka.admin.ConfigCommand$.$anonfun$describeResourceConfig$5$adapted(ConfigCommand.scala:560)
at scala.collection.immutable.List.foreach(List.scala:333)
at
kafka.admin.ConfigCommand$.describeResourceConfig(ConfigCommand.scala:560)
at kafka.admin.ConfigCommand$.describeConfig(ConfigCommand.scala:538)
at kafka.admin.ConfigCommand$.processCommand(ConfigCommand.scala:343)
at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:97)
at kafka.admin.ConfigCommand.main(ConfigCommand.scala)
Caused by: org.apache.kafka.common.errors.InvalidRequestException: Unexpected
broker id, expected 1 or empty string, but received 2 {code}
Where exactly did I specify "2"?
If I want to describe configs for node 4 (broker), no matter what node I use as
{{{}--bootstrap-controller{}}}, I get a timeout:
{code:java}
[appuser@e4bbc669d343 ~]$ kafka-configs --describe --bootstrap-controller
kafka3:9093 --command-config /tmp/kafka-client.properties --entity-type brokers
--entity-name 4
Dynamic configs for broker 4 are:
Error while executing config command with args '--describe
--bootstrap-controller kafka3:9093 --command-config
/tmp/kafka-client.properties --entity-type brokers --entity-name 4'
java.util.concurrent.TimeoutException
at
java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1960)
at
java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2095)
at
org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:180)
at kafka.admin.ConfigCommand$.getResourceConfig(ConfigCommand.scala:610)
at
kafka.admin.ConfigCommand$.$anonfun$describeResourceConfig$5(ConfigCommand.scala:568)
at
kafka.admin.ConfigCommand$.$anonfun$describeResourceConfig$5$adapted(ConfigCommand.scala:560)
at scala.collection.immutable.List.foreach(List.scala:333)
at
kafka.admin.ConfigCommand$.describeResourceConfig(ConfigCommand.scala:560)
at kafka.admin.ConfigCommand$.describeConfig(ConfigCommand.scala:538)
at kafka.admin.ConfigCommand$.processCommand(ConfigCommand.scala:343)
at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:97)
at kafka.admin.ConfigCommand.main(ConfigCommand.scala){code}
If I specify any of the bootstrap servers, it works fine for brokers:
{code:java}
[appuser@e4bbc669d343 ~]$ kafka-configs --describe --bootstrap-server
kafka4:9092 --entity-type brokers --entity-name 4
Dynamic configs for broker 4 are:
[appuser@e4bbc669d343 ~]$ kafka-configs --describe --bootstrap-server
kafka4:9092 --entity-type brokers --entity-name 5
Dynamic configs for broker 5 are:
[appuser@e4bbc669d343 ~]$ kafka-configs --describe --bootstrap-server
kafka4:9092 --entity-type brokers --entity-name 6
Dynamic configs for broker 6 are:
[appuser@e4bbc669d343 ~]$ {code}
but describing config for controller node(s) fails:
{code:java}
[appuser@e4bbc669d343 ~]$ kafka-configs --describe --bootstrap-server
kafka4:9092 --entity-type brokers --entity-name 1
Dynamic configs for broker 1 are:
Error while executing config command with args '--describe --bootstrap-server
kafka4:9092 --entity-type brokers --entity-name 1'
java.util.concurrent.TimeoutException
at
java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1960)
at
java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2095)
at
org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:180)
at kafka.admin.ConfigCommand$.getResourceConfig(ConfigCommand.scala:610)
at
kafka.admin.ConfigCommand$.$anonfun$describeResourceConfig$5(ConfigCommand.scala:568)
at
kafka.admin.ConfigCommand$.$anonfun$describeResourceConfig$5$adapted(ConfigCommand.scala:560)
at scala.collection.immutable.List.foreach(List.scala:333)
at
kafka.admin.ConfigCommand$.describeResourceConfig(ConfigCommand.scala:560)
at kafka.admin.ConfigCommand$.describeConfig(ConfigCommand.scala:538)
at kafka.admin.ConfigCommand$.processCommand(ConfigCommand.scala:343)
at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:97)
at kafka.admin.ConfigCommand.main(ConfigCommand.scala)
[appuser@e4bbc669d343 ~]$ {code}
And the last stuff to be reported today is a SSL handshake problem for
kafka-features:
{code:java}
[appuser@e4bbc669d343 ~]$ kafka-features --bootstrap-controller kafka1:9093
--command-config /tmp/kafka-client.properties describe
[2024-09-11 10:13:42,466] ERROR [AdminClient clientId=adminclient-1] Connection
to node 2 (/127.0.0.1:9093) failed authentication due to: SSL handshake failed
(org.apache.kafka.clients.NetworkClient)
[2024-09-11 10:13:42,467] WARN [AdminClient clientId=adminclient-1] Metadata
update failed due to authentication error
(org.apache.kafka.clients.admin.internals.AdminMetadataManager)
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLHandshakeException: No subject alternative names
matching IP address 127.0.0.1 found
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
at
java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:378)
at
java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:321)
at
java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:316)
at
java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1351)
at
java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1226)
at
java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1169)
at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396)
at
java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480)
at
java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277)
at
java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264)
at
java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
at
java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209)
at
org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:443)
at
org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:532)
at
org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:381)
at
org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:301)
at
org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178)
at
org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543)
at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:585)
at
org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1504)
at
org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1435)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.security.cert.CertificateException: No subject alternative
names matching IP address 127.0.0.1 found
at
java.base/sun.security.util.HostnameChecker.matchIP(HostnameChecker.java:165)
at
java.base/sun.security.util.HostnameChecker.match(HostnameChecker.java:101)
at
java.base/sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:458)
at
java.base/sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:432)
at
java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:292)
at
java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
at
java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1329)
... 19 more
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
[appuser@e4bbc669d343 ~]${code}
The weird thing here would be, why the error is reporting that 127.0.0.1 is
missing in AltNames ... while I am using FQDNS:
{code:java}
[appuser@e4bbc669d343 ~]$ ping -c1 kafka1
PING kafka1 (172.30.0.2) 56(84) bytes of data.
64 bytes from e4bbc669d343 (172.30.0.2): icmp_seq=1 ttl=64 time=0.108 ms---
kafka1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.108/0.108/0.108/0.000 ms{code}
This being AltNames in the certificate:
{code:java}
X509v3 Subject Alternative Name:
DNS:kafka1.cd4460cf-3d86-4f1b-ad25-a7ec66cecbb8, DNS:kafka1
{code}
None of the names or config refering to localhost, no dns aliases ...
{code:java}
[appuser@e4bbc669d343 ~]$ cat /etc/hosts
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.30.0.2 e4bbc669d343 {code}
Finally, let me tell you what is my configuration for controllers (1,2,3, the
snippet shows config for node 1, other nodes just having respective numbers:
{code:java}
listener.name.controller.ssl.truststore.type=PEM
listener.name.controller.ssl.keystore.type=PEM
listener.name.controller.ssl.keystore.certificate.chain=<redacted>
transaction.state.log.min.isr=1
process.roles=controller
controller.listener.names=CONTROLLER
group.initial.rebalance.delay.ms=0
controller.quorum.voters=1@kafka1:9093,2@kafka2:9093,3@kafka3:9093
listener.name.controller.ssl.keystore.key=<redacted>
node.id=1
listener.name.controller.ssl.client.auth=required
kraft.mode=true
listener.name.controller.ssl.truststore.certificates=<redacted>
listener.security.protocol.map=CONTROLLER:SSL
listener.name.controller.ssl.endpoint.identification.algorithm=https
transaction.state.log.replication.factor=1
listeners=CONTROLLER://0.0.0.0:9093
zookeeper.connect=
log.dirs=/var/lib/kafka/data-1
offsets.topic.replication.factor=1 {code}
... and brokers:
{code:java}
listener.name.controller.ssl.truststore.type=PEM
listener.name.controller.ssl.keystore.type=PEM
listener.name.controller.ssl.keystore.certificate.chain=<redacted>
transaction.state.log.min.isr=1
process.roles=broker
controller.listener.names=CONTROLLER
group.initial.rebalance.delay.ms=0
controller.quorum.voters=1@kafka1:9093,2@kafka2:9093,3@kafka3:9093
listener.name.controller.ssl.keystore.key=<redacted>
node.id=4
listener.name.controller.ssl.client.auth=required
advertised.listeners=PLAINTEXT://kafka4:9092
kraft.mode=true
listener.name.controller.ssl.truststore.certificates=<redacted>
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,CONTROLLER:SSL
listener.name.controller.ssl.endpoint.identification.algorithm=https
transaction.state.log.replication.factor=1
listeners=PLAINTEXT://0.0.0.0:9092
zookeeper.connect=
log.dirs=/var/lib/kafka/data-4
offsets.topic.replication.factor=1 {code}
... with respective numbers (5,6) for another two instances.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
