Hi all,
We've got a problem with high CPU usage on a 0.9 client. We've got a monitoring
system that polls kafka topics for metadata (to get the last message offset)
every so often, and this has started using very high CPU continuously. We're
seeing the following being spammed in the logs every 100ms:
2016-06-21T13:21:10,355 | DEBUG | o.a.k.c.NetworkClient [pool-11-thread-8] |
Initialize connection to node -1 for sending metadata request
2016-06-21T13:21:10,355 | DEBUG | o.a.k.c.NetworkClient [pool-11-thread-8] |
Initiating connection to node -1 at <hostname>:9092.
2016-06-21T13:21:10,355 | DEBUG | o.a.k.c.NetworkClient [pool-11-thread-8] |
Error connecting to node -1 at <hostname>:9092:
java.nio.channels.ClosedByInterruptException: null
at
java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
~[na:1.8.0_60-ea]
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:659)
~[na:1.8.0_60-ea]
at org.apache.kafka.common.network.Selector.connect(Selector.java:153)
~[monitoring-collector.jar:na]
at
org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:489)
[monitoring-collector.jar:na]
at
org.apache.kafka.clients.NetworkClient.access$400(NetworkClient.java:47)
[monitoring-collector.jar:na]
at
org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:624)
[monitoring-collector.jar:na]
at
org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:543)
[monitoring-collector.jar:na]
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:254)
[monitoring-collector.jar:na]
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:320)
[monitoring-collector.jar:na]
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:213)
[monitoring-collector.jar:na]
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:193)
[monitoring-collector.jar:na]
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitMetadataUpdate(ConsumerNetworkClient.java:134)
[monitoring-collector.jar:na]
at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorKnown(AbstractCoordinator.java:184)
[monitoring-collector.jar:na]
at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.fetchCommittedOffsets(ConsumerCoordinator.java:290)
[monitoring-collector.jar:na]
at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.refreshCommittedOffsetsIfNeeded(ConsumerCoordinator.java:272)
[monitoring-collector.jar:na]
at
org.apache.kafka.clients.consumer.KafkaConsumer.updateFetchPositions(KafkaConsumer.java:1299)
[monitoring-collector.jar:na]
at
org.apache.kafka.clients.consumer.KafkaConsumer.position(KafkaConsumer.java:1106)
[monitoring-collector.jar:na]
...
2016-06-21T13:21:10,355 | DEBUG | o.a.k.c.NetworkClient [pool-11-thread-8] |
Give up sending metadata request since no node is available
This is a single-broker cluster (id: 1), all on a single machine. There is
nothing being logged in the broker logs.
Can anyone help work out what is going wrong, and how we could fix it? In
particular, the '-1' node id is suspicious, but we can't work out where this
value is coming from.
Thanks,
SimonC
