[
https://issues.apache.org/jira/browse/KAFKA-9250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vijay resolved KAFKA-9250.
--------------------------
Resolution: Resolved
> Kafka streams stuck in rebalancing state after UnknownHostException
> -------------------------------------------------------------------
>
> Key: KAFKA-9250
> URL: https://issues.apache.org/jira/browse/KAFKA-9250
> Project: Kafka
> Issue Type: Bug
> Components: clients, config, network, streams
> Affects Versions: 2.2.0
> Reporter: Vijay
> Priority: Critical
>
> We are using kafka streams (2.2.0) application for reading messages from
> source topic do some transformation and send to destination topic.
> Application started fine and processed messages till it encountered
> UnknownHostException, after which application is hung in rebalancing state
> and not processing messages.
>
> Below are the properties we have configured :
> application.id = *****
> bootstrap.servers = "hostname1:port1,hostname2:port2,hostname3:port3"
> num.stream.threads=3
> replication.factor=3
> num.standby.replicas=1
> max.block.ms=86400000
> acks=all
> auto.offset.reset=earliest
> processing.guarantee=exactly_once
>
> Additional details.
> Number of brokers - 3
> Source topic partition count - 12 and replication factor of 3
> Destination topic partition count - 12 and replication factor of 3
> 4 instances of stream application are deployed in docker containers.
>
> Below are the some of the logs :
> {noformat}
> [WARN] [streams-example-9213da8c-22ad-4116-82bd-47abf80bbf15-StreamThread-1]
> o.a.k.clients.NetworkClient - [Consumer
> clientId=streams-example-9213da8c-22ad-4116-82bd-47abf80bbf15-StreamThread-1-consumer,
> groupId=streams-example] Error connecting to node hostname1:port1 (id:
> 2147438464 rack: null)
> java.net.UnknownHostException: hostname1
> at java.net.InetAddress.getAllByName0(InetAddress.java:1280)
> at java.net.InetAddress.getAllByName(InetAddress.java:1192)
> at java.net.InetAddress.getAllByName(InetAddress.java:1126)
> at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:117)
> at
> org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.moveToNextAddress(ClusterConnectionStates.java:387)
>
> at
> org.apache.kafka.clients.ClusterConnectionStates.connecting(ClusterConnectionStates.java:121)
>
> at
> org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:917)
>
> at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:287)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.tryConnect(ConsumerNetworkClient.java:548)
>
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$FindCoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:676)
>
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$FindCoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:656)
>
> at
> org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:204)
>
> at
> org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:167)
>
> at
> org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:127)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.fireCompletion(ConsumerNetworkClient.java:575)
>
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.firePendingCompletedRequests(ConsumerNetworkClient.java:389)
>
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:297)
>
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236)
>
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:215)
>
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:235)
>
> at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:317)
>
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1226)
>
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1191)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1176)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:941)
>
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:850)
>
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:805)
>
> at
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:774){noformat}
>
>
>
> {noformat}
> [ERROR] [kafka-producer-network-thread |
> streams-example-e3628ed4-d2f5-44c2-a9cd-31de847589f6-StreamThread-1-0_8-producer]
> o.a.k.c.p.internals.Sender - [Producer
> clientId=streams-example-e3628ed4-d2f5-44c2-a9cd-31de847589f6-StreamThread-1-0_8-producer,
> transactionalId=streams-example-0_8] Uncaught error in kafka producer I/O
> thread:
> java.lang.IllegalStateException: No entry found for connection 2
> at
> org.apache.kafka.clients.ClusterConnectionStates.nodeState(ClusterConnectionStates.java:339)
>
> at
> org.apache.kafka.clients.ClusterConnectionStates.disconnected(ClusterConnectionStates.java:143)
>
> at
> org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:926)
>
> at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:287)
> at
> org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:65)
> at
> org.apache.kafka.clients.producer.internals.Sender.maybeSendTransactionalRequest(Sender.java:421)
>
> at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:286)
> at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
> at java.lang.Thread.run(Thread.java:748){noformat}
>
>
>
> {noformat}
> [WARN] [streams-example-e3628ed4-d2f5-44c2-a9cd-31de847589f6-StreamThread-1]
> o.a.kafka.clients.ClientUtils - Couldn't resolve server hostname1:port1 from
> bootstrap.servers as DNS resolution failed for hostname1{noformat}
>
>
> {noformat}
> [INFO] [kafka-coordinator-heartbeat-thread | streams-example]
> o.a.k.c.c.i.AbstractCoordinator - [Consumer
> clientId=streams-example-9213da8c-22ad-4116-82bd-47abf80bbf15-StreamThread-3-consumer,
> groupId=streams-example] Group coordinator hostname1:port1 (id: 2147438646
> rack: null) is unavailable or invalid, will attempt rediscovery
> [INFO] [kafka-coordinator-heartbeat-thread | streams-example]
> o.a.k.c.c.i.AbstractCoordinator - [Consumer
> clientId=streams-example-9213da8c-22ad-4116-82bd-47abf80bbf15-StreamThread-3-consumer,
> groupId=streams-example] Discovered group coordinator hostname1:port1 (id:
> 2147438646 rack: null)
> [INFO] [kafka-coordinator-heartbeat-thread | streams-example]
> o.a.k.c.c.i.AbstractCoordinator - [Consumer
> clientId=streams-example-9213da8c-22ad-4116-82bd-47abf80bbf15-StreamThread-3-consumer,
> groupId=streams-example] Group coordinator hostname1:port1 (id: 2147438646
> rack: null) is unavailable or invalid, will attempt rediscovery
> [INFO] [kafka-coordinator-heartbeat-thread | streams-example]
> o.a.k.c.c.i.AbstractCoordinator - [Consumer
> clientId=streams-example-9213da8c-22ad-4116-82bd-47abf80bbf15-StreamThread-3-consumer,
> groupId=streams-example] Discovered group coordinator hostname1:port1 (id:
> 2147438646 rack: null){noformat}
>
>
> {noformat}
> [INFO] [kafka-coordinator-heartbeat-thread | streams-example]
> o.a.k.c.c.i.AbstractCoordinator - [Consumer
> clientId=streams-example-9213da8c-22ad-4116-82bd-47abf80bbf15-StreamThread-2-consumer,
> groupId=streams-example] Attempt to heartbeat failed since group is
> rebalancing{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
