Can someone help me on this. Thanks, Tony
On Fri, Jul 9, 2021 at 8:15 PM Tony John <tonyjohnant...@gmail.com> wrote: > Hi All, > > I am trying to upgrade my Kafka streams application to 2.7.1 version of > Kafka. The brokers are upgraded to 2.7.1 and kafka dependencies are also on > 2.7.1. But when I start the application, rebalance is failing with the > following message > > Rebalance failed. org.apache.kafka.common.errors.DisconnectException > > I am also seeing Group coordinator broker102:9092 (id: 2147483645 rack: > null) is unavailable or invalid due to cause: coordinator > unavailable.isDisconnected: false. Rediscovery will be attempted. > > The full set of logs (which gets printed every 30 seconds) is given below > > INFO 2021-07-09 09:33:20.229 | internals.AbstractCoordinator [Consumer > clientId=consumer-my-app-v1-task-coordinator-consumer-app-node100-9, > groupId=my-app-v1-task-coordinator-consumer-app-node100] Group coordinator > broker102:9092 (id: 2147483645 rack: null) is unavailable or invalid due to > cause: null.isDisconnected: true. Rediscovery will be attempted. > INFO 2021-07-09 09:33:20.230 | internals.AbstractCoordinator [Consumer > clientId=consumer-my-app-v1-task-coordinator-consumer-app-node100-9, > groupId=my-app-v1-task-coordinator-consumer-app-node100] Discovered group > coordinator broker102:9092 (id: 2147483645 rack: null) > INFO 2021-07-09 09:33:20.230 | internals.AbstractCoordinator [Consumer > clientId=consumer-my-app-v1-task-coordinator-consumer-app-node100-9, > groupId=my-app-v1-task-coordinator-consumer-app-node100] Group coordinator > broker102:9092 (id: 2147483645 rack: null) is unavailable or invalid due to > cause: coordinator unavailable.isDisconnected: false. Rediscovery will be > attempted. > INFO 2021-07-09 09:33:20.330 | internals.AbstractCoordinator [Consumer > clientId=consumer-my-app-v1-task-coordinator-consumer-app-node100-9, > groupId=my-app-v1-task-coordinator-consumer-app-node100] Discovered group > coordinator broker102:9092 (id: 2147483645 rack: null) > INFO 2021-07-09 09:33:20.331 | internals.AbstractCoordinator [Consumer > clientId=consumer-my-app-v1-task-coordinator-consumer-app-node100-9, > groupId=my-app-v1-task-coordinator-consumer-app-node100] Rebalance failed. > org.apache.kafka.common.errors.DisconnectException > INFO 2021-07-09 09:33:20.331 | internals.AbstractCoordinator [Consumer > clientId=consumer-my-app-v1-task-coordinator-consumer-app-node100-9, > groupId=my-app-v1-task-coordinator-consumer-app-node100] (Re-)joining group > INFO 2021-07-09 09:33:20.333 | internals.AbstractCoordinator [Consumer > clientId=consumer-my-app-v1-task-coordinator-consumer-app-node100-9, > groupId=my-app-v1-task-coordinator-consumer-app-node100] (Re-)joining group > INFO 2021-07-09 09:33:20.419 | internals.AbstractCoordinator [Consumer > clientId=consumer-my-app-v1-master-coordinator-consumer-8, > groupId=my-app-v1-master-coordinator-consumer] Group coordinator > broker101:9092 (id: 2147483646 rack: null) is unavailable or invalid due to > cause: null.isDisconnected: true. Rediscovery will be attempted. > INFO 2021-07-09 09:33:20.420 | internals.AbstractCoordinator [Consumer > clientId=consumer-my-app-v1-master-coordinator-consumer-8, > groupId=my-app-v1-master-coordinator-consumer] Discovered group coordinator > broker101:9092 (id: 2147483646 rack: null) > INFO 2021-07-09 09:33:20.420 | internals.AbstractCoordinator [Consumer > clientId=consumer-my-app-v1-master-coordinator-consumer-8, > groupId=my-app-v1-master-coordinator-consumer] Group coordinator > broker101:9092 (id: 2147483646 rack: null) is unavailable or invalid due to > cause: coordinator unavailable.isDisconnected: false. Rediscovery will be > attempted. > INFO 2021-07-09 09:33:20.521 | internals.AbstractCoordinator [Consumer > clientId=consumer-my-app-v1-master-coordinator-consumer-8, > groupId=my-app-v1-master-coordinator-consumer] Discovered group coordinator > broker101:9092 (id: 2147483646 rack: null) > INFO 2021-07-09 09:33:20.522 | internals.AbstractCoordinator [Consumer > clientId=consumer-my-app-v1-master-coordinator-consumer-8, > groupId=my-app-v1-master-coordinator-consumer] Rebalance failed. > org.apache.kafka.common.errors.DisconnectException > INFO 2021-07-09 09:33:20.523 | internals.AbstractCoordinator [Consumer > clientId=consumer-my-app-v1-master-coordinator-consumer-8, > groupId=my-app-v1-master-coordinator-consumer] (Re-)joining group > INFO 2021-07-09 09:33:20.524 | internals.AbstractCoordinator [Consumer > clientId=consumer-my-app-v1-master-coordinator-consumer-8, > groupId=my-app-v1-master-coordinator-consumer] (Re-)joining group > > The application was working fine on 2.5.1. Please note with 2.5.1 the > build used was kafka_*2.12*-2.5.1, but with 2.7.1 I used kafka_*2.13* > -2.7.1 > > *The broker config is * > > broker.id=2 > listeners=PLAINTEXT://broker102:9092 > advertised.listeners=PLAINTEXT://broker102:9092 > num.network.threads=4 > num.io.threads=4 > socket.send.buffer.bytes=102400 > socket.receive.buffer.bytes=102400 > socket.request.max.bytes=104857600 > log.dirs=/mnt/store/kafka/kafka-logs > num.partitions=1 > num.recovery.threads.per.data.dir=4 > offsets.topic.replication.factor=2 > transaction.state.log.replication.factor=2 > transaction.state.log.min.isr=2 > log.retention.hours=1 > log.segment.bytes=1073741824 > log.retention.check.interval.ms=300000 > > zookeeper.connect=broker100:2181,broker101:2181,broker102:2181,broker103:2181,broker104:2181 > zookeeper.connection.timeout.ms=30000 > group.initial.rebalance.delay.ms=120000 > offsets.retention.minutes=10080 > message.max.bytes=31457280 > replica.fetch.max.bytes=31457280 > group.max.session.timeout.ms=1200000 > request.timeout.ms=600000 > connections.max.idle.ms=1080000 > > What could be wrong? Should I switch to kafka_*2.12*-2.7.1 ? > > Thanks, > Tony > >
