Hi All, Any pointers to the above issue that I can explore further.
-Sameer. On Tue, Dec 12, 2017 at 5:40 PM, Sameer Kumar <sam.kum.w...@gmail.com> wrote: > HI Ismael, > > This is what I see in the logs, I tried this twice and got the same > exception. > > This is only reproducible when I have 3 nodes in my cluster even in the > start of the app, everything works fine on 2 nodes. > > [2017-12-12 17:28:36,822] INFO [GroupCoordinator 90]: Preparing to > rebalance group c-7-aq32 with old generation 25 (__consumer_offsets-39) > (kafka.coordinator.group.GroupCoordinator) > [2017-12-12 17:28:40,500] INFO [GroupCoordinator 90]: Stabilized group > c-7-aq32 generation 26 (__consumer_offsets-39) (kafka.coordinator.group. > GroupCoordinator) > [2017-12-12 17:28:42,290] INFO [GroupCoordinator 90]: Assignment received > from leader for group c-7-aq32 for generation 26 (kafka.coordinator.group. > GroupCoordinator) > *[2017-12-12 17:28:42,300] ERROR [GroupMetadataManager brokerId=90] > Appending metadata message for group c-7-aq32 generation 26 failed due to > org.apache.kafka.common.errors.RecordTooLargeException, returning UNKNOWN > error code to the client (kafka.coordinator.group.GroupMetadataManager)* > [2017-12-12 17:28:42,301] INFO [GroupCoordinator 90]: Preparing to > rebalance group c-7-aq32 with old generation 26 (__consumer_offsets-39) > (kafka.coordinator.group.GroupCoordinator) > [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member > c-7-aq32-6138ec53-4aff-4596-8f4b-44ae6f5d72da-StreamThread-13-consumer- > e0cc0931-0619-4908-82c2-28f7bf9bace9 in group c-7-aq32 has failed, > removing it from the group (kafka.coordinator.group.GroupCoordinator) > [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member > c-7-aq32-6c5469ff-501c-4bae-aa3e-2a4b8fff9949-StreamThread-12-consumer- > b3435055-3e95-47e3-b54d-1aff2b668a8e in group c-7-aq32 has failed, > removing it from the group (kafka.coordinator.group.GroupCoordinator) > [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member > c-7-aq32-e78df498-16ea-4be0-9c39-7851cf90682b-StreamThread-8-consumer- > 1385f1ce-cd8e-411c-acdf-6e51e0ae889e in group c-7-aq32 has failed, > removing it from the group (kafka.coordinator.group.GroupCoordinator) > [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member > c-7-aq32-6138ec53-4aff-4596-8f4b-44ae6f5d72da-StreamThread-2-consumer- > be3a159b-bb84-405a-954d-1f36ee22abe6 in group c-7-aq32 has failed, > removing it from the group (kafka.coordinator.group.GroupCoordinator) > [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member > c-7-aq32-e78df498-16ea-4be0-9c39-7851cf90682b-StreamThread-5-consumer- > f0b62435-a2db-48f0-aae9-a36c0a01ab41 in group c-7-aq32 has failed, > removing it from the group (kafka.coordinator.group.GroupCoordinator) > [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member > c-7-aq32-6138ec53-4aff-4596-8f4b-44ae6f5d72da-StreamThread-4-consumer- > a36855f9-e015-444b-a5f9-684a59ada6c1 in group c-7-aq32 has failed, > removing it from the group (kafka.coordinator.group.GroupCoordinator) > [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member > c-7-aq32-6138ec53-4aff-4596-8f4b-44ae6f5d72da-StreamThread-5-consumer- > 6b8a4555-6ec4-409b-8437-98711af5ad69 in group c-7-aq32 has failed, > removing it from the group (kafka.coordinator.group.GroupCoordinator) > [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member > c-7-aq32-6c5469ff-501c-4bae-aa3e-2a4b8fff9949-StreamThread-2-consumer- > d62b0d68-9802-4d3c-9316-64ac5e4b755c in group c-7-aq32 has failed, > removing it from the group (kafka.coordinator.group.GroupCoordinator) > [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member > c-7-aq32-6c5469ff-501c-4bae-aa3e-2a4b8fff9949-StreamThread-5-consumer- > c608b36e-6e83-49bd-9f75-7de86ca83bdd in group c-7-aq32 has failed, > removing it from the group (kafka.coordinator.group.GroupCoordinator) > [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member > c-7-aq32-6c5469ff-501c-4bae-aa3e-2a4b8fff9949-StreamThread-10-consumer- > 9ba5e1e2-0478-4acd-b08d-ff37fb6f4e96 in group c-7-aq32 has failed, > removing it from the group (kafka.coordinator.group.GroupCoordinator) > [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member > c-7-aq32-6138ec53-4aff-4596-8f4b-44ae6f5d72da-StreamThread-1-consumer- > 10e57922-235c-454d-ba7d-d3d3737f2465 in group c-7-aq32 has failed, > removing it from the group (kafka.coordinator.group.GroupCoordinator) > [2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member > c-7-aq32-6138ec53-4aff-4596-8f4b-44ae6f5d72da-StreamThread-8-consumer- > 6befe815-3f9e-4d40-a2b1-e6f7ccc28a39 in group c-7-aq32 has failed, > removing it from the group (kafka.coordinator.group.GroupCoordinator) > [2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member > c-7-aq32-6c5469ff-501c-4bae-aa3e-2a4b8fff9949-StreamThread-14-consumer- > 58ae1271-5e3e-4ed6-a259-25c989faa9e0 in group c-7-aq32 has failed, > removing it from the group (kafka.coordinator.group.GroupCoordinator) > [2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member > c-7-aq32-6c5469ff-501c-4bae-aa3e-2a4b8fff9949-StreamThread-9-consumer- > f5c39812-6843-4d31-9755-4ff39b4f6c7d in group c-7-aq32 has failed, > removing it from the group (kafka.coordinator.group.GroupCoordinator) > [2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member > c-7-aq32-6c5469ff-501c-4bae-aa3e-2a4b8fff9949-StreamThread-3-consumer- > 7a97d164-dc77-4071-8a88-711fee624233 in group c-7-aq32 has failed, > removing it from the group (kafka.coordinator.group.GroupCoordinator) > [2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member > c-7-aq32-6138ec53-4aff-4596-8f4b-44ae6f5d72da-StreamThread-12-consumer- > e24b7bc8-000e-4ce5-9e2e-603832ffbeed in group c-7-aq32 has failed, > removing it from the group (kafka.coordinator.group.GroupCoordinator) > [2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member > c-7-aq32-e78df498-16ea-4be0-9c39-7851cf90682b-StreamThread-14-consumer- > f41cff13-5346-431b-b8f7-209d5df18651 in group c-7-aq32 has failed, > removing it from the group (kafka.coordinator.group.GroupCoordinator) > [2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member > c-7-aq32-e78df498-16ea-4be0-9c39-7851cf90682b-StreamThread-4-consumer- > f590b670-c43a-423d-a3d7-b36765d9ed57 in group c-7-aq32 has failed, > removing it from the group (kafka.coordinator.group.GroupCoordinator) > [2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member > c-7-aq32-e78df498-16ea-4be0-9c39-7851cf90682b-StreamThread-1-consumer- > fc145893-a22e-4338-a4a2-7b46370060f6 in group c-7-aq32 has failed, > removing it from the group (kafka.coordinator.group.GroupCoordinator) > [2017- > > On Tue, Dec 12, 2017 at 4:07 PM, Ismael Juma <ism...@juma.me.uk> wrote: > >> Can you please check the broker logs for errors? >> >> Ismael >> >> On Tue, Dec 12, 2017 at 12:10 PM, Sameer Kumar <sam.kum.w...@gmail.com> >> wrote: >> >> > Hi All, >> > >> > Facing an strange exception while running Kafka Streams. I am reading >> from >> > a topic of 60 partitions. I am using exactly once in Kafka 1.0.0. >> > >> > Now, this error has started appearing recently. The application runs >> fine >> > on 2 nodes, but as soon as a 3rd node is added, it starts throwing >> > exception. Please find the stacktrace attached. >> > >> > 2017-12-12 15:32:55 ERROR Kafka010Base:47 - Exception caught in thread >> > c-7-aq32-5648256f-9142-49e2-98c0-da792e6da48e-StreamThread-5 >> > org.apache.kafka.common.KafkaException: Unexpected error from >> SyncGroup: >> > The server experienced an unexpected error when processing the request >> > at org.apache.kafka.clients.consumer.internals. >> > AbstractCoordinator$SyncGroupResponseHandler.handle( >> AbstractCoordinator. >> > java:566) >> > at org.apache.kafka.clients.consumer.internals. >> > AbstractCoordinator$SyncGroupResponseHandler.handle( >> AbstractCoordinator. >> > java:539) >> > at org.apache.kafka.clients.consumer.internals. >> > AbstractCoordinator$CoordinatorResponseHandler. >> > onSuccess(AbstractCoordinator.java:808) >> > at org.apache.kafka.clients.consumer.internals. >> > AbstractCoordinator$CoordinatorResponseHandler. >> > onSuccess(AbstractCoordinator.java:788) >> > at org.apache.kafka.clients.consumer.internals. >> > RequestFuture$1.onSuccess(RequestFuture.java:204) >> > at org.apache.kafka.clients.consumer.internals. >> > RequestFuture.fireSuccess(RequestFuture.java:167) >> > at org.apache.kafka.clients.consumer.internals. >> > RequestFuture.complete(RequestFuture.java:127) >> > at org.apache.kafka.clients.consumer.internals. >> > ConsumerNetworkClient$RequestFutureCompletionHandler.fireCompletion( >> > ConsumerNetworkClient.java:506) >> > at org.apache.kafka.clients.consumer.internals. >> > ConsumerNetworkClient.firePendingCompletedRequests( >> > ConsumerNetworkClient.java:353) >> > at org.apache.kafka.clients.consumer.internals. >> > ConsumerNetworkClient.poll(ConsumerNetworkClient.java:268) >> > at org.apache.kafka.clients.consumer.internals. >> > ConsumerNetworkClient.poll(ConsumerNetworkClient.java:214) >> > at org.apache.kafka.clients.consumer.internals. >> > ConsumerNetworkClient.poll(ConsumerNetworkClient.java:174) >> > at org.apache.kafka.clients.consumer.internals. >> > AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:364) >> > at org.apache.kafka.clients.consumer.internals. >> > AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:316) >> > at org.apache.kafka.clients.consumer.internals. >> > ConsumerCoordinator.poll(ConsumerCoordinator.java:295) >> > at org.apache.kafka.clients.consumer.KafkaConsumer. >> > pollOnce(KafkaConsumer.java:1138) >> > at org.apache.kafka.clients.consumer.KafkaConsumer.poll( >> > KafkaConsumer.java:1103) >> > at org.apache.kafka.streams.processor.internals. >> > StreamThread.pollRequests(StreamThread.java:851) >> > at org.apache.kafka.streams.processor.internals. >> > StreamThread.runOnce(StreamThread.java:808) >> > at org.apache.kafka.streams.processor.internals. >> > StreamThread.runLoop(StreamThread.java:774) >> > at org.apache.kafka.streams.processor.internals. >> > StreamThread.run(StreamThread.java:744) >> > >> > >> > >> > -Sameer. >> > >> > >
