HI Ismael, This is what I see in the logs, I tried this twice and got the same exception.
This is only reproducible when I have 3 nodes in my cluster even in the start of the app, everything works fine on 2 nodes. [2017-12-12 17:28:36,822] INFO [GroupCoordinator 90]: Preparing to rebalance group c-7-aq32 with old generation 25 (__consumer_offsets-39) (kafka.coordinator.group.GroupCoordinator) [2017-12-12 17:28:40,500] INFO [GroupCoordinator 90]: Stabilized group c-7-aq32 generation 26 (__consumer_offsets-39) (kafka.coordinator.group.GroupCoordinator) [2017-12-12 17:28:42,290] INFO [GroupCoordinator 90]: Assignment received from leader for group c-7-aq32 for generation 26 (kafka.coordinator.group.GroupCoordinator) *[2017-12-12 17:28:42,300] ERROR [GroupMetadataManager brokerId=90] Appending metadata message for group c-7-aq32 generation 26 failed due to org.apache.kafka.common.errors.RecordTooLargeException, returning UNKNOWN error code to the client (kafka.coordinator.group.GroupMetadataManager)* [2017-12-12 17:28:42,301] INFO [GroupCoordinator 90]: Preparing to rebalance group c-7-aq32 with old generation 26 (__consumer_offsets-39) (kafka.coordinator.group.GroupCoordinator) [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member c-7-aq32-6138ec53-4aff-4596-8f4b-44ae6f5d72da-StreamThread-13-consumer-e0cc0931-0619-4908-82c2-28f7bf9bace9 in group c-7-aq32 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator) [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member c-7-aq32-6c5469ff-501c-4bae-aa3e-2a4b8fff9949-StreamThread-12-consumer-b3435055-3e95-47e3-b54d-1aff2b668a8e in group c-7-aq32 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator) [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member c-7-aq32-e78df498-16ea-4be0-9c39-7851cf90682b-StreamThread-8-consumer-1385f1ce-cd8e-411c-acdf-6e51e0ae889e in group c-7-aq32 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator) [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member c-7-aq32-6138ec53-4aff-4596-8f4b-44ae6f5d72da-StreamThread-2-consumer-be3a159b-bb84-405a-954d-1f36ee22abe6 in group c-7-aq32 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator) [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member c-7-aq32-e78df498-16ea-4be0-9c39-7851cf90682b-StreamThread-5-consumer-f0b62435-a2db-48f0-aae9-a36c0a01ab41 in group c-7-aq32 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator) [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member c-7-aq32-6138ec53-4aff-4596-8f4b-44ae6f5d72da-StreamThread-4-consumer-a36855f9-e015-444b-a5f9-684a59ada6c1 in group c-7-aq32 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator) [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member c-7-aq32-6138ec53-4aff-4596-8f4b-44ae6f5d72da-StreamThread-5-consumer-6b8a4555-6ec4-409b-8437-98711af5ad69 in group c-7-aq32 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator) [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member c-7-aq32-6c5469ff-501c-4bae-aa3e-2a4b8fff9949-StreamThread-2-consumer-d62b0d68-9802-4d3c-9316-64ac5e4b755c in group c-7-aq32 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator) [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member c-7-aq32-6c5469ff-501c-4bae-aa3e-2a4b8fff9949-StreamThread-5-consumer-c608b36e-6e83-49bd-9f75-7de86ca83bdd in group c-7-aq32 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator) [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member c-7-aq32-6c5469ff-501c-4bae-aa3e-2a4b8fff9949-StreamThread-10-consumer-9ba5e1e2-0478-4acd-b08d-ff37fb6f4e96 in group c-7-aq32 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator) [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member c-7-aq32-6138ec53-4aff-4596-8f4b-44ae6f5d72da-StreamThread-1-consumer-10e57922-235c-454d-ba7d-d3d3737f2465 in group c-7-aq32 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator) [2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member c-7-aq32-6138ec53-4aff-4596-8f4b-44ae6f5d72da-StreamThread-8-consumer-6befe815-3f9e-4d40-a2b1-e6f7ccc28a39 in group c-7-aq32 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator) [2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member c-7-aq32-6c5469ff-501c-4bae-aa3e-2a4b8fff9949-StreamThread-14-consumer-58ae1271-5e3e-4ed6-a259-25c989faa9e0 in group c-7-aq32 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator) [2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member c-7-aq32-6c5469ff-501c-4bae-aa3e-2a4b8fff9949-StreamThread-9-consumer-f5c39812-6843-4d31-9755-4ff39b4f6c7d in group c-7-aq32 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator) [2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member c-7-aq32-6c5469ff-501c-4bae-aa3e-2a4b8fff9949-StreamThread-3-consumer-7a97d164-dc77-4071-8a88-711fee624233 in group c-7-aq32 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator) [2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member c-7-aq32-6138ec53-4aff-4596-8f4b-44ae6f5d72da-StreamThread-12-consumer-e24b7bc8-000e-4ce5-9e2e-603832ffbeed in group c-7-aq32 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator) [2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member c-7-aq32-e78df498-16ea-4be0-9c39-7851cf90682b-StreamThread-14-consumer-f41cff13-5346-431b-b8f7-209d5df18651 in group c-7-aq32 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator) [2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member c-7-aq32-e78df498-16ea-4be0-9c39-7851cf90682b-StreamThread-4-consumer-f590b670-c43a-423d-a3d7-b36765d9ed57 in group c-7-aq32 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator) [2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member c-7-aq32-e78df498-16ea-4be0-9c39-7851cf90682b-StreamThread-1-consumer-fc145893-a22e-4338-a4a2-7b46370060f6 in group c-7-aq32 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator) [2017- On Tue, Dec 12, 2017 at 4:07 PM, Ismael Juma <ism...@juma.me.uk> wrote: > Can you please check the broker logs for errors? > > Ismael > > On Tue, Dec 12, 2017 at 12:10 PM, Sameer Kumar <sam.kum.w...@gmail.com> > wrote: > > > Hi All, > > > > Facing an strange exception while running Kafka Streams. I am reading > from > > a topic of 60 partitions. I am using exactly once in Kafka 1.0.0. > > > > Now, this error has started appearing recently. The application runs fine > > on 2 nodes, but as soon as a 3rd node is added, it starts throwing > > exception. Please find the stacktrace attached. > > > > 2017-12-12 15:32:55 ERROR Kafka010Base:47 - Exception caught in thread > > c-7-aq32-5648256f-9142-49e2-98c0-da792e6da48e-StreamThread-5 > > org.apache.kafka.common.KafkaException: Unexpected error from SyncGroup: > > The server experienced an unexpected error when processing the request > > at org.apache.kafka.clients.consumer.internals. > > AbstractCoordinator$SyncGroupResponseHandler.handle(AbstractCoordinator. > > java:566) > > at org.apache.kafka.clients.consumer.internals. > > AbstractCoordinator$SyncGroupResponseHandler.handle(AbstractCoordinator. > > java:539) > > at org.apache.kafka.clients.consumer.internals. > > AbstractCoordinator$CoordinatorResponseHandler. > > onSuccess(AbstractCoordinator.java:808) > > at org.apache.kafka.clients.consumer.internals. > > AbstractCoordinator$CoordinatorResponseHandler. > > onSuccess(AbstractCoordinator.java:788) > > at org.apache.kafka.clients.consumer.internals. > > RequestFuture$1.onSuccess(RequestFuture.java:204) > > at org.apache.kafka.clients.consumer.internals. > > RequestFuture.fireSuccess(RequestFuture.java:167) > > at org.apache.kafka.clients.consumer.internals. > > RequestFuture.complete(RequestFuture.java:127) > > at org.apache.kafka.clients.consumer.internals. > > ConsumerNetworkClient$RequestFutureCompletionHandler.fireCompletion( > > ConsumerNetworkClient.java:506) > > at org.apache.kafka.clients.consumer.internals. > > ConsumerNetworkClient.firePendingCompletedRequests( > > ConsumerNetworkClient.java:353) > > at org.apache.kafka.clients.consumer.internals. > > ConsumerNetworkClient.poll(ConsumerNetworkClient.java:268) > > at org.apache.kafka.clients.consumer.internals. > > ConsumerNetworkClient.poll(ConsumerNetworkClient.java:214) > > at org.apache.kafka.clients.consumer.internals. > > ConsumerNetworkClient.poll(ConsumerNetworkClient.java:174) > > at org.apache.kafka.clients.consumer.internals. > > AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:364) > > at org.apache.kafka.clients.consumer.internals. > > AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:316) > > at org.apache.kafka.clients.consumer.internals. > > ConsumerCoordinator.poll(ConsumerCoordinator.java:295) > > at org.apache.kafka.clients.consumer.KafkaConsumer. > > pollOnce(KafkaConsumer.java:1138) > > at org.apache.kafka.clients.consumer.KafkaConsumer.poll( > > KafkaConsumer.java:1103) > > at org.apache.kafka.streams.processor.internals. > > StreamThread.pollRequests(StreamThread.java:851) > > at org.apache.kafka.streams.processor.internals. > > StreamThread.runOnce(StreamThread.java:808) > > at org.apache.kafka.streams.processor.internals. > > StreamThread.runLoop(StreamThread.java:774) > > at org.apache.kafka.streams.processor.internals. > > StreamThread.run(StreamThread.java:744) > > > > > > > > -Sameer. > > >
