Hi All,

Any pointers to the above issue that I can explore further.

-Sameer.

On Tue, Dec 12, 2017 at 5:40 PM, Sameer Kumar <sam.kum.w...@gmail.com>
wrote:

> HI Ismael,
>
> This is what I see in the logs, I tried this twice and got the same
> exception.
>
> This is only reproducible when I have 3 nodes in my cluster even in the
> start of the app, everything works fine on 2 nodes.
>
> [2017-12-12 17:28:36,822] INFO [GroupCoordinator 90]: Preparing to
> rebalance group c-7-aq32 with old generation 25 (__consumer_offsets-39)
> (kafka.coordinator.group.GroupCoordinator)
> [2017-12-12 17:28:40,500] INFO [GroupCoordinator 90]: Stabilized group
> c-7-aq32 generation 26 (__consumer_offsets-39) (kafka.coordinator.group.
> GroupCoordinator)
> [2017-12-12 17:28:42,290] INFO [GroupCoordinator 90]: Assignment received
> from leader for group c-7-aq32 for generation 26 (kafka.coordinator.group.
> GroupCoordinator)
> *[2017-12-12 17:28:42,300] ERROR [GroupMetadataManager brokerId=90]
> Appending metadata message for group c-7-aq32 generation 26 failed due to
> org.apache.kafka.common.errors.RecordTooLargeException, returning UNKNOWN
> error code to the client (kafka.coordinator.group.GroupMetadataManager)*
> [2017-12-12 17:28:42,301] INFO [GroupCoordinator 90]: Preparing to
> rebalance group c-7-aq32 with old generation 26 (__consumer_offsets-39)
> (kafka.coordinator.group.GroupCoordinator)
> [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member
> c-7-aq32-6138ec53-4aff-4596-8f4b-44ae6f5d72da-StreamThread-13-consumer-
> e0cc0931-0619-4908-82c2-28f7bf9bace9 in group c-7-aq32 has failed,
> removing it from the group (kafka.coordinator.group.GroupCoordinator)
> [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member
> c-7-aq32-6c5469ff-501c-4bae-aa3e-2a4b8fff9949-StreamThread-12-consumer-
> b3435055-3e95-47e3-b54d-1aff2b668a8e in group c-7-aq32 has failed,
> removing it from the group (kafka.coordinator.group.GroupCoordinator)
> [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member
> c-7-aq32-e78df498-16ea-4be0-9c39-7851cf90682b-StreamThread-8-consumer-
> 1385f1ce-cd8e-411c-acdf-6e51e0ae889e in group c-7-aq32 has failed,
> removing it from the group (kafka.coordinator.group.GroupCoordinator)
> [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member
> c-7-aq32-6138ec53-4aff-4596-8f4b-44ae6f5d72da-StreamThread-2-consumer-
> be3a159b-bb84-405a-954d-1f36ee22abe6 in group c-7-aq32 has failed,
> removing it from the group (kafka.coordinator.group.GroupCoordinator)
> [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member
> c-7-aq32-e78df498-16ea-4be0-9c39-7851cf90682b-StreamThread-5-consumer-
> f0b62435-a2db-48f0-aae9-a36c0a01ab41 in group c-7-aq32 has failed,
> removing it from the group (kafka.coordinator.group.GroupCoordinator)
> [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member
> c-7-aq32-6138ec53-4aff-4596-8f4b-44ae6f5d72da-StreamThread-4-consumer-
> a36855f9-e015-444b-a5f9-684a59ada6c1 in group c-7-aq32 has failed,
> removing it from the group (kafka.coordinator.group.GroupCoordinator)
> [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member
> c-7-aq32-6138ec53-4aff-4596-8f4b-44ae6f5d72da-StreamThread-5-consumer-
> 6b8a4555-6ec4-409b-8437-98711af5ad69 in group c-7-aq32 has failed,
> removing it from the group (kafka.coordinator.group.GroupCoordinator)
> [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member
> c-7-aq32-6c5469ff-501c-4bae-aa3e-2a4b8fff9949-StreamThread-2-consumer-
> d62b0d68-9802-4d3c-9316-64ac5e4b755c in group c-7-aq32 has failed,
> removing it from the group (kafka.coordinator.group.GroupCoordinator)
> [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member
> c-7-aq32-6c5469ff-501c-4bae-aa3e-2a4b8fff9949-StreamThread-5-consumer-
> c608b36e-6e83-49bd-9f75-7de86ca83bdd in group c-7-aq32 has failed,
> removing it from the group (kafka.coordinator.group.GroupCoordinator)
> [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member
> c-7-aq32-6c5469ff-501c-4bae-aa3e-2a4b8fff9949-StreamThread-10-consumer-
> 9ba5e1e2-0478-4acd-b08d-ff37fb6f4e96 in group c-7-aq32 has failed,
> removing it from the group (kafka.coordinator.group.GroupCoordinator)
> [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member
> c-7-aq32-6138ec53-4aff-4596-8f4b-44ae6f5d72da-StreamThread-1-consumer-
> 10e57922-235c-454d-ba7d-d3d3737f2465 in group c-7-aq32 has failed,
> removing it from the group (kafka.coordinator.group.GroupCoordinator)
> [2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member
> c-7-aq32-6138ec53-4aff-4596-8f4b-44ae6f5d72da-StreamThread-8-consumer-
> 6befe815-3f9e-4d40-a2b1-e6f7ccc28a39 in group c-7-aq32 has failed,
> removing it from the group (kafka.coordinator.group.GroupCoordinator)
> [2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member
> c-7-aq32-6c5469ff-501c-4bae-aa3e-2a4b8fff9949-StreamThread-14-consumer-
> 58ae1271-5e3e-4ed6-a259-25c989faa9e0 in group c-7-aq32 has failed,
> removing it from the group (kafka.coordinator.group.GroupCoordinator)
> [2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member
> c-7-aq32-6c5469ff-501c-4bae-aa3e-2a4b8fff9949-StreamThread-9-consumer-
> f5c39812-6843-4d31-9755-4ff39b4f6c7d in group c-7-aq32 has failed,
> removing it from the group (kafka.coordinator.group.GroupCoordinator)
> [2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member
> c-7-aq32-6c5469ff-501c-4bae-aa3e-2a4b8fff9949-StreamThread-3-consumer-
> 7a97d164-dc77-4071-8a88-711fee624233 in group c-7-aq32 has failed,
> removing it from the group (kafka.coordinator.group.GroupCoordinator)
> [2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member
> c-7-aq32-6138ec53-4aff-4596-8f4b-44ae6f5d72da-StreamThread-12-consumer-
> e24b7bc8-000e-4ce5-9e2e-603832ffbeed in group c-7-aq32 has failed,
> removing it from the group (kafka.coordinator.group.GroupCoordinator)
> [2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member
> c-7-aq32-e78df498-16ea-4be0-9c39-7851cf90682b-StreamThread-14-consumer-
> f41cff13-5346-431b-b8f7-209d5df18651 in group c-7-aq32 has failed,
> removing it from the group (kafka.coordinator.group.GroupCoordinator)
> [2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member
> c-7-aq32-e78df498-16ea-4be0-9c39-7851cf90682b-StreamThread-4-consumer-
> f590b670-c43a-423d-a3d7-b36765d9ed57 in group c-7-aq32 has failed,
> removing it from the group (kafka.coordinator.group.GroupCoordinator)
> [2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member
> c-7-aq32-e78df498-16ea-4be0-9c39-7851cf90682b-StreamThread-1-consumer-
> fc145893-a22e-4338-a4a2-7b46370060f6 in group c-7-aq32 has failed,
> removing it from the group (kafka.coordinator.group.GroupCoordinator)
> [2017-
>
> On Tue, Dec 12, 2017 at 4:07 PM, Ismael Juma <ism...@juma.me.uk> wrote:
>
>> Can you please check the broker logs for errors?
>>
>> Ismael
>>
>> On Tue, Dec 12, 2017 at 12:10 PM, Sameer Kumar <sam.kum.w...@gmail.com>
>> wrote:
>>
>> > Hi All,
>> >
>> > Facing an strange exception while running Kafka Streams. I am reading
>> from
>> > a topic of 60 partitions. I am using exactly once in Kafka 1.0.0.
>> >
>> > Now, this error has started appearing recently. The application runs
>> fine
>> > on 2 nodes, but as soon as a 3rd node is added, it starts throwing
>> > exception. Please find the stacktrace attached.
>> >
>> > 2017-12-12 15:32:55 ERROR Kafka010Base:47 - Exception caught in thread
>> > c-7-aq32-5648256f-9142-49e2-98c0-da792e6da48e-StreamThread-5
>> > org.apache.kafka.common.KafkaException: Unexpected error from
>> SyncGroup:
>> > The server experienced an unexpected error when processing the request
>> >         at org.apache.kafka.clients.consumer.internals.
>> > AbstractCoordinator$SyncGroupResponseHandler.handle(
>> AbstractCoordinator.
>> > java:566)
>> >         at org.apache.kafka.clients.consumer.internals.
>> > AbstractCoordinator$SyncGroupResponseHandler.handle(
>> AbstractCoordinator.
>> > java:539)
>> >         at org.apache.kafka.clients.consumer.internals.
>> > AbstractCoordinator$CoordinatorResponseHandler.
>> > onSuccess(AbstractCoordinator.java:808)
>> >         at org.apache.kafka.clients.consumer.internals.
>> > AbstractCoordinator$CoordinatorResponseHandler.
>> > onSuccess(AbstractCoordinator.java:788)
>> >         at org.apache.kafka.clients.consumer.internals.
>> > RequestFuture$1.onSuccess(RequestFuture.java:204)
>> >         at org.apache.kafka.clients.consumer.internals.
>> > RequestFuture.fireSuccess(RequestFuture.java:167)
>> >         at org.apache.kafka.clients.consumer.internals.
>> > RequestFuture.complete(RequestFuture.java:127)
>> >         at org.apache.kafka.clients.consumer.internals.
>> > ConsumerNetworkClient$RequestFutureCompletionHandler.fireCompletion(
>> > ConsumerNetworkClient.java:506)
>> >         at org.apache.kafka.clients.consumer.internals.
>> > ConsumerNetworkClient.firePendingCompletedRequests(
>> > ConsumerNetworkClient.java:353)
>> >         at org.apache.kafka.clients.consumer.internals.
>> > ConsumerNetworkClient.poll(ConsumerNetworkClient.java:268)
>> >         at org.apache.kafka.clients.consumer.internals.
>> > ConsumerNetworkClient.poll(ConsumerNetworkClient.java:214)
>> >         at org.apache.kafka.clients.consumer.internals.
>> > ConsumerNetworkClient.poll(ConsumerNetworkClient.java:174)
>> >         at org.apache.kafka.clients.consumer.internals.
>> > AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:364)
>> >         at org.apache.kafka.clients.consumer.internals.
>> > AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:316)
>> >         at org.apache.kafka.clients.consumer.internals.
>> > ConsumerCoordinator.poll(ConsumerCoordinator.java:295)
>> >         at org.apache.kafka.clients.consumer.KafkaConsumer.
>> > pollOnce(KafkaConsumer.java:1138)
>> >         at org.apache.kafka.clients.consumer.KafkaConsumer.poll(
>> > KafkaConsumer.java:1103)
>> >         at org.apache.kafka.streams.processor.internals.
>> > StreamThread.pollRequests(StreamThread.java:851)
>> >         at org.apache.kafka.streams.processor.internals.
>> > StreamThread.runOnce(StreamThread.java:808)
>> >         at org.apache.kafka.streams.processor.internals.
>> > StreamThread.runLoop(StreamThread.java:774)
>> >         at org.apache.kafka.streams.processor.internals.
>> > StreamThread.run(StreamThread.java:744)
>> >
>> >
>> >
>> > -Sameer.
>> >
>>
>
>

Reply via email to