logs shows "failed due to org.apache.kafka.common.errors.RecordTooLargeException, returning UNKNOWN error code to the client (kafka.coordinator.group. GroupMetadataManager)"
Try increasing max.message.bytes config on broker and/or enable compression on the offsets topic via the broker config `offsets.topic.compression.codec`. On Tue, Dec 19, 2017 at 11:18 AM, Sameer Kumar <sam.kum.w...@gmail.com> wrote: > Hi All, > > I faced this exception yesterday, not sure why I got this error. Attached > are the kafka specific logs. > Mentioned below are logs on broker and application. > > On broker side, there are the logs: > [2017-12-19 03:02:44,054] INFO [GroupMetadataManager brokerId=90] Removed > 0 expired offsets in 0 milliseconds. (kafka.coordinator.group. > GroupMetadataManager) > [2017-12-19 03:05:23,682] INFO [GroupCoordinator 90]: Member > c-7-aq41-49e73818-e6fd-471d-af25-20310786a6e5-StreamThread-14-consumer- > e74f6be8-393e-4ec3-bb34-57badc0638a1 in group c-7-aq41 has failed, > removing it from the group (kafka.coordinator.group.GroupCoordinator) > [2017-12-19 03:05:23,690] INFO [GroupCoordinator 90]: Preparing to > rebalance group c-7-aq41 with old generation 9 (__consumer_offsets-9) > (kafka.coordinator.group.GroupCoordinator) > [2017-12-19 03:05:24,265] INFO [GroupCoordinator 90]: Member > c-7-aq41-49e73818-e6fd-471d-af25-20310786a6e5-StreamThread-13-consumer- > e31185e6-a0e8-4e04-96c3-6e219090cdee in group c-7-aq41 has failed, > removing it from the group (kafka.coordinator.group.GroupCoordinator) > [2017-12-19 03:05:24,285] INFO [GroupCoordinator 90]: Member > c-7-aq41-49e73818-e6fd-471d-af25-20310786a6e5-StreamThread-5-consumer- > ea825dbd-a2af-4fad-b35f-ed10f1f6345f in group c-7-aq41 has failed, > removing it from the group (kafka.coordinator.group.GroupCoordinator) > [2017-12-19 03:05:24,349] INFO [GroupCoordinator 90]: Member > c-7-aq41-49e73818-e6fd-471d-af25-20310786a6e5-StreamThread-6-consumer- > 631ac688-eac8-4940-ada0-7e22a9355ebd in group c-7-aq41 has failed, > removing it from the group (kafka.coordinator.group.GroupCoordinator) > [2017-12-19 03:05:24,681] INFO [GroupCoordinator 90]: Member > c-7-aq41-49e73818-e6fd-471d-af25-20310786a6e5-StreamThread-4-consumer- > 4d949b0e-4204-4aa7-a776-c170ff0bf9b2 in group c-7-aq41 has failed, > removing it from the group (kafka.coordinator.group.GroupCoordinator) > [2017-12-19 03:05:24,710] INFO [GroupCoordinator 90]: Member > c-7-aq41-49e73818-e6fd-471d-af25-20310786a6e5-StreamThread-3-consumer- > d7eadb21-cdd1-4f8c-8d08-ea9fdc59d809 in group c-7-aq41 has failed, > removing it from the group (kafka.coordinator.group.GroupCoordinator) > [2017-12-19 03:05:24,783] INFO [GroupCoordinator 90]: Member > c-7-aq41-49e73818-e6fd-471d-af25-20310786a6e5-StreamThread-1-consumer- > 770cd546-f14c-47db-9376-c107e8beba71 in group c-7-aq41 has failed, > removing it from the group (kafka.coordinator.group.GroupCoordinator) > [2017-12-19 03:05:24,839] INFO [GroupCoordinator 90]: Member > c-7-aq41-49e73818-e6fd-471d-af25-20310786a6e5-StreamThread-11-consumer- > 6b510fee-155b-40c5-8842-7450a45156d6 in group c-7-aq41 has failed, > removing it from the group (kafka.coordinator.group.GroupCoordinator) > [2017-12-19 03:05:32,070] INFO [GroupCoordinator 90]: Stabilized group > c-7-aq41 generation 10 (__consumer_offsets-9) (kafka.coordinator.group. > GroupCoordinator) > [2017-12-19 03:05:32,177] INFO [GroupCoordinator 90]: Preparing to > rebalance group c-7-aq41 with old generation 10 (__consumer_offsets-9) > (kafka.coordinator.group.GroupCoordinator) > [2017-12-19 03:05:42,070] INFO [GroupCoordinator 90]: Member > c-7-aq41-49e73818-e6fd-471d-af25-20310786a6e5-StreamThread-5-consumer- > 2db7dd39-1045-4732-9d08-9badd63ddf1d in group c-7-aq41 has failed, > removing it from the group (kafka.coordinator.group.GroupCoordinator) > [2017-12-19 03:05:42,070] INFO [GroupCoordinator 90]: Member > c-7-aq41-49e73818-e6fd-471d-af25-20310786a6e5-StreamThread-3-consumer- > 2cccd0d6-b0cb-44b9-a2aa-2cd68f148870 in group c-7-aq41 has failed, > removing it from the group (kafka.coordinator.group.GroupCoordinator) > [2017-12-19 03:05:42,071] INFO [GroupCoordinator 90]: Member > c-7-aq41-49e73818-e6fd-471d-af25-20310786a6e5-StreamThread-1-consumer- > 16b528ff-41b6-42b8-b222-bb84c5d2b7cb in group c-7-aq41 has failed, > removing it from the group (kafka.coordinator.group.GroupCoordinator) > [2017-12-19 03:05:42,071] INFO [GroupCoordinator 90]: Member > c-7-aq41-49e73818-e6fd-471d-af25-20310786a6e5-StreamThread-13-consumer- > d81ade49-0bc9-4374-80f4-d82495b6135a in group c-7-aq41 has failed, > removing it from the group (kafka.coordinator.group.GroupCoordinator) > [2017-12-19 03:05:42,071] INFO [GroupCoordinator 90]: Member > c-7-aq41-49e73818-e6fd-471d-af25-20310786a6e5-StreamThread-4-consumer- > 6167070d-5536-4ad6-b8c5-7fb2a62c465f in group c-7-aq41 has failed, > removing it from the group (kafka.coordinator.group.GroupCoordinator) > [2017-12-19 03:05:42,073] INFO [GroupCoordinator 90]: Stabilized group > c-7-aq41 generation 11 (__consumer_offsets-9) (kafka.coordinator.group. > GroupCoordinator) > [2017-12-19 03:05:43,445] INFO [GroupCoordinator 90]: Assignment received > from leader for group c-7-aq41 for generation 11 (kafka.coordinator.group. > GroupCoordinator) > [2017-12-19 03:05:43,448] ERROR [GroupMetadataManager brokerId=90] > Appending metadata message for group c-7-aq41 generation 11 failed due to > org.apache.kafka.common.errors.RecordTooLargeException, returning UNKNOWN > error code to the client (kafka.coordinator.group.GroupMetadataManager) > [2017-12-19 03:05:43,449] INFO [GroupCoordinator 90]: Preparing to > rebalance group c-7-aq41 with old generation 11 (__consumer_offsets-9) > (kafka.coordinator.group.GroupCoordinator) > > > > On stream application:- > 2017-12-19 03:05:45 ERROR Kafka010Base:47 - Exception caught in thread > c-7-aq41-49e73818-e6fd-471d-af25-20310786a6e5-StreamThread-6 > org.apache.kafka.common.KafkaException: Unexpected error from SyncGroup: > The server experienced an unexpected error when processing the request > at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$ > SyncGroupResponseHandler.handle(AbstractCoordinator.java:566) > at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$ > SyncGroupResponseHandler.handle(AbstractCoordinator.java:539) > at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$ > CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:808) > at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$ > CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:788) > at org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess( > RequestFuture.java:204) > at org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess( > RequestFuture.java:167) > at org.apache.kafka.clients.consumer.internals.RequestFuture.complete( > RequestFuture.java:127) > at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$ > RequestFutureCompletionHandler.fireCompletion(ConsumerNetworkClient.java: > 506) > at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient. > firePendingCompletedRequests(ConsumerNetworkClient.java:353) > at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll( > ConsumerNetworkClient.java:268) > at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll( > ConsumerNetworkClient.java:214) > at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll( > ConsumerNetworkClient.java:174) > at org.apache.kafka.clients.consumer.internals.AbstractCoordinator. > joinGroupIfNeeded(AbstractCoordinator.java:364) > at org.apache.kafka.clients.consumer.internals.AbstractCoordinator. > ensureActiveGroup(AbstractCoordinator.java:316) > at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll( > ConsumerCoordinator.java:295) > at org.apache.kafka.clients.consumer.KafkaConsumer. > pollOnce(KafkaConsumer.java:1138) > at org.apache.kafka.clients.consumer.KafkaConsumer.poll( > KafkaConsumer.java:1103) > at org.apache.kafka.streams.processor.internals.StreamThread.pollRequests( > StreamThread.java:851) > at org.apache.kafka.streams.processor.internals.StreamThread.runOnce( > StreamThread.java:808) > at org.apache.kafka.streams.processor.internals.StreamThread.runLoop( > StreamThread.java:774) > at org.apache.kafka.streams.processor.internals. > StreamThread.run(StreamThread.java:744) > > Any help would be appreciated. > > Thanks, > -Sameer. >
