I am facing too many problems these days. Now one of our consumer groups
is rebalancing every now and then. And rebalance takes very low, more than
5-10 minutes. Even after re-balancing I see that only half of the consumers
are active/receive assignment. Its all going haywire.
I am seeing these logs in kafka consumer logs. Can anyone help me
understand what is going on here? It is a very long piece of log, but
someone please help me. I am desperately looking for any solution since
more than 2 months now. But to no avail.
[2018-08-09 11:39:51] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:39:53] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25465113 for partition bulk-email-8
[2018-08-09 11:39:53] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:39:53] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25463566 for partition bulk-email-6
[2018-08-09 11:39:53] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:39:53] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25444488 for partition bulk-email-9
[2018-08-09 11:39:53] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:39:54] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:39:54] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:39:54] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:39:54] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:39:54] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:39:54] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:39:54] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:39:54] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:39:56] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25463566 for partition bulk-email-6
[2018-08-09 11:39:56] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25444488 for partition bulk-email-9
[2018-08-09 11:39:56] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25465113 for partition bulk-email-8
[2018-08-09 11:39:56] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:39:56] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:39:56] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:39:57] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:39:57] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:39:57] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:39:57] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:39:57] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:39:57] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:39:57] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:39:57] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:00] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:00] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:00] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:03] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:03] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:03] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:04] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:04] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:04] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:664 - Offset commit for
group bulk-email-consumer failed: The request timed out.
[2018-08-09 11:40:04] :: INFO :: AbstractCoordinator:600 - Marking the
coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) dead for group
bulk-email-consumer
[2018-08-09 11:40:04] :: WARN :: ConsumerCoordinator$4:535 - Auto offset
commit failed for group bulk-email-consumer: Offset commit failed with a
retriable exception. You should retry committing offsets.
[2018-08-09 11:40:04] :: DEBUG :: AbstractCoordinator:531 - Sending
coordinator request for group bulk-email-consumer to broker
10.189.179.117:9092 (id: 1 rack: null)
[2018-08-09 11:40:04] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:04] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:664 - Offset commit for
group bulk-email-consumer failed: The request timed out.
[2018-08-09 11:40:04] :: INFO :: AbstractCoordinator:600 - Marking the
coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) dead for group
bulk-email-consumer
[2018-08-09 11:40:04] :: WARN :: ConsumerCoordinator$4:535 - Auto offset
commit failed for group bulk-email-consumer: Offset commit failed with a
retriable exception. You should retry committing offsets.
[2018-08-09 11:40:04] :: DEBUG :: AbstractCoordinator:531 - Sending
coordinator request for group bulk-email-consumer to broker
10.189.179.118:9092 (id: 2 rack: null)
[2018-08-09 11:40:04] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:04] :: DEBUG ::
AbstractCoordinator$GroupCoordinatorResponseHandler:541 - Received group
coordinator response ClientResponse(receivedTimeMs=1533795004158,
disconnected=false, request=ClientRequest(expectResponse=true,
callback=org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler@3a3e5740,
request=RequestSend(header={api_key=10,api_version=0,correlation_id=389784,client_id=consumer-26},
body={group_id=bulk-email-consumer}), createdTimeMs=1533795004157,
sendTimeMs=1533795004157),
responseBody={error_code=0,coordinator={node_id=0,host=10.189.179.116,port=9092}})
[2018-08-09 11:40:04] :: INFO ::
AbstractCoordinator$GroupCoordinatorResponseHandler:555 - Discovered
coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) for group
bulk-emai -consumer.
[2018-08-09 11:40:04] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:04] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:04] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25463566 for partition bulk-email-6
[2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:40:04] :: DEBUG ::
AbstractCoordinator$GroupCoordinatorResponseHandler:541 - Received group
coordinator response ClientResponse(receivedTimeMs=1533795004718,
disconnected=false, request=ClientRequest(expectResponse=true,
callback=org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler@66889bbd,
request=RequestSend(header={api_key=10,api_version=0,correlation_id=363859,client_id=consumer-25},
body={group_id=bulk-email-consumer}), createdTimeMs=1533795004145,
sendTimeMs=1533795004145),
responseBody={error_code=0,coordinator={node_id=0,host=10.189.179.116,port=9092}})
[2018-08-09 11:40:04] :: INFO ::
AbstractCoordinator$GroupCoordinatorResponseHandler:555 - Discovered
coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) for group
bulk-emai -consumer.
[2018-08-09 11:40:04] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:04] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25444488 for partition bulk-email-9
[2018-08-09 11:40:04] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25465113 for partition bulk-email-8
[2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:40:04] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:04] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:04] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25463566 for partition bulk-email-6
[2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:40:04] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:04] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25465113 for partition bulk-email-8
[2018-08-09 11:40:04] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25444488 for partition bulk-email-9
[2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:40:05] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25463566 for partition bulk-email-6
[2018-08-09 11:40:05] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:40:06] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:06] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:06] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:06] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:06] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:06] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:07] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:07] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25444488 for partition bulk-email-9
[2018-08-09 11:40:07] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:40:07] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:07] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25465113 for partition bulk-email-8
[2018-08-09 11:40:07] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:40:08] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25463566 for partition bulk-email-6
[2018-08-09 11:40:08] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:40:09] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:09] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:09] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:09] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:09] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:09] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:10] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:10] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25444488 for partition bulk-email-9
[2018-08-09 11:40:10] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:40:10] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25465113 for partition bulk-email-8
[2018-08-09 11:40:10] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:40:10] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:11] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25463566 for partition bulk-email-6
[2018-08-09 11:40:11] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:40:12] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:12] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:12] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:12] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:12] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:12] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:13] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25444488 for partition bulk-email-9
[2018-08-09 11:40:13] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:40:13] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:14] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25465113 for partition bulk-email-8
[2018-08-09 11:40:14] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25463566 for partition bulk-email-6
[2018-08-09 11:40:14] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:40:14] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:40:14] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:15] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:15] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:15] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:15] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:15] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:15] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:16] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25444488 for partition bulk-email-9
[2018-08-09 11:40:16] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:40:16] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:16] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25465113 for partition bulk-email-8
[2018-08-09 11:40:16] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:40:16] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:17] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25463566 for partition bulk-email-6
[2018-08-09 11:40:17] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:40:18] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:18] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:18] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:18] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:18] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:18] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:19] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25444488 for partition bulk-email-9
[2018-08-09 11:40:19] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:40:19] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:19] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25465113 for partition bulk-email-8
[2018-08-09 11:40:19] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:40:19] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:20] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25463566 for partition bulk-email-6
[2018-08-09 11:40:20] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:40:21] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:21] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:21] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:21] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:21] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:21] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:21] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25463566 for partition bulk-email-6
[2018-08-09 11:40:21] :: INFO :: ConsumerCoordinator:333 - Revoking
previously assigned partitions [bulk-email-6] for group bulk-email-consumer
[2018-08-09 11:40:21] :: INFO :: AbstractCoordinator:381 - (Re-)joining
group bulk-email-consumer
[2018-08-09 11:40:21] :: DEBUG :: AbstractCoordinator:390 - Sending
JoinGroup
({group_id=bulk-email-consumer,session_timeout=300000,rebalance_timeout=300000,member_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,protocol_type=consumer,group_protocols=[{protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id: 2147483647 rack:
null)
[2018-08-09 11:40:22] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25444488 for partition bulk-email-9
[2018-08-09 11:40:22] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:40:22] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25465113 for partition bulk-email-8
[2018-08-09 11:40:22] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:40:22] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:22] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:22] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25444488 for partition bulk-email-9
[2018-08-09 11:40:22] :: INFO :: ConsumerCoordinator:333 - Revoking
previously assigned partitions [bulk-email-9] for group bulk-email-consumer
[2018-08-09 11:40:22] :: INFO :: AbstractCoordinator:381 - (Re-)joining
group bulk-email-consumer
[2018-08-09 11:40:22] :: DEBUG :: AbstractCoordinator:390 - Sending
JoinGroup
({group_id=bulk-email-consumer,session_timeout=300000,rebalance_timeout=300000,member_id=consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09,protocol_type=consumer,group_protocols=[{protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id: 2147483647 rack:
null)
[2018-08-09 11:40:22] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25465113 for partition bulk-email-8
[2018-08-09 11:40:23] :: INFO :: ConsumerCoordinator:333 - Revoking
previously assigned partitions [bulk-email-8] for group bulk-email-consumer
[2018-08-09 11:40:23] :: INFO :: AbstractCoordinator:381 - (Re-)joining
group bulk-email-consumer
[2018-08-09 11:40:23] :: DEBUG :: AbstractCoordinator:390 - Sending
JoinGroup
({group_id=bulk-email-consumer,session_timeout=300000,rebalance_timeout=300000,member_id=consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa,protocol_type=consumer,group_protocols=[{protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id: 2147483647 rack:
null)
[2018-08-09 11:40:24] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:24] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:24] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:24] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:24] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:26] :: DEBUG :: NetworkClient$DefaultMetadataUpdater:640
- Sending metadata request {topics=[bulk-email]} to node 1
[2018-08-09 11:40:26] :: DEBUG :: Metadata :241 - Updated cluster metadata
version 642 to Cluster(id = 09FjKmqPQvat0c0dyPptIw, nodes = [
10.189.179.118:9092 (id: 2 rack: null), 10.189.179.116:9092 (id: 0 rack:
null), 10.189.179.117:9092 (id: 1 rack: null)], partitions =
[Partition(topic = bulk-email, partition = 8, leader = 2, replicas =
[0,1,2,], isr = [0,2,1,]), Partition(topic = bulk-email, partition = 7,
leader = 1, replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic =
bulk-email, partition = 9, leader = 0, replicas = [0,1,2,], isr =
[0,1,2,]), Partition(topic = bulk-email, partition = 4, leader = 1,
replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic = bulk-email,
partition = 3, leader = 0, replicas = [0,1,2,], isr = [0,1,2,]),
Partition(topic = bulk-email, partition = 6, leader = 0, replicas =
[0,1,2,], isr = [0,1,2,]), Partition(topic = bulk-email, partition = 5,
leader = 2, replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic =
bulk-email, partition = 0, leader = 0, replicas = [0,1,2,], isr =
[0,1,2,]), Partition(topic = bulk-email, partition = 2, leader = 2,
replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic = bulk-email,
partition = 1, leader = 1, replicas = [0,1,2,], isr = [1,0,2,])])
[2018-08-09 11:40:27] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:27] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:27] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:27] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:27] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:28] :: DEBUG :: NetworkClient$DefaultMetadataUpdater:640
- Sending metadata request {topics=[bulk-email]} to node 1
[2018-08-09 11:40:28] :: DEBUG :: Metadata :241 - Updated cluster metadata
version 641 to Cluster(id = 09FjKmqPQvat0c0dyPptIw, nodes = [
10.189.179.116:9092 (id: 0 rack: null), 10.189.179.118:9092 (id: 2 rack:
null), 10.189.179.117:9092 (id: 1 rack: null)], partitions =
[Partition(topic = bulk-email, partition = 8, leader = 2, replicas =
[0,1,2,], isr = [0,2,1,]), Partition(topic = bulk-email, partition = 7,
leader = 1, replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic =
bulk-email, partition = 9, leader = 0, replicas = [0,1,2,], isr =
[0,1,2,]), Partition(topic = bulk-email, partition = 4, leader = 1,
replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic = bulk-email,
partition = 3, leader = 0, replicas = [0,1,2,], isr = [0,1,2,]),
Partition(topic = bulk-email, partition = 6, leader = 0, replicas =
[0,1,2,], isr = [0,1,2,]), Partition(topic = bulk-email, partition = 5,
leader = 2, replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic =
bulk-email, partition = 0, leader = 0, replicas = [0,1,2,], isr =
[0,1,2,]), Partition(topic = bulk-email, partition = 2, leader = 2,
replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic = bulk-email,
partition = 1, leader = 1, replicas = [0,1,2,], isr = [1,0,2,])])
[2018-08-09 11:40:30] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:30] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:30] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:30] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:31] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:33] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:33] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:33] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:33] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:33] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:34] :: DEBUG :: NetworkClient$DefaultMetadataUpdater:640
- Sending metadata request {topics=[bulk-email]} to node 1
[2018-08-09 11:40:34] :: DEBUG :: Metadata :241 - Updated cluster metadata
version 646 to Cluster(id = 09FjKmqPQvat0c0dyPptIw, nodes = [
10.189.179.117:9092 (id: 1 rack: null), 10.189.179.116:9092 (id: 0 rack:
null), 10.189.179.118:9092 (id: 2 rack: null)], partitions =
[Partition(topic = bulk-email, partition = 8, leader = 2, replicas =
[0,1,2,], isr = [0,2,1,]), Partition(topic = bulk-email, partition = 7,
leader = 1, replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic =
bulk-email, partition = 9, leader = 0, replicas = [0,1,2,], isr =
[0,1,2,]), Partition(topic = bulk-email, partition = 4, leader = 1,
replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic = bulk-email,
partition = 3, leader = 0, replicas = [0,1,2,], isr = [0,1,2,]),
Partition(topic = bulk-email, partition = 6, leader = 0, replicas =
[0,1,2,], isr = [0,1,2,]), Partition(topic = bulk-email, partition = 5,
leader = 2, replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic =
bulk-email, partition = 0, leader = 0, replicas = [0,1,2,], isr =
[0,1,2,]), Partition(topic = bulk-email, partition = 2, leader = 2,
replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic = bulk-email,
partition = 1, leader = 1, replicas = [0,1,2,], isr = [1,0,2,])])
[2018-08-09 11:40:36] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:36] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:36] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:36] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:36] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:39] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:39] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:39] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:39] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:39] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:42] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:42] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:42] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:42] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:42] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:45] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:45] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:45] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:45] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:45] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:48] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:48] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:48] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:48] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:48] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:50] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-0=OffsetAndMetadata{offset=25390438,
metadata=''}, bulk-email-2=OffsetAndMetadata{offset=25368327, metadata=''},
bulk-email-1=OffsetAndMetadata{offset=25369219, metadata=''}} for group
bulk-email-consumer
[2018-08-09 11:40:50] :: WARN :: ConsumerCoordinator:554 - Auto offset
commit failed for group bulk-email-consumer: Commit cannot be completed
since the group has already rebalanced and assigned the partitions to
another member. This means that the time between subsequent calls to poll()
was longer than the configured max.poll.interval.ms, which typically
implies that the poll loop is spending too much time message processing.
You can address this either by increasing the session timeout or by
reducing the maximum size of batches returned in poll() with
max.poll.records.
[2018-08-09 11:40:50] :: INFO :: ConsumerCoordinator:333 - Revoking
previously assigned partitions [bulk-email-0, bulk-email-2, bulk-email-1]
for group bulk-email-consumer
[2018-08-09 11:40:50] :: INFO :: AbstractCoordinator:381 - (Re-)joining
group bulk-email-consumer
[2018-08-09 11:40:50] :: DEBUG :: AbstractCoordinator:390 - Sending
JoinGroup
({group_id=bulk-email-consumer,session_timeout=300000,rebalance_timeout=300000,member_id=,protocol_type=consumer,group_protocols=[{protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id: 2147483647 rack:
null)
[2018-08-09 11:40:50] :: DEBUG :: NetworkClient$DefaultMetadataUpdater:640
- Sending metadata request {topics=[bulk-email]} to node 0
[2018-08-09 11:40:50] :: DEBUG ::
ConsumerNetworkClient$RequestFutureCompletionHandler:475 - Cancelled
LEAVE_GROUP request ClientRequest(expectResponse=true,
callback=org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler@60f41e0b,
request=RequestSend(header={api_key=13,api_version=0,correlation_id=349313,client_id=consumer-19},
body={group_id=bulk-email-consumer,member_id=consumer-19-34903180-9cb3-496d-a913-b782476d9a52}),
createdTimeMs=1533794419195, sendTimeMs=1533794419195) with correlation id
349313 due to node 2147483647 being disconnected
[2018-08-09 11:40:50] :: INFO :: AbstractCoordinator:600 - Marking the
coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) dead for group
bulk-email-consumer
[2018-08-09 11:40:50] :: DEBUG ::
ConsumerNetworkClient$RequestFutureCompletionHandler:475 - Cancelled
JOIN_GROUP request ClientRequest(expectResponse=true,
callback=org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler@3bd6f5c9,
request=RequestSend(header={api_key=11,api_version=1,correlation_id=349314,client_id=consumer-19},
body={group_id=bulk-email-consumer,session_timeout=300000,rebalance_timeout=300000,member_id=,protocol_type=consumer,group_protocols=[{protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
lim=22 cap=22]}]}), createdTimeMs=1533795050857, sendTimeMs=0) with
correlation id 349314 due to node 2147483647 being disconnected
[2018-08-09 11:40:50] :: INFO :: AbstractCoordinator:600 - Marking the
coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) dead for group
bulk-email-consumer
[2018-08-09 11:40:50] :: DEBUG :: AbstractCoordinator:531 - Sending
coordinator request for group bulk-email-consumer to broker
10.189.179.118:9092 (id: 2 rack: null)
[2018-08-09 11:40:51] :: DEBUG :: NetworkClient$DefaultMetadataUpdater:640
- Sending metadata request {topics=[bulk-email]} to node 2
[2018-08-09 11:40:51] :: DEBUG :: Metadata :241 - Updated cluster metadata
version 641 to Cluster(id = 09FjKmqPQvat0c0dyPptIw, nodes = [
10.189.179.117:9092 (id: 1 rack: null), 10.189.179.116:9092 (id: 0 rack:
null), 10.189.179.118:9092 (id: 2 rack: null)], partitions =
[Partition(topic = bulk-email, partition = 8, leader = 2, replicas =
[0,1,2,], isr = [0,2,1,]), Partition(topic = bulk-email, partition = 7,
leader = 1, replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic =
bulk-email, partition = 9, leader = 0, replicas = [0,1,2,], isr =
[0,1,2,]), Partition(topic = bulk-email, partition = 4, leader = 1,
replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic = bulk-email,
partition = 3, leader = 0, replicas = [0,1,2,], isr = [0,1,2,]),
Partition(topic = bulk-email, partition = 6, leader = 0, replicas =
[0,1,2,], isr = [0,1,2,]), Partition(topic = bulk-email, partition = 5,
leader = 2, replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic =
bulk-email, partition = 0, leader = 0, replicas = [0,1,2,], isr =
[0,1,2,]), Partition(topic = bulk-email, partition = 2, leader = 2,
replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic = bulk-email,
partition = 1, leader = 1, replicas = [0,1,2,], isr = [1,0,2,])])
[2018-08-09 11:40:51] :: DEBUG ::
AbstractCoordinator$GroupCoordinatorResponseHandler:541 - Received group
coordinator response ClientResponse(receivedTimeMs=1533795051063,
disconnected=false, request=ClientRequest(expectResponse=true,
callback=org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler@21a34a4e,
request=RequestSend(header={api_key=10,api_version=0,correlation_id=349316,client_id=consumer-19},
body={group_id=bulk-email-consumer}), createdTimeMs=1533795050961,
sendTimeMs=1533795051061),
responseBody={error_code=0,coordinator={node_id=0,host=10.189.179.116,port=9092}})
[2018-08-09 11:40:51] :: INFO ::
AbstractCoordinator$GroupCoordinatorResponseHandler:555 - Discovered
coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) for group
bulk-emai -consumer.
[2018-08-09 11:40:51] :: INFO :: AbstractCoordinator:381 - (Re-)joining
group bulk-email-consumer
[2018-08-09 11:40:51] :: DEBUG :: AbstractCoordinator:390 - Sending
JoinGroup
({group_id=bulk-email-consumer,session_timeout=300000,rebalance_timeout=300000,member_id=,protocol_type=consumer,group_protocols=[{protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id: 2147483647 rack:
null)
[2018-08-09 11:40:51] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:51] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:51] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:51] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:54] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:54] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:54] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:54] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:57] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:57] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:57] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:57] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:00] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:00] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:03] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:03] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:06] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:06] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:09] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:09] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:12] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:15] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:17] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:17] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:18] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:18] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:21] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:21] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:24] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:24] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:27] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:27] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:31] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:31] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:33] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:33] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:36] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:36] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:39] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:39] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:42] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:42] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:46] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:46] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:48] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:48] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:51] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:51] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:54] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:54] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:57] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:57] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:42:01] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:42:03] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:42:06] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:42:09] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:42:12] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:42:15] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:42:18] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:42:21] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:42:24] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:42:27] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:42:31] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:42:34] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:42:37] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:42:40] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:42:43] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:42:46] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:42:49] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:42:52] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:42:55] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:42:58] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:43:01] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:43:04] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:43:07] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:43:10] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:43:13] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:43:16] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:43:19] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:43:22] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:43:25] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:43:26] :: DEBUG ::
AbstractCoordinator$JoinGroupResponseHandler:406 - Received successful join
group response for group bulk-email-consumer:
{error_code=0,generation_id=1225,group_protocol=range,leader_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_id=consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09,members=[]}
[2018-08-09 11:43:26] :: DEBUG ::
AbstractCoordinator$JoinGroupResponseHandler:406 - Received successful join
group response for group bulk-email-consumer:
{error_code=0,generation_id=1225,group_protocol=range,leader_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_id=consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa,members=[]}
[2018-08-09 11:43:26] :: DEBUG ::
AbstractCoordinator$JoinGroupResponseHandler:406 - Received successful join
group response for group bulk-email-consumer:
{error_code=0,generation_id=1225,group_protocol=range,leader_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_id=consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa,members=[]}
[2018-08-09 11:43:26] :: DEBUG ::
AbstractCoordinator$JoinGroupResponseHandler:406 - Received successful join
group response for group bulk-email-consumer:
{error_code=0,generation_id=1225,group_protocol=range,leader_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,members=[{member_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_metadata=java.nio.HeapByteBuffer[pos=0
lim=22
cap=250]},{member_id=consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa,member_metadata=java.nio.HeapByteBuffer[pos=0
lim=22
cap=174]},{member_id=consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa,member_metadata=java.nio.HeapByteBuffer[pos=0
lim=22
cap=98]},{member_id=consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09,member_metadata=java.nio.HeapByteBuffer[pos=0
lim=22 cap=22]}]}
[2018-08-09 11:43:26] :: DEBUG :: AbstractCoordinator:460 - Sending
follower SyncGroup for group bulk-email-consumer to coordinator
10.189.179.116:9092 (id: 2147483647 rack: null):
{group_id=bulk-email-consumer,generation_id=1225,member_id=consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09,group_assignment=[]}
[2018-08-09 11:43:26] :: DEBUG :: AbstractCoordinator:460 - Sending
follower SyncGroup for group bulk-email-consumer to coordinator
10.189.179.116:9092 (id: 2147483647 rack: null):
{group_id=bulk-email-consumer,generation_id=1225,member_id=consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa,group_assignment=[]}
[2018-08-09 11:43:26] :: DEBUG :: AbstractCoordinator:460 - Sending
follower SyncGroup for group bulk-email-consumer to coordinator
10.189.179.116:9092 (id: 2147483647 rack: null):
{group_id=bulk-email-consumer,generation_id=1225,member_id=consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa,group_assignment=[]}
[2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:310 - Performing
assignment for group bulk-email-consumer using strategy range with
subscriptions
{consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1=Subscription(topics=[bulk-email]),
consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa=Subscription(topics=[bulk-email]),
consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa=Subscription(topics=[bulk-email]),
consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09=Subscription(topics=[bulk-email])}
[2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:315 - Finished
assignment for group bulk-email-consumer:
{consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1=Assignment(partitions=[bulk-email-3,
bulk-email-4, bulk-email-5]),
consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa=Assignment(partitions=[bulk-email-0,
bulk-email-1, bulk-email-2]),
consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa=Assignment(partitions=[bulk-email-6,
bulk-email-7]),
consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09=Assignment(partitions=[bulk-email-8,
bulk-email-9])}
[2018-08-09 11:43:26] :: DEBUG :: AbstractCoordinator:471 - Sending leader
SyncGroup for group bulk-email-consumer to coordinator 10.189.179.116:9092
(id: 2147483647 rack: null):
{group_id=bulk-email-consumer,generation_id=1225,member_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,group_assignment=[{member_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_assignment=java.nio.HeapByteBuffer[pos=0
lim=38
cap=38]},{member_id=consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa,member_assignment=java.nio.HeapByteBuffer[pos=0
lim=38
cap=38]},{member_id=consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa,member_assignment=java.nio.HeapByteBuffer[pos=0
lim=34
cap=34]},{member_id=consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09,member_assignment=java.nio.HeapByteBuffer[pos=0
lim=34 cap=34]}]}
[2018-08-09 11:43:26] :: INFO :: AbstractCoordinator$1:349 - Successfully
joined group bulk-email-consumer with generation 1225
[2018-08-09 11:43:26] :: INFO :: AbstractCoordinator$1:349 - Successfully
joined group bulk-email-consumer with generation 1225
[2018-08-09 11:43:26] :: INFO :: AbstractCoordinator$1:349 - Successfully
joined group bulk-email-consumer with generation 1225
[2018-08-09 11:43:26] :: INFO :: AbstractCoordinator$1:349 - Successfully
joined group bulk-email-consumer with generation 1225
[2018-08-09 11:43:26] :: INFO :: ConsumerCoordinator:225 - Setting newly
assigned partitions [bulk-email-7, bulk-email-6] for group
bulk-email-consumer
[2018-08-09 11:43:26] :: INFO :: ConsumerCoordinator:225 - Setting newly
assigned partitions [bulk-email-0, bulk-email-2, bulk-email-1] for group
bulk-email-consumer
[2018-08-09 11:43:26] :: INFO :: ConsumerCoordinator:225 - Setting newly
assigned partitions [bulk-email-4, bulk-email-3, bulk-email-5] for group
bulk-email-consumer
[2018-08-09 11:43:26] :: INFO :: ConsumerCoordinator:225 - Setting newly
assigned partitions [bulk-email-8, bulk-email-9] for group
bulk-email-consumer
[2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:708 - Group
bulk-email-consumer fetching committed offsets for partitions:
[bulk-email-7, bulk-email-6]
[2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:708 - Group
bulk-email-consumer fetching committed offsets for partitions:
[bulk-email-0, bulk-email-2, bulk-email-1]
[2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:708 - Group
bulk-email-consumer fetching committed offsets for partitions:
[bulk-email-4, bulk-email-3, bulk-email-5]
[2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:708 - Group
bulk-email-consumer fetching committed offsets for partitions:
[bulk-email-8, bulk-email-9]
[2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
partition bulk-email-7 to the committed offset 25435998
[2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
partition bulk-email-0 to the committed offset 25390842
[2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
partition bulk-email-6 to the committed offset 25463566
[2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
partition bulk-email-4 to the committed offset 25409183
[2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
partition bulk-email-8 to the committed offset 25465113
[2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
partition bulk-email-2 to the committed offset 25368327
[2018-08-09 11:43:26] :: DEBUG :: Fetcher :665 - Ignoring fetched records
for partition bulk-email-8 since it is no longer fetchable
[2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
partition bulk-email-3 to the committed offset 25422008
[2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
partition bulk-email-9 to the committed offset 25444488
[2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
partition bulk-email-1 to the committed offset 25369263
[2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
partition bulk-email-5 to the committed offset 25456518
[2018-08-09 11:43:26] :: DEBUG :: Fetcher :470 - Ignoring fetched records
for bulk-email-0 at offset 25390442 since the current position is 25390842
[2018-08-09 11:43:26] :: DEBUG :: Fetcher :665 - Ignoring fetched records
for partition bulk-email-6 since it is no longer fetchable
[2018-08-09 11:43:26] :: DEBUG :: Fetcher :671 - Discarding stale fetch
response for partition bulk-email-1 since its offset 25369219 does not
match the expected offset 25369263
[2018-08-09 11:43:29] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25465113 for partition bulk-email-8
[2018-08-09 11:43:29] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25444488 for partition bulk-email-9
[2018-08-09 11:43:29] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
metadata=''}, bulk-email-9=OffsetAndMetadata{offset=25444488, metadata=''}}
for group bulk-email-consumer
[2018-08-09 11:43:29] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:43:29] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:43:29] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25436018 for partition bulk-email-7
[2018-08-09 11:43:29] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:43:29] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25463566 for partition bulk-email-6
[2018-08-09 11:43:29] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:43:29] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25409183 for partition bulk-email-4
[2018-08-09 11:43:29] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25422008 for partition bulk-email-3
[2018-08-09 11:43:29] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25456530 for partition bulk-email-5
[2018-08-09 11:43:29] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25390842 for partition bulk-email-0
[2018-08-09 11:43:29] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25368339 for partition bulk-email-2
[2018-08-09 11:43:29] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25369263 for partition bulk-email-1
[2018-08-09 11:43:29] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-4=OffsetAndMetadata{offset=25409183,
metadata=''}, bulk-email-3=OffsetAndMetadata{offset=25422008, metadata=''},
bulk-email-5=OffsetAndMetadata{offset=25456530, metadata=''}} for group
bulk-email-consumer
On Thu, Jul 12, 2018 at 9:27 PM Steve Tian <steve.cs.t...@gmail.com> wrote:
> It's a very good and important doc so I think you should read it all. You
> should get some idea from sections like *Detecting Consumer Failures* and
> *Multi-threaded Processing* for your case.
>
> On Thu, Jul 12, 2018, 3:17 PM Shantanu Deshmukh <shantanu...@gmail.com>
> wrote:
>
> > Hi Steve,
> >
> > Could you please shed more light on this? What section should I revisit?
> I
> > am using high-level consumer. So I am simply calling consumer.close()
> when
> > I am shutting down the process. Is there any other method to be called
> > before calling close()?
> >
> > On Mon, Jul 9, 2018 at 5:58 PM Steve Tian <steve.cs.t...@gmail.com>
> wrote:
> >
> > > Please re-read the javadoc of KafkaConsumer, make sure you know how to
> > > wakeup/close consumer properly while shutting down your application.
> Try
> > > to understand the motivation of KIP-62 and adjust related timeout.
> > >
> > > On Mon, Jul 9, 2018, 8:05 PM harish lohar <hklo...@gmail.com> wrote:
> > >
> > > > Try reducing below timer
> > > > metadata.max.age.ms = 300000
> > > >
> > > >
> > > > On Fri, Jul 6, 2018 at 5:55 AM Shantanu Deshmukh <
> > shantanu...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hello everyone,
> > > > >
> > > > > We are running a 3 broker Kafka 0.10.0.1 cluster. We have a java
> app
> > > > which
> > > > > spawns many consumer threads consuming from different topics. For
> > every
> > > > > topic we have specified different consumer-group. A lot of times I
> > see
> > > > that
> > > > > whenever this application is restarted a CG on one or two topics
> > takes
> > > > more
> > > > > than 5 minutes to receive partition assignment. Till that time
> > > consumers
> > > > > for that topic don't consumer anything. If I go to Kafka broker and
> > run
> > > > > consumer-groups.sh and describe that particular CG I see that it is
> > > > > rebalancing. There is time critical data stored in that topic and
> we
> > > > cannot
> > > > > tolerate such long delays. What can be the reason for such long
> > > > rebalances.
> > > > >
> > > > > Here's our consumer config
> > > > >
> > > > >
> > > > > auto.commit.interval.ms = 3000
> > > > > auto.offset.reset = latest
> > > > > bootstrap.servers = [x.x.x.x:9092, x.x.x.x:9092, x.x.x.x:9092]
> > > > > check.crcs = true
> > > > > client.id =
> > > > > connections.max.idle.ms = 540000
> > > > > enable.auto.commit = true
> > > > > exclude.internal.topics = true
> > > > > fetch.max.bytes = 52428800
> > > > > fetch.max.wait.ms = 500
> > > > > fetch.min.bytes = 1
> > > > > group.id = otp-notifications-consumer
> > > > > heartbeat.interval.ms = 3000
> > > > > interceptor.classes = null
> > > > > key.deserializer = class
> > > > > org.apache.kafka.common.serialization.StringDeserializer
> > > > > max.partition.fetch.bytes = 1048576
> > > > > max.poll.interval.ms = 300000
> > > > > max.poll.records = 50
> > > > > metadata.max.age.ms = 300000
> > > > > metric.reporters = []
> > > > > metrics.num.samples = 2
> > > > > metrics.sample.window.ms = 30000
> > > > > partition.assignment.strategy = [class
> > > > > org.apache.kafka.clients.consumer.RangeAssignor]
> > > > > receive.buffer.bytes = 65536
> > > > > reconnect.backoff.ms = 50
> > > > > request.timeout.ms = 305000
> > > > > retry.backoff.ms = 100
> > > > > sasl.kerberos.kinit.cmd = /usr/bin/kinit
> > > > > sasl.kerberos.min.time.before.relogin = 60000
> > > > > sasl.kerberos.service.name = null
> > > > > sasl.kerberos.ticket.renew.jitter = 0.05
> > > > > sasl.kerberos.ticket.renew.window.factor = 0.8
> > > > > sasl.mechanism = GSSAPI
> > > > > security.protocol = SSL
> > > > > send.buffer.bytes = 131072
> > > > > session.timeout.ms = 300000
> > > > > ssl.cipher.suites = null
> > > > > ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
> > > > > ssl.endpoint.identification.algorithm = null
> > > > > ssl.key.password = null
> > > > > ssl.keymanager.algorithm = SunX509
> > > > > ssl.keystore.location = null
> > > > > ssl.keystore.password = null
> > > > > ssl.keystore.type = JKS
> > > > > ssl.protocol = TLS
> > > > > ssl.provider = null
> > > > > ssl.secure.random.implementation = null
> > > > > ssl.trustmanager.algorithm = PKIX
> > > > > ssl.truststore.location = /x/x/client.truststore.jks
> > > > > ssl.truststore.password = [hidden]
> > > > > ssl.truststore.type = JKS
> > > > > value.deserializer = class
> > > > > org.apache.kafka.common.serialization.StringDeserializer
> > > > >
> > > > > Please help.
> > > > >
> > > > > *Thanks & Regards,*
> > > > > *Shantanu Deshmukh*
> > > > >
> > > >
> > >
> >
>
