if you can upgrade, I would say upgrading to 0.10.2.x would be better for
you (or even higher, 2.0.0). Otherwise you have to play around with
max.poll.records and session.timeout.ms.
As the doc says (or newer versions), the adjustment should be such that
request.timeout.ms >= max.poll.interval.ms. Also, heartbeat.interval.ms
should be curbed at (rule of thumb) 30% of session.timeout.ms.
Lastly, all these have to be within the bounds of
group.min.session.timeout.ms and group.max.session.timeout.ms.
You can check all these, tune them as necessary, and retry. Some of these
configs may or may not be applicable at runtime. so a rolling restart may
be required before all changes take place.
On 9 August 2018 at 13:48, Shantanu Deshmukh <shantanu...@gmail.com> wrote:
> Hi,
>
> Yes my consumer application works like below
>
> 1. Reads how many workers are required to process each topics from
> properties file
> 2. As many threads are spawned as there are workers mentioned in
> properties file, topic name is passed to this thread. FixedThreadPool
> implementation is used.
> 3. Each worker thread initializes one consumer object and subscribes to
> given topic. Consumer group is simply <topic-name>-consumer. So if my
> topic
> bulk-email, then consumer group for all those threads is
> bulk-email-consumer
> 4. Once this is done, inside an infinite while loop consumer.poll(100)
> method keeps running. So this application is a daemon. Shuts down only
> when
> server shuts down or in case of kill command.
>
> I have configured session.timeout.ms in consumer properties. I haven't
> done
> anything about zookeeper timeout. Is it required now? Since consumer
> accesses only the brokers.
>
> On Thu, Aug 9, 2018 at 3:03 PM M. Manna <manme...@gmail.com> wrote:
>
> > In the simplest way, how have you implemented your consumer?
> >
> > 1) Does your consumers join a designated group, process messages, and
> then
> > closes all connection? Or does it stay open perpetually until server
> > shutdown?
> > 2) Have you configured the session timeouts for client and zookeeper
> > accordingly?
> >
> > Regards,
> >
> > On 9 August 2018 at 08:00, Shantanu Deshmukh <shantanu...@gmail.com>
> > wrote:
> >
> > > I am facing too many problems these days. Now one of our consumer
> groups
> > > is rebalancing every now and then. And rebalance takes very low, more
> > than
> > > 5-10 minutes. Even after re-balancing I see that only half of the
> > consumers
> > > are active/receive assignment. Its all going haywire.
> > >
> > > I am seeing these logs in kafka consumer logs. Can anyone help me
> > > understand what is going on here? It is a very long piece of log, but
> > > someone please help me. I am desperately looking for any solution since
> > > more than 2 months now. But to no avail.
> > >
> > > [2018-08-09 11:39:51] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:39:53] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25465113 for partition
> bulk-email-8
> > > [2018-08-09 11:39:53] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:39:53] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25463566 for partition
> bulk-email-6
> > > [2018-08-09 11:39:53] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:39:53] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25444488 for partition
> bulk-email-9
> > > [2018-08-09 11:39:53] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:39:54] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:39:54] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:39:54] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:39:54] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:39:54] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:39:54] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:39:54] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:39:54] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:39:56] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25463566 for partition
> bulk-email-6
> > > [2018-08-09 11:39:56] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25444488 for partition
> bulk-email-9
> > > [2018-08-09 11:39:56] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25465113 for partition
> bulk-email-8
> > > [2018-08-09 11:39:56] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:39:56] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:39:56] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:39:57] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:39:57] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:39:57] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:39:57] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:39:57] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:39:57] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:39:57] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:39:57] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:00] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:00] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:00] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:03] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:03] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:03] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:04] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:04] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:04] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:664 - Offset commit
> for
> > > group bulk-email-consumer failed: The request timed out.
> > > [2018-08-09 11:40:04] :: INFO :: AbstractCoordinator:600 - Marking the
> > > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) dead for
> > group
> > > bulk-email-consumer
> > > [2018-08-09 11:40:04] :: WARN :: ConsumerCoordinator$4:535 - Auto
> offset
> > > commit failed for group bulk-email-consumer: Offset commit failed with
> a
> > > retriable exception. You should retry committing offsets.
> > > [2018-08-09 11:40:04] :: DEBUG :: AbstractCoordinator:531 - Sending
> > > coordinator request for group bulk-email-consumer to broker
> > > 10.189.179.117:9092 (id: 1 rack: null)
> > > [2018-08-09 11:40:04] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:04] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:664 - Offset commit
> for
> > > group bulk-email-consumer failed: The request timed out.
> > > [2018-08-09 11:40:04] :: INFO :: AbstractCoordinator:600 - Marking the
> > > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) dead for
> > group
> > > bulk-email-consumer
> > > [2018-08-09 11:40:04] :: WARN :: ConsumerCoordinator$4:535 - Auto
> offset
> > > commit failed for group bulk-email-consumer: Offset commit failed with
> a
> > > retriable exception. You should retry committing offsets.
> > > [2018-08-09 11:40:04] :: DEBUG :: AbstractCoordinator:531 - Sending
> > > coordinator request for group bulk-email-consumer to broker
> > > 10.189.179.118:9092 (id: 2 rack: null)
> > > [2018-08-09 11:40:04] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:04] :: DEBUG ::
> > > AbstractCoordinator$GroupCoordinatorResponseHandler:541 - Received
> group
> > > coordinator response ClientResponse(receivedTimeMs=1533795004158,
> > > disconnected=false, request=ClientRequest(expectResponse=true,
> > > callback=org.apache.kafka.clients.consumer.internals.
> > > ConsumerNetworkClient$RequestFutureCompletionHandler@3a3e5740,
> > > request=RequestSend(header={api_key=10,api_version=0,
> > > correlation_id=389784,client_id=consumer-26},
> > > body={group_id=bulk-email-consumer}), createdTimeMs=1533795004157,
> > > sendTimeMs=1533795004157),
> > > responseBody={error_code=0,coordinator={node_id=0,host=
> > > 10.189.179.116,port=9092}})
> > > [2018-08-09 11:40:04] :: INFO ::
> > > AbstractCoordinator$GroupCoordinatorResponseHandler:555 - Discovered
> > > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) for group
> > > bulk-emai -consumer.
> > > [2018-08-09 11:40:04] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:04] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:04] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25463566 for partition
> bulk-email-6
> > > [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:40:04] :: DEBUG ::
> > > AbstractCoordinator$GroupCoordinatorResponseHandler:541 - Received
> group
> > > coordinator response ClientResponse(receivedTimeMs=1533795004718,
> > > disconnected=false, request=ClientRequest(expectResponse=true,
> > > callback=org.apache.kafka.clients.consumer.internals.
> > > ConsumerNetworkClient$RequestFutureCompletionHandler@66889bbd,
> > > request=RequestSend(header={api_key=10,api_version=0,
> > > correlation_id=363859,client_id=consumer-25},
> > > body={group_id=bulk-email-consumer}), createdTimeMs=1533795004145,
> > > sendTimeMs=1533795004145),
> > > responseBody={error_code=0,coordinator={node_id=0,host=
> > > 10.189.179.116,port=9092}})
> > > [2018-08-09 11:40:04] :: INFO ::
> > > AbstractCoordinator$GroupCoordinatorResponseHandler:555 - Discovered
> > > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) for group
> > > bulk-emai -consumer.
> > > [2018-08-09 11:40:04] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:04] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25444488 for partition
> bulk-email-9
> > > [2018-08-09 11:40:04] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25465113 for partition
> bulk-email-8
> > > [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:40:04] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:04] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:04] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25463566 for partition
> bulk-email-6
> > > [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:40:04] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:04] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25465113 for partition
> bulk-email-8
> > > [2018-08-09 11:40:04] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25444488 for partition
> bulk-email-9
> > > [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:40:05] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25463566 for partition
> bulk-email-6
> > > [2018-08-09 11:40:05] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:40:06] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:06] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:06] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:06] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:06] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:06] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:07] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:07] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25444488 for partition
> bulk-email-9
> > > [2018-08-09 11:40:07] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:40:07] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:07] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25465113 for partition
> bulk-email-8
> > > [2018-08-09 11:40:07] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:40:08] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25463566 for partition
> bulk-email-6
> > > [2018-08-09 11:40:08] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:40:09] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:09] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:09] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:09] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:09] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:09] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:10] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:10] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25444488 for partition
> bulk-email-9
> > > [2018-08-09 11:40:10] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:40:10] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25465113 for partition
> bulk-email-8
> > > [2018-08-09 11:40:10] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:40:10] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:11] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25463566 for partition
> bulk-email-6
> > > [2018-08-09 11:40:11] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:40:12] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:12] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:12] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:12] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:12] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:12] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:13] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25444488 for partition
> bulk-email-9
> > > [2018-08-09 11:40:13] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:40:13] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:14] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25465113 for partition
> bulk-email-8
> > > [2018-08-09 11:40:14] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25463566 for partition
> bulk-email-6
> > > [2018-08-09 11:40:14] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:40:14] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:40:14] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:15] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:15] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:15] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:15] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:15] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:15] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:16] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25444488 for partition
> bulk-email-9
> > > [2018-08-09 11:40:16] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:40:16] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:16] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25465113 for partition
> bulk-email-8
> > > [2018-08-09 11:40:16] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:40:16] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:17] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25463566 for partition
> bulk-email-6
> > > [2018-08-09 11:40:17] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:40:18] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:18] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:18] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:18] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:18] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:18] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:19] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25444488 for partition
> bulk-email-9
> > > [2018-08-09 11:40:19] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:40:19] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:19] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25465113 for partition
> bulk-email-8
> > > [2018-08-09 11:40:19] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:40:19] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:20] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25463566 for partition
> bulk-email-6
> > > [2018-08-09 11:40:20] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:40:21] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:21] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:21] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:21] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:21] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:21] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:21] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25463566 for partition
> bulk-email-6
> > > [2018-08-09 11:40:21] :: INFO :: ConsumerCoordinator:333 - Revoking
> > > previously assigned partitions [bulk-email-6] for group
> > bulk-email-consumer
> > > [2018-08-09 11:40:21] :: INFO :: AbstractCoordinator:381 -
> (Re-)joining
> > > group bulk-email-consumer
> > > [2018-08-09 11:40:21] :: DEBUG :: AbstractCoordinator:390 - Sending
> > > JoinGroup
> > > ({group_id=bulk-email-consumer,session_timeout=
> 300000,rebalance_timeout=
> > > 300000,member_id=consumer-23-bdc073b6-1f90-4ac5-a801-
> > > 869377f9ead1,protocol_type=consumer,group_protocols=[{
> > > protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
> > > lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id: 2147483647
> > > rack:
> > > null)
> > > [2018-08-09 11:40:22] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25444488 for partition
> bulk-email-9
> > > [2018-08-09 11:40:22] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:40:22] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25465113 for partition
> bulk-email-8
> > > [2018-08-09 11:40:22] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:40:22] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:22] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:22] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25444488 for partition
> bulk-email-9
> > > [2018-08-09 11:40:22] :: INFO :: ConsumerCoordinator:333 - Revoking
> > > previously assigned partitions [bulk-email-9] for group
> > bulk-email-consumer
> > > [2018-08-09 11:40:22] :: INFO :: AbstractCoordinator:381 -
> (Re-)joining
> > > group bulk-email-consumer
> > > [2018-08-09 11:40:22] :: DEBUG :: AbstractCoordinator:390 - Sending
> > > JoinGroup
> > > ({group_id=bulk-email-consumer,session_timeout=
> 300000,rebalance_timeout=
> > > 300000,member_id=consumer-26-f5a1293d-7d14-4519-8378-
> > > 1c4dd363ba09,protocol_type=consumer,group_protocols=[{
> > > protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
> > > lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id: 2147483647
> > > rack:
> > > null)
> > > [2018-08-09 11:40:22] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25465113 for partition
> bulk-email-8
> > > [2018-08-09 11:40:23] :: INFO :: ConsumerCoordinator:333 - Revoking
> > > previously assigned partitions [bulk-email-8] for group
> > bulk-email-consumer
> > > [2018-08-09 11:40:23] :: INFO :: AbstractCoordinator:381 -
> (Re-)joining
> > > group bulk-email-consumer
> > > [2018-08-09 11:40:23] :: DEBUG :: AbstractCoordinator:390 - Sending
> > > JoinGroup
> > > ({group_id=bulk-email-consumer,session_timeout=
> 300000,rebalance_timeout=
> > > 300000,member_id=consumer-25-fa10026f-031e-49aa-9df1-
> > > 9d083b256ffa,protocol_type=consumer,group_protocols=[{
> > > protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
> > > lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id: 2147483647
> > > rack:
> > > null)
> > > [2018-08-09 11:40:24] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:24] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:24] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:24] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:24] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:26] :: DEBUG ::
> > NetworkClient$DefaultMetadataUpdater:640
> > > - Sending metadata request {topics=[bulk-email]} to node 1
> > > [2018-08-09 11:40:26] :: DEBUG :: Metadata :241 - Updated cluster
> > metadata
> > > version 642 to Cluster(id = 09FjKmqPQvat0c0dyPptIw, nodes = [
> > > 10.189.179.118:9092 (id: 2 rack: null), 10.189.179.116:9092 (id: 0
> rack:
> > > null), 10.189.179.117:9092 (id: 1 rack: null)], partitions =
> > > [Partition(topic = bulk-email, partition = 8, leader = 2, replicas =
> > > [0,1,2,], isr = [0,2,1,]), Partition(topic = bulk-email, partition = 7,
> > > leader = 1, replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic =
> > > bulk-email, partition = 9, leader = 0, replicas = [0,1,2,], isr =
> > > [0,1,2,]), Partition(topic = bulk-email, partition = 4, leader = 1,
> > > replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic = bulk-email,
> > > partition = 3, leader = 0, replicas = [0,1,2,], isr = [0,1,2,]),
> > > Partition(topic = bulk-email, partition = 6, leader = 0, replicas =
> > > [0,1,2,], isr = [0,1,2,]), Partition(topic = bulk-email, partition = 5,
> > > leader = 2, replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic =
> > > bulk-email, partition = 0, leader = 0, replicas = [0,1,2,], isr =
> > > [0,1,2,]), Partition(topic = bulk-email, partition = 2, leader = 2,
> > > replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic = bulk-email,
> > > partition = 1, leader = 1, replicas = [0,1,2,], isr = [1,0,2,])])
> > > [2018-08-09 11:40:27] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:27] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:27] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:27] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:27] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:28] :: DEBUG ::
> > NetworkClient$DefaultMetadataUpdater:640
> > > - Sending metadata request {topics=[bulk-email]} to node 1
> > > [2018-08-09 11:40:28] :: DEBUG :: Metadata :241 - Updated cluster
> > metadata
> > > version 641 to Cluster(id = 09FjKmqPQvat0c0dyPptIw, nodes = [
> > > 10.189.179.116:9092 (id: 0 rack: null), 10.189.179.118:9092 (id: 2
> rack:
> > > null), 10.189.179.117:9092 (id: 1 rack: null)], partitions =
> > > [Partition(topic = bulk-email, partition = 8, leader = 2, replicas =
> > > [0,1,2,], isr = [0,2,1,]), Partition(topic = bulk-email, partition = 7,
> > > leader = 1, replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic =
> > > bulk-email, partition = 9, leader = 0, replicas = [0,1,2,], isr =
> > > [0,1,2,]), Partition(topic = bulk-email, partition = 4, leader = 1,
> > > replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic = bulk-email,
> > > partition = 3, leader = 0, replicas = [0,1,2,], isr = [0,1,2,]),
> > > Partition(topic = bulk-email, partition = 6, leader = 0, replicas =
> > > [0,1,2,], isr = [0,1,2,]), Partition(topic = bulk-email, partition = 5,
> > > leader = 2, replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic =
> > > bulk-email, partition = 0, leader = 0, replicas = [0,1,2,], isr =
> > > [0,1,2,]), Partition(topic = bulk-email, partition = 2, leader = 2,
> > > replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic = bulk-email,
> > > partition = 1, leader = 1, replicas = [0,1,2,], isr = [1,0,2,])])
> > > [2018-08-09 11:40:30] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:30] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:30] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:30] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:31] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:33] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:33] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:33] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:33] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:33] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:34] :: DEBUG ::
> > NetworkClient$DefaultMetadataUpdater:640
> > > - Sending metadata request {topics=[bulk-email]} to node 1
> > > [2018-08-09 11:40:34] :: DEBUG :: Metadata :241 - Updated cluster
> > metadata
> > > version 646 to Cluster(id = 09FjKmqPQvat0c0dyPptIw, nodes = [
> > > 10.189.179.117:9092 (id: 1 rack: null), 10.189.179.116:9092 (id: 0
> rack:
> > > null), 10.189.179.118:9092 (id: 2 rack: null)], partitions =
> > > [Partition(topic = bulk-email, partition = 8, leader = 2, replicas =
> > > [0,1,2,], isr = [0,2,1,]), Partition(topic = bulk-email, partition = 7,
> > > leader = 1, replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic =
> > > bulk-email, partition = 9, leader = 0, replicas = [0,1,2,], isr =
> > > [0,1,2,]), Partition(topic = bulk-email, partition = 4, leader = 1,
> > > replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic = bulk-email,
> > > partition = 3, leader = 0, replicas = [0,1,2,], isr = [0,1,2,]),
> > > Partition(topic = bulk-email, partition = 6, leader = 0, replicas =
> > > [0,1,2,], isr = [0,1,2,]), Partition(topic = bulk-email, partition = 5,
> > > leader = 2, replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic =
> > > bulk-email, partition = 0, leader = 0, replicas = [0,1,2,], isr =
> > > [0,1,2,]), Partition(topic = bulk-email, partition = 2, leader = 2,
> > > replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic = bulk-email,
> > > partition = 1, leader = 1, replicas = [0,1,2,], isr = [1,0,2,])])
> > > [2018-08-09 11:40:36] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:36] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:36] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:36] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:36] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:39] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:39] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:39] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:39] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:39] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:42] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:42] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:42] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:42] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:42] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:45] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:45] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:45] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:45] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:45] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:48] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:48] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:48] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:48] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:48] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:50] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-0=OffsetAndMetadata{offset=25390438,
> > > metadata=''}, bulk-email-2=OffsetAndMetadata{offset=25368327,
> > > metadata=''},
> > > bulk-email-1=OffsetAndMetadata{offset=25369219, metadata=''}} for
> group
> > > bulk-email-consumer
> > > [2018-08-09 11:40:50] :: WARN :: ConsumerCoordinator:554 - Auto offset
> > > commit failed for group bulk-email-consumer: Commit cannot be completed
> > > since the group has already rebalanced and assigned the partitions to
> > > another member. This means that the time between subsequent calls to
> > poll()
> > > was longer than the configured max.poll.interval.ms, which typically
> > > implies that the poll loop is spending too much time message
> processing.
> > > You can address this either by increasing the session timeout or by
> > > reducing the maximum size of batches returned in poll() with
> > > max.poll.records.
> > > [2018-08-09 11:40:50] :: INFO :: ConsumerCoordinator:333 - Revoking
> > > previously assigned partitions [bulk-email-0, bulk-email-2,
> bulk-email-1]
> > > for group bulk-email-consumer
> > > [2018-08-09 11:40:50] :: INFO :: AbstractCoordinator:381 -
> (Re-)joining
> > > group bulk-email-consumer
> > > [2018-08-09 11:40:50] :: DEBUG :: AbstractCoordinator:390 - Sending
> > > JoinGroup
> > > ({group_id=bulk-email-consumer,session_timeout=
> 300000,rebalance_timeout=
> > > 300000,member_id=,protocol_type=consumer,group_protocols=
> > > [{protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
> > > lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id: 2147483647
> > > rack:
> > > null)
> > > [2018-08-09 11:40:50] :: DEBUG ::
> > NetworkClient$DefaultMetadataUpdater:640
> > > - Sending metadata request {topics=[bulk-email]} to node 0
> > > [2018-08-09 11:40:50] :: DEBUG ::
> > > ConsumerNetworkClient$RequestFutureCompletionHandler:475 - Cancelled
> > > LEAVE_GROUP request ClientRequest(expectResponse=true,
> > > callback=org.apache.kafka.clients.consumer.internals.
> > > ConsumerNetworkClient$RequestFutureCompletionHandler@60f41e0b,
> > > request=RequestSend(header={api_key=13,api_version=0,
> > > correlation_id=349313,client_id=consumer-19},
> > > body={group_id=bulk-email-consumer,member_id=consumer-
> > > 19-34903180-9cb3-496d-a913-b782476d9a52}),
> > > createdTimeMs=1533794419195, sendTimeMs=1533794419195) with correlation
> > id
> > > 349313 due to node 2147483647 being disconnected
> > > [2018-08-09 11:40:50] :: INFO :: AbstractCoordinator:600 - Marking the
> > > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) dead for
> > group
> > > bulk-email-consumer
> > > [2018-08-09 11:40:50] :: DEBUG ::
> > > ConsumerNetworkClient$RequestFutureCompletionHandler:475 - Cancelled
> > > JOIN_GROUP request ClientRequest(expectResponse=true,
> > > callback=org.apache.kafka.clients.consumer.internals.
> > > ConsumerNetworkClient$RequestFutureCompletionHandler@3bd6f5c9,
> > > request=RequestSend(header={api_key=11,api_version=1,
> > > correlation_id=349314,client_id=consumer-19},
> > > body={group_id=bulk-email-consumer,session_timeout=
> > > 300000,rebalance_timeout=300000,member_id=,protocol_
> > > type=consumer,group_protocols=[{protocol_name=range,
> > > protocol_metadata=java.nio.HeapByteBuffer[pos=0
> > > lim=22 cap=22]}]}), createdTimeMs=1533795050857, sendTimeMs=0) with
> > > correlation id 349314 due to node 2147483647 being disconnected
> > > [2018-08-09 11:40:50] :: INFO :: AbstractCoordinator:600 - Marking the
> > > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) dead for
> > group
> > > bulk-email-consumer
> > > [2018-08-09 11:40:50] :: DEBUG :: AbstractCoordinator:531 - Sending
> > > coordinator request for group bulk-email-consumer to broker
> > > 10.189.179.118:9092 (id: 2 rack: null)
> > > [2018-08-09 11:40:51] :: DEBUG ::
> > NetworkClient$DefaultMetadataUpdater:640
> > > - Sending metadata request {topics=[bulk-email]} to node 2
> > > [2018-08-09 11:40:51] :: DEBUG :: Metadata :241 - Updated cluster
> > metadata
> > > version 641 to Cluster(id = 09FjKmqPQvat0c0dyPptIw, nodes = [
> > > 10.189.179.117:9092 (id: 1 rack: null), 10.189.179.116:9092 (id: 0
> rack:
> > > null), 10.189.179.118:9092 (id: 2 rack: null)], partitions =
> > > [Partition(topic = bulk-email, partition = 8, leader = 2, replicas =
> > > [0,1,2,], isr = [0,2,1,]), Partition(topic = bulk-email, partition = 7,
> > > leader = 1, replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic =
> > > bulk-email, partition = 9, leader = 0, replicas = [0,1,2,], isr =
> > > [0,1,2,]), Partition(topic = bulk-email, partition = 4, leader = 1,
> > > replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic = bulk-email,
> > > partition = 3, leader = 0, replicas = [0,1,2,], isr = [0,1,2,]),
> > > Partition(topic = bulk-email, partition = 6, leader = 0, replicas =
> > > [0,1,2,], isr = [0,1,2,]), Partition(topic = bulk-email, partition = 5,
> > > leader = 2, replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic =
> > > bulk-email, partition = 0, leader = 0, replicas = [0,1,2,], isr =
> > > [0,1,2,]), Partition(topic = bulk-email, partition = 2, leader = 2,
> > > replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic = bulk-email,
> > > partition = 1, leader = 1, replicas = [0,1,2,], isr = [1,0,2,])])
> > > [2018-08-09 11:40:51] :: DEBUG ::
> > > AbstractCoordinator$GroupCoordinatorResponseHandler:541 - Received
> group
> > > coordinator response ClientResponse(receivedTimeMs=1533795051063,
> > > disconnected=false, request=ClientRequest(expectResponse=true,
> > > callback=org.apache.kafka.clients.consumer.internals.
> > > ConsumerNetworkClient$RequestFutureCompletionHandler@21a34a4e,
> > > request=RequestSend(header={api_key=10,api_version=0,
> > > correlation_id=349316,client_id=consumer-19},
> > > body={group_id=bulk-email-consumer}), createdTimeMs=1533795050961,
> > > sendTimeMs=1533795051061),
> > > responseBody={error_code=0,coordinator={node_id=0,host=
> > > 10.189.179.116,port=9092}})
> > > [2018-08-09 11:40:51] :: INFO ::
> > > AbstractCoordinator$GroupCoordinatorResponseHandler:555 - Discovered
> > > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) for group
> > > bulk-emai -consumer.
> > > [2018-08-09 11:40:51] :: INFO :: AbstractCoordinator:381 -
> (Re-)joining
> > > group bulk-email-consumer
> > > [2018-08-09 11:40:51] :: DEBUG :: AbstractCoordinator:390 - Sending
> > > JoinGroup
> > > ({group_id=bulk-email-consumer,session_timeout=
> 300000,rebalance_timeout=
> > > 300000,member_id=,protocol_type=consumer,group_protocols=
> > > [{protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
> > > lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id: 2147483647
> > > rack:
> > > null)
> > > [2018-08-09 11:40:51] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:51] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:51] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:51] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:54] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:54] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:54] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:54] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:57] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:57] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:57] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:57] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:00] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:00] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:03] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:03] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:06] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:06] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:09] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:09] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:12] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:15] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:17] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:17] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:18] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:18] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:21] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:21] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:24] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:24] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:27] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:27] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:31] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:31] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:33] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:33] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:36] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:36] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:39] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:39] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:42] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:42] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:46] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:46] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:48] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:48] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:51] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:51] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:54] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:54] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:57] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:57] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:42:01] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:42:03] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:42:06] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:42:09] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:42:12] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:42:15] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:42:18] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:42:21] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:42:24] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:42:27] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:42:31] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:42:34] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:42:37] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:42:40] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:42:43] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:42:46] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:42:49] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:42:52] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:42:55] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:42:58] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:43:01] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:43:04] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:43:07] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:43:10] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:43:13] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:43:16] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:43:19] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:43:22] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:43:25] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:43:26] :: DEBUG ::
> > > AbstractCoordinator$JoinGroupResponseHandler:406 - Received successful
> > > join
> > > group response for group bulk-email-consumer:
> > > {error_code=0,generation_id=1225,group_protocol=range,
> > > leader_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_id=
> > > consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09,members=[]}
> > > [2018-08-09 11:43:26] :: DEBUG ::
> > > AbstractCoordinator$JoinGroupResponseHandler:406 - Received successful
> > > join
> > > group response for group bulk-email-consumer:
> > > {error_code=0,generation_id=1225,group_protocol=range,
> > > leader_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_id=
> > > consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa,members=[]}
> > > [2018-08-09 11:43:26] :: DEBUG ::
> > > AbstractCoordinator$JoinGroupResponseHandler:406 - Received successful
> > > join
> > > group response for group bulk-email-consumer:
> > > {error_code=0,generation_id=1225,group_protocol=range,
> > > leader_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_id=
> > > consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa,members=[]}
> > > [2018-08-09 11:43:26] :: DEBUG ::
> > > AbstractCoordinator$JoinGroupResponseHandler:406 - Received successful
> > > join
> > > group response for group bulk-email-consumer:
> > > {error_code=0,generation_id=1225,group_protocol=range,
> > > leader_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_id=
> > > consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,
> > > members=[{member_id=consumer-23-bdc073b6-1f90-4ac5-a801-
> > > 869377f9ead1,member_metadata=java.nio.HeapByteBuffer[pos=0
> > > lim=22
> > > cap=250]},{member_id=consumer-19-59fa9871-2dcc-4713-9c4c-
> > > 6f0621f10afa,member_metadata=java.nio.HeapByteBuffer[pos=0
> > > lim=22
> > > cap=174]},{member_id=consumer-25-fa10026f-031e-49aa-9df1-
> > > 9d083b256ffa,member_metadata=java.nio.HeapByteBuffer[pos=0
> > > lim=22
> > > cap=98]},{member_id=consumer-26-f5a1293d-7d14-4519-8378-
> > > 1c4dd363ba09,member_metadata=java.nio.HeapByteBuffer[pos=0
> > > lim=22 cap=22]}]}
> > > [2018-08-09 11:43:26] :: DEBUG :: AbstractCoordinator:460 - Sending
> > > follower SyncGroup for group bulk-email-consumer to coordinator
> > > 10.189.179.116:9092 (id: 2147483647 rack: null):
> > > {group_id=bulk-email-consumer,generation_id=1225,member_id=
> > > consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09,group_assignment=[]}
> > > [2018-08-09 11:43:26] :: DEBUG :: AbstractCoordinator:460 - Sending
> > > follower SyncGroup for group bulk-email-consumer to coordinator
> > > 10.189.179.116:9092 (id: 2147483647 rack: null):
> > > {group_id=bulk-email-consumer,generation_id=1225,member_id=
> > > consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa,group_assignment=[]}
> > > [2018-08-09 11:43:26] :: DEBUG :: AbstractCoordinator:460 - Sending
> > > follower SyncGroup for group bulk-email-consumer to coordinator
> > > 10.189.179.116:9092 (id: 2147483647 rack: null):
> > > {group_id=bulk-email-consumer,generation_id=1225,member_id=
> > > consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa,group_assignment=[]}
> > > [2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:310 - Performing
> > > assignment for group bulk-email-consumer using strategy range with
> > > subscriptions
> > > {consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1=
> > > Subscription(topics=[bulk-email]),
> > > consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa=
> > > Subscription(topics=[bulk-email]),
> > > consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa=
> > > Subscription(topics=[bulk-email]),
> > > consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09=
> > > Subscription(topics=[bulk-email])}
> > > [2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:315 - Finished
> > > assignment for group bulk-email-consumer:
> > > {consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1=
> > > Assignment(partitions=[bulk-email-3,
> > > bulk-email-4, bulk-email-5]),
> > > consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa=
> > > Assignment(partitions=[bulk-email-0,
> > > bulk-email-1, bulk-email-2]),
> > > consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa=
> > > Assignment(partitions=[bulk-email-6,
> > > bulk-email-7]),
> > > consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09=
> > > Assignment(partitions=[bulk-email-8,
> > > bulk-email-9])}
> > > [2018-08-09 11:43:26] :: DEBUG :: AbstractCoordinator:471 - Sending
> > leader
> > > SyncGroup for group bulk-email-consumer to coordinator
> > 10.189.179.116:9092
> > > (id: 2147483647 rack: null):
> > > {group_id=bulk-email-consumer,generation_id=1225,member_id=
> > > consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,group_
> > > assignment=[{member_id=consumer-23-bdc073b6-1f90-
> > > 4ac5-a801-869377f9ead1,member_assignment=java.nio.HeapByteBuffer[pos=0
> > > lim=38
> > > cap=38]},{member_id=consumer-19-59fa9871-2dcc-4713-9c4c-
> > > 6f0621f10afa,member_assignment=java.nio.HeapByteBuffer[pos=0
> > > lim=38
> > > cap=38]},{member_id=consumer-25-fa10026f-031e-49aa-9df1-
> > > 9d083b256ffa,member_assignment=java.nio.HeapByteBuffer[pos=0
> > > lim=34
> > > cap=34]},{member_id=consumer-26-f5a1293d-7d14-4519-8378-
> > > 1c4dd363ba09,member_assignment=java.nio.HeapByteBuffer[pos=0
> > > lim=34 cap=34]}]}
> > > [2018-08-09 11:43:26] :: INFO :: AbstractCoordinator$1:349 -
> > Successfully
> > > joined group bulk-email-consumer with generation 1225
> > > [2018-08-09 11:43:26] :: INFO :: AbstractCoordinator$1:349 -
> > Successfully
> > > joined group bulk-email-consumer with generation 1225
> > > [2018-08-09 11:43:26] :: INFO :: AbstractCoordinator$1:349 -
> > Successfully
> > > joined group bulk-email-consumer with generation 1225
> > > [2018-08-09 11:43:26] :: INFO :: AbstractCoordinator$1:349 -
> > Successfully
> > > joined group bulk-email-consumer with generation 1225
> > > [2018-08-09 11:43:26] :: INFO :: ConsumerCoordinator:225 - Setting
> newly
> > > assigned partitions [bulk-email-7, bulk-email-6] for group
> > > bulk-email-consumer
> > > [2018-08-09 11:43:26] :: INFO :: ConsumerCoordinator:225 - Setting
> newly
> > > assigned partitions [bulk-email-0, bulk-email-2, bulk-email-1] for
> group
> > > bulk-email-consumer
> > > [2018-08-09 11:43:26] :: INFO :: ConsumerCoordinator:225 - Setting
> newly
> > > assigned partitions [bulk-email-4, bulk-email-3, bulk-email-5] for
> group
> > > bulk-email-consumer
> > > [2018-08-09 11:43:26] :: INFO :: ConsumerCoordinator:225 - Setting
> newly
> > > assigned partitions [bulk-email-8, bulk-email-9] for group
> > > bulk-email-consumer
> > > [2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:708 - Group
> > > bulk-email-consumer fetching committed offsets for partitions:
> > > [bulk-email-7, bulk-email-6]
> > > [2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:708 - Group
> > > bulk-email-consumer fetching committed offsets for partitions:
> > > [bulk-email-0, bulk-email-2, bulk-email-1]
> > > [2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:708 - Group
> > > bulk-email-consumer fetching committed offsets for partitions:
> > > [bulk-email-4, bulk-email-3, bulk-email-5]
> > > [2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:708 - Group
> > > bulk-email-consumer fetching committed offsets for partitions:
> > > [bulk-email-8, bulk-email-9]
> > > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
> > > partition bulk-email-7 to the committed offset 25435998
> > > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
> > > partition bulk-email-0 to the committed offset 25390842
> > > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
> > > partition bulk-email-6 to the committed offset 25463566
> > > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
> > > partition bulk-email-4 to the committed offset 25409183
> > > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
> > > partition bulk-email-8 to the committed offset 25465113
> > > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
> > > partition bulk-email-2 to the committed offset 25368327
> > > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :665 - Ignoring fetched
> > records
> > > for partition bulk-email-8 since it is no longer fetchable
> > > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
> > > partition bulk-email-3 to the committed offset 25422008
> > > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
> > > partition bulk-email-9 to the committed offset 25444488
> > > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
> > > partition bulk-email-1 to the committed offset 25369263
> > > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
> > > partition bulk-email-5 to the committed offset 25456518
> > > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :470 - Ignoring fetched
> > records
> > > for bulk-email-0 at offset 25390442 since the current position is
> > 25390842
> > > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :665 - Ignoring fetched
> > records
> > > for partition bulk-email-6 since it is no longer fetchable
> > > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :671 - Discarding stale
> fetch
> > > response for partition bulk-email-1 since its offset 25369219 does not
> > > match the expected offset 25369263
> > > [2018-08-09 11:43:29] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25465113 for partition
> bulk-email-8
> > > [2018-08-09 11:43:29] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25444488 for partition
> bulk-email-9
> > > [2018-08-09 11:43:29] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > metadata=''}, bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > metadata=''}}
> > > for group bulk-email-consumer
> > > [2018-08-09 11:43:29] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:43:29] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:43:29] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25436018 for partition
> bulk-email-7
> > > [2018-08-09 11:43:29] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:43:29] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25463566 for partition
> bulk-email-6
> > > [2018-08-09 11:43:29] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:43:29] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25409183 for partition
> bulk-email-4
> > > [2018-08-09 11:43:29] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25422008 for partition
> bulk-email-3
> > > [2018-08-09 11:43:29] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25456530 for partition
> bulk-email-5
> > > [2018-08-09 11:43:29] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25390842 for partition
> bulk-email-0
> > > [2018-08-09 11:43:29] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25368339 for partition
> bulk-email-2
> > > [2018-08-09 11:43:29] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25369263 for partition
> bulk-email-1
> > > [2018-08-09 11:43:29] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-4=OffsetAndMetadata{offset=25409183,
> > > metadata=''}, bulk-email-3=OffsetAndMetadata{offset=25422008,
> > > metadata=''},
> > > bulk-email-5=OffsetAndMetadata{offset=25456530, metadata=''}} for
> group
> > > bulk-email-consumer
> > >
> > > On Thu, Jul 12, 2018 at 9:27 PM Steve Tian <steve.cs.t...@gmail.com>
> > > wrote:
> > >
> > > > It's a very good and important doc so I think you should read it all.
> > > You
> > > > should get some idea from sections like *Detecting Consumer Failures*
> > and
> > > > *Multi-threaded Processing* for your case.
> > > >
> > > > On Thu, Jul 12, 2018, 3:17 PM Shantanu Deshmukh <
> shantanu...@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Hi Steve,
> > > > >
> > > > > Could you please shed more light on this? What section should I
> > > revisit?
> > > > I
> > > > > am using high-level consumer. So I am simply calling
> consumer.close()
> > > > when
> > > > > I am shutting down the process. Is there any other method to be
> > called
> > > > > before calling close()?
> > > > >
> > > > > On Mon, Jul 9, 2018 at 5:58 PM Steve Tian <steve.cs.t...@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > > Please re-read the javadoc of KafkaConsumer, make sure you know
> how
> > > to
> > > > > > wakeup/close consumer properly while shutting down your
> > application.
> > > > Try
> > > > > > to understand the motivation of KIP-62 and adjust related
> timeout.
> > > > > >
> > > > > > On Mon, Jul 9, 2018, 8:05 PM harish lohar <hklo...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > Try reducing below timer
> > > > > > > metadata.max.age.ms = 300000
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Jul 6, 2018 at 5:55 AM Shantanu Deshmukh <
> > > > > shantanu...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hello everyone,
> > > > > > > >
> > > > > > > > We are running a 3 broker Kafka 0.10.0.1 cluster. We have a
> > java
> > > > app
> > > > > > > which
> > > > > > > > spawns many consumer threads consuming from different topics.
> > For
> > > > > every
> > > > > > > > topic we have specified different consumer-group. A lot of
> > times
> > > I
> > > > > see
> > > > > > > that
> > > > > > > > whenever this application is restarted a CG on one or two
> > topics
> > > > > takes
> > > > > > > more
> > > > > > > > than 5 minutes to receive partition assignment. Till that
> time
> > > > > > consumers
> > > > > > > > for that topic don't consumer anything. If I go to Kafka
> broker
> > > and
> > > > > run
> > > > > > > > consumer-groups.sh and describe that particular CG I see that
> > it
> > > is
> > > > > > > > rebalancing. There is time critical data stored in that topic
> > and
> > > > we
> > > > > > > cannot
> > > > > > > > tolerate such long delays. What can be the reason for such
> long
> > > > > > > rebalances.
> > > > > > > >
> > > > > > > > Here's our consumer config
> > > > > > > >
> > > > > > > >
> > > > > > > > auto.commit.interval.ms = 3000
> > > > > > > > auto.offset.reset = latest
> > > > > > > > bootstrap.servers = [x.x.x.x:9092, x.x.x.x:9092,
> x.x.x.x:9092]
> > > > > > > > check.crcs = true
> > > > > > > > client.id =
> > > > > > > > connections.max.idle.ms = 540000
> > > > > > > > enable.auto.commit = true
> > > > > > > > exclude.internal.topics = true
> > > > > > > > fetch.max.bytes = 52428800
> > > > > > > > fetch.max.wait.ms = 500
> > > > > > > > fetch.min.bytes = 1
> > > > > > > > group.id = otp-notifications-consumer
> > > > > > > > heartbeat.interval.ms = 3000
> > > > > > > > interceptor.classes = null
> > > > > > > > key.deserializer = class
> > > > > > > > org.apache.kafka.common.serialization.StringDeserializer
> > > > > > > > max.partition.fetch.bytes = 1048576
> > > > > > > > max.poll.interval.ms = 300000
> > > > > > > > max.poll.records = 50
> > > > > > > > metadata.max.age.ms = 300000
> > > > > > > > metric.reporters = []
> > > > > > > > metrics.num.samples = 2
> > > > > > > > metrics.sample.window.ms = 30000
> > > > > > > > partition.assignment.strategy = [class
> > > > > > > > org.apache.kafka.clients.consumer.RangeAssignor]
> > > > > > > > receive.buffer.bytes = 65536
> > > > > > > > reconnect.backoff.ms = 50
> > > > > > > > request.timeout.ms = 305000
> > > > > > > > retry.backoff.ms = 100
> > > > > > > > sasl.kerberos.kinit.cmd = /usr/bin/kinit
> > > > > > > > sasl.kerberos.min.time.before.relogin = 60000
> > > > > > > > sasl.kerberos.service.name = null
> > > > > > > > sasl.kerberos.ticket.renew.jitter = 0.05
> > > > > > > > sasl.kerberos.ticket.renew.window.factor = 0.8
> > > > > > > > sasl.mechanism = GSSAPI
> > > > > > > > security.protocol = SSL
> > > > > > > > send.buffer.bytes = 131072
> > > > > > > > session.timeout.ms = 300000
> > > > > > > > ssl.cipher.suites = null
> > > > > > > > ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
> > > > > > > > ssl.endpoint.identification.algorithm = null
> > > > > > > > ssl.key.password = null
> > > > > > > > ssl.keymanager.algorithm = SunX509
> > > > > > > > ssl.keystore.location = null
> > > > > > > > ssl.keystore.password = null
> > > > > > > > ssl.keystore.type = JKS
> > > > > > > > ssl.protocol = TLS
> > > > > > > > ssl.provider = null
> > > > > > > > ssl.secure.random.implementation = null
> > > > > > > > ssl.trustmanager.algorithm = PKIX
> > > > > > > > ssl.truststore.location = /x/x/client.truststore.jks
> > > > > > > > ssl.truststore.password = [hidden]
> > > > > > > > ssl.truststore.type = JKS
> > > > > > > > value.deserializer = class
> > > > > > > > org.apache.kafka.common.serialization.StringDeserializer
> > > > > > > >
> > > > > > > > Please help.
> > > > > > > >
> > > > > > > > *Thanks & Regards,*
> > > > > > > > *Shantanu Deshmukh*
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
