I saw a few topics with segment.ms and retention.ms property set. Can that
be causing any issue? I remember that this is the only change I carried out
to the cluster in last couple of months after which the problem started.
On Fri, Aug 10, 2018 at 2:55 PM M. Manna <manme...@gmail.com> wrote:
> if you can upgrade, I would say upgrading to 0.10.2.x would be better for
> you (or even higher, 2.0.0). Otherwise you have to play around with
> max.poll.records and session.timeout.ms.
>
> As the doc says (or newer versions), the adjustment should be such that
> request.timeout.ms >= max.poll.interval.ms. Also, heartbeat.interval.ms
> should be curbed at (rule of thumb) 30% of session.timeout.ms.
>
> Lastly, all these have to be within the bounds of
> group.min.session.timeout.ms and group.max.session.timeout.ms.
>
> You can check all these, tune them as necessary, and retry. Some of these
> configs may or may not be applicable at runtime. so a rolling restart may
> be required before all changes take place.
>
> On 9 August 2018 at 13:48, Shantanu Deshmukh <shantanu...@gmail.com>
> wrote:
>
> > Hi,
> >
> > Yes my consumer application works like below
> >
> > 1. Reads how many workers are required to process each topics from
> > properties file
> > 2. As many threads are spawned as there are workers mentioned in
> > properties file, topic name is passed to this thread. FixedThreadPool
> > implementation is used.
> > 3. Each worker thread initializes one consumer object and subscribes
> to
> > given topic. Consumer group is simply <topic-name>-consumer. So if my
> > topic
> > bulk-email, then consumer group for all those threads is
> > bulk-email-consumer
> > 4. Once this is done, inside an infinite while loop consumer.poll(100)
> > method keeps running. So this application is a daemon. Shuts down only
> > when
> > server shuts down or in case of kill command.
> >
> > I have configured session.timeout.ms in consumer properties. I haven't
> > done
> > anything about zookeeper timeout. Is it required now? Since consumer
> > accesses only the brokers.
> >
> > On Thu, Aug 9, 2018 at 3:03 PM M. Manna <manme...@gmail.com> wrote:
> >
> > > In the simplest way, how have you implemented your consumer?
> > >
> > > 1) Does your consumers join a designated group, process messages, and
> > then
> > > closes all connection? Or does it stay open perpetually until server
> > > shutdown?
> > > 2) Have you configured the session timeouts for client and zookeeper
> > > accordingly?
> > >
> > > Regards,
> > >
> > > On 9 August 2018 at 08:00, Shantanu Deshmukh <shantanu...@gmail.com>
> > > wrote:
> > >
> > > > I am facing too many problems these days. Now one of our consumer
> > groups
> > > > is rebalancing every now and then. And rebalance takes very low, more
> > > than
> > > > 5-10 minutes. Even after re-balancing I see that only half of the
> > > consumers
> > > > are active/receive assignment. Its all going haywire.
> > > >
> > > > I am seeing these logs in kafka consumer logs. Can anyone help me
> > > > understand what is going on here? It is a very long piece of log, but
> > > > someone please help me. I am desperately looking for any solution
> since
> > > > more than 2 months now. But to no avail.
> > > >
> > > > [2018-08-09 11:39:51] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:39:53] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25465113 for partition
> > bulk-email-8
> > > > [2018-08-09 11:39:53] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:39:53] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25463566 for partition
> > bulk-email-6
> > > > [2018-08-09 11:39:53] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:39:53] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25444488 for partition
> > bulk-email-9
> > > > [2018-08-09 11:39:53] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:39:54] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:39:54] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:39:54] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:39:54] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:39:54] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:39:54] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:39:54] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:39:54] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:39:56] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25463566 for partition
> > bulk-email-6
> > > > [2018-08-09 11:39:56] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25444488 for partition
> > bulk-email-9
> > > > [2018-08-09 11:39:56] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25465113 for partition
> > bulk-email-8
> > > > [2018-08-09 11:39:56] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:39:56] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:39:56] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:39:57] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:39:57] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:39:57] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:39:57] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:39:57] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:39:57] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:39:57] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:39:57] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:00] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:00] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:00] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:03] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:03] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:03] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:664 - Offset commit
> > for
> > > > group bulk-email-consumer failed: The request timed out.
> > > > [2018-08-09 11:40:04] :: INFO :: AbstractCoordinator:600 - Marking
> the
> > > > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) dead for
> > > group
> > > > bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: WARN :: ConsumerCoordinator$4:535 - Auto
> > offset
> > > > commit failed for group bulk-email-consumer: Offset commit failed
> with
> > a
> > > > retriable exception. You should retry committing offsets.
> > > > [2018-08-09 11:40:04] :: DEBUG :: AbstractCoordinator:531 - Sending
> > > > coordinator request for group bulk-email-consumer to broker
> > > > 10.189.179.117:9092 (id: 1 rack: null)
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:664 - Offset commit
> > for
> > > > group bulk-email-consumer failed: The request timed out.
> > > > [2018-08-09 11:40:04] :: INFO :: AbstractCoordinator:600 - Marking
> the
> > > > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) dead for
> > > group
> > > > bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: WARN :: ConsumerCoordinator$4:535 - Auto
> > offset
> > > > commit failed for group bulk-email-consumer: Offset commit failed
> with
> > a
> > > > retriable exception. You should retry committing offsets.
> > > > [2018-08-09 11:40:04] :: DEBUG :: AbstractCoordinator:531 - Sending
> > > > coordinator request for group bulk-email-consumer to broker
> > > > 10.189.179.118:9092 (id: 2 rack: null)
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > AbstractCoordinator$GroupCoordinatorResponseHandler:541 - Received
> > group
> > > > coordinator response ClientResponse(receivedTimeMs=1533795004158,
> > > > disconnected=false, request=ClientRequest(expectResponse=true,
> > > > callback=org.apache.kafka.clients.consumer.internals.
> > > > ConsumerNetworkClient$RequestFutureCompletionHandler@3a3e5740,
> > > > request=RequestSend(header={api_key=10,api_version=0,
> > > > correlation_id=389784,client_id=consumer-26},
> > > > body={group_id=bulk-email-consumer}), createdTimeMs=1533795004157,
> > > > sendTimeMs=1533795004157),
> > > > responseBody={error_code=0,coordinator={node_id=0,host=
> > > > 10.189.179.116,port=9092}})
> > > > [2018-08-09 11:40:04] :: INFO ::
> > > > AbstractCoordinator$GroupCoordinatorResponseHandler:555 - Discovered
> > > > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) for
> group
> > > > bulk-emai -consumer.
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25463566 for partition
> > bulk-email-6
> > > > [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > AbstractCoordinator$GroupCoordinatorResponseHandler:541 - Received
> > group
> > > > coordinator response ClientResponse(receivedTimeMs=1533795004718,
> > > > disconnected=false, request=ClientRequest(expectResponse=true,
> > > > callback=org.apache.kafka.clients.consumer.internals.
> > > > ConsumerNetworkClient$RequestFutureCompletionHandler@66889bbd,
> > > > request=RequestSend(header={api_key=10,api_version=0,
> > > > correlation_id=363859,client_id=consumer-25},
> > > > body={group_id=bulk-email-consumer}), createdTimeMs=1533795004145,
> > > > sendTimeMs=1533795004145),
> > > > responseBody={error_code=0,coordinator={node_id=0,host=
> > > > 10.189.179.116,port=9092}})
> > > > [2018-08-09 11:40:04] :: INFO ::
> > > > AbstractCoordinator$GroupCoordinatorResponseHandler:555 - Discovered
> > > > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) for
> group
> > > > bulk-emai -consumer.
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25444488 for partition
> > bulk-email-9
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25465113 for partition
> > bulk-email-8
> > > > [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25463566 for partition
> > bulk-email-6
> > > > [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25465113 for partition
> > bulk-email-8
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25444488 for partition
> > bulk-email-9
> > > > [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:05] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25463566 for partition
> > bulk-email-6
> > > > [2018-08-09 11:40:05] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:06] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:06] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:06] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:06] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:06] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:06] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:07] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:07] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25444488 for partition
> > bulk-email-9
> > > > [2018-08-09 11:40:07] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:07] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:07] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25465113 for partition
> > bulk-email-8
> > > > [2018-08-09 11:40:07] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:08] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25463566 for partition
> > bulk-email-6
> > > > [2018-08-09 11:40:08] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:09] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:09] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:09] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:09] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:09] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:09] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:10] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:10] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25444488 for partition
> > bulk-email-9
> > > > [2018-08-09 11:40:10] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:10] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25465113 for partition
> > bulk-email-8
> > > > [2018-08-09 11:40:10] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:10] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:11] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25463566 for partition
> > bulk-email-6
> > > > [2018-08-09 11:40:11] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:12] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:12] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:12] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:12] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:12] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:12] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:13] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25444488 for partition
> > bulk-email-9
> > > > [2018-08-09 11:40:13] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:13] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:14] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25465113 for partition
> > bulk-email-8
> > > > [2018-08-09 11:40:14] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25463566 for partition
> > bulk-email-6
> > > > [2018-08-09 11:40:14] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:14] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:14] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:15] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:15] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:15] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:15] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:15] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:15] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:16] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25444488 for partition
> > bulk-email-9
> > > > [2018-08-09 11:40:16] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:16] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:16] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25465113 for partition
> > bulk-email-8
> > > > [2018-08-09 11:40:16] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:16] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:17] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25463566 for partition
> > bulk-email-6
> > > > [2018-08-09 11:40:17] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:18] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:18] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:18] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:18] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:18] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:18] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:19] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25444488 for partition
> > bulk-email-9
> > > > [2018-08-09 11:40:19] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:19] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:19] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25465113 for partition
> > bulk-email-8
> > > > [2018-08-09 11:40:19] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:19] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:20] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25463566 for partition
> > bulk-email-6
> > > > [2018-08-09 11:40:20] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:21] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:21] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:21] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:21] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:21] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:21] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:21] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25463566 for partition
> > bulk-email-6
> > > > [2018-08-09 11:40:21] :: INFO :: ConsumerCoordinator:333 - Revoking
> > > > previously assigned partitions [bulk-email-6] for group
> > > bulk-email-consumer
> > > > [2018-08-09 11:40:21] :: INFO :: AbstractCoordinator:381 -
> > (Re-)joining
> > > > group bulk-email-consumer
> > > > [2018-08-09 11:40:21] :: DEBUG :: AbstractCoordinator:390 - Sending
> > > > JoinGroup
> > > > ({group_id=bulk-email-consumer,session_timeout=
> > 300000,rebalance_timeout=
> > > > 300000,member_id=consumer-23-bdc073b6-1f90-4ac5-a801-
> > > > 869377f9ead1,protocol_type=consumer,group_protocols=[{
> > > > protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
> > > > lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id:
> 2147483647
> > > > rack:
> > > > null)
> > > > [2018-08-09 11:40:22] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25444488 for partition
> > bulk-email-9
> > > > [2018-08-09 11:40:22] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:22] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25465113 for partition
> > bulk-email-8
> > > > [2018-08-09 11:40:22] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:22] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:22] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:22] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25444488 for partition
> > bulk-email-9
> > > > [2018-08-09 11:40:22] :: INFO :: ConsumerCoordinator:333 - Revoking
> > > > previously assigned partitions [bulk-email-9] for group
> > > bulk-email-consumer
> > > > [2018-08-09 11:40:22] :: INFO :: AbstractCoordinator:381 -
> > (Re-)joining
> > > > group bulk-email-consumer
> > > > [2018-08-09 11:40:22] :: DEBUG :: AbstractCoordinator:390 - Sending
> > > > JoinGroup
> > > > ({group_id=bulk-email-consumer,session_timeout=
> > 300000,rebalance_timeout=
> > > > 300000,member_id=consumer-26-f5a1293d-7d14-4519-8378-
> > > > 1c4dd363ba09,protocol_type=consumer,group_protocols=[{
> > > > protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
> > > > lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id:
> 2147483647
> > > > rack:
> > > > null)
> > > > [2018-08-09 11:40:22] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25465113 for partition
> > bulk-email-8
> > > > [2018-08-09 11:40:23] :: INFO :: ConsumerCoordinator:333 - Revoking
> > > > previously assigned partitions [bulk-email-8] for group
> > > bulk-email-consumer
> > > > [2018-08-09 11:40:23] :: INFO :: AbstractCoordinator:381 -
> > (Re-)joining
> > > > group bulk-email-consumer
> > > > [2018-08-09 11:40:23] :: DEBUG :: AbstractCoordinator:390 - Sending
> > > > JoinGroup
> > > > ({group_id=bulk-email-consumer,session_timeout=
> > 300000,rebalance_timeout=
> > > > 300000,member_id=consumer-25-fa10026f-031e-49aa-9df1-
> > > > 9d083b256ffa,protocol_type=consumer,group_protocols=[{
> > > > protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
> > > > lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id:
> 2147483647
> > > > rack:
> > > > null)
> > > > [2018-08-09 11:40:24] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:24] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:24] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:24] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:24] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:26] :: DEBUG ::
> > > NetworkClient$DefaultMetadataUpdater:640
> > > > - Sending metadata request {topics=[bulk-email]} to node 1
> > > > [2018-08-09 11:40:26] :: DEBUG :: Metadata :241 - Updated cluster
> > > metadata
> > > > version 642 to Cluster(id = 09FjKmqPQvat0c0dyPptIw, nodes = [
> > > > 10.189.179.118:9092 (id: 2 rack: null), 10.189.179.116:9092 (id: 0
> > rack:
> > > > null), 10.189.179.117:9092 (id: 1 rack: null)], partitions =
> > > > [Partition(topic = bulk-email, partition = 8, leader = 2, replicas =
> > > > [0,1,2,], isr = [0,2,1,]), Partition(topic = bulk-email, partition =
> 7,
> > > > leader = 1, replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic =
> > > > bulk-email, partition = 9, leader = 0, replicas = [0,1,2,], isr =
> > > > [0,1,2,]), Partition(topic = bulk-email, partition = 4, leader = 1,
> > > > replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic = bulk-email,
> > > > partition = 3, leader = 0, replicas = [0,1,2,], isr = [0,1,2,]),
> > > > Partition(topic = bulk-email, partition = 6, leader = 0, replicas =
> > > > [0,1,2,], isr = [0,1,2,]), Partition(topic = bulk-email, partition =
> 5,
> > > > leader = 2, replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic =
> > > > bulk-email, partition = 0, leader = 0, replicas = [0,1,2,], isr =
> > > > [0,1,2,]), Partition(topic = bulk-email, partition = 2, leader = 2,
> > > > replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic = bulk-email,
> > > > partition = 1, leader = 1, replicas = [0,1,2,], isr = [1,0,2,])])
> > > > [2018-08-09 11:40:27] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:27] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:27] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:27] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:27] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:28] :: DEBUG ::
> > > NetworkClient$DefaultMetadataUpdater:640
> > > > - Sending metadata request {topics=[bulk-email]} to node 1
> > > > [2018-08-09 11:40:28] :: DEBUG :: Metadata :241 - Updated cluster
> > > metadata
> > > > version 641 to Cluster(id = 09FjKmqPQvat0c0dyPptIw, nodes = [
> > > > 10.189.179.116:9092 (id: 0 rack: null), 10.189.179.118:9092 (id: 2
> > rack:
> > > > null), 10.189.179.117:9092 (id: 1 rack: null)], partitions =
> > > > [Partition(topic = bulk-email, partition = 8, leader = 2, replicas =
> > > > [0,1,2,], isr = [0,2,1,]), Partition(topic = bulk-email, partition =
> 7,
> > > > leader = 1, replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic =
> > > > bulk-email, partition = 9, leader = 0, replicas = [0,1,2,], isr =
> > > > [0,1,2,]), Partition(topic = bulk-email, partition = 4, leader = 1,
> > > > replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic = bulk-email,
> > > > partition = 3, leader = 0, replicas = [0,1,2,], isr = [0,1,2,]),
> > > > Partition(topic = bulk-email, partition = 6, leader = 0, replicas =
> > > > [0,1,2,], isr = [0,1,2,]), Partition(topic = bulk-email, partition =
> 5,
> > > > leader = 2, replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic =
> > > > bulk-email, partition = 0, leader = 0, replicas = [0,1,2,], isr =
> > > > [0,1,2,]), Partition(topic = bulk-email, partition = 2, leader = 2,
> > > > replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic = bulk-email,
> > > > partition = 1, leader = 1, replicas = [0,1,2,], isr = [1,0,2,])])
> > > > [2018-08-09 11:40:30] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:30] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:30] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:30] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:31] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:33] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:33] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:33] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:33] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:33] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:34] :: DEBUG ::
> > > NetworkClient$DefaultMetadataUpdater:640
> > > > - Sending metadata request {topics=[bulk-email]} to node 1
> > > > [2018-08-09 11:40:34] :: DEBUG :: Metadata :241 - Updated cluster
> > > metadata
> > > > version 646 to Cluster(id = 09FjKmqPQvat0c0dyPptIw, nodes = [
> > > > 10.189.179.117:9092 (id: 1 rack: null), 10.189.179.116:9092 (id: 0
> > rack:
> > > > null), 10.189.179.118:9092 (id: 2 rack: null)], partitions =
> > > > [Partition(topic = bulk-email, partition = 8, leader = 2, replicas =
> > > > [0,1,2,], isr = [0,2,1,]), Partition(topic = bulk-email, partition =
> 7,
> > > > leader = 1, replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic =
> > > > bulk-email, partition = 9, leader = 0, replicas = [0,1,2,], isr =
> > > > [0,1,2,]), Partition(topic = bulk-email, partition = 4, leader = 1,
> > > > replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic = bulk-email,
> > > > partition = 3, leader = 0, replicas = [0,1,2,], isr = [0,1,2,]),
> > > > Partition(topic = bulk-email, partition = 6, leader = 0, replicas =
> > > > [0,1,2,], isr = [0,1,2,]), Partition(topic = bulk-email, partition =
> 5,
> > > > leader = 2, replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic =
> > > > bulk-email, partition = 0, leader = 0, replicas = [0,1,2,], isr =
> > > > [0,1,2,]), Partition(topic = bulk-email, partition = 2, leader = 2,
> > > > replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic = bulk-email,
> > > > partition = 1, leader = 1, replicas = [0,1,2,], isr = [1,0,2,])])
> > > > [2018-08-09 11:40:36] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:36] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:36] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:36] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:36] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:39] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:39] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:39] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:39] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:39] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:42] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:42] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:42] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:42] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:42] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:45] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:45] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:45] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:45] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:45] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:48] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:48] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:48] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:48] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:48] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:50] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-0=OffsetAndMetadata{offset=25390438,
> > > > metadata=''}, bulk-email-2=OffsetAndMetadata{offset=25368327,
> > > > metadata=''},
> > > > bulk-email-1=OffsetAndMetadata{offset=25369219, metadata=''}} for
> > group
> > > > bulk-email-consumer
> > > > [2018-08-09 11:40:50] :: WARN :: ConsumerCoordinator:554 - Auto
> offset
> > > > commit failed for group bulk-email-consumer: Commit cannot be
> completed
> > > > since the group has already rebalanced and assigned the partitions to
> > > > another member. This means that the time between subsequent calls to
> > > poll()
> > > > was longer than the configured max.poll.interval.ms, which typically
> > > > implies that the poll loop is spending too much time message
> > processing.
> > > > You can address this either by increasing the session timeout or by
> > > > reducing the maximum size of batches returned in poll() with
> > > > max.poll.records.
> > > > [2018-08-09 11:40:50] :: INFO :: ConsumerCoordinator:333 - Revoking
> > > > previously assigned partitions [bulk-email-0, bulk-email-2,
> > bulk-email-1]
> > > > for group bulk-email-consumer
> > > > [2018-08-09 11:40:50] :: INFO :: AbstractCoordinator:381 -
> > (Re-)joining
> > > > group bulk-email-consumer
> > > > [2018-08-09 11:40:50] :: DEBUG :: AbstractCoordinator:390 - Sending
> > > > JoinGroup
> > > > ({group_id=bulk-email-consumer,session_timeout=
> > 300000,rebalance_timeout=
> > > > 300000,member_id=,protocol_type=consumer,group_protocols=
> > > > [{protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
> > > > lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id:
> 2147483647
> > > > rack:
> > > > null)
> > > > [2018-08-09 11:40:50] :: DEBUG ::
> > > NetworkClient$DefaultMetadataUpdater:640
> > > > - Sending metadata request {topics=[bulk-email]} to node 0
> > > > [2018-08-09 11:40:50] :: DEBUG ::
> > > > ConsumerNetworkClient$RequestFutureCompletionHandler:475 - Cancelled
> > > > LEAVE_GROUP request ClientRequest(expectResponse=true,
> > > > callback=org.apache.kafka.clients.consumer.internals.
> > > > ConsumerNetworkClient$RequestFutureCompletionHandler@60f41e0b,
> > > > request=RequestSend(header={api_key=13,api_version=0,
> > > > correlation_id=349313,client_id=consumer-19},
> > > > body={group_id=bulk-email-consumer,member_id=consumer-
> > > > 19-34903180-9cb3-496d-a913-b782476d9a52}),
> > > > createdTimeMs=1533794419195, sendTimeMs=1533794419195) with
> correlation
> > > id
> > > > 349313 due to node 2147483647 being disconnected
> > > > [2018-08-09 11:40:50] :: INFO :: AbstractCoordinator:600 - Marking
> the
> > > > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) dead for
> > > group
> > > > bulk-email-consumer
> > > > [2018-08-09 11:40:50] :: DEBUG ::
> > > > ConsumerNetworkClient$RequestFutureCompletionHandler:475 - Cancelled
> > > > JOIN_GROUP request ClientRequest(expectResponse=true,
> > > > callback=org.apache.kafka.clients.consumer.internals.
> > > > ConsumerNetworkClient$RequestFutureCompletionHandler@3bd6f5c9,
> > > > request=RequestSend(header={api_key=11,api_version=1,
> > > > correlation_id=349314,client_id=consumer-19},
> > > > body={group_id=bulk-email-consumer,session_timeout=
> > > > 300000,rebalance_timeout=300000,member_id=,protocol_
> > > > type=consumer,group_protocols=[{protocol_name=range,
> > > > protocol_metadata=java.nio.HeapByteBuffer[pos=0
> > > > lim=22 cap=22]}]}), createdTimeMs=1533795050857, sendTimeMs=0) with
> > > > correlation id 349314 due to node 2147483647 being disconnected
> > > > [2018-08-09 11:40:50] :: INFO :: AbstractCoordinator:600 - Marking
> the
> > > > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) dead for
> > > group
> > > > bulk-email-consumer
> > > > [2018-08-09 11:40:50] :: DEBUG :: AbstractCoordinator:531 - Sending
> > > > coordinator request for group bulk-email-consumer to broker
> > > > 10.189.179.118:9092 (id: 2 rack: null)
> > > > [2018-08-09 11:40:51] :: DEBUG ::
> > > NetworkClient$DefaultMetadataUpdater:640
> > > > - Sending metadata request {topics=[bulk-email]} to node 2
> > > > [2018-08-09 11:40:51] :: DEBUG :: Metadata :241 - Updated cluster
> > > metadata
> > > > version 641 to Cluster(id = 09FjKmqPQvat0c0dyPptIw, nodes = [
> > > > 10.189.179.117:9092 (id: 1 rack: null), 10.189.179.116:9092 (id: 0
> > rack:
> > > > null), 10.189.179.118:9092 (id: 2 rack: null)], partitions =
> > > > [Partition(topic = bulk-email, partition = 8, leader = 2, replicas =
> > > > [0,1,2,], isr = [0,2,1,]), Partition(topic = bulk-email, partition =
> 7,
> > > > leader = 1, replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic =
> > > > bulk-email, partition = 9, leader = 0, replicas = [0,1,2,], isr =
> > > > [0,1,2,]), Partition(topic = bulk-email, partition = 4, leader = 1,
> > > > replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic = bulk-email,
> > > > partition = 3, leader = 0, replicas = [0,1,2,], isr = [0,1,2,]),
> > > > Partition(topic = bulk-email, partition = 6, leader = 0, replicas =
> > > > [0,1,2,], isr = [0,1,2,]), Partition(topic = bulk-email, partition =
> 5,
> > > > leader = 2, replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic =
> > > > bulk-email, partition = 0, leader = 0, replicas = [0,1,2,], isr =
> > > > [0,1,2,]), Partition(topic = bulk-email, partition = 2, leader = 2,
> > > > replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic = bulk-email,
> > > > partition = 1, leader = 1, replicas = [0,1,2,], isr = [1,0,2,])])
> > > > [2018-08-09 11:40:51] :: DEBUG ::
> > > > AbstractCoordinator$GroupCoordinatorResponseHandler:541 - Received
> > group
> > > > coordinator response ClientResponse(receivedTimeMs=1533795051063,
> > > > disconnected=false, request=ClientRequest(expectResponse=true,
> > > > callback=org.apache.kafka.clients.consumer.internals.
> > > > ConsumerNetworkClient$RequestFutureCompletionHandler@21a34a4e,
> > > > request=RequestSend(header={api_key=10,api_version=0,
> > > > correlation_id=349316,client_id=consumer-19},
> > > > body={group_id=bulk-email-consumer}), createdTimeMs=1533795050961,
> > > > sendTimeMs=1533795051061),
> > > > responseBody={error_code=0,coordinator={node_id=0,host=
> > > > 10.189.179.116,port=9092}})
> > > > [2018-08-09 11:40:51] :: INFO ::
> > > > AbstractCoordinator$GroupCoordinatorResponseHandler:555 - Discovered
> > > > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) for
> group
> > > > bulk-emai -consumer.
> > > > [2018-08-09 11:40:51] :: INFO :: AbstractCoordinator:381 -
> > (Re-)joining
> > > > group bulk-email-consumer
> > > > [2018-08-09 11:40:51] :: DEBUG :: AbstractCoordinator:390 - Sending
> > > > JoinGroup
> > > > ({group_id=bulk-email-consumer,session_timeout=
> > 300000,rebalance_timeout=
> > > > 300000,member_id=,protocol_type=consumer,group_protocols=
> > > > [{protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
> > > > lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id:
> 2147483647
> > > > rack:
> > > > null)
> > > > [2018-08-09 11:40:51] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:51] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:51] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:51] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:54] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:54] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:54] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:54] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:57] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:57] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:57] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:57] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:00] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:00] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:03] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:03] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:06] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:06] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:09] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:09] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:12] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:15] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:17] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:17] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:18] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:18] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:21] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:21] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:24] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:24] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:27] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:27] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:31] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:31] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:33] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:33] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:36] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:36] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:39] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:39] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:42] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:42] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:46] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:46] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:48] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:48] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:51] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:51] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:54] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:54] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:57] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:57] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:01] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:03] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:06] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:09] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:12] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:15] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:18] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:21] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:24] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:27] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:31] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:34] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:37] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:40] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:43] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:46] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:49] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:52] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:55] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:58] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:43:01] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:43:04] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:43:07] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:43:10] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:43:13] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:43:16] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:43:19] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:43:22] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:43:25] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:43:26] :: DEBUG ::
> > > > AbstractCoordinator$JoinGroupResponseHandler:406 - Received
> successful
> > > > join
> > > > group response for group bulk-email-consumer:
> > > > {error_code=0,generation_id=1225,group_protocol=range,
> > > > leader_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_id=
> > > > consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09,members=[]}
> > > > [2018-08-09 11:43:26] :: DEBUG ::
> > > > AbstractCoordinator$JoinGroupResponseHandler:406 - Received
> successful
> > > > join
> > > > group response for group bulk-email-consumer:
> > > > {error_code=0,generation_id=1225,group_protocol=range,
> > > > leader_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_id=
> > > > consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa,members=[]}
> > > > [2018-08-09 11:43:26] :: DEBUG ::
> > > > AbstractCoordinator$JoinGroupResponseHandler:406 - Received
> successful
> > > > join
> > > > group response for group bulk-email-consumer:
> > > > {error_code=0,generation_id=1225,group_protocol=range,
> > > > leader_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_id=
> > > > consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa,members=[]}
> > > > [2018-08-09 11:43:26] :: DEBUG ::
> > > > AbstractCoordinator$JoinGroupResponseHandler:406 - Received
> successful
> > > > join
> > > > group response for group bulk-email-consumer:
> > > > {error_code=0,generation_id=1225,group_protocol=range,
> > > > leader_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_id=
> > > > consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,
> > > > members=[{member_id=consumer-23-bdc073b6-1f90-4ac5-a801-
> > > > 869377f9ead1,member_metadata=java.nio.HeapByteBuffer[pos=0
> > > > lim=22
> > > > cap=250]},{member_id=consumer-19-59fa9871-2dcc-4713-9c4c-
> > > > 6f0621f10afa,member_metadata=java.nio.HeapByteBuffer[pos=0
> > > > lim=22
> > > > cap=174]},{member_id=consumer-25-fa10026f-031e-49aa-9df1-
> > > > 9d083b256ffa,member_metadata=java.nio.HeapByteBuffer[pos=0
> > > > lim=22
> > > > cap=98]},{member_id=consumer-26-f5a1293d-7d14-4519-8378-
> > > > 1c4dd363ba09,member_metadata=java.nio.HeapByteBuffer[pos=0
> > > > lim=22 cap=22]}]}
> > > > [2018-08-09 11:43:26] :: DEBUG :: AbstractCoordinator:460 - Sending
> > > > follower SyncGroup for group bulk-email-consumer to coordinator
> > > > 10.189.179.116:9092 (id: 2147483647 rack: null):
> > > > {group_id=bulk-email-consumer,generation_id=1225,member_id=
> > > > consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09,group_assignment=[]}
> > > > [2018-08-09 11:43:26] :: DEBUG :: AbstractCoordinator:460 - Sending
> > > > follower SyncGroup for group bulk-email-consumer to coordinator
> > > > 10.189.179.116:9092 (id: 2147483647 rack: null):
> > > > {group_id=bulk-email-consumer,generation_id=1225,member_id=
> > > > consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa,group_assignment=[]}
> > > > [2018-08-09 11:43:26] :: DEBUG :: AbstractCoordinator:460 - Sending
> > > > follower SyncGroup for group bulk-email-consumer to coordinator
> > > > 10.189.179.116:9092 (id: 2147483647 rack: null):
> > > > {group_id=bulk-email-consumer,generation_id=1225,member_id=
> > > > consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa,group_assignment=[]}
> > > > [2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:310 -
> Performing
> > > > assignment for group bulk-email-consumer using strategy range with
> > > > subscriptions
> > > > {consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1=
> > > > Subscription(topics=[bulk-email]),
> > > > consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa=
> > > > Subscription(topics=[bulk-email]),
> > > > consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa=
> > > > Subscription(topics=[bulk-email]),
> > > > consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09=
> > > > Subscription(topics=[bulk-email])}
> > > > [2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:315 - Finished
> > > > assignment for group bulk-email-consumer:
> > > > {consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1=
> > > > Assignment(partitions=[bulk-email-3,
> > > > bulk-email-4, bulk-email-5]),
> > > > consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa=
> > > > Assignment(partitions=[bulk-email-0,
> > > > bulk-email-1, bulk-email-2]),
> > > > consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa=
> > > > Assignment(partitions=[bulk-email-6,
> > > > bulk-email-7]),
> > > > consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09=
> > > > Assignment(partitions=[bulk-email-8,
> > > > bulk-email-9])}
> > > > [2018-08-09 11:43:26] :: DEBUG :: AbstractCoordinator:471 - Sending
> > > leader
> > > > SyncGroup for group bulk-email-consumer to coordinator
> > > 10.189.179.116:9092
> > > > (id: 2147483647 rack: null):
> > > > {group_id=bulk-email-consumer,generation_id=1225,member_id=
> > > > consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,group_
> > > > assignment=[{member_id=consumer-23-bdc073b6-1f90-
> > > >
> 4ac5-a801-869377f9ead1,member_assignment=java.nio.HeapByteBuffer[pos=0
> > > > lim=38
> > > > cap=38]},{member_id=consumer-19-59fa9871-2dcc-4713-9c4c-
> > > > 6f0621f10afa,member_assignment=java.nio.HeapByteBuffer[pos=0
> > > > lim=38
> > > > cap=38]},{member_id=consumer-25-fa10026f-031e-49aa-9df1-
> > > > 9d083b256ffa,member_assignment=java.nio.HeapByteBuffer[pos=0
> > > > lim=34
> > > > cap=34]},{member_id=consumer-26
