[ https://issues.apache.org/jira/browse/KAFKA-10253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Lalafaryan updated KAFKA-10253: ------------------------------------------ Description: Hello everyone! We are running kafka-connect cluster (3 workers) and very often it gets into an infinite rebalance loop. {code:java} 2020-07-09 08:51:25,731 INFO [Worker clientId=connect-1, groupId= kafka-connect] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,731 INFO [Worker clientId=connect-1, groupId= kafka-connect] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,733 INFO [Worker clientId=connect-1, groupId= kafka-connect] Was selected to perform assignments, but do not have latest config found in sync request. Returning an empty configuration to trigger re-sync. (org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignor) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,735 INFO [Worker clientId=connect-1, groupId= kafka-connect] Successfully joined group with generation 305655831 (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,735 INFO [Worker clientId=connect-1, groupId= kafka-connect] Joined group at generation 305655831 with protocol version 2 and got assignment: Assignment{error=1, leader='connect-1-0008abc5-a152-42fe-a697-a4a4641f72bb', leaderUrl='http://10.20.30.221:8083/', offset=12, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,735 INFO [Worker clientId=connect-1, groupId= kafka-connect] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,735 INFO [Worker clientId=connect-1, groupId= kafka-connect] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,736 INFO [Worker clientId=connect-1, groupId= kafka-connect] Was selected to perform assignments, but do not have latest config found in sync request. Returning an empty configuration to trigger re-sync. (org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignor) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,739 INFO [Worker clientId=connect-1, groupId= kafka-connect] Successfully joined group with generation 305655832 (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,739 INFO [Worker clientId=connect-1, groupId= kafka-connect] Joined group at generation 305655832 with protocol version 2 and got assignment: Assignment{error=1, leader='connect-1-0008abc5-a152-42fe-a697-a4a4641f72bb', leaderUrl='http://10.20.30.221:8083/', offset=12, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,739 INFO [Worker clientId=connect-1, groupId= kafka-connect] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,739 INFO [Worker clientId=connect-1, groupId= kafka-connect] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,740 INFO [Worker clientId=connect-1, groupId= kafka-connect] Was selected to perform assignments, but do not have latest config found in sync request. Returning an empty configuration to trigger re-sync. (org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignor) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,742 INFO [Worker clientId=connect-1, groupId= kafka-connect] Successfully joined group with generation 305655833 (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,742 INFO [Worker clientId=connect-1, groupId= kafka-connect] Joined group at generation 305655833 with protocol version 2 and got assignment: Assignment{error=1, leader='connect-1-0008abc5-a152-42fe-a697-a4a4641f72bb', leaderUrl='http://10.20.30.221:8083/', offset=12, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,742 INFO [Worker clientId=connect-1, groupId= kafka-connect] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,742 INFO [Worker clientId=connect-1, groupId= kafka-connect] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,744 INFO [Worker clientId=connect-1, groupId= kafka-connect] Was selected to perform assignments, but do not have latest config found in sync request. Returning an empty configuration to trigger re-sync. (org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignor) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,746 INFO [Worker clientId=connect-1, groupId= kafka-connect] Successfully joined group with generation 305655834 (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,746 INFO [Worker clientId=connect-1, groupId= kafka-connect] Joined group at generation 305655834 with protocol version 2 and got assignment: Assignment{error=1, leader='connect-1-0008abc5-a152-42fe-a697-a4a4641f72bb', leaderUrl='http://10.20.30.221:8083/', offset=12, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,746 INFO [Worker clientId=connect-1, groupId= kafka-connect] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,746 INFO [Worker clientId=connect-1, groupId= kafka-connect] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,748 INFO [Worker clientId=connect-1, groupId= kafka-connect] Was selected to perform assignments, but do not have latest config found in sync request. Returning an empty configuration to trigger re-sync. (org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignor) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,750 INFO [Worker clientId=connect-1, groupId= kafka-connect] Successfully joined group with generation 305655835 (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,750 INFO [Worker clientId=connect-1, groupId= kafka-connect] Joined group at generation 305655835 with protocol version 2 and got assignment: Assignment{error=1, leader='connect-1-0008abc5-a152-42fe-a697-a4a4641f72bb', leaderUrl='http://10.20.30.221:8083/', offset=12, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,750 INFO [Worker clientId=connect-1, groupId= kafka-connect] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,750 INFO [Worker clientId=connect-1, groupId= kafka-connect] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,751 INFO [Worker clientId=connect-1, groupId= kafka-connect] Was selected to perform assignments, but do not have latest config found in sync request. Returning an empty configuration to trigger re-sync. (org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignor) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,754 INFO [Worker clientId=connect-1, groupId= kafka-connect] Successfully joined group with generation 305655836 (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,754 INFO [Worker clientId=connect-1, groupId= kafka-connect] Joined group at generation 305655836 with protocol version 2 and got assignment: Assignment{error=1, leader='connect-1-0008abc5-a152-42fe-a697-a4a4641f72bb', leaderUrl='http://10.20.30.221:8083/', offset=12, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,754 INFO [Worker clientId=connect-1, groupId= kafka-connect] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,754 INFO [Worker clientId=connect-1, groupId= kafka-connect] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,755 INFO [Worker clientId=connect-1, groupId= kafka-connect] Was selected to perform assignments, but do not have latest config found in sync request. Returning an empty configuration to trigger re-sync. (org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignor) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,757 INFO [Worker clientId=connect-1, groupId= kafka-connect] Successfully joined group with generation 305655837 (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,757 INFO [Worker clientId=connect-1, groupId= kafka-connect] Joined group at generation 305655837 with protocol version 2 and got assignment: Assignment{error=1, leader='connect-1-0008abc5-a152-42fe-a697-a4a4641f72bb', leaderUrl='http://10.20.30.221:8083/', offset=12, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,757 INFO [Worker clientId=connect-1, groupId= kafka-connect] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,757 INFO [Worker clientId=connect-1, groupId= kafka-connect] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,759 INFO [Worker clientId=connect-1, groupId= kafka-connect] Was selected to perform assignments, but do not have latest config found in sync request. Returning an empty configuration to trigger re-sync. (org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignor) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,761 INFO [Worker clientId=connect-1, groupId= kafka-connect] Successfully joined group with generation 305655838 (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,761 INFO [Worker clientId=connect-1, groupId= kafka-connect] Joined group at generation 305655838 with protocol version 2 and got assignment: Assignment{error=1, leader='connect-1-0008abc5-a152-42fe-a697-a4a4641f72bb', leaderUrl='http://10.20.30.221:8083/', offset=12, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,761 INFO [Worker clientId=connect-1, groupId= kafka-connect] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,761 INFO [Worker clientId=connect-1, groupId= kafka-connect] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,763 INFO [Worker clientId=connect-1, groupId= kafka-connect] Was selected to perform assignments, but do not have latest config found in sync request. Returning an empty configuration to trigger re-sync. (org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignor) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,766 INFO [Worker clientId=connect-1, groupId= kafka-connect] Successfully joined group with generation 305655839 (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,766 INFO [Worker clientId=connect-1, groupId= kafka-connect] Joined group at generation 305655839 with protocol version 2 and got assignment: Assignment{error=1, leader='connect-1-0008abc5-a152-42fe-a697-a4a4641f72bb', leaderUrl='http://10.20.30.221:8083/', offset=12, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,766 INFO [Worker clientId=connect-1, groupId= kafka-connect] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,766 INFO [Worker clientId=connect-1, groupId= kafka-connect] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,768 INFO [Worker clientId=connect-1, groupId= kafka-connect] Was selected to perform assignments, but do not have latest config found in sync request. Returning an empty configuration to trigger re-sync. (org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignor) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,771 INFO [Worker clientId=connect-1, groupId= kafka-connect] Successfully joined group with generation 305655840 (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,771 INFO [Worker clientId=connect-1, groupId= kafka-connect] Joined group at generation {code} It is happening in all 3 workers. Any feedback is appreciated! Thanks! was: Hello everyone! We are running kafka-connect cluster (3 workers) and very often it gets into an infinite rebalance loop. {code:java} 2020-07-09 08:51:25,731 INFO [Worker clientId=connect-1, groupId= kafka-connect] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,731 INFO [Worker clientId=connect-1, groupId= kafka-connect] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,733 INFO [Worker clientId=connect-1, groupId= kafka-connect] Was selected to perform assignments, but do not have latest config found in sync request. Returning an empty configuration to trigger re-sync. (org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignor) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,735 INFO [Worker clientId=connect-1, groupId= kafka-connect] Successfully joined group with generation 305655831 (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,735 INFO [Worker clientId=connect-1, groupId= kafka-connect] Joined group at generation 305655831 with protocol version 2 and got assignment: Assignment{error=1, leader='connect-1-0008abc5-a152-42fe-a697-a4a4641f72bb', leaderUrl='http://10.20.30.221:8083/', offset=12, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,735 INFO [Worker clientId=connect-1, groupId= kafka-connect] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,735 INFO [Worker clientId=connect-1, groupId= kafka-connect] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,736 INFO [Worker clientId=connect-1, groupId= kafka-connect] Was selected to perform assignments, but do not have latest config found in sync request. Returning an empty configuration to trigger re-sync. (org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignor) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,739 INFO [Worker clientId=connect-1, groupId= kafka-connect] Successfully joined group with generation 305655832 (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,739 INFO [Worker clientId=connect-1, groupId= kafka-connect] Joined group at generation 305655832 with protocol version 2 and got assignment: Assignment{error=1, leader='connect-1-0008abc5-a152-42fe-a697-a4a4641f72bb', leaderUrl='http://10.20.30.221:8083/', offset=12, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,739 INFO [Worker clientId=connect-1, groupId= kafka-connect] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,739 INFO [Worker clientId=connect-1, groupId= kafka-connect] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,740 INFO [Worker clientId=connect-1, groupId= kafka-connect] Was selected to perform assignments, but do not have latest config found in sync request. Returning an empty configuration to trigger re-sync. (org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignor) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,742 INFO [Worker clientId=connect-1, groupId= kafka-connect] Successfully joined group with generation 305655833 (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,742 INFO [Worker clientId=connect-1, groupId= kafka-connect] Joined group at generation 305655833 with protocol version 2 and got assignment: Assignment{error=1, leader='connect-1-0008abc5-a152-42fe-a697-a4a4641f72bb', leaderUrl='http://10.20.30.221:8083/', offset=12, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,742 INFO [Worker clientId=connect-1, groupId= kafka-connect] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,742 INFO [Worker clientId=connect-1, groupId= kafka-connect] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,744 INFO [Worker clientId=connect-1, groupId= kafka-connect] Was selected to perform assignments, but do not have latest config found in sync request. Returning an empty configuration to trigger re-sync. (org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignor) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,746 INFO [Worker clientId=connect-1, groupId= kafka-connect] Successfully joined group with generation 305655834 (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,746 INFO [Worker clientId=connect-1, groupId= kafka-connect] Joined group at generation 305655834 with protocol version 2 and got assignment: Assignment{error=1, leader='connect-1-0008abc5-a152-42fe-a697-a4a4641f72bb', leaderUrl='http://10.20.30.221:8083/', offset=12, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,746 INFO [Worker clientId=connect-1, groupId= kafka-connect] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,746 INFO [Worker clientId=connect-1, groupId= kafka-connect] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,748 INFO [Worker clientId=connect-1, groupId= kafka-connect] Was selected to perform assignments, but do not have latest config found in sync request. Returning an empty configuration to trigger re-sync. (org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignor) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,750 INFO [Worker clientId=connect-1, groupId= kafka-connect] Successfully joined group with generation 305655835 (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,750 INFO [Worker clientId=connect-1, groupId= kafka-connect] Joined group at generation 305655835 with protocol version 2 and got assignment: Assignment{error=1, leader='connect-1-0008abc5-a152-42fe-a697-a4a4641f72bb', leaderUrl='http://10.20.30.221:8083/', offset=12, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,750 INFO [Worker clientId=connect-1, groupId= kafka-connect] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,750 INFO [Worker clientId=connect-1, groupId= kafka-connect] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,751 INFO [Worker clientId=connect-1, groupId= kafka-connect] Was selected to perform assignments, but do not have latest config found in sync request. Returning an empty configuration to trigger re-sync. (org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignor) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,754 INFO [Worker clientId=connect-1, groupId= kafka-connect] Successfully joined group with generation 305655836 (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,754 INFO [Worker clientId=connect-1, groupId= kafka-connect] Joined group at generation 305655836 with protocol version 2 and got assignment: Assignment{error=1, leader='connect-1-0008abc5-a152-42fe-a697-a4a4641f72bb', leaderUrl='http://10.20.30.221:8083/', offset=12, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,754 INFO [Worker clientId=connect-1, groupId= kafka-connect] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,754 INFO [Worker clientId=connect-1, groupId= kafka-connect] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,755 INFO [Worker clientId=connect-1, groupId= kafka-connect] Was selected to perform assignments, but do not have latest config found in sync request. Returning an empty configuration to trigger re-sync. (org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignor) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,757 INFO [Worker clientId=connect-1, groupId= kafka-connect] Successfully joined group with generation 305655837 (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,757 INFO [Worker clientId=connect-1, groupId= kafka-connect] Joined group at generation 305655837 with protocol version 2 and got assignment: Assignment{error=1, leader='connect-1-0008abc5-a152-42fe-a697-a4a4641f72bb', leaderUrl='http://10.20.30.221:8083/', offset=12, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,757 INFO [Worker clientId=connect-1, groupId= kafka-connect] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,757 INFO [Worker clientId=connect-1, groupId= kafka-connect] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,759 INFO [Worker clientId=connect-1, groupId= kafka-connect] Was selected to perform assignments, but do not have latest config found in sync request. Returning an empty configuration to trigger re-sync. (org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignor) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,761 INFO [Worker clientId=connect-1, groupId= kafka-connect] Successfully joined group with generation 305655838 (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,761 INFO [Worker clientId=connect-1, groupId= kafka-connect] Joined group at generation 305655838 with protocol version 2 and got assignment: Assignment{error=1, leader='connect-1-0008abc5-a152-42fe-a697-a4a4641f72bb', leaderUrl='http://10.20.30.221:8083/', offset=12, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,761 INFO [Worker clientId=connect-1, groupId= kafka-connect] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,761 INFO [Worker clientId=connect-1, groupId= kafka-connect] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,763 INFO [Worker clientId=connect-1, groupId= kafka-connect] Was selected to perform assignments, but do not have latest config found in sync request. Returning an empty configuration to trigger re-sync. (org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignor) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,766 INFO [Worker clientId=connect-1, groupId= kafka-connect] Successfully joined group with generation 305655839 (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,766 INFO [Worker clientId=connect-1, groupId= kafka-connect] Joined group at generation 305655839 with protocol version 2 and got assignment: Assignment{error=1, leader='connect-1-0008abc5-a152-42fe-a697-a4a4641f72bb', leaderUrl='http://10.20.30.221:8083/', offset=12, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,766 INFO [Worker clientId=connect-1, groupId= kafka-connect] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,766 INFO [Worker clientId=connect-1, groupId= kafka-connect] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,768 INFO [Worker clientId=connect-1, groupId= kafka-connect] Was selected to perform assignments, but do not have latest config found in sync request. Returning an empty configuration to trigger re-sync. (org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignor) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,771 INFO [Worker clientId=connect-1, groupId= kafka-connect] Successfully joined group with generation 305655840 (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,771 INFO [Worker clientId=connect-1, groupId= kafka-connect] Joined group at generation {code} Any feedback is appreciated! Thanks! > Kafka Connect gets into an infinite rebalance loop > -------------------------------------------------- > > Key: KAFKA-10253 > URL: https://issues.apache.org/jira/browse/KAFKA-10253 > Project: Kafka > Issue Type: Bug > Affects Versions: 2.5.0 > Reporter: Konstantin Lalafaryan > Priority: Blocker > > Hello everyone! > > We are running kafka-connect cluster (3 workers) and very often it gets into > an infinite rebalance loop. > > {code:java} > 2020-07-09 08:51:25,731 INFO [Worker clientId=connect-1, groupId= > kafka-connect] Rebalance started > (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) > [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,731 INFO [Worker > clientId=connect-1, groupId= kafka-connect] (Re-)joining group > (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) > [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,733 INFO [Worker > clientId=connect-1, groupId= kafka-connect] Was selected to perform > assignments, but do not have latest config found in sync request. Returning > an empty configuration to trigger re-sync. > (org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignor) > [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,735 INFO [Worker > clientId=connect-1, groupId= kafka-connect] Successfully joined group with > generation 305655831 > (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) > [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,735 INFO [Worker > clientId=connect-1, groupId= kafka-connect] Joined group at generation > 305655831 with protocol version 2 and got assignment: Assignment{error=1, > leader='connect-1-0008abc5-a152-42fe-a697-a4a4641f72bb', > leaderUrl='http://10.20.30.221:8083/', offset=12, connectorIds=[], > taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with > rebalance delay: 0 > (org.apache.kafka.connect.runtime.distributed.DistributedHerder) > [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,735 INFO [Worker > clientId=connect-1, groupId= kafka-connect] Rebalance started > (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) > [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,735 INFO [Worker > clientId=connect-1, groupId= kafka-connect] (Re-)joining group > (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) > [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,736 INFO [Worker > clientId=connect-1, groupId= kafka-connect] Was selected to perform > assignments, but do not have latest config found in sync request. Returning > an empty configuration to trigger re-sync. > (org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignor) > [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,739 INFO [Worker > clientId=connect-1, groupId= kafka-connect] Successfully joined group with > generation 305655832 > (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) > [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,739 INFO [Worker > clientId=connect-1, groupId= kafka-connect] Joined group at generation > 305655832 with protocol version 2 and got assignment: Assignment{error=1, > leader='connect-1-0008abc5-a152-42fe-a697-a4a4641f72bb', > leaderUrl='http://10.20.30.221:8083/', offset=12, connectorIds=[], > taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with > rebalance delay: 0 > (org.apache.kafka.connect.runtime.distributed.DistributedHerder) > [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,739 INFO [Worker > clientId=connect-1, groupId= kafka-connect] Rebalance started > (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) > [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,739 INFO [Worker > clientId=connect-1, groupId= kafka-connect] (Re-)joining group > (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) > [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,740 INFO [Worker > clientId=connect-1, groupId= kafka-connect] Was selected to perform > assignments, but do not have latest config found in sync request. Returning > an empty configuration to trigger re-sync. > (org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignor) > [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,742 INFO [Worker > clientId=connect-1, groupId= kafka-connect] Successfully joined group with > generation 305655833 > (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) > [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,742 INFO [Worker > clientId=connect-1, groupId= kafka-connect] Joined group at generation > 305655833 with protocol version 2 and got assignment: Assignment{error=1, > leader='connect-1-0008abc5-a152-42fe-a697-a4a4641f72bb', > leaderUrl='http://10.20.30.221:8083/', offset=12, connectorIds=[], > taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with > rebalance delay: 0 > (org.apache.kafka.connect.runtime.distributed.DistributedHerder) > [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,742 INFO [Worker > clientId=connect-1, groupId= kafka-connect] Rebalance started > (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) > [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,742 INFO [Worker > clientId=connect-1, groupId= kafka-connect] (Re-)joining group > (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) > [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,744 INFO [Worker > clientId=connect-1, groupId= kafka-connect] Was selected to perform > assignments, but do not have latest config found in sync request. Returning > an empty configuration to trigger re-sync. > (org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignor) > [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,746 INFO [Worker > clientId=connect-1, groupId= kafka-connect] Successfully joined group with > generation 305655834 > (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) > [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,746 INFO [Worker > clientId=connect-1, groupId= kafka-connect] Joined group at generation > 305655834 with protocol version 2 and got assignment: Assignment{error=1, > leader='connect-1-0008abc5-a152-42fe-a697-a4a4641f72bb', > leaderUrl='http://10.20.30.221:8083/', offset=12, connectorIds=[], > taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with > rebalance delay: 0 > (org.apache.kafka.connect.runtime.distributed.DistributedHerder) > [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,746 INFO [Worker > clientId=connect-1, groupId= kafka-connect] Rebalance started > (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) > [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,746 INFO [Worker > clientId=connect-1, groupId= kafka-connect] (Re-)joining group > (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) > [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,748 INFO [Worker > clientId=connect-1, groupId= kafka-connect] Was selected to perform > assignments, but do not have latest config found in sync request. Returning > an empty configuration to trigger re-sync. > (org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignor) > [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,750 INFO [Worker > clientId=connect-1, groupId= kafka-connect] Successfully joined group with > generation 305655835 > (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) > [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,750 INFO [Worker > clientId=connect-1, groupId= kafka-connect] Joined group at generation > 305655835 with protocol version 2 and got assignment: Assignment{error=1, > leader='connect-1-0008abc5-a152-42fe-a697-a4a4641f72bb', > leaderUrl='http://10.20.30.221:8083/', offset=12, connectorIds=[], > taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with > rebalance delay: 0 > (org.apache.kafka.connect.runtime.distributed.DistributedHerder) > [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,750 INFO [Worker > clientId=connect-1, groupId= kafka-connect] Rebalance started > (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) > [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,750 INFO [Worker > clientId=connect-1, groupId= kafka-connect] (Re-)joining group > (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) > [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,751 INFO [Worker > clientId=connect-1, groupId= kafka-connect] Was selected to perform > assignments, but do not have latest config found in sync request. Returning > an empty configuration to trigger re-sync. > (org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignor) > [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,754 INFO [Worker > clientId=connect-1, groupId= kafka-connect] Successfully joined group with > generation 305655836 > (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) > [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,754 INFO [Worker > clientId=connect-1, groupId= kafka-connect] Joined group at generation > 305655836 with protocol version 2 and got assignment: Assignment{error=1, > leader='connect-1-0008abc5-a152-42fe-a697-a4a4641f72bb', > leaderUrl='http://10.20.30.221:8083/', offset=12, connectorIds=[], > taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with > rebalance delay: 0 > (org.apache.kafka.connect.runtime.distributed.DistributedHerder) > [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,754 INFO [Worker > clientId=connect-1, groupId= kafka-connect] Rebalance started > (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) > [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,754 INFO [Worker > clientId=connect-1, groupId= kafka-connect] (Re-)joining group > (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) > [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,755 INFO [Worker > clientId=connect-1, groupId= kafka-connect] Was selected to perform > assignments, but do not have latest config found in sync request. Returning > an empty configuration to trigger re-sync. > (org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignor) > [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,757 INFO [Worker > clientId=connect-1, groupId= kafka-connect] Successfully joined group with > generation 305655837 > (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) > [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,757 INFO [Worker > clientId=connect-1, groupId= kafka-connect] Joined group at generation > 305655837 with protocol version 2 and got assignment: Assignment{error=1, > leader='connect-1-0008abc5-a152-42fe-a697-a4a4641f72bb', > leaderUrl='http://10.20.30.221:8083/', offset=12, connectorIds=[], > taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with > rebalance delay: 0 > (org.apache.kafka.connect.runtime.distributed.DistributedHerder) > [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,757 INFO [Worker > clientId=connect-1, groupId= kafka-connect] Rebalance started > (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) > [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,757 INFO [Worker > clientId=connect-1, groupId= kafka-connect] (Re-)joining group > (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) > [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,759 INFO [Worker > clientId=connect-1, groupId= kafka-connect] Was selected to perform > assignments, but do not have latest config found in sync request. Returning > an empty configuration to trigger re-sync. > (org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignor) > [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,761 INFO [Worker > clientId=connect-1, groupId= kafka-connect] Successfully joined group with > generation 305655838 > (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) > [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,761 INFO [Worker > clientId=connect-1, groupId= kafka-connect] Joined group at generation > 305655838 with protocol version 2 and got assignment: Assignment{error=1, > leader='connect-1-0008abc5-a152-42fe-a697-a4a4641f72bb', > leaderUrl='http://10.20.30.221:8083/', offset=12, connectorIds=[], > taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with > rebalance delay: 0 > (org.apache.kafka.connect.runtime.distributed.DistributedHerder) > [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,761 INFO [Worker > clientId=connect-1, groupId= kafka-connect] Rebalance started > (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) > [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,761 INFO [Worker > clientId=connect-1, groupId= kafka-connect] (Re-)joining group > (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) > [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,763 INFO [Worker > clientId=connect-1, groupId= kafka-connect] Was selected to perform > assignments, but do not have latest config found in sync request. Returning > an empty configuration to trigger re-sync. > (org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignor) > [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,766 INFO [Worker > clientId=connect-1, groupId= kafka-connect] Successfully joined group with > generation 305655839 > (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) > [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,766 INFO [Worker > clientId=connect-1, groupId= kafka-connect] Joined group at generation > 305655839 with protocol version 2 and got assignment: Assignment{error=1, > leader='connect-1-0008abc5-a152-42fe-a697-a4a4641f72bb', > leaderUrl='http://10.20.30.221:8083/', offset=12, connectorIds=[], > taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with > rebalance delay: 0 > (org.apache.kafka.connect.runtime.distributed.DistributedHerder) > [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,766 INFO [Worker > clientId=connect-1, groupId= kafka-connect] Rebalance started > (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) > [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,766 INFO [Worker > clientId=connect-1, groupId= kafka-connect] (Re-)joining group > (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) > [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,768 INFO [Worker > clientId=connect-1, groupId= kafka-connect] Was selected to perform > assignments, but do not have latest config found in sync request. Returning > an empty configuration to trigger re-sync. > (org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignor) > [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,771 INFO [Worker > clientId=connect-1, groupId= kafka-connect] Successfully joined group with > generation 305655840 > (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) > [DistributedHerder-connect-1-1] 2020-07-09 08:51:25,771 INFO [Worker > clientId=connect-1, groupId= kafka-connect] Joined group at generation > {code} > > It is happening in all 3 workers. > > Any feedback is appreciated! > Thanks! -- This message was sent by Atlassian Jira (v8.3.4#803005)