Hi
I have been using distributed Kafka Connect 0.10.0.0 very successfully
for a while now. But then after a restart both machines get stuck as
they try to join the group and marks the coordinator dead. I found no
way of getting out of this state. Restarting the machines does not
help.
Here are some relevant parts of the configuration.
request.timeout.ms=610000
heartbeat.interval.ms=60000
session.timeout.ms=300000
max.poll.records=10000
consumer.request.timeout.ms=610000
consumer.session.timeout.ms=300000
Any advice?
Cheers,
-Kristoffer
[2016-08-11 13:47:36,236] INFO Finished reading KafkaBasedLog for
topic kafka-connect-config
(org.apache.kafka.connect.util.KafkaBasedLog:148)
[2016-08-11 13:47:36,237] INFO Started KafkaBasedLog for topic
kafka-connect-config (org.apache.kafka.connect.util.KafkaBasedLog:150)
[2016-08-11 13:47:36,239] INFO Started KafkaConfigBackingStore
(org.apache.kafka.connect.storage.KafkaConfigBackingStore:261)
[2016-08-11 13:47:36,243] INFO Herder started
(org.apache.kafka.connect.runtime.distributed.DistributedHerder:173)
[2016-08-11 13:47:36,245] INFO Discovered coordinator
kafka-ix03.se-ix.delta.prod:6667 (id: 2137162607 rack: null) for group
kafka-connect.
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator:505)
[2016-08-11 13:47:36,246] INFO (Re-)joining group kafka-connect
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator:326)
[2016-08-11 13:47:42,163] INFO Successfully joined group kafka-connect
with generation 24
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator:434)
[2016-08-11 13:47:42,166] INFO Joined group and got assignment:
Assignment{error=0,
leader='connect-1-b144d3b4-333c-4fe7-8ca3-7b53940cfea9',
leaderUrl='http://10.3.20.31:8083/', offset=-1, connectorIds=[],
taskIds=[]}
(org.apache.kafka.connect.runtime.distributed.DistributedHerder:1023)
[2016-08-11 13:47:42,167] INFO Starting connectors and tasks using
config offset -1
(org.apache.kafka.connect.runtime.distributed.DistributedHerder:752)
[2016-08-11 13:47:42,168] INFO Finished starting connectors and tasks
(org.apache.kafka.connect.runtime.distributed.DistributedHerder:769)
[2016-08-11 13:47:45,426] INFO Reflections took 9072 ms to scan 268
urls, producing 12203 keys and 80967 values
(org.reflections.Reflections:229)
[2016-08-11 13:56:34,698] INFO Marking the coordinator
kafka-ix03.se-ix.delta.prod:6667 (id: 2137162607 rack: null) dead for
group kafka-connect
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator:542)
[2016-08-11 13:56:34,702] INFO Discovered coordinator
kafka-ix03.se-ix.delta.prod:6667 (id: 2137162607 rack: null) for group
kafka-connect.
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator:505)
[2016-08-11 13:56:34,702] INFO Marking the coordinator
kafka-ix03.se-ix.delta.prod:6667 (id: 2137162607 rack: null) dead for
group kafka-connect
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator:542)
[2016-08-11 13:56:34,804] INFO Discovered coordinator
kafka-ix03.se-ix.delta.prod:6667 (id: 2137162607 rack: null) for group
kafka-connect.
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator:505)
[2016-08-11 13:56:35,197] INFO Marking the coordinator
kafka-ix03.se-ix.delta.prod:6667 (id: 2137162607 rack: null) dead for
group kafka-connect
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator:542)
[2016-08-11 13:56:35,199] INFO Discovered coordinator
kafka-ix03.se-ix.delta.prod:6667 (id: 2137162607 rack: null) for group
kafka-connect.
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator:505)
[2016-08-11 13:56:35,199] INFO Marking the coordinator
kafka-ix03.se-ix.delta.prod:6667 (id: 2137162607 rack: null) dead for
group kafka-connect
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator:542)
[2016-08-11 13:56:35,301] INFO Discovered coordinator
kafka-ix03.se-ix.delta.prod:6667 (id: 2137162607 rack: null) for group
kafka-connect.
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator:505)
[2016-08-11 13:56:36,477] INFO Marking the coordinator
kafka-ix03.se-ix.delta.prod:6667 (id: 2137162607 rack: null) dead for
group kafka-connect
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator:542)
[2016-08-11 13:56:36,478] INFO Discovered coordinator
kafka-ix03.se-ix.delta.prod:6667 (id: 2137162607 rack: null) for group
kafka-connect.
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator:505)
[2016-08-11 13:56:36,478] INFO Marking the coordinator
kafka-ix03.se-ix.delta.prod:6667 (id: 2137162607 rack: null) dead for
group kafka-connect
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator:542)
[2016-08-11 13:56:36,580] INFO Discovered coordinator
kafka-ix03.se-ix.delta.prod:6667 (id: 2137162607 rack: null) for group
kafka-connect.
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator:505)
