Re: Kafka Connect - offset.storage.topic reuse across clusters

2022-03-30 Thread Chris Egerton
Connectors overwriting each other's offsets is the primary concern. If you have a guarantee that there will only ever be one connector with a given name running at once on any of the Connect clusters that use the same offsets topic, and you want offsets to be shared for all source connectors on any

Re: Kafka Connect - offset.storage.topic reuse across clusters

2022-03-30 Thread Jordan Wyatt
Hi Robin, I'm interested in a use case in which I need to be able to have a connect cluster fail, and then bring up a new cluster with the same offset topics and connectors. By new cluster I mean a cluster with a new `group.id`. I am aware I could just use the same group id as before but I would l

Re: Kafka Connect - offset.storage.topic reuse across clusters

2022-03-30 Thread Robin Moffatt
Hi Jordan, Is there a good reason for wanting to do this? I can think of multiple reasons why you shouldn't do this even if technically it works in some cases. Or it's just curiosity as to whether you can/should? thanks, Robin. -- Robin Moffatt | Principal Developer Advocate | ro...@confluent

Kafka Connect - offset.storage.topic reuse across clusters

2022-03-30 Thread Jordan Wyatt
Hi, I've recently been experimenting with setting the values of the `offset,` `storage` and `status` topics within Kafka Connect. I'm aware from various sources (Robin Moffatt blogs, StackOverflow, Confluent Kafka Connect docs) that these topics should not be shared across different connect **clu