Hi Ashwin, Thanks for taking a look and sharing your thoughts!
1. Yes, the request / response formats of the two APIs were intentionally made identical for such use-cases. [1] 2. I'm assuming you're referring to retaining the offset / config topic records for a connector when it is deleted by a user? Firstly, a connector's offsets aren't actually currently deleted when the connector is deleted - it was listed as potential future work in KIP-875 [2]. Secondly, the config topic is the source of truth for the state of a Connect cluster and a connector deletion is done by writing a null-valued record (tombstone) with the connector key to the config topic. We could potentially modify the delete mechanism to use a special new record (instead of a tombstone with the connector key) in order to retain the latest configuration of a connector while still deleting the actual connector - however, this would have its downsides and I don't see too many benefits. Furthermore, connector migration between different Kafka clusters was just used as a representational use case for creating connectors in a stopped state - it isn't the primary focus of this KIP as such. 3. KIP-875 goes into some more details about this [3], but the TLDR is that the STOPPED state will be treated like the PAUSED state on older workers that don't recognize the STOPPED state. Thanks, Yash [1] - https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect#KIP875:FirstclassoffsetssupportinKafkaConnect-Request/responseformat [2] - https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect#KIP875:FirstclassoffsetssupportinKafkaConnect-Automaticallydeleteoffsetswithconnectors [3] - https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect#KIP875:FirstclassoffsetssupportinKafkaConnect-STOPPEDtargetstate On Wed, Sep 20, 2023 at 7:24 PM Ashwin <apan...@confluent.io.invalid> wrote: > Thanks Yash! This is very useful for migrating connectors from one cluster > to another. > > I had the following comments/questions > > 1. Is the offset read using `GET /offsets` api always guaranteed to be in a > format accepted by `PATCH /offsets` ? > 2. I had to tackle a similar migration situation but the two connect > clusters in question were using the same backing Kafka cluster. The > challenge in this case is that when I delete the original connector, I want > to retain offsets and config topics. Do you think we should support > deletion of a connector without removal of these topics as part of this KIP > ? > 3. In the case of a downgrade, how will Connect worker handle the optional > “state” field in config topic ? > > Thanks, > Ashwin > > > > > On Sun, Sep 17, 2023 at 11:09 PM Yash Mayya <yash.ma...@gmail.com> wrote: > > > Hi all, > > > > I'd like to begin discussion on a KIP to allow creating connectors in a > > stopped state - > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-980%3A+Allow+creating+connectors+in+a+stopped+state > > > > > > Thanks, > > Yash > > >
