Hey Chris, Thanks for the KIP!
I think this is an important feature for both development and operations use-cases, and it's an obvious gap in the REST feature set. I also appreciate the incremental nature of the KIP and the future extensions that will now be possible. I had a couple of questions about the design and it's extensibility: 1. How do you imagine the API will behave with connectors that have extremely large numbers of partitions (thousands or more) and/or source connectors with large amounts of data per partition? 2. Does the new STOPPED state need any special integration with the rebalance subsystem, or can the rebalance algorithms remain ignorant of the target state of connectors? And about the implementation: 1. For my own edification, what is the difference between deleting a consumer group and deleting all known offsets for that group? Does deleting the group offer better/easier atomicity? 2. For EOS sources, will stopping the connector and tombstoning the task configs perform a fence-out, or will that fence-out only occur when performing the offsets DELETE operation? Thanks! Greg On 2022/10/13 20:52:26 Chris Egerton wrote: > Hi all, > > I noticed a fairly large gap in the first version of this KIP that I > published last Friday, which has to do with accommodating connectors > that target different Kafka clusters than the one that the Kafka Connect > cluster uses for its internal topics and source connectors with dedicated > offsets topics. I've since updated the KIP to address this gap, which has > substantially altered the design. Wanted to give a heads-up to anyone > that's already started reviewing. > > Cheers, > > Chris > > On Fri, Oct 7, 2022 at 1:29 PM Chris Egerton <ch...@aiven.io> wrote: > > > Hi all, > > > > I'd like to begin discussion on a KIP to add offsets support to the Kafka > > Connect REST API: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect > > > > Cheers, > > > > Chris > > >
