Hi Ryanne, Thanks for the KIP, this is an interesting use case.
I agree it's best to have the runtime take care of the deletion. However, I wonder if it would make more sense to have the configuration on connectors instead of the runtime. While Workers perform the deletion, it's very much driven by connectors. It's true that when having multiple sink connectors that use the same topics, you'd need to enable deletion on each but using connector configuration looks like a more natural way to defining how a pipeline should work. Also if configured on the runtime, each time a new pipeline is setup, it may require reconfiguring and restarting the Connect cluster. What do you think? On Sat, Jul 17, 2021 at 3:16 PM Ryanne Dolan <ryannedo...@gmail.com> wrote: > > Whoops, looks like I got the KIP number wrong in the original email subject > line. Please use this corrected thread. > > Ryanne > > On Fri, Jul 16, 2021, 3:45 PM Ryanne Dolan <ryannedo...@gmail.com> wrote: > > > Hey y'all, please review the following small proposal: > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-762%3A+Delete+Committed+Connect+Records > > > > The idea is to enable Source->Sink pipelines to clean up after themselves > > by automatically truncating intermediate topics. > > > > Ryanne > > > >
