Mickael, thanks for taking a look. I wouldn't be opposed to making this a per-connector configuration. I guess each topic-partition is consumed by one WorkerSinkTask, so it should be safe for the worker to truncate such partitions if all associated Connectors opt-in.
I think in that case it may make sense to remove the regex configuration part, which appeals to me. Thanks for the suggestion. Ryanne On Thu, Aug 19, 2021, 9:18 AM Mickael Maison <mickael.mai...@gmail.com> wrote: > Hi Ryanne, > > Thanks for the KIP, this is an interesting use case. > > I agree it's best to have the runtime take care of the deletion. > However, I wonder if it would make more sense to have the > configuration on connectors instead of the runtime. > > While Workers perform the deletion, it's very much driven by > connectors. It's true that when having multiple sink connectors that > use the same topics, you'd need to enable deletion on each but using > connector configuration looks like a more natural way to defining how > a pipeline should work. > Also if configured on the runtime, each time a new pipeline is setup, > it may require reconfiguring and restarting the Connect cluster. > > What do you think? > > On Sat, Jul 17, 2021 at 3:16 PM Ryanne Dolan <ryannedo...@gmail.com> > wrote: > > > > Whoops, looks like I got the KIP number wrong in the original email > subject > > line. Please use this corrected thread. > > > > Ryanne > > > > On Fri, Jul 16, 2021, 3:45 PM Ryanne Dolan <ryannedo...@gmail.com> > wrote: > > > > > Hey y'all, please review the following small proposal: > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-762%3A+Delete+Committed+Connect+Records > > > > > > The idea is to enable Source->Sink pipelines to clean up after > themselves > > > by automatically truncating intermediate topics. > > > > > > Ryanne > > > > > > >
