Hi Greg,
I hadn't considered the implications W/R/T zombie fencing. I agree that
using the transactional ID for task 0 is better in that case.
Yash (who is implementing this part, cc'd), does this seem reasonable to
you?
Cheers,
Chris
On Tue, Apr 11, 2023 at 3:23 PM Greg Harris <greg.har...@aiven.io.invalid>
wrote:
> Chris & Yash,
>
> 1. Since the global offsets topic does not have transactions on it already,
> I don't think adding transactions just for these reset operations would be
> an improvement. The transactional produce would not exclude other
> non-transactional producers, but hanging transactions on the global offsets
> topic would negatively impact the general cluster health. Your proposed
> strategy seems reasonable to me.
>
> 2. While it may be the connector performing the offset reset and not the
> task, I think it would be preferable for the connector to use task 0's
> task-id and 'impersonate' the task for the purpose of changing the offsets.
> I think the complication elsewhere (getting users to provide a new ACL,
> expanding fencing to also fence the connector transaction id, etc) is not
> practically worth it to change 1 string value in the logs.
> I would find a separate transaction ID beneficial if the connector could be
> given a different principal from the task, and be given distinct ACLs.
> However, I don't think this is possible or desirable, and so I don't think
> it's relevant right now. Let me know if there are any other ways that the
> connector transaction ID would be useful.
>
> Thanks for all the effort on this feature!
> Greg
>
> On Tue, Apr 11, 2023 at 7:52 AM Chris Egerton <chr...@aiven.io.invalid>
> wrote:
>
> > Hi all,
> >
> > A couple slight tweaks to the design have been proposed during
> > implementation and I'd like to report them here to make sure that they're
> > acceptable to all who previously voted for this KIP. I've updated the KIP
> > to include these changes but will be happy to revert and/or amend if
> there
> > are any concerns.
> >
> > 1. We would like to refrain from using a transaction when resetting
> source
> > connector offsets in the worker's global offsets topic when exactly-once
> > support is enabled. We would continue to use a transaction when resetting
> > offsets in the connector's offsets topic. Discussed in [1].
> >
> > 2. We would like to use a transactional ID of ${groupId}-${connector} to
> > alter/reset source connector offsets when exactly-once support is
> enabled,
> > where ${groupId} is the group ID of the Connect cluster and ${connector}
> is
> > the name of the connector. This is raised here because it would introduce
> > an additional ACL requirement for this API. A less-elegant alternative
> that
> > would obviate the additional ACL requirement is to use the transactional
> ID
> > that would be used by task 0 of the connector, but this may be confusing
> to
> > users as it could indicate that the task is actually running. Discussed
> in
> > [2].
> >
> > [1] -
> https://github.com/apache/kafka/pull/13465/#issuecomment-1486718538
> > [2] - https://github.com/apache/kafka/pull/13465/#discussion_r1159694956
> >
> > Cheers,
> >
> > Chris
> >
> > On Fri, Mar 3, 2023 at 10:22 AM Chris Egerton <chr...@aiven.io> wrote:
> >
> > > Hi all,
> > >
> > > Thanks for the votes! I'll cast a final +1 myself and close the vote
> out.
> > >
> > > This KIP passes with the following +1 votes (and no +0 or -1 votes):
> > >
> > > • Greg Harris
> > > • Yash Mayya
> > > • Knowles Atchison Jr
> > > • Mickael Maison (binding)
> > > • Tom Bentley (binding)
> > > • Josep Prat (binding)
> > > • Chris Egerton (binding, author)
> > >
> > > I'll write up Jira tickets and begin implementing things next week.
> > >
> > > Cheers,
> > >
> > > Chris
> > >
> > > On Fri, Mar 3, 2023 at 10:07 AM Josep Prat <josep.p...@aiven.io.invalid
> >
> > > wrote:
> > >
> > >> Hi Chris,
> > >>
> > >> Thanks for the KIP. I have a non-blocking comment on the DISCUSS
> thread.
> > >>
> > >> +1 (binding).
> > >>
> > >> Best,
> > >>
> > >> On Wed, Mar 1, 2023 at 12:16 PM Tom Bentley <tbent...@redhat.com>
> > wrote:
> > >>
> > >> > Hi Chris,
> > >> >
> > >> > Thanks for the KIP.
> > >> >
> > >> > +1 (binding).
> > >> >
> > >> > Cheers,
> > >> >
> > >> > Tom
> > >> >
> > >> > On Wed, 15 Feb 2023 at 16:11, Chris Egerton <chr...@aiven.io.invalid
> >
> > >> > wrote:
> > >> >
> > >> > > Hi all,
> > >> > >
> > >> > > Thanks to everyone who's voted so far! Just wanted to bump this
> > thread
> > >> > and
> > >> > > see if we could get a few more votes; currently we're at +3
> > >> non-binding
> > >> > > and +1 binding. Hoping we can get this approved, reviewed, and
> > merged
> > >> in
> > >> > > time for 3.5.0.
> > >> > >
> > >> > > Cheers,
> > >> > >
> > >> > > Chris
> > >> > >
> > >> > > On Tue, Jan 31, 2023 at 2:52 AM Mickael Maison <
> > >> mickael.mai...@gmail.com
> > >> > >
> > >> > > wrote:
> > >> > >
> > >> > > > Thanks Chris for the KIP, this is a much needed feature!
> > >> > > >
> > >> > > > +1 (binding)
> > >> > > >
> > >> > > >
> > >> > > > On Tue, Jan 24, 2023 at 3:45 PM Knowles Atchison Jr
> > >> > > > <katchiso...@gmail.com> wrote:
> > >> > > > >
> > >> > > > > +1 (non binding)
> > >> > > > >
> > >> > > > > On Tue, Jan 24, 2023 at 5:24 AM Yash Mayya <
> > yash.ma...@gmail.com>
> > >> > > wrote:
> > >> > > > >
> > >> > > > > > Hi Chris,
> > >> > > > > >
> > >> > > > > > I'm +1 (non-binding). Thanks again for proposing this
> > extremely
> > >> > > > > > valuable addition to Kafka Connect!
> > >> > > > > >
> > >> > > > > > Thanks,
> > >> > > > > > Yash
> > >> > > > > >
> > >> > > > > > On Thu, Jan 19, 2023 at 12:11 AM Chris Egerton
> > >> > > <chr...@aiven.io.invalid
> > >> > > > >
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > > > Hi all,
> > >> > > > > > >
> > >> > > > > > > I'd like to call for a vote on KIP-875, which adds support
> > for
> > >> > > > viewing
> > >> > > > > > and
> > >> > > > > > > manipulating the offsets of connectors to the Kafka
> Connect
> > >> REST
> > >> > > API.
> > >> > > > > > >
> > >> > > > > > > The KIP:
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect
> > >> > > > > > >
> > >> > > > > > > The discussion thread:
> > >> > > > > > >
> > >> https://lists.apache.org/thread/m5bklnh5w4mwr9nbzrmfk0pftpxfjd02
> > >> > > > > > >
> > >> > > > > > > Cheers,
> > >> > > > > > >
> > >> > > > > > > Chris
> > >> > > > > > >
> > >> > > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >>
> > >> --
> > >> [image: Aiven] <https://www.aiven.io>
> > >>
> > >> *Josep Prat*
> > >> Open Source Engineering Director, *Aiven*
> > >> josep.p...@aiven.io | +491715557497
> > >> aiven.io <https://www.aiven.io> | <
> > >> https://www.facebook.com/aivencloud>
> > >> <https://www.linkedin.com/company/aiven/> <
> > >> https://twitter.com/aiven_io>
> > >> *Aiven Deutschland GmbH*
> > >> Alexanderufer 3-7, 10117 Berlin
> > >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > >> Amtsgericht Charlottenburg, HRB 209739 B
> > >>
> > >
> >
>
