Hi Chris and Greg,
The current implementation does already use the transactional ID for task 0
so no complaints from me. Although I'm not sure I follow the concerns w.r.t
zombie fencing? In which cases would we need to fence out the transactional
producer instantiated for altering offsets?
Thanks,
Yash
On Wed, Apr 12, 2023 at 9:02 PM Chris Egerton <chr...@aiven.io> wrote:
> Hi Greg,
>
> I hadn't considered the implications W/R/T zombie fencing. I agree that
> using the transactional ID for task 0 is better in that case.
>
> Yash (who is implementing this part, cc'd), does this seem reasonable to
> you?
>
> Cheers,
>
> Chris
>
> On Tue, Apr 11, 2023 at 3:23 PM Greg Harris <greg.har...@aiven.io.invalid>
> wrote:
>
>> Chris & Yash,
>>
>> 1. Since the global offsets topic does not have transactions on it
>> already,
>> I don't think adding transactions just for these reset operations would be
>> an improvement. The transactional produce would not exclude other
>> non-transactional producers, but hanging transactions on the global
>> offsets
>> topic would negatively impact the general cluster health. Your proposed
>> strategy seems reasonable to me.
>>
>> 2. While it may be the connector performing the offset reset and not the
>> task, I think it would be preferable for the connector to use task 0's
>> task-id and 'impersonate' the task for the purpose of changing the
>> offsets.
>> I think the complication elsewhere (getting users to provide a new ACL,
>> expanding fencing to also fence the connector transaction id, etc) is not
>> practically worth it to change 1 string value in the logs.
>> I would find a separate transaction ID beneficial if the connector could
>> be
>> given a different principal from the task, and be given distinct ACLs.
>> However, I don't think this is possible or desirable, and so I don't think
>> it's relevant right now. Let me know if there are any other ways that the
>> connector transaction ID would be useful.
>>
>> Thanks for all the effort on this feature!
>> Greg
>>
>> On Tue, Apr 11, 2023 at 7:52 AM Chris Egerton <chr...@aiven.io.invalid>
>> wrote:
>>
>> > Hi all,
>> >
>> > A couple slight tweaks to the design have been proposed during
>> > implementation and I'd like to report them here to make sure that
>> they're
>> > acceptable to all who previously voted for this KIP. I've updated the
>> KIP
>> > to include these changes but will be happy to revert and/or amend if
>> there
>> > are any concerns.
>> >
>> > 1. We would like to refrain from using a transaction when resetting
>> source
>> > connector offsets in the worker's global offsets topic when exactly-once
>> > support is enabled. We would continue to use a transaction when
>> resetting
>> > offsets in the connector's offsets topic. Discussed in [1].
>> >
>> > 2. We would like to use a transactional ID of ${groupId}-${connector} to
>> > alter/reset source connector offsets when exactly-once support is
>> enabled,
>> > where ${groupId} is the group ID of the Connect cluster and
>> ${connector} is
>> > the name of the connector. This is raised here because it would
>> introduce
>> > an additional ACL requirement for this API. A less-elegant alternative
>> that
>> > would obviate the additional ACL requirement is to use the
>> transactional ID
>> > that would be used by task 0 of the connector, but this may be
>> confusing to
>> > users as it could indicate that the task is actually running. Discussed
>> in
>> > [2].
>> >
>> > [1] -
>> https://github.com/apache/kafka/pull/13465/#issuecomment-1486718538
>> > [2] -
>> https://github.com/apache/kafka/pull/13465/#discussion_r1159694956
>> >
>> > Cheers,
>> >
>> > Chris
>> >
>> > On Fri, Mar 3, 2023 at 10:22 AM Chris Egerton <chr...@aiven.io> wrote:
>> >
>> > > Hi all,
>> > >
>> > > Thanks for the votes! I'll cast a final +1 myself and close the vote
>> out.
>> > >
>> > > This KIP passes with the following +1 votes (and no +0 or -1 votes):
>> > >
>> > > • Greg Harris
>> > > • Yash Mayya
>> > > • Knowles Atchison Jr
>> > > • Mickael Maison (binding)
>> > > • Tom Bentley (binding)
>> > > • Josep Prat (binding)
>> > > • Chris Egerton (binding, author)
>> > >
>> > > I'll write up Jira tickets and begin implementing things next week.
>> > >
>> > > Cheers,
>> > >
>> > > Chris
>> > >
>> > > On Fri, Mar 3, 2023 at 10:07 AM Josep Prat
>> <josep.p...@aiven.io.invalid>
>> > > wrote:
>> > >
>> > >> Hi Chris,
>> > >>
>> > >> Thanks for the KIP. I have a non-blocking comment on the DISCUSS
>> thread.
>> > >>
>> > >> +1 (binding).
>> > >>
>> > >> Best,
>> > >>
>> > >> On Wed, Mar 1, 2023 at 12:16 PM Tom Bentley <tbent...@redhat.com>
>> > wrote:
>> > >>
>> > >> > Hi Chris,
>> > >> >
>> > >> > Thanks for the KIP.
>> > >> >
>> > >> > +1 (binding).
>> > >> >
>> > >> > Cheers,
>> > >> >
>> > >> > Tom
>> > >> >
>> > >> > On Wed, 15 Feb 2023 at 16:11, Chris Egerton
>> <chr...@aiven.io.invalid>
>> > >> > wrote:
>> > >> >
>> > >> > > Hi all,
>> > >> > >
>> > >> > > Thanks to everyone who's voted so far! Just wanted to bump this
>> > thread
>> > >> > and
>> > >> > > see if we could get a few more votes; currently we're at +3
>> > >> non-binding
>> > >> > > and +1 binding. Hoping we can get this approved, reviewed, and
>> > merged
>> > >> in
>> > >> > > time for 3.5.0.
>> > >> > >
>> > >> > > Cheers,
>> > >> > >
>> > >> > > Chris
>> > >> > >
>> > >> > > On Tue, Jan 31, 2023 at 2:52 AM Mickael Maison <
>> > >> mickael.mai...@gmail.com
>> > >> > >
>> > >> > > wrote:
>> > >> > >
>> > >> > > > Thanks Chris for the KIP, this is a much needed feature!
>> > >> > > >
>> > >> > > > +1 (binding)
>> > >> > > >
>> > >> > > >
>> > >> > > > On Tue, Jan 24, 2023 at 3:45 PM Knowles Atchison Jr
>> > >> > > > <katchiso...@gmail.com> wrote:
>> > >> > > > >
>> > >> > > > > +1 (non binding)
>> > >> > > > >
>> > >> > > > > On Tue, Jan 24, 2023 at 5:24 AM Yash Mayya <
>> > yash.ma...@gmail.com>
>> > >> > > wrote:
>> > >> > > > >
>> > >> > > > > > Hi Chris,
>> > >> > > > > >
>> > >> > > > > > I'm +1 (non-binding). Thanks again for proposing this
>> > extremely
>> > >> > > > > > valuable addition to Kafka Connect!
>> > >> > > > > >
>> > >> > > > > > Thanks,
>> > >> > > > > > Yash
>> > >> > > > > >
>> > >> > > > > > On Thu, Jan 19, 2023 at 12:11 AM Chris Egerton
>> > >> > > <chr...@aiven.io.invalid
>> > >> > > > >
>> > >> > > > > > wrote:
>> > >> > > > > >
>> > >> > > > > > > Hi all,
>> > >> > > > > > >
>> > >> > > > > > > I'd like to call for a vote on KIP-875, which adds
>> support
>> > for
>> > >> > > > viewing
>> > >> > > > > > and
>> > >> > > > > > > manipulating the offsets of connectors to the Kafka
>> Connect
>> > >> REST
>> > >> > > API.
>> > >> > > > > > >
>> > >> > > > > > > The KIP:
>> > >> > > > > > >
>> > >> > > > > > >
>> > >> > > > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect
>> > >> > > > > > >
>> > >> > > > > > > The discussion thread:
>> > >> > > > > > >
>> > >> https://lists.apache.org/thread/m5bklnh5w4mwr9nbzrmfk0pftpxfjd02
>> > >> > > > > > >
>> > >> > > > > > > Cheers,
>> > >> > > > > > >
>> > >> > > > > > > Chris
>> > >> > > > > > >
>> > >> > > > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> > >>
>> > >> --
>> > >> [image: Aiven] <https://www.aiven.io>
>> > >>
>> > >> *Josep Prat*
>> > >> Open Source Engineering Director, *Aiven*
>> > >> josep.p...@aiven.io | +491715557497
>> > >> aiven.io <https://www.aiven.io> | <
>> > >> https://www.facebook.com/aivencloud>
>> > >> <https://www.linkedin.com/company/aiven/> <
>> > >> https://twitter.com/aiven_io>
>> > >> *Aiven Deutschland GmbH*
>> > >> Alexanderufer 3-7, 10117 Berlin
>> > >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>> > >> Amtsgericht Charlottenburg, HRB 209739 B
>> > >>
>> > >
>> >
>>
>
