Hi Erik, Thanks for the KIP. I haven’t fully read the KIP yet but I agree with the weaknesses that you point out in it. I will continue to read it.
For your information, we are working full speed on implementing KIP-848 while also changing the internal threading model of consumer. Those changes are already extremely large so I would rather prefer to complete them before adding more on top of them. Moreover, I think that this KIP should build on top of KIP-848 now. Would you agree with this? Best, David Le ven. 13 oct. 2023 à 20:44, Erik van Oosten <e.vanoos...@grons.nl.invalid> a écrit : > Thanks Philip, > > No worries, I am not in a hurry. Knowing this is not forgotten is enough > for me. If there is anything I can do to help the process please let me > know. > > Kind regards, > Erik. > > > Op 13-10-2023 om 20:29 schreef Philip Nee: > > Hi Erik, > > > > Sorry for the delay, I have not finished reviewing the KIP, but I also > have > > not forgotten about it! > > > > In general, KIP review process can be lengthy, so I think mailing list is > > the best bet to get the committer's attention. > > > > P > > > > On Fri, Oct 13, 2023 at 10:55 AM Erik van Oosten > > <e.vanoos...@grons.nl.invalid> wrote: > > > >> Hi client developers, > >> > >> The text is updated so that it is more clear that you can only use > >> auto-commit when doing synchronous processing (approach 1). I am > >> assuming that auto-commit commits whatever was consumed in the previous > >> poll. > >> > >> I am wondering why this KIP doesn't get more attention. Is async > >> processing not something that the kafka client wants to support? > >> > >> Kind regards, > >> Erik. > >> > >> > >> Op 25-09-2023 om 18:17 schreef Erik van Oosten: > >>> Hi Viktor, > >>> > >>> Good questions! > >>> > >>> 1. Auto-commits would only work with approach 1 in the KIP. Any async > >>> solution is incompatible with auto-commits. Do you think the text will > >>> improve when this is mentioned? > >>> > >>> 2. That is entirely correct. If you use async commits you can await > >>> completion by doing a single sync commit with an empty offsets Map > >>> (this will work as of Kafka 3.6.0). > >>> > >>> Is there anything I can do to make the text clearer? > >>> > >>> Kind regards, > >>> Erik. > >>> > >>> > >>> Op 25-09-2023 om 17:04 schreef Viktor Somogyi-Vass: > >>>> Hi Erik, > >>>> > >>>> I'm still trying to wrap my head around the KIP, however I have a few > >>>> questions that weren't clear to me regarding offset commits: > >>>> 1. Would auto-commits interfere with the behavior defined in your KIP > or > >>>> would it work the same as manual commits? > >>>> 2. As I see you don't separate offset commits by whether they're sync > or > >>>> async. For sync commits timing isn't really a problem but how would > you > >>>> change work in case of async offset commits? There can be a few > caveats > >>>> there as you may not know whether a commit is finished or not until > your > >>>> callback is called. > >>>> > >>>> Thanks, > >>>> Viktor > >>>> > >>>> On Sat, Sep 23, 2023 at 4:00 PM Erik van Oosten > >>>> <e.vanoos...@grons.nl.invalid> wrote: > >>>> > >>>>> Hi all, > >>>>> > >>>>> I would like to start the discussion on KIP-983: Full speed async > >>>>> processing during rebalance [1]. > >>>>> > >>>>> The idea is that we can prevent the drop in throughput during a > >>>>> cooperative rebalance. > >>>>> > >>>>> I am curious to your ideas and comments. > >>>>> > >>>>> Kind regards, > >>>>> Erik. > >>>>> > >>>>> [1] > >>>>> > >>>>> > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-983%3A+Full+speed+async+processing+during+rebalance > >> -- > >> Erik van Oosten > >> e.vanoos...@grons.nl > >> https://day-to-day-stuff.blogspot.com > >> > >> > -- > Erik van Oosten > e.vanoos...@grons.nl > https://day-to-day-stuff.blogspot.com > >
