Hi Richard, Thanks for the KIP.
After reading it over, I'm wondering if you saw this one: https://cwiki.apache.org/confluence/display/KAFKA/KIP-424%3A+Allow+suppression+of+intermediate+events+based+on+wall+clock+time It seems like it overlaps with your use case, where you're willing to get events flushed out of the suppression buffer after some wall clock time passes, rather than solely based on stream time. WDYT? Thanks, -John On Sun, Oct 20, 2019 at 11:50 AM Richard Yu <yohan.richard...@gmail.com> wrote: > > Hi Bill, John, > > I've made some changes to the KIP after some thinking. I've come up with a > reasonable solution, I believe, to relieve the problem that is associated > with low traffic suppression buffers. Would be great if I can get your > input on this. :) > > Cheers, > Richard > > On Fri, Oct 18, 2019 at 7:16 PM Richard Yu <yohan.richard...@gmail.com> > wrote: > > > Hi Bill, > > > > Thanks for the input! > > TBH, I am think that suppression buffers are not used *in response *to > > low traffic conditions. > > Rather, we are trying to fix the situation when low traffic conditions > > occur in a suppression buffer (for example, previously, the same > > suppression buffer had a decent volume of records entering it, thus > > advancing the stream time). > > In summary, when these conditions do hit, we want to advance the stream > > time somehow to resolve this issue. > > Reflecting on this though, I am not completely certain if this really > > stops us from implementing per key stream time tracking because the problem > > wouldn't be made *that *much worse. > > > > Cheers, > > Richard > > > > On Fri, Oct 18, 2019 at 11:46 AM Bill Bejeck <bbej...@gmail.com> wrote: > > > >> Hi Richard, > >> > >> Thanks for the KIP proposal. I understand the situation you are > >> describing. > >> But in my mind, if there is a low traffic condition and you need to keep > >> records going downstream at regular intervals, I'm wondering if using > >> suppression is the correct approach. > >> IMHO it seems it would be better to use the PAPI or a Transform on the DSL > >> with a scheduled punctuation call. > >> > >> Just my 2 cents. > >> > >> Thanks, > >> Bill > >> > >> On Thu, Oct 17, 2019 at 7:42 PM Richard Yu <yohan.richard...@gmail.com> > >> wrote: > >> > >> > Hi all, > >> > > >> > I wish to discuss this KIP which would help us in resolving some issues > >> we > >> > have with suppression buffers. > >> > Below is the link: > >> > > >> > > >> > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-539%3A+Implement+mechanism+to+flush+out+records+in+low+volume+suppression+buffers > >> > > >> > @John Roesler if you have time, would be great if we could get your > >> input. > >> > > >> > Cheers, > >> > Richard > >> > > >> > >
