Thanks for the PR! On 4/19/20 10:04 PM, Liam Clarke-Hutchinson wrote: > PR submitted :) https://github.com/apache/kafka/pull/8520 > > On Mon, Apr 20, 2020 at 2:34 PM John Roesler <vvcep...@apache.org> wrote: > >> Hi Liam, >> >> That sounds like a good idea to me. In fact, I’d go so far as to say we >> should just change the existing example to include a grace period, and not >> bother with an extra example. That would put it front and center. >> >> A PR would be greatly appreciated! Thanks for the offer! >> >> Thanks, >> John >> >> On Sun, Apr 19, 2020, at 19:58, Liam Clarke wrote: >>> Hi Matthias, >>> >>> I think as an interim measure, if the windowing samples in the docs >> showed >>> an additional example where the grace period was set (with perhaps a >>> comment about the current default grace period, and planned future >>> changes?) it would make it sufficiently visible - happy to submit a PR >> with >>> those changes if it seems appropriate. >>> >>> Cheers, >>> >>> Liam Clarke-Hutchinson >>> >>> On Mon, Apr 20, 2020 at 12:12 PM Matthias J. Sax <mj...@apache.org> >> wrote: >>> >>>> I would prefer to not make the grace-period a mandatory argument and >>>> keep the API as-is. I understand the issue of backward compatibility, >>>> but I would still argue that we should just change the default grace >>>> period to 0 in the 3.0 release. It's a major release and thus it seems >>>> to be fine. To prepare for this change, we could start to log a WARN >>>> message, if a user does not set the grace period explicitly for now. >>>> >>>> Just my 2 ct. Thoughts? >>>> >>>> -Matthias >>>> >>>> On 4/19/20 7:40 AM, John Roesler wrote: >>>>> Oh, man, that’s a good idea. >>>>> >>>>> I can propose to deprecate (not remove) the existing ‘of’ factory >> method >>>> and add one with a mandatory grace period. Not sure why I didn’t think >> of >>>> that before. Probably too caught up in looking for something “smart”. >>>>> >>>>> Thanks! >>>>> John >>>>> >>>>> On Sun, Apr 19, 2020, at 02:27, Liam Clarke wrote: >>>>>> Hi John, >>>>>> >>>>>> I can't really think of a way to make it more obvious without >> breaking >>>>>> backwards compatibility - e.g., obvious easy fix is that grace >> period >>>> is a >>>>>> mandatory arg to TimeWindows, but that would definitely break >>>> compatibility. >>>>>> >>>>>> Cheers, >>>>>> >>>>>> Liam Clarke-Hutchinson >>>>>> >>>>>> On Thu, Apr 16, 2020 at 1:59 AM John Roesler <vvcep...@apache.org> >>>> wrote: >>>>>> >>>>>>> Boom, you got it, Liam! Nice debugging work. >>>>>>> >>>>>>> This is a pretty big bummer, but I had to do it that way for >>>>>>> compatibility. I added a log message to try and help reduce the >> risk, >>>> but >>>>>>> it’s still kind of a trap. >>>>>>> >>>>>>> I’d like to do a KIP at some point to consider changing the default >>>> grace >>>>>>> period, but haven’t done it because it’s not clear what the default >>>> should >>>>>>> be. >>>>>>> >>>>>>> Please let me know if you have any ideas! >>>>>>> Thanks, >>>>>>> -John >>>>>>> >>>>>>> >>>>>>> On Tue, Apr 14, 2020, at 23:44, Liam Clarke wrote: >>>>>>>> And the answer is to change >>>>>>>> .windowedBy(TimeWindows.of(Duration.ofMillis(5000))) >>>>>>>> and specify the grace period: >>>>>>>> >>>>>>>> >>>>>>> >>>> >> windowedBy(TimeWindows.of(Duration.ofMillis(5000)).grace(Duration.ofMillis(100))) >>>>>>>> >>>>>>>> On Wed, Apr 15, 2020 at 4:34 PM Liam Clarke < >>>> liam.cla...@adscale.co.nz> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Okay, doing some debugging it looks like I'm seeing this >> behaviour >>>>>>> because >>>>>>>>> it's picking up a grace duration of 86,395,000 ms in >>>>>>>>> KTableImpl.buildSuppress, which would happen to be 5000 millis >> (my >>>>>>> window >>>>>>>>> size) off 24 hours, so I've got some clues! >>>>>>>>> >>>>>>>>> On Wed, Apr 15, 2020 at 3:43 PM Liam Clarke < >>>> liam.cla...@adscale.co.nz >>>>>>>> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi all, >>>>>>>>>> >>>>>>>>>> I have a case where I want to consume from a topic, count the >> number >>>>>>> of >>>>>>>>>> certain ids in a given time period X, and emit a new record to a >>>>>>> different >>>>>>>>>> topic after that same time period X has elapsed containing the >>>>>>> aggregated >>>>>>>>>> value. >>>>>>>>>> >>>>>>>>>> I'm using suppress with Suppressed.untilWindowCloses, but >> nothing is >>>>>>> ever >>>>>>>>>> emitted, nor is my peek placed after the suppress ever being >> hit. >>>>>>>>>> My code is in the below Gist - I've hardcoded the durations for >> 5 >>>>>>> seconds >>>>>>>>>> after testing purposes: >>>>>>>>>> >>>> https://gist.github.com/LiamClarkeNZ/24121ccf0f09e4530749cbd92633fa46 >>>>>>>>>> >>>>>>>>>> I'm assuming I've misunderstood something drastically, and would >>>>>>> greatly >>>>>>>>>> appreciate a pointer on where I may have gone wrong. I'm >> wondering >>>> if >>>>>>> I >>>>>>>>>> need a larger retention on the persistent store? >>>>>>>>>> >>>>>>>>>> I understand that events have to arrive in order for windows to >>>>>>> close, so >>>>>>>>>> I've sent events after the window has expired to attempt to >> move the >>>>>>> window >>>>>>>>>> on, and my first peek (before the suppression) is emitting as I >> do: >>>>>>>>>> >>>>>>>>>> 1. 2020-04-15T03:36:48.569Z >> e2442bef-72bf-4424-b94e-7e4743e03c5e - 1 >>>>>>>>>> 1. 2020-04-15T03:37:11.682Z >> e2442bef-72bf-4424-b94e-7e4743e03c5e - 1 >>>>>>>>>> 1. 2020-04-15T03:39:18.882Z aqgzftnvyn - 1 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Any guidance greatfully appreciated. >>>>>>>>>> >>>>>>>>>> Kind regards, >>>>>>>>>> >>>>>>>>>> Liam Clarke >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>> >>>> >>> >> >
signature.asc
Description: OpenPGP digital signature
