Yes, thanks, Liam! By the way, There's actually already a ticket to try and improve the API, and the discussed solution is basically the same thing I said had never occurred to me before, so I'm not sure what to say about that...
https://issues.apache.org/jira/browse/KAFKA-8924 The ticket seems abandoned, though, so it might be up for grabs if you want to make sure it gets resolved asap. I'll add a comment with my proposal. Thanks, -John On Mon, Apr 20, 2020, at 01:08, Matthias J. Sax wrote: > Thanks for the PR! > > On 4/19/20 10:04 PM, Liam Clarke-Hutchinson wrote: > > PR submitted :) https://github.com/apache/kafka/pull/8520 > > > > On Mon, Apr 20, 2020 at 2:34 PM John Roesler <vvcep...@apache.org> wrote: > > > >> Hi Liam, > >> > >> That sounds like a good idea to me. In fact, I’d go so far as to say we > >> should just change the existing example to include a grace period, and not > >> bother with an extra example. That would put it front and center. > >> > >> A PR would be greatly appreciated! Thanks for the offer! > >> > >> Thanks, > >> John > >> > >> On Sun, Apr 19, 2020, at 19:58, Liam Clarke wrote: > >>> Hi Matthias, > >>> > >>> I think as an interim measure, if the windowing samples in the docs > >> showed > >>> an additional example where the grace period was set (with perhaps a > >>> comment about the current default grace period, and planned future > >>> changes?) it would make it sufficiently visible - happy to submit a PR > >> with > >>> those changes if it seems appropriate. > >>> > >>> Cheers, > >>> > >>> Liam Clarke-Hutchinson > >>> > >>> On Mon, Apr 20, 2020 at 12:12 PM Matthias J. Sax <mj...@apache.org> > >> wrote: > >>> > >>>> I would prefer to not make the grace-period a mandatory argument and > >>>> keep the API as-is. I understand the issue of backward compatibility, > >>>> but I would still argue that we should just change the default grace > >>>> period to 0 in the 3.0 release. It's a major release and thus it seems > >>>> to be fine. To prepare for this change, we could start to log a WARN > >>>> message, if a user does not set the grace period explicitly for now. > >>>> > >>>> Just my 2 ct. Thoughts? > >>>> > >>>> -Matthias > >>>> > >>>> On 4/19/20 7:40 AM, John Roesler wrote: > >>>>> Oh, man, that’s a good idea. > >>>>> > >>>>> I can propose to deprecate (not remove) the existing ‘of’ factory > >> method > >>>> and add one with a mandatory grace period. Not sure why I didn’t think > >> of > >>>> that before. Probably too caught up in looking for something “smart”. > >>>>> > >>>>> Thanks! > >>>>> John > >>>>> > >>>>> On Sun, Apr 19, 2020, at 02:27, Liam Clarke wrote: > >>>>>> Hi John, > >>>>>> > >>>>>> I can't really think of a way to make it more obvious without > >> breaking > >>>>>> backwards compatibility - e.g., obvious easy fix is that grace > >> period > >>>> is a > >>>>>> mandatory arg to TimeWindows, but that would definitely break > >>>> compatibility. > >>>>>> > >>>>>> Cheers, > >>>>>> > >>>>>> Liam Clarke-Hutchinson > >>>>>> > >>>>>> On Thu, Apr 16, 2020 at 1:59 AM John Roesler <vvcep...@apache.org> > >>>> wrote: > >>>>>> > >>>>>>> Boom, you got it, Liam! Nice debugging work. > >>>>>>> > >>>>>>> This is a pretty big bummer, but I had to do it that way for > >>>>>>> compatibility. I added a log message to try and help reduce the > >> risk, > >>>> but > >>>>>>> it’s still kind of a trap. > >>>>>>> > >>>>>>> I’d like to do a KIP at some point to consider changing the default > >>>> grace > >>>>>>> period, but haven’t done it because it’s not clear what the default > >>>> should > >>>>>>> be. > >>>>>>> > >>>>>>> Please let me know if you have any ideas! > >>>>>>> Thanks, > >>>>>>> -John > >>>>>>> > >>>>>>> > >>>>>>> On Tue, Apr 14, 2020, at 23:44, Liam Clarke wrote: > >>>>>>>> And the answer is to change > >>>>>>>> .windowedBy(TimeWindows.of(Duration.ofMillis(5000))) > >>>>>>>> and specify the grace period: > >>>>>>>> > >>>>>>>> > >>>>>>> > >>>> > >> windowedBy(TimeWindows.of(Duration.ofMillis(5000)).grace(Duration.ofMillis(100))) > >>>>>>>> > >>>>>>>> On Wed, Apr 15, 2020 at 4:34 PM Liam Clarke < > >>>> liam.cla...@adscale.co.nz> > >>>>>>>> wrote: > >>>>>>>> > >>>>>>>>> Okay, doing some debugging it looks like I'm seeing this > >> behaviour > >>>>>>> because > >>>>>>>>> it's picking up a grace duration of 86,395,000 ms in > >>>>>>>>> KTableImpl.buildSuppress, which would happen to be 5000 millis > >> (my > >>>>>>> window > >>>>>>>>> size) off 24 hours, so I've got some clues! > >>>>>>>>> > >>>>>>>>> On Wed, Apr 15, 2020 at 3:43 PM Liam Clarke < > >>>> liam.cla...@adscale.co.nz > >>>>>>>> > >>>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>>> Hi all, > >>>>>>>>>> > >>>>>>>>>> I have a case where I want to consume from a topic, count the > >> number > >>>>>>> of > >>>>>>>>>> certain ids in a given time period X, and emit a new record to a > >>>>>>> different > >>>>>>>>>> topic after that same time period X has elapsed containing the > >>>>>>> aggregated > >>>>>>>>>> value. > >>>>>>>>>> > >>>>>>>>>> I'm using suppress with Suppressed.untilWindowCloses, but > >> nothing is > >>>>>>> ever > >>>>>>>>>> emitted, nor is my peek placed after the suppress ever being > >> hit. > >>>>>>>>>> My code is in the below Gist - I've hardcoded the durations for > >> 5 > >>>>>>> seconds > >>>>>>>>>> after testing purposes: > >>>>>>>>>> > >>>> https://gist.github.com/LiamClarkeNZ/24121ccf0f09e4530749cbd92633fa46 > >>>>>>>>>> > >>>>>>>>>> I'm assuming I've misunderstood something drastically, and would > >>>>>>> greatly > >>>>>>>>>> appreciate a pointer on where I may have gone wrong. I'm > >> wondering > >>>> if > >>>>>>> I > >>>>>>>>>> need a larger retention on the persistent store? > >>>>>>>>>> > >>>>>>>>>> I understand that events have to arrive in order for windows to > >>>>>>> close, so > >>>>>>>>>> I've sent events after the window has expired to attempt to > >> move the > >>>>>>> window > >>>>>>>>>> on, and my first peek (before the suppression) is emitting as I > >> do: > >>>>>>>>>> > >>>>>>>>>> 1. 2020-04-15T03:36:48.569Z > >> e2442bef-72bf-4424-b94e-7e4743e03c5e - 1 > >>>>>>>>>> 1. 2020-04-15T03:37:11.682Z > >> e2442bef-72bf-4424-b94e-7e4743e03c5e - 1 > >>>>>>>>>> 1. 2020-04-15T03:39:18.882Z aqgzftnvyn - 1 > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> Any guidance greatfully appreciated. > >>>>>>>>>> > >>>>>>>>>> Kind regards, > >>>>>>>>>> > >>>>>>>>>> Liam Clarke > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>> > >>>> > >>> > >> > > > > > Attachments: > * signature.asc
