Hi Aljoscha, unfortunately, I think, FLINK-4994 would not solve our issue. What does "on the very end" mean in case of a GlobalWindow?
FLINK-4369 would fix my workaround though. Is there already a timeline for Flink 1.2? Cheers, Konst On 10.11.2016 10:19, Aljoscha Krettek wrote: > Hi Konstantin, > evicting elements not being evicted is a bug that should be fixed for > Flink 1.2: https://issues.apache.org/jira/browse/FLINK-4369. > > The check about non-existing window state when triggering was introduced > because otherwise a Trigger could return FIRE and then there would be > nothing to fire. I guess if we did indeed fire the trigger even with > non-existing state then some people might wonder why no emission is > triggered when their trigger returns FIRE. I see your point though, that > the omitted firing is problematic for some cases. > > I think having clear() as proposed > in https://issues.apache.org/jira/browse/FLINK-4994 would solve your > case. You were using your own cleanup timer as a workaround because > clear() is currently also called on PURGE. With clear() only being > called at the very end this should work, correct? > > Cheers, > Aljoscha > > On Wed, 9 Nov 2016 at 19:53 Konstantin Knauf > <konstantin.kn...@tngtech.com <mailto:konstantin.kn...@tngtech.com>> wrote: > > Hi Aljoscha, > > as it turns out the "workaround" I was thinking was functionally > working, but had a so to say memory leak. I was under the impression > that evicted elements will be removed from the window state... > > Anyway, I think that this (triggers not being evaluated when the window > state is null) turns out to be blocker for us. > > Why is this check done? Since a user can do basically whatever she likes > in onProcessingTimeTimer() the comment > > // if we have no state, there is nothing to do > > is, well, just not true in some cases (e.g. state updates in our case). > > Cheers, > > Konstantin > > On 09.11.2016 14:17, Aljoscha Krettek wrote: > > Could you go into some detail of why you need to keep the trigger > state? > > > > Just the basics because you probably cannot (should not) talk > about your > > internal stuff. > > > > On Wed, 9 Nov 2016 at 13:16 Konstantin Knauf > > <konstantin.kn...@tngtech.com > <mailto:konstantin.kn...@tngtech.com> > <mailto:konstantin.kn...@tngtech.com > <mailto:konstantin.kn...@tngtech.com>>> wrote: > > > > Sounds good Aljoscha. > > > > sent from my phone. Plz excuse brevity and tpyos. > > --- > > Konstantin Knauf *konstantin.kn...@tngtech.com > <mailto:konstantin.kn...@tngtech.com> > > <mailto:konstantin.kn...@tngtech.com > <mailto:konstantin.kn...@tngtech.com>> * +49-174-3413182 > <tel:0174%203413182> > > <tel:0174%203413182> > > > > TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring > > Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert > Dahlke > > > > ---- Aljoscha Krettek schrieb ---- > > > > > > Hi, > > exactly for this case I want to make a change to when > > Trigger.clear() is > > called: https://issues.apache.org/jira/browse/FLINK-4994 > > > > Right now, clear is called when the window is being garbage > > collected because we passed the allowed lateness (after this, > > nothing will ever be added to a window again) and also when the > > Trigger returns PURGE or FIRE_AND_PURGE. > > > > I want to change it to only be called in the former case. We could > > possibly add an onPurge() callback to allow cleaning state on > purge > > or require people to put the code that they want to run on > PURGE in > > the Trigger method that returns the PURGE. > > > > What do you think? > > > > Cheers, > > Aljoscha > > > > On Tue, 8 Nov 2016 at 18:46 Konstantin Knauf > > <konstantin.kn...@tngtech.com > <mailto:konstantin.kn...@tngtech.com> > <mailto:konstantin.kn...@tngtech.com > <mailto:konstantin.kn...@tngtech.com>>> > > wrote: > > > > Hi Aljoscha, > > > > interesting, this explains it. Well, in our case the PURGE > in the > > onProcessingTimeTimer is only used to clear > KeyValueStates*, and > > at this > > point there are usually no records in the window state. > > > > Any Ideas? > > > > I do have a workaround with an evictor, but it seemed to be > > unnecessarily complicated. > > > > *We can not use clear()-callback for that, since this > state should > > survive the FIRE_AND_PURGEs in the onElement()-calls. > > > > Cheers, > > > > Konstantin > > > > > > On 08.11.2016 18:31, Aljoscha Krettek wrote: > > > Hi, > > > the timers are not actually deleted but the WindowOperator > > will check > > > whether there is any window state associated with the window > > for which > > > the timer fires. If there is no window state the timer will > > silently be > > > ignored. > > > > > > Is this a problem for you or did you just want to > clarify? If > > yes, then > > > we should work on finding a solution. > > > > > > Cheers, > > > Aljoscha > > > > > > On Tue, 8 Nov 2016 at 18:18 Konstantin Knauf > > > <konstantin.kn...@tngtech.com > <mailto:konstantin.kn...@tngtech.com> > > <mailto:konstantin.kn...@tngtech.com > <mailto:konstantin.kn...@tngtech.com>> > > <mailto:konstantin.kn...@tngtech.com > <mailto:konstantin.kn...@tngtech.com> > > <mailto:konstantin.kn...@tngtech.com > <mailto:konstantin.kn...@tngtech.com>>>> wrote: > > > > > > Hi everyone, > > > > > > I just migrated a streaming Job from 1.0.2 to 1.1.3 and > > stumbled across > > > a problem concerning one of our custom triggers. > > > > > > The trigger basically FIRE_AND_PURGEs multiple times in > > onElement() and > > > the window is PURGEd onProcessingTimeTimer(), but it > seems > > that the all > > > registered processing time timers are deleted everytime > > the window is > > > PURGEd. > > > > > > clear() is the default implementation, i.e. no-op. > > > > > > Just wanted to, if this is the expected behavior > > (processing time timers > > > being deleted on PURGE or FIRE_AND_PURGE) from Flink > 1.1 on? > > > > > > Cheers, > > > > > > Konstantin > > > > > > -- > > > Konstantin Knauf * konstantin.kn...@tngtech.com > <mailto:konstantin.kn...@tngtech.com> > > <mailto:konstantin.kn...@tngtech.com > <mailto:konstantin.kn...@tngtech.com>> > > > <mailto:konstantin.kn...@tngtech.com > <mailto:konstantin.kn...@tngtech.com> > > <mailto:konstantin.kn...@tngtech.com > <mailto:konstantin.kn...@tngtech.com>>> * +49-174-3413182 > <tel:0174%203413182> > > <tel:0174%203413182> > > > <tel:0174%203413182> > > > TNG Technology Consulting GmbH, Betastr. 13a, 85774 > > Unterföhring > > > Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. > > Robert Dahlke > > > Sitz: Unterföhring * Amtsgericht München * HRB 135082 > > > > > > > > > > -- > > Konstantin Knauf * konstantin.kn...@tngtech.com > <mailto:konstantin.kn...@tngtech.com> > > <mailto:konstantin.kn...@tngtech.com > <mailto:konstantin.kn...@tngtech.com>> * +49-174-3413182 > <tel:0174%203413182> > > <tel:0174%203413182> > > TNG Technology Consulting GmbH, Betastr. 13a, 85774 > Unterföhring > > Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. > Robert Dahlke > > Sitz: Unterföhring * Amtsgericht München * HRB 135082 > > > > -- > Konstantin Knauf * konstantin.kn...@tngtech.com > <mailto:konstantin.kn...@tngtech.com> * +49-174-3413182 > <tel:0174%203413182> > TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring > Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke > Sitz: Unterföhring * Amtsgericht München * HRB 135082 > -- Konstantin Knauf * konstantin.kn...@tngtech.com * +49-174-3413182 TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke Sitz: Unterföhring * Amtsgericht München * HRB 135082
signature.asc
Description: OpenPGP digital signature
