Re: Window PURGE Behaviour 1.0.2 vs 1.1.3

2016-11-14 Thread Aljoscha Krettek
Not sure, because there's going to be two WindwoOperator classes on the class path when deserialising this on the cluster. On Mon, 14 Nov 2016 at 10:07 Ufuk Celebi wrote: > On 11 November 2016 at 18:19:30, Aljoscha Krettek (aljos...@apache.org) > wrote: > > Hi, > > I think the jar files in the l

Re: Window PURGE Behaviour 1.0.2 vs 1.1.3

2016-11-14 Thread Ufuk Celebi
On 11 November 2016 at 18:19:30, Aljoscha Krettek (aljos...@apache.org) wrote: > Hi, > I think the jar files in the lib folder are checked first so shipping the > WindowOperator with the job should not work. The WindowOperator is instantiated on the client side and shipped as user code to the clu

Re: Window PURGE Behaviour 1.0.2 vs 1.1.3

2016-11-11 Thread Aljoscha Krettek
Hi, I think the jar files in the lib folder are checked first so shipping the WindowOperator with the job should not work. Cheers, Aljoscha On Thu, 10 Nov 2016 at 17:48 Konstantin Knauf wrote: > Hi Aljoscha, > > alright, for the time being I have modified the WindowOperator and built > flink-st

Re: Window PURGE Behaviour 1.0.2 vs 1.1.3

2016-11-10 Thread Konstantin Knauf
Hi Aljoscha, alright, for the time being I have modified the WindowOperator and built flink-streaming-java for our team. When you only change the WindowOperator class, is it safe to just bundle it with the job? I.e. does this class have precedence over the class in the binary bundle of flink? Che

Re: Window PURGE Behaviour 1.0.2 vs 1.1.3

2016-11-10 Thread Aljoscha Krettek
Hi, there were some discussions on the ML and it seems that the consensus is to aim for a release this year. Let me think a bit more and get back to you on the other issues. Cheers, Aljoscha On Thu, 10 Nov 2016 at 11:47 Konstantin Knauf wrote: > Hi Aljoscha, > > unfortunately, I think, FLINK-4

Re: Window PURGE Behaviour 1.0.2 vs 1.1.3

2016-11-10 Thread Konstantin Knauf
Hi Aljoscha, unfortunately, I think, FLINK-4994 would not solve our issue. What does "on the very end" mean in case of a GlobalWindow? FLINK-4369 would fix my workaround though. Is there already a timeline for Flink 1.2? Cheers, Konst On 10.11.2016 10:19, Aljoscha Krettek wrote: > Hi Konstant

Re: Window PURGE Behaviour 1.0.2 vs 1.1.3

2016-11-10 Thread Aljoscha Krettek
Hi Konstantin, evicting elements not being evicted is a bug that should be fixed for Flink 1.2: https://issues.apache.org/jira/browse/FLINK-4369. The check about non-existing window state when triggering was introduced because otherwise a Trigger could return FIRE and then there would be nothing t

Re: Window PURGE Behaviour 1.0.2 vs 1.1.3

2016-11-09 Thread Konstantin Knauf
Hi Aljoscha, as it turns out the "workaround" I was thinking was functionally working, but had a so to say memory leak. I was under the impression that evicted elements will be removed from the window state... Anyway, I think that this (triggers not being evaluated when the window state is null)

Re: Re: Window PURGE Behaviour 1.0.2 vs 1.1.3

2016-11-09 Thread Aljoscha Krettek
Could you go into some detail of why you need to keep the trigger state? Just the basics because you probably cannot (should not) talk about your internal stuff. On Wed, 9 Nov 2016 at 13:16 Konstantin Knauf wrote: > Sounds good Aljoscha. > > sent from my phone. Plz excuse brevity and tpyos. > -

AW: Re: Window PURGE Behaviour 1.0.2 vs 1.1.3

2016-11-09 Thread Konstantin Knauf
Sounds good Aljoscha. sent from my phone. Plz excuse brevity and tpyos. --- Konstantin Knauf *konstantin.kn...@tngtech.com * +49-174-3413182 TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke Aljoscha Krettek s

Re: Window PURGE Behaviour 1.0.2 vs 1.1.3

2016-11-09 Thread Aljoscha Krettek
Hi, exactly for this case I want to make a change to when Trigger.clear() is called: https://issues.apache.org/jira/browse/FLINK-4994 Right now, clear is called when the window is being garbage collected because we passed the allowed lateness (after this, nothing will ever be added to a window aga

Re: Window PURGE Behaviour 1.0.2 vs 1.1.3

2016-11-08 Thread Konstantin Knauf
Hi Aljoscha, interesting, this explains it. Well, in our case the PURGE in the onProcessingTimeTimer is only used to clear KeyValueStates*, and at this point there are usually no records in the window state. Any Ideas? I do have a workaround with an evictor, but it seemed to be unnecessarily com

Re: Window PURGE Behaviour 1.0.2 vs 1.1.3

2016-11-08 Thread Aljoscha Krettek
Hi, the timers are not actually deleted but the WindowOperator will check whether there is any window state associated with the window for which the timer fires. If there is no window state the timer will silently be ignored. Is this a problem for you or did you just want to clarify? If yes, then