Hi Aljoscha, This looks like the bug that we discussed, as part of Enhance window evictor JIRA
Thanks, Vishnu On Wed, Aug 10, 2016 at 1:18 PM, Aljoscha Krettek <aljos...@apache.org> wrote: > Hi, > from your mail I'm gathering that you are in fact using an Evictor, is > that correct? If not, then the window operator should not keep all the > elements ever received for a window but only the aggregated result. > > Side note, there seems to be a bug in EvictingWindowOperator that causes > evicted elements to not actually be removed from the state. They are only > filtered from the Iterable that is given to the WindowFunction. I opened a > Jira issue for that: https://issues.apache.org/jira/browse/FLINK-4369 > > Cheers, > Aljoscha > > On Wed, 10 Aug 2016 at 18:19 Shannon Carey <sca...@expedia.com> wrote: > >> One unfortunate aspect of using a fold() instead of a window is that the >> fold function has no knowledge of the watermarks. As a result, it is >> difficult to ensure that only items before the current watermark are >> included in the aggregation, and that old items are evicted correctly. This >> fact lends more support to the idea of using a custom operator (though that >> is more complex) or adding support for this use case within Flink. >> >> -Shannon >> >
