Re: Windows, watermarks, and late data

{ > > // FLINK-2870 should provide a more idiomatic way to ignore late > > arrivals > >} > > } > >} > > > > } > > ``` > > > > > > > > -- > > View this message in context: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Windows-watermarks-and-late-data-tp5239p5291.html > > Sent from the Apache Flink User Mailing List archive. mailing list > archive at Nabble.com. > >

Re: Windows, watermarks, and late data

tream[T] = { > stream.filterStrictlyAscendingTime(timestampExtractor) { >(element, timestamp) => { > // FLINK-2870 should provide a more idiomatic way to ignore late > arrivals >} > } >} > > } > ``` > > > > -- &g

Re: Windows, watermarks, and late data

to ignore late arrivals } } } } ``` -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Windows-watermarks-and-late-data-tp5239p5291.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: Windows, watermarks, and late data

Ha, never mind, I realized I can just put the unique key into the aggregate object maintained by the FoldFunction. I'm still curious why RichWindowFunction (and RichFoldFunction) aren't supported for Scala WindowedStream.apply. Mike On Thu, Mar 3, 2016 at 4:50 PM, Michael Radford wrote: > Thank

Re: Windows, watermarks, and late data

Thank you, that was helpful. I didn't appreciate that a Trigger is fully in control of when to fire / purge regardless of the watermark. Now I am wondering the best way to distinguish different instances of the same time window with completely different data, vs. repeated fires that include data u

Re: Windows, watermarks, and late data

Hello Mike, The code that Aljiosha mentioned is here: https://github.com/kl0u/flink-examples/blob/master/src/main/java/com/dataartisans/flinksolo/customTriggers/EventTimeTriggerWithEarlyAndLateFiring.java

Re: Windows, watermarks, and late data

Hi, I did some initial work on extending the EventTimeTrigger a bit to allow more complex behavior. Specifically, this allows setting an “allowed lateness” after which elements should no longer lead to windows being emitted. Also, it allows to specify to keep an emitted window in memory and when

Windows, watermarks, and late data

I'm evaluating Flink for a reporting application that will keep various aggregates updated in a database. It will be consuming from Kafka queues that are replicated from remote data centers, so in case there is a long outage in replication, I need to decide what to do about windowing and late data.