Hi, would these super-structure events occur per key? If yes, then I think you can process this using the currently available windowing mechanism by writing a custom WindowAssigner and Trigger. This, of course, assumes that the events arrive in-order, i.e. if A-End arrives before A-Start or if elements that should fall inside the A window arrive after A-End then I don't see an easy way to do it.
Let me know if you need to know more about assigners/triggers. Cheers, Aljoscha On Mon, 13 Jun 2016 at 16:29 Paul Wilson <paulalexwil...@gmail.com> wrote: > Hi, > > I've been evaluating Flink and wondering if it was possible to define a > window that is based on characteristics of the data (data driven) but not > contained in the data stream directly. > > Consider 'nested events' where lower level events belong to a wider event > where the wider event serves only to define a boundary (or window) over the > lower level events. I was wondering if there was some way to communicate > this super-structure in the stream somehow? > > I know that Flink users 'barriers' to define snapshot boundaries, but it > might it be possible to communicate a 'window end' in a similar fashion? > > I guess I could attach an additional value to each event using a stateful > map function and then define the window on that? > > e.g. A-Start, 1, 2, 3, A-End, B-Start, 1, 2, 3, B-End > > Regards, > Paul >
