Re: Keyby multiple elements

2018-02-16 Thread Fabian Hueske
Yes, you can do that. You would have to define a custom trigger. Alternatively, you can also generate more conservative watermarks. That would have the same effect. Best, Fabian 2018-02-16 12:25 GMT+01:00 Björn Zachrisson : > Hi Fabian, > > It does not since my events are out of order within a c

Re: Keyby multiple elements

2018-02-16 Thread Björn Zachrisson
Hi Fabian, It does not since my events are out of order within a certain interval and removing allowedLateness reduces the elements processed with 99.5%. Is it possible to trigger the window first when the allowed latness value has been passed? Regards Björn Zachrisson On 16 February 2018 at 12

Re: Keyby multiple elements

2018-02-16 Thread Fabian Hueske
Hi Björn, You configured an allowed lateness, so this might be caused by late arriving data. In case a late record is received within the allowed lateness, the window function will fire again for the same window. Does that also happen if you remove the allowedLateness() call? Best, Fabian 2018-

Keyby multiple elements

2018-02-16 Thread Björn Zachrisson
Hi, I have something weird probably a user error :) I'm running a keyby on multiple elements val keyedStream = nonKeyedStream .keyBy(m => (m.1, m.2, m.3.getOrElse(-1), m.4)) then apply a window function val appliedWindow = keyedStream .timeWindow(minutes(WindowTimeMinutes)) .allowedLaten