Re: TimeWindowAll doeesn't assign properly

2016-08-29 Thread Sendoh
Hi, As I found, the problem was rebalance() because the data arrives in 1 minute (it re-processes old events) and it's a bit strange that when configured watermark as 10 minutes it worked. After removing rebalance(), it works as expected that setting watermark less. DataStream streams = env.addS

Re: TimeWindowAll doeesn't assign properly

2016-08-01 Thread Aljoscha Krettek
Hi, yes, if you set the delay to high you will have to wait a long time until your windows are emitted. Cheers, Aljoscha On Mon, 1 Aug 2016 at 04:52 Sendoh wrote: > Probably `processAt` is not used adequately because after increasing > maxDelay > in watermark to 10 minutes it works as expected.

Re: TimeWindowAll doeesn't assign properly

2016-08-01 Thread Sendoh
Probably `processAt` is not used adequately because after increasing maxDelay in watermark to 10 minutes it works as expected. Is there any upper limit of setting this maxDelay? Because there might be too many windows are waiting for the last instance? Best, Sendoh -- View this message in con

Re: TimeWindowAll doeesn't assign properly

2016-08-01 Thread Sendoh
Thank you for helping the issue. Those single-element-windows arrive within seconds and delay is configured with watermark as 6 seconds. Following are some samples after investigated. ... {"hashCode":-1798107744,"count":1,"processAt":"2016-08-01T11:08:05.846","startDate":"2016-07-19T21:34:00.

Re: TimeWindowAll doeesn't assign properly

2016-07-29 Thread Aljoscha Krettek
Hi, the single-element-windows to me indicate that these originate from elements that arrived at the window operator after the watermark. In the current version of Flink these elements will be emitted as a single-element window. You can avoid this by writing a custom EventTimeTrigger that does not

TimeWindowAll doeesn't assign properly

2016-07-29 Thread Sendoh
Hi Flink users, We have an issue that TimeWindowAll() doesn't assign properly. The sum should be in the same window but is generated in separate windows. For example in the following, window 832348384 has window start time 2016-07-20T05:57:00.000 with counts 36, and there is another window 832348