Re: Doubt about window and count trigger

2015-11-26 Thread Matthias J. Sax
Hi, a Trigger is an *additional* condition for intermediate (early) evaluation of the window. Thus, it is not "or-ed" to the basic window definition. If you want to have an or-ed window condition, you can customize it by specifying your own window definition. > dataStream.window(new MyOwnWindow(

Doubt about window and count trigger

2015-11-26 Thread Anwar Rizal
Hi all, >From the documentation: "The Trigger specifies when the function that comes after the window clause (e.g., sum, count) is evaluated (“fires”) for each window." So, basically, if I specify: keyedStream .window(TumblingTimeWindows.of(Time.of(5, TimeUnit.SECONDS)) .trigger(CountTri

Cleanup of OperatorStates?

2015-11-26 Thread Niels Basjes
Hi, I'm working on a streaming application that ingests clickstream data. In a specific part of the flow I need to retain a little bit of state per visitor (i.e. keyBy(sessionid) ) So I'm using the Key/Value state interface (i.e. OperatorState) in a map function. Now in my application I expect t

Working with the Windowing functionality

2015-11-26 Thread Niels Basjes
Hi, I'm trying to build something in Flink that relies heavily on the Windowing features. In essence what I want to build: I have clickstream data coming in via Kafka. Each record (click) has a sessionid and a timestamp. I want to create a window for each session and after 30 minutes idle I want

Re: Working with State example /flink streaming

2015-11-26 Thread Stephan Ewen
Hi! In streaming, there is no "end" of the stream when you would emit the final sum. That's why there are windows. If you do not want the partial sums, but only the final sum, you need to define what window in which the sum is computed. At the end of that window, that value is emitted. The window

Re: Working with State example /flink streaming

2015-11-26 Thread Lopez, Javier
Hi, thanks for the answer. It worked but not in the way we expected. We expect to have only one sum per ID and we are getting all the consecutive sums, for example: We expect this: (11,6) but we get this (11,1) (11,3) (11,6) (the initial values are ID -> 11, values -> 1,2,3). Here is the code we a