Re: flink streaming - window chaining example

2016-03-27 Thread Balaji Rajagopalan
val stream:DataStream[String] = env .addSource(new FlinkKafkaConsumer08[String]("topic_name", new SimpleStringSchema, prop)) val event:DataStream[SomeEventObj] = stream.map(MyMapFunction) val tenMinute:DataStream[AggEvents] = ridesByDeviceIdStream.timeWindowAll(Time.of(10, TimeUnit.MINUTES).tri

Storm topologies compatibility and exactly-once

2016-03-27 Thread Olivier Mallassi
hello all I was reading the apache flink documentation and was particularly interested in two things - compatibility with storm api. AFAIU(i need to do more tests), you can reuse storm defined topologies and run them on a Flink cluster. I was wondering if this is "still a bĂȘta" or production ready

Re: for loop slow

2016-03-27 Thread Chiwan Park
Hi Lydia, To build iterative algorithm on Flink, using API for iterations [1] would be better than using for-loop. Your program triggers multiple executions by multiple calling `next.gap.print()`. In each execution, Flink reads whole data redundantly and it cause performance to decrease. Regar

flink streaming - window chaining example

2016-03-27 Thread Chen Bekor
hi all! I'm just starting my way with flink and I have a design question. I'm trying to aggregate incoming events (source: kafka topic) on a 10min tumbling window in order to calculate the incoming events rate (total per minute). I would like to take this window and perform an additional window

Re: Memory Leak using ProcessingTimeTimers?

2016-03-27 Thread Aljoscha Krettek
Hi, you are right, this is a problem. In an earlier version we were only setting very few actual timers using the RuntimeContext because a firing timer will trigger all the timers with a lower timestamp that we have stored in the trigger queue. We have to change the lower level trigger service (in