Re: Flink Dashboard

Hi Till, thanks for the clarification. It all makes sense now. So the keyBy call is more a partitioning scheme and less of an operator, similar to Storm's field grouping, and Flink's other schemes such as forward and broadcast. The difference is that it produces KeyedStreams, which are a prere

Why would slot sharing be undesirable?

Hi, i am deploying a 5 stage pipeline, with varying DOPs for experimental purposes. The benefits of slot sharing are clear. What is unclear to me is when slot sharing would be undesirable. Since slot sharing enables bypassing of additional buffers (buffers between slots) as well as memory base

Re: Internal buffers

Dear Ufuk, the wiki entry is exactly what i was looking for. I found it quite complicated to understand on a first attempt but i will dedicate some more time for it in the future. Thanks. Regards Leon 1. Jun 2016 13:06 by u...@apache.org: > There is this in the Wiki: > https://cwiki.apache.

Re: Maintaining watermarks per key, instead of per operator instance

Hi again Aljoscha, understood. Thanks for the link. I really like the straightforward approach concerning storing state. It makes things very easy. The improvements are very interesting, particularly the composite triggers. That would significantly improve flexibility. Kind regards Leon 1. Ju

Internal buffers

I have a question regarding how tuples are buffered between (possibly chained) subtasks. Is it correct that there is a buffer for each vertex in the DAG of subtasks? Regardless of task slot sharing? If yes, then the primary optimization in this regard is operator chaining. Furthermore, how do

Re: Maintaining watermarks per key, instead of per operator instance

Hi Aljoscha, thanks for the speedy reply. I am processing measurements delivered by smart meters. I use windows to gather measurements and calculate values such as average consumption. The key is simply the meter ID. The challenge is that meters may undergo network partitioning, under which t

Maintaining watermarks per key, instead of per operator instance

My use case primarily concerns applying transformations per key, with the keys remaining fixed throughout the topology. I am using event time for my windows. The problem i am currently facing is that watermarks in windows propagate per operator instance, meaning the operator event time increase

Re: re: Elegantly sharing state in a streaming environment

Dear Philippe, that is exactly what i need. Thank you for the concise explanation. This approach is excellent, as it also permits the values to be easily updated externally. Kind regards Leon 30. May 2016 14:31 by philippe.capar...@orange.fr: > > > Just transform the list in a DataStream. A

Elegantly sharing state in a streaming environment

Hello Flink team, How can i partition and share static state among instances of a streaming operator? I have a huge list of keys and values, which are used to filter tuples in a stream. The list does not change. Currently i am sharing the list with each operator instance via the constructor, a

Key factors for Flink's performance

Hello Flink team, i am currently playing around with Storm and Flink in the context of a smart home. The primary functional requirement is to quickly react to certain properties in stream tuples. I was looking at some benchmarks from the two systems, and generally Flink has the upper hand, in bo