Re: Apache Flink and serious streaming stateful processing

Hi! As Gyula mentioned an upcoming Pull Request will make the state backend pluggable. We would like to add the following state holders into Flink: (1) Small state in memory (local execution / debugging) : State maintained in a heap hash map, checkpoints to JobManager. This is in there now. (2)

Re: Apache Flink and serious streaming stateful processing

Hi! You have an interesting use case that I think comes up in many applications (in fact I will be working on something very similar shortly). Stephan has made some nice changes (this PR) to the State interfaces supporting flexible backends, which can b

Re: Apache Flink and serious streaming stateful processing

Hi guys! I'm sorry I have abandoned this thread but I had to give up Flink for some time. Now I'm back and would like to resurrect this thread. Flink has rapidly evolved in this time too, so maybe new features will allow me what I want to do. By the way, I heard really only good stuff about you fro

Re: Apache Flink and serious streaming stateful processing

Hi Krzysztof, Thanks for the kind words! I think that Flink is to a good extend set up and provide what you are looking for. The remaining gaps are WIP. Let me elaborate a bit on Gyula's answers: 1) Backpressure is very much there, it has always been working well, also better than in Storm, as f

Re: Apache Flink and serious streaming stateful processing

On 30 Jun 2015, at 14:23, Gyula Fóra wrote: > 2. We have support for stateful processing in Flink in many ways you have > described in your question. Unfortunately the docs are down currently but you > should check out the 'Stateful processing' section in the 0.10 docs (once its > back online)

Re: Apache Flink and serious streaming stateful processing

Hi Krzysztof, Thank you for your questions, we are happy to help you getting started. Regarding your questions: 1. There is backpressure for the streams, so if the downstream operators cannot keep up the sources will slow down. 2. We have support for stateful processing in Flink in many ways yo

Apache Flink and serious streaming stateful processing

Greetings! I'm extremely interested in Apache Flink, I think you're doing really a great job! But please allow me to share two things that I would require from Apache Flink to consider it as groundbreaking (it is what I need for Streaming framework): 1. Stream backpressure. When stream processing