Re: Communicating between nodes at runtime

2015-07-21 Thread Sachin Goel
@Stephan, why would there be a problem with the reliability of updates? Also, what would be the best way to achieve this functionality? Preferably using the current API. Cheers! Sachin -- Sachin Goel Computer Science, IIT Delhi m. +91-9871457685 On Tue, Jul 21, 2015 at 4:17 PM, Stephan Ewen wr

Re: Communicating between nodes at runtime

2015-07-21 Thread Stephan Ewen
I see the use case for that, but I don't think that this should be realized through Flink's network stack. The network stack is designed for continuous streams, backpressure, and configurable throughput/latency. I think it is not the best solution for asynchronous messaging. Also, for the ML field

Re: Communicating between nodes at runtime

2015-07-20 Thread Sachin Goel
@Max, broadcast variables have to be declared before the program is executed. I want to be able to do something whereby I can send data inside a map operation to all nodes. This would perhaps have an effect similar to the recent discussion on making Accumulators available before Job completion. [I

Re: Communicating between nodes at runtime

2015-07-20 Thread Stephan Ewen
You are probably looking for a parameter server tool. How about setting up one of these memory grids to use that? Apache Ignite, or Apache Geode, or one of those. On Fri, Jul 17, 2015 at 6:19 PM, Sachin Goel wrote: > Hi all > Is it possible to send and receive key,value pairs at runtime? I woul

Re: Communicating between nodes at runtime

2015-07-20 Thread Maximilian Michels
Hi Sachin, Do you know about Broadcast Variables? They allow you to transfer a DataSet to all nodes. https://ci.apache.org/projects/flink/flink-docs-master/apis/programming_guide.html#broadcast-variables Let us know if that fits your needs. Cheers, Max On Fri, Jul 17, 2015 at 6:19 PM, Sachin G

Communicating between nodes at runtime

2015-07-17 Thread Sachin Goel
Hi all Is it possible to send and receive key,value pairs at runtime? I would like to broadcast values at runtime so they are available on every node. This somehow seems essential for an implementation of Asynchronous batch iterations. What I would like is to have two functions in the RuntimeConte