@Max, broadcast variables have to be declared before the program is executed. I want to be able to do something whereby I can send data inside a map operation to all nodes. This would perhaps have an effect similar to the recent discussion on making Accumulators available before Job completion. [I don't know if it has been merged yet.]
@Stephan, I was thinking more in the direction of adding a task at every worker which works as an Event handler to receive data from other nodes. I got the idea after going through the iterative runtime module which has a separate task for handling synchronization. I followed it up with going through the optimizer routine wherein it is added to some *auxiliary vertices *list. Can you elaborate on that? Further, is it possible to add an event handler to the runtime context itself? I wrote the boilerplate stuff for this but I haven't defined the channels where the event would be written to the network so other workers can also listen to it. [ https://github.com/sachingoel0101/flink/tree/async_iter]. I want to be able to call broadcast and receive as getRuntimeContext.broadcast and getRuntimeContext.receive, which ensures complete data transmission across nodes, even at runtime. I'm not sure how to add a separate task which handles event over the network like the IterationSynchronization task [which is the best use I could find of using event handlers]. Let me know your thoughts. Or if you can think of a better way to achieve sharing of data which is *only *evaluated at runtime. Cheers! Sachin -- Sachin Goel Computer Science, IIT Delhi m. +91-9871457685 On Mon, Jul 20, 2015 at 3:34 PM, Stephan Ewen <se...@apache.org> wrote: > You are probably looking for a parameter server tool. > > How about setting up one of these memory grids to use that? Apache Ignite, > or Apache Geode, or one of those. > > On Fri, Jul 17, 2015 at 6:19 PM, Sachin Goel <sachingoel0...@gmail.com> > wrote: > > > Hi all > > Is it possible to send and receive key,value pairs at runtime? I would > like > > to broadcast values at runtime so they are available on every node. This > > somehow seems essential for an implementation of Asynchronous batch > > iterations. > > > > What I would like is to have two functions in the RuntimeContext, > > broadcast(key,value) and receive(key). > > If such a thing doesn't exist now, how should I approach implementing it? > > > > Cheers! > > Sachin > > -- Sachin Goel > > Computer Science, IIT Delhi > > m. +91-9871457685 > > >
