Re: schedule tasks `inside` Flink

2016-02-25 Thread Michal Fijolek
Thanks for help guys! Eventually I did implemented it as a RichFunction using open() and closed() methods. MichaƂ 2016-02-25 19:00 GMT+01:00 Stephan Ewen : > Fabian's suggestion with the co-map is good. You can use a "broadcast()" > connect to make sure the dictionary gets to all nodes. > > If y

Re: schedule tasks `inside` Flink

2016-02-25 Thread Stephan Ewen
Fabian's suggestion with the co-map is good. You can use a "broadcast()" connect to make sure the dictionary gets to all nodes. If you want full control about how and when to read the data, a scheduled task is not that bad even as a solution. Make sure you implement this as a "RichFunction", so y

Re: schedule tasks `inside` Flink

2016-02-15 Thread Fabian Hueske
Hi Michal, If I got your requirements right, you could try to solve this issue by serving the updates through a regular DataStream. You could add a SourceFunction which periodically emits a new version of the cache and a CoFlatMap operator which receives on the first input the regular streamed inp

schedule tasks `inside` Flink

2016-02-14 Thread Michal Fijolek
Hello. My app needs Map[K, V] as simple cache for business data, which needs to be invalidated periodically, lets say once per day. Right now I'm using rather naive approach which is trait Dictionary[K, V] extends Serializable { @volatile private var cache: Map[K, V] = Map() def lookup(id: K):