Re: repartion locally to task manager

2015-06-22 Thread Stephan Ewen
Hi Ventura! Concerning (1) : What would be good is to make the "org.apache.flink.runtime.instance.InstanceConnectionInfo" in the getruntimeContext()'s RuntimeContext object. In order to do that, we could need to move that into the flink-core package. We could also rename it simply to "ConnectionI

Re: repartion locally to task manager

2015-06-08 Thread Ventura Del Monte
Hi Stephan, Many thank for your reply! 1) This would be a nice feature. I have already done something similar, if you told me which informations you would like to export in the runtime context, I could add them to my code, update unit tests and share them. 2) Yes, I have figured that out. Howeve

Re: repartion locally to task manager

2015-06-03 Thread Stephan Ewen
Hi Ventura! Sorry for the late response. Here are a few ideas or comments that may help you: 1) We want to make it possible for a function (such as MapFunction) to figure out on which TaskManager it is running. The mechanism would be something like "getRuntimeContext().getTaskManagerInformation()

repartion locally to task manager

2015-05-22 Thread Ventura Del Monte
Hello, I am trying to introduce a new feature in my flink project, I would like to shuffle (random repartition) my dataset only locally to a task manager, so that each internal worker will have a different set of objects to work on. I have looked to internal flink mechanism, and I know (i hope) ho