Re: [DISCUSS] FLIP-5 Only send data to each taskmanager once for broadcasts

Hi everybody, I implemented the second approach (see https://cwiki.apache.org/confl uence/display/FLINK/FLIP-5%3A+Only+send+data+to+each+taskman ager+once+for+broadcasts). So each subpartition will be read by m tasks (m = number of task managers) and the other tasks will notify the subpartition th

Re: [DISCUSS] FLIP-5 Only send data to each taskmanager once for broadcasts

Hi everybody, the previous approach turned out to have an issue. Since we only write to one subpartition, we have N-1 empty subpartitions per Task (where N = degree of parallelism). In the current approach I didn't consume these empty subpartitions. When you don't consume a subpartition it won't b

Re: [DISCUSS] FLIP-5 Only send data to each taskmanager once for broadcasts

Cool first version Felix :-) On Wed, Aug 10, 2016 at 6:48 PM, Stephan Ewen wrote: > Cool, nice results! > > For the iteration unspecialization - we probably should design this hand in > hand with the streaming fault tolerance, as they share the notion of > "intermediate result versions". > > > O

Re: [DISCUSS] FLIP-5 Only send data to each taskmanager once for broadcasts

Cool, nice results! For the iteration unspecialization - we probably should design this hand in hand with the streaming fault tolerance, as they share the notion of "intermediate result versions". On Wed, Aug 10, 2016 at 6:09 PM, Felix Neutatz wrote: > Hi everybody, > > I found a quick and dir

Re: [DISCUSS] FLIP-5 Only send data to each taskmanager once for broadcasts

Hi everybody, I found a quick and dirty way to make the blocking subpartition readable by multiple readers. In the JobGraph generation I make all broadcast partitions blocking (see more details here: https://github.com/FelixNeutatz/incubator-flink/commits/blockingMultipleReads). I want to point ou

Re: [DISCUSS] FLIP-5 Only send data to each taskmanager once for broadcasts

I agree with Till. Changing the basic data exchange mechanism would screw up many other ongoing efforts, like more incremental recovery. It seems to make this properly applicable, we need to first un-specialize the iterations. (1) Allow for "versioned" intermediate results, i.e., result-x-superst

Re: [DISCUSS] FLIP-5 Only send data to each taskmanager once for broadcasts

Hi Felix, if we cannot work around the problem with blocking intermediate results in iterations, then we have to make FLINK-1713 a blocker for this new issue. But maybe you can also keep the current broadcasting mechanism to be used within iterations only. Then we can address the iteration problem

Re: [DISCUSS] FLIP-5 Only send data to each taskmanager once for broadcasts

Hi Till, thanks for the fast answer. I also think this should be the way to go. So should I open a new jira "Make blocking SpillableSubpartition able to be read multiple times". Moreover should I mark this jira and FLINK-1713 as blocking for the b

Re: [DISCUSS] FLIP-5 Only send data to each taskmanager once for broadcasts

Hi Felix, I'm not sure whether PipelinedSubpartition should be readable more than once because then it would effectively mean that we materialize the elements of the pipelined subpartition for stragglers. Therefore, I think that we should make blocking intermediate results readable more than once.

Re: [DISCUSS] FLIP-5 Only send data to each taskmanager once for broadcasts

Hi Stephan, I did some research about blocking intermediate results. It turns out that neither PipelinedSubpartition (see line 178) nor blocking intermediate results (see SpillableSubpartition line: 189) can be read multiple times. Moreover blocking intermediate results are currently not supported

Re: [DISCUSS] FLIP-5 Only send data to each taskmanager once for broadcasts

Hi Felix! Hope this helps_ Concerning (1.1) - The producer does not think in term of number of target TaskManagers. That number can, after all, change in the presence of a failure and recovery. The producer should, for its own result, not care how many consumers it will have (Tasks), but produce

Re: [DISCUSS] FLIP-5 Only send data to each taskmanager once for broadcasts

Hi Stephan, thanks for the great ideas. First I have some questions: 1.1) Does every task generate an intermediate result partition for every target task or is that already implemented in a way so that there are only as many intermediate result partitions per task manager as target tasks? (Exampl

Re: [DISCUSS] FLIP-5 Only send data to each taskmanager once for broadcasts

Hi Felix! Interesting suggestion. Here are some thoughts on the design. The two core changes needed to send data once to the TaskManagers are: (1) Every sender needs to produce its stuff once (rather than for every target task), there should not be redundancy there. (2) Every TaskManager sho