On Sunday, July 7, 2013 9:20:11 AM UTC+2, Maarten Derickx wrote: > > This indeed seems like a good addition. I'm not aware yet of a place in > sage where something like this is done. > > I would chose a different name though, something like "bundled_parallel", > because naming this one unordered seems to indirectly imply that the other > one is ordered, which it is absolutely not. >
Perhaps sliced_parallel or parallel_slices or something similar? I don't think I've seen the term "bundled" for this operation before. Note that the semantics are going to be slightly difficult to explain: contrary to the normal parallel, state of one data point calculation does propagate to the next data point in the same slice. Finally, I think the right level to split this from the usual "parallel" implementation is p_iter_fork. In fact, you may be able to borrow its interface entirely and just slice using the "ncpus" argument. In fact, here you already see that slicing the iterator in a fixed way may not be the best interface. Perhaps a better approach is to have "ncpus" worker processes but instead of forking new ones every time, as p_iter_fork does, leave them running and feed them data via some inter process communication protocol (pipe or otherwise). Then you can schedule the jobs more dynamically than by slicing the work load beforehand. In fact, have you checked that what you end up implementing isn't essentially what "sage.parallel.multiprocessing_sage" does? In particular: sage: @parallel(ncpus=4,p_iter='multiprocessing') ....: def square_m(x): ....: return x^2 ....: sage: @parallel(ncpus=4,p_iter='fork') ....: def square_f(x): ....: return x^2 ....: sage: %time Lm=list(square_m(range(1000))) CPU times: user 0.06 s, sys: 0.02 s, total: 0.08 s Wall time: 0.12 s sage: %time Lf=list(square_f(range(1000))) CPU times: user 0.21 s, sys: 3.00 s, total: 3.21 s Wall time: 5.89 s suggests that with multiprocessing, the overhead is already much better (but perhaps you can improve even further). In any case, I'd think the appropriate interface would be to extend @parallel with another p_iter option, not to make another decorator. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/groups/opt_out.
