On Sat, Jun 20, 2015 at 1:41 AM, Steven D'Aprano <st...@pearwood.info> wrote: > On Sat, 20 Jun 2015 12:01 am, Fabien wrote: > >> Folks, >> >> I am developing a tool which works on individual entities (glaciers) and >> do a lot of operations on them. There are many tasks to do, one after >> each other, and each task follows the same interface: > > I'm afraid your description is contradictory. Here you say the tasks run one > after another, but then you say: > >> This way, the tasks can be run in parallel very easily: > > and then later still you contradict this: > >> Also, the task2 should not be run if task1 threw an error. > > > If task2 relies on task1, then you *cannot* run them in parallel. You have > to run them one after the other, sequentially.
AIUI what he's doing is all the subparts of task1 in parallel, then all the subparts of task2: pool.map(task1, dirs, chunksize=1) pool.map(task2, dirs, chunksize=1) pool.map(task3, dirs, chunksize=1) task1 can be done on all of dirs in parallel, as no instance of task1 depends on any other instance of task1; but task2 should be started only if all task1s finish successfully. OP, is this how it is? If not, I apologize for the noise. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
