Hi, I have a master script that executes two sequences (lists) of child
scripts, i.e. script_1 to script_3, and script_4 to_script_6 (the structure
is attached as a png file).
The execution is sequential, e.g. running script_1, then 2 then 3.
After executing the 1st sequence (script_1 to 3), master will execute the
2nd sequence (script_4 to 6).
Each child script will be calling a multiprocessing function to process a
task.
Master script is like,
for seq in seqs_to_launch:
for script in seq:
script().execute(data)
Each child script is like,
import multi_process_update
class Script_n():
def execute(self, data):
# some data processing
multi_process_task(data_task, task_name)
The multiprocessing function is like,
def multi_process_task(tasks, task_name):
cores_to_use = how_many_core()
handler = task_handling(task_name) # task handling class
task_blocks = slice_list(tasks, cores_to_use)
for block in task_blocks:
# spawn processes for each row block assigned to every cpu core
p = multiprocessing.Process(target=handler.execute, args=(block,))
p.start()
I like to know how to notify the master when the 2nd sequence is finished
that all multiprocess tasks (seq 1 and 2) are completed, and then maybe
close the multiprocessing.
thanks
--
https://mail.python.org/mailman/listinfo/python-list
