John Kearney wrote:
> The example is bad anyway as you normally don't want to parallelize disk
> io , due to seek overhead and io bottle neck congestion. This example
> will be slower and more likely to damage your disk than simply using mv
> on its own. but thats another discussion.
---
That depends on how many IOPS your disk subsystem can
handle and how much cpu is between each of the IO calls.
Generally, unless you have a really old, non-queuing disk,
>1 procs will be of help. If you have a RAID, it can go
up with # of data spindles (as a max, though if all are reading
from the same area, not so much...;-))...
Case in point, I wanted to compare rpm versions of files
on disk in a dir to see if there were duplicate version, and if so,
only keep the newest (highest numbered) version) (with the rest
going into a per-disk recycling bin (a fall-out of sharing
those disks to windows and implementing undo abilities on
the shares (samba, vfs_recycle).
I was working directories with 1000's of files -- (1 dir,
after pruning has 10,312 entries). Sequential reading of those files
was DOG slow.
I parallelized it (using perl) first by sorting all the names,
then breaking it into 'N' lists -- doing those in parallel, then
merging the results (and comparing end-points -- like end of one list
might have been diff-ver from beginning of next). I found a dynamic
'N' based on max cpu load v.disk (i.e. no matter how many procs I
threw at it, it still used about 75% cpu).
So I chose 9:
Hot cache:
Read 12161 rpm names.
Use 1 procs w/12162 items/process
#pkgs=10161, #deletes=2000, total=12161
Recycling 2000 duplicates...Done
Cumulative This Phase ID
0.000s 0.000s Init
0.000s 0.000s start_program
0.038s 0.038s starting_children
0.038s 0.001s end_starting_children
8.653s 8.615s endRdFrmChldrn_n_start_re_sort
10.733s 2.079s afterFinalSort
17.94sec 3.71usr 6.21sys (55.29% cpu)
---------------
Read 12161 rpm names.
Use 9 procs w/1353 items/process
#pkgs=10161, #deletes=2000, total=12161
Recycling 2000 duplicates...Done
Cumulative This Phase ID
0.000s 0.000s Init
0.000s 0.000s start_program
0.032s 0.032s starting_children
0.036s 0.004s end_starting_children
1.535s 1.500s endRdFrmChldrn_n_start_re_sort
3.722s 2.187s afterFinalSort
10.36sec 3.31usr 4.47sys (75.09% cpu)
Cold Cache:
============
Read 12161 rpm names.
Use 1 procs w/12162 items/process
#pkgs=10161, #deletes=2000, total=12161
Recycling 2000 duplicates...Done
Cumulative This Phase ID
0.000s 0.000s Init
0.000s 0.000s start_program
0.095s 0.095s starting_children
0.096s 0.001s end_starting_children
75.067s 74.971s endRdFrmChldrn_n_start_re_sort
77.140s 2.073s afterFinalSort
84.52sec 3.62usr 6.26sys (11.70% cpu)
----
Read 12161 rpm names.
Use 9 procs w/1353 items/process
#pkgs=10161, #deletes=2000, total=12161
Recycling 2000 duplicates...Done
Cumulative This Phase ID
0.000s 0.000s Init
0.000s 0.000s start_program
0.107s 0.107s starting_children
0.112s 0.005s end_starting_children
29.350s 29.238s endRdFrmChldrn_n_start_re_sort
31.497s 2.147s afterFinalSort
38.27sec 3.35usr 4.47sys (20.47% cpu)
---
hot cache savings: 42%
cold cache savings: 55%
