Hello.
As I promised, I send you results about different simulations and parameters
according
to the MPI options :TEST | DESCRIPTION
SHARING MPI | WITH PBS | ELAPSE TIME 1ST ITERATION
1 Node 2
12 process yes no 0.21875E+03
2 Node 1
12 process yes no 0.21957E+03
3 Node 1, with 24 process to test multithreadin 24 process
yes no 0.20613E+03
4 Node 2
12 process yes yes 0.22130E+03
5 Node 2, with 24 process to test multithreadin 24
process yes no 0.27300E+03
6
7 Nodes 1, 2
2 x 6 process yes yes 0.17304E+03
8 Nodes 1, 2
2 x 11 process yes yes 0.12395E+03
9 Nodes 1, 2
2 x 12 process yes yes 0.11812E+03
10 Nodes 3, 4
2 x 12 process yes yes 0.11237E+03
11 Nodes 1,2,3 with 1 more process upon node 3 2 x 12 + 1
proces yes yes 0.56223E+03
12 Nodes 1,2,3;MPI options --bycore --bind-to-core 2 x 12 + 1 proces
yes yes 0.32452E+03
13 Nodes 1,4,3 with 1 more process upon node 3 2 x 12 + 1
proces yes yes 0.37252E+03
14 Nodes 1,4,3;MPI options --bysocket --bind-to-sock 2 x 12 + 1 proces
yes yes 0.56666E+03
15 Nodes 1,4,3;MPI options --bycore --bind-to-core 2 x 12 + 1 proces
yes yes 0.39983E+03
16 Nodes 2,3,4
3 x 12 process yes yes 0.85723E+03
17 Nodes 2,3,4
3 x 8 process yes yes 0.49378E+03
18 Nodes 1,2,3
3 x 8 process yes yes 0.51863E+03
19 Nodes 1,2,3,4
4 x 6 process yes yes 0.73272E+03
20
21 1,2,3,4; MPI options --bysocket --bind-to-socke 4 x 6 process
yes yes 0.67739E+03
22 1,2,3,4; MPI options --bycore --bind-to-core 4 x 6 process
yes yes 0.69612E+03
The more surprising, even by taking in account latency between the nodes, are
the tests
11 to 15. By adding only 1 process on the node 3, elapse time becomes 0.56e+03,
i.e.
5 times the case 9 and 10. When partitioning upon 25 processors : 1 node
represents 4%
of the simulation (I have verified each partitions : they contain
approximatively the sames number
of elements plus or minus 8%), even one takes in account a latency factor of
10, i.e 40% more,
one should obtain (for test 10) : 0.11e+03 x 1.40 ~= 0.154e+03 sec.
In addition, when I observe the data transfers upon the eth0 connexion during
an iteration, I see that
when node 1 and 2 transfer, for example 5 Mo, then node 3 transfers 2,5 Mo. But
if we consider that
node 3 is concerned by 4% of the data simulation, it should only need 200 Ko !
Results are very differents too, between options with binding socket or binding
core, as tests 13, 14 and 15 show.
Regards.
Albert
