Hi Graham,
here are the all-to-all test results with the modification to the decision
routine you suggested yesterday. Now the routine behaves nicely for 128
and 256 float messages on 128 CPUs! For the other sizes one probably wants
to keep the original algorithm, since it is faster there. However I have
the feeling that for messages >= 4096 floats there still exists the old
problem since the execution times are so variable there (note that the
standard deviation rises by more than a factor of 10 when going from 2048
to 4096 floats.) If you need additional test results to tune the decision
functions please let me know.
Carsten
OMPI tuned all-to-all with modification:
========================================
mesg size time in seconds
#CPUs floats average std.dev. min. max.
128 1 0.001253 0.000074 0.001141 0.001470
128 2 0.023507 0.000563 0.022562 0.024761
128 4 0.023435 0.000426 0.022582 0.024166
128 8 0.023438 0.000359 0.022904 0.024104
128 16 0.023664 0.000438 0.022844 0.024670
128 32 0.024136 0.000463 0.023297 0.025117
128 64 0.024704 0.000535 0.023727 0.026030
128 128 0.025750 0.000525 0.024592 0.026799 *
128 256 0.028862 0.000683 0.027389 0.030168 *
128 512 0.035869 0.001214 0.034067 0.038655
128 1024 0.046528 0.001722 0.043549 0.050432
128 2048 0.072388 0.007032 0.066708 0.104358
128 4096 0.217678 0.097312 0.135113 0.409431
128 8192 0.378586 0.090267 0.297878 0.577771
128 16384 0.567473 0.105083 0.483573 0.735509
128 32768 1.151343 0.146547 0.937150 1.404478
128 65536 2.298998 0.169669 1.983286 2.572027
128 131072 4.070989 0.159958 3.691039 4.373587
> > OMPI tuned all-to-all:
> > ======================
> > mesg size time in seconds
> > #CPUs floats average std.dev. min. max.
> > 128 1 0.001288 0.000102 0.001077 0.001512
> > 128 2 0.008391 0.000400 0.007861 0.009958
> > 128 4 0.008403 0.000237 0.008095 0.009018
> > 128 8 0.008228 0.000942 0.003801 0.008810
> > 128 16 0.008503 0.000191 0.008233 0.008839
> > 128 32 0.008656 0.000271 0.008084 0.009177
> > 128 64 0.009085 0.000209 0.008757 0.009603
> > 128 128 0.251414 0.073069 0.011547 0.506703 !
> > 128 256 0.385515 0.127661 0.251431 0.578955 !
> > 128 512 0.035111 0.000872 0.033358 0.036262
> > 128 1024 0.046028 0.002116 0.043381 0.052602
> > 128 2048 0.073392 0.007745 0.066432 0.104531
> > 128 4096 0.165052 0.072889 0.124589 0.404213
> > 128 8192 0.341377 0.041815 0.309457 0.530409
> > 128 16384 0.507200 0.050872 0.492307 0.750956
> > 128 32768 1.050291 0.132867 0.954496 1.344978
> > 128 65536 2.213977 0.154987 1.962907 2.492560
> > 128 131072 4.026107 0.147103 3.800191 4.336205
> >
> > alternative all-to-all:
> > ======================
> > 128 1 0.012584 0.000724 0.011073 0.015331
> > 128 2 0.012506 0.000444 0.011707 0.013461
> > 128 4 0.012412 0.000511 0.011157 0.013413
> > 128 8 0.012488 0.000455 0.011767 0.013746
> > 128 16 0.012664 0.000416 0.011745 0.013362
> > 128 32 0.012878 0.000410 0.012157 0.013609
> > 128 64 0.013138 0.000417 0.012452 0.013826
> > 128 128 0.014016 0.000505 0.013195 0.014942 +
> > 128 256 0.015843 0.000521 0.015107 0.016725 +
> > 128 512 0.052240 0.079323 0.027019 0.320653 !
> > 128 1024 0.123884 0.121560 0.038062 0.308929 !
> > 128 2048 0.176877 0.125229 0.074457 0.387276 !
> > 128 4096 0.305030 0.121716 0.176640 0.496375 !
> > 128 8192 0.546405 0.108007 0.415272 0.899858 !
> > 128 16384 0.604844 0.056576 0.558657 0.843943 !
> > 128 32768 1.235298 0.097969 1.094720 1.451241 !
> > 128 65536 2.926902 0.312733 2.458742 3.895563 !
> > 128 131072 6.208087 0.472115 5.354304 7.317153 !
---------------------------------------------------
Dr. Carsten Kutzner
Max Planck Institute for Biophysical Chemistry
Theoretical and Computational Biophysics Department
Am Fassberg 11
37077 Goettingen, Germany
Tel. +49-551-2012313, Fax: +49-551-2012302
eMail ckut...@gwdg.de
http://www.gwdg.de/~ckutzne
