On Mar 22, 2010, at 4:41 PM, <ananda.mu...@wipro.com> <ananda.mu...@wipro.com
> wrote:
Hi
If the run my compute intensive openmpi based program using regular
invocation of mpirun (ie; mpirun –host <hostnamelist> -np <number of
cores>), it gets completed in few seconds but if I run the same
program with “-am ft-enable-cr” option, the program takes 10x time
to complete.
If I enable hyperthreading on my cluster nodes and then call mpirun
with “-am ft-enable-cr” option, the program gets completed with few
additional seconds than the normal mpirun!!
How can I improve the performance of mpirun with “-am ft-enable-cr”
option when I disable hyperthreading on my cluster nodes? Any
pointers will be really useful.
FYI, I am using openmpi 1.3.4 library and BLCR 0.8.2. Cluster nodes
are Nehalem based nodes with 8 cores.
I have not done any performance studies focused on hyperthreading, so
I can not say specifically what is happening.
The 10x slowdown is certainly unexpected (I don't see this in my
testing). There usually is a small slowdown (few microseconds) because
of the message tracking technique used to support the checkpoint
coordination protocol. I suspect that the cause of your problem is the
C/R thread which is probably too aggressive for your system. The
improvement with hyperthreading may be that this thread is able to sit
on one of the hardware threads and not completely steal the CPU from
the main application.
You can change how aggressive the thread is by adjusting the two
parameters below:
http://osl.iu.edu/research/ft/ompi-cr/api.php#mca-opal_cr_thread_sleep_check
http://osl.iu.edu/research/ft/ompi-cr/api.php#mca-opal_cr_thread_sleep_wait
I usually set the latter to:
opal_cr_thread_sleep_wait=1000
Give that a try and let me know is that helps. You might also try to
upgrade to the 1.4 series, or even the upcoming v1.5.0 release and see
if the problem persists there.
-- Josh
Thanks
Anand
Please do not print this email unless it is absolutely necessary.
The information contained in this electronic message and any
attachments to this message are intended for the exclusive use of
the addressee(s) and may contain proprietary, confidential or
privileged information. If you are not the intended recipient, you
should not disseminate, distribute or copy this e-mail. Please
notify the sender immediately and destroy all copies of this message
and any attachments.
WARNING: Computer viruses can be transmitted via email. The
recipient should check this email and any attachments for the
presence of viruses. The company accepts no liability for any damage
caused by any virus transmitted by this email.
www.wipro.com
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
