I am trying to run an MPI code across 136 processing using an appfile (attached), since every process needs to be run with a host/process dependent parameter.
This whole system works wonderfully for up to around 100 processes but usually fails with a segfault, apparently in in mca_odls_default.so, during initialization. The attached appfile is an attempt at 136 processes. If I split the appfile into two, both halves will initialize OK and successfully pass an MPI_Barrier() (the program won't actually work without all 136 nodes, but I'm happy MPI is doing its job). Because both halves work, I think it has to be related to the number of processes - not a problem with a specific appfile entry or machine. The cluster I am running it on has openmpi-1.3.3 but I have also installed 1.4.1 from the website in my home dir and that does the same (and is from which the attached data comes). The actual segfault is: [jac-11:12300] *** Process received signal *** [jac-11:12300] Signal: Segmentation fault (11) [jac-11:12300] Signal code: Address not mapped (1) [jac-11:12300] Failing at address: 0x40 [jac-11:12300] [ 0] [0x74640c] [jac-11:12300] [ 1] /home/oford/openmpi/lib/openmpi/mca_odls_default.so [0x8863d4] [jac-11:12300] [ 2] /home/oford/openmpi/lib/libopen-rte.so.0 [0x76ffe9] [jac-11:12300] [ 3] /home/oford/openmpi/lib/libopen-rte.so.0(orte_daemon_cmd_processor+0x2f6) [0x771b86] [jac-11:12300] [ 4] /home/oford/openmpi/lib/libopen-pal.so.0 [0x5d6ba8] [jac-11:12300] [ 5] /home/oford/openmpi/lib/libopen-pal.so.0(opal_event_loop+0x27) [0x5d6e47] [jac-11:12300] [ 6] /home/oford/openmpi/lib/libopen-pal.so.0(opal_progress+0xce) [0x5ca00e] [jac-11:12300] [ 7] /home/oford/openmpi/lib/libopen-rte.so.0(orte_plm_base_launch_apps+0x355) [0x7815f5] [jac-11:12300] [ 8] /home/oford/openmpi/lib/openmpi/mca_plm_rsh.so [0xc73d1b] [jac-11:12300] [ 9] mpirun [0x804a8f0] [jac-11:12300] [10] mpirun [0x8049ef6] [jac-11:12300] [11] /lib/libc.so.6(__libc_start_main+0xe5) [0x1406e5] [jac-11:12300] [12] mpirun [0x8049e41] [jac-11:12300] *** End of error message *** Segmentation fault The full output with '-d' and the config.log from the build of 1.4.1 are also attached. I don't know the exact setup of the network, but I can ask our sysadmin anything else that might help. Thanks in advance, Oliver Ford Culham Centre for Fusion Energy Oxford, UK
-np 1 --host jac-11 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-39-80115 Y 11 11 133 debug -np 1 --host jac-5 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-26-81244 N 11 11 133 debug -np 1 --host batch-020 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-122-75993 N 11 11 133 debug -np 1 --host batch-037 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-157-15286 N 11 11 133 debug -np 1 --host batch-042 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-114-89529 N 11 11 133 debug -np 1 --host jac-9 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-35-90257 N 11 11 133 debug -np 1 --host batch-020 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-151-56062 N 11 11 133 debug -np 1 --host batch-004 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-16-2723 N 11 11 133 debug -np 1 --host batch-003 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-156-65790 N 11 11 133 debug -np 1 --host jac-11 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-198-63239 N 11 11 133 debug -np 1 --host batch-046 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-105-12753 N 11 11 133 debug -np 1 --host batch-015 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-12-25631 N 11 11 133 debug -np 1 --host jac-12 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-196-35421 N 11 11 133 debug -np 1 --host batch-045 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-103-98246 N 11 11 133 debug -np 1 --host batch-006 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-142-44009 N 11 11 133 debug -np 1 --host batch-044 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-117-30325 N 11 11 133 debug -np 1 --host batch-003 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-143-21739 N 11 11 133 debug -np 1 --host batch-042 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-112-64293 N 11 11 133 debug -np 1 --host batch-041 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-57-11238 N 11 11 133 debug -np 1 --host batch-025 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-170-80831 N 11 11 133 debug -np 1 --host jac-6 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-32-47578 N 11 11 133 debug -np 1 --host batch-027 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-146-27711 N 11 11 133 debug -np 1 --host jac-8 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-25-36312 N 11 11 133 debug -np 1 --host batch-023 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-154-17110 N 11 11 133 debug -np 1 --host batch-023 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-136-3872 N 11 11 133 debug -np 1 --host batch-036 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-132-1105 N 11 11 133 debug -np 1 --host jac-10 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-192-11991 N 11 11 133 debug -np 1 --host batch-045 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-115-41110 N 11 11 133 debug -np 1 --host batch-036 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-155-67802 N 11 11 133 debug -np 1 --host jac-2 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-40-46953 N 11 11 133 debug -np 1 --host batch-006 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-125-79465 N 11 11 133 debug -np 1 --host batch-022 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-128-79309 N 11 11 133 debug -np 1 --host batch-033 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-152-82320 N 11 11 133 debug -np 1 --host batch-026 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-158-68939 N 11 11 133 debug -np 1 --host batch-020 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-137-41369 N 11 11 133 debug -np 1 --host jac-5 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-2-66663 N 11 11 133 debug -np 1 --host batch-046 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-52-62955 N 11 11 133 debug -np 1 --host jac-13 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-195-10991 N 11 11 133 debug -np 1 --host jac-9 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-193-50774 N 11 11 133 debug -np 1 --host jac-3 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-177-45516 N 11 11 133 debug -np 1 --host batch-032 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-171-96942 N 11 11 133 debug -np 1 --host batch-038 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-175-99761 N 11 11 133 debug -np 1 --host batch-006 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-118-35065 N 11 11 133 debug -np 1 --host batch-014 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-17-49750 N 11 11 133 debug -np 1 --host batch-030 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-169-10604 N 11 11 133 debug -np 1 --host jac-2 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-46-74041 N 11 11 133 debug -np 1 --host jac-13 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-187-7833 N 11 11 133 debug -np 1 --host jac-7 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-200-36312 N 11 11 133 debug -np 1 --host batch-027 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-159-37278 N 11 11 133 debug -np 1 --host batch-045 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-11-37181 N 11 11 133 debug -np 1 --host batch-013 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-149-11377 N 11 11 133 debug -np 1 --host jac-3 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-179-64715 N 11 11 133 debug -np 1 --host jac-2 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-44-46953 N 11 11 133 debug -np 1 --host batch-017 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-130-64980 N 11 11 133 debug -np 1 --host batch-015 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-13-39413 N 11 11 133 debug -np 1 --host jac-8 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-34-46573 N 11 11 133 debug -np 1 --host batch-027 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-134-85354 N 11 11 133 debug -np 1 --host jac-7 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-194-95760 N 11 11 133 debug -np 1 --host batch-013 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-133-18010 N 11 11 133 debug -np 1 --host batch-024 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-173-20377 N 11 11 133 debug -np 1 --host batch-013 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-160-43487 N 11 11 133 debug -np 1 --host batch-046 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-106-65988 N 11 11 133 debug -np 1 --host jac-3 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-18-18834 N 11 11 133 debug -np 1 --host jac-3 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-180-19539 N 11 11 133 debug -np 1 --host batch-022 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-140-53706 N 11 11 133 debug -np 1 --host jac-7 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-184-80559 N 11 11 133 debug -np 1 --host batch-023 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-126-6794 N 11 11 133 debug -np 1 --host batch-044 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-111-83886 N 11 11 133 debug -np 1 --host batch-005 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-150-26152 N 11 11 133 debug -np 1 --host jac-9 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-29-81244 N 11 11 133 debug -np 1 --host jac-10 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-30-5208 N 11 11 133 debug -np 1 --host jac-6 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-185-82710 N 11 11 133 debug -np 1 --host batch-022 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-153-49288 N 11 11 133 debug -np 1 --host batch-026 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-131-48260 N 11 11 133 debug -np 1 --host jac-10 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-21-85978 N 11 11 133 debug -np 1 --host jac-12 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-186-67937 N 11 11 133 debug -np 1 --host batch-033 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-135-94983 N 11 11 133 debug -np 1 --host batch-044 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-107-10462 N 11 11 133 debug -np 1 --host batch-046 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-113-29689 N 11 11 133 debug -np 1 --host jac-4 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-27-60637 N 11 11 133 debug -np 1 --host jac-2 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-36-26250 N 11 11 133 debug -np 1 --host batch-017 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-119-1554 N 11 11 133 debug -np 1 --host jac-6 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-24-17859 N 11 11 133 debug -np 1 --host jac-12 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-31-5480 N 11 11 133 debug -np 1 --host jac-13 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-23-24746 N 11 11 133 debug -np 1 --host batch-005 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-138-53784 N 11 11 133 debug -np 1 --host jac-7 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-33-43810 N 11 11 133 debug -np 1 --host batch-005 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-121-38059 N 11 11 133 debug -np 1 --host batch-036 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-145-79515 N 11 11 133 debug -np 1 --host batch-042 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-109-66314 N 11 11 133 debug -np 1 --host batch-028 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-168-62760 N 11 11 133 debug -np 1 --host batch-001 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-166-35091 N 11 11 133 debug -np 1 --host jac-13 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-3-30335 N 11 11 133 debug -np 1 --host batch-043 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-108-43856 N 11 11 133 debug -np 1 --host batch-041 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-116-98652 N 11 11 133 debug -np 1 --host jac-8 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-189-70592 N 11 11 133 debug -np 1 --host batch-004 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-148-12571 N 11 11 133 debug -np 1 --host jac-4 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-38-79978 N 11 11 133 debug -np 1 --host batch-003 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-129-21483 N 11 11 133 debug -np 1 --host batch-033 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-124-3865 N 11 11 133 debug -np 1 --host jac-11 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-191-79048 N 11 11 133 debug -np 1 --host jac-4 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-188-93446 N 11 11 133 debug -np 1 --host jac-10 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-37-71686 N 11 11 133 debug -np 1 --host batch-037 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-162-83683 N 11 11 133 debug -np 1 --host batch-043 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-1-48706 N 11 11 133 debug -np 1 --host jac-8 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-199-95838 N 11 11 133 debug -np 1 --host jac-3 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-178-23994 N 11 11 133 debug -np 1 --host batch-026 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-144-79515 N 11 11 133 debug -np 1 --host batch-008 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-15-46689 N 11 11 133 debug -np 1 --host jac-4 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-197-46966 N 11 11 133 debug -np 1 --host jac-2 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-41-82354 N 11 11 133 debug -np 1 --host jac-9 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-20-1112 N 11 11 133 debug -np 1 --host batch-039 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-176-80381 N 11 11 133 debug -np 1 --host batch-041 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-101-91749 N 11 11 133 debug -np 1 --host batch-043 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-10-96608 N 11 11 133 debug -np 1 --host batch-017 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-139-62526 N 11 11 133 debug -np 1 --host batch-007 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-164-93486 N 11 11 133 debug -np 1 --host batch-018 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-174-37716 N 11 11 133 debug -np 1 --host batch-041 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-104-25353 N 11 11 133 debug -np 1 --host batch-031 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-172-7131 N 11 11 133 debug -np 1 --host batch-006 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-127-18397 N 11 11 133 debug -np 1 --host batch-034 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-147-65530 N 11 11 133 debug -np 1 --host batch-015 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-14-40975 N 11 11 133 debug -np 1 --host batch-017 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-123-45216 N 11 11 133 debug -np 1 --host jac-6 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-190-27454 N 11 11 133 debug -np 1 --host batch-007 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-141-10541 N 11 11 133 debug -np 1 --host batch-043 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-102-77190 N 11 11 133 debug -np 1 --host batch-015 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-120-13012 N 11 11 133 debug -np 1 --host jac-5 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-19-29582 N 11 11 133 debug -np 1 --host batch-008 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-161-20135 N 11 11 133 debug -np 1 --host batch-040 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-163-95832 N 11 11 133 debug -np 1 --host batch-019 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-167-60329 N 11 11 133 debug -np 1 --host jac-11 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-28-77824 N 11 11 133 debug -np 1 --host batch-034 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-165-57829 N 11 11 133 debug -np 1 --host jac-5 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-4-1153 N 11 11 133 debug -np 1 --host jac-12 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-22-4961 N 11 11 133 debug
[jac-11:15014] procdir: /tmp/openmpi-sessions-oford@jac-11_0/48099/0/0 [jac-11:15014] jobdir: /tmp/openmpi-sessions-oford@jac-11_0/48099/0 [jac-11:15014] top: openmpi-sessions-oford@jac-11_0 [jac-11:15014] tmp: /tmp [jac-11:15014] mpirun: reset PATH: /home/oford/openmpi/bin:/home/oford/openmpi/bin:/home/oford/bin:/usr/lib/qt-3.3/bin:/usr/bin:/bin:/usr/local/depot/PGI/pgi-7.0/linux86/7.0/bin:/usr/local/bin:/jet/share/root/bin:/jet/share/bin:.:/u/sim/sh:/u/sim/cmg/sh:/home/oford/sh:/u/sim/sh/linux:/home/oford/sh/linux:/u/sim/edge2d/sh:/u/sim/edge2d/sh/linux:/home/oford/edge2d/sh:/home/oford/edge2d/sh/linux:/u/sim/jetto/sh [jac-11:15014] mpirun: reset LD_LIBRARY_PATH: /home/oford/openmpi/lib:/home/oford/openmpi/lib:/u/sim/mdsplus/default/lib/linux:/u/sim/jproc/default/lib/linux:/u/sim/eproc/default/lib/linux/fixed:/home/oford/oldXlib:/home/oford/hdf5:/jet/share/lib:/usr/local/lib [batch-020:17451] procdir: /tmp/openmpi-sessions-oford@batch-020_0/48099/0/2 [batch-020:17451] jobdir: /tmp/openmpi-sessions-oford@batch-020_0/48099/0 [batch-020:17451] top: openmpi-sessions-oford@batch-020_0 [batch-020:17451] tmp: /tmp [batch-042:15862] procdir: /tmp/openmpi-sessions-oford@batch-042_0/48099/0/4 [batch-042:15862] jobdir: /tmp/openmpi-sessions-oford@batch-042_0/48099/0 [batch-042:15862] top: openmpi-sessions-oford@batch-042_0 [batch-042:15862] tmp: /tmp [batch-023:09760] procdir: /tmp/openmpi-sessions-oford@batch-023_0/48099/0/19 [batch-023:09760] jobdir: /tmp/openmpi-sessions-oford@batch-023_0/48099/0 [batch-023:09760] top: openmpi-sessions-oford@batch-023_0 [batch-023:09760] tmp: /tmp [batch-025:05770] procdir: /tmp/openmpi-sessions-oford@batch-025_0/48099/0/15 [batch-025:05770] jobdir: /tmp/openmpi-sessions-oford@batch-025_0/48099/0 [batch-025:05770] top: openmpi-sessions-oford@batch-025_0 [batch-025:05770] tmp: /tmp [batch-037:17611] procdir: /tmp/openmpi-sessions-oford@batch-037_0/48099/0/3 [batch-037:17611] jobdir: /tmp/openmpi-sessions-oford@batch-037_0/48099/0 [batch-037:17611] top: openmpi-sessions-oford@batch-037_0 [batch-037:17611] tmp: /tmp [jac-5:15183] procdir: /tmp/openmpi-sessions-oford@jac-5_0/48099/0/1 [jac-5:15183] jobdir: /tmp/openmpi-sessions-oford@jac-5_0/48099/0 [jac-5:15183] top: openmpi-sessions-oford@jac-5_0 [jac-5:15183] tmp: /tmp [jac-9:22204] procdir: /tmp/openmpi-sessions-oford@jac-9_0/48099/0/5 [jac-9:22204] jobdir: /tmp/openmpi-sessions-oford@jac-9_0/48099/0 [jac-9:22204] top: openmpi-sessions-oford@jac-9_0 [jac-9:22204] tmp: /tmp [batch-013:23865] procdir: /tmp/openmpi-sessions-oford@batch-013_0/48099/0/33 [batch-013:23865] jobdir: /tmp/openmpi-sessions-oford@batch-013_0/48099/0 [batch-013:23865] top: openmpi-sessions-oford@batch-013_0 [batch-013:23865] tmp: /tmp [batch-022:15225] procdir: /tmp/openmpi-sessions-oford@batch-022_0/48099/0/23 [batch-022:15225] jobdir: /tmp/openmpi-sessions-oford@batch-022_0/48099/0 [batch-022:15225] top: openmpi-sessions-oford@batch-022_0 [batch-022:15225] tmp: /tmp [batch-018:19925] procdir: /tmp/openmpi-sessions-oford@batch-018_0/48099/0/44 [batch-018:19925] jobdir: /tmp/openmpi-sessions-oford@batch-018_0/48099/0 [batch-018:19925] top: openmpi-sessions-oford@batch-018_0 [batch-018:19925] tmp: /tmp [jac-4:06627] procdir: /tmp/openmpi-sessions-oford@jac-4_0/48099/0/37 [jac-4:06627] jobdir: /tmp/openmpi-sessions-oford@jac-4_0/48099/0 [jac-4:06627] top: openmpi-sessions-oford@jac-4_0 [jac-4:06627] tmp: /tmp [batch-004:30088] procdir: /tmp/openmpi-sessions-oford@batch-004_0/48099/0/6 [batch-004:30088] jobdir: /tmp/openmpi-sessions-oford@batch-004_0/48099/0 [batch-004:30088] top: openmpi-sessions-oford@batch-004_0 [batch-004:30088] tmp: /tmp [batch-046:00820] procdir: /tmp/openmpi-sessions-oford@batch-046_0/48099/0/8 [batch-046:00820] jobdir: /tmp/openmpi-sessions-oford@batch-046_0/48099/0 [batch-046:00820] top: openmpi-sessions-oford@batch-046_0 [batch-046:00820] tmp: /tmp [batch-036:16988] procdir: /tmp/openmpi-sessions-oford@batch-036_0/48099/0/20 [batch-036:16988] jobdir: /tmp/openmpi-sessions-oford@batch-036_0/48099/0 [batch-036:16988] top: openmpi-sessions-oford@batch-036_0 [batch-036:16988] tmp: /tmp [batch-030:25412] procdir: /tmp/openmpi-sessions-oford@batch-030_0/48099/0/31 [batch-030:25412] jobdir: /tmp/openmpi-sessions-oford@batch-030_0/48099/0 [batch-030:25412] top: openmpi-sessions-oford@batch-030_0 [batch-030:25412] tmp: /tmp [jac-6:04101] procdir: /tmp/openmpi-sessions-oford@jac-6_0/48099/0/16 [jac-6:04101] jobdir: /tmp/openmpi-sessions-oford@jac-6_0/48099/0 [jac-6:04101] top: openmpi-sessions-oford@jac-6_0 [jac-6:04101] tmp: /tmp [batch-006:16136] procdir: /tmp/openmpi-sessions-oford@batch-006_0/48099/0/12 [batch-006:16136] jobdir: /tmp/openmpi-sessions-oford@batch-006_0/48099/0 [batch-006:16136] top: openmpi-sessions-oford@batch-006_0 [batch-006:16136] tmp: /tmp [jac-7:17379] procdir: /tmp/openmpi-sessions-oford@jac-7_0/48099/0/32 [jac-7:17379] jobdir: /tmp/openmpi-sessions-oford@jac-7_0/48099/0 [jac-7:17379] top: openmpi-sessions-oford@jac-7_0 [jac-7:17379] tmp: /tmp [batch-003:23921] procdir: /tmp/openmpi-sessions-oford@batch-003_0/48099/0/7 [batch-003:23921] jobdir: /tmp/openmpi-sessions-oford@batch-003_0/48099/0 [batch-003:23921] top: openmpi-sessions-oford@batch-003_0 [batch-003:23921] tmp: /tmp [batch-044:01101] procdir: /tmp/openmpi-sessions-oford@batch-044_0/48099/0/13 [batch-044:01101] jobdir: /tmp/openmpi-sessions-oford@batch-044_0/48099/0 [batch-044:01101] top: openmpi-sessions-oford@batch-044_0 [batch-044:01101] tmp: /tmp [batch-039:02035] procdir: /tmp/openmpi-sessions-oford@batch-039_0/48099/0/42 [batch-039:02035] jobdir: /tmp/openmpi-sessions-oford@batch-039_0/48099/0 [batch-039:02035] top: openmpi-sessions-oford@batch-039_0 [batch-039:02035] tmp: /tmp [batch-033:02863] procdir: /tmp/openmpi-sessions-oford@batch-033_0/48099/0/24 [batch-033:02863] jobdir: /tmp/openmpi-sessions-oford@batch-033_0/48099/0 [batch-033:02863] top: openmpi-sessions-oford@batch-033_0 [batch-033:02863] tmp: /tmp [jac-8:26567] procdir: /tmp/openmpi-sessions-oford@jac-8_0/48099/0/18 [jac-8:26567] jobdir: /tmp/openmpi-sessions-oford@jac-8_0/48099/0 [jac-8:26567] top: openmpi-sessions-oford@jac-8_0 [jac-8:26567] tmp: /tmp [batch-028:14068] procdir: /tmp/openmpi-sessions-oford@batch-028_0/48099/0/38 [batch-028:14068] jobdir: /tmp/openmpi-sessions-oford@batch-028_0/48099/0 [batch-028:14068] top: openmpi-sessions-oford@batch-028_0 [batch-028:14068] tmp: /tmp [batch-019:32641] procdir: /tmp/openmpi-sessions-oford@batch-019_0/48099/0/48 [batch-019:32641] jobdir: /tmp/openmpi-sessions-oford@batch-019_0/48099/0 [batch-019:32641] top: openmpi-sessions-oford@batch-019_0 [batch-019:32641] tmp: /tmp [batch-038:07716] procdir: /tmp/openmpi-sessions-oford@batch-038_0/48099/0/29 [batch-038:07716] jobdir: /tmp/openmpi-sessions-oford@batch-038_0/48099/0 [batch-038:07716] top: openmpi-sessions-oford@batch-038_0 [batch-038:07716] tmp: /tmp [jac-2:05183] procdir: /tmp/openmpi-sessions-oford@jac-2_0/48099/0/22 [jac-2:05183] jobdir: /tmp/openmpi-sessions-oford@jac-2_0/48099/0 [jac-2:05183] top: openmpi-sessions-oford@jac-2_0 [jac-2:05183] tmp: /tmp [batch-017:25718] procdir: /tmp/openmpi-sessions-oford@batch-017_0/48099/0/34 [batch-017:25718] jobdir: /tmp/openmpi-sessions-oford@batch-017_0/48099/0 [batch-017:25718] top: openmpi-sessions-oford@batch-017_0 [batch-017:25718] tmp: /tmp [batch-026:09510] procdir: /tmp/openmpi-sessions-oford@batch-026_0/48099/0/25 [batch-026:09510] jobdir: /tmp/openmpi-sessions-oford@batch-026_0/48099/0 [batch-026:09510] top: openmpi-sessions-oford@batch-026_0 [batch-026:09510] tmp: /tmp [batch-043:01668] procdir: /tmp/openmpi-sessions-oford@batch-043_0/48099/0/40 [batch-043:01668] jobdir: /tmp/openmpi-sessions-oford@batch-043_0/48099/0 [batch-043:01668] top: openmpi-sessions-oford@batch-043_0 [batch-043:01668] tmp: /tmp [batch-008:17836] procdir: /tmp/openmpi-sessions-oford@batch-008_0/48099/0/41 [batch-008:17836] jobdir: /tmp/openmpi-sessions-oford@batch-008_0/48099/0 [batch-008:17836] top: openmpi-sessions-oford@batch-008_0 [batch-008:17836] tmp: /tmp [jac-12:17562] procdir: /tmp/openmpi-sessions-oford@jac-12_0/48099/0/10 [jac-12:17562] jobdir: /tmp/openmpi-sessions-oford@jac-12_0/48099/0 [jac-12:17562] top: openmpi-sessions-oford@jac-12_0 [jac-12:17562] tmp: /tmp [batch-005:03532] procdir: /tmp/openmpi-sessions-oford@batch-005_0/48099/0/36 [batch-005:03532] jobdir: /tmp/openmpi-sessions-oford@batch-005_0/48099/0 [batch-005:03532] top: openmpi-sessions-oford@batch-005_0 [batch-005:03532] tmp: /tmp [batch-045:04261] procdir: /tmp/openmpi-sessions-oford@batch-045_0/48099/0/11 [batch-045:04261] jobdir: /tmp/openmpi-sessions-oford@batch-045_0/48099/0 [batch-045:04261] top: openmpi-sessions-oford@batch-045_0 [batch-045:04261] tmp: /tmp [batch-001:22016] procdir: /tmp/openmpi-sessions-oford@batch-001_0/48099/0/39 [batch-001:22016] jobdir: /tmp/openmpi-sessions-oford@batch-001_0/48099/0 [batch-001:22016] top: openmpi-sessions-oford@batch-001_0 [batch-001:22016] tmp: /tmp [batch-040:08361] procdir: /tmp/openmpi-sessions-oford@batch-040_0/48099/0/47 [batch-040:08361] jobdir: /tmp/openmpi-sessions-oford@batch-040_0/48099/0 [batch-040:08361] top: openmpi-sessions-oford@batch-040_0 [batch-040:08361] tmp: /tmp [batch-007:05689] procdir: /tmp/openmpi-sessions-oford@batch-007_0/48099/0/43 [batch-007:05689] jobdir: /tmp/openmpi-sessions-oford@batch-007_0/48099/0 [batch-007:05689] top: openmpi-sessions-oford@batch-007_0 [batch-007:05689] tmp: /tmp [batch-032:02059] procdir: /tmp/openmpi-sessions-oford@batch-032_0/48099/0/28 [batch-032:02059] jobdir: /tmp/openmpi-sessions-oford@batch-032_0/48099/0 [batch-032:02059] top: openmpi-sessions-oford@batch-032_0 [batch-032:02059] tmp: /tmp [batch-031:01049] procdir: /tmp/openmpi-sessions-oford@batch-031_0/48099/0/45 [batch-031:01049] jobdir: /tmp/openmpi-sessions-oford@batch-031_0/48099/0 [batch-031:01049] top: openmpi-sessions-oford@batch-031_0 [batch-031:01049] tmp: /tmp [batch-024:05301] procdir: /tmp/openmpi-sessions-oford@batch-024_0/48099/0/35 [batch-024:05301] jobdir: /tmp/openmpi-sessions-oford@batch-024_0/48099/0 [batch-024:05301] top: openmpi-sessions-oford@batch-024_0 [batch-024:05301] tmp: /tmp [batch-034:24766] procdir: /tmp/openmpi-sessions-oford@batch-034_0/48099/0/46 [batch-034:24766] jobdir: /tmp/openmpi-sessions-oford@batch-034_0/48099/0 [batch-034:24766] top: openmpi-sessions-oford@batch-034_0 [batch-034:24766] tmp: /tmp [jac-10:27707] procdir: /tmp/openmpi-sessions-oford@jac-10_0/48099/0/21 [jac-10:27707] jobdir: /tmp/openmpi-sessions-oford@jac-10_0/48099/0 [jac-10:27707] top: openmpi-sessions-oford@jac-10_0 [jac-10:27707] tmp: /tmp [batch-014:30991] procdir: /tmp/openmpi-sessions-oford@batch-014_0/48099/0/30 [batch-014:30991] jobdir: /tmp/openmpi-sessions-oford@batch-014_0/48099/0 [batch-014:30991] top: openmpi-sessions-oford@batch-014_0 [batch-014:30991] tmp: /tmp [jac-3:20114] procdir: /tmp/openmpi-sessions-oford@jac-3_0/48099/0/27 [jac-3:20114] jobdir: /tmp/openmpi-sessions-oford@jac-3_0/48099/0 [jac-3:20114] top: openmpi-sessions-oford@jac-3_0 [jac-3:20114] tmp: /tmp [batch-015:01829] procdir: /tmp/openmpi-sessions-oford@batch-015_0/48099/0/9 [batch-015:01829] jobdir: /tmp/openmpi-sessions-oford@batch-015_0/48099/0 [batch-015:01829] top: openmpi-sessions-oford@batch-015_0 [batch-015:01829] tmp: /tmp [batch-041:17727] procdir: /tmp/openmpi-sessions-oford@batch-041_0/48099/0/14 [batch-041:17727] jobdir: /tmp/openmpi-sessions-oford@batch-041_0/48099/0 [batch-041:17727] top: openmpi-sessions-oford@batch-041_0 [batch-041:17727] tmp: /tmp [jac-13:26153] procdir: /tmp/openmpi-sessions-oford@jac-13_0/48099/0/26 [jac-13:26153] jobdir: /tmp/openmpi-sessions-oford@jac-13_0/48099/0 [jac-13:26153] top: openmpi-sessions-oford@jac-13_0 [jac-13:26153] tmp: /tmp [batch-027:23746] procdir: /tmp/openmpi-sessions-oford@batch-027_0/48099/0/17 [batch-027:23746] jobdir: /tmp/openmpi-sessions-oford@batch-027_0/48099/0 [batch-027:23746] top: openmpi-sessions-oford@batch-027_0 [batch-027:23746] tmp: /tmp [jac-11:15014] [[48099,0],0] node[0].name jac-11 daemon 0 arch ffca0200 [jac-11:15014] [[48099,0],0] node[1].name jac-5 daemon 1 arch ffca0200 [jac-11:15014] [[48099,0],0] node[2].name batch-020 daemon 2 arch ffca0200 [jac-11:15014] [[48099,0],0] node[3].name batch-037 daemon 3 arch ffca0200 [jac-11:15014] [[48099,0],0] node[4].name batch-042 daemon 4 arch ffca0200 [jac-11:15014] [[48099,0],0] node[5].name jac-9 daemon 5 arch ffca0200 [jac-11:15014] [[48099,0],0] node[6].name batch-004 daemon 6 arch ffca0200 [jac-11:15014] [[48099,0],0] node[7].name batch-003 daemon 7 arch ffca0200 [jac-11:15014] [[48099,0],0] node[8].name batch-046 daemon 8 arch ffca0200 [jac-11:15014] [[48099,0],0] node[9].name batch-015 daemon 9 arch ffca0200 [jac-11:15014] [[48099,0],0] node[10].name jac-12 daemon 10 arch ffca0200 [jac-11:15014] [[48099,0],0] node[11].name batch-045 daemon 11 arch ffca0200 [jac-11:15014] [[48099,0],0] node[12].name batch-006 daemon 12 arch ffca0200 [jac-11:15014] [[48099,0],0] node[13].name batch-044 daemon 13 arch ffca0200 [jac-11:15014] [[48099,0],0] node[14].name batch-041 daemon 14 arch ffca0200 [jac-11:15014] [[48099,0],0] node[15].name batch-025 daemon 15 arch ffca0200 [jac-11:15014] [[48099,0],0] node[16].name jac-6 daemon 16 arch ffca0200 [jac-11:15014] [[48099,0],0] node[17].name batch-027 daemon 17 arch ffca0200 [jac-11:15014] [[48099,0],0] node[18].name jac-8 daemon 18 arch ffca0200 [jac-11:15014] [[48099,0],0] node[19].name batch-023 daemon 19 arch ffca0200 [jac-11:15014] [[48099,0],0] node[20].name batch-036 daemon 20 arch ffca0200 [jac-11:15014] [[48099,0],0] node[21].name jac-10 daemon 21 arch ffca0200 [jac-11:15014] [[48099,0],0] node[22].name jac-2 daemon 22 arch ffca0200 [jac-11:15014] [[48099,0],0] node[23].name batch-022 daemon 23 arch ffca0200 [jac-11:15014] [[48099,0],0] node[24].name batch-033 daemon 24 arch ffca0200 [jac-11:15014] [[48099,0],0] node[25].name batch-026 daemon 25 arch ffca0200 [jac-11:15014] [[48099,0],0] node[26].name jac-13 daemon 26 arch ffca0200 [jac-11:15014] [[48099,0],0] node[27].name jac-3 daemon 27 arch ffca0200 [jac-11:15014] [[48099,0],0] node[28].name batch-032 daemon 28 arch ffca0200 [jac-11:15014] [[48099,0],0] node[29].name batch-038 daemon 29 arch ffca0200 [jac-11:15014] [[48099,0],0] node[30].name batch-014 daemon 30 arch ffca0200 [jac-11:15014] [[48099,0],0] node[31].name batch-030 daemon 31 arch ffca0200 [jac-11:15014] [[48099,0],0] node[32].name jac-7 daemon 32 arch ffca0200 [jac-11:15014] [[48099,0],0] node[33].name batch-013 daemon 33 arch ffca0200 [jac-11:15014] [[48099,0],0] node[34].name batch-017 daemon 34 arch ffca0200 [jac-11:15014] [[48099,0],0] node[35].name batch-024 daemon 35 arch ffca0200 [jac-11:15014] [[48099,0],0] node[36].name batch-005 daemon 36 arch ffca0200 [jac-11:15014] [[48099,0],0] node[37].name jac-4 daemon 37 arch ffca0200 [jac-11:15014] [[48099,0],0] node[38].name batch-028 daemon 38 arch ffca0200 [jac-11:15014] [[48099,0],0] node[39].name batch-001 daemon 39 arch ffca0200 [jac-11:15014] [[48099,0],0] node[40].name batch-043 daemon 40 arch ffca0200 [jac-11:15014] [[48099,0],0] node[41].name batch-008 daemon 41 arch ffca0200 [jac-11:15014] [[48099,0],0] node[42].name batch-039 daemon 42 arch ffca0200 [jac-11:15014] [[48099,0],0] node[43].name batch-007 daemon 43 arch ffca0200 [jac-11:15014] [[48099,0],0] node[44].name batch-018 daemon 44 arch ffca0200 [jac-11:15014] [[48099,0],0] node[45].name batch-031 daemon 45 arch ffca0200 [jac-11:15014] [[48099,0],0] node[46].name batch-034 daemon 46 arch ffca0200 [jac-11:15014] [[48099,0],0] node[47].name batch-040 daemon 47 arch ffca0200 [jac-11:15014] [[48099,0],0] node[48].name batch-019 daemon 48 arch ffca0200 [jac-11:15014] *** Process received signal *** [jac-11:15014] Signal: Segmentation fault (11) [jac-11:15014] Signal code: Address not mapped (1) [jac-11:15014] Failing at address: 0x302c5d70 [jac-11:15014] [ 0] [0x9f540c] [jac-11:15014] [ 1] /home/oford/openmpi/lib/openmpi/mca_odls_default.so [0x52c3d4] [jac-11:15014] [ 2] /home/oford/openmpi/lib/libopen-rte.so.0 [0x22afe9] [jac-11:15014] [ 3] /home/oford/openmpi/lib/libopen-rte.so.0(orte_daemon_cmd_processor+0x2f6) [0x22cb86] [jac-11:15014] [ 4] /home/oford/openmpi/lib/libopen-pal.so.0 [0x129ba8] [jac-11:15014] [ 5] /home/oford/openmpi/lib/libopen-pal.so.0(opal_event_loop+0x27) [0x129e47] [jac-11:15014] [ 6] /home/oford/openmpi/lib/libopen-pal.so.0(opal_progress+0xce) [0x11d00e] [jac-11:15014] [ 7] /home/oford/openmpi/lib/libopen-rte.so.0(orte_plm_base_launch_apps+0x355) [0x23c5f5] [jac-11:15014] [ 8] /home/oford/openmpi/lib/openmpi/mca_plm_rsh.so [0x391d1b] [jac-11:15014] [ 9] mpirun [0x804a8f0] [jac-11:15014] [10] mpirun [0x8049ef6] [jac-11:15014] [11] /lib/libc.so.6(__libc_start_main+0xe5) [0x5b36e5] [jac-11:15014] [12] mpirun [0x8049e41] [jac-11:15014] *** End of error message *** [batch-041:17727] [[48099,0],14] routed:binomial: Connection to lifeline [[48099,0],0] lo[jac-6:04101] sess_dir_finalize: proc session dir not empty - leaving eaving batch-034:24766] sess_dir_finalize: proc session dir not empty - leaving [jac-12:17562] sess_dir_finalize: proc session dir not empty - leaving ng
Package: Open MPI oford@jac-12 Distribution
Open MPI: 1.4.1
Open MPI SVN revision: r22421
Open MPI release date: Jan 14, 2010
Open RTE: 1.4.1
Open RTE SVN revision: r22421
Open RTE release date: Jan 14, 2010
OPAL: 1.4.1
OPAL SVN revision: r22421
OPAL release date: Jan 14, 2010
Ident string: 1.4.1
Prefix: /home/oford/openmpi
Configured architecture: i686-pc-linux-gnu
Configure host: jac-12
Configured by: oford
Configured on: Thu Feb 25 22:46:23 GMT 2010
Configure host: jac-12
Built by: oford
Built on: Thu Feb 25 23:03:10 GMT 2010
Built host: jac-2
C bindings: yes
C++ bindings: yes
Fortran77 bindings: yes (all)
Fortran90 bindings: yes
Fortran90 bindings size: small
C compiler: gcc
C compiler absolute: /usr/bin/gcc
C++ compiler: g++
C++ compiler absolute: /usr/bin/g++
Fortran77 compiler: gfortran
Fortran77 compiler abs: /usr/bin/gfortran
Fortran90 compiler: gfortran
Fortran90 compiler abs: /usr/bin/gfortran
C profiling: yes
C++ profiling: yes
Fortran77 profiling: yes
Fortran90 profiling: yes
C++ exceptions: no
Thread support: posix (mpi: no, progress: no)
Sparse Groups: no
Internal debug support: no
MPI parameter check: runtime
Memory profiling support: no
Memory debugging support: no
libltdl support: yes
Heterogeneous support: no
mpirun default --prefix: yes
MPI I/O support: yes
MPI_WTIME support: gettimeofday
Symbol visibility support: yes
FT Checkpoint support: no (checkpoint thread: no)
MCA backtrace: execinfo (MCA v2.0, API v2.0, Component v1.4.1)
MCA memory: ptmalloc2 (MCA v2.0, API v2.0, Component v1.4.1)
MCA paffinity: linux (MCA v2.0, API v2.0, Component v1.4.1)
MCA carto: auto_detect (MCA v2.0, API v2.0, Component v1.4.1)
MCA carto: file (MCA v2.0, API v2.0, Component v1.4.1)
MCA maffinity: first_use (MCA v2.0, API v2.0, Component v1.4.1)
MCA timer: linux (MCA v2.0, API v2.0, Component v1.4.1)
MCA installdirs: env (MCA v2.0, API v2.0, Component v1.4.1)
MCA installdirs: config (MCA v2.0, API v2.0, Component v1.4.1)
MCA dpm: orte (MCA v2.0, API v2.0, Component v1.4.1)
MCA pubsub: orte (MCA v2.0, API v2.0, Component v1.4.1)
MCA allocator: basic (MCA v2.0, API v2.0, Component v1.4.1)
MCA allocator: bucket (MCA v2.0, API v2.0, Component v1.4.1)
MCA coll: basic (MCA v2.0, API v2.0, Component v1.4.1)
MCA coll: hierarch (MCA v2.0, API v2.0, Component v1.4.1)
MCA coll: inter (MCA v2.0, API v2.0, Component v1.4.1)
MCA coll: self (MCA v2.0, API v2.0, Component v1.4.1)
MCA coll: sm (MCA v2.0, API v2.0, Component v1.4.1)
MCA coll: sync (MCA v2.0, API v2.0, Component v1.4.1)
MCA coll: tuned (MCA v2.0, API v2.0, Component v1.4.1)
MCA io: romio (MCA v2.0, API v2.0, Component v1.4.1)
MCA mpool: fake (MCA v2.0, API v2.0, Component v1.4.1)
MCA mpool: rdma (MCA v2.0, API v2.0, Component v1.4.1)
MCA mpool: sm (MCA v2.0, API v2.0, Component v1.4.1)
MCA pml: cm (MCA v2.0, API v2.0, Component v1.4.1)
MCA pml: csum (MCA v2.0, API v2.0, Component v1.4.1)
MCA pml: ob1 (MCA v2.0, API v2.0, Component v1.4.1)
MCA pml: v (MCA v2.0, API v2.0, Component v1.4.1)
MCA bml: r2 (MCA v2.0, API v2.0, Component v1.4.1)
MCA rcache: vma (MCA v2.0, API v2.0, Component v1.4.1)
MCA btl: self (MCA v2.0, API v2.0, Component v1.4.1)
MCA btl: sm (MCA v2.0, API v2.0, Component v1.4.1)
MCA btl: tcp (MCA v2.0, API v2.0, Component v1.4.1)
MCA topo: unity (MCA v2.0, API v2.0, Component v1.4.1)
MCA osc: pt2pt (MCA v2.0, API v2.0, Component v1.4.1)
MCA osc: rdma (MCA v2.0, API v2.0, Component v1.4.1)
MCA iof: hnp (MCA v2.0, API v2.0, Component v1.4.1)
MCA iof: orted (MCA v2.0, API v2.0, Component v1.4.1)
MCA iof: tool (MCA v2.0, API v2.0, Component v1.4.1)
MCA oob: tcp (MCA v2.0, API v2.0, Component v1.4.1)
MCA odls: default (MCA v2.0, API v2.0, Component v1.4.1)
MCA ras: gridengine (MCA v2.0, API v2.0, Component v1.4.1)
MCA ras: slurm (MCA v2.0, API v2.0, Component v1.4.1)
MCA rmaps: load_balance (MCA v2.0, API v2.0, Component v1.4.1)
MCA rmaps: rank_file (MCA v2.0, API v2.0, Component v1.4.1)
MCA rmaps: round_robin (MCA v2.0, API v2.0, Component v1.4.1)
MCA rmaps: seq (MCA v2.0, API v2.0, Component v1.4.1)
MCA rml: oob (MCA v2.0, API v2.0, Component v1.4.1)
MCA routed: binomial (MCA v2.0, API v2.0, Component v1.4.1)
MCA routed: direct (MCA v2.0, API v2.0, Component v1.4.1)
MCA routed: linear (MCA v2.0, API v2.0, Component v1.4.1)
MCA plm: rsh (MCA v2.0, API v2.0, Component v1.4.1)
MCA plm: slurm (MCA v2.0, API v2.0, Component v1.4.1)
MCA filem: rsh (MCA v2.0, API v2.0, Component v1.4.1)
MCA errmgr: default (MCA v2.0, API v2.0, Component v1.4.1)
MCA ess: env (MCA v2.0, API v2.0, Component v1.4.1)
MCA ess: hnp (MCA v2.0, API v2.0, Component v1.4.1)
MCA ess: singleton (MCA v2.0, API v2.0, Component v1.4.1)
MCA ess: slurm (MCA v2.0, API v2.0, Component v1.4.1)
MCA ess: tool (MCA v2.0, API v2.0, Component v1.4.1)
MCA grpcomm: bad (MCA v2.0, API v2.0, Component v1.4.1)
MCA grpcomm: basic (MCA v2.0, API v2.0, Component v1.4.1)
config.log.gz
Description: GNU Zip compressed data
