Hi Ralph I have a motherboard with 2 X6580 chips, each with 6 cores 2 way hyperthreading, so /proc/cpuinfo reports 24 cores
Doing a pure compute OpenMP loop where I'd expect the number of iterations in 10s to rise with number of threads with gnu and mpich OMP_NUM_THREADS=1 -n 1 : 112 iterations OMP_NUM_THREADS=2 -n 1 : 224 iterations OMP_NUM_THREADS=6 -n 1 : 644 iterations OMP_NUM_THREADS=12 -n 1 : 1287 iterations OMP_NUM_THREADS=22 -n 1 : 1182 iterations OMP_NUM_THREADS=24 -n 1 : 454 iterations which shows that mpich is spreading across the cores, but hyperthreading is not useful, and using the whole node counterproductive with gnu and openmpi 1.8.3 OMP_NUM_THREADS=1 mpiexec -n 1 : 112 OMP_NUM_THREADS=2 mpiexec -n 1 : 113 which suggests you aren't allowing the threads to spread across cores adding --cpus-per-node I gain access to the resources on one chip OMP_NUM_THREADS=1 mpiexec --cpus-per-proc 1 -n 1 : 112 OMP_NUM_THREADS=2 mpiexec --cpus-per-proc 2 -n 1 : 224 OMP_NUM_THREADS=6 mpiexec --cpus-per-proc 2 -n 1 : 644 then OMP_NUM_THREADS=12 mpiexec --cpus-per-proc 12 -n 1 A request for multiple cpus-per-proc was given, but a directive was also give to map to an object level that has less cpus than requested ones: #cpus-per-proc: 12 number of cpus: 6 map-by: BYNUMA So you aren't happy using both chips for one process OMP_NUM_THREADS=1 mpiexec -n 1 --cpus-per-proc 1 --use-hwthread-cpus : 112 OMP_NUM_THREADS=2 mpiexec -n 1 --cpus-per-proc 2 --use-hwthread-cpus : 112 OMP_NUM_THREADS=4 mpiexec -n 1 --cpus-per-proc 4 --use-hwthread-cpus : 224 OMP_NUM_THREADS=6 mpiexec -n 1 --cpus-per-proc 6 --use-hwthread-cpus : 324 OMP_NUM_THREADS=6 mpiexec -n 1 --cpus-per-proc 12 --use-hwthread-cpus : 631 OMP_NUM_THREADS=12 mpiexec -n 1 --cpus-per-proc 12 --use-hwthread-cpus : 647 OMP_NUM_THREADS=24 mpiexec -n 1 --cpus-per-proc 12 --use-hwthread-cpus A request for multiple cpus-per-proc was given, but a directive was also give to map to an object level that has less cpus than requested ones: #cpus-per-proc: 24 number of cpus: 12 map-by: BYNUMA OMP_NUM_THREADS=1 mpiexec -n 1 --cpus-per-proc 2 --use-hwthread-cpus : 112 OMP_NUM_THREADS=2 mpiexec -n 1 --cpus-per-proc 4 --use-hwthread-cpus : 224 OMP_NUM_THREADS=6 mpiexec -n 1 --cpus-per-proc 12 --use-hwthread-cpus :: 644 OMP_NUM_THREADS=12 mpiexec -n 1 --cpus-per-proc 24 --use-hwthread-cpus :: 644 A request for multiple cpus-per-proc was given, but a directive was also give to map to an object level that has less cpus than requested ones: #cpus-per-proc: 24 number of cpus: 12 map-by: BYNUMA So it seems that --use-hwthread-cpus means that --cpus-per-proc changes from physical cores to hyperthreaded cores, but I can't get both chips working on the problem in way mpich can John
