Hi Mostyn,
On Tue, 24 Apr 2007, Mostyn Lewis wrote:
> Well, I'm sorry to have caused even a smidgen of grief here.
> I moved aside the *paffinity_linux* module and la and it still
> bound. I was using InfiniPath HCAs and beta software and eventually found
> (sigh) a variable to stop the affine - IPATH_NO_CPUAFFINITY.
Ah! I actually followed the entire thread, but didn't know you were
working with InfiniPath.
> So, a
>
> export IPATH_NO_CPUAFFINITY=1
> $OPENMPI_GCC/bin/mpirun -x IPATH_NO_CPUAFFINITY -np 1 -host s0158 ./cpi
>
>
> showed me what I wanted to see:
>
> 18236:cpi *->0 (f=noaffinity,0,1,2,3)
>
> This, in the jargon of my utility, says the mask for taskset is 0xf
> and so is not affined and the ->0 says it's on CPU 0.
>
> The reason all this comes about is I do endless benchmarks for my
> employer and get to use Scali, QuickSilver(SilverStorm), Qlogic(InfiniPath),
> all the ethernet MPICHes and LAMs (fading fast) - even HP MPI - on
> our racks which have x cores / socket and sometimes we like to use
> our own methodoligies to choose where to bind and in that case need to
> switch off any supplied binding. I really wish the default was no
> binding like OpenMPI with docs that point out the variables but it's
> not always the case.
Turns out that for performance, it's extremely important to set the
affinity as early as possible to get all potential library (and even
driver) allocated memory correctly bound on a NUMA machine. So we do
it for performance and in a mode we hope to be a safe default. This
last goal is fairly difficult since different linux kernels number
the actual cores in different numerical orders. Then there's the
case where OpenMP is thrown into the mix with its own affinity
requirements.
Our User Guide (section 2.10.5) acknowledges that CPU is always set
on InfiniPath and suggest how to change the affinity but don't say
how to turn it off completely (with IPATH_NO_CPUAFFINITY). I'll
suggest that this be updated in the documentation and meanwhile,
mention that I'm coming off as defensive about our User guide but
like many developers, am not very good about reading other people's
documentation.
. . christian
--
christian.b...@qlogic.com
(QLogic SIG, formerly Pathscale)
