To close out this issue and eat some crow. It ended up the issue I saw
below was caused by a messed up version of BLACS and that using the
scalapack installer to retrieve BLACS and the options mentioned in the
Open MPI FAQ does produce a working BLACS. So there is no need for
changing WHATMPI.
Sorry for the misinformation,
--td
Terry Dontje wrote:
Ok, I think I found the cause of the SPARC segv when trying to use a
64-bit compiled Open MPI library. If one does not set the WHATMPI
variable in the Bmake.inc it defaults to UseF77Mpi which assumes all
handles are ints. This is a correct assumption if you are using the
F77 interfaces but the way BLACS seems to compile for Open MPI it uses
the C versions. So the handles are stored as 32 bits in BLACS and
passed to the C Open MPI interfaces which expects 64 bits. In cases
where your addresses need more than 32 bits this will cause MPI to
segv when passed an invalid address due to this coersion.
So by setting "WHATMPI= -DUseCMpi" I've gotten the SPARC version of
BLACS compiled for 64 bits to pass its tests without segv'ing. I do
believe this issue actually exists for other platforms (ie AMD64 and
IA64) with other OSes and compilers. Just that we've been lucky that
MPI_COMM_WORLD is allocated such that it has an address that fits in
32 bits. I am amazed still that we haven't seen this fail in user
codes. Note, I have not confirmed this failure with a test case but
the code stack in dbx looks the same on X64 platforms as the code on
SPARC except the address is smaller on the former.
Greg, I would be interested in knowing if you are still seeing the
problem on Leopard and whether the above setting helps any.
--td
*
*Subject:* Re: [OMPI users] ScaLapack and BLACS on Leopard
*From:* Terry Dontje (/Terry.Dontje_at_[hidden]/)
*Date:* 2008-03-03 07:34:17
*
What kind of system lib errors are you seeing and do you have a stack
trace? Note, I was trying something similar with Solaris and 64-bit on
a SPARC machine and was seeing segv's inside the MPI Library due to a
pointer being passed through an integer (thus dropping the upper 32
bits). Funny thing is it all works under Solaris on AMD64 or IA-64
platforms.
--td
> Date: Thu, 28 Feb 2008 17:50:28 -0500
> From: Gregory John Orris <gregory.orris_at_[hidden]>
> Subject: [OMPI users] ScaLapack and BLACS on Leopard
> To: Open MPI Users <users_at_[hidden]>
> Message-ID: <528FD4C0-6157-49CB-80E6-1C62684E4545_at_[hidden]>
> Content-Type: text/plain; charset="us-ascii"
>
> Hey Folks,
>
> Anyone got ScaLapack and BLACS working and not just compiled under
> OSX10.5 in 64-bit mode?
> The FAQ site directions were followed and every thing compiles just
> fine. But ALL of the single precision routines and many of the double
> precisions routines in the TESTING directory fail with system lib
> errors.
>
> I've gotten some interesting errors and am wondering what the magic
> touch is.
>
> Regards,
> Greg
>
