Brock,
On the node you are running ompi_info is there a Truescale card present? i.e.
is /dev/ipath created and the ib_qib driver loaded? If not can you rerun
ompi_info on a node that does have it loaded (compute nodes?) and see if it
shows up. This is expected in the above scenario as the PSM m
Anton,
On the node that you saw the failure (u02n065) can you verify what the max
locked memory limit is set to? In a bash shell you can do this with ulimit -l.
It should be set to at least 128K. Also please verify that the available
memory on the node (/proc/meminfo shows this) is sufficient
