Matt, I just found something else with your job script. You are defining the LD_LIBRARY_PATH environment variable, and *then* loading the openmpi environment module:
>>> #!/bin/bash >>> export TMPDIR=$SCRATCH/abyss_tmp/ >>> LD_LIBRARY_PATH=/work/01301/mmacmane >>> LD_LIBRARY_PATH=/work/01301/mmacmane/bin >>> LD_LIBRARY_PATH=/work/01301/mmacmane/include >>> LD_LIBRARY_PATH=/work/01301/mmacmane/etc >>> LD_LIBRARY_PATH=/work/01301/mmacmane/lib >>> LD_LIBRARY_PATH=/work/01301/mmacmane/openmpi-1.4.1 >>> cd /work/01301/mmacmane/Ray-0.0.6 >>> module load openmpi This is bad. Not only are you incorrectly setting your LD_LIBRARY_PATH, but after you modify it, the command module load openmpi is overwriting your customizations with what I presume is the default evironment for the officially supported openmpi version at TACC/on Ranger. Also from this line: >>> Module mvapich superceded it looks like you already had the mvapich module loaded before you submitted your openmpi job with qsub. This isn't a good idea, either. I don't know how your modules are written, but it's possible that your mvapich module has settings that are incompatible with your home-grown openmpi setup. You should make sure you remove your mvapich module before submitting your script. I don't use GNU environment modules (yet), so you'll have to contact your local system admin or read the module man page to learn how to remove the mvapich module. Just to be clear, these problems are problems with your basic Linux/Unix environment, and aren't really SGE- or OpenMPI-specific problems. -- Prentice
