Yep. I thought it was the startup script, but it was merely the fact I restarted it that got it going.
I wonder if adding ulimit -l unlimited to the startup script will help, tho. Brian Andrus perotsystems Site Manager | Sr. Computer Scientist Naval Research Lab 7 Grace Hopper Ave, Monterey, CA 93943 Phone (831) 656-4839 | Fax (831) 656-4866 -----Original Message----- From: users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] On Behalf Of pat.o'bry...@exxonmobil.com Sent: Thursday, November 08, 2007 5:55 AM To: Open MPI Users Subject: Re: [OMPI users] openib errors as user, but not root What we discovered is that our PBS mom daemon did not have unlimited locked memory. So, when your job is created by the mom daemon it inherits the memory limits. The fix was to cycle the PBS mom daemon after a boot (and yes, we do start the mom daemon at boot but for some reason it doesn't inherit unlimited locked memory). The way to determine if this is the problem is to place a "ulimit -a" in the text of your PBS job. Run your job and you will see a limit of 32K. Next cycle the mom daemon on the node(s) of interest and re-run your job. You will now see unlimited memory. Thanks, Pat O'Bryant J.W. (Pat) O'Bryant,Jr. Business Line Infrastructure Technical Systems, HPC Jeff Squyres <jsquyres@cisc o.com> To Sent by: Open MPI Users <us...@open-mpi.org> users-bounces@ cc open-mpi.org Subject Re: [OMPI users] openib errors as 11/07/07 06:25 user, but not root PM Please respond to Open MPI Users <users@open-mp i.org> Check out: http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages-more In particular, see the stuff about using resource managers. On Nov 7, 2007, at 7:22 PM, Andrus, Mr. Brian (Contractor) wrote: > Ok, I am having some difficulty troubleshooting this. > > If I run my hello program without torque, it works fine: > [root@login1 root]# mpirun --mca btl openib,self -host > n01,n02,n03,n04,n05 /data/root/hello > Hello from process 0 of 5 on node n01 > Hello from process 1 of 5 on node n02 > Hello from process 2 of 5 on node n03 > Hello from process 3 of 5 on node n04 > Hello from process 4 of 5 on node n05 > > If I submit it as root, it seems happy: > [root@login1 root]# qsub > #!/bin/bash > #PBS -j oe > #PBS -l nodes=5:ppn=1 > #PBS -W x=NACCESSPOLICY:SINGLEJOB > #PBS -N TestJob > #PBS -q long > #PBS -o output.txt > #PBS -V > cd $PBS_O_WORKDIR > rm -f output.txt > date > mpirun --mca btl openib,self /data/root/hello > 103.cluster.default.domain > [root@login1 root]# cat output.txt > Wed Nov 7 16:20:33 PST 2007 > Hello from process 0 of 5 on node n05 > Hello from process 1 of 5 on node n04 > Hello from process 2 of 5 on node n03 > Hello from process 3 of 5 on node n02 > Hello from process 4 of 5 on node n01 > > If I do it as me, not so good: > [andrus@login1 data]$ qsub > [andrus@login1 data]$ qsub > #!/bin/bash > #PBS -j oe > #PBS -l nodes=1:ppn=1 > #PBS -W x=NACCESSPOLICY:SINGLEJOB > #PBS -N TestJob > #PBS -q long > #PBS -o output.txt > #PBS -V > cd $PBS_O_WORKDIR > rm -f output.txt > date > mpirun --mca btl openib,self /data/root/hello > 105.littlemac.default.domain > [andrus@login1 data]$ cat output.txt > Wed Nov 7 16:23:00 PST 2007 > ------------------------------------------------------------------------ -- > The OpenIB BTL failed to initialize while trying to allocate some > locked memory. This typically can indicate that the memlock limits > are set too low. For most HPC installations, the memlock limits > should be set to "unlimited". The failure occured here: > > Host: n01 > OMPI source: btl_openib.c:828 > Function: ibv_create_cq() > Device: mthca0 > Memlock limit: 32768 > > You may need to consult with your system administrator to get this > problem fixed. This FAQ entry on the Open MPI web site may also be > helpful: > > http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages > ------------------------------------------------------------------------ -- > ------------------------------------------------------------------------ -- > It looks like MPI_INIT failed for some reason; your parallel process > is likely to abort. There are many reasons that a parallel process > can fail during MPI_INIT; some of which are due to configuration or > environment problems. This failure appears to be an internal failure; > here's some additional information (which may only be relevant to an > Open MPI > developer): > > PML add procs failed > --> Returned "Error" (-1) instead of "Success" (0) > ------------------------------------------------------------------------ -- > *** An error occurred in MPI_Init > *** before MPI was initialized > *** MPI_ERRORS_ARE_FATAL (goodbye) > > > > I have checked that ulimit is unlimited. I cannot seem to figure this. > Any help? > Brian Andrus perotsystems > Site Manager | Sr. Computer Scientist > Naval Research Lab > 7 Grace Hopper Ave, Monterey, CA 93943 Phone (831) 656-4839 | Fax > (831) 656-4866 _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users -- Jeff Squyres Cisco Systems _______________________________________________ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users _______________________________________________ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users
