> Message: 4 > Date: Tue, 06 Dec 2005 12:15:37 -0700 > From: "Tim S. Woodall" <twood...@lanl.gov> > Subject: Re: [O-MPI users] error creating high priority cq for mthca0 > To: Open MPI Users <us...@open-mpi.org> > Message-ID: <4395e359.4060...@lanl.gov> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > Daryl, > > Try this: > > > -------- Original Message -------- > Subject: RE: only root running mpi jobs with 1.0.1rc5 > Date: Thu, 01 Dec 2005 18:49:46 -0700 > From: Joshua Aune <lu...@lnxi.com> > Reply-To: lu...@lnxi.com > Organization: Linux Networx > To: Todd Wilde <t...@mellanox.com> > CC: Matthew Finlay <m...@mellanox.com>, twood...@lanl.gov, Robert > Cummins <rcumm...@lnxi.com>, Pat Lindsay <plind...@lnxi.com> > References: <25AE7F432672D511B8DC00B0D0DF11DA05FC26CB@MTIEX01> > > Sounds like you were right > > * soft memlock 8388608 # 8 GB > * hard memlock 8388608 # 8 GB
Yes! This fixes the problem. Thanks, Daryl > and now I get no errors :) Looks like the limits were propigated to the > back end nodes. > > Tim, this should fix your problem as well? > > On Thu, 2005-12-01 at 17:26 -0800, Todd Wilde wrote: > > How about this one: > > > > For Redhat AS4.0 and Fedora Core 3 or a newer kernel, edit the > > file /etc/security/limits.conf and add the following two lines: > > > > soft memlock <number> > > > > hard memlock <number> > > > > The <number> value denotes the number of kilobytes that may be locked > > by a process. > > > > > -----Original Message----- > > > From: Joshua Aune [mailto:lu...@lnxi.com] > > > Sent: Thursday, December 01, 2005 3:50 PM > > > To: Todd Wilde > > > Cc: Matthew Finlay; twood...@lanl.gov; Robert Cummins; Pat Lindsay > > > Subject: RE: only root running mpi jobs with 1.0.1rc5 > > > > > > On Thu, 2005-12-01 at 15:39 -0800, Todd Wilde wrote: > > > > It may be a permissions issue with normal users locking memory. > > I've > > > > seen this in the past. Try adding the following command at boot: > > > > > > > > > > > > sysctl -w vm.disable_cap_mlock=1 > > > > > > This doesn't exist in 2.6.14... > > > > > > > > > > > > > > > > -----Original Message----- > > > > > From: Joshua Aune [mailto:lu...@lnxi.com] > > > > > Sent: Thursday, December 01, 2005 1:56 PM > > > > > To: Matthew Finlay; Todd Wilde; twood...@lanl.gov > > > > > Cc: Robert Cummins; Pat Lindsay > > > > > Subject: only root running mpi jobs with 1.0.1rc5 > > > > > > > > > > Root runs jobs fine but users don't. > > > > > > > > > > Any thoughts? > > > > > > > > > > Thanks, > > > > > josh > > > > > > > > > > coyote2-compute# module purge > > > > > coyote2-compute# module load compiler/gcc mpi/openmpi-1.0.1rc5 > > > > > coyote2-compute# cd /home/luken/hello > > > > > coyote2-compute# mpirun -np 2 -H 201,202 mpi_hello > > > > > n201: I am rank 0 > > > > > n202: I am rank 1 > > > > > > > > > > > > > > > coyote2-compute$ su - luken > > > > > coyote2-compute$ module purge > > > > > coyote2-compute$ module load compiler/gcc mpi/openmpi-1.0.1rc5 > > > > > coyote2-compute$ cd /home/luken/hello > > > > > coyote2-compute$ mpirun -np 2 -H 201,202 mpi_hello > > > > > [0,1,0][btl_openib.c:803:mca_btl_openib_module_init] error > > creating > > > > high > > > > > priority cq for mthca0 errno says Cannot allocate memory > > > > > [0,1,1][btl_openib.c:803:mca_btl_openib_module_init] error > > creating > > > > high > > > > > priority cq for mthca0 errno says Cannot allocate memory > > > > > > > > > > n201: I am rank 0 > > > > > > > > > > n202: I am rank 1 > > > > > > > > > > > Daryl W. Grunau wrote: > > Hi, I'm running OMPI 1.1a1r8378 on 2.6.14 + recent OpenIB stack and getting > > the following runtime error: > > > > [0,1,0][btl_openib.c:803:mca_btl_openib_module_init] error creating high > > priority cq for mthca0 errno says Cannot allocate memory > > [0,1,3][btl_openib.c:803:mca_btl_openib_module_init] error creating high > > priority cq for mthca0 errno says Cannot allocate memory > > [0,1,1][btl_openib.c:803:mca_btl_openib_module_init] error creating high > > priority cq for mthca0 errno says Cannot allocate memory > > [0,1,2][btl_openib.c:803:mca_btl_openib_module_init] error creating high > > priority cq for mthca0 errno says Cannot allocate memory > > > > > > Strange thing is that it works properly when I run as root. A permissions > > problem on my part? My devices look like: > > > > # ls -l /dev/infiniband/* > > crw------- 1 root root 231, 64 Dec 5 17:16 /dev/infiniband/issm0 > > crw------- 1 root root 231, 65 Dec 5 17:16 /dev/infiniband/issm1 > > crw------- 1 root root 231, 0 Dec 5 17:16 /dev/infiniband/umad0 > > crw------- 1 root root 231, 1 Dec 5 17:16 /dev/infiniband/umad1 > > crw-rw-rw- 1 root root 231, 192 Dec 5 17:16 /dev/infiniband/uverbs0 > > > > Daryl
