[OMPI users] control openmpi or force to use pbs?

[Duke Nguyen]

Hi all,

Please advise me how to force our users to use pbs instead of "mpirun
--hostfile"? Or how do I control mpirun so that any user using "mpirun
--hostfile" will not overload the cluster? We have OpenMPI installed
with Torque/Maui and we can control users's limits (total number of
procs, total nodes, total memories etc) with Torque/Maui, but if a user
knows the cluster, and creates himself a hostfile with all the available
nodes then he can use them all.

Thanks,

D.



signature.asc
Description: OpenPGP digital signature

Re: [OMPI users] control openmpi or force to use pbs?

[Duke Nguyen]

On 2/6/13 1:03 AM, Gus Correa wrote:
>
> On 02/05/2013 08:52 AM, Jeff Squyres (jsquyres) wrote:
>> To add to what Reuti said, if you enable PBS support in Open MPI,
when users "mpirun ..." in a PBS job, Open MPI will automatically use
the PBS native launching mechanism, which won't let you run outside of
the servers allocated to that job.
>>
>> Concrete example: if you qsub a job and are allocated node A, B, and
C, but then try to run with "mpirun --host D,E,F ...", you'll get an error.
>>
>> That being said -- keep in mind what Reuti said: if users are allowed
to ssh between nodes that are not allocated to them, then they can
always bypass this behavior and use just Open MPI's ssh support to
launch on nodes D, E, F (etc.).

>> On Feb 5, 2013, at 2:46 AM, Reuti wrote:
> Can the users use a plain ssh between the nodes? If they are forced to
> use the TM of Torque instead, it should be impossible to start a job
> on a non-granted machine.
>
Thanks Reuti and Jeff, you are right, users should not be allowed to ssh
to all nodes, which is how our cluster was set up: users can even
password-less ssh to any node. I know this is not appropriate question
in OpenMPI forum, but how can we setup so that user can only ssh (with
password) to nodes that are allocated to them at the time of qsub'ing? I
am still new to all of this cluster thing :)


>
> Besides Reuti's and Jeff's suggestions.
>
> If you build your own Torque/PBS with PAM support
> (./configure --with-pam [other configure flags]),
> you can prevent users that are not running a Torque/PBS job
> on a node to launch processes in that node.
>
> See this:
>
http://docs.adaptivecomputing.com/torque/4-1-3/help.htm#topics/1-installConfig/customizingTheInstall.htm
>
> Of course you will need to rebuild your OpenMPI with Torque
> support again, after you install a version of Torque with PAM
> support.
>
> This is mostly a Torque/Maui issue, with a bit of an MPI issue.
> You may get more help about this on the Torque and Maui
> mailing lists, and in their archives you may find more specific
> guidance on what you need to add to the pam/security
> files to make it work.

Thanks for this! I will try this --with-pam when our test job finishes.

>
> Torque with PAM support is not 100% foolproof,
> because users that *are* running a Torque/PBS job on
> a node can still cheat and launch more processes there,
> but it helps restrict the problem to this case.
>
> Some sys admins also add a cleanup/sweep routine to the
> Torque epilogue script to kill any processes belonging to
> the user whose job just finished.
> However, this not very good because that user may have another
> legitimate job still running there.
> Other cleanup strategies are possible, and you may find some
> suggestions and even scripts if you google around.
>
> Moreover, if you configure your scheduler (Maui?) to
> assign full nodes to jobs (no node sharing),
> the cheaters will be cheating on
> themselves, not stepping on other users' toes.
> Look for "JOBNODEMATCHPOLICY" here:
> http://docs.adaptivecomputing.com/maui/a.fparameters.php
>
> Assigning full nodes to jobs ("EXACTNODE") may or may not be a
> good choice for you.
> E.g. you may consider it wasteful, if there are many serial
> jobs or parallel jobs running only on a few processors, in
> which case you may want to pack those jobs in the fewest
> nodes possible ("EXACTPROC"), so as to have a maximum throughput.
> However, "no node sharing" helps preventing cheaters
> to bother other users that are running jobs on the same node,
> and it is not bad at all if most of the jobs are parallel
> and use many cores (say, >= number of cores per node).

Yes, we use Maui, but I let it at its' default settings since I have no
idea and experience about maui's options. I like EXACTPROC option. Our
cluster is small so that might be better for us.





signature.asc
Description: OpenPGP digital signature

Re: [OMPI users] control openmpi or force to use pbs?

[Duke Nguyen]

On 2/5/13 11:20 PM, John Hearns wrote:
> Lart your users. Its the only way.
> they will thank you for it it, eventually.
>  
>  
> www.catb.org/jargon/html/L/LART.html
> 
>  

Not sure if I get this right (first time heard of LART anyway)... Do you
mean that we should *educate* our users?

Thanks,

D.


signature.asc
Description: OpenPGP digital signature

Re: [OMPI users] control openmpi or force to use pbs?

[Duke Nguyen]

On 2/6/13 10:06 PM, Jeff Squyres (jsquyres) wrote:
> On Feb 6, 2013, at 5:11 AM, Reuti  wrote:
>
>>> Thanks Reuti and Jeff, you are right, users should not be allowed to ssh to 
>>> all nodes, which is how our cluster was set up: users can even 
>>> password-less ssh to any node. I know this is not appropriate question in 
>>> OpenMPI forum, but how can we setup so that user can only ssh (with 
>>> password) to nodes that are allocated to them at the time of qsub'ing? I am 
>>> still new to all of this cluster thing :)
>> I even disallow this. Only admin staff is allowed to login to the nodes. 
>> This forces also the admin to look for a tight integration of the user's 
>> software into the queuing system.
>
> +1
>
> FWIW, that makes one-more-thing that you have to setup and maintain (because 
> it doesn't happen by default -- you'd have to add some extra scripting in the 
> ssh authentication stuff to enable that functionality).
>

Thanks, that what I want to do too, but I thought if it is impossible
because ssh is needed for seting up a cluster. From what I understand:

 * for an user to run pbs jobs, master and clients should have that user
on their passwd/shadow/group files
 * configure ssh server on clients to prohibit certain users

Is that right?



signature.asc
Description: OpenPGP digital signature

Re: [OMPI users] control openmpi or force to use pbs?

[Duke Nguyen]

Sorry for the late update. Anyway, per suggestions, here is what I did:
 * prevent ssh-login to the nodes except admins
 * reconfigure torque with --with-pam (then reinstall torque, openmpi
etc...)

After testing for a few days with some intensive jobs, everything looks
fine :)

Thanks for all the helps/suggestsions/comments,

D.

On 2/6/13 10:58 PM, Reuti wrote:
> Am 06.02.2013 um 16:45 schrieb Duke Nguyen:
>
> > On 2/6/13 10:06 PM, Jeff Squyres (jsquyres) wrote:
> >> On Feb 6, 2013, at 5:11 AM, Reuti  wrote:
> >>
> >>>> Thanks Reuti and Jeff, you are right, users should not be allowed
> to ssh to all nodes, which is how our cluster was set up: users can
> even password-less ssh to any node. I know this is not appropriate
> question in OpenMPI forum, but how can we setup so that user can only
> ssh (with password) to nodes that are allocated to them at the time of
> qsub'ing? I am still new to all of this cluster thing :)
> >>> I even disallow this. Only admin staff is allowed to login to the
> nodes. This forces also the admin to look for a tight integration of
> the user's software into the queuing system.
> >>
> >> +1
> >>
> >> FWIW, that makes one-more-thing that you have to setup and maintain
> (because it doesn't happen by default -- you'd have to add some extra
> scripting in the ssh authentication stuff to enable that functionality).
> >>
>
> > Thanks, that what I want to do too, but I thought if it is impossible
> > because ssh is needed for seting up a cluster. From what I understand:
>
> > * for an user to run pbs jobs, master and clients should have that user
> > on their passwd/shadow/group files
>
> Or use NIS / LDAP to have a central location for this information.
>
>
> > * configure ssh server on clients to prohibit certain users
>
> Correct, like a line in /etc/ssh/sshd_config:
>
> AllowGroups admin
>
> and only admin staff has this group as one of their secondary groups
> attached.
>
> -- Reuti
>
>
> > Is that right?
>
> > ___
> > users mailing list
> > us...@open-mpi.org
> > http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>




signature.asc
Description: OpenPGP digital signature

[OMPI users] memory per core/process

[Duke Nguyen]

Hi folks,

I am sorry if this question had been asked before, but after ten days of 
searching/working on the system, I surrender :(. We try to use mpirun to 
run abinit (abinit.org) which in turns will call an input file to run 
some simulation. The command to run is pretty simple


$ mpirun -np 4 /opt/apps/abinit/bin/abinit < input.files >& output.log

We ran this command on a server with two quad core x5420 and 16GB of 
memory. I called only 4 core, and I guess in theory each of the core 
should take up to 2GB each.


In the output of the log, there is something about memory:

P This job should need less than 717.175 Mbytes of 
memory.

  Rough estimation (10% accuracy) of disk space for files :
  WF disk file : 69.524 Mbytes ; DEN or POT disk file : 14.240 Mbytes.

So basically it reported that the above job should not take more than 
718MB each core.


But I still have the Segmentation Fault error:

mpirun noticed that process rank 0 with PID 16099 on node biobos exited 
on signal 11 (Segmentation fault).


The system already has limits up to unlimited:

$ cat /etc/security/limits.conf | grep -v '#'
* soft memlock unlimited
* hard memlock unlimited

I also tried to run

$ ulimit -l unlimited

before the mpirun command above, but it did not help at all.

If we adjust the parameters of the input.files to give the reported mem 
per core is less than 512MB, then the job runs fine.


Please help,

Thanks,

D.

Re: [OMPI users] memory per core/process

[Duke Nguyen]

On 3/30/13 3:13 PM, Patrick Bégou wrote:

I do not know about your code but:

1) did you check stack limitations ? Typically intel fortran codes 
needs large amount of stack when the problem size increase.

Check ulimit -a


First time I heard of stack limitations. Anyway, ulimit -a gives

$ ulimit -a
core file size  (blocks, -c) 0
data seg size   (kbytes, -d) unlimited
scheduling priority (-e) 0
file size   (blocks, -f) unlimited
pending signals (-i) 127368
max locked memory   (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files  (-n) 1024
pipe size(512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority  (-r) 0
stack size  (kbytes, -s) 10240
cpu time   (seconds, -t) unlimited
max user processes  (-u) 1024
virtual memory  (kbytes, -v) unlimited
file locks  (-x) unlimited

So stack size is 10MB??? Does this one create problem? How do I change this?



2) did your node uses cpuset and memory limitation like fake numa to 
set the maximum amount of memory available for a job ?


Not really understand (also first time heard of fake numa), but I am 
pretty sure we do not have such things. The server I tried was a 
dedicated server with 2 x5420 and 16GB physical memory.




Patrick

Duke Nguyen a écrit :

Hi folks,

I am sorry if this question had been asked before, but after ten days 
of searching/working on the system, I surrender :(. We try to use 
mpirun to run abinit (abinit.org) which in turns will call an input 
file to run some simulation. The command to run is pretty simple


$ mpirun -np 4 /opt/apps/abinit/bin/abinit < input.files >& output.log

We ran this command on a server with two quad core x5420 and 16GB of 
memory. I called only 4 core, and I guess in theory each of the core 
should take up to 2GB each.


In the output of the log, there is something about memory:

P This job should need less than 717.175 Mbytes 
of memory.

  Rough estimation (10% accuracy) of disk space for files :
  WF disk file : 69.524 Mbytes ; DEN or POT disk file : 14.240 
Mbytes.


So basically it reported that the above job should not take more than 
718MB each core.


But I still have the Segmentation Fault error:

mpirun noticed that process rank 0 with PID 16099 on node biobos 
exited on signal 11 (Segmentation fault).


The system already has limits up to unlimited:

$ cat /etc/security/limits.conf | grep -v '#'
* soft memlock unlimited
* hard memlock unlimited

I also tried to run

$ ulimit -l unlimited

before the mpirun command above, but it did not help at all.

If we adjust the parameters of the input.files to give the reported 
mem per core is less than 512MB, then the job runs fine.


Please help,

Thanks,

D.


___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users


___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] memory per core/process

[Duke Nguyen]

On 3/30/13 5:22 PM, Duke Nguyen wrote:

On 3/30/13 3:13 PM, Patrick Bégou wrote:

I do not know about your code but:

1) did you check stack limitations ? Typically intel fortran codes 
needs large amount of stack when the problem size increase.

Check ulimit -a


First time I heard of stack limitations. Anyway, ulimit -a gives

$ ulimit -a
core file size  (blocks, -c) 0
data seg size   (kbytes, -d) unlimited
scheduling priority (-e) 0
file size   (blocks, -f) unlimited
pending signals (-i) 127368
max locked memory   (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files  (-n) 1024
pipe size(512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority  (-r) 0
stack size  (kbytes, -s) 10240
cpu time   (seconds, -t) unlimited
max user processes  (-u) 1024
virtual memory  (kbytes, -v) unlimited
file locks  (-x) unlimited

So stack size is 10MB??? Does this one create problem? How do I change 
this?


I did $ ulimit -s unlimited to have stack size to be unlimited, and the 
job ran fine!!! So it looks like stack limit is the problem. Questions are:


 * how do I set this automatically (and permanently)?
 * should I set all other ulimits to be unlimited?

Thanks,

D.





2) did your node uses cpuset and memory limitation like fake numa to 
set the maximum amount of memory available for a job ?


Not really understand (also first time heard of fake numa), but I am 
pretty sure we do not have such things. The server I tried was a 
dedicated server with 2 x5420 and 16GB physical memory.




Patrick

Duke Nguyen a écrit :

Hi folks,

I am sorry if this question had been asked before, but after ten 
days of searching/working on the system, I surrender :(. We try to 
use mpirun to run abinit (abinit.org) which in turns will call an 
input file to run some simulation. The command to run is pretty simple


$ mpirun -np 4 /opt/apps/abinit/bin/abinit < input.files >& output.log

We ran this command on a server with two quad core x5420 and 16GB of 
memory. I called only 4 core, and I guess in theory each of the core 
should take up to 2GB each.


In the output of the log, there is something about memory:

P This job should need less than 717.175 Mbytes 
of memory.

  Rough estimation (10% accuracy) of disk space for files :
  WF disk file : 69.524 Mbytes ; DEN or POT disk file : 14.240 
Mbytes.


So basically it reported that the above job should not take more 
than 718MB each core.


But I still have the Segmentation Fault error:

mpirun noticed that process rank 0 with PID 16099 on node biobos 
exited on signal 11 (Segmentation fault).


The system already has limits up to unlimited:

$ cat /etc/security/limits.conf | grep -v '#'
* soft memlock unlimited
* hard memlock unlimited

I also tried to run

$ ulimit -l unlimited

before the mpirun command above, but it did not help at all.

If we adjust the parameters of the input.files to give the reported 
mem per core is less than 512MB, then the job runs fine.


Please help,

Thanks,

D.


___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users


___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] memory per core/process

[Duke Nguyen]

On 3/30/13 8:20 PM, Reuti wrote:

Am 30.03.2013 um 13:26 schrieb Tim Prince:


On 03/30/2013 06:36 AM, Duke Nguyen wrote:

On 3/30/13 5:22 PM, Duke Nguyen wrote:

On 3/30/13 3:13 PM, Patrick Bégou wrote:

I do not know about your code but:

1) did you check stack limitations ? Typically intel fortran codes needs large 
amount of stack when the problem size increase.
Check ulimit -a

First time I heard of stack limitations. Anyway, ulimit -a gives

$ ulimit -a
core file size  (blocks, -c) 0
data seg size   (kbytes, -d) unlimited
scheduling priority (-e) 0
file size   (blocks, -f) unlimited
pending signals (-i) 127368
max locked memory   (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files  (-n) 1024
pipe size(512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority  (-r) 0
stack size  (kbytes, -s) 10240
cpu time   (seconds, -t) unlimited
max user processes  (-u) 1024
virtual memory  (kbytes, -v) unlimited
file locks  (-x) unlimited

So stack size is 10MB??? Does this one create problem? How do I change this?

I did $ ulimit -s unlimited to have stack size to be unlimited, and the job ran 
fine!!! So it looks like stack limit is the problem. Questions are:

* how do I set this automatically (and permanently)?
* should I set all other ulimits to be unlimited?


In our environment, the only solution we found is to have mpirun run a script 
on each node which sets ulimit (as well as environment variables which are more 
convenient to set there than in the mpirun), before starting the executable.  
We had expert recommendations against this but no other working solution.  It 
seems unlikely that you would want to remove any limits which work at default.
Stack size unlimited in reality is not unlimited; it may be limited by a system 
limit or implementation.  As we run up to 120 threads per rank and many 
applications have threadprivate data regions, ability to run without 
considering stack limit is the exception rather than the rule.

Even if I would be the only user on a cluster of machines, I would define this 
in any queuingsystem to set the limits for the job.


Sorry if I dont get this correctly, but do you mean I should set this 
using Torque/Maui (our queuing manager) instead of the system itself 
(/etc/security/limits.conf and /etc/profile.d/)?

Re: [OMPI users] memory per core/process

[Duke Nguyen]

I should really have asked earlier. Thanks for all the helps.

D.

On 3/30/13 10:28 PM, Ralph Castain wrote:

FWIW: there is an MCA param that helps with such problems:

 opal_set_max_sys_limits
  "Set to non-zero to automatically set any system-imposed limits to 
the maximum allowed",

At the moment, it only sets the limits on number of files open, and max size of 
a file we can create. Easy enough to add the stack size, though as someone 
pointed out, it has some negatives as well.


On Mar 30, 2013, at 7:35 AM, Gustavo Correa  wrote:


On Mar 30, 2013, at 10:02 AM, Duke Nguyen wrote:


On 3/30/13 8:20 PM, Reuti wrote:

Am 30.03.2013 um 13:26 schrieb Tim Prince:


On 03/30/2013 06:36 AM, Duke Nguyen wrote:

On 3/30/13 5:22 PM, Duke Nguyen wrote:

On 3/30/13 3:13 PM, Patrick Bégou wrote:

I do not know about your code but:

1) did you check stack limitations ? Typically intel fortran codes needs large 
amount of stack when the problem size increase.
Check ulimit -a

First time I heard of stack limitations. Anyway, ulimit -a gives

$ ulimit -a
core file size  (blocks, -c) 0
data seg size   (kbytes, -d) unlimited
scheduling priority (-e) 0
file size   (blocks, -f) unlimited
pending signals (-i) 127368
max locked memory   (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files  (-n) 1024
pipe size(512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority  (-r) 0
stack size  (kbytes, -s) 10240
cpu time   (seconds, -t) unlimited
max user processes  (-u) 1024
virtual memory  (kbytes, -v) unlimited
file locks  (-x) unlimited

So stack size is 10MB??? Does this one create problem? How do I change this?

I did $ ulimit -s unlimited to have stack size to be unlimited, and the job ran 
fine!!! So it looks like stack limit is the problem. Questions are:

* how do I set this automatically (and permanently)?
* should I set all other ulimits to be unlimited?


In our environment, the only solution we found is to have mpirun run a script 
on each node which sets ulimit (as well as environment variables which are more 
convenient to set there than in the mpirun), before starting the executable.  
We had expert recommendations against this but no other working solution.  It 
seems unlikely that you would want to remove any limits which work at default.
Stack size unlimited in reality is not unlimited; it may be limited by a system 
limit or implementation.  As we run up to 120 threads per rank and many 
applications have threadprivate data regions, ability to run without 
considering stack limit is the exception rather than the rule.

Even if I would be the only user on a cluster of machines, I would define this 
in any queuingsystem to set the limits for the job.

Sorry if I dont get this correctly, but do you mean I should set this using 
Torque/Maui (our queuing manager) instead of the system itself 
(/etc/security/limits.conf and /etc/profile.d/)?

Hi Duke

We do both.
Set memlock and stacksize to unlimited, and increase the maximum number of
open files  in the pbs_mom script in /etc/init.d, and do the same in 
/etc/security/limits.conf.
This maybe an overzealous  "belt and suspenders" policy, but it works.
As everybody else said, a small stacksize is a common cause of segmentation 
fault in
large codes.
Basically all codes that we run here have this problem, with too many
automatic arrays, structures, etc in functions and subroutines.
But also a small memlock is trouble for OFED/Infinband, and the small (default)
max number of open file handles may hit the limit easily if many programs
(or poorly written  programs) are running in the same node.
The default Linux distribution limits don't seem to be tailored for HPC, I 
guess.

I hope this helps,
Gus Correa


___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users


___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] memory per core/process

[Duke Nguyen]

On 3/31/13 12:20 AM, Duke Nguyen wrote:

I should really have asked earlier. Thanks for all the helps.


I think I was excited too soon :). Increasing stacksize does help if I 
run a job in a dedicated server. Today I tried to modify the cluster 
(/etc/security/limits.conf, /etc/init.d/pbs_mom) and tried to run a 
different job with 4 nodes/8 core each (nodes=4:ppn=8), but I still get 
the mpirun error. My ulimit now reads:


$ ulimit -a
core file size  (blocks, -c) 0
data seg size   (kbytes, -d) unlimited
scheduling priority (-e) 0
file size   (blocks, -f) unlimited
pending signals (-i) 8271027
max locked memory   (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files  (-n) 32768
pipe size(512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority  (-r) 0
stack size  (kbytes, -s) unlimited
cpu time   (seconds, -t) unlimited
max user processes  (-u) 8192
virtual memory  (kbytes, -v) unlimited
file locks  (-x) unlimited

Any other advice???



On 3/30/13 10:28 PM, Ralph Castain wrote:

FWIW: there is an MCA param that helps with such problems:

 opal_set_max_sys_limits
  "Set to non-zero to automatically set any 
system-imposed limits to the maximum allowed",


At the moment, it only sets the limits on number of files open, and 
max size of a file we can create. Easy enough to add the stack size, 
though as someone pointed out, it has some negatives as well.



On Mar 30, 2013, at 7:35 AM, Gustavo Correa  
wrote:



On Mar 30, 2013, at 10:02 AM, Duke Nguyen wrote:


On 3/30/13 8:20 PM, Reuti wrote:

Am 30.03.2013 um 13:26 schrieb Tim Prince:


On 03/30/2013 06:36 AM, Duke Nguyen wrote:

On 3/30/13 5:22 PM, Duke Nguyen wrote:

On 3/30/13 3:13 PM, Patrick Bégou wrote:

I do not know about your code but:

1) did you check stack limitations ? Typically intel fortran 
codes needs large amount of stack when the problem size increase.

Check ulimit -a

First time I heard of stack limitations. Anyway, ulimit -a gives

$ ulimit -a
core file size  (blocks, -c) 0
data seg size   (kbytes, -d) unlimited
scheduling priority (-e) 0
file size   (blocks, -f) unlimited
pending signals (-i) 127368
max locked memory   (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files  (-n) 1024
pipe size(512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority  (-r) 0
stack size  (kbytes, -s) 10240
cpu time   (seconds, -t) unlimited
max user processes  (-u) 1024
virtual memory  (kbytes, -v) unlimited
file locks  (-x) unlimited

So stack size is 10MB??? Does this one create problem? How do I 
change this?
I did $ ulimit -s unlimited to have stack size to be unlimited, 
and the job ran fine!!! So it looks like stack limit is the 
problem. Questions are:


* how do I set this automatically (and permanently)?
* should I set all other ulimits to be unlimited?

In our environment, the only solution we found is to have mpirun 
run a script on each node which sets ulimit (as well as 
environment variables which are more convenient to set there than 
in the mpirun), before starting the executable.  We had expert 
recommendations against this but no other working solution.  It 
seems unlikely that you would want to remove any limits which 
work at default.
Stack size unlimited in reality is not unlimited; it may be 
limited by a system limit or implementation.  As we run up to 120 
threads per rank and many applications have threadprivate data 
regions, ability to run without considering stack limit is the 
exception rather than the rule.
Even if I would be the only user on a cluster of machines, I would 
define this in any queuingsystem to set the limits for the job.
Sorry if I dont get this correctly, but do you mean I should set 
this using Torque/Maui (our queuing manager) instead of the system 
itself (/etc/security/limits.conf and /etc/profile.d/)?

Hi Duke

We do both.
Set memlock and stacksize to unlimited, and increase the maximum 
number of
open files  in the pbs_mom script in /etc/init.d, and do the same in 
/etc/security/limits.conf.

This maybe an overzealous  "belt and suspenders" policy, but it works.
As everybody else said, a small stacksize is a common cause of 
segmentation fault in

large codes.
Basically all codes that we run here have this problem, with too many
automatic arrays, structures, etc in functions and subroutines.
But also a small memlock is trouble for OFED/Infinband, and the 
small (default)
max number of open file handles may hit the limit easily if many 
programs

(or poorly written  programs) are running in the same node.
The default Linux

Re: [OMPI users] memory per core/process

[Duke Nguyen]

On 4/1/13 9:20 PM, Ralph Castain wrote:

It's probably the same problem - try running 'mpirun -npernode 1 -tag-output ulimit 
-a"  on the remote nodes and see what it says. I suspect you'll find that they 
aren't correct.


Somehow I could not run your advised CMD:

$ qsub -l nodes=4:ppn=8 -I
qsub: waiting for job 481.biobos to start
qsub: job 481.biobos ready

$ /usr/local/bin/mpirun -npernode 1 -tag-output ulimit -a
--
mpirun was unable to launch the specified application as it could not 
find an executable:


Executable: ulimit
Node: node0108.biobos

while attempting to start process rank 0.
--
4 total processes failed to start

But anyway, I figured out the reason. Yes, it is the cluster nodes that 
did not update ulimit settings (our system is a diskless node with 
warewulf so basically we have to update the vnfs and reboot all nodes 
before the nodes can run with new settings).


Thanks for all the helps :)

D.



BTW: the "-tag-output'" option marks each line of output with the rank of the 
process. Since all the outputs will be interleaved, this will help you identify what came 
from each node.


On Mar 31, 2013, at 11:30 PM, Duke Nguyen  wrote:


On 3/31/13 12:20 AM, Duke Nguyen wrote:

I should really have asked earlier. Thanks for all the helps.

I think I was excited too soon :). Increasing stacksize does help if I run a 
job in a dedicated server. Today I tried to modify the cluster 
(/etc/security/limits.conf, /etc/init.d/pbs_mom) and tried to run a different 
job with 4 nodes/8 core each (nodes=4:ppn=8), but I still get the mpirun error. 
My ulimit now reads:

$ ulimit -a
core file size  (blocks, -c) 0
data seg size   (kbytes, -d) unlimited
scheduling priority (-e) 0
file size   (blocks, -f) unlimited
pending signals (-i) 8271027
max locked memory   (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files  (-n) 32768
pipe size(512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority  (-r) 0
stack size  (kbytes, -s) unlimited
cpu time   (seconds, -t) unlimited
max user processes  (-u) 8192
virtual memory  (kbytes, -v) unlimited
file locks  (-x) unlimited

Any other advice???


On 3/30/13 10:28 PM, Ralph Castain wrote:

FWIW: there is an MCA param that helps with such problems:

 opal_set_max_sys_limits
  "Set to non-zero to automatically set any system-imposed limits to 
the maximum allowed",

At the moment, it only sets the limits on number of files open, and max size of 
a file we can create. Easy enough to add the stack size, though as someone 
pointed out, it has some negatives as well.


On Mar 30, 2013, at 7:35 AM, Gustavo Correa  wrote:


On Mar 30, 2013, at 10:02 AM, Duke Nguyen wrote:


On 3/30/13 8:20 PM, Reuti wrote:

Am 30.03.2013 um 13:26 schrieb Tim Prince:


On 03/30/2013 06:36 AM, Duke Nguyen wrote:

On 3/30/13 5:22 PM, Duke Nguyen wrote:

On 3/30/13 3:13 PM, Patrick Bégou wrote:

I do not know about your code but:

1) did you check stack limitations ? Typically intel fortran codes needs large 
amount of stack when the problem size increase.
Check ulimit -a

First time I heard of stack limitations. Anyway, ulimit -a gives

$ ulimit -a
core file size  (blocks, -c) 0
data seg size   (kbytes, -d) unlimited
scheduling priority (-e) 0
file size   (blocks, -f) unlimited
pending signals (-i) 127368
max locked memory   (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files  (-n) 1024
pipe size(512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority  (-r) 0
stack size  (kbytes, -s) 10240
cpu time   (seconds, -t) unlimited
max user processes  (-u) 1024
virtual memory  (kbytes, -v) unlimited
file locks  (-x) unlimited

So stack size is 10MB??? Does this one create problem? How do I change this?

I did $ ulimit -s unlimited to have stack size to be unlimited, and the job ran 
fine!!! So it looks like stack limit is the problem. Questions are:

* how do I set this automatically (and permanently)?
* should I set all other ulimits to be unlimited?


In our environment, the only solution we found is to have mpirun run a script 
on each node which sets ulimit (as well as environment variables which are more 
convenient to set there than in the mpirun), before starting the executable.  
We had expert recommendations against this but no other working solution.  It 
seems unlikely that you would want to remove any limits which work at default.
Stack siz

Re: [OMPI users] memory per core/process

[Duke Nguyen]

On 4/2/13 6:42 PM, Reuti wrote:

/usr/local/bin/mpirun -npernode 1 -tag-output  sh -c "ulimit -a"

You are right :)

$ /usr/local/bin/mpirun -npernode 1 -tag-output  sh -c "ulimit -a"
[1,0]:core file size  (blocks, -c) 0
[1,0]:data seg size   (kbytes, -d) unlimited
[1,0]:scheduling priority (-e) 0
[1,0]:file size   (blocks, -f) unlimited
[1,0]:pending signals (-i) 8271027
[1,0]:max locked memory   (kbytes, -l) unlimited
[1,0]:max memory size (kbytes, -m) unlimited
[1,0]:open files  (-n) 32768
[1,0]:pipe size(512 bytes, -p) 8
[1,0]:POSIX message queues (bytes, -q) 819200
[1,0]:real-time priority  (-r) 0
[1,0]:stack size  (kbytes, -s) unlimited
[1,0]:cpu time   (seconds, -t) unlimited
[1,0]:max user processes  (-u) 8192
[1,0]:virtual memory  (kbytes, -v) unlimited
[1,0]:file locks  (-x) unlimited
[1,1]:core file size  (blocks, -c) 0
[1,1]:data seg size   (kbytes, -d) unlimited
[1,1]:scheduling priority (-e) 0
[1,1]:file size   (blocks, -f) unlimited
[1,1]:pending signals (-i) 8271027
[1,1]:max locked memory   (kbytes, -l) unlimited
[1,1]:max memory size (kbytes, -m) unlimited
[1,1]:open files  (-n) 32768
[1,1]:pipe size(512 bytes, -p) 8
[1,1]:POSIX message queues (bytes, -q) 819200
[1,1]:real-time priority  (-r) 0
[1,1]:stack size  (kbytes, -s) unlimited
[1,1]:cpu time   (seconds, -t) unlimited
[1,1]:max user processes  (-u) 8192
[1,1]:virtual memory  (kbytes, -v) unlimited
[1,1]:file locks  (-x) unlimited
[1,2]:core file size  (blocks, -c) 0
[1,2]:data seg size   (kbytes, -d) unlimited
[1,2]:scheduling priority (-e) 0
[1,2]:file size   (blocks, -f) unlimited
[1,2]:pending signals (-i) 8271027
[1,2]:max locked memory   (kbytes, -l) unlimited
[1,2]:max memory size (kbytes, -m) unlimited
[1,2]:open files  (-n) 32768
[1,2]:pipe size(512 bytes, -p) 8
[1,2]:POSIX message queues (bytes, -q) 819200
[1,2]:real-time priority  (-r) 0
[1,2]:stack size  (kbytes, -s) unlimited
[1,2]:cpu time   (seconds, -t) unlimited
[1,2]:max user processes  (-u) 8192
[1,2]:virtual memory  (kbytes, -v) unlimited
[1,2]:file locks  (-x) unlimited
[1,3]:core file size  (blocks, -c) 0
[1,3]:data seg size   (kbytes, -d) unlimited
[1,3]:scheduling priority (-e) 0
[1,3]:file size   (blocks, -f) unlimited
[1,3]:pending signals (-i) 8271027
[1,3]:max locked memory   (kbytes, -l) unlimited
[1,3]:max memory size (kbytes, -m) unlimited
[1,3]:open files  (-n) 32768
[1,3]:pipe size(512 bytes, -p) 8
[1,3]:POSIX message queues (bytes, -q) 819200
[1,3]:real-time priority  (-r) 0
[1,3]:stack size  (kbytes, -s) unlimited
[1,3]:cpu time   (seconds, -t) unlimited
[1,3]:max user processes  (-u) 8192
[1,3]:virtual memory  (kbytes, -v) unlimited
[1,3]:file locks  (-x) unlimited

Re: [OMPI users] memory per core/process

[Duke Nguyen]

On 4/2/13 6:50 PM, Reuti wrote:

Hi,

Am 30.03.2013 um 14:46 schrieb Patrick Bégou:


Ok, so your problem is identified as a stack size problem. I went into these 
limitations using Intel fortran compilers on large data problems.

First, it seems you can increase your stack size as "ulimit -s unlimited" works 
(you didn't enforce the system hard limit). The best way  is to set this setting in your 
.bashrc file so it will works on every node.
But setting it to unlimited may not be really safe. IE, if you got in a badly 
coded recursive function calling itself without a stop condition you can 
request all the system memory and crash the node. So set a large but limited 
value, it's safer.

I'm managing a cluster and I always set a maximum value to stack size. I also 
limit the memory available for each core for system stability. If a user 
request only one of the 12 cores of a node he can only access 1/12 of the node 
memory amount. If he needs more memory he has to request 2 cores, even if he 
uses a sequential code. This avoid crashing jobs of other users on the same 
node with memory requirements. But this is not configured on your node.

This is one way to implement memory limits as a policy - it's up to the user to 
request the correct number of cores then although he wants to run a serial job 
only. Personally I prefer that the user specifies the requested memory in such 
a case. It's up to the queuingsystem then to avoid that additional jobs are 
scheduled to a machine unless the remaining memory is sufficient for their 
execution in such a situation.


We use Torque/Maui and I want to do similar with Torque/Maui (still 
learning - those, together with openmpi are new to me). Unfortunately 
posting to Torque/Maui forums are somehow too difficult (my posts were 
moderated since I am the newcomer, but it seems nobody is managing those 
forums so my posts were never able to get through...). I wish they were 
as active as this forum...


D.



-- Reuti



Duke Nguyen a écrit :

On 3/30/13 3:13 PM, Patrick Bégou wrote:

I do not know about your code but:

1) did you check stack limitations ? Typically intel fortran codes needs large 
amount of stack when the problem size increase.
Check ulimit -a

First time I heard of stack limitations. Anyway, ulimit -a gives

$ ulimit -a
core file size  (blocks, -c) 0
data seg size   (kbytes, -d) unlimited
scheduling priority (-e) 0
file size   (blocks, -f) unlimited
pending signals (-i) 127368
max locked memory   (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files  (-n) 1024
pipe size(512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority  (-r) 0
stack size  (kbytes, -s) 10240
cpu time   (seconds, -t) unlimited
max user processes  (-u) 1024
virtual memory  (kbytes, -v) unlimited
file locks  (-x) unlimited

So stack size is 10MB??? Does this one create problem? How do I change this?


2) did your node uses cpuset and memory limitation like fake numa to set the 
maximum amount of memory available for a job ?

Not really understand (also first time heard of fake numa), but I am pretty 
sure we do not have such things. The server I tried was a dedicated server with 
2 x5420 and 16GB physical memory.


Patrick

Duke Nguyen a écrit :

Hi folks,

I am sorry if this question had been asked before, but after ten days of 
searching/working on the system, I surrender :(. We try to use mpirun to run 
abinit (abinit.org) which in turns will call an input file to run some 
simulation. The command to run is pretty simple

$ mpirun -np 4 /opt/apps/abinit/bin/abinit < input.files >& output.log

We ran this command on a server with two quad core x5420 and 16GB of memory. I 
called only 4 core, and I guess in theory each of the core should take up to 
2GB each.

In the output of the log, there is something about memory:

P This job should need less than 717.175 Mbytes of memory.
  Rough estimation (10% accuracy) of disk space for files :
  WF disk file : 69.524 Mbytes ; DEN or POT disk file : 14.240 Mbytes.

So basically it reported that the above job should not take more than 718MB 
each core.

But I still have the Segmentation Fault error:

mpirun noticed that process rank 0 with PID 16099 on node biobos exited on 
signal 11 (Segmentation fault).

The system already has limits up to unlimited:

$ cat /etc/security/limits.conf | grep -v '#'
* soft memlock unlimited
* hard memlock unlimited

I also tried to run

$ ulimit -l unlimited

before the mpirun command above, but it did not help at all.

If we adjust the parameters of the input.files to give the reported mem per 
core is less than 512MB, then the job runs fine.

Please help,

Thanks,

D.


___

Re: [OMPI users] memory per core/process

[Duke Nguyen]

On 3/30/13 8:46 PM, Patrick Bégou wrote:
Ok, so your problem is identified as a stack size problem. I went into 
these limitations using Intel fortran compilers on large data problems.


First, it seems you can increase your stack size as "ulimit -s 
unlimited" works (you didn't enforce the system hard limit). The best 
way  is to set this setting in your .bashrc file so it will works on 
every node.
But setting it to unlimited may not be really safe. IE, if you got in 
a badly coded recursive function calling itself without a stop 
condition you can request all the system memory and crash the node. So 
set a large but limited value, it's safer.




Now I feel the pain you mentioned :). With -s unlimited now some of our 
nodes are easily down (completely) and needed to be hard reset!!! 
(whereas we never had any node down like that before even with the 
killed or badly coded jobs).


Looking for a safer number of ulimit -s other than "unlimited" now... :(

I'm managing a cluster and I always set a maximum value to stack size. 
I also limit the memory available for each core for system stability. 
If a user request only one of the 12 cores of a node he can only 
access 1/12 of the node memory amount. If he needs more memory he has 
to request 2 cores, even if he uses a sequential code. This avoid 
crashing jobs of other users on the same node with memory 
requirements. But this is not configured on your node.


Duke Nguyen a écrit :

On 3/30/13 3:13 PM, Patrick Bégou wrote:

I do not know about your code but:

1) did you check stack limitations ? Typically intel fortran codes 
needs large amount of stack when the problem size increase.

Check ulimit -a


First time I heard of stack limitations. Anyway, ulimit -a gives

$ ulimit -a
core file size  (blocks, -c) 0
data seg size   (kbytes, -d) unlimited
scheduling priority (-e) 0
file size   (blocks, -f) unlimited
pending signals (-i) 127368
max locked memory   (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files  (-n) 1024
pipe size(512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority  (-r) 0
stack size  (kbytes, -s) 10240
cpu time   (seconds, -t) unlimited
max user processes  (-u) 1024
virtual memory  (kbytes, -v) unlimited
file locks  (-x) unlimited

So stack size is 10MB??? Does this one create problem? How do I 
change this?




2) did your node uses cpuset and memory limitation like fake numa to 
set the maximum amount of memory available for a job ?


Not really understand (also first time heard of fake numa), but I am 
pretty sure we do not have such things. The server I tried was a 
dedicated server with 2 x5420 and 16GB physical memory.




Patrick

Duke Nguyen a écrit :

Hi folks,

I am sorry if this question had been asked before, but after ten 
days of searching/working on the system, I surrender :(. We try to 
use mpirun to run abinit (abinit.org) which in turns will call an 
input file to run some simulation. The command to run is pretty simple


$ mpirun -np 4 /opt/apps/abinit/bin/abinit < input.files >& output.log

We ran this command on a server with two quad core x5420 and 16GB 
of memory. I called only 4 core, and I guess in theory each of the 
core should take up to 2GB each.


In the output of the log, there is something about memory:

P This job should need less than 717.175 Mbytes 
of memory.

  Rough estimation (10% accuracy) of disk space for files :
  WF disk file : 69.524 Mbytes ; DEN or POT disk file : 14.240 
Mbytes.


So basically it reported that the above job should not take more 
than 718MB each core.


But I still have the Segmentation Fault error:

mpirun noticed that process rank 0 with PID 16099 on node biobos 
exited on signal 11 (Segmentation fault).


The system already has limits up to unlimited:

$ cat /etc/security/limits.conf | grep -v '#'
* soft memlock unlimited
* hard memlock unlimited

I also tried to run

$ ulimit -l unlimited

before the mpirun command above, but it did not help at all.

If we adjust the parameters of the input.files to give the reported 
mem per core is less than 512MB, then the job runs fine.


Please help,

Thanks,

D.


___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users


___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users



___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users



___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] memory per core/process

[Duke Nguyen]

On 4/2/13 10:45 PM, Ralph Castain wrote:

Hmmm...tell you what. I'll add the ability for OMPI to set the limit to a 
user-specified level upon launch of each process. This will give you some 
protection and flexibility.


That would be excellent ;)



I forget, so please forgive the old man's fading memory - what version of OMPI 
are you using? I'll backport a patch for you.


It's openmpi-1.6.3-x86_64, if that helps...



On Apr 2, 2013, at 8:40 AM, Duke Nguyen  wrote:


On 3/30/13 8:46 PM, Patrick Bégou wrote:

Ok, so your problem is identified as a stack size problem. I went into these 
limitations using Intel fortran compilers on large data problems.

First, it seems you can increase your stack size as "ulimit -s unlimited" works 
(you didn't enforce the system hard limit). The best way  is to set this setting in your 
.bashrc file so it will works on every node.
But setting it to unlimited may not be really safe. IE, if you got in a badly 
coded recursive function calling itself without a stop condition you can 
request all the system memory and crash the node. So set a large but limited 
value, it's safer.


Now I feel the pain you mentioned :). With -s unlimited now some of our nodes 
are easily down (completely) and needed to be hard reset!!! (whereas we never 
had any node down like that before even with the killed or badly coded jobs).

Looking for a safer number of ulimit -s other than "unlimited" now... :(


I'm managing a cluster and I always set a maximum value to stack size. I also 
limit the memory available for each core for system stability. If a user 
request only one of the 12 cores of a node he can only access 1/12 of the node 
memory amount. If he needs more memory he has to request 2 cores, even if he 
uses a sequential code. This avoid crashing jobs of other users on the same 
node with memory requirements. But this is not configured on your node.

Duke Nguyen a écrit :

On 3/30/13 3:13 PM, Patrick Bégou wrote:

I do not know about your code but:

1) did you check stack limitations ? Typically intel fortran codes needs large 
amount of stack when the problem size increase.
Check ulimit -a

First time I heard of stack limitations. Anyway, ulimit -a gives

$ ulimit -a
core file size  (blocks, -c) 0
data seg size   (kbytes, -d) unlimited
scheduling priority (-e) 0
file size   (blocks, -f) unlimited
pending signals (-i) 127368
max locked memory   (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files  (-n) 1024
pipe size(512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority  (-r) 0
stack size  (kbytes, -s) 10240
cpu time   (seconds, -t) unlimited
max user processes  (-u) 1024
virtual memory  (kbytes, -v) unlimited
file locks  (-x) unlimited

So stack size is 10MB??? Does this one create problem? How do I change this?


2) did your node uses cpuset and memory limitation like fake numa to set the 
maximum amount of memory available for a job ?

Not really understand (also first time heard of fake numa), but I am pretty 
sure we do not have such things. The server I tried was a dedicated server with 
2 x5420 and 16GB physical memory.


Patrick

Duke Nguyen a écrit :

Hi folks,

I am sorry if this question had been asked before, but after ten days of 
searching/working on the system, I surrender :(. We try to use mpirun to run 
abinit (abinit.org) which in turns will call an input file to run some 
simulation. The command to run is pretty simple

$ mpirun -np 4 /opt/apps/abinit/bin/abinit < input.files >& output.log

We ran this command on a server with two quad core x5420 and 16GB of memory. I 
called only 4 core, and I guess in theory each of the core should take up to 
2GB each.

In the output of the log, there is something about memory:

P This job should need less than 717.175 Mbytes of memory.
  Rough estimation (10% accuracy) of disk space for files :
  WF disk file : 69.524 Mbytes ; DEN or POT disk file : 14.240 Mbytes.

So basically it reported that the above job should not take more than 718MB 
each core.

But I still have the Segmentation Fault error:

mpirun noticed that process rank 0 with PID 16099 on node biobos exited on 
signal 11 (Segmentation fault).

The system already has limits up to unlimited:

$ cat /etc/security/limits.conf | grep -v '#'
* soft memlock unlimited
* hard memlock unlimited

I also tried to run

$ ulimit -l unlimited

before the mpirun command above, but it did not help at all.

If we adjust the parameters of the input.files to give the reported mem per 
core is less than 512MB, then the job runs fine.

Please help,

Thanks,

D.


___
users mailing

Re: [OMPI users] memory per core/process

[Duke Nguyen]

On 4/2/13 11:03 PM, Gus Correa wrote:

On 04/02/2013 11:40 AM, Duke Nguyen wrote:

On 3/30/13 8:46 PM, Patrick Bégou wrote:

Ok, so your problem is identified as a stack size problem. I went into
these limitations using Intel fortran compilers on large data problems.

First, it seems you can increase your stack size as "ulimit -s
unlimited" works (you didn't enforce the system hard limit). The best
way is to set this setting in your .bashrc file so it will works on
every node.
But setting it to unlimited may not be really safe. IE, if you got in
a badly coded recursive function calling itself without a stop
condition you can request all the system memory and crash the node. So
set a large but limited value, it's safer.



Now I feel the pain you mentioned :). With -s unlimited now some of our
nodes are easily down (completely) and needed to be hard reset!!!
(whereas we never had any node down like that before even with the
killed or badly coded jobs).

Looking for a safer number of ulimit -s other than "unlimited" now... :(



In my opinion this is a trade off between who feels the pain.
It can be you (sys admin) feeling the pain of having
to power up offline nodes,
or it could be the user feeling the pain for having
her/his code killed by segmentation fault due to small memory
available for the stack.


... in case that user is at a large institute that promises to provide 
best service, unlimited resources/unlimited *everything* to end users. 
If not, user should really think of how to make use the best of 
available resources. Unfortunately many (most?) end users don't.



There is only so much that can be done to make everybody happy.


So true... especially HPC resource is still luxurious here in Vietnam, 
and we have a quite small (and not-so-strong) cluster.



If you share the nodes among jobs, you could set the
stack size limit to
some part of the physical_memory divided by the number_of_cores,
saving some memory for the OS etc beforehand.
However, this can be a straitjacket for jobs that could run with
a bit more memory, and won't because of this limit.
If you do not share the nodes, then you could make stacksize
closer to physical memory.


Great. Thanks for this advice Gus.



Anyway, this is less of an OpenMPI than of a
resource manager / queuing system conversation.


Yeah, and I have learned a lot other than just openmpi stuffs here :)



Best,
Gus Correa


I'm managing a cluster and I always set a maximum value to stack size.
I also limit the memory available for each core for system stability.
If a user request only one of the 12 cores of a node he can only
access 1/12 of the node memory amount. If he needs more memory he has
to request 2 cores, even if he uses a sequential code. This avoid
crashing jobs of other users on the same node with memory
requirements. But this is not configured on your node.

Duke Nguyen a écrit :

On 3/30/13 3:13 PM, Patrick Bégou wrote:

I do not know about your code but:

1) did you check stack limitations ? Typically intel fortran codes
needs large amount of stack when the problem size increase.
Check ulimit -a


First time I heard of stack limitations. Anyway, ulimit -a gives

$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 127368
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 1024
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

So stack size is 10MB??? Does this one create problem? How do I
change this?



2) did your node uses cpuset and memory limitation like fake numa to
set the maximum amount of memory available for a job ?


Not really understand (also first time heard of fake numa), but I am
pretty sure we do not have such things. The server I tried was a
dedicated server with 2 x5420 and 16GB physical memory.



Patrick

Duke Nguyen a écrit :

Hi folks,

I am sorry if this question had been asked before, but after ten
days of searching/working on the system, I surrender :(. We try to
use mpirun to run abinit (abinit.org) which in turns will call an
input file to run some simulation. The command to run is pretty 
simple


$ mpirun -np 4 /opt/apps/abinit/bin/abinit < input.files >& 
output.log


We ran this command on a server with two quad core x5420 and 16GB
of memory. I called only 4 core, and I guess in theory each of the
core should take up to 2GB each.

In the output of the log, there is something about memory:

P This job should need less than 717.175 Mbytes of memory.
Rough estimation (10% accuracy) of disk space for files :
WF disk file : 69.524 Mbytes ; DEN or POT disk file : 14.240 Mbytes.

So basically it reported that the abov

[OMPI users] running openmpi with specified lib path

[Duke Nguyen]

Hi folks,

I am testing our cluster with module environment, and am having a
headache to understand openmpi 1.7.2!!! So our system currently has
openmpi 1.6.3 (at default location /usr/local), 1.6.4 and 1.7.2 compiled
with intel compilers (installed at /opt/apps). In order to use openmpi
1.7.2 for example, I tried:

$ module load mpi/openmpi-1.7.2_composer_xe_2013.3.163
$ module load
apps/abinit-7.2.1_composer_xe_2013.3.163_openmpi-1.7.2_intel_fftw3-mkl
$ mpirun ./mpihello_intel
mca: base: component_find: unable to open
/usr/local/lib/openmpi/mca_ess_hnp:
/usr/local/lib/openmpi/mca_ess_hnp.so: undefined symbol:
orte_local_jobdata (ignored)
mca: base: component_find: unable to open
/usr/local/lib/openmpi/mca_ess_slurm:
/usr/local/lib/openmpi/mca_ess_slurm.so: undefined symbol:
orte_orted_exit_with_barrier (ignored)
mca: base: component_find: unable to open
/usr/local/lib/openmpi/mca_ess_slurmd:
/usr/local/lib/openmpi/mca_ess_slurmd.so: undefined symbol:
orte_pmap_t_class (ignored)
mpirun: symbol lookup error:
/usr/local/lib/openmpi/mca_ess_singleton.so: undefined symbol:
orte_util_setup_local_nidmap_entries

To investigate the error, I tried:

$ echo $LD_LIBRARY_PATH
/opt/apps/abinit/abinit-7.2.1_composer_xe_2013.3.163_openmpi-1.7.2_intel_fftw3-mkl/usr/local/lib:/opt/apps/openmpi/openmpi-1.7.2r28341_composer_xe_2013.3.163/usr/local/lib/openmpi:/opt/apps/openmpi/openmpi-1.7.2r28341_composer_xe_2013.3.163/usr/local/lib:/usr/local/lib:/opt/intel/lib/intel64:/opt/intel/mkl/lib/intel64
$ ls
/opt/apps/openmpi/openmpi-1.7.2r28341_composer_xe_2013.3.163/usr/local/lib/openmpi
| grep mca_ess
mca_ess_env.la
mca_ess_env.so
mca_ess_hnp.la
mca_ess_hnp.so
mca_ess_singleton.la
mca_ess_singleton.so
mca_ess_slurm.la
mca_ess_slurm.so
mca_ess_tm.la
mca_ess_tm.so
mca_ess_tool.la
mca_ess_tool.so

So apparently openmpi 1.7.2 looks for the old library at
/usr/loca/lib/openmpi for 1.6.3 instead of at
/opt/apps/openmpi/openmpi-1.7.2r28341_composer_xe_2013.3.163/usr/local/lib/openmpi.
Is there away to force openmpi 1.7.2 look at
/opt/apps/openmpi/openmpi-1.7.2r28341_composer_xe_2013.3.163/usr/local/lib/openmpi
first before looking at other locations?

Thanks,

D.




signature.asc
Description: OpenPGP digital signature

Re: [OMPI users] running openmpi with specified lib path

[Duke Nguyen]

On 5/7/13 7:02 PM, Reuti wrote:
> Hi,
> 
> Am 07.05.2013 um 13:36 schrieb Duke Nguyen:
> 
>> I am testing our cluster with module environment, and am having a
>> headache to understand openmpi 1.7.2!!! So our system currently has
>> openmpi 1.6.3 (at default location /usr/local), 1.6.4 and 1.7.2 compiled
>> with intel compilers (installed at /opt/apps). In order to use openmpi
>> 1.7.2 for example, I tried:
>>
>> $ module load mpi/openmpi-1.7.2_composer_xe_2013.3.163
>> $ module load
>> apps/abinit-7.2.1_composer_xe_2013.3.163_openmpi-1.7.2_intel_fftw3-mkl
>> $ mpirun ./mpihello_intel
> 
> What does:
> 
> $ which mpirun


$ which mpirun
/opt/apps/openmpi/openmpi-1.7.2r28341_composer_xe_2013.3.163/usr/local/bin/mpirun

> 
> show? Did you also recompile your application?


Yes, I did. That is why abinit has that long's name when I call it using
module
(abinit-7.2.1_composer_xe_2013.3.163_openmpi-1.7.2_intel_fftw3-mkl ->
abinit compiled with intel xe 2013.3.163 and openmpi 1.7.2 and
fftw3-mkl; also openmpi 1.7.2 was compiled with intel xe 2013.3.163).

> 
> -- Reuti
> 
> 
>> mca: base: component_find: unable to open
>> /usr/local/lib/openmpi/mca_ess_hnp:
>> /usr/local/lib/openmpi/mca_ess_hnp.so: undefined symbol:
>> orte_local_jobdata (ignored)
>> mca: base: component_find: unable to open
>> /usr/local/lib/openmpi/mca_ess_slurm:
>> /usr/local/lib/openmpi/mca_ess_slurm.so: undefined symbol:
>> orte_orted_exit_with_barrier (ignored)
>> mca: base: component_find: unable to open
>> /usr/local/lib/openmpi/mca_ess_slurmd:
>> /usr/local/lib/openmpi/mca_ess_slurmd.so: undefined symbol:
>> orte_pmap_t_class (ignored)
>> mpirun: symbol lookup error:
>> /usr/local/lib/openmpi/mca_ess_singleton.so: undefined symbol:
>> orte_util_setup_local_nidmap_entries
>>
>> To investigate the error, I tried:
>>
>> $ echo $LD_LIBRARY_PATH
>> /opt/apps/abinit/abinit-7.2.1_composer_xe_2013.3.163_openmpi-1.7.2_intel_fftw3-mkl/usr/local/lib:/opt/apps/openmpi/openmpi-1.7.2r28341_composer_xe_2013.3.163/usr/local/lib/openmpi:/opt/apps/openmpi/openmpi-1.7.2r28341_composer_xe_2013.3.163/usr/local/lib:/usr/local/lib:/opt/intel/lib/intel64:/opt/intel/mkl/lib/intel64
>> $ ls
>> /opt/apps/openmpi/openmpi-1.7.2r28341_composer_xe_2013.3.163/usr/local/lib/openmpi
>> | grep mca_ess
>> mca_ess_env.la
>> mca_ess_env.so
>> mca_ess_hnp.la
>> mca_ess_hnp.so
>> mca_ess_singleton.la
>> mca_ess_singleton.so
>> mca_ess_slurm.la
>> mca_ess_slurm.so
>> mca_ess_tm.la
>> mca_ess_tm.so
>> mca_ess_tool.la
>> mca_ess_tool.so
>>
>> So apparently openmpi 1.7.2 looks for the old library at
>> /usr/loca/lib/openmpi for 1.6.3 instead of at
>> /opt/apps/openmpi/openmpi-1.7.2r28341_composer_xe_2013.3.163/usr/local/lib/openmpi.
>> Is there away to force openmpi 1.7.2 look at
>> /opt/apps/openmpi/openmpi-1.7.2r28341_composer_xe_2013.3.163/usr/local/lib/openmpi
>> first before looking at other locations?
>>
>> Thanks,
>>
>> D.
>>
>>
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> 
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
> 




signature.asc
Description: OpenPGP digital signature

Re: [OMPI users] running openmpi with specified lib path

[Duke Nguyen]

On 5/7/13 8:10 PM, Jeff Squyres (jsquyres) wrote:
> On May 7, 2013, at 7:36 AM, Duke Nguyen  wrote:
> 
>> So apparently openmpi 1.7.2 looks for the old library at
>> /usr/loca/lib/openmpi for 1.6.3 instead of at
>> /opt/apps/openmpi/openmpi-1.7.2r28341_composer_xe_2013.3.163/usr/local/lib/openmpi.
>> Is there away to force openmpi 1.7.2 look at
>> /opt/apps/openmpi/openmpi-1.7.2r28341_composer_xe_2013.3.163/usr/local/lib/openmpi
>> first before looking at other locations?
> 
> 
> This is *usually* a problem of local settings.  Make sure you don't have both 
> the 1.6 and 1.7 library directories listed in LD_LIBRARY_PATH, for example.
> 

I also thought of that, but removing /usr/local does not help.

$ echo $PATH
/usr/lib64/qt-3.3/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/apps/abinit/bin:/opt/torque/sbin:/opt/torque/bin:/usr/local/bin:/opt/apps/bin
$ export PATH=/usr/lib64/qt-3.3/bin:/bin:/usr/bin:/usr/sbin:/sbin
$ echo $LD_LIBRARY_PATH
:/usr/local/lib:/opt/intel/lib/intel64:/opt/intel/mkl/lib/intel64
$ export LD_LIBRARY_PATH=
$ module load mpi/openmpi-1.7.2_composer_xe_2013.3.163
$ module load
apps/abinit-7.2.1_composer_xe_2013.3.163_openmpi-1.7.2_intel_fftw3-mkl
$ mpirun ./mpihello_intel
mpirun: error while loading shared libraries: libimf.so: cannot open
shared object file: No such file or directory
$ echo $LD_LIBRARY_PATH
/opt/apps/abinit/abinit-7.2.1_composer_xe_2013.3.163_openmpi-1.7.2_intel_fftw3-mkl/usr/local/lib:/opt/apps/openmpi/openmpi-1.7.2r28341_composer_xe_2013.3.163/usr/local/lib/openmpi:/opt/apps/openmpi/openmpi-1.7.2r28341_composer_xe_2013.3.163/usr/local/lib
$ export
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/opt/intel/composer_xe_2013/lib/intel64
$ mpirun ./mpihello_intel
[node0111.biobos.grih.org:18979] mca: base: component_find: unable to
open /usr/local/lib/openmpi/mca_ess_hnp:
/usr/local/lib/openmpi/mca_ess_hnp.so: undefined symbol:
orte_local_jobdata (ignored)
[node0111.biobos.grih.org:18979] mca: base: component_find: unable to
open /usr/local/lib/openmpi/mca_ess_slurm:
/usr/local/lib/openmpi/mca_ess_slurm.so: undefined symbol:
orte_orted_exit_with_barrier (ignored)
[node0111.biobos.grih.org:18979] mca: base: component_find: unable to
open /usr/local/lib/openmpi/mca_ess_slurmd:
/usr/local/lib/openmpi/mca_ess_slurmd.so: undefined symbol:
orte_pmap_t_class (ignored)
mpirun: symbol lookup error:
/usr/local/lib/openmpi/mca_ess_singleton.so: undefined symbol:
orte_util_setup_local_nidmap_entries



signature.asc
Description: OpenPGP digital signature

Re: [OMPI users] running openmpi with specified lib path

[Duke Nguyen]

On 5/7/13 9:07 PM, Ralph Castain wrote:
> Is there a typo here?
> 
> /opt/apps/abinit/abinit-7.2.1_composer_xe_2013.3.163_openmpi-1.7.2_intel_fftw3-mkl/usr/local/lib:/opt/apps/openmpi/openmpi-1.7.2r28341_composer_xe_2013.3.163/usr/local/lib/openmpi:/opt/apps/openmpi/openmpi-1.7.2r28341_composer_xe_2013.3.163/usr/local/lib:/usr/local/lib:/opt/intel/lib/intel64:/opt/intel/mkl/lib/intel64
> 
> I don't see a colon after the initial /opt/apps path - looks like it is 
> missing


I dont have any lib installed at /opt/apps. All libs with full
appropriate paths will be loaded once the module is called.

D.


> 
> 
> On May 7, 2013, at 7:01 AM, Duke Nguyen  wrote:
> 
>> On 5/7/13 7:02 PM, Reuti wrote:
>>> Hi,
>>>
>>> Am 07.05.2013 um 13:36 schrieb Duke Nguyen:
>>>
>>>> I am testing our cluster with module environment, and am having a
>>>> headache to understand openmpi 1.7.2!!! So our system currently has
>>>> openmpi 1.6.3 (at default location /usr/local), 1.6.4 and 1.7.2 compiled
>>>> with intel compilers (installed at /opt/apps). In order to use openmpi
>>>> 1.7.2 for example, I tried:
>>>>
>>>> $ module load mpi/openmpi-1.7.2_composer_xe_2013.3.163
>>>> $ module load
>>>> apps/abinit-7.2.1_composer_xe_2013.3.163_openmpi-1.7.2_intel_fftw3-mkl
>>>> $ mpirun ./mpihello_intel
>>>
>>> What does:
>>>
>>> $ which mpirun
>>
>>
>> $ which mpirun
>> /opt/apps/openmpi/openmpi-1.7.2r28341_composer_xe_2013.3.163/usr/local/bin/mpirun
>>
>>>
>>> show? Did you also recompile your application?
>>
>>
>> Yes, I did. That is why abinit has that long's name when I call it using
>> module
>> (abinit-7.2.1_composer_xe_2013.3.163_openmpi-1.7.2_intel_fftw3-mkl ->
>> abinit compiled with intel xe 2013.3.163 and openmpi 1.7.2 and
>> fftw3-mkl; also openmpi 1.7.2 was compiled with intel xe 2013.3.163).
>>
>>>
>>> -- Reuti
>>>
>>>
>>>> mca: base: component_find: unable to open
>>>> /usr/local/lib/openmpi/mca_ess_hnp:
>>>> /usr/local/lib/openmpi/mca_ess_hnp.so: undefined symbol:
>>>> orte_local_jobdata (ignored)
>>>> mca: base: component_find: unable to open
>>>> /usr/local/lib/openmpi/mca_ess_slurm:
>>>> /usr/local/lib/openmpi/mca_ess_slurm.so: undefined symbol:
>>>> orte_orted_exit_with_barrier (ignored)
>>>> mca: base: component_find: unable to open
>>>> /usr/local/lib/openmpi/mca_ess_slurmd:
>>>> /usr/local/lib/openmpi/mca_ess_slurmd.so: undefined symbol:
>>>> orte_pmap_t_class (ignored)
>>>> mpirun: symbol lookup error:
>>>> /usr/local/lib/openmpi/mca_ess_singleton.so: undefined symbol:
>>>> orte_util_setup_local_nidmap_entries
>>>>
>>>> To investigate the error, I tried:
>>>>
>>>> $ echo $LD_LIBRARY_PATH
>>>> /opt/apps/abinit/abinit-7.2.1_composer_xe_2013.3.163_openmpi-1.7.2_intel_fftw3-mkl/usr/local/lib:/opt/apps/openmpi/openmpi-1.7.2r28341_composer_xe_2013.3.163/usr/local/lib/openmpi:/opt/apps/openmpi/openmpi-1.7.2r28341_composer_xe_2013.3.163/usr/local/lib:/usr/local/lib:/opt/intel/lib/intel64:/opt/intel/mkl/lib/intel64
>>>> $ ls
>>>> /opt/apps/openmpi/openmpi-1.7.2r28341_composer_xe_2013.3.163/usr/local/lib/openmpi
>>>> | grep mca_ess
>>>> mca_ess_env.la
>>>> mca_ess_env.so
>>>> mca_ess_hnp.la
>>>> mca_ess_hnp.so
>>>> mca_ess_singleton.la
>>>> mca_ess_singleton.so
>>>> mca_ess_slurm.la
>>>> mca_ess_slurm.so
>>>> mca_ess_tm.la
>>>> mca_ess_tm.so
>>>> mca_ess_tool.la
>>>> mca_ess_tool.so
>>>>
>>>> So apparently openmpi 1.7.2 looks for the old library at
>>>> /usr/loca/lib/openmpi for 1.6.3 instead of at
>>>> /opt/apps/openmpi/openmpi-1.7.2r28341_composer_xe_2013.3.163/usr/local/lib/openmpi.
>>>> Is there away to force openmpi 1.7.2 look at
>>>> /opt/apps/openmpi/openmpi-1.7.2r28341_composer_xe_2013.3.163/usr/local/lib/openmpi
>>>> first before looking at other locations?
>>>>
>>>> Thanks,
>>>>
>>>> D.
>>>>
>>>>
>>>> ___
>>>> users mailing list
>>>> us...@open-mpi.org
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>
>>>
>>> ___
>>> users mailing list
>>> us...@open-mpi.org
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>
>>
>>
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> 
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
> 




signature.asc
Description: OpenPGP digital signature

Re: [OMPI users] running openmpi with specified lib path

[Duke Nguyen]

On 5/7/13 9:25 PM, Ralph Castain wrote:
> Look at your ld-library-path. Breaking it down, you have the following 
> elements
> 
> /opt/apps/abinit/abinit-7.2.1_composer_xe_2013.3.163_openmpi-1.7.2_intel_fftw3-mkl/usr/local/lib

$ export LD_LIBRARY_PATH=
$ module load mpi/openmpi-1.7.2_composer_xe_2013.3.163
$ module load
apps/abinit-7.2.1_composer_xe_2013.3.163_openmpi-1.7.2_intel_fftw3-mkl
$ echo $LD_LIBRARY_PATH
/opt/apps/abinit/abinit-7.2.1_composer_xe_2013.3.163_openmpi-1.7.2_intel_fftw3-mkl/usr/local/lib:/opt/apps/openmpi/openmpi-1.7.2r28341_composer_xe_2013.3.163/usr/local/lib/openmpi:/opt/apps/openmpi/openmpi-1.7.2r28341_composer_xe_2013.3.163/usr/local/lib

The LD_LIBRARY_PATH above contains three different locations:

/opt/apps/abinit/abinit-7.2.1_composer_xe_2013.3.163_openmpi-1.7.2_intel_fftw3-mkl/usr/local/lib
/opt/apps/openmpi/openmpi-1.7.2r28341_composer_xe_2013.3.163/usr/local/lib/openmpi
/opt/apps/openmpi/openmpi-1.7.2r28341_composer_xe_2013.3.163/usr/local/lib


> 
> repeated three times. Did you actually configure with
> 
> --prefix=/opt/apps/openmpi/openmpi-1.7.2r28341_composer_xe_2013.3.163/usr/local


No I didnt (and that maybe the reason, I am not really sure I was
correct when installation these things). What I did was:

$ cd /opt/apps/openmpi/openmpi-1.7.2r28341_composer_xe_2013.3.163
$ ./configure CC=/opt/intel/composer_xe_2013.3.163/bin/intel64/icc
CXX=/opt/intel/composer_xe_2013.3.163/bin/intel64/icpc
F77=/opt/intel/composer_xe_2013.3.163/bin/intel64/ifort
FC=/opt/intel/composer_xe_2013.3.163/bin/intel64/ifort
$ make -j8
$ DESTDIR=`pwd` make -j8 install

so all the compiled openmpi components were installed at

$ ls /opt/apps/openmpi/openmpi-1.7.2r28341_composer_xe_2013.3.163/usr/local/
bin  etc  include  lib  share

Am I doing wrong??? Please advise.

> 
> Note that the prefix path *must* be the first element in your ld-libary-path 
> or you'll pickup one of your other installations. When you did your test, you 
> goofed:
> 
>> $ export
>> LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/opt/intel/composer_xe_2013/lib/intel64
> 
> 
> This put the new path at the end, where it would never be used


What I thought was if the needed lib or exec is not found on the first
path, then the next will be used. Since libimf was not found from all
previous path, added the last one helped. Anyway, try adding to the
beginning:

$ mpirun ./mpihello_intel
[node0111.biobos.grih.org:19243] mca: base: component_find: unable to
open /usr/local/lib/openmpi/mca_ess_hnp:
/usr/local/lib/openmpi/mca_ess_hnp.so: undefined symbol:
orte_local_jobdata (ignored)
[node0111.biobos.grih.org:19243] mca: base: component_find: unable to
open /usr/local/lib/openmpi/mca_ess_slurm:
/usr/local/lib/openmpi/mca_ess_slurm.so: undefined symbol:
orte_orted_exit_with_barrier (ignored)
[node0111.biobos.grih.org:19243] mca: base: component_find: unable to
open /usr/local/lib/openmpi/mca_ess_slurmd:
/usr/local/lib/openmpi/mca_ess_slurmd.so: undefined symbol:
orte_pmap_t_class (ignored)
mpirun: symbol lookup error:
/usr/local/lib/openmpi/mca_ess_singleton.so: undefined symbol:
orte_util_setup_local_nidmap_entries

gave the same error.


> 
> 
> On May 7, 2013, at 7:11 AM, Duke Nguyen  wrote:
> 
>> On 5/7/13 8:10 PM, Jeff Squyres (jsquyres) wrote:
>>> On May 7, 2013, at 7:36 AM, Duke Nguyen  wrote:
>>>
>>>> So apparently openmpi 1.7.2 looks for the old library at
>>>> /usr/loca/lib/openmpi for 1.6.3 instead of at
>>>> /opt/apps/openmpi/openmpi-1.7.2r28341_composer_xe_2013.3.163/usr/local/lib/openmpi.
>>>> Is there away to force openmpi 1.7.2 look at
>>>> /opt/apps/openmpi/openmpi-1.7.2r28341_composer_xe_2013.3.163/usr/local/lib/openmpi
>>>> first before looking at other locations?
>>>
>>>
>>> This is *usually* a problem of local settings.  Make sure you don't have 
>>> both the 1.6 and 1.7 library directories listed in LD_LIBRARY_PATH, for 
>>> example.
>>>
>>
>> I also thought of that, but removing /usr/local does not help.
>>
>> $ echo $PATH
>> /usr/lib64/qt-3.3/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/apps/abinit/bin:/opt/torque/sbin:/opt/torque/bin:/usr/local/bin:/opt/apps/bin
>> $ export PATH=/usr/lib64/qt-3.3/bin:/bin:/usr/bin:/usr/sbin:/sbin
>> $ echo $LD_LIBRARY_PATH
>> :/usr/local/lib:/opt/intel/lib/intel64:/opt/intel/mkl/lib/intel64
>> $ export LD_LIBRARY_PATH=
>> $ module load mpi/openmpi-1.7.2_composer_xe_2013.3.163
>> $ module load
>> apps/abinit-7.2.1_composer_xe_2013.3.163_openmpi-1.7.2_intel_fftw3-mkl
>> $ mpirun ./mpihello_intel
>> mpirun: error while loading shared libraries: libimf.so: cannot open
>> shared object file: No such file or directory
>> $ echo $LD_LIBRARY_PATH
>> /o

Re: [OMPI users] running openmpi with specified lib path

[Duke Nguyen]

On 5/7/13 9:48 PM, Ralph Castain wrote:
> 
> On May 7, 2013, at 7:39 AM, Duke Nguyen  wrote:
> 
>>
>>
>> No I didnt (and that maybe the reason, I am not really sure I was
>> correct when installation these things). What I did was:
>>
>> $ cd /opt/apps/openmpi/openmpi-1.7.2r28341_composer_xe_2013.3.163
>> $ ./configure CC=/opt/intel/composer_xe_2013.3.163/bin/intel64/icc
>> CXX=/opt/intel/composer_xe_2013.3.163/bin/intel64/icpc
>> F77=/opt/intel/composer_xe_2013.3.163/bin/intel64/ifort
>> FC=/opt/intel/composer_xe_2013.3.163/bin/intel64/ifort
>> $ make -j8
>> $ DESTDIR=`pwd` make -j8 install
>>
>> so all the compiled openmpi components were installed at
>>
>> $ ls /opt/apps/openmpi/openmpi-1.7.2r28341_composer_xe_2013.3.163/usr/local/
>> bin  etc  include  lib  share
>>
>> Am I doing wrong??? Please advise.
> 
> That's the problem - you have to give a prefix that matches the 
> ld-libary-path. This installation is looking for its libs in /usr/local, not 
> in your actual location
> 

You SAVED me a whole week of headache! I was messing between DESTDIR
and PREFIX. I think it is sorted out now :).



signature.asc
Description: OpenPGP digital signature