Re: [OMPI users] General ORTE questions
Hi Ralph: Thanks for your information. You said I could ask more so I am! See below. Ralph Castain wrote On 03/30/06 16:51,: Hi Rolf I apologize for the scarce documentation - we are working on it, but have a ways to go. I've tried to address your questions below. Please feel free to ask more! Ralph Rolf Vandevaart wrote: Greetings: I am new to the Open MPI world, and I have been trying to get a better understanding of the ORTE environment. At this point, I have a few questions that I was hoping someone could answer. 1. I have heard mention of running the ORTE daemons in persistent mode, however, I can find no details of how to do this. Are there arguments to either orted or mpirun to make this work right? Normally, we start a persistent daemon with: orted --seed --persistent --scope=public This will start the daemon and "daemonize" it so it keeps running until told to die. The arguments worth noting are: (a) --persistent. Tells the daemon to "stay alive" until specifically told to "die" (b) --scope=[public, private, exclusive]. This actually pertains to the universe, but you'll need to provide it anyway to ensure proper connectivity to anything you try to run. Right now, the daemons default to "exclusive", which means nothing can connect to them except the application that spawned them - no value to anyone if started with the above command! Private would exclude them to contact only from you - I haven't tested this enough to guarantee its functionality. I usually run them as "public" since security isn't a big concern right now - all this means is that anyone who can read the session directory tree (which is normally "locked" to only you anyway) would be able to connect to the daemon. (c) --seed. Indicates that this daemon is the first one and therefore will host the data storage for the registry and other central services (d) --universe=userid@hostname:universe_name. Allows you to name your universe to whatever you like. We use this to allow you to have multiple universes co-existing but separate - I've been explaining the reasons for that elsewhere, but will send them to this list if desired. You don't have to provide this, nor do you have to provide all the fields (e.g., you could just say "--universe=foo" to set the universe name). You can provide the same options to mpirun, if you like - mpirun will simply start an orted and pass those parameters along, and the orted will merrily stay alive after the specified application completes. While I understand all that has been written here in theory, I am still struggling to get things to work. The persistent daemon seems to be ignored when I do an mpirun. I have watched the system calls and looked at the process tree, and the persistent daemon does not seem to be part of the fun. So, I will be specific about what I am doing, and maybe you can point out what I am doing wrong. I have a 3 node cluster. ct2, ct4, and ct5. I am launching the job from ct2 and trying to run on ct4 and ct5 which have persistent daemons on them. I have selected the daemon on ct4 to be the seed. ct4> orted --seed --persistent --scope public -universe foo ct5> orted --persistent --scope public -universe foo ct2> mpirun --mca pls_rsh_agent rsh -np 4 -host ct4,ct5 -universe foo my_connectivity -v While the program is running, I see this on ct4 and ct5. ps -ef | grep orted rolfv 9456 1 0 11:24:26 ? 0:00 orted --bootproxy 1 --name 0.0.2 --num_procs 3 --vpid_start 0 --nodename ct4 rolfv 9386 1 0 11:21:30 ? 0:00 orted --seed --persistent --scope public --universe foo Thanks for any additional details. *snip* 3. I have a similar question about orteprobe. Is this something we should know about? Yes and no - there's nothing secret about it. We use it internally to OpenRTE to "probe" a machine and see if we have a daemon/universe operating on it. Basically, we launch orteprobe on the remote machine - it checks to see if a session directory exists on it, attempts to connect to any universes it finds, and then reports back on its findings. Based on that report, we either launch an orted on the remote machine (to act as our surrogate so we can launch an application on that cell) or connect to an existing universe on the remote machine (and then tell it to launch the application for us). 4. Is there an easy way to view the data in the General Purpose Registry? This may be related to my first question, in that I could imagine having persistent daemons and then I would like to see what is stored in the registry. Well, yes and no. Ideally, that would be a command from within the orteconsole function, but I don't think that has been implemented yet. I'd be happy to do so, if that is something you would like (shouldn't take long at all). There are a set of "dump" functions in the registry API for just that purpose. I usually access them via gdb - I att
Re: [OMPI users] General ORTE questions
> -Original Message- > From: users-boun...@open-mpi.org > [mailto:users-boun...@open-mpi.org] On Behalf Of Rolf Vandevaart > Sent: Thursday, March 30, 2006 3:43 PM > To: us...@open-mpi.org > Subject: [OMPI users] General ORTE questions > > 6. From what I can tell, there is no way to specify the slots argument > with the -host argument. For example, I cannot do this: > mpirun -np 8 -host node1:slots=4,node2:slots=4 a.out > Just wanted to confirm that. Right now, I do believe that if you list a host more than once, it bumps the slots count. This is somewhat inelegant, but something like: shell$ mpirun -host node1 -host node1 ... Same as: shell% mpirun -host node1,node1 ... Will make node1 have a slot count of 2. We can certainly change this, though -- it wasn't implemented simply because I didn't think of this case. :-) -- Jeff Squyres Server Virtualization Business Unit Cisco Systems
Re: [OMPI users] General ORTE questions
Hi Rolf See below! Rolf Vandevaart wrote: Hi Ralph: Thanks for your information. You said I could ask more so I am! See below. Ralph Castain wrote On 03/30/06 16:51,: Hi Rolf I apologize for the scarce documentation - we are working on it, but have a ways to go. I've tried to address your questions below. Please feel free to ask more! Ralph Rolf Vandevaart wrote: Greetings: I am new to the Open MPI world, and I have been trying to get a better understanding of the ORTE environment. At this point, I have a few questions that I was hoping someone could answer. 1. I have heard mention of running the ORTE daemons in persistent mode, however, I can find no details of how to do this. Are there arguments to either orted or mpirun to make this work right? Normally, we start a persistent daemon with: orted --seed --persistent --scope=public This will start the daemon and "daemonize" it so it keeps running until told to die. The arguments worth noting are: (a) --persistent. Tells the daemon to "stay alive" until specifically told to "die" (b) --scope=[public, private, exclusive]. This actually pertains to the universe, but you'll need to provide it anyway to ensure proper connectivity to anything you try to run. Right now, the daemons default to "exclusive", which means nothing can connect to them except the application that spawned them - no value to anyone if started with the above command! Private would exclude them to contact only from you - I haven't tested this enough to guarantee its functionality. I usually run them as "public" since security isn't a big concern right now - all this means is that anyone who can read the session directory tree (which is normally "locked" to only you anyway) would be able to connect to the daemon. (c) --seed. Indicates that this daemon is the first one and therefore will host the data storage for the registry and other central services (d) --universe=userid@hostname:universe_name. Allows you to name your universe to whatever you like. We use this to allow you to have multiple universes co-existing but separate - I've been explaining the reasons for that elsewhere, but will send them to this list if desired. You don't have to provide this, nor do you have to provide all the fields (e.g., you could just say "--universe=foo" to set the universe name). You can provide the same options to mpirun, if you like - mpirun will simply start an orted and pass those parameters along, and the orted will merrily stay alive after the specified application completes. While I understand all that has been written here in theory, I am still struggling to get things to work. The persistent daemon seems to be ignored when I do an mpirun. I have watched the system calls and looked at the process tree, and the persistent daemon does not seem to be part of the fun. So, I will be specific about what I am doing, and maybe you can point out what I am doing wrong. I have a 3 node cluster. ct2, ct4, and ct5. I am launching the job from ct2 and trying to run on ct4 and ct5 which have persistent daemons on them. I have selected the daemon on ct4 to be the seed. ct4> orted --seed --persistent --scope public -universe foo ct5> orted --persistent --scope public -universe foo ct2> mpirun --mca pls_rsh_agent rsh -np 4 -host ct4,ct5 -universe foo my_connectivity -v While the program is running, I see this on ct4 and ct5. ps -ef | grep orted rolfv 9456 1 0 11:24:26 ? 0:00 orted --bootproxy 1 --name 0.0.2 --num_procs 3 --vpid_start 0 --nodename ct4 rolfv 9386 1 0 11:21:30 ? 0:00 orted --seed --persistent --scope public --universe foo Thanks for any additional details. I see the problem - really just a current limit on our implementation. At the moment, our launchers don't know how to take advantage of existing daemons on the remote nodes. Your mpirun is correctly connecting to the persistent daemon on ct2, but will launch its own daemons on the other nodes. This is a known issue that we need to address - just low on the priority list right now. *snip* 3. I have a similar question about orteprobe. Is this something we should know about? Yes and no - there's nothing secret about it. We use it internally to OpenRTE to "probe" a machine and see if we have a daemon/universe operating on it. Basically, we launch orteprobe on the remote machine - it checks to see if a session directory exists on it, attempts to connect to any universes it finds, and then reports back on its findings. Based on that report, we either launch an orted on the remote machine (to act as our surrogate so we can launch an application on that cell) or connect to an existing universe on the remote machine (and then tell it to launch the application for us). 4. Is there an easy way to view the data in the General Pur
[OMPI users] HPCWire article on OpenRTE
Hi folks Just as an FYI: HPCWire has released a Q&A article on OpenRTE in today's edition. You can access it at http://www.hpcwire.com/hpc/608536.html. Ralph
[OMPI users] MPI_Testsome with no requests
I have an algorithm that collects information in a tree like manner using
nonblocking communication. Some nodes do not receive information from other
nodes, so there are no outstanding requests on those nodes. On all
processors, I check for the incoming messages using MPI_Testsome().
MPI_Testsome fails with OpenMPI, however if the request length is zero.
Here is a code that can be run with only one processor that shows the same
behavior:
///
#include "mpi.h"
#include
int main( int argc, char *argv[])
{
int myid, numprocs;
MPI_Init(&argc,&argv);
MPI_Comm_size(MPI_COMM_WORLD,&numprocs);
MPI_Comm_rank(MPI_COMM_WORLD,&myid);
printf("Hello from processor %i of %i\n", myid, numprocs);
int size = 0;
int num_done = 0;
MPI_Status* stat = 0;
MPI_Request* req = 0;
int* done_indices = 0;
MPI_Testsome( size, req, &num_done, done_indices, stat);
printf("Finalizing on processor %i of %i\n", myid, numprocs);
MPI_Finalize();
return 0;
}
/
The output using OpenMPI is:
Hello from processor 0 of 1
[mymachine:09115] *** An error occurred in MPI_Testsome
[mymachine:09115] *** on communicator MPI_COMM_WORLD
[mymachine:09115] *** MPI_ERR_REQUEST: invalid request
[mymachine:09115] *** MPI_ERRORS_ARE_FATAL (goodbye)
1 process killed (possibly by Open MPI)
Many other MPI implementations support this, and reading the standard, it
seems like it should be OK.
Thanks,
Tom
config.log.bz2
Description: Binary data
testsome_test.out
Description: Binary data
testsome_test.c
Description: Binary data
ompi_info.out
Description: Binary data
Re: [OMPI users] MPI_Testsome with no requests
When we're checking the arguments, we check for the request array to
not be NULL without looking to the number of requests. I think it
make sense, as I don't see why the user would call these functions
with 0 requests ... But, the other way around make sense too. As I
don't find anything in the MPI standard that stop the user doing that
I add the additional check to all MPI_Wait* and MPI_Test* functions.
Please get the version from trunk after revision 9504.
Thanks,
george.
On Mar 31, 2006, at 2:56 PM, Brunner, Thomas A. wrote:
I have an algorithm that collects information in a tree like manner
using
nonblocking communication. Some nodes do not receive information
from other
nodes, so there are no outstanding requests on those nodes. On all
processors, I check for the incoming messages using MPI_Testsome().
MPI_Testsome fails with OpenMPI, however if the request length is
zero.
Here is a code that can be run with only one processor that shows
the same
behavior:
///
#include "mpi.h"
#include
int main( int argc, char *argv[])
{
int myid, numprocs;
MPI_Init(&argc,&argv);
MPI_Comm_size(MPI_COMM_WORLD,&numprocs);
MPI_Comm_rank(MPI_COMM_WORLD,&myid);
printf("Hello from processor %i of %i\n", myid, numprocs);
int size = 0;
int num_done = 0;
MPI_Status* stat = 0;
MPI_Request* req = 0;
int* done_indices = 0;
MPI_Testsome( size, req, &num_done, done_indices, stat);
printf("Finalizing on processor %i of %i\n", myid, numprocs);
MPI_Finalize();
return 0;
}
/
The output using OpenMPI is:
Hello from processor 0 of 1
[mymachine:09115] *** An error occurred in MPI_Testsome
[mymachine:09115] *** on communicator MPI_COMM_WORLD
[mymachine:09115] *** MPI_ERR_REQUEST: invalid request
[mymachine:09115] *** MPI_ERRORS_ARE_FATAL (goodbye)
1 process killed (possibly by Open MPI)
Many other MPI implementations support this, and reading the
standard, it
seems like it should be OK.
Thanks,
Tom
___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
Re: [OMPI users] MPI_Testsome with no requests
Compiling revision 9505 of the trunk and building my original test code now
core dumps. I can run the test code with the Testsome line commented out.
Here is the output from a brief gdb session:
--
gdb a.out /cores/core.28141
GNU gdb 6.1-20040303 (Apple version gdb-437) (Sun Dec 25 08:31:29 GMT 2005)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "powerpc-apple-darwin"...Reading symbols for
shared libraries . done
Core was generated by `a.out'.
#0 0x010b2a90 in ?? ()
(gdb) bt
#0 0x010b2a90 in ?? ()
#1 0x010b2a3c in ?? ()
warning: Previous frame identical to this frame (corrupt stack?)
#2 0x2c18 in grow_table (table=0x1, soft=3221222188, hard=0) at
class/ompi_pointer_array.c:352
(gdb) up
#1 0x010b2a3c in ?? ()
(gdb) up
#2 0x2c18 in grow_table (table=0x1, soft=3221222188, hard=0) at
class/ompi_pointer_array.c:352
352 if (table->size >= OMPI_FORTRAN_HANDLE_MAX) {
---
This is the output from the code.
Hello from processor 0 of 1
Signal:10 info.si_errno:0(Unknown error: 0) si_code:1(BUS_ADRALN)
Failing at addr:0x0
*** End of error message ***
Perhaps in the MPI_Wait* and MPI_Test* functions, if incount==0, then
*outcount should be set to zero and immediately return? (Of course checking
that outcount !=0 too.)
Tom
On 3/31/06 1:35 PM, "George Bosilca" wrote:
> When we're checking the arguments, we check for the request array to
> not be NULL without looking to the number of requests. I think it
> make sense, as I don't see why the user would call these functions
> with 0 requests ... But, the other way around make sense too. As I
> don't find anything in the MPI standard that stop the user doing that
> I add the additional check to all MPI_Wait* and MPI_Test* functions.
>
> Please get the version from trunk after revision 9504.
>
>Thanks,
> george.
>
> On Mar 31, 2006, at 2:56 PM, Brunner, Thomas A. wrote:
>
>>
>> I have an algorithm that collects information in a tree like manner
>> using
>> nonblocking communication. Some nodes do not receive information
>> from other
>> nodes, so there are no outstanding requests on those nodes. On all
>> processors, I check for the incoming messages using MPI_Testsome().
>> MPI_Testsome fails with OpenMPI, however if the request length is
>> zero.
>> Here is a code that can be run with only one processor that shows
>> the same
>> behavior:
>>
>> ///
>>
>> #include "mpi.h"
>> #include
>>
>> int main( int argc, char *argv[])
>> {
>> int myid, numprocs;
>>
>> MPI_Init(&argc,&argv);
>> MPI_Comm_size(MPI_COMM_WORLD,&numprocs);
>> MPI_Comm_rank(MPI_COMM_WORLD,&myid);
>>
>> printf("Hello from processor %i of %i\n", myid, numprocs);
>>
>> int size = 0;
>> int num_done = 0;
>> MPI_Status* stat = 0;
>> MPI_Request* req = 0;
>> int* done_indices = 0;
>>
>> MPI_Testsome( size, req, &num_done, done_indices, stat);
>>
>> printf("Finalizing on processor %i of %i\n", myid, numprocs);
>>
>> MPI_Finalize();
>>
>> return 0;
>> }
>>
>> /
>>
>> The output using OpenMPI is:
>>
>> Hello from processor 0 of 1
>> [mymachine:09115] *** An error occurred in MPI_Testsome
>> [mymachine:09115] *** on communicator MPI_COMM_WORLD
>> [mymachine:09115] *** MPI_ERR_REQUEST: invalid request
>> [mymachine:09115] *** MPI_ERRORS_ARE_FATAL (goodbye)
>> 1 process killed (possibly by Open MPI)
>>
>>
>> Many other MPI implementations support this, and reading the
>> standard, it
>> seems like it should be OK.
>>
>> Thanks,
>> Tom
>>
>>
>>
>>
>>
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
