[OMPI users] Efficient polling for both incoming messages and request completion
Hello, Is it possible to efficiently poll for both incoming messages and request completion using only one thread? As far as I know, busy waiting with alternate MPI_Iprobe and MPI_Testsome calls is the only way to do this. Is that approach dangerous to do performance-wise? Background: my application is memory constrained, so when requests complete I may suddenly be able to schedule new computation. At the same time, I need to be responding to a variety of asynchronous messages from unknown processors with unknown message sizes, which as far as I know I can't turn into a request to poll on. Thanks, Geoffrey
[OMPI users] openmpi MPI_Init doesn't work after fork under gdb
The attached program illustrates the problem. It forks, and the child
calls MPI_Init. This works fine unless I'm inside gdb. Inside gdb,
MPI_Init silently crashes.
I'm using OpenMPI 1.6.0 on Mac 10.8.2. I'm running the program
directly, not through mpirun.
Any ideas what might be wrong?
Thanks,
Geoffrey
cone:scratch% /usr/local/bin/mpicc -o fork-bug fork-bug.c
cone:scratch% ./fork-bug
We're an MPI program!
child status = 0
cone:scratch% gdb ./fork-bug
gdb ./fork-bug
GNU gdb 6.3.50-20050815 (Apple version gdb-1824) (Thu Nov 15 10:42:43 UTC 2012)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "x86_64-apple-darwin"...Reading symbols for
shared libraries ... done
(gdb) run
Reading symbols for shared libraries ++. done
child status = 5
Program exited normally.
(gdb) cone:scratch%
#include
#include
#include
#include
#include
#include
#define CAREFUL 0
#if CAREFUL
int main(int argc, char** argv) {
int pid = fork();
if (pid < 0) {
printf("fork failed: %s\n",strerror(errno));
return 1;
}
if (pid) { // Parent
int status;
pid = wait(&status);
if (pid < 0) {
printf("wait failed: %s\n",strerror(errno));
return 1;
}
printf("child status = %d\n",status);
} else { // Child
int r = MPI_Init(&argc,&argv);
if (r != MPI_SUCCESS) {
printf("MPI_Init failed: r = %d\n",r);
return 1;
}
printf("We're an MPI program!\n");
r = MPI_Finalize();
if (r != MPI_SUCCESS) {
printf("MPI_Finalized failed: r = %d\n",r);
return 1;
}
}
// Everything worked
return 0;
}
#else
int main(int argc, char** argv) {
int pid = fork();
if (pid) { // Parent
int status;
wait(&status);
printf("child status = %d\n",status);
} else { // Child
MPI_Init(&argc,&argv);
printf("We're an MPI program!\n");
MPI_Finalize();
}
return 0;
}
#endif
Re: [OMPI users] openmpi MPI_Init doesn't work after fork under gdb
The singleton fork/exec itself is fine, since normal MPI programs work under gdb (e.g., fork-bug.c without the fork). gdb is has follow-fork-mode set to parent, so it's odd that gdb is looking at the child process's trickery at all. I've confirmed that it's still broken under 1.6.4, unfortunately. Geoffrey On Thu, Feb 21, 2013 at 3:36 PM, Ralph Castain wrote: > Singletons fork/exec a daemon to support them - my guess is that gdb may not > like it on your machine? > > FWIW - it runs fine for me using the developer's trunk. You might try with > 1.6.4 in case it's a bug in 1.6.0 > > > On Feb 21, 2013, at 3:18 PM, Geoffrey Irving wrote: > >> The attached program illustrates the problem. It forks, and the child >> calls MPI_Init. This works fine unless I'm inside gdb. Inside gdb, >> MPI_Init silently crashes. >> >> I'm using OpenMPI 1.6.0 on Mac 10.8.2. I'm running the program >> directly, not through mpirun. >> >> Any ideas what might be wrong? >> >> Thanks, >> Geoffrey >> >> cone:scratch% /usr/local/bin/mpicc -o fork-bug fork-bug.c >> cone:scratch% ./fork-bug >> We're an MPI program! >> child status = 0 >> cone:scratch% gdb ./fork-bug >> gdb ./fork-bug >> GNU gdb 6.3.50-20050815 (Apple version gdb-1824) (Thu Nov 15 10:42:43 UTC >> 2012) >> Copyright 2004 Free Software Foundation, Inc. >> GDB is free software, covered by the GNU General Public License, and you are >> welcome to change it and/or distribute copies of it under certain conditions. >> Type "show copying" to see the conditions. >> There is absolutely no warranty for GDB. Type "show warranty" for details. >> This GDB was configured as "x86_64-apple-darwin"...Reading symbols for >> shared libraries ... done >> >> (gdb) run >> Reading symbols for shared libraries ++. done >> child status = 5 >> >> Program exited normally. >> (gdb) cone:scratch% >> ___ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users > > > ___ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
Re: [OMPI users] openmpi MPI_Init doesn't work after fork under gdb
Actually, I don't see it printing "We're an MPI program!" under gdb, which means it isn't working. Geoffrey On Thu, Feb 21, 2013 at 4:07 PM, Ralph Castain wrote: > Hmmm...works with 1.6.4 for me on Mac 10.8.2: > > Ralphs-iMac:v1.6 rhc$ ./fork-bug > We're an MPI program! > child status = 0 > Ralphs-iMac:v1.6 rhc$ gdb ./fork-bug > GNU gdb 6.3.50-20050815 (Apple version gdb-1820) (Sat Jun 16 02:40:11 UTC > 2012) > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and you are > welcome to change it and/or distribute copies of it under certain conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for details. > This GDB was configured as "x86_64-apple-darwin"...Reading symbols for shared > libraries ... done > > (gdb) r > Starting program: /Volumes/RHCHD/rhc/openmpi/v1.6/fork-bug > Reading symbols for shared libraries ++. done > child status = 0 > > Program exited normally. > (gdb) > > Afraid I have no idea why you might be hitting the problem, though... > > > On Feb 21, 2013, at 3:50 PM, Geoffrey Irving wrote: > >> The singleton fork/exec itself is fine, since normal MPI programs work >> under gdb (e.g., fork-bug.c without the fork). gdb is has >> follow-fork-mode set to parent, so it's odd that gdb is looking at the >> child process's trickery at all. >> >> I've confirmed that it's still broken under 1.6.4, unfortunately. >> >> Geoffrey >> >> On Thu, Feb 21, 2013 at 3:36 PM, Ralph Castain wrote: >>> Singletons fork/exec a daemon to support them - my guess is that gdb may >>> not like it on your machine? >>> >>> FWIW - it runs fine for me using the developer's trunk. You might try with >>> 1.6.4 in case it's a bug in 1.6.0 >>> >>> >>> On Feb 21, 2013, at 3:18 PM, Geoffrey Irving wrote: >>> >>>> The attached program illustrates the problem. It forks, and the child >>>> calls MPI_Init. This works fine unless I'm inside gdb. Inside gdb, >>>> MPI_Init silently crashes. >>>> >>>> I'm using OpenMPI 1.6.0 on Mac 10.8.2. I'm running the program >>>> directly, not through mpirun. >>>> >>>> Any ideas what might be wrong? >>>> >>>> Thanks, >>>> Geoffrey >>>> >>>> cone:scratch% /usr/local/bin/mpicc -o fork-bug fork-bug.c >>>> cone:scratch% ./fork-bug >>>> We're an MPI program! >>>> child status = 0 >>>> cone:scratch% gdb ./fork-bug >>>> gdb ./fork-bug >>>> GNU gdb 6.3.50-20050815 (Apple version gdb-1824) (Thu Nov 15 10:42:43 UTC >>>> 2012) >>>> Copyright 2004 Free Software Foundation, Inc. >>>> GDB is free software, covered by the GNU General Public License, and you >>>> are >>>> welcome to change it and/or distribute copies of it under certain >>>> conditions. >>>> Type "show copying" to see the conditions. >>>> There is absolutely no warranty for GDB. Type "show warranty" for details. >>>> This GDB was configured as "x86_64-apple-darwin"...Reading symbols for >>>> shared libraries ... done >>>> >>>> (gdb) run >>>> Reading symbols for shared libraries ++. done >>>> child status = 5 >>>> >>>> Program exited normally. >>>> (gdb) cone:scratch% >>>> ___ >>>> users mailing list >>>> us...@open-mpi.org >>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>> >>> >>> ___ >>> users mailing list >>> us...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/users >> ___ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users > > > ___ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
Re: [OMPI users] openmpi MPI_Init doesn't work after fork under gdb
No, I don't really have any idea what it's doing. You have to add manual sleeps and attach another instance of gdb if you want to step through the child, since I also can't get follow-fork-mode child to work. I only tried that once, and it segfaulted accessing the zero pointer destroying the stack in the process. I haven't done manual stepping or printf searches, so I don't know where in MPI_Init it breaks. I'm trying to install gdb 7.5.1 now (a nontrivial process on Mac, unfortunately) to see if that fixes it. Geoffrey On Thu, Feb 21, 2013 at 4:28 PM, Ralph Castain wrote: > Hmmm...how about that? Yeah, it doesn't work with the devel trunk either - > I'd missed that point. > > No idea why, I'm afraid - never tried it before. Are you sure it "crashes"? > I'm still getting a child status of "0", but no message output. My guess is > that the I/O is being lost for some reason. > > On Feb 21, 2013, at 4:09 PM, Geoffrey Irving wrote: > >> Actually, I don't see it printing "We're an MPI program!" under gdb, >> which means it isn't working. >> >> Geoffrey >> >> On Thu, Feb 21, 2013 at 4:07 PM, Ralph Castain wrote: >>> Hmmm...works with 1.6.4 for me on Mac 10.8.2: >>> >>> Ralphs-iMac:v1.6 rhc$ ./fork-bug >>> We're an MPI program! >>> child status = 0 >>> Ralphs-iMac:v1.6 rhc$ gdb ./fork-bug >>> GNU gdb 6.3.50-20050815 (Apple version gdb-1820) (Sat Jun 16 02:40:11 UTC >>> 2012) >>> Copyright 2004 Free Software Foundation, Inc. >>> GDB is free software, covered by the GNU General Public License, and you are >>> welcome to change it and/or distribute copies of it under certain >>> conditions. >>> Type "show copying" to see the conditions. >>> There is absolutely no warranty for GDB. Type "show warranty" for details. >>> This GDB was configured as "x86_64-apple-darwin"...Reading symbols for >>> shared libraries ... done >>> >>> (gdb) r >>> Starting program: /Volumes/RHCHD/rhc/openmpi/v1.6/fork-bug >>> Reading symbols for shared libraries ++. done >>> child status = 0 >>> >>> Program exited normally. >>> (gdb) >>> >>> Afraid I have no idea why you might be hitting the problem, though... >>> >>> >>> On Feb 21, 2013, at 3:50 PM, Geoffrey Irving wrote: >>> >>>> The singleton fork/exec itself is fine, since normal MPI programs work >>>> under gdb (e.g., fork-bug.c without the fork). gdb is has >>>> follow-fork-mode set to parent, so it's odd that gdb is looking at the >>>> child process's trickery at all. >>>> >>>> I've confirmed that it's still broken under 1.6.4, unfortunately. >>>> >>>> Geoffrey >>>> >>>> On Thu, Feb 21, 2013 at 3:36 PM, Ralph Castain wrote: >>>>> Singletons fork/exec a daemon to support them - my guess is that gdb may >>>>> not like it on your machine? >>>>> >>>>> FWIW - it runs fine for me using the developer's trunk. You might try >>>>> with 1.6.4 in case it's a bug in 1.6.0 >>>>> >>>>> >>>>> On Feb 21, 2013, at 3:18 PM, Geoffrey Irving wrote: >>>>> >>>>>> The attached program illustrates the problem. It forks, and the child >>>>>> calls MPI_Init. This works fine unless I'm inside gdb. Inside gdb, >>>>>> MPI_Init silently crashes. >>>>>> >>>>>> I'm using OpenMPI 1.6.0 on Mac 10.8.2. I'm running the program >>>>>> directly, not through mpirun. >>>>>> >>>>>> Any ideas what might be wrong? >>>>>> >>>>>> Thanks, >>>>>> Geoffrey >>>>>> >>>>>> cone:scratch% /usr/local/bin/mpicc -o fork-bug fork-bug.c >>>>>> cone:scratch% ./fork-bug >>>>>> We're an MPI program! >>>>>> child status = 0 >>>>>> cone:scratch% gdb ./fork-bug >>>>>> gdb ./fork-bug >>>>>> GNU gdb 6.3.50-20050815 (Apple version gdb-1824) (Thu Nov 15 10:42:43 >>>>>> UTC 2012) >>>>>> Copyright 2004 Free Software Foundation, Inc. >>>>>> GDB is free software, covered by the GNU General Public License, and you >>>>>> are >>>&
Re: [OMPI users] openmpi MPI_Init doesn't work after fork under gdb
No luck with gdb 7.5.1. I can get it to run, but it hits an internal error and bails out. Thus, I'm stuck with Apple's 6.3.50. I suppose I'll try to restructure the app so that I don't have to spawn the mpi process with a fork from the other one. Geoffrey On Thu, Feb 21, 2013 at 4:38 PM, Geoffrey Irving wrote: > No, I don't really have any idea what it's doing. You have to add > manual sleeps and attach another instance of gdb if you want to step > through the child, since I also can't get follow-fork-mode child to > work. I only tried that once, and it segfaulted accessing the zero > pointer destroying the stack in the process. I haven't done manual > stepping or printf searches, so I don't know where in MPI_Init it > breaks. > > I'm trying to install gdb 7.5.1 now (a nontrivial process on Mac, > unfortunately) to see if that fixes it. > > Geoffrey > > On Thu, Feb 21, 2013 at 4:28 PM, Ralph Castain wrote: >> Hmmm...how about that? Yeah, it doesn't work with the devel trunk either - >> I'd missed that point. >> >> No idea why, I'm afraid - never tried it before. Are you sure it "crashes"? >> I'm still getting a child status of "0", but no message output. My guess is >> that the I/O is being lost for some reason. >> >> On Feb 21, 2013, at 4:09 PM, Geoffrey Irving wrote: >> >>> Actually, I don't see it printing "We're an MPI program!" under gdb, >>> which means it isn't working. >>> >>> Geoffrey >>> >>> On Thu, Feb 21, 2013 at 4:07 PM, Ralph Castain wrote: >>>> Hmmm...works with 1.6.4 for me on Mac 10.8.2: >>>> >>>> Ralphs-iMac:v1.6 rhc$ ./fork-bug >>>> We're an MPI program! >>>> child status = 0 >>>> Ralphs-iMac:v1.6 rhc$ gdb ./fork-bug >>>> GNU gdb 6.3.50-20050815 (Apple version gdb-1820) (Sat Jun 16 02:40:11 UTC >>>> 2012) >>>> Copyright 2004 Free Software Foundation, Inc. >>>> GDB is free software, covered by the GNU General Public License, and you >>>> are >>>> welcome to change it and/or distribute copies of it under certain >>>> conditions. >>>> Type "show copying" to see the conditions. >>>> There is absolutely no warranty for GDB. Type "show warranty" for details. >>>> This GDB was configured as "x86_64-apple-darwin"...Reading symbols for >>>> shared libraries ... done >>>> >>>> (gdb) r >>>> Starting program: /Volumes/RHCHD/rhc/openmpi/v1.6/fork-bug >>>> Reading symbols for shared libraries ++. done >>>> child status = 0 >>>> >>>> Program exited normally. >>>> (gdb) >>>> >>>> Afraid I have no idea why you might be hitting the problem, though... >>>> >>>> >>>> On Feb 21, 2013, at 3:50 PM, Geoffrey Irving wrote: >>>> >>>>> The singleton fork/exec itself is fine, since normal MPI programs work >>>>> under gdb (e.g., fork-bug.c without the fork). gdb is has >>>>> follow-fork-mode set to parent, so it's odd that gdb is looking at the >>>>> child process's trickery at all. >>>>> >>>>> I've confirmed that it's still broken under 1.6.4, unfortunately. >>>>> >>>>> Geoffrey >>>>> >>>>> On Thu, Feb 21, 2013 at 3:36 PM, Ralph Castain wrote: >>>>>> Singletons fork/exec a daemon to support them - my guess is that gdb may >>>>>> not like it on your machine? >>>>>> >>>>>> FWIW - it runs fine for me using the developer's trunk. You might try >>>>>> with 1.6.4 in case it's a bug in 1.6.0 >>>>>> >>>>>> >>>>>> On Feb 21, 2013, at 3:18 PM, Geoffrey Irving wrote: >>>>>> >>>>>>> The attached program illustrates the problem. It forks, and the child >>>>>>> calls MPI_Init. This works fine unless I'm inside gdb. Inside gdb, >>>>>>> MPI_Init silently crashes. >>>>>>> >>>>>>> I'm using OpenMPI 1.6.0 on Mac 10.8.2. I'm running the program >>>>>>> directly, not through mpirun. >>>>>>> >>>>>>> Any ideas what might be wrong? >>>>>>> >>>&
Re: [OMPI users] openmpi MPI_Init doesn't work after fork under gdb
More info: it works fine if I exec after the fork (or equivalently, run anything via system). This provides a sufficient workaround at least for me, so it's probably not worth complaining further that a proprietarily modified decade old version of gdb doesn't work. :) Thanks, Geoffrey On Fri, Feb 22, 2013 at 11:18 AM, George Bosilca wrote: > Geoffrey, > > If I understand correctly your problem, I think it is a gdb issue. You should > inform gdb that after the fork call instead of following the parent you > expect to follow the child. > > set follow-fork-mode child (should do the trick). > > For more info: http://sourceware.org/gdb/onlinedocs/gdb/Forks.html > > George. > > On Feb 22, 2013, at 19:55 , Geoffrey Irving wrote: > >> No luck with gdb 7.5.1. I can get it to run, but it hits an internal >> error and bails out. Thus, I'm stuck with Apple's 6.3.50. I suppose >> I'll try to restructure the app so that I don't have to spawn the mpi >> process with a fork from the other one. >> >> Geoffrey >> >> On Thu, Feb 21, 2013 at 4:38 PM, Geoffrey Irving wrote: >>> No, I don't really have any idea what it's doing. You have to add >>> manual sleeps and attach another instance of gdb if you want to step >>> through the child, since I also can't get follow-fork-mode child to >>> work. I only tried that once, and it segfaulted accessing the zero >>> pointer destroying the stack in the process. I haven't done manual >>> stepping or printf searches, so I don't know where in MPI_Init it >>> breaks. >>> >>> I'm trying to install gdb 7.5.1 now (a nontrivial process on Mac, >>> unfortunately) to see if that fixes it. >>> >>> Geoffrey >>> >>> On Thu, Feb 21, 2013 at 4:28 PM, Ralph Castain wrote: >>>> Hmmm...how about that? Yeah, it doesn't work with the devel trunk either - >>>> I'd missed that point. >>>> >>>> No idea why, I'm afraid - never tried it before. Are you sure it >>>> "crashes"? I'm still getting a child status of "0", but no message output. >>>> My guess is that the I/O is being lost for some reason. >>>> >>>> On Feb 21, 2013, at 4:09 PM, Geoffrey Irving wrote: >>>> >>>>> Actually, I don't see it printing "We're an MPI program!" under gdb, >>>>> which means it isn't working. >>>>> >>>>> Geoffrey >>>>> >>>>> On Thu, Feb 21, 2013 at 4:07 PM, Ralph Castain wrote: >>>>>> Hmmm...works with 1.6.4 for me on Mac 10.8.2: >>>>>> >>>>>> Ralphs-iMac:v1.6 rhc$ ./fork-bug >>>>>> We're an MPI program! >>>>>> child status = 0 >>>>>> Ralphs-iMac:v1.6 rhc$ gdb ./fork-bug >>>>>> GNU gdb 6.3.50-20050815 (Apple version gdb-1820) (Sat Jun 16 02:40:11 >>>>>> UTC 2012) >>>>>> Copyright 2004 Free Software Foundation, Inc. >>>>>> GDB is free software, covered by the GNU General Public License, and you >>>>>> are >>>>>> welcome to change it and/or distribute copies of it under certain >>>>>> conditions. >>>>>> Type "show copying" to see the conditions. >>>>>> There is absolutely no warranty for GDB. Type "show warranty" for >>>>>> details. >>>>>> This GDB was configured as "x86_64-apple-darwin"...Reading symbols for >>>>>> shared libraries ... done >>>>>> >>>>>> (gdb) r >>>>>> Starting program: /Volumes/RHCHD/rhc/openmpi/v1.6/fork-bug >>>>>> Reading symbols for shared libraries ++. done >>>>>> child status = 0 >>>>>> >>>>>> Program exited normally. >>>>>> (gdb) >>>>>> >>>>>> Afraid I have no idea why you might be hitting the problem, though... >>>>>> >>>>>> >>>>>> On Feb 21, 2013, at 3:50 PM, Geoffrey Irving wrote: >>>>>> >>>>>>> The singleton fork/exec itself is fine, since normal MPI programs work >>>>>>> under gdb (e.g., fork-bug.c without the fork). gdb is has >>>>>>> follow-fork-mode set to parent, so it's odd that gdb is looking at the >>>>>>> child process&#x
[OMPI users] open mpi doesn't have OMPI_VERSION
Hello, LAM had a preprocessor symbol LAM_VERSION giving its version. Open MPI appears to have this only in Fortran. It would be great if you could add the version symbols in a C++ include file. This is useful since it allows me to check whether the installed version of open mpi is the same as the compiled version to prevent silent undefined behavior and crashes. Alternatively, could a version matching check be added to Open MPI internally? Geoffrey
