Re: [OMPI users] www.open-mpi.org certificate error?
Hi Jeff and all Disclaimer: I know next to nothing about how the web works. Having said that, would it not be possible to redirect an https request to a http request? I believe apache mod-rewrite can do it. Or does this certificate check happens even before the rewrite? Regards Durga The woods are lovely, dark and deep; but I have promises to keep. And kilometers to go before I sleep; and kilometers to go before I sleep. On Sat, Jul 30, 2016 at 12:31 PM, Jeff Squyres (jsquyres) < jsquy...@cisco.com> wrote: > Meh. That's a good point. We might have to pony up the cost for the > certificates, then. :-( > > (Indiana University provided all this stuff to us for free; now that the > community has to pay for our own hosting, the funding has to come from some > where). > > Please bear with us -- all this sysadmin/infrastructure stuff is > completely unrelated to do with our real jobs (i.e., software development > of Open MPI); we're doing all this migration work on nights, weekends, and > sometimes while waiting for lengthy compiles. We didn't think of the > Google-will-have-https-links issue. :-\ > > > > > On Jul 30, 2016, at 12:27 PM, Bennet Fauber wrote: > > > > Thanks, Jeff, > > > > Just to note, though, many, many links in Google searches will have > > the https address. > > > > -- bennet > > > > > > On Sat, Jul 30, 2016 at 12:21 PM, Jeff Squyres (jsquyres) > > wrote: > >> Hmm. Sorry about this; we just moved the web site from Indiana > University to Host Gator (per > http://www.open-mpi.org/community/lists/devel/2016/06/19139.php). > >> > >> I thought I had disabled https for the web site last night when I did > the move -- I'll have to check into this. > >> > >> For the meantime, please just use http://www.open-mpi.org/. > >> > >> > >> > >>> On Jul 30, 2016, at 11:25 AM, Bennet Fauber wrote: > >>> > >>> I am getting a certificate error from https://www.open-mpi.org/ > >>> > >>> The owner of www.open-mpi.org has configured their website improperly. > >>> To protect your information from being stolen, Firefox has not > >>> connected to this website. > >>> > >>> and if I go to advanced and ask about the certificate, it says > >>> > >>> The certificate is only valid for the following names: > >>> *.hostgator.com, hostgator.com > >>> > >>> > >>> Is this something I have done to myself? > >>> ___ > >>> users mailing list > >>> users@lists.open-mpi.org > >>> https://rfd.newmexicoconsortium.org/mailman/listinfo/users > >> > >> > >> -- > >> Jeff Squyres > >> jsquy...@cisco.com > >> For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > >> > >> ___ > >> users mailing list > >> users@lists.open-mpi.org > >> https://rfd.newmexicoconsortium.org/mailman/listinfo/users > > ___ > > users mailing list > > users@lists.open-mpi.org > > https://rfd.newmexicoconsortium.org/mailman/listinfo/users > > > -- > Jeff Squyres > jsquy...@cisco.com > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > > ___ > users mailing list > users@lists.open-mpi.org > https://rfd.newmexicoconsortium.org/mailman/listinfo/users > ___ users mailing list users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/users
Re: [OMPI users] open-mpi: all-recursive error when compiling
In addition to what Gilles said, could you also check if selinux is enabled, and if so, if disabling it makes a difference? Thanks Durga On Thu, Aug 4, 2016 at 8:33 PM, Gilles Gouaillardet wrote: > The error message is related to a permission issue (which is very puzzling > in itself ...) > > can you manually check the permissions ? > > cd /home/pi/Downloads/openmpi-2.0.0/opal/asm > > ls -l .deps/atomic-asm.Tpo atomic-asm.S > > > then you can > > make clean > > make V=1 atomic-asm.lo > > > and post the output > > > meanwhile, you can check your umask is correct (so files are created with > the right permissions). > > note unless you plan to install openmpi in /usr/local (e.g. sudo make > install, after make) > > you need to ./configure --prefix= > > i also recommend you also --enable-mpirun-prefix-by-default > > > Cheers, > > > Gilles > > On 8/5/2016 9:09 AM, Christiano SA wrote: > > Hi, > > I've downloaded the openmpi-2.0.0.tar.gz and done this: > $ tar -zxvf openmpi-2.0.0.tar.gz > $ cd openmpi-2.0.0 > $ ./configure > (...) > $ make > but an error is happening: > (...) > Making all in include > make[2]: Entering directory '/home/pi/Downloads/openmpi-2. > 0.0/opal/include' > make all-am > make[3]: Entering directory '/home/pi/Downloads/openmpi-2. > 0.0/opal/include' > make[3]: Leaving directory '/home/pi/Downloads/openmpi-2.0.0/opal/include' > make[2]: Leaving directory '/home/pi/Downloads/openmpi-2.0.0/opal/include' > Making all in asm > make[2]: Entering directory '/home/pi/Downloads/openmpi-2.0.0/opal/asm' > CPPASatomic-asm.lo > atomic-asm.S:1:0: fatal error: opening dependency file > .deps/atomic-asm.Tpo: Permission denied > .text > ^ > compilation terminated. > make[2]: *** [Makefile:1735: atomic-asm.lo] Error 1 > make[2]: Leaving directory '/home/pi/Downloads/openmpi-2.0.0/opal/asm' > make[1]: *** [Makefile:2301: all-recursive] Error 1 > make[1]: Leaving directory '/home/pi/Downloads/openmpi-2.0.0/opal' > make: *** [Makefile:1800: all-recursive] Error 1 > > I use raspberry pi 2 (raspbian) as my desktop computer, however, I've done > a test in my Solaris 13.1 x86 with Oracle Developer Studio 12.5 and the > result is similar: an error around "all-recursive". I alread tried install > the gnu-make lastest version and didn't work, same thing with dmake and > gmake: error around "all-recursive". > > More information: The .deb (raspbian) packages are working and packages to > solaris too, however I would like to compile the lastest version. > > What did I do wrong? > > > ___ > users mailing > listus...@lists.open-mpi.orghttps://rfd.newmexicoconsortium.org/mailman/listinfo/users > > > > ___ > users mailing list > users@lists.open-mpi.org > https://rfd.newmexicoconsortium.org/mailman/listinfo/users > ___ users mailing list users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/users
[OMPI users] No core dump in some cases
Hello all I run MPI jobs (for test purpose only) on two different 'clusters'. Both 'clusters' have two nodes only, connected back-to-back. The two are very similar, but not identical, both software and hardware wise. Both have ulimit -c set to unlimited. However, only one of the two creates core files when an MPI job crashes. The other creates a text file named something like .80s-,.btr I'd much prefer a core file because that allows me to debug with a lot more options than a static text file with addresses. How do I get a core file in all situations? I am using MPI source from the master branch. Thanks in advance Durga The surgeon general advises you to eat right, exercise regularly and quit ageing.
Re: [OMPI users] mpirun command won't run unless the firewalld daemon is disabled
Hello Llolsten Is there a specific reason you run as root? This practice is discouraged, isn't it? Also, isn't it true that OMPI uses ephemeral (i.e. 'user level, randomly chosen') ports for TCP transport? In that case, how did this ever worked with a firewall enabled? I have, in the past, have faced similar situations, thus I am curious to know the answer as well. Thanks Durga The surgeon general advises you to eat right, exercise regularly and quit ageing. On Mon, May 9, 2016 at 2:31 PM, Llolsten Kaonga wrote: > Hello all, > > > > We’ve been running openmpi for a long time and up to version 1.8.2 and > CentOS 6.7 with commands such as the one below: > > > > usr/local/bin/mpirun --allow-run-as-root --mca btl openib,self,sm --mca > pml ob1 -np 2 -np 8 -hostfile /root/mpi-hosts /usr/local/bin/IMB-MPI1 > > > > To be able to run the above command, we normally just disabled the IPv4 > firewall and SELinux. > > > > We recently made the following updates: > > > > OS: CentOS 7.2 > > IMB: 4.1 > > OpenMPI: 1.10.2 > > > > When we tried to execute the above mpirun command, we got a TCP Broken > Pipe error. There was no IP assignment conflict and eventually, we narrowed > down the problem to the firewall. Disabling the firewalld daemon allows the > command to run to completion. We would prefer not to disable the daemon as > our servers may sometimes be connected to the rest of our subnet. > > > > Are there other options such as perhaps specifying a port (I am guessing, > so specific instructions will be greatly appreciated). > > > > I thank you. > > ___ > users mailing list > us...@open-mpi.org > Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2016/05/29143.php >
Re: [OMPI users] No core dump in some cases
Hi Gus Thanks for your suggestion. But I am not using any resource manager (i.e. I am launching mpirun from the bash shell.). In fact, both of the two clusters I talked about run CentOS 7 and I launch the job the same way on both of these, yet one of them creates standard core files and the other creates the 'btr; files. Strange thing is, I could not find anything on the .btr (= Backtrace?) files on Google, which is any I asked on this forum. Best regards Durga The surgeon general advises you to eat right, exercise regularly and quit ageing. On Mon, May 9, 2016 at 12:04 PM, Gus Correa wrote: > Hi Durga > > Just in case ... > If you're using a resource manager to start the jobs (Torque, etc), > you need to have them set the limits (for coredump size, stacksize, locked > memory size, etc). > This way the jobs will inherit the limits from the > resource manager daemon. > On Torque (which I use) I do this on the pbs_mom daemon > init script (I am still before the systemd era, that lovely POS). > And set the hard/soft limits on /etc/security/limits.conf as well. > > I hope this helps, > Gus Correa > > On 05/07/2016 12:27 PM, Jeff Squyres (jsquyres) wrote: > >> I'm afraid I don't know what a .btr file is -- that is not something that >> is controlled by Open MPI. >> >> You might want to look into your OS settings to see if it has some kind >> of alternate corefile mechanism...? >> >> >> On May 6, 2016, at 8:58 PM, dpchoudh . wrote: >>> >>> Hello all >>> >>> I run MPI jobs (for test purpose only) on two different 'clusters'. Both >>> 'clusters' have two nodes only, connected back-to-back. The two are very >>> similar, but not identical, both software and hardware wise. >>> >>> Both have ulimit -c set to unlimited. However, only one of the two >>> creates core files when an MPI job crashes. The other creates a text file >>> named something like >>> >>> .80s-,.btr >>> >>> I'd much prefer a core file because that allows me to debug with a lot >>> more options than a static text file with addresses. How do I get a core >>> file in all situations? I am using MPI source from the master branch. >>> >>> Thanks in advance >>> Durga >>> >>> The surgeon general advises you to eat right, exercise regularly and >>> quit ageing. >>> ___ >>> users mailing list >>> us...@open-mpi.org >>> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users >>> Link to this post: >>> http://www.open-mpi.org/community/lists/users/2016/05/29124.php >>> >> >> >> > ___ > users mailing list > us...@open-mpi.org > Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2016/05/29141.php >
Re: [OMPI users] No core dump in some cases
Hello Nathan Thank you for your response. Could you please be more specific? Adding the following after MPI_Init() does not seem to make a difference. MPI_Init(&argc, &argv); * signal(SIGABRT, SIG_DFL); signal(SIGTERM, SIG_DFL);* I also find it puzzling that nearly identical OMPI distro running on a different machine shows different behaviour. Best regards Durga The surgeon general advises you to eat right, exercise regularly and quit ageing. On Tue, May 10, 2016 at 10:02 AM, Hjelm, Nathan Thomas wrote: > btr files are indeed created by open mpi's backtrace mechanism. I think we > should revisit it at some point but for now the only effective way i have > found to prevent it is to restore the default signal handlers after > MPI_Init. > > Excuse the quoting style. Good sucks. > > > > From: users on behalf of dpchoudh . > Sent: Monday, May 09, 2016 2:59:37 PM > To: Open MPI Users > Subject: Re: [OMPI users] No core dump in some cases > > Hi Gus > > Thanks for your suggestion. But I am not using any resource manager (i.e. > I am launching mpirun from the bash shell.). In fact, both of the two > clusters I talked about run CentOS 7 and I launch the job the same way on > both of these, yet one of them creates standard core files and the other > creates the 'btr; files. Strange thing is, I could not find anything on the > .btr (= Backtrace?) files on Google, which is any I asked on this forum. > > Best regards > Durga > > The surgeon general advises you to eat right, exercise regularly and quit > ageing. > > On Mon, May 9, 2016 at 12:04 PM, Gus Correa g...@ldeo.columbia.edu>> wrote: > Hi Durga > > Just in case ... > If you're using a resource manager to start the jobs (Torque, etc), > you need to have them set the limits (for coredump size, stacksize, locked > memory size, etc). > This way the jobs will inherit the limits from the > resource manager daemon. > On Torque (which I use) I do this on the pbs_mom daemon > init script (I am still before the systemd era, that lovely POS). > And set the hard/soft limits on /etc/security/limits.conf as well. > > I hope this helps, > Gus Correa > > On 05/07/2016 12:27 PM, Jeff Squyres (jsquyres) wrote: > I'm afraid I don't know what a .btr file is -- that is not something that > is controlled by Open MPI. > > You might want to look into your OS settings to see if it has some kind of > alternate corefile mechanism...? > > > On May 6, 2016, at 8:58 PM, dpchoudh . dpcho...@gmail.com>> wrote: > > Hello all > > I run MPI jobs (for test purpose only) on two different 'clusters'. Both > 'clusters' have two nodes only, connected back-to-back. The two are very > similar, but not identical, both software and hardware wise. > > Both have ulimit -c set to unlimited. However, only one of the two creates > core files when an MPI job crashes. The other creates a text file named > something like > > .80s-,.btr > > I'd much prefer a core file because that allows me to debug with a lot > more options than a static text file with addresses. How do I get a core > file in all situations? I am using MPI source from the master branch. > > Thanks in advance > Durga > > The surgeon general advises you to eat right, exercise regularly and quit > ageing. > ___ > users mailing list > us...@open-mpi.org<mailto:us...@open-mpi.org> > Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2016/05/29124.php > > > > ___ > users mailing list > us...@open-mpi.org<mailto:us...@open-mpi.org> > Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2016/05/29141.php > > ___ > users mailing list > us...@open-mpi.org > Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2016/05/29154.php >
Re: [OMPI users] No core dump in some cases
Hello Gilles Thank you for the advice. However, that did not seem to make any difference. Here is what I did (on the cluster that generates .btr files for core dumps): [durga@smallMPI git]$ ompi_info --all | grep opal_signal MCA opal base: parameter "opal_signal" (current value: "6,7,8,11", data source: default, level: 3 user/all, type: string) [durga@smallMPI git]$ According to , signals 6.7,8,11 are this: #defineSIGABRT6/* Abort (ANSI). */ #defineSIGBUS7/* BUS error (4.2 BSD). */ #defineSIGFPE8/* Floating-point exception (ANSI). */ #defineSIGSEGV11/* Segmentation violation (ANSI). */ And thus I added the following just after MPI_Init() MPI_Init(&argc, &argv); signal(SIGABRT, SIG_DFL); signal(SIGBUS, SIG_DFL); signal(SIGFPE, SIG_DFL); signal(SIGSEGV, SIG_DFL); signal(SIGTERM, SIG_DFL); (I added the 'SIGTERM' part later, just in case it would make a difference; i didn't) The resulting code still generates .btr files instead of core files. It looks like the 'execinfo' MCA component is being used as the backtrace mechanism: [durga@smallMPI git]$ ompi_info | grep backtrace MCA backtrace: execinfo (MCA v2.1.0, API v2.0.0, Component v3.0.0) However, I could not find any way to choose 'none' instead of 'execinfo' And the strange thing is, on the cluster where regular core dump is happening, the output of $ ompi_info | grep backtrace is identical to the above. (Which kind of makes sense because they were created from the same source with the same configure options.) Sorry to harp on this, but without a core file it is hard to debug the application (e.g. examine stack variables). Thank you Durga The surgeon general advises you to eat right, exercise regularly and quit ageing. On Wed, May 11, 2016 at 3:37 AM, Gilles Gouaillardet < gilles.gouaillar...@gmail.com> wrote: > Durga, > > you might wanna try to restore the signal handler for other signals as well > (SIGSEGV, SIGBUS, ...) > ompi_info --all | grep opal_signal > does list the signal you should restore the handler > > > only one backtrace component is built (out of several candidates : > execinfo, none, printstack) > nm -l libopen-pal.so | grep backtrace > will hint you which component was built > > your two similar distros might have different backtrace component > > > > Gus, > > btr is a plain text file with a back trace "ala" gdb > > > > Nathan, > > i did a 'grep btr' and could not find anything :-( > opal_backtrace_buffer and opal_backtrace_print are only used with stderr. > so i am puzzled who creates the tracefile name and where ... > also, no stack is printed by default unless opal_abort_print_stack is true > > Cheers, > > Gilles > > > On Wed, May 11, 2016 at 3:43 PM, dpchoudh . wrote: > > Hello Nathan > > > > Thank you for your response. Could you please be more specific? Adding > the > > following after MPI_Init() does not seem to make a difference. > > > > MPI_Init(&argc, &argv); > > signal(SIGABRT, SIG_DFL); > > signal(SIGTERM, SIG_DFL); > > > > I also find it puzzling that nearly identical OMPI distro running on a > > different machine shows different behaviour. > > > > Best regards > > Durga > > > > The surgeon general advises you to eat right, exercise regularly and quit > > ageing. > > > > On Tue, May 10, 2016 at 10:02 AM, Hjelm, Nathan Thomas > > wrote: > >> > >> btr files are indeed created by open mpi's backtrace mechanism. I think > we > >> should revisit it at some point but for now the only effective way i > have > >> found to prevent it is to restore the default signal handlers after > >> MPI_Init. > >> > >> Excuse the quoting style. Good sucks. > >> > >> > >> > >> From: users on behalf of dpchoudh . > >> Sent: Monday, May 09, 2016 2:59:37 PM > >> To: Open MPI Users > >> Subject: Re: [OMPI users] No core dump in some cases > >> > >> Hi Gus > >> > >> Thanks for your suggestion. But I am not using any resource manager > (i.e. > >> I am launching mpirun from the bash shell.). In fact, both of the two > >> clusters I talked about run CentOS 7 and I launch the job the same way > on > >> both of these, yet one of them creates standard core files and the other > >> creates the 'btr; files. Strange thing is, I could not find anything on > the > >> .btr (= Backtrace?) files on Google, which is any I as
Re: [OMPI users] No core dump in some cases
est[0x400829]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7fdfe2d8eb15]
./btrtest[0x4006d9]
after MPI_Init : -1 -1
after MPI_Init : -1 -1
---
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
---
--
mpirun detected that one or more processes exited with non-zero status,
thus causing
the job to be terminated. The first process to do so was:
Process name: [[9384,1],1]
Exit code:1
--
[durga@smallMPI ~]$ ulimit -a
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 216524
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size(512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 4096
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
[durga@smallMPI ~]$
I do realize that my setup is very unusual (I am a quasi-developer of MPI
whereas most other folks in this list are likely end-users), but somehow
just disabling this 'execinfo' MCA would allow me to make progress (and
also find out why/where MPI_Init() is crashing!). Is there any way I can do
that?
Thank you
Durga
The surgeon general advises you to eat right, exercise regularly and quit
ageing.
On Wed, May 11, 2016 at 8:58 PM, Gilles Gouaillardet
wrote:
> Are you sure ulimit -c unlimited is *really* applied on all hosts
>
>
> can you please run the simple program below and confirm that ?
>
>
> Cheers,
>
>
> Gilles
>
>
> #include
> #include
> #include
> #include
>
> int main(int argc, char *argv[]) {
> struct rlimit rlim;
> char * c = (char *)0;
> getrlimit(RLIMIT_CORE, &rlim);
> printf ("before MPI_Init : %d %d\n", rlim.rlim_cur, rlim.rlim_max);
> MPI_Init(&argc, &argv);
> getrlimit(RLIMIT_CORE, &rlim);
> printf ("after MPI_Init : %d %d\n", rlim.rlim_cur, rlim.rlim_max);
> *c = 0;
> MPI_Finalize();
> return 0;
> }
>
>
> On 5/12/2016 4:22 AM, dpchoudh . wrote:
>
> Hello Gilles
>
> Thank you for the advice. However, that did not seem to make any
> difference. Here is what I did (on the cluster that generates .btr files
> for core dumps):
>
> [durga@smallMPI git]$ ompi_info --all | grep opal_signal
>MCA opal base: parameter "opal_signal" (current value:
> "6,7,8,11", data source: default, level: 3 user/all, type: string)
> [durga@smallMPI git]$
>
>
> According to , signals 6.7,8,11 are this:
>
> #defineSIGABRT6/* Abort (ANSI). */
> #defineSIGBUS7/* BUS error (4.2 BSD). */
> #defineSIGFPE8/* Floating-point exception (ANSI). */
> #defineSIGSEGV11/* Segmentation violation (ANSI). */
>
> And thus I added the following just after MPI_Init()
>
> MPI_Init(&argc, &argv);
> signal(SIGABRT, SIG_DFL);
> signal(SIGBUS, SIG_DFL);
> signal(SIGFPE, SIG_DFL);
> signal(SIGSEGV, SIG_DFL);
> signal(SIGTERM, SIG_DFL);
>
> (I added the 'SIGTERM' part later, just in case it would make a
> difference; i didn't)
>
> The resulting code still generates .btr files instead of core files.
>
> It looks like the 'execinfo' MCA component is being used as the backtrace
> mechanism:
>
> [durga@smallMPI git]$ ompi_info | grep backtrace
>MCA backtrace: execinfo (MCA v2.1.0, API v2.0.0, Component
> v3.0.0)
>
> However, I could not find any way to choose 'none' instead of 'execinfo'
>
> And the strange thing is, on the cluster where regular core dump is
> happening, the output of
> $ ompi_info | grep backtrace
> is identical to the above. (Which kind of makes sense because they were
> created from the same source with the same configure options.)
>
> Sorry to harp on this, but without a core file it is hard to debug the
> application (e.g. examine stack variables).
>
> Thank you
> Durga
>
>
> The surgeon general advises you to eat right, exercise regularly and quit
> ageing.
>
> On Wed, May 11, 2016 at 3:37 AM, Gilles Gouaillardet <
> gilles.
Re: [OMPI users] No core dump in some cases
Hello Gilles and all Here is an update: looks like I have root-caused it: Disabling MPI's signal handlers after MPI_Init() AND mentioning both the PML AND BTL explicitly does generate a core dump. (Note that just mentioning -mca pml ob1 alone does not do it, which I find strange) I believe there was a similar post recently. To be more explicit, the following command generates a core file mpirun -np 2 -mca pml ob1 -mca btl self,sm ./btrtest only if I have the following 4 lines after MPI_Init() signal(SIGABRT, SIG_DFL); signal(SIGBUS, SIG_DFL); signal(SIGFPE, SIG_DFL); signal(SIGSEGV, SIG_DFL); So, whatever default MCA parameters OMPI is choosing seems to be incorrect, on a machine that has the extra NICs, link off. That still leaves open the question as to why the default signal handlers are not required in the other cluster. To investigate what parameters OMPI is choosing my default, I did the following: mpirun -np 2 -mca pml_base_verbose 100 -mca bml_base_verbose 100 -mca btl_base_verboe 100 ./btrtest which did not seem to return anything very insightful: before MPI_Init : -1 -1 before MPI_Init : -1 -1 [smallMPI:15125] mca: base: components_register: registering framework bml components [smallMPI:15125] mca: base: components_register: found loaded component r2 [smallMPI:15125] mca: base: components_register: component r2 register function successful [smallMPI:15125] mca: base: components_open: opening bml components [smallMPI:15125] mca: base: components_open: found loaded component r2 [smallMPI:15125] mca: base: components_open: component r2 open function successful [smallMPI:15126] mca: base: components_register: registering framework bml components [smallMPI:15126] mca: base: components_register: found loaded component r2 [smallMPI:15126] mca: base: components_register: component r2 register function successful [smallMPI:15126] mca: base: components_open: opening bml components [smallMPI:15126] mca: base: components_open: found loaded component r2 [smallMPI:15126] mca: base: components_open: component r2 open function successful btrtest:15125 terminated with signal 11 at PC=7ff5d13927d8 SP=7ffe4fe833d8. Backtrace: /lib64/libc.so.6(+0x3ba7d8)[0x7ff5d13927d8] btrtest:15126 terminated with signal 11 at PC=7f568715c7d8 SP=7ffe5057b518. Backtrace: /lib64/libc.so.6(+0x3ba7d8)[0x7f568715c7d8] I'll be more than happy to run any command in this machine that might help the developers narrow down the issue; please let me know. Thank you Durga The surgeon general advises you to eat right, exercise regularly and quit ageing. On Wed, May 11, 2016 at 10:34 PM, dpchoudh . wrote: > Hello Gilles > > Thank you for your continued support. With your help, I have a better > understanding of what is happening. Here are the details. > > 1. Yes, I am sure that ulimit -c is 'unlimited' (and for the test in > question, I am running it on a single node, so there are no other nodes) > > 2. The command I am running is possibly the simplest MPI command: > mpirun -np 2 > > It looks to me, after running your test code, that what is crashing is > MPI_Init() itself. The output from your code (I called it 'btrtest') is as > follows: > > [durga@smallMPI ~]$ mpirun -np 2 ./btrtest > before MPI_Init : -1 -1 > before MPI_Init : -1 -1 > > btrtest:7275 terminated with signal 11 at PC=7f401f49e7d8 > SP=7ffec47e7578. Backtrace: > /lib64/libc.so.6(+0x3ba7d8)[0x7f401f49e7d8] > > btrtest:7274 terminated with signal 11 at PC=7f1ba21897d8 > SP=7ffc516ac8d8. Backtrace: > /lib64/libc.so.6(+0x3ba7d8)[0x7f1ba21897d8] > --- > Primary job terminated normally, but 1 process returned > a non-zero exit code. Per user-direction, the job has been aborted. > --- > -- > mpirun detected that one or more processes exited with non-zero status, > thus causing > the job to be terminated. The first process to do so was: > > Process name: [[7936,1],1] > Exit code:1 > -- > > So obviously the code does not make it past MPI_Init() > > This is an issue that I have been observing for quite a while in different > forms and have reported on the forum a few times also. Let me elaborate: > > Both my 'well-behaving' and crashing clusters run CentOS 7 (the crashing > one has the latest updates, the well-behaving one does not as I am not > allowed to apply updates on that). They both have OMPI, from the master > branch, compiled from the source. Both consist of 64 bit Dell servers, > although not identical models (I doubt if that matters) > > The only significant difference between the two is this: > > The w
Re: [OMPI users] No core dump in some cases
Hello Ralph I have the latest from master, but I still see this old behaviour. Is your code available on master? Thank you Durga The surgeon general advises you to eat right, exercise regularly and quit ageing. On Wed, May 11, 2016 at 10:53 PM, Ralph Castain wrote: > This is a known problem - I committed the fix for PSM with a link down > just today. > > > On May 11, 2016, at 7:34 PM, dpchoudh . wrote: > > Hello Gilles > > Thank you for your continued support. With your help, I have a better > understanding of what is happening. Here are the details. > > 1. Yes, I am sure that ulimit -c is 'unlimited' (and for the test in > question, I am running it on a single node, so there are no other nodes) > > 2. The command I am running is possibly the simplest MPI command: > mpirun -np 2 > > It looks to me, after running your test code, that what is crashing is > MPI_Init() itself. The output from your code (I called it 'btrtest') is as > follows: > > [durga@smallMPI ~]$ mpirun -np 2 ./btrtest > before MPI_Init : -1 -1 > before MPI_Init : -1 -1 > > btrtest:7275 terminated with signal 11 at PC=7f401f49e7d8 > SP=7ffec47e7578. Backtrace: > /lib64/libc.so.6(+0x3ba7d8)[0x7f401f49e7d8] > > btrtest:7274 terminated with signal 11 at PC=7f1ba21897d8 > SP=7ffc516ac8d8. Backtrace: > /lib64/libc.so.6(+0x3ba7d8)[0x7f1ba21897d8] > --- > Primary job terminated normally, but 1 process returned > a non-zero exit code. Per user-direction, the job has been aborted. > --- > -- > mpirun detected that one or more processes exited with non-zero status, > thus causing > the job to be terminated. The first process to do so was: > > Process name: [[7936,1],1] > Exit code:1 > -- > > So obviously the code does not make it past MPI_Init() > > This is an issue that I have been observing for quite a while in different > forms and have reported on the forum a few times also. Let me elaborate: > > Both my 'well-behaving' and crashing clusters run CentOS 7 (the crashing > one has the latest updates, the well-behaving one does not as I am not > allowed to apply updates on that). They both have OMPI, from the master > branch, compiled from the source. Both consist of 64 bit Dell servers, > although not identical models (I doubt if that matters) > > The only significant difference between the two is this: > > The well behaved one (if it does core dump, that is because there is a bug > in the MPI app) has very simple network hardware: two different NICs (one > Broadcom GbE, one proprietary NIC that is currently being exposed as an IP > interface). There is no RDMA capability there at all. > > The crashing one have 4 different NICs: > 1. Broadcom GbE > 2. Chelsio T3 based 10Gb iWARP NIC > 3. QLogic 20Gb Infiniband (PSM capable) > 4. LSI logic Fibre channel > > In this situation, WITH ALL BUT THE GbE LINK DOWN (the GbE connects the > machine to the WAN link), it seems just the presence of these NICs matter. > > Here are the various commands and outputs: > > [durga@smallMPI ~]$ mpirun -np 2 ./btrtest > before MPI_Init : -1 -1 > before MPI_Init : -1 -1 > > btrtest:10032 terminated with signal 11 at PC=7f6897c197d8 > SP=7ffcae2b2ef8. Backtrace: > /lib64/libc.so.6(+0x3ba7d8)[0x7f6897c197d8] > > btrtest:10033 terminated with signal 11 at PC=7fb035c3e7d8 > SP=7ffe61a92088. Backtrace: > /lib64/libc.so.6(+0x3ba7d8)[0x7fb035c3e7d8] > --- > Primary job terminated normally, but 1 process returned > a non-zero exit code. Per user-direction, the job has been aborted. > --- > -- > mpirun detected that one or more processes exited with non-zero status, > thus causing > the job to be terminated. The first process to do so was: > > Process name: [[9294,1],0] > Exit code:1 > -- > > [durga@smallMPI ~]$ mpirun -np 2 -mca pml ob1 ./btrtest > before MPI_Init : -1 -1 > before MPI_Init : -1 -1 > > btrtest:10076 terminated with signal 11 at PC=7fa92d20b7d8 > SP=7ffebb106028. Backtrace: > /lib64/libc.so.6(+0x3ba7d8)[0x7fa92d20b7d8] > > btrtest:10077 terminated with signal 11 at PC=7f5012fa57d8 > SP=7ffea4f4fdf8. Backtrace: > /lib64/libc.so.6(+0x3ba7d8)[0x7f5012fa57d8] >
Re: [OMPI users] No core dump in some cases
Hello Gilles Mystery solved! In fact, this one line is exactly what was needed!! It turns out the OMPI signal handlers are irrelevant. (i.e. don't make any difference when this env variable is set) This explains: 1. The difference in the behaviour in the two clusters (one has PSM, the other does not) 2. Why you couldn't find where in OMPI code the .btr files are being generated (looks like they are being generated in PSM library) And, now that I can get a core file (finally!), I can present the back trace where the crash in MPI_Init() is happening. It is as follows: #0 0x7f1c114977d8 in main_arena () from /lib64/libc.so.6 #1 0x7f1c106719ac in device_destruct (device=0x1c85b70) at btl_openib_component.c:985 #2 0x7f1c1066d0ae in opal_obj_run_destructors (object=0x1c85b70) at ../../../../opal/class/opal_object.h:460 #3 0x7f1c10674d3c in init_one_device (btl_list=0x7ffd00dada50, ib_dev=0x1c85430) at btl_openib_component.c:2255 #4 0x7f1c10676800 in btl_openib_component_init (num_btl_modules=0x7ffd00dadb80, enable_progress_threads=true, enable_mpi_threads=false) at btl_openib_component.c:2752 #5 0x7f1c10633971 in mca_btl_base_select (enable_progress_threads=true, enable_mpi_threads=false) at base/btl_base_select.c:110 #6 0x7f1c117fb0a0 in mca_bml_r2_component_init (priority=0x7ffd00dadc4c, enable_progress_threads=true, enable_mpi_threads=false) at bml_r2_component.c:86 #7 0x7f1c117f8033 in mca_bml_base_init (enable_progress_threads=true, enable_mpi_threads=false) at base/bml_base_init.c:74 #8 0x7f1c1173f675 in ompi_mpi_init (argc=1, argv=0x7ffd00dae008, requested=0, provided=0x7ffd00daddbc) at runtime/ompi_mpi_init.c:590 #9 0x7f1c1177c8b7 in PMPI_Init (argc=0x7ffd00daddfc, argv=0x7ffd00daddf0) at pinit.c:66 #10 0x00400aa0 in main (argc=1, argv=0x7ffd00dae008) at mpitest.c:17 This is with the absolute latest code from master. Thanks everyone for their help. Durga The surgeon general advises you to eat right, exercise regularly and quit ageing. On Wed, May 11, 2016 at 10:55 PM, Gilles Gouaillardet wrote: > Note the psm library sets its own signal handler, possibly after the > OpenMPI one. > > that can be disabled by > > export IPATH_NO_BACKTRACE=1 > > Cheers, > > Gilles > > > On 5/12/2016 11:34 AM, dpchoudh . wrote: > > Hello Gilles > > Thank you for your continued support. With your help, I have a better > understanding of what is happening. Here are the details. > > 1. Yes, I am sure that ulimit -c is 'unlimited' (and for the test in > question, I am running it on a single node, so there are no other nodes) > > 2. The command I am running is possibly the simplest MPI command: > mpirun -np 2 > > It looks to me, after running your test code, that what is crashing is > MPI_Init() itself. The output from your code (I called it 'btrtest') is as > follows: > > [durga@smallMPI ~]$ mpirun -np 2 ./btrtest > before MPI_Init : -1 -1 > before MPI_Init : -1 -1 > > btrtest:7275 terminated with signal 11 at PC=7f401f49e7d8 > SP=7ffec47e7578. Backtrace: > /lib64/libc.so.6(+0x3ba7d8)[0x7f401f49e7d8] > > btrtest:7274 terminated with signal 11 at PC=7f1ba21897d8 > SP=7ffc516ac8d8. Backtrace: > /lib64/libc.so.6(+0x3ba7d8)[0x7f1ba21897d8] > --- > Primary job terminated normally, but 1 process returned > a non-zero exit code. Per user-direction, the job has been aborted. > --- > -- > mpirun detected that one or more processes exited with non-zero status, > thus causing > the job to be terminated. The first process to do so was: > > Process name: [[7936,1],1] > Exit code:1 > -- > > So obviously the code does not make it past MPI_Init() > > This is an issue that I have been observing for quite a while in different > forms and have reported on the forum a few times also. Let me elaborate: > > Both my 'well-behaving' and crashing clusters run CentOS 7 (the crashing > one has the latest updates, the well-behaving one does not as I am not > allowed to apply updates on that). They both have OMPI, from the master > branch, compiled from the source. Both consist of 64 bit Dell servers, > although not identical models (I doubt if that matters) > > The only significant difference between the two is this: > > The well behaved one (if it does core dump, that is because there is a bug > in the MPI app) has very simple network hardware: two different NICs (one > Broadcom GbE, one proprietary NIC that is currently being exposed as an IP > interface). There is no RDMA cap
Re: [OMPI users] No core dump in some cases
Sorry for belabouring on this, but this (hopefully final!) piece of information might be of interest to the developers: There must be a reason why PSM is installing its signal handlers; often this is done to modify the permission of a page in response to a SEGV and attempt access again. By disabling the handlers, I am preventing the library from doing that, and here is what it tells me: [durga@smallMPI ~]$ mpirun -np 2 ./mpitest [smallMPI:20496] *** Process received signal *** [smallMPI:20496] Signal: Segmentation fault (11) [smallMPI:20496] Signal code: Invalid permissions (2) [smallMPI:20496] Failing at address: 0x7f0b2fdb57d8 [smallMPI:20496] [ 0] /lib64/libpthread.so.0(+0xf100)[0x7f0b2fdcb100] [smallMPI:20496] [ 1] /lib64/libc.so.6(+0x3ba7d8)[0x7f0b2fdb57d8] [smallMPI:20496] *** End of error message *** [smallMPI:20497] *** Process received signal *** [smallMPI:20497] Signal: Segmentation fault (11) [smallMPI:20497] Signal code: Invalid permissions (2) [smallMPI:20497] Failing at address: 0x7fbfe2b387d8 [smallMPI:20497] [ 0] /lib64/libpthread.so.0(+0xf100)[0x7fbfe2b4e100] [smallMPI:20497] [ 1] /lib64/libc.so.6(+0x3ba7d8)[0x7fbfe2b387d8] [smallMPI:20497] *** End of error message *** --- Primary job terminated normally, but 1 process returned a non-zero exit code. Per user-direction, the job has been aborted. --- However, even without disabling it, it crashes anyway, as follows: unset IPATH_NO_BACKTRACE [durga@smallMPI ~]$ mpirun -np 2 ./mpitest mpitest:22009 terminated with signal 11 at PC=7f908bb2a7d8 SP=7ffebb4ee5b8. Backtrace: /lib64/libc.so.6(+0x3ba7d8)[0x7f908bb2a7d8] mpitest:22010 terminated with signal 11 at PC=7f7a2caa67d8 SP=7ffd73dec3e8. Backtrace: /lib64/libc.so.6(+0x3ba7d8)[0x7f7a2caa67d8] The PC is at a different location but I do not have any more information without a core file. It seems like some interaction between OMPI and PSM library is incorrect. I'll let the developers figure it out :-) Thanks Durga The surgeon general advises you to eat right, exercise regularly and quit ageing. On Wed, May 11, 2016 at 11:23 PM, dpchoudh . wrote: > Hello Gilles > > Mystery solved! In fact, this one line is exactly what was needed!! It > turns out the OMPI signal handlers are irrelevant. (i.e. don't make any > difference when this env variable is set) > > This explains: > > 1. The difference in the behaviour in the two clusters (one has PSM, the > other does not) > 2. Why you couldn't find where in OMPI code the .btr files are being > generated (looks like they are being generated in PSM library) > > And, now that I can get a core file (finally!), I can present the back > trace where the crash in MPI_Init() is happening. It is as follows: > > #0 0x7f1c114977d8 in main_arena () from /lib64/libc.so.6 > #1 0x7f1c106719ac in device_destruct (device=0x1c85b70) at > btl_openib_component.c:985 > #2 0x7f1c1066d0ae in opal_obj_run_destructors (object=0x1c85b70) at > ../../../../opal/class/opal_object.h:460 > #3 0x7f1c10674d3c in init_one_device (btl_list=0x7ffd00dada50, > ib_dev=0x1c85430) at btl_openib_component.c:2255 > #4 0x7f1c10676800 in btl_openib_component_init > (num_btl_modules=0x7ffd00dadb80, enable_progress_threads=true, > enable_mpi_threads=false) > at btl_openib_component.c:2752 > #5 0x7f1c10633971 in mca_btl_base_select > (enable_progress_threads=true, enable_mpi_threads=false) at > base/btl_base_select.c:110 > #6 0x7f1c117fb0a0 in mca_bml_r2_component_init > (priority=0x7ffd00dadc4c, enable_progress_threads=true, > enable_mpi_threads=false) > at bml_r2_component.c:86 > #7 0x7f1c117f8033 in mca_bml_base_init (enable_progress_threads=true, > enable_mpi_threads=false) at base/bml_base_init.c:74 > #8 0x7f1c1173f675 in ompi_mpi_init (argc=1, argv=0x7ffd00dae008, > requested=0, provided=0x7ffd00daddbc) at runtime/ompi_mpi_init.c:590 > #9 0x7f1c1177c8b7 in PMPI_Init (argc=0x7ffd00daddfc, > argv=0x7ffd00daddf0) at pinit.c:66 > #10 0x00400aa0 in main (argc=1, argv=0x7ffd00dae008) at > mpitest.c:17 > > This is with the absolute latest code from master. > > Thanks everyone for their help. > > Durga > > The surgeon general advises you to eat right, exercise regularly and quit > ageing. > > On Wed, May 11, 2016 at 10:55 PM, Gilles Gouaillardet > wrote: > >> Note the psm library sets its own signal handler, possibly after the >> OpenMPI one. >> >> that can be disabled by >> >> export IPATH_NO_BACKTRACE=1 >> >> Cheers, >> >> Gilles >> >> >> On 5/12/2016 11:34 AM, dpchoudh . wrote: >> >> Hello Gilles >> >> Thank you for your continued supp
Re: [OMPI users] No core dump in some cases
*not* to be removed in the destructor. > that means that if the signal handler is invoked *after* the pml MTL > is unloaded, a crash will likely occur because the psm signal handler > is likely pointing to unmapped memory. > > on top of that, there used to be a bug if some PSM device is detected > but with no link (e.g. crash) > with the latest ompi master, this bug should be fixed (e.g. no crash) > this means the PSM mtl should disqualify itself if there is no link on > any of the PSM ports, so, unless your infinipath library is fixed or > you configure'd with --disable-dlopen, you will run into trouble if > the ipath signal handler is invoked. > > can you confirm you have the latest master and there is no link on > your ipath device ? > > what does > grep ACTIVE /sys/class/infiniband/qib*/ports/*/state > returns ? > > if you did not configure with --disable-dlopen *and* you do not need > the psm mtl, you can > mpirun --mca mtl ^psm ... > or if you do not need any mtl at all > mpirun --mca pml ob1 ... > should be enough > > Cheers, > > Gilles > > commit 4d026e223ce717345712e669d26f78ed49082df6 > Merge: f8facb1 4071719 > Author: rhc54 > Date: Wed May 11 17:43:17 2016 -0700 > > Merge pull request #1661 from matcabral/master > > PSM and PSM2 MTLs to detect drivers and link > > > On Thu, May 12, 2016 at 12:42 PM, dpchoudh . wrote: > > Sorry for belabouring on this, but this (hopefully final!) piece of > > information might be of interest to the developers: > > > > There must be a reason why PSM is installing its signal handlers; often > this > > is done to modify the permission of a page in response to a SEGV and > attempt > > access again. By disabling the handlers, I am preventing the library from > > doing that, and here is what it tells me: > > > > [durga@smallMPI ~]$ mpirun -np 2 ./mpitest > > [smallMPI:20496] *** Process received signal *** > > [smallMPI:20496] Signal: Segmentation fault (11) > > [smallMPI:20496] Signal code: Invalid permissions (2) > > [smallMPI:20496] Failing at address: 0x7f0b2fdb57d8 > > [smallMPI:20496] [ 0] /lib64/libpthread.so.0(+0xf100)[0x7f0b2fdcb100] > > [smallMPI:20496] [ 1] /lib64/libc.so.6(+0x3ba7d8)[0x7f0b2fdb57d8] > > [smallMPI:20496] *** End of error message *** > > [smallMPI:20497] *** Process received signal *** > > [smallMPI:20497] Signal: Segmentation fault (11) > > [smallMPI:20497] Signal code: Invalid permissions (2) > > [smallMPI:20497] Failing at address: 0x7fbfe2b387d8 > > [smallMPI:20497] [ 0] /lib64/libpthread.so.0(+0xf100)[0x7fbfe2b4e100] > > [smallMPI:20497] [ 1] /lib64/libc.so.6(+0x3ba7d8)[0x7fbfe2b387d8] > > [smallMPI:20497] *** End of error message *** > > --- > > Primary job terminated normally, but 1 process returned > > a non-zero exit code. Per user-direction, the job has been aborted. > > --- > > > > However, even without disabling it, it crashes anyway, as follows: > > > > unset IPATH_NO_BACKTRACE > > > > [durga@smallMPI ~]$ mpirun -np 2 ./mpitest > > > > mpitest:22009 terminated with signal 11 at PC=7f908bb2a7d8 > SP=7ffebb4ee5b8. > > Backtrace: > > /lib64/libc.so.6(+0x3ba7d8)[0x7f908bb2a7d8] > > > > mpitest:22010 terminated with signal 11 at PC=7f7a2caa67d8 > SP=7ffd73dec3e8. > > Backtrace: > > /lib64/libc.so.6(+0x3ba7d8)[0x7f7a2caa67d8] > > > > The PC is at a different location but I do not have any more information > > without a core file. > > > > It seems like some interaction between OMPI and PSM library is incorrect. > > I'll let the developers figure it out :-) > > > > > > Thanks > > Durga > > > > > > > > > > The surgeon general advises you to eat right, exercise regularly and quit > > ageing. > > > > On Wed, May 11, 2016 at 11:23 PM, dpchoudh . wrote: > >> > >> Hello Gilles > >> > >> Mystery solved! In fact, this one line is exactly what was needed!! It > >> turns out the OMPI signal handlers are irrelevant. (i.e. don't make any > >> difference when this env variable is set) > >> > >> This explains: > >> > >> 1. The difference in the behaviour in the two clusters (one has PSM, the > >> other does not) > >> 2. Why you couldn't find where in OMPI code the .btr files are being > >> generated (looks like they are being generated in PSM library) > >> > >> And, now that I can get a core file
Re: [OMPI users] No core dump in some cases
If you configure with --disable-dlopen, then libinfinipath.so is slurped and hence the infinipath signal handler is always set, even if you disable the psm mtl or choose to only use the ob1 pml. if you do not configure with --disable-dlopen, then the infinipath signal handler is set when mca_mtl_psm.so is loaded. and it is not loaded if it is disabled or if only ob1 is used. Aah, I see. But you said that this was recently fixed, right? (I mean, the signal handlers are now uninstalled if PSM is unloaded). I do have the latest from master. I ran your patches, and *both* of them fix the crash. In case it is useful, I am attaching the console output after applying the patch (the output from the app proper is omitted.) [durga@smallMPI ~]$ mpirun -np 2 ./mpitest -- WARNING: There is at least non-excluded one OpenFabrics device found, but there are no active ports detected (or Open MPI was unable to use them). This is most certainly not what you wanted. Check your cables, subnet manager configuration, etc. The openib BTL will be ignored for this job. Local host: smallMPI -- smallMPI.26487PSM found 0 available contexts on InfiniPath device(s). (err=21) smallMPI.26488PSM found 0 available contexts on InfiniPath device(s). (err=21) [durga@smallMPI ~]$ mpirun -np 2 ./mpitest -- WARNING: There is at least non-excluded one OpenFabrics device found, but there are no active ports detected (or Open MPI was unable to use them). This is most certainly not what you wanted. Check your cables, subnet manager configuration, etc. The openib BTL will be ignored for this job. Local host: smallMPI -- smallMPI.7486PSM found 0 available contexts on InfiniPath device(s). (err=21) smallMPI.7487PSM found 0 available contexts on InfiniPath device(s). (err=21) The surgeon general advises you to eat right, exercise regularly and quit ageing. On Thu, May 12, 2016 at 4:29 AM, Gilles Gouaillardet wrote: > If you configure with --disable-dlopen, then libinfinipath.so is slurped > and hence the infinipath signal handler is always set, even if you disable > the psm mtl or choose to only use the ob1 pml. > > if you do not configure with --disable-dlopen, then the infinipath signal > handler is set when mca_mtl_psm.so is loaded. and it is not loaded if it is > disabled or if only ob1 is used. > > it seems some verbs destructors are called twice here. > > can you please give the attached patches a try ? > > /* they are exclusive, e.g. you should only apply one at a time */ > > > Cheers, > > > Gilles > On 5/12/2016 4:54 PM, dpchoudh . wrote: > > Hello Gilles > > I am not sure if I understand you correctly, but let me answer based on > what I think you mean: > > > the infinipath signal handler only dump the stack (into a .btr file, yeah > !) > so if your application crashes without it, you should examine the core > file and see what is going wrong. > > > If this is true, then there is a bug in OMPI proper, since it is crashing > inside MPI_Init(). Here is the stack: > > (gdb) bt > #0 0x7ff3104ac7d8 in main_arena () from /lib64/libc.so.6 > #1 0x7ff30f6869ac in device_destruct (device=0x1284b30) at > btl_openib_component.c:985 > #2 0x7ff30f6820ae in opal_obj_run_destructors (object=0x1284b30) at > ../../../../opal/class/opal_object.h:460 > #3 0x7ff30f689d3c in init_one_device (btl_list=0x7fff96c3a200, > ib_dev=0x12843f0) at btl_openib_component.c:2255 > #4 0x7ff30f68b800 in btl_openib_component_init > (num_btl_modules=0x7fff96c3a330, enable_progress_threads=true, > enable_mpi_threads=false) at btl_openib_component.c:2752 > #5 0x7ff30f648971 in mca_btl_base_select > (enable_progress_threads=true, enable_mpi_threads=false) at > base/btl_base_select.c:110 > #6 0x7ff3108100a0 in mca_bml_r2_component_init > (priority=0x7fff96c3a3fc, enable_progress_threads=true, > enable_mpi_threads=false) > at bml_r2_component.c:86 > #7 0x7ff31080d033 in mca_bml_base_init (enable_progress_threads=true, > enable_mpi_threads=false) at base/bml_base_init.c:74 > #8 0x7ff310754675 in ompi_mpi_init (argc=1, argv=0x7fff96c3a7b8, > requested=0, provided=0x7fff96c3a56c) > at runtime/ompi_mpi_init.c:590 > #9 0x7ff3107918b7 in PMPI_Init (argc=0x7fff96c3a5ac, > argv=0x7fff96c3a5a0) at pinit.c:66 > #10 0x00400aa0 in main (argc=1, argv=0x7fff96c3a7b8) at > mpitest.c:17 > > As you can see, the crash happens inside the verbs library and the > following gets printed to the console: > > [durga@smallMP
[OMPI users] One more (possible) bug report
Dear developers
I have been observing this issue all along on the master branch, but have
been brushing off as something to do with my installation.
Right now, I just downloaded a fresh checkout (via git pull), built and
installed it (after deleting /usr/local/lib/openmpi/) and I can reproduce
the hang 100% of the time.
Description of the setup:
1. Two x86_64 boxes (dual xeons, 6 core each)
2. Four network interfaces, 3 running IP:
Broadcom GbE (IP 10.01.10.X/24) BW 1 Gbps
Chelsio iWARP (IP 10.10.10.X/24) BW 10 Gbps
Qlogic Infiniband (IP 10.01.11.X/24) BW 20Gbps
LSI logic Fibre channel (not running IP, I don't think this matters)
All of the NICs have their link UP. All the NICs are in separate IP
subnets, connected back to back.
With this, the following command hangs:
(The hostfile is this:
10.10.10.10 slots=1
10.10.10.11 slots=1
[durga@smallMPI ~]$ mpirun -np 2 -hostfile ~/hostfile -mca btl self,tcp
-mca pml ob1 ./mpitest
with the following output:
Hello world from processor smallMPI, rank 0 out of 2 processors
Hello world from processor bigMPI, rank 1 out of 2 processors
smallMPI sent haha!, rank 0
bigMPI received haha!, rank 1
The stack trace at rank 0 is:
(gdb) bt
#0 0x7f9cb844769d in poll () from /lib64/libc.so.6
#1 0x7f9cb79354d6 in poll_dispatch (base=0xddb540, tv=0x7ffc065d01b0)
at poll.c:165
#2 0x7f9cb792d180 in opal_libevent2022_event_base_loop (base=0xddb540,
flags=2) at event.c:1630
#3 0x7f9cb7851e74 in opal_progress () at runtime/opal_progress.c:171
#4 0x7f9cb89bc47d in opal_condition_wait (c=0x7f9cb8f37c40
, m=0x7f9cb8f37bc0 ) at
../opal/threads/condition.h:76
#5 0x7f9cb89bcadf in ompi_request_default_wait_all (count=2,
requests=0x7ffc065d0360, statuses=0x7ffc065d0330) at request/req_wait.c:287
#6 0x7f9cb8a95469 in ompi_coll_base_sendrecv_zero (dest=1, stag=-16,
source=1, rtag=-16, comm=0x601280 )
at base/coll_base_barrier.c:63
#7 0x7f9cb8a95b86 in ompi_coll_base_barrier_intra_two_procs
(comm=0x601280 , module=0xeb4a00) at
base/coll_base_barrier.c:313
#8 0x7f9cb8ac6d1c in ompi_coll_tuned_barrier_intra_dec_fixed
(comm=0x601280 , module=0xeb4a00) at
coll_tuned_decision_fixed.c:196
#9 0x7f9cb89dc689 in PMPI_Barrier (comm=0x601280
) at pbarrier.c:63
#10 0x00400b11 in main (argc=1, argv=0x7ffc065d0648) at mpitest.c:27
and at rank 1 is:
(gdb) bt
#0 0x7f1101e7d69d in poll () from /lib64/libc.so.6
#1 0x7f110136b4d6 in poll_dispatch (base=0x1d54540, tv=0x7ffd73013710)
at poll.c:165
#2 0x7f1101363180 in opal_libevent2022_event_base_loop
(base=0x1d54540, flags=2) at event.c:1630
#3 0x7f1101287e74 in opal_progress () at runtime/opal_progress.c:171
#4 0x7f11023f247d in opal_condition_wait (c=0x7f110296dc40
, m=0x7f110296dbc0 ) at
../opal/threads/condition.h:76
#5 0x7f11023f2adf in ompi_request_default_wait_all (count=2,
requests=0x7ffd730138c0, statuses=0x7ffd73013890) at request/req_wait.c:287
#6 0x7f11024cb469 in ompi_coll_base_sendrecv_zero (dest=0, stag=-16,
source=0, rtag=-16, comm=0x601280 )
at base/coll_base_barrier.c:63
#7 0x7f11024cbb86 in ompi_coll_base_barrier_intra_two_procs
(comm=0x601280 , module=0x1e2ebc0) at
base/coll_base_barrier.c:313
#8 0x7f11024cde3c in ompi_coll_tuned_barrier_intra_dec_fixed
(comm=0x601280 , module=0x1e2ebc0) at
coll_tuned_decision_fixed.c:196
#9 0x7f1102412689 in PMPI_Barrier (comm=0x601280
) at pbarrier.c:63
#10 0x00400b11 in main (argc=1, argv=0x7ffd73013ba8) at mpitest.c:27
The code for the test program is:
#include
#include
#include
#include
int main(int argc, char *argv[])
{
int world_size, world_rank, name_len;
char hostname[MPI_MAX_PROCESSOR_NAME], buf[8];
MPI_Init(&argc, &argv);
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
MPI_Get_processor_name(hostname, &name_len);
printf("Hello world from processor %s, rank %d out of %d processors\n",
hostname, world_rank, world_size);
if (world_rank == 1)
{
MPI_Recv(buf, 6, MPI_CHAR, 0, 99, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
printf("%s received %s, rank %d\n", hostname, buf, world_rank);
}
else
{
strcpy(buf, "haha!");
MPI_Send(buf, 6, MPI_CHAR, 1, 99, MPI_COMM_WORLD);
printf("%s sent %s, rank %d\n", hostname, buf, world_rank);
}
MPI_Barrier(MPI_COMM_WORLD);
MPI_Finalize();
return 0;
}
I have a strong feeling that there is an issue in this kind of situation.
I'll be more than happy to run further tests if someone asks me to.
Thank you
Durga
The surgeon general advises you to eat right, exercise regularly and quit
ageing.
Re: [OMPI users] One more (possible) bug report
An update on this issue:
If I choose only the IP interface matching the address that was specified
in the hostfile, the program terminates successfully; i.e. the following
command works:
mpirun -np 2 -hostfile ~/hostfile -mca btl self,tcp -mca pml ob1 -mca
btl_tcp_if_include enp35s0 ./mpitest
(enp35s0 is in the 10.10.10.X network)
If my run method is incorrect, please let me know.
An unrelated issue: a pull from github.com seems unusually slow (a simple
'already uptodate' message takes minutes to complete.) Are others
experiencing the same?
The surgeon general advises you to eat right, exercise regularly and quit
ageing.
On Fri, May 13, 2016 at 11:24 PM, dpchoudh . wrote:
> Dear developers
>
> I have been observing this issue all along on the master branch, but have
> been brushing off as something to do with my installation.
>
> Right now, I just downloaded a fresh checkout (via git pull), built and
> installed it (after deleting /usr/local/lib/openmpi/) and I can reproduce
> the hang 100% of the time.
>
> Description of the setup:
>
> 1. Two x86_64 boxes (dual xeons, 6 core each)
> 2. Four network interfaces, 3 running IP:
> Broadcom GbE (IP 10.01.10.X/24) BW 1 Gbps
> Chelsio iWARP (IP 10.10.10.X/24) BW 10 Gbps
> Qlogic Infiniband (IP 10.01.11.X/24) BW 20Gbps
> LSI logic Fibre channel (not running IP, I don't think this matters)
>
> All of the NICs have their link UP. All the NICs are in separate IP
> subnets, connected back to back.
>
> With this, the following command hangs:
> (The hostfile is this:
> 10.10.10.10 slots=1
> 10.10.10.11 slots=1
>
> [durga@smallMPI ~]$ mpirun -np 2 -hostfile ~/hostfile -mca btl self,tcp
> -mca pml ob1 ./mpitest
>
> with the following output:
>
> Hello world from processor smallMPI, rank 0 out of 2 processors
> Hello world from processor bigMPI, rank 1 out of 2 processors
> smallMPI sent haha!, rank 0
> bigMPI received haha!, rank 1
>
> The stack trace at rank 0 is:
>
> (gdb) bt
> #0 0x7f9cb844769d in poll () from /lib64/libc.so.6
> #1 0x7f9cb79354d6 in poll_dispatch (base=0xddb540, tv=0x7ffc065d01b0)
> at poll.c:165
> #2 0x7f9cb792d180 in opal_libevent2022_event_base_loop
> (base=0xddb540, flags=2) at event.c:1630
> #3 0x7f9cb7851e74 in opal_progress () at runtime/opal_progress.c:171
> #4 0x7f9cb89bc47d in opal_condition_wait (c=0x7f9cb8f37c40
> , m=0x7f9cb8f37bc0 ) at
> ../opal/threads/condition.h:76
> #5 0x7f9cb89bcadf in ompi_request_default_wait_all (count=2,
> requests=0x7ffc065d0360, statuses=0x7ffc065d0330) at request/req_wait.c:287
> #6 0x7f9cb8a95469 in ompi_coll_base_sendrecv_zero (dest=1, stag=-16,
> source=1, rtag=-16, comm=0x601280 )
> at base/coll_base_barrier.c:63
> #7 0x7f9cb8a95b86 in ompi_coll_base_barrier_intra_two_procs
> (comm=0x601280 , module=0xeb4a00) at
> base/coll_base_barrier.c:313
> #8 0x7f9cb8ac6d1c in ompi_coll_tuned_barrier_intra_dec_fixed
> (comm=0x601280 , module=0xeb4a00) at
> coll_tuned_decision_fixed.c:196
> #9 0x7f9cb89dc689 in PMPI_Barrier (comm=0x601280
> ) at pbarrier.c:63
> #10 0x00400b11 in main (argc=1, argv=0x7ffc065d0648) at
> mpitest.c:27
>
> and at rank 1 is:
>
> (gdb) bt
> #0 0x7f1101e7d69d in poll () from /lib64/libc.so.6
> #1 0x7f110136b4d6 in poll_dispatch (base=0x1d54540,
> tv=0x7ffd73013710) at poll.c:165
> #2 0x7f1101363180 in opal_libevent2022_event_base_loop
> (base=0x1d54540, flags=2) at event.c:1630
> #3 0x7f1101287e74 in opal_progress () at runtime/opal_progress.c:171
> #4 0x7f11023f247d in opal_condition_wait (c=0x7f110296dc40
> , m=0x7f110296dbc0 ) at
> ../opal/threads/condition.h:76
> #5 0x7f11023f2adf in ompi_request_default_wait_all (count=2,
> requests=0x7ffd730138c0, statuses=0x7ffd73013890) at request/req_wait.c:287
> #6 0x7f11024cb469 in ompi_coll_base_sendrecv_zero (dest=0, stag=-16,
> source=0, rtag=-16, comm=0x601280 )
> at base/coll_base_barrier.c:63
> #7 0x7f11024cbb86 in ompi_coll_base_barrier_intra_two_procs
> (comm=0x601280 , module=0x1e2ebc0) at
> base/coll_base_barrier.c:313
> #8 0x7f11024cde3c in ompi_coll_tuned_barrier_intra_dec_fixed
> (comm=0x601280 , module=0x1e2ebc0) at
> coll_tuned_decision_fixed.c:196
> #9 0x7f1102412689 in PMPI_Barrier (comm=0x601280
> ) at pbarrier.c:63
> #10 0x00400b11 in main (argc=1, argv=0x7ffd73013ba8) at
> mpitest.c:27
>
> The code for the test program is:
>
> #include
> #include
> #include
> #include
>
> int main(int argc, char *argv[])
> {
> int world_size, world_rank, name_len;
> char hostname[MPI_MAX_PROCESSOR_NAME], buf[8];
>
> MPI_Init(&argc, &argv);
&
Re: [OMPI users] One more (possible) bug report
Hello Gilles Thanks for your prompt follow up. It looks this this issue is somehow specific to the Broadcom NIC. If I take it out, the rest of them work in any combination. On further investigation, I found that the name that 'ifconfig' shows for this intterface is different from what it is named in internal scripts. Could be a bug in CentOS, but at least does not look like an OpenMPI issue. Sorry for raising the false alarm. Durga The surgeon general advises you to eat right, exercise regularly and quit ageing. On Sat, May 14, 2016 at 12:02 AM, Gilles Gouaillardet < gilles.gouaillar...@gmail.com> wrote: > at first I recommend you test 7 cases > - one network only (3 cases) > - two networks ony (3 cases) > - three networks (1 case) > > and see when things hang > > you might also want to > mpirun --mca oob_tcp_if_include 10.1.10.0/24 ... > to ensure no hang will happen in oob > > as usual, double check no firewall is running, and your hosts can ping > each other > > Cheers, > > Gilles > > On Saturday, May 14, 2016, dpchoudh . wrote: > >> Dear developers >> >> I have been observing this issue all along on the master branch, but have >> been brushing off as something to do with my installation. >> >> Right now, I just downloaded a fresh checkout (via git pull), built and >> installed it (after deleting /usr/local/lib/openmpi/) and I can reproduce >> the hang 100% of the time. >> >> Description of the setup: >> >> 1. Two x86_64 boxes (dual xeons, 6 core each) >> 2. Four network interfaces, 3 running IP: >> Broadcom GbE (IP 10.01.10.X/24) BW 1 Gbps >> Chelsio iWARP (IP 10.10.10.X/24) BW 10 Gbps >> Qlogic Infiniband (IP 10.01.11.X/24) BW 20Gbps >> LSI logic Fibre channel (not running IP, I don't think this matters) >> >> All of the NICs have their link UP. All the NICs are in separate IP >> subnets, connected back to back. >> >> With this, the following command hangs: >> (The hostfile is this: >> 10.10.10.10 slots=1 >> 10.10.10.11 slots=1 >> >> [durga@smallMPI ~]$ mpirun -np 2 -hostfile ~/hostfile -mca btl self,tcp >> -mca pml ob1 ./mpitest >> >> with the following output: >> >> Hello world from processor smallMPI, rank 0 out of 2 processors >> Hello world from processor bigMPI, rank 1 out of 2 processors >> smallMPI sent haha!, rank 0 >> bigMPI received haha!, rank 1 >> >> The stack trace at rank 0 is: >> >> (gdb) bt >> #0 0x7f9cb844769d in poll () from /lib64/libc.so.6 >> #1 0x7f9cb79354d6 in poll_dispatch (base=0xddb540, >> tv=0x7ffc065d01b0) at poll.c:165 >> #2 0x7f9cb792d180 in opal_libevent2022_event_base_loop >> (base=0xddb540, flags=2) at event.c:1630 >> #3 0x7f9cb7851e74 in opal_progress () at runtime/opal_progress.c:171 >> #4 0x7f9cb89bc47d in opal_condition_wait (c=0x7f9cb8f37c40 >> , m=0x7f9cb8f37bc0 ) at >> ../opal/threads/condition.h:76 >> #5 0x7f9cb89bcadf in ompi_request_default_wait_all (count=2, >> requests=0x7ffc065d0360, statuses=0x7ffc065d0330) at request/req_wait.c:287 >> #6 0x7f9cb8a95469 in ompi_coll_base_sendrecv_zero (dest=1, stag=-16, >> source=1, rtag=-16, comm=0x601280 ) >> at base/coll_base_barrier.c:63 >> #7 0x7f9cb8a95b86 in ompi_coll_base_barrier_intra_two_procs >> (comm=0x601280 , module=0xeb4a00) at >> base/coll_base_barrier.c:313 >> #8 0x7f9cb8ac6d1c in ompi_coll_tuned_barrier_intra_dec_fixed >> (comm=0x601280 , module=0xeb4a00) at >> coll_tuned_decision_fixed.c:196 >> #9 0x7f9cb89dc689 in PMPI_Barrier (comm=0x601280 >> ) at pbarrier.c:63 >> #10 0x00400b11 in main (argc=1, argv=0x7ffc065d0648) at >> mpitest.c:27 >> >> and at rank 1 is: >> >> (gdb) bt >> #0 0x7f1101e7d69d in poll () from /lib64/libc.so.6 >> #1 0x7f110136b4d6 in poll_dispatch (base=0x1d54540, >> tv=0x7ffd73013710) at poll.c:165 >> #2 0x7f1101363180 in opal_libevent2022_event_base_loop >> (base=0x1d54540, flags=2) at event.c:1630 >> #3 0x7f1101287e74 in opal_progress () at runtime/opal_progress.c:171 >> #4 0x7f11023f247d in opal_condition_wait (c=0x7f110296dc40 >> , m=0x7f110296dbc0 ) at >> ../opal/threads/condition.h:76 >> #5 0x7f11023f2adf in ompi_request_default_wait_all (count=2, >> requests=0x7ffd730138c0, statuses=0x7ffd73013890) at request/req_wait.c:287 >> #6 0x7f11024cb469 in ompi_coll_base_sendrecv_zero (dest=0, stag=-16, >> source=0, rtag=-16, comm=0x601280 ) >> at base/coll_base_barrier.c:63 >> #7 0x7f11024cbb8
Re: [OMPI users] One more (possible) bug report
No, I used IP addresses in all my tests. What I found that if I used the IP address of the Broadcom NIC in hostfile and used that network exclusively (btl_tcp_if_include), the mpirun command hung silently. If I used the IP address of another NIC in the host file (and Broadcom NIC exclusively), mpirun crashed saying the remote process is unreachable. If I used the other two networks exclusively (and any of their IP addresses in the host file) it works fine. Since TCP itself does not care what the underlying NIC is, it is most likely some kind of firewall issue, as you guessed (I did disable it, but there could be other related issues). In any case, I believe it has nothing to do with OMPI. One thing that is different between the Broadcom NIC and the rest is that the Broadcom NIC is connected to the WAN side and thus gets its IP via DHCP, where as the rest have static IPs. I don't see why that would make a difference, but it is possible that CentOS is enforcing some kind of security policy that I am not aware of. Thank you for for feedback. Durga The surgeon general advises you to eat right, exercise regularly and quit ageing. On Sat, May 14, 2016 at 1:13 AM, Gilles Gouaillardet < gilles.gouaillar...@gmail.com> wrote: > iirc, ompi internally uses networks and not interface names. > what did you use in your tests ? > can you try with networks ? > > Cheers, > > Gilles > > On Saturday, May 14, 2016, dpchoudh . wrote: > >> Hello Gilles >> >> Thanks for your prompt follow up. It looks this this issue is somehow >> specific to the Broadcom NIC. If I take it out, the rest of them work in >> any combination. On further investigation, I found that the name that >> 'ifconfig' shows for this intterface is different from what it is named in >> internal scripts. Could be a bug in CentOS, but at least does not look like >> an OpenMPI issue. >> >> Sorry for raising the false alarm. >> >> Durga >> >> The surgeon general advises you to eat right, exercise regularly and quit >> ageing. >> >> On Sat, May 14, 2016 at 12:02 AM, Gilles Gouaillardet < >> gilles.gouaillar...@gmail.com> wrote: >> >>> at first I recommend you test 7 cases >>> - one network only (3 cases) >>> - two networks ony (3 cases) >>> - three networks (1 case) >>> >>> and see when things hang >>> >>> you might also want to >>> mpirun --mca oob_tcp_if_include 10.1.10.0/24 ... >>> to ensure no hang will happen in oob >>> >>> as usual, double check no firewall is running, and your hosts can ping >>> each other >>> >>> Cheers, >>> >>> Gilles >>> >>> On Saturday, May 14, 2016, dpchoudh . wrote: >>> >>>> Dear developers >>>> >>>> I have been observing this issue all along on the master branch, but >>>> have been brushing off as something to do with my installation. >>>> >>>> Right now, I just downloaded a fresh checkout (via git pull), built and >>>> installed it (after deleting /usr/local/lib/openmpi/) and I can reproduce >>>> the hang 100% of the time. >>>> >>>> Description of the setup: >>>> >>>> 1. Two x86_64 boxes (dual xeons, 6 core each) >>>> 2. Four network interfaces, 3 running IP: >>>> Broadcom GbE (IP 10.01.10.X/24) BW 1 Gbps >>>> Chelsio iWARP (IP 10.10.10.X/24) BW 10 Gbps >>>> Qlogic Infiniband (IP 10.01.11.X/24) BW 20Gbps >>>> LSI logic Fibre channel (not running IP, I don't think this matters) >>>> >>>> All of the NICs have their link UP. All the NICs are in separate IP >>>> subnets, connected back to back. >>>> >>>> With this, the following command hangs: >>>> (The hostfile is this: >>>> 10.10.10.10 slots=1 >>>> 10.10.10.11 slots=1 >>>> >>>> [durga@smallMPI ~]$ mpirun -np 2 -hostfile ~/hostfile -mca btl >>>> self,tcp -mca pml ob1 ./mpitest >>>> >>>> with the following output: >>>> >>>> Hello world from processor smallMPI, rank 0 out of 2 processors >>>> Hello world from processor bigMPI, rank 1 out of 2 processors >>>> smallMPI sent haha!, rank 0 >>>> bigMPI received haha!, rank 1 >>>> >>>> The stack trace at rank 0 is: >>>> >>>> (gdb) bt >>>> #0 0x7f9cb844769d in poll () from /lib64/libc.so.6 >>>> #1 0x7f9cb79354d6 in poll_dispatch (base=0xddb540, >>
[OMPI users] Possible (minor) bug?
Hello all I have started noticing this message since yesterday on builds from the master branch. Any simple mpirun command, such as: mpirun -np 2 -hostfile ~/hostfile -mca btl self,tcp hostname generates a warning/error like this: *Duplicate cmd line entry mca* The hostfile, in my case, is just this: 10.10.10.10 slots=1 10.10.10.11 slots=1 Thanks Durga We learn from history that we never learn from history.
[OMPI users] How to see the output from OPAL_OUTPUT_VERBOSE?
Hello all I have built and installed OMPI with --enable-debug set. What runtime parameter do I need to see the output from OPAL_DEBUG_OUTPUT? Thank you Durga We learn from history that we never learn from history.
Re: [OMPI users] How to see the output from OPAL_OUTPUT_VERBOSE?
Hello Gilles and Nathan Thank you very much for the answer. The reason I asked this question is that I wanted to see the debug output from OPAL_MODEX_RECV()/OPAL_MODEX_SEND(). As I reported in an earlier mail, in my case, the modex data from OPAL_MODEX_RECV() is getting corrupted and I wanted to see how the other BTLs are doing it. However, setting the _base_verbose to 100 does not seem to include any debug message coming from modex exchange, either for tcp or for openib: please see the output below: [durga@bigMPI ~]$ mpirun --verbose -np 2 -hostfile ~/hostfile -mca btl_base_verbose 100 -mca pml_base_verbose 100 -mca bml_base_verbose 100 -mca btl self,tcp -mca btl_tcp_if_include enp35s0 -mca opal_base_verbose 100 ./hello_c 2>&1 | grep -i modex [bigMPI:12791] check:select: modex data not found [durga@bigMPI ~]$ mpirun --verbose -np 2 -hostfile ~/hostfile -mca btl_base_verbose 100 -mca pml_base_verbose 100 -mca bml_base_verbose 100 -mca btl self,openib -mca btl_openib_if_include qib0 -mca opal_base_verbose 100 ./hello_c 2>&1 | grep -i modex [bigMPI][[27674,1],1][connect/btl_openib_connect_udcm.c:743:udcm_module_init] my modex = LID: 2, Port: 1, QPN: 19 [smallMPI][[27674,1],0][connect/btl_openib_connect_udcm.c:743:udcm_module_init] my modex = LID: 1, Port: 1, QPN: 19 [bigMPI:13479] check:select: modex data not found Please note that the two lines coming from udcm_module_init are not part of OPAL_MODEX_SEND() or OPAL_MODEX_RECV() Why am I not able to see the debug output from the above two macros? Thanks in advance Durga We learn from history that we never learn from history. On Sun, May 22, 2016 at 10:55 AM, Nathan Hjelm wrote: > You use the *_base_verbose MCA variables. For example, if you want to see > output from the btl use -mca btl_base_verbose x. The number x controls the > verbosity level. Starting with 2.x are named levels but now many components > conform to the names yet. In general components use use numbers between 0 > and 100 (inclusive) with 100 being very verbose. > > -Nathan > > > On May 22, 2016, at 12:36 AM, dpchoudh . wrote: > > > > Hello all > > > > I have built and installed OMPI with --enable-debug set. What runtime > parameter do I need to see the output from OPAL_DEBUG_OUTPUT? > > > > Thank you > > Durga > > > > We learn from history that we never learn from history. > > ___ > > users mailing list > > us...@open-mpi.org > > Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users > > Link to this post: > http://www.open-mpi.org/community/lists/users/2016/05/29271.php > > > ___ > users mailing list > us...@open-mpi.org > Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2016/05/29273.php >
[OMPI users] PSM vs PSM2
Hello all What is the difference between PSM and PSM2? Any pointer to more information is appreciated. Also, the PSM2 MTL does not seem to have a owner.txt file (on master, at least). Why is that? Thanks Durga We learn from history that we never learn from history.
[OMPI users] QP creation failure on iWARP adapter
Dear all This is a slightly off-topic post, and hopefully people won't mind helping me out. I have a very simple setup with two PCs, both with identical Chelsio 10GE iWARP adapter connected back-to-back. With this setup, the TCP channel works fine (with MPI or otherwise). But somehow, using RDMA semantics, the QP creation fails. This has nothing to do with OpenMPI per se; using standard benchmarking tools such as qperf also shows the same behavior. I have set both the hard and soft limit on locked memory to 'unlimited' and have rebooted both PCs and have verified that 'ulimit -l' shows 'unlimited' on both nodes. Yet, the QP creation fails. In OpenMPI this shows up if I try to use the openib BTL. The OS (on both nodes) is CentOS 7, if that matters. What am I doing wrong? Thanks a lot for any advice. Durga Life is complex. It has real and imaginary parts.
[OMPI users] Release vs git trunk directory tree
Hello developers The directory structure of the latest release and what exists on the git trunk seems to be very different, at least in the ompi/mca/ branch. In particular, the btl/ subtree does not even exist in the trunk. The reason I went looking for it is I am trying to implement a BTL for a new, proprietary NIC, and from a Google search, I found that the git trunk has a subdirectory called 'template' under btl/ containing a stub implementation that can be used as a starting point. But it looks like even the parent btl/ directory is missing in the trunk. Any help and pointers in this regards is greatly appreciated. Durga Life is complex. It has real and imaginary parts.
[OMPI users] Adding a new BTL
Hello all I am not sure if this question belongs in the user list or the developer list, but because it is a simpler question I am trying the user list first. I am trying to add a new BTL for a proprietary transport. As step #0, I copied the BTL template, renamed the 'template' to something else, and ran autogen.sh at the top level directory (of openMPI 1.10.2). The Makefile.am is identical to what is provided in the template except that all the 'template' has been substituted with 'lf', the name of the fabric. With that, I get the following error: autoreconf: running: /usr/bin/autoconf --include=config --force --warnings=all,no-obsolete,no-override autoreconf: running: /usr/bin/autoheader --include=config --force --warnings=all,no-obsolete,no-override autoreconf: running: automake --add-missing --copy --force-missing --warnings=all,no-obsolete,no-override configure.ac:320: installing 'config/compile' configure.ac:73: installing 'config/config.guess' configure.ac:73: installing 'config/config.sub' configure.ac:93: installing 'config/install-sh' configure.ac:93: installing 'config/missing' ompi/Makefile.am: installing 'config/depcomp' ompi/mca/btl/lf/Makefile.am:33: error: MCA_BUILD_opal_btl_lf_DSO does not appear in AM_CONDITIONAL I tried adding a configure.m4 file to the btl directory with the following content: # MCA_btl_lf_CONFIG([action-if-can-compile], # [action-if-cant-compile]) # AC_DEFUN([MCA_ompi_btl_lf_CONFIG],[ AC_CONFIG_FILES([ompi/mca/btl/lf/Makefile]) AC_MSG_FAILURE ])dnl but the error remains. I am sure I am missing at least one step, but am lost in the huge codebase. Please help. Thank you Durga Life is complex. It has real and imaginary parts.
Re: [OMPI users] Adding a new BTL
Hello Gilles Thank you very much for your advice. Yes, I copied the templates from the master branch to the 1.10.2 release, since the release does not have them. And yes, changing the Makefile.am as you suggest did make the autogen error go away. However, in the master branch, the autotools seem to be ignoring the new btl directory altogether; i.e. I do not get a Makefile.in from the Makefile.am. In the 1.10.2 release, doing an identical sequence of steps do create a Makefile.in from Makefile.am (via autogen) and a Makefile from Makefile.in (via configure), but of course, the new BTL does not build because the include paths in master and 1.10.2 are different. My Makefile.am and configure.m4 are as follows. Any thoughts on what it would take in the master branch to hook up the new BTL directory? opal/mca/btl/lf/configure.m4 # AC_DEFUN([MCA_opal_btl_lf_CONFIG],[ AC_CONFIG_FILES([opal/mca/btl/lf/Makefile]) ])dnl opal/mca/btl/lf/Makefile.am--- amca_paramdir = $(AMCA_PARAM_SETS_DIR) dist_amca_param_DATA = netpipe-btl-lf.txt sources = \ btl_lf.c \ btl_lf.h \ btl_lf_component.c \ btl_lf_endpoint.c \ btl_lf_endpoint.h \ btl_lf_frag.c \ btl_lf_frag.h \ btl_lf_proc.c \ btl_lf_proc.h # Make the output library in this directory, and name it either # mca__.la (for DSO builds) or libmca__.la # (for static builds). if MCA_BUILD_opal_btl_lf_DSO lib = lib_sources = component = mca_btl_lf.la component_sources = $(sources) else lib = libmca_btl_lf.la lib_sources = $(sources) component = component_sources = endif mcacomponentdir = $(opallibdir) mcacomponent_LTLIBRARIES = $(component) mca_btl_lf_la_SOURCES = $(component_sources) mca_btl_lf_la_LDFLAGS = -module -avoid-version noinst_LTLIBRARIES = $(lib) libmca_btl_lf_la_SOURCES = $(lib_sources) libmca_btl_lf_la_LDFLAGS = -module -avoid-version - Life is complex. It has real and imaginary parts. On Thu, Feb 25, 2016 at 3:10 AM, Gilles Gouaillardet < gilles.gouaillar...@gmail.com> wrote: > Did you copy the template from the master branch into the v1.10 branch ? > if so, you need to replacing MCA_BUILD_opal_btl_lf_DSO with > MCA_BUILD_ompi_btl_lf_DSO will likely solve your issue. > you do need a configure.m4 (otherwise your btl will not be built) but > you do not need AC_MSG_FAILURE > > as far as i am concerned, i would develop in the master branch, and > then back port it into the v1.10 branch when it is ready. > > fwiw, btl used to be in ompi/mca/btl (still the case in v1.10) and > have been moved into opal/mca/btl since v2.x > so it is quite common a bit of porting is required, most of the time, > it consists in replacing OMPI like macros by OPAL like macros > > Cheers, > > Gilles > > On Thu, Feb 25, 2016 at 3:54 PM, dpchoudh . wrote: > > Hello all > > > > I am not sure if this question belongs in the user list or the > > developer list, but because it is a simpler question I am trying the > > user list first. > > > > I am trying to add a new BTL for a proprietary transport. > > > > As step #0, I copied the BTL template, renamed the 'template' to > > something else, and ran autogen.sh at the top level directory (of > > openMPI 1.10.2). The Makefile.am is identical to what is provided in > > the template except that all the 'template' has been substituted with > > 'lf', the name of the fabric. > > > > With that, I get the following error: > > > > > > > > autoreconf: running: /usr/bin/autoconf --include=config --force > > --warnings=all,no-obsolete,no-override > > autoreconf: running: /usr/bin/autoheader --include=config --force > > --warnings=all,no-obsolete,no-override > > autoreconf: running: automake --add-missing --copy --force-missing > > --warnings=all,no-obsolete,no-override > > configure.ac:320: installing 'config/compile' > > configure.ac:73: installing 'config/config.guess' > > configure.ac:73: installing 'config/config.sub' > > configure.ac:93: installing 'config/install-sh' > > configure.ac:93: installing 'config/missing' > > ompi/Makefile.am: installing 'config/depcomp' > > ompi/mca/btl/lf/Makefile.am:33: error: MCA_BUILD_opal_btl_lf_DSO does > > not appear in AM_CONDITIONAL > > > > > > I tried adding a configure.m4 file to the btl directory with the > > following content: > > > > # MCA_btl_lf_CONFIG([action-if-can-compile], > > # [action-if-cant-compile]) > > # > > AC_DEFUN([MCA_ompi_btl_lf_CONFIG],[ > > AC_CONFIG_FILE
Re: [OMPI users] Adding a new BTL
Thank you, Jeff. I'll start a new thread on this on the devel list. Durga Life is complex. It has real and imaginary parts. On Thu, Feb 25, 2016 at 5:15 PM, Jeff Squyres (jsquyres) wrote: > Can you send the full output from autogen and configure? > > Also, this is probably better suited for the Devel list, since we're > talking about OMPI internals. > > Sent from my phone. No type good. > > On Feb 25, 2016, at 2:06 PM, dpchoudh . wrote: > > Hello Gilles > > Thank you very much for your advice. Yes, I copied the templates from the > master branch to the 1.10.2 release, since the release does not have them. > And yes, changing the Makefile.am as you suggest did make the autogen error > go away. > > However, in the master branch, the autotools seem to be ignoring the new > btl directory altogether; i.e. I do not get a Makefile.in from the > Makefile.am. > > In the 1.10.2 release, doing an identical sequence of steps do create a > Makefile.in from Makefile.am (via autogen) and a Makefile from Makefile.in > (via configure), but of course, the new BTL does not build because the > include paths in master and 1.10.2 are different. > > My Makefile.am and configure.m4 are as follows. Any thoughts on what it > would take in the master branch to hook up the new BTL directory? > > opal/mca/btl/lf/configure.m4 > # > AC_DEFUN([MCA_opal_btl_lf_CONFIG],[ > AC_CONFIG_FILES([opal/mca/btl/lf/Makefile]) > ])dnl > > opal/mca/btl/lf/Makefile.am--- > amca_paramdir = $(AMCA_PARAM_SETS_DIR) > dist_amca_param_DATA = netpipe-btl-lf.txt > > sources = \ > btl_lf.c \ > btl_lf.h \ > btl_lf_component.c \ > btl_lf_endpoint.c \ > btl_lf_endpoint.h \ > btl_lf_frag.c \ > btl_lf_frag.h \ > btl_lf_proc.c \ > btl_lf_proc.h > > # Make the output library in this directory, and name it either > # mca__.la (for DSO builds) or libmca__.la > # (for static builds). > > if MCA_BUILD_opal_btl_lf_DSO > lib = > lib_sources = > component = mca_btl_lf.la > component_sources = $(sources) > else > lib = libmca_btl_lf.la > lib_sources = $(sources) > component = > component_sources = > endif > > mcacomponentdir = $(opallibdir) > mcacomponent_LTLIBRARIES = $(component) > mca_btl_lf_la_SOURCES = $(component_sources) > mca_btl_lf_la_LDFLAGS = -module -avoid-version > > noinst_LTLIBRARIES = $(lib) > libmca_btl_lf_la_SOURCES = $(lib_sources) > libmca_btl_lf_la_LDFLAGS = -module -avoid-version > > - > > Life is complex. It has real and imaginary parts. > > On Thu, Feb 25, 2016 at 3:10 AM, Gilles Gouaillardet < > gilles.gouaillar...@gmail.com> wrote: > >> Did you copy the template from the master branch into the v1.10 branch ? >> if so, you need to replacing MCA_BUILD_opal_btl_lf_DSO with >> MCA_BUILD_ompi_btl_lf_DSO will likely solve your issue. >> you do need a configure.m4 (otherwise your btl will not be built) but >> you do not need AC_MSG_FAILURE >> >> as far as i am concerned, i would develop in the master branch, and >> then back port it into the v1.10 branch when it is ready. >> >> fwiw, btl used to be in ompi/mca/btl (still the case in v1.10) and >> have been moved into opal/mca/btl since v2.x >> so it is quite common a bit of porting is required, most of the time, >> it consists in replacing OMPI like macros by OPAL like macros >> >> Cheers, >> >> Gilles >> >> On Thu, Feb 25, 2016 at 3:54 PM, dpchoudh . wrote: >> > Hello all >> > >> > I am not sure if this question belongs in the user list or the >> > developer list, but because it is a simpler question I am trying the >> > user list first. >> > >> > I am trying to add a new BTL for a proprietary transport. >> > >> > As step #0, I copied the BTL template, renamed the 'template' to >> > something else, and ran autogen.sh at the top level directory (of >> > openMPI 1.10.2). The Makefile.am is identical to what is provided in >> > the template except that all the 'template' has been substituted with >> > 'lf', the name of the fabric. >> > >> > With that, I get the following error: >> > >> > >> > >> > autoreconf: running: /usr/bin/autoconf --include=config --force >> > --warnings=all,no-obsolete,no-override >> > autoreconf: running: /usr/bin/autoheader --include=config --force >> > --warnings=all,no-obsolete,no-override >> > autoreconf: runni
Re: [OMPI users] General Questions
I don't think the Open MPI TCP BTL will pass the SDP socket type when creating sockets -- SDP is much lower performance than native verbs/RDMA. You should use a "native" interface to your RDMA network instead (which one you use depends on which kind of network you have). I have a rather naive follow-up question along this line: why is there not a native mode for (garden variety) Ethernet? Is it because it lacks the end-to-end guarantees of TCP, Infiniband and the like? These days, switched Ethernet is very reliable, isn't it? (I mean by the rate of packet drop because of congestion). So if the application only needs data chunks of around 8KB max, which would not need to be fragmented (using jumbo frames), won't a native ethernet be much more efficient? Or perhaps these constraints are too limiting in practice? Thanks Durga Life is complex. It has real and imaginary parts. On Tue, Mar 1, 2016 at 9:54 PM, Jeff Squyres (jsquyres) wrote: > On Mar 1, 2016, at 6:55 PM, Matthew Larkin wrote: > > > > As far as PCIe, I am looking into: > > > > 1. Dolphin's implementation of IPoPCIe > > If it provides a TCP stack and an IP interface, you should be able to use > Open MPI's TCP BTL interface over it. > > > 2. SDP protocol and how it can be utilized, mapping TCP to RDMA > > I don't think the Open MPI TCP BTL will pass the SDP socket type when > creating sockets -- SDP is much lower performance than native verbs/RDMA. > You should use a "native" interface to your RDMA network instead (which one > you use depends on which kind of network you have). > > > Not sure if the only answer for this is a custom stack, API/kernel > module. > > > > Do you have any input on the above mentioned things? > > > > On Tuesday, March 1, 2016 6:42 AM, Jeff Squyres (jsquyres) < > jsquy...@cisco.com> wrote: > > > > > > On Feb 29, 2016, at 6:48 PM, Matthew Larkin wrote: > > > > > > 1. I know OpenMPI supports ethernet, but where does it clearly state > that? > > > - I see on the FAQ on the web page there is a whole list of network > interconnect, but how do I relate that to Ethernet network etc.? > > > > Open MPI actually supports multiple Ethernet-based interconnects: Cisco > usNIC, iWARP, Mellanox RoCE, and TCP sockets. > > > > I suspect the one you are asking about is TCP sockets (which technically > doesn't need to run over Ethernet, but TCP-over-Ethernet is probably its > most common use case). > > > > > > > 2. Does OpenMPI work with PCIe and PCIe switch? > > > - Is there any specific configuration to get it to work? > > > > > > Do you have a specific vendor device / networking stack in mind? In > general, Open MPI will use: > > > > - some kind of local IPC mechanism (e.g., some flavor of shared memory) > for intra-node communication > > - some kind of networking API for inter-node communication > > > > Extending PCIe between servers blurs this line a bit -- peer MPI > processes on a remote server can make it look like they are actually > local. So it depends on your network stack: do you have some kind of API > that effects messaging transport over PCIe? > > > > -- > > Jeff Squyres > > jsquy...@cisco.com > > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > > > > > > > > > -- > Jeff Squyres > jsquy...@cisco.com > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > > ___ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2016/03/28613.php >
[OMPI users] iWARP usage issue
Hello all I am asking for help for the following situation: I have two (mostly identical) nodes. Each of them have (completely identical) 1. qlogic 4x DDR infiniband, AND 2. Chelsio S310E (T3 chip based) 10GE iWARP cards. Both are connected back-to-back, without a switch. The connection is physically OK and IP traffic can flow on both of them without issues. The issue is, I can run MPI programs using the openib BTL using the qlogic card, but not the Chelsio card. Here are the commands: [durga@smallMPI ~]$ ibv_devices device node GUID -- cxgb3_0 00074306cd3b <-- Chelsio qib0001175ff831d <-- Qlogic The following command works: mpirun -np 2 --hostfile ~/hostfile -mca btl_openib_if_include qib0 ./osu_acc_latency And the following do not: mpirun -np 2 --hostfile ~/hostfile -mca btl_openib_if_include cxgb3_0 ./osu_acc_latency mpirun -np 2 --hostfile ~/hostfile -mca pml ob1 -mca btl_openib_if_include cxgb3_0 ./osu_acc_latency mpirun -np 2 --hostfile ~/hostfile -mca pml ^cm -mca btl_openib_if_include cxgb3_0 ./osu_acc_latency The error I get is the following (in all of the non-working cases): WARNING: The largest queue pair buffer size specified in the btl_openib_receive_queues MCA parameter is smaller than the maximum send size (i.e., the btl_openib_max_send_size MCA parameter), meaning that no queue is large enough to receive the largest possible incoming message fragment. The OpenFabrics (openib) BTL will therefore be deactivated for this run. Local host: smallMPI Largest buffer size: 65536 Maximum send fragment size: 131072 -- -- No OpenFabrics connection schemes reported that they were able to be used on a specific port. As such, the openib BTL (OpenFabrics support) will be disabled for this port. Local host: bigMPI Local device: cxgb3_0 Local port: 1 CPCs attempted: udcm -- I have a vague understanding of what the message is trying to say, but I do not know which file or configuration parameters to change to fix the situation. Thanks in advance Durga Life is complex. It has real and imaginary parts.
Re: [OMPI users] Communication problem (on one node) when network interface is down
Hello all >From a user standpoint, that does not seem right to me. Why should one need any kind of network at all if one is entirely dealing with a single node? Is there any particular reason OpenMPI does not/cannot use the lo (loopback) interface? I'd think it is there for exactly this kind of situation. Thanks Durga Life is complex. It has real and imaginary parts. On Fri, Mar 11, 2016 at 6:08 AM, Gilles Gouaillardet < gilles.gouaillar...@gmail.com> wrote: > Spawned tasks cannot use the sm nor vader btl so you need an other one > (tcp, openib, ...) > self btl is only to send/recvcount with oneself (e.g. it does not work for > inter process and intra node communications. > > I am pretty sure the lo interface is always discarded by openmpi, so I > have no solution on top of my head that involves openmpi. > maybe your bed bet is to use a "dummy" interface, for example tan or tun > or even a bridge. > > Cheers, > > Gilles > > > > On Friday, March 11, 2016, Rémy Grünblatt > wrote: > >> Hello, >> I'm having communications problem between two processes (with one being >> spawned by the other, on the *same* physical machine). Everything works >> as expected when I have network interface such as eth0 or wlo1 up, but >> as soon as they are down, I get errors (such as « At least one pair of >> MPI processes are unable to reach each other for MPI communications […] >> »). >> I tried to specify a set of mca parameters including the btl "self" >> parameter and including the lo interface in btl_tcp_if_include list, as >> advised by https://www.open-mpi.org/faq/?category=tcp but I didn't reach >> any working state for this code with "external" network interface down. >> >> Got any idea about what I might do wrong ? >> >> Example code that triggers the problem: https://ptpb.pw/YOjr.tar.gz >> Ompi_info: https://ptpb.pw/Vt_V.txt >> Full log: https://ptpb.pw/JCXn.txt >> >> Rémy >> >> >> > ___ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2016/03/28684.php >
[OMPI users] IB question (slightly off topic)
Hello all I have a question, that I do realize, is somewhat off topic to this list. But I do not know who to approach for an answer. Hopefully the community here will help me out. I know that Infiniband is a 'standard' interface (standardized by IETF? IEEE? or some similar body), much like Ethernet. However, I do see that they come in different 'flavors', (and have different feature set?) such as Qlogic PSM or Mellanox ConnectX, that have *user space* "drivers" and even OpenMPI treats them differently (preferring Qlogic PSM over other IB, as a default behavior). For someone very new to the Infiniband world, what are the differences? How can they be different and yet confirm to the (supposed) standard? Any pointer to appropriate literature is appreciated. Thanks in advance Durga Life is complex. It has real and imaginary parts.
[OMPI users] Issue about cm PML
Hello all I have a simple test setup, consisting of two Dell workstation nodes with similar hardware profile. Both the nodes have (identical) 1. Qlogic 4x DDR infiniband 2. Chelsio C310 iWARP ethernet. Both of these cards are connected back to back, without a switch. With this setup, I can run OpenMPI over TCP and openib BTL. However, if I try to use the PSM MTL (excluding the Chelsio NIC, of course, since it does not support PSM), I get an error from one of the nodes (details below), which makes me think that a required library or package is not installed, but I can't figure out what it might be. Note that the test program is a simple 'hello world' program. The following work: mpirun -np 2 --hostfile ~/hostfile -mca btl tcp,self ./mpitest mpirun -np 2 --hostfile ~/hostfile -mca btl self,openib -mca btl_openib_if_exclude cxgb3_0 ./mpitest (I had to exclude the Chelsio card because of this issue: https://www.open-mpi.org/community/lists/users/2016/03/28661.php ) Here is what does NOT work: mpirun -np 2 --hostfile ~/hostfile -mca mtl psm -mca btl_openib_if_exclude cxgb3_0 ./mpitest The error (from both nodes) is: mca: base: components_open: component pml / cm open function failed However, I still see the "Hello, world" output indicating that the program ran to completion. Here is also another command that does NOT work: mpirun -np 2 --hostfile ~/hostfile -mca pml cm -mca btl_openib_if_exclude cxgb3_0 ./mpitest The error is: (from the root node) PML cm cannot be selected However, this time, I see no output from the program, indicating it did not run. The following command also fails in a similar way: mpirun -np 2 --hostfile ~/hostfile -mca pml cm -mca mtl psm -mca btl_openib_if_exclude cxgb3_0 ./mpitest I have verified that infinipath-psm is installed on both nodes. Both nodes run identical CentOS 7 and the libraries were installed from the CentOS repositories (i.e. were not compiled from source) Both nodes run OMPI 1.10.2, compiled from the source RPM. What am I doing wrong? Thanks Durga Life is complex. It has real and imaginary parts.
Re: [OMPI users] Issue about cm PML
Thank you everybody. With your help I was able to resolve the issue. For the sake of completeness, this is what I had to do: infinipath-psm was already installed in my system when OpenMPI was built from source. However, infinipath-psm-devel was NOT installed. I suppose that's why openMPI could not add support for PSM when built from source, and, following Jeff's advice, I ran ompi_info | grep psm which showed no output. I had to install infinipath-psm-devel and rebuild OpenMPI. That fixed it. Durga Life is complex. It has real and imaginary parts. On Thu, Mar 17, 2016 at 9:17 AM, Jeff Squyres (jsquyres) wrote: > Additionally, if you run > > ompi_info | grep psm > > Do you see the PSM MTL listed? > > To force the CM MTL, you can run: > > mpirun --mca pml cm ... > > That won't let any BTLs be selected (because only ob1 uses the BTLs). > > > > On Mar 17, 2016, at 8:07 AM, Gilles Gouaillardet < > gilles.gouaillar...@gmail.com> wrote: > > > > can you try to add > > --mca mtl psm > > to your mpirun command line ? > > > > you might also have to blacklist the opening btl > > > > Cheers, > > > > Gilles > > > > On Thursday, March 17, 2016, dpchoudh . wrote: > > Hello all > > I have a simple test setup, consisting of two Dell workstation nodes > with similar hardware profile. > > > > Both the nodes have (identical) > > 1. Qlogic 4x DDR infiniband > > 2. Chelsio C310 iWARP ethernet. > > > > Both of these cards are connected back to back, without a switch. > > > > With this setup, I can run OpenMPI over TCP and openib BTL. However, if > I try to use the PSM MTL (excluding the Chelsio NIC, of course, since it > does not support PSM), I get an error from one of the nodes (details > below), which makes me think that a required library or package is not > installed, but I can't figure out what it might be. > > > > Note that the test program is a simple 'hello world' program. > > > > The following work: > > mpirun -np 2 --hostfile ~/hostfile -mca btl tcp,self ./mpitest > > mpirun -np 2 --hostfile ~/hostfile -mca btl self,openib -mca > btl_openib_if_exclude cxgb3_0 ./mpitest > > > > (I had to exclude the Chelsio card because of this issue: > > https://www.open-mpi.org/community/lists/users/2016/03/28661.php ) > > > > Here is what does NOT work: > > mpirun -np 2 --hostfile ~/hostfile -mca mtl psm -mca > btl_openib_if_exclude cxgb3_0 ./mpitest > > > > The error (from both nodes) is: > > mca: base: components_open: component pml / cm open function failed > > > > However, I still see the "Hello, world" output indicating that the > program ran to completion. > > > > Here is also another command that does NOT work: > > > > mpirun -np 2 --hostfile ~/hostfile -mca pml cm -mca > btl_openib_if_exclude cxgb3_0 ./mpitest > > > > The error is: (from the root node) > > PML cm cannot be selected > > > > However, this time, I see no output from the program, indicating it did > not run. > > > > The following command also fails in a similar way: > > mpirun -np 2 --hostfile ~/hostfile -mca pml cm -mca mtl psm -mca > btl_openib_if_exclude cxgb3_0 ./mpitest > > > > I have verified that infinipath-psm is installed on both nodes. Both > nodes run identical CentOS 7 and the libraries were installed from the > CentOS repositories (i.e. were not compiled from source) > > > > Both nodes run OMPI 1.10.2, compiled from the source RPM. > > > > What am I doing wrong? > > > > Thanks > > Durga > > > > > > > > > > Life is complex. It has real and imaginary parts. > > ___ > > users mailing list > > us...@open-mpi.org > > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > > Link to this post: > http://www.open-mpi.org/community/lists/users/2016/03/28725.php > > > -- > Jeff Squyres > jsquy...@cisco.com > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > > ___ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2016/03/28727.php >
[OMPI users] Why does 'self' needs to be explicitly mentioned?
Hello all I am wondering as to: 1. Why 'self' needs to be explicitly mentioned when using the BTL communication? Since it must always be there for MPI communication to work, should it not be implicit? I am sure there is some architectural rationale behind this; could someone please elaborate? 2. Why is this (using 'self') not needed when using MTL communication? 3. Is it possible to mix BTL and MTL on a single OMPI job? For example, if I have a card, let's say Texas Instruments Hyperlink, that is only available as BTL, and I also have Qlogic cards on the same nodes, can I use something like this: mpirun -np 3 -H h1,h2,h3 -mca MTL psm -mca BTL self,hlink 4. If the above answer is yes, then would OpenMPI use both cards *simultaneously* (and stripe messages) or would it use the one with higher exclusivity and put the other as standby for failure recovery? Thanks in advance Durga Life is complex. It has real and imaginary parts.
[OMPI users] Why do I need a C++ linker while linking in MPI C code with CUDA?
Hello all I downloaded some code samples from here: https://github.com/parallel-forall/code-samples/ and tried to build the subdirectory posts/cuda-aware-mpi-example/src in my CentOS 7 machine. I had to make several changes to the Makefile before it would build. The modified Makefile is attached (the make targets I am talking about are the 3rd and 4th from the bottom). Most of the modifications can be explained as possible platform specific variations (such as path differences betwen Ubuntu and CentOS), except the following: I had to use a C++ linker (mpic++) to link in the object files that were produced with C host compiler (mpicc) and CUDA compiler (nvcc). If I did not do this, (i.e. I stuck to mpicc for linking), I got the following link error: mpicc -L/usr/local/cuda/lib64 -lcudart -lm -o ../bin/jacobi_cuda_normal_mpi jacobi.o input.o host.o device.o cuda_normal_mpi.o device.o: In function `__static_initialization_and_destruction_0(int, int)': tmpxft_4651_-4_Device.cudafe1.cpp:(.text+0xd1e): undefined reference to `std::ios_base::Init::Init()' tmpxft_4651_-4_Device.cudafe1.cpp:(.text+0xd2d): undefined reference to `std::ios_base::Init::~Init()' collect2: error: ld returned 1 exit status Can someone please explain why would I need a C++ linker for object files that were generated using C compiler? Note that if I use mpic++ both for compiling and linking, there are no errors either. Thanks in advance Durga Life is complex. It has real and imaginary parts. Makefile Description: Binary data
Re: [OMPI users] Why do I need a C++ linker while linking in MPI C code with CUDA?
I'd tend to agree with Gilles. I have written CUDA programs in pure C (i.e. neither involving MPI nor C++) and a pure C based tool chain builds the code successfully. So I don't see why CUDA should be intrinsically C++. >From the Makefile (that I had attached in my previous mail) the only CUDA library being linked against is this: /usr/local/cuda/lib64/libcudart.so and ldd on that shows this: [durga@smallMPI lib64]$ ldd libcudart.so linux-vdso.so.1 => (0x7ffe1e7f1000) libc.so.6 => /lib64/libc.so.6 (0x7ff7e4493000) libdl.so.2 => /lib64/libdl.so.2 (0x7ff7e428f000) libpthread.so.0 => /lib64/libpthread.so.0 (0x7ff7e4072000) librt.so.1 => /lib64/librt.so.1 (0x7ff7e3e6a000) /lib64/ld-linux-x86-64.so.2 (0x7ff7e4af3000) I don't see any C++ dependency here either. And finally, I don't think there is any version issue. This is a clean CUDA 7.5 install directly from NVIDIA CUDA repo (for Redhat) and all provided examples run fine with this installation. I believe there are NVIDIA employees in this list; hopefully one of them will clarify. Thanks Durga Life is complex. It has real and imaginary parts. On Sun, Mar 20, 2016 at 10:23 PM, Gilles Gouaillardet < gilles.gouaillar...@gmail.com> wrote: > I am a bit puzzled... > > if only cuda uses the c++ std libraries, then it should depend on them > (ldd libcudaxyz.so can be used to confirm that) > and then linking with cuda lib should pull the c++ libs > > could there be a version issue ? > e.g. the missing symbol is not provided by the version of the c++ lib that > is pulled. > that might occur if you are using cuda built for distro X on distro Y > > could you please double check this ? > if everything should work, then i recommend you report this to nvidia > > Cheers, > > Gilles > > On Monday, March 21, 2016, Damien Hocking wrote: > >> Durga, >> >> The Cuda libraries use the C++ std libraries. That's the std::ios_base >> errors.. You need the C++ linker to bring those in. >> >> Damien >> >> On March 20, 2016 9:15:47 AM "dpchoudh ." wrote: >> >>> Hello all >>> >>> I downloaded some code samples from here: >>> >>> https://github.com/parallel-forall/code-samples/ >>> >>> and tried to build the subdirectory >>> >>> posts/cuda-aware-mpi-example/src >>> >>> in my CentOS 7 machine. >>> >>> I had to make several changes to the Makefile before it would build. The >>> modified Makefile is attached (the make targets I am talking about are the >>> 3rd and 4th from the bottom). Most of the modifications can be explained as >>> possible platform specific variations (such as path differences betwen >>> Ubuntu and CentOS), except the following: >>> >>> I had to use a C++ linker (mpic++) to link in the object files that were >>> produced with C host compiler (mpicc) and CUDA compiler (nvcc). If I did >>> not do this, (i.e. I stuck to mpicc for linking), I got the following link >>> error: >>> >>> mpicc -L/usr/local/cuda/lib64 -lcudart -lm -o >>> ../bin/jacobi_cuda_normal_mpi jacobi.o input.o host.o device.o >>> cuda_normal_mpi.o >>> device.o: In function `__static_initialization_and_destruction_0(int, >>> int)': >>> tmpxft_4651_-4_Device.cudafe1.cpp:(.text+0xd1e): undefined >>> reference to `std::ios_base::Init::Init()' >>> tmpxft_4651_-4_Device.cudafe1.cpp:(.text+0xd2d): undefined >>> reference to `std::ios_base::Init::~Init()' >>> collect2: error: ld returned 1 exit status >>> >>> Can someone please explain why would I need a C++ linker for object >>> files that were generated using C compiler? Note that if I use mpic++ both >>> for compiling and linking, there are no errors either. >>> >>> Thanks in advance >>> Durga >>> >>> Life is complex. It has real and imaginary parts. >>> ___ >>> users mailing list >>> us...@open-mpi.org >>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>> Link to this post: >>> http://www.open-mpi.org/community/lists/users/2016/03/28760.php >>> >> > ___ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2016/03/28762.php >
[OMPI users] Existing and emerging interconnects for commodity PCs
Hello all I don't mean this to be a political conversation, but more of a research type. >From what I have been observing, some of the interconnects that had very good technological features as well as popularity in the past have basically gone down the history book and some others, with comparable feature set, have gained (although I won't put any names here, neither of these are commodity gigabit Ethernet). Any comments on what drives these factors? Put another way, if I am to architect a system consisting of commodity nodes today, how can I reasonably be sure that the interconnect will be a good choice, in all sense of the word 'good', say, 10 years down the road? Thanks Durga We learn from history that we never learn from history.
[OMPI users] Error running mpicc
Hello all The system in question is a CentOS 7 box, that has been running OpenMPI, both the master branch and the 1.10.2 release happily until now. Just now, in order to debug something, I recompiled with the following options: $ ./configure --enable-debug --enable-debug-symbols --disable-dlopen The compilation and install was successful; however, mpicc now crashes like this: [durga@smallMPI ~]$ mpicc -Wall -Wextra -o mpitest mpitest.c mpicc: route/tc.c:973: rtnl_tc_register: Assertion `0' failed. Aborted (core dumped) Searching the mailing archive, I found two posts that describe similar situations: https://www.open-mpi.org/community/lists/devel/2015/08/17812.php http://www.open-mpi.org/community/lists/users/2015/11/28016.php However, the solution proposed in these, to disable verbs, is not acceptable to me for the following reasons: I am trying to implement a new BTL by reverse engineering the openib BTL. I am using a Qlogic HCA for this purpose. (Please note that I cannot use PSM as I am writing code for a BTL) As there any more acceptable solutions for this? Here are the list of nl libraries on my box: [durga@smallMPI ~]$ sudo yum list installed | grep libnl libnl.x86_64 1.1.4-3.el7 @anaconda libnl-devel.x86_64 1.1.4-3.el7 @anaconda libnl3.x86_64 3.2.21-10.el7 @base libnl3-cli.x86_64 3.2.21-10.el7 @base and uninstalling libnl3 is not an option either: it seems yum wants to uninstall around 100 odd other packages because of dependency which will essentially render the machine unusable. Please help! Thanks in advance Durga We learn from history that we never learn from history.
Re: [OMPI users] Error running mpicc
Hello Gilles Thank you very much for your prompt response! Here are the answers to your questions: [durga@smallMPI ~]$ ldd `which mpicc` | grep libnl libnl.so.1 => /lib64/libnl.so.1 (0x7f79b2d8a000) libnl-route-3.so.200 => /lib64/libnl-route-3.so.200 (0x7f79b1c44000) libnl-3.so.200 => /lib64/libnl-3.so.200 (0x7f79b1a28000) So yes, mpicc does seem to need both libnl and libnl3. And this is even though libnl3-devel is NOT installed on my system: [durga@smallMPI ~]$ sudo yum list installed | grep libnl libnl.x86_64 1.1.4-3.el7 @anaconda libnl-devel.x86_64 1.1.4-3.el7 @anaconda libnl3.x86_64 3.2.21-10.el7 @base libnl3-cli.x86_64 3.2.21-10.el7 @base Could it be because of the --disable-dlopen switch to ./configure? The other two switches (--enable-debug and --enable-debug-symbols seem pretty harmless). I'll try your other suggestion and let you know the outcome shortly. Thanks Durga We learn from history that we never learn from history. On Mon, Mar 28, 2016 at 2:37 AM, Gilles Gouaillardet wrote: > Does this happen only with master ? > > what does > ldd mpicc > says ? > does it require both libnl and libnl3 ? > > libnl3 is used by OpenMPI if libnl3-devel package is installed, > and this is not the case on your system > > a possible root cause is third party libs use libnl3, and the > reachable/netlink component > tries to use libnl, in this case, installing libnl3-devel should fix your > issue > /* you will need to re-configure after that */ > > an other possible root cause is some third party libs use libnl and other > use libnl3, > and in this case, i am afraid there is no simple workaround. > if installing libnl3-devel did not solve your issue, you can give a try to > https://github.com/open-mpi/ompi/pull/1014 > at least, it will abort with an error message that states which lib is > using libnl and which is using libnl3 > > i am afraid the only option is to manually disable some components, so > only one flavor of lib nl is used. > that can be achieved by adding a .opal_ignore empty file in the dir of the > components you want to disable. > /* you will need to rerun autogen.pl after that */ > > Cheers, > > Gilles > > On 3/28/2016 3:16 PM, dpchoudh . wrote: > > Hello all > > The system in question is a CentOS 7 box, that has been running OpenMPI, > both the master branch and the 1.10.2 release happily until now. > > Just now, in order to debug something, I recompiled with the following > options: > > $ ./configure --enable-debug --enable-debug-symbols --disable-dlopen > > The compilation and install was successful; however, mpicc now crashes > like this: > > [durga@smallMPI ~]$ mpicc -Wall -Wextra -o mpitest mpitest.c > mpicc: route/tc.c:973: rtnl_tc_register: Assertion `0' failed. > Aborted (core dumped) > > > Searching the mailing archive, I found two posts that describe similar > situations: > > https://www.open-mpi.org/community/lists/devel/2015/08/17812.php > http://www.open-mpi.org/community/lists/users/2015/11/28016.php > > However, the solution proposed in these, to disable verbs, is not > acceptable to me for the following reasons: I am trying to implement a new > BTL by reverse engineering the openib BTL. I am using a Qlogic HCA for this > purpose. (Please note that I cannot use PSM as I am writing code for a BTL) > > As there any more acceptable solutions for this? Here are the list of nl > libraries on my box: > > [durga@smallMPI ~]$ sudo yum list installed | grep libnl > libnl.x86_64 1.1.4-3.el7 > @anaconda > libnl-devel.x86_64 1.1.4-3.el7 > @anaconda > libnl3.x86_64 3.2.21-10.el7 > @base > libnl3-cli.x86_64 3.2.21-10.el7 > @base > > and uninstalling libnl3 is not an option either: it seems yum wants to > uninstall around 100 odd other packages because of dependency which will > essentially render the machine unusable. > > Please help! > > Thanks in advance > Durga > > We learn from history that we never learn from history. > > > ___ > users mailing listus...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2016/03/28855.php > > > > ___ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2016/03/28856.php >
Re: [OMPI users] Error running mpicc
Hello Gilles
Per your suggestion, installing libnl3-devel does fixes the mpicc issue,
but there still seems to be another issue down the road: the generated
executable seems to hang. I have tried sm, tcp and openib BTLs, all with
the same result:
[durga@smallMPI ~]$ mpirun -np 2 -H smallMPI,bigMPI -mca btl self,sm
./mpitest <--- Hangs
The source code for the simple test is as follows:
#include
#include
#include
int main(int argc, char** argv)
{
int world_size, world_rank, name_len;
char hostname[MPI_MAX_PROCESSOR_NAME];
MPI_Init(&argc, &argv);
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
MPI_Get_processor_name(hostname, &name_len);
printf("Hello world from processor %s, rank %d out of %d processors\n",
hostname, world_rank, world_size);
MPI_Finalize();
return 0;
}
What do I do now?
Thanks
Durga
We learn from history that we never learn from history.
On Mon, Mar 28, 2016 at 2:37 AM, Gilles Gouaillardet
wrote:
> Does this happen only with master ?
>
> what does
> ldd mpicc
> says ?
> does it require both libnl and libnl3 ?
>
> libnl3 is used by OpenMPI if libnl3-devel package is installed,
> and this is not the case on your system
>
> a possible root cause is third party libs use libnl3, and the
> reachable/netlink component
> tries to use libnl, in this case, installing libnl3-devel should fix your
> issue
> /* you will need to re-configure after that */
>
> an other possible root cause is some third party libs use libnl and other
> use libnl3,
> and in this case, i am afraid there is no simple workaround.
> if installing libnl3-devel did not solve your issue, you can give a try to
> https://github.com/open-mpi/ompi/pull/1014
> at least, it will abort with an error message that states which lib is
> using libnl and which is using libnl3
>
> i am afraid the only option is to manually disable some components, so
> only one flavor of lib nl is used.
> that can be achieved by adding a .opal_ignore empty file in the dir of the
> components you want to disable.
> /* you will need to rerun autogen.pl after that */
>
> Cheers,
>
> Gilles
>
> On 3/28/2016 3:16 PM, dpchoudh . wrote:
>
> Hello all
>
> The system in question is a CentOS 7 box, that has been running OpenMPI,
> both the master branch and the 1.10.2 release happily until now.
>
> Just now, in order to debug something, I recompiled with the following
> options:
>
> $ ./configure --enable-debug --enable-debug-symbols --disable-dlopen
>
> The compilation and install was successful; however, mpicc now crashes
> like this:
>
> [durga@smallMPI ~]$ mpicc -Wall -Wextra -o mpitest mpitest.c
> mpicc: route/tc.c:973: rtnl_tc_register: Assertion `0' failed.
> Aborted (core dumped)
>
>
> Searching the mailing archive, I found two posts that describe similar
> situations:
>
> https://www.open-mpi.org/community/lists/devel/2015/08/17812.php
> http://www.open-mpi.org/community/lists/users/2015/11/28016.php
>
> However, the solution proposed in these, to disable verbs, is not
> acceptable to me for the following reasons: I am trying to implement a new
> BTL by reverse engineering the openib BTL. I am using a Qlogic HCA for this
> purpose. (Please note that I cannot use PSM as I am writing code for a BTL)
>
> As there any more acceptable solutions for this? Here are the list of nl
> libraries on my box:
>
> [durga@smallMPI ~]$ sudo yum list installed | grep libnl
> libnl.x86_64 1.1.4-3.el7
> @anaconda
> libnl-devel.x86_64 1.1.4-3.el7
> @anaconda
> libnl3.x86_64 3.2.21-10.el7
> @base
> libnl3-cli.x86_64 3.2.21-10.el7
> @base
>
> and uninstalling libnl3 is not an option either: it seems yum wants to
> uninstall around 100 odd other packages because of dependency which will
> essentially render the machine unusable.
>
> Please help!
>
> Thanks in advance
> Durga
>
> We learn from history that we never learn from history.
>
>
> ___
> users mailing listus...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2016/03/28855.php
>
>
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2016/03/28856.php
>
Re: [OMPI users] Error running mpicc
Hello Gilles
Thanks for your prompt response and apologies for the delayed response.
The hang issue is fixed now. It seems that OpenMPI seems to prefer PSM when
it detects Qlogic HCAs, even when I pass -mca btl openib,self. Adding
another parameter, -mca pml ob1 fixed the issue. There is nothing wrong
with the code base (except for the conflicting libnl dependency), and sorry
for confusing you.
Durga
We learn from history that we never learn from history.
On Mon, Mar 28, 2016 at 3:44 AM, Gilles Gouaillardet
wrote:
> at first, does it hang when running on only one node ?
>
> when the hang occur, you can collect stack traces
> (run pstack on mpitest)
> to see where it hangs.
>
> since you configure'd with --disable-dlopen, it means your btl has been
> slurped into openmpi.
> that means some parts of it are executed, and it could be responsible for
> the hang.
>
> note if you
> mpirun --mca btl self,sm -np 2 ...
> on two hosts, then it will never work since no btl can be used so mpi
> tasks can communicate.
>
> it seems something is wrong on master, i will check from now
>
> do smallMPI and bigMPI implies on host is little endian and the other is
> big endian ?
> if yes, then you need to configure with --enable-heterogeneous
>
> Cheers,
>
> Gilles
>
>
> On 3/28/2016 4:26 PM, dpchoudh . wrote:
>
> Hello Gilles
>
> Per your suggestion, installing libnl3-devel does fixes the mpicc issue,
> but there still seems to be another issue down the road: the generated
> executable seems to hang. I have tried sm, tcp and openib BTLs, all with
> the same result:
>
> [durga@smallMPI ~]$ mpirun -np 2 -H smallMPI,bigMPI -mca btl self,sm
> ./mpitest <--- Hangs
>
> The source code for the simple test is as follows:
>
> #include
> #include
> #include
>
> int main(int argc, char** argv)
> {
> int world_size, world_rank, name_len;
> char hostname[MPI_MAX_PROCESSOR_NAME];
> MPI_Init(&argc, &argv);
> MPI_Comm_size(MPI_COMM_WORLD, &world_size);
> MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
> MPI_Get_processor_name(hostname, &name_len);
> printf("Hello world from processor %s, rank %d out of %d
> processors\n", hostname, world_rank, world_size);
> MPI_Finalize();
> return 0;
> }
>
>
> What do I do now?
>
> Thanks
> Durga
>
> We learn from history that we never learn from history.
>
> On Mon, Mar 28, 2016 at 2:37 AM, Gilles Gouaillardet
> wrote:
>
>> Does this happen only with master ?
>>
>> what does
>> ldd mpicc
>> says ?
>> does it require both libnl and libnl3 ?
>>
>> libnl3 is used by OpenMPI if libnl3-devel package is installed,
>> and this is not the case on your system
>>
>> a possible root cause is third party libs use libnl3, and the
>> reachable/netlink component
>> tries to use libnl, in this case, installing libnl3-devel should fix your
>> issue
>> /* you will need to re-configure after that */
>>
>> an other possible root cause is some third party libs use libnl and other
>> use libnl3,
>> and in this case, i am afraid there is no simple workaround.
>> if installing libnl3-devel did not solve your issue, you can give a try to
>> https://github.com/open-mpi/ompi/pull/1014
>> at least, it will abort with an error message that states which lib is
>> using libnl and which is using libnl3
>>
>> i am afraid the only option is to manually disable some components, so
>> only one flavor of lib nl is used.
>> that can be achieved by adding a .opal_ignore empty file in the dir of
>> the components you want to disable.
>> /* you will need to rerun autogen.pl after that */
>>
>> Cheers,
>>
>> Gilles
>>
>> On 3/28/2016 3:16 PM, dpchoudh . wrote:
>>
>> Hello all
>>
>> The system in question is a CentOS 7 box, that has been running OpenMPI,
>> both the master branch and the 1.10.2 release happily until now.
>>
>> Just now, in order to debug something, I recompiled with the following
>> options:
>>
>> $ ./configure --enable-debug --enable-debug-symbols --disable-dlopen
>>
>> The compilation and install was successful; however, mpicc now crashes
>> like this:
>>
>> [durga@smallMPI ~]$ mpicc -Wall -Wextra -o mpitest mpitest.c
>> mpicc: route/tc.c:973: rtnl_tc_register: Assertion `0' failed.
>> Aborted (core dumped)
>>
>>
>> Searching the mailing archive, I found two posts that describe similar
>> situations:
>>
>> https://www.open-mpi.org/community/lists/devel/2015/08/17812.php
>> http:/
[OMPI users] libfabric verb provider for iWARP RNIC
Hello all My machine has 3 network cards: 1. Broadcom GbE (vanilla type, with some offload capability) 2. Chelsion S310 10Gb iWARP 3. Qlogic DDR 4X Infiniband. With this setup, I built libfabric like this: ./configure --enable-udp=auto --enable-gni=auto --enable-mxm=auto --enable-usnic=auto --enable-verbs=auto --enable-sockets=auto --enable-psm2=auto --enable-psm=auto && make && sudo make install However, in the built libfabric, I do not see a verb provider, which I'd expect for the iWARP card, at least. [durga@smallMPI libfabric]$ fi_info psm: psm version: 0.9 type: FI_EP_RDM protocol: FI_PROTO_PSMX UDP: UDP-IP version: 1.0 type: FI_EP_DGRAM protocol: FI_PROTO_UDP sockets: IP version: 1.0 type: FI_EP_MSG protocol: FI_PROTO_SOCK_TCP sockets: IP version: 1.0 type: FI_EP_DGRAM protocol: FI_PROTO_SOCK_TCP sockets: IP version: 1.0 type: FI_EP_RDM protocol: FI_PROTO_SOCK_TCP Am I doing something wrong or misunderstanding how libfabric works? Thanks in advance Durga We learn from history that we never learn from history.
[OMPI users] Newbie question
Hello all
I don't mean to be competing for the 'silliest question of the year award',
but I can't figure this out on my own:
My 'cluster' has 2 machines, bigMPI and smallMPI. They are connected via
several (types of) networks and the connectivity is OK.
In this setup, the following program hangs after printing
Hello world from processor smallMPI, rank 0 out of 2 processors
Hello world from processor bigMPI, rank 1 out of 2 processors
smallMPI sent haha!
Obviously it is hanging at MPI_Recv(). But why? My command line is as
follows, but this happens if I try openib BTL (instead of TCP) as well.
mpirun -np 2 -hostfile ~/hostfile -mca btl self,tcp ./mpitest
It must be something *really* trivial, but I am drawing a blank right now.
Please help!
#include
#include
#include
int main(int argc, char** argv)
{
int world_size, world_rank, name_len;
char hostname[MPI_MAX_PROCESSOR_NAME], buf[8];
MPI_Init(&argc, &argv);
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
MPI_Get_processor_name(hostname, &name_len);
printf("Hello world from processor %s, rank %d out of %d processors\n",
hostname, world_rank, world_size);
if (world_rank == 1)
{
MPI_Recv(buf, 6, MPI_CHAR, 0, 99, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
printf("%s received %s\n", hostname, buf);
}
else
{
strcpy(buf, "haha!");
MPI_Send(buf, 6, MPI_CHAR, 1, 99, MPI_COMM_WORLD);
printf("%s sent %s\n", hostname, buf);
}
MPI_Barrier(MPI_COMM_WORLD);
MPI_Finalize();
return 0;
}
We learn from history that we never learn from history.
Re: [OMPI users] Newbie question
Hello Gilles
Thanks for your help.
My question was more of a sanity check on myself. That little program I
sent looked correct to me; do you see anything wrong with it?
What I am running on my setup is an instrumented OMPI stack, taken from git
HEAD, in an attempt to understand how some of the internals work. If you
think the code is correct, it is quite possible that one of those
'instrumentations' is causing this.
And BTW, adding -mca pml ob1 makes the code hang at MPI_Send (as opposed to
MPI_Recv())
[smallMPI:51673] mca: bml: Using tcp btl for send to [[51894,1],1] on node
10.10.10.11
[smallMPI:51673] mca: bml: Using tcp btl for send to [[51894,1],1] on node
10.10.10.11
[smallMPI:51673] mca: bml: Using tcp btl for send to [[51894,1],1] on node
10.10.10.11
[smallMPI:51673] mca: bml: Using tcp btl for send to [[51894,1],1] on node
10.10.10.11
[smallMPI:51673] btl: tcp: attempting to connect() to [[51894,1],1] address
10.10.10.11 on port 1024 <--- Hangs here
But 10.10.10.11 is pingable:
[durga@smallMPI ~]$ ping bigMPI
PING bigMPI (10.10.10.11) 56(84) bytes of data.
64 bytes from bigMPI (10.10.10.11): icmp_seq=1 ttl=64 time=0.247 ms
We learn from history that we never learn from history.
On Sun, Apr 3, 2016 at 8:04 PM, Gilles Gouaillardet
wrote:
> Hi,
>
> per a previous message, can you give a try to
> mpirun -np 2 -hostfile ~/hostfile -mca btl self,tcp --mca pml ob1 ./mpitest
>
> if it still hangs, the issue could be OpenMPI think some subnets are
> reachable but they are not.
>
> for diagnostic :
> mpirun --mca btl_base_verbose 100 ...
>
> you can explicitly include/exclude subnets with
> --mca btl_tcp_if_include xxx
> or
> --mca btl_tcp_if_exclude yyy
>
> for example,
> mpirun --mca btl_btp_if_include 192.168.0.0/24 -np 2 -hostfile ~/hostfile
> --mca btl self,tcp --mca pml ob1 ./mpitest
> should do the trick
>
> Cheers,
>
> Gilles
>
>
>
>
> On 4/4/2016 8:32 AM, dpchoudh . wrote:
>
> Hello all
>
> I don't mean to be competing for the 'silliest question of the year
> award', but I can't figure this out on my own:
>
> My 'cluster' has 2 machines, bigMPI and smallMPI. They are connected via
> several (types of) networks and the connectivity is OK.
>
> In this setup, the following program hangs after printing
>
> Hello world from processor smallMPI, rank 0 out of 2 processors
> Hello world from processor bigMPI, rank 1 out of 2 processors
> smallMPI sent haha!
>
>
> Obviously it is hanging at MPI_Recv(). But why? My command line is as
> follows, but this happens if I try openib BTL (instead of TCP) as well.
>
> mpirun -np 2 -hostfile ~/hostfile -mca btl self,tcp ./mpitest
>
> It must be something *really* trivial, but I am drawing a blank right now.
>
> Please help!
>
> #include
> #include
> #include
>
> int main(int argc, char** argv)
> {
> int world_size, world_rank, name_len;
> char hostname[MPI_MAX_PROCESSOR_NAME], buf[8];
>
> MPI_Init(&argc, &argv);
> MPI_Comm_size(MPI_COMM_WORLD, &world_size);
> MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
> MPI_Get_processor_name(hostname, &name_len);
> printf("Hello world from processor %s, rank %d out of %d
> processors\n", hostname, world_rank, world_size);
> if (world_rank == 1)
> {
> MPI_Recv(buf, 6, MPI_CHAR, 0, 99, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
> printf("%s received %s\n", hostname, buf);
> }
> else
> {
> strcpy(buf, "haha!");
> MPI_Send(buf, 6, MPI_CHAR, 1, 99, MPI_COMM_WORLD);
> printf("%s sent %s\n", hostname, buf);
> }
> MPI_Barrier(MPI_COMM_WORLD);
> MPI_Finalize();
> return 0;
> }
>
>
>
> We learn from history that we never learn from history.
>
>
> ___
> users mailing listus...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2016/04/28876.php
>
>
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2016/04/28877.php
>
Re: [OMPI users] Newbie question
Hello Gilles
Thanks again for your inputs. Since that code snippet works for you, I am
now fairly certain that my 'instrumentation' has broken something; sorry
for troubling the whole community while I climb the learning curve. The
netcat script that you mention does work correctly; that and that fact that
the issue happens even when I use the openib BTL makes me convinced it is
not a firewall issue.
Best regards
Durga
We learn from history that we never learn from history.
On Sun, Apr 3, 2016 at 9:05 PM, Gilles Gouaillardet
wrote:
> your program works fine on my environment.
>
> this is typical of a firewall running on your host(s), can you double
> check that ?
>
> a simple way to do that is to
> 10.10.10.11# nc -l 1024
>
> and on the other node
> echo ahah | nc 10.10.10.11 1024
>
> the first command should print "ahah" unless the host is unreachable
> and/or the tcp connection is denied by the firewall.
>
> Cheers,
>
> Gilles
>
>
>
> On 4/4/2016 9:44 AM, dpchoudh . wrote:
>
> Hello Gilles
>
> Thanks for your help.
>
> My question was more of a sanity check on myself. That little program I
> sent looked correct to me; do you see anything wrong with it?
>
> What I am running on my setup is an instrumented OMPI stack, taken from
> git HEAD, in an attempt to understand how some of the internals work. If
> you think the code is correct, it is quite possible that one of those
> 'instrumentations' is causing this.
>
> And BTW, adding -mca pml ob1 makes the code hang at MPI_Send (as opposed
> to MPI_Recv())
>
> [smallMPI:51673] mca: bml: Using tcp btl for send to [[51894,1],1] on node
> 10.10.10.11
> [smallMPI:51673] mca: bml: Using tcp btl for send to [[51894,1],1] on node
> 10.10.10.11
> [smallMPI:51673] mca: bml: Using tcp btl for send to [[51894,1],1] on node
> 10.10.10.11
> [smallMPI:51673] mca: bml: Using tcp btl for send to [[51894,1],1] on node
> 10.10.10.11
> [smallMPI:51673] btl: tcp: attempting to connect() to [[51894,1],1]
> address 10.10.10.11 on port 1024 <--- Hangs here
>
> But 10.10.10.11 is pingable:
> [durga@smallMPI ~]$ ping bigMPI
> PING bigMPI (10.10.10.11) 56(84) bytes of data.
> 64 bytes from bigMPI (10.10.10.11): icmp_seq=1 ttl=64 time=0.247 ms
>
>
> We learn from history that we never learn from history.
>
> On Sun, Apr 3, 2016 at 8:04 PM, Gilles Gouaillardet
> wrote:
>
>> Hi,
>>
>> per a previous message, can you give a try to
>> mpirun -np 2 -hostfile ~/hostfile -mca btl self,tcp --mca pml ob1
>> ./mpitest
>>
>> if it still hangs, the issue could be OpenMPI think some subnets are
>> reachable but they are not.
>>
>> for diagnostic :
>> mpirun --mca btl_base_verbose 100 ...
>>
>> you can explicitly include/exclude subnets with
>> --mca btl_tcp_if_include xxx
>> or
>> --mca btl_tcp_if_exclude yyy
>>
>> for example,
>> mpirun --mca btl_btp_if_include 192.168.0.0/24 -np 2 -hostfile
>> ~/hostfile --mca btl self,tcp --mca pml ob1 ./mpitest
>> should do the trick
>>
>> Cheers,
>>
>> Gilles
>>
>>
>>
>>
>> On 4/4/2016 8:32 AM, dpchoudh . wrote:
>>
>> Hello all
>>
>> I don't mean to be competing for the 'silliest question of the year
>> award', but I can't figure this out on my own:
>>
>> My 'cluster' has 2 machines, bigMPI and smallMPI. They are connected via
>> several (types of) networks and the connectivity is OK.
>>
>> In this setup, the following program hangs after printing
>>
>> Hello world from processor smallMPI, rank 0 out of 2 processors
>> Hello world from processor bigMPI, rank 1 out of 2 processors
>> smallMPI sent haha!
>>
>>
>> Obviously it is hanging at MPI_Recv(). But why? My command line is as
>> follows, but this happens if I try openib BTL (instead of TCP) as well.
>>
>> mpirun -np 2 -hostfile ~/hostfile -mca btl self,tcp ./mpitest
>>
>> It must be something *really* trivial, but I am drawing a blank right now.
>>
>> Please help!
>>
>> #include
>> #include
>> #include
>>
>> int main(int argc, char** argv)
>> {
>> int world_size, world_rank, name_len;
>> char hostname[MPI_MAX_PROCESSOR_NAME], buf[8];
>>
>> MPI_Init(&argc, &argv);
>> MPI_Comm_size(MPI_COMM_WORLD, &world_size);
>> MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
>> MPI_Get_processor_name(hostname, &name_len);
>> printf("Hello world from processor %s, rank %d out of %d
>> processors\n", hostname,
Re: [OMPI users] libfabric verb provider for iWARP RNIC
Hi Howard Thank you very much for your suggestions. All the installation location in my case are the default ones, so that is likely not the issue. What I find a bit confusing is this: As I mentioned, my cluster has both Qlogic Infiniband and Chelsio iWARP (which are exposed to OpenMPI natively as well as an IP interface) With this configuration, if I build libfabric with configure options --with-psm=auto --with-verbs=auto, as I mentioned earlier, only the PSM interface shows up in the fi_info listing, and OpenMPI programs using ofi MTL *do* work. However, I do not know if the traffic is going through the Qlogic card or the Chelsio card; it is likely the former. I am going to ask this on the libfabric list, but perhaps the following question is relevant in the OpenMPI list: My understanding about the OFI MTL is the following: please correct me where I am wrong: As I understand, ANY type of transport that exposes a verbs interface (iWARP, RoCE, Infiniband from any manufacturer) can become a libfabric provider (when libfabric is compiled with --with-verbs option) and thus support the OFI MTL (and thus the cm PML?) Is the above true? Best regards Durga We learn from history that we never learn from history. On Mon, Apr 4, 2016 at 7:29 AM, Howard Pritchard wrote: > Hi Durga, > > I'd suggest reposting this to the libfabric-users mail list. > You can join that list at > http://lists.openfabrics.org/mailman/listinfo/libfabric-users > > I'd suggest including the output of config.log. If you installed > ofed in non-canonical location, you may need to give an explicit > path as an argument to the --enable-verbs configury option. > > Note if you're trying to use libfabric with the Open MPI ofi > mtl, you will need to get literally the freshest version of > libfabric, either at github or the 1.3rc2 tarball at > > http://www.openfabrics.org/downloads/ofi/ > > Good luck, > > Howard > > > 2016-04-02 13:41 GMT-06:00 dpchoudh . : > >> Hello all >> >> My machine has 3 network cards: >> >> 1. Broadcom GbE (vanilla type, with some offload capability) >> 2. Chelsion S310 10Gb iWARP >> 3. Qlogic DDR 4X Infiniband. >> >> With this setup, I built libfabric like this: >> >> ./configure --enable-udp=auto --enable-gni=auto --enable-mxm=auto >> --enable-usnic=auto --enable-verbs=auto --enable-sockets=auto >> --enable-psm2=auto --enable-psm=auto && make && sudo make install >> >> However, in the built libfabric, I do not see a verb provider, which I'd >> expect for the iWARP card, at least. >> >> [durga@smallMPI libfabric]$ fi_info >> psm: psm >> version: 0.9 >> type: FI_EP_RDM >> protocol: FI_PROTO_PSMX >> UDP: UDP-IP >> version: 1.0 >> type: FI_EP_DGRAM >> protocol: FI_PROTO_UDP >> sockets: IP >> version: 1.0 >> type: FI_EP_MSG >> protocol: FI_PROTO_SOCK_TCP >> sockets: IP >> version: 1.0 >> type: FI_EP_DGRAM >> protocol: FI_PROTO_SOCK_TCP >> sockets: IP >> version: 1.0 >> type: FI_EP_RDM >> protocol: FI_PROTO_SOCK_TCP >> >> >> Am I doing something wrong or misunderstanding how libfabric works? >> >> Thanks in advance >> Durga >> >> We learn from history that we never learn from history. >> >> ___ >> users mailing list >> us...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >> Link to this post: >> http://www.open-mpi.org/community/lists/users/2016/04/28870.php >> > > > ___ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2016/04/28884.php >
Re: [OMPI users] libfabric verb provider for iWARP RNIC
For the sake of completeness, let me put the answer here. I posted the question on libfabric mailing list and with their input, installed librdmacm-devel. After that and a rebuild, the issue went away. Thanks Durga We learn from history that we never learn from history. On Mon, Apr 4, 2016 at 3:20 PM, dpchoudh . wrote: > Hi Howard > > Thank you very much for your suggestions. All the installation location in > my case are the default ones, so that is likely not the issue. > > What I find a bit confusing is this: > > As I mentioned, my cluster has both Qlogic Infiniband and Chelsio iWARP > (which are exposed to OpenMPI natively as well as an IP interface) > > With this configuration, if I build libfabric with configure options > --with-psm=auto --with-verbs=auto, as I mentioned earlier, only the PSM > interface shows up in the fi_info listing, and OpenMPI programs using ofi > MTL *do* work. However, I do not know if the traffic is going through the > Qlogic card or the Chelsio card; it is likely the former. > > I am going to ask this on the libfabric list, but perhaps the following > question is relevant in the OpenMPI list: > > My understanding about the OFI MTL is the following: please correct me > where I am wrong: > As I understand, ANY type of transport that exposes a verbs interface > (iWARP, RoCE, Infiniband from any manufacturer) can become a libfabric > provider (when libfabric is compiled with --with-verbs option) and thus > support the OFI MTL (and thus the cm PML?) > > Is the above true? > > Best regards > Durga > > We learn from history that we never learn from history. > > On Mon, Apr 4, 2016 at 7:29 AM, Howard Pritchard > wrote: > >> Hi Durga, >> >> I'd suggest reposting this to the libfabric-users mail list. >> You can join that list at >> http://lists.openfabrics.org/mailman/listinfo/libfabric-users >> >> I'd suggest including the output of config.log. If you installed >> ofed in non-canonical location, you may need to give an explicit >> path as an argument to the --enable-verbs configury option. >> >> Note if you're trying to use libfabric with the Open MPI ofi >> mtl, you will need to get literally the freshest version of >> libfabric, either at github or the 1.3rc2 tarball at >> >> http://www.openfabrics.org/downloads/ofi/ >> >> Good luck, >> >> Howard >> >> >> 2016-04-02 13:41 GMT-06:00 dpchoudh . : >> >>> Hello all >>> >>> My machine has 3 network cards: >>> >>> 1. Broadcom GbE (vanilla type, with some offload capability) >>> 2. Chelsion S310 10Gb iWARP >>> 3. Qlogic DDR 4X Infiniband. >>> >>> With this setup, I built libfabric like this: >>> >>> ./configure --enable-udp=auto --enable-gni=auto --enable-mxm=auto >>> --enable-usnic=auto --enable-verbs=auto --enable-sockets=auto >>> --enable-psm2=auto --enable-psm=auto && make && sudo make install >>> >>> However, in the built libfabric, I do not see a verb provider, which I'd >>> expect for the iWARP card, at least. >>> >>> [durga@smallMPI libfabric]$ fi_info >>> psm: psm >>> version: 0.9 >>> type: FI_EP_RDM >>> protocol: FI_PROTO_PSMX >>> UDP: UDP-IP >>> version: 1.0 >>> type: FI_EP_DGRAM >>> protocol: FI_PROTO_UDP >>> sockets: IP >>> version: 1.0 >>> type: FI_EP_MSG >>> protocol: FI_PROTO_SOCK_TCP >>> sockets: IP >>> version: 1.0 >>> type: FI_EP_DGRAM >>> protocol: FI_PROTO_SOCK_TCP >>> sockets: IP >>> version: 1.0 >>> type: FI_EP_RDM >>> protocol: FI_PROTO_SOCK_TCP >>> >>> >>> Am I doing something wrong or misunderstanding how libfabric works? >>> >>> Thanks in advance >>> Durga >>> >>> We learn from history that we never learn from history. >>> >>> ___ >>> users mailing list >>> us...@open-mpi.org >>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>> Link to this post: >>> http://www.open-mpi.org/community/lists/users/2016/04/28870.php >>> >> >> >> ___ >> users mailing list >> us...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >> Link to this post: >> http://www.open-mpi.org/community/lists/users/2016/04/28884.php >> > >
Re: [OMPI users] libfabric verb provider for iWARP RNIC
Hi Howard and all Thank you very much for the information. I have a follow up question: If the vendor of a new type of fabric wants to include support for OpenMPI, then, as long as they can implement a libfabric provider, they can use the OFI MTL without adding any code to the OpenMPI source tree itself. Is the above statement correct? Note that I am trying to contrast against the other possibility, implementing a BTL, that does not add anything to the libfabrics source tree but adds to OpenMPI source tree instead. Thanks Durga We learn from history that we never learn from history. On Mon, Apr 4, 2016 at 7:29 AM, Howard Pritchard wrote: > Hi Durga, > > I'd suggest reposting this to the libfabric-users mail list. > You can join that list at > http://lists.openfabrics.org/mailman/listinfo/libfabric-users > > I'd suggest including the output of config.log. If you installed > ofed in non-canonical location, you may need to give an explicit > path as an argument to the --enable-verbs configury option. > > Note if you're trying to use libfabric with the Open MPI ofi > mtl, you will need to get literally the freshest version of > libfabric, either at github or the 1.3rc2 tarball at > > http://www.openfabrics.org/downloads/ofi/ > > Good luck, > > Howard > > > 2016-04-02 13:41 GMT-06:00 dpchoudh . : > >> Hello all >> >> My machine has 3 network cards: >> >> 1. Broadcom GbE (vanilla type, with some offload capability) >> 2. Chelsion S310 10Gb iWARP >> 3. Qlogic DDR 4X Infiniband. >> >> With this setup, I built libfabric like this: >> >> ./configure --enable-udp=auto --enable-gni=auto --enable-mxm=auto >> --enable-usnic=auto --enable-verbs=auto --enable-sockets=auto >> --enable-psm2=auto --enable-psm=auto && make && sudo make install >> >> However, in the built libfabric, I do not see a verb provider, which I'd >> expect for the iWARP card, at least. >> >> [durga@smallMPI libfabric]$ fi_info >> psm: psm >> version: 0.9 >> type: FI_EP_RDM >> protocol: FI_PROTO_PSMX >> UDP: UDP-IP >> version: 1.0 >> type: FI_EP_DGRAM >> protocol: FI_PROTO_UDP >> sockets: IP >> version: 1.0 >> type: FI_EP_MSG >> protocol: FI_PROTO_SOCK_TCP >> sockets: IP >> version: 1.0 >> type: FI_EP_DGRAM >> protocol: FI_PROTO_SOCK_TCP >> sockets: IP >> version: 1.0 >> type: FI_EP_RDM >> protocol: FI_PROTO_SOCK_TCP >> >> >> Am I doing something wrong or misunderstanding how libfabric works? >> >> Thanks in advance >> Durga >> >> We learn from history that we never learn from history. >> >> ___ >> users mailing list >> us...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >> Link to this post: >> http://www.open-mpi.org/community/lists/users/2016/04/28870.php >> > > > ___ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2016/04/28884.php >
[OMPI users] Debugging help
Hello all I am trying to set a breakpoint during the modex exchange process so I can see the data being passed for different transport type. I assume that this is being done in the context of orted since this is part of process launch. Here is what I did: (All of this pertains to the master branch and NOT the 1.10 release) 1. Built and installed OpenMPI like this: (on two nodes) ./configure --enable-debug --enable-debug-symbols --disable-dlopen && make && sudo make install 2. Compiled a tiny hello-world MPI program, mpitest (on both nodes) 3. Since the modex exchange is a macro now, (it used to be a function call before), I have to put the breakpoint inside a line of code in the macro; I chose the function mca_base_component_to_string(). I hoped that choosing --enable-debug-symbols and --disable-dlopen will make this function visible, but may be I am wrong. Do I need to explicitly add a DLSPEC to libtools? 4. I launched gdb like this: gdb mpirun set args -np 2 -H bigMPI,smallMPI -mca btl tcp,self ./mpitest b mca_base_component_to_string run This told me that the breakpoint is not present in the executable and gdb will try to load a shared object if needed; I chose 'yes'. However, the breakpoint never triggers and the program runs to completion and exits. I have two requests: 1. Please help me understand what I am doing wrong. 2. Is there a (perhaps a sequence of) switch to 'configure' that will create the most debuggable image, while throwing all performance optimization out of the window? This would be a great thing for a developer. Thank you Durga We learn from history that we never learn from history.
[OMPI users] Possible bug in MPI_Barrier() ?
Hi all
I have reported this issue before, but then had brushed it off as something
that was caused by my modifications to the source tree. It looks like that
is not the case.
Just now, I did the following:
1. Cloned a fresh copy from master.
2. Configured with the following flags, built and installed it in my
two-node "cluster".
--enable-debug --enable-debug-symbols --disable-dlopen
3. Compiled the following program, mpitest.c with these flags: -g3 -Wall
-Wextra
4. Ran it like this:
[durga@smallMPI ~]$ mpirun -np 2 -hostfile ~/hostfile -mca btl self,tcp
-mca pml ob1 ./mpitest
With this, the code hangs at MPI_Barrier() on both nodes, after generating
the following output:
Hello world from processor smallMPI, rank 0 out of 2 processors
Hello world from processor bigMPI, rank 1 out of 2 processors
smallMPI sent haha!
bigMPI received haha!
Attaching to the hung process at one node gives the following backtrace:
(gdb) bt
#0 0x7f55b0f41c3d in poll () from /lib64/libc.so.6
#1 0x7f55b03ccde6 in poll_dispatch (base=0x70e7b0, tv=0x7ffd1bb551c0)
at poll.c:165
#2 0x7f55b03c4a90 in opal_libevent2022_event_base_loop (base=0x70e7b0,
flags=2) at event.c:1630
#3 0x7f55b02f0144 in opal_progress () at runtime/opal_progress.c:171
#4 0x7f55b14b4d8b in opal_condition_wait (c=0x7f55b19fec40
, m=0x7f55b19febc0 ) at
../opal/threads/condition.h:76
#5 0x7f55b14b531b in ompi_request_default_wait_all (count=2,
requests=0x7ffd1bb55370, statuses=0x7ffd1bb55340) at request/req_wait.c:287
#6 0x7f55b157a225 in ompi_coll_base_sendrecv_zero (dest=1, stag=-16,
source=1, rtag=-16, comm=0x601280 )
at base/coll_base_barrier.c:63
#7 0x7f55b157a92a in ompi_coll_base_barrier_intra_two_procs
(comm=0x601280 , module=0x7c2630) at
base/coll_base_barrier.c:308
#8 0x7f55b15aafec in ompi_coll_tuned_barrier_intra_dec_fixed
(comm=0x601280 , module=0x7c2630) at
coll_tuned_decision_fixed.c:196
#9 0x7f55b14d36fd in PMPI_Barrier (comm=0x601280
) at pbarrier.c:63
#10 0x00400b0b in main (argc=1, argv=0x7ffd1bb55658) at mpitest.c:26
(gdb)
Thinking that this might be a bug in tuned collectives, since that is what
the stack shows, I ran the program like this (basically adding the ^tuned
part)
[durga@smallMPI ~]$ mpirun -np 2 -hostfile ~/hostfile -mca btl self,tcp
-mca pml ob1 -mca coll ^tuned ./mpitest
It still hangs, but now with a different stack trace:
(gdb) bt
#0 0x7f910d38ac3d in poll () from /lib64/libc.so.6
#1 0x7f910c815de6 in poll_dispatch (base=0x1a317b0, tv=0x7fff43ee3610)
at poll.c:165
#2 0x7f910c80da90 in opal_libevent2022_event_base_loop
(base=0x1a317b0, flags=2) at event.c:1630
#3 0x7f910c739144 in opal_progress () at runtime/opal_progress.c:171
#4 0x7f910db130f7 in opal_condition_wait (c=0x7f910de47c40
, m=0x7f910de47bc0 )
at ../../../../opal/threads/condition.h:76
#5 0x7f910db132d8 in ompi_request_wait_completion (req=0x1b07680) at
../../../../ompi/request/request.h:383
#6 0x7f910db1533b in mca_pml_ob1_send (buf=0x0, count=0,
datatype=0x7f910de1e340 , dst=1, tag=-16,
sendmode=MCA_PML_BASE_SEND_STANDARD,
comm=0x601280 ) at pml_ob1_isend.c:259
#7 0x7f910d9c3b38 in ompi_coll_base_barrier_intra_basic_linear
(comm=0x601280 , module=0x1b092c0) at
base/coll_base_barrier.c:368
#8 0x7f910d91c6fd in PMPI_Barrier (comm=0x601280
) at pbarrier.c:63
#9 0x00400b0b in main (argc=1, argv=0x7fff43ee3a58) at mpitest.c:26
(gdb)
The mpitest.c program is as follows:
#include
#include
#include
int main(int argc, char** argv)
{
int world_size, world_rank, name_len;
char hostname[MPI_MAX_PROCESSOR_NAME], buf[8];
MPI_Init(&argc, &argv);
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
MPI_Get_processor_name(hostname, &name_len);
printf("Hello world from processor %s, rank %d out of %d processors\n",
hostname, world_rank, world_size);
if (world_rank == 1)
{
MPI_Recv(buf, 6, MPI_CHAR, 0, 99, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
printf("%s received %s\n", hostname, buf);
}
else
{
strcpy(buf, "haha!");
MPI_Send(buf, 6, MPI_CHAR, 1, 99, MPI_COMM_WORLD);
printf("%s sent %s\n", hostname, buf);
}
MPI_Barrier(MPI_COMM_WORLD);
MPI_Finalize();
return 0;
}
The hostfile is as follows:
10.10.10.10 slots=1
10.10.10.11 slots=1
The two nodes are connected by three physical and 3 logical networks:
Physical: Gigabit Ethernet, 10G iWARP, 20G Infiniband
Logical: IP (all 3), PSM (Qlogic Infiniband), Verbs (iWARP and Infiniband)
Please note again that this is a fresh, brand new clone.
Is this a bug (perhaps a side effect of --disable-dlopen) or something I am
doing wrong?
Thanks
Durga
We learn from history that we never learn from history.
[OMPI users] Build on FreeBSD
Hello all I understand that FreeBSD is not a supported platform, so this may be an irrelevant piece of information, but let me pass it on anyway in the hope that it might be useful to somebody. OpenMPI 1.10.2 (release) successfully compiles on FreeBSD 10.2 (except for a minor issue of setting LD_LIBRARY_PATH for Fortran compilation). However, the trunk does not build, and the error is as follows: Making all in mca/patcher/linux CC patcher_linux_module.lo patcher_linux_module.c:44:5: error: expected specifier-qualifier-list before 'ElfW' ElfW(Xword)size; ^ patcher_linux_module.c:48:5: error: expected specifier-qualifier-list before 'ElfW' ElfW(Rela) *tab; It looks like the file and directory where this issue occurs is a new addition in the master branch (for release 2.x, I presume); it does not exist in 1.10.2 I know nothing about the intent of the new code, and it looks like it is meant for Linux (only?) anyway, but somehow the autotools is pulling it in for FreeBSD as well. Thanks Durga 1% of the executables have 99% of CPU privilege! Userspace code! Unite!! Occupy the kernel!!!
[OMPI users] openib failover
Hello all As I understand, the openib BTL supports NIC failover, but I am confused about the scope of this support. Let me elaborate: 1. Is the failover support part of MPI specification? 2. Is it an openMPI-specific addition to MPI implementation? 3. Is it a verb-API specification? Since the openib BTL uses only verbs APIs under the hood, it should work on any NIC (e,g. iWARP or RoCE) that support verbs, isn't it? 4. Is it an Infiniband specification? Going through my old mail archive, it seems that openMPI 1.2 release supported this without relying on the IB automatic path migration functionality, so it seems to me that what openMPI provides is independent of IB APM (that plus the openib BTL runs on link types other than Infiniband) 4.1 If it is based on infiniband APM, is this available if I chose to run a MTL (e.g. PSM) instead of the openib BTL? 5. If my understanding is correct on point #4 above (i.e. the openMPI failover is independent of any link specific capability of Infiniband), then why is a similar functionality not provided for other transport type? The only non-verb transport that I currently have access to is TCP, and I don't think the TCP transport provides auto-failover. Can someone with expertise on this please elaborate? Thanks in advance Durga 1% of the executables have 99% of CPU privilege! Userspace code! Unite!! Occupy the kernel!!!
Re: [OMPI users] Possible bug in MPI_Barrier() ?
:ff:fe33: prefixlen 64 scopeid 0x20 ether 32:02:00:33:33:33 txqueuelen 1000 (Ethernet) RX packets 10 bytes 512 (512.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 22 bytes 1536 (1.5 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 lo: flags=73 mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 inet6 ::1 prefixlen 128 scopeid 0x10 loop txqueuelen 0 (Local Loopback) RX packets 26 bytes 1378 (1.3 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 26 bytes 1378 (1.3 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 Please help me with this. I am stuck with the TCP transport, which is the most basic of all transports. Thanks in advance Durga 1% of the executables have 99% of CPU privilege! Userspace code! Unite!! Occupy the kernel!!! On Tue, Apr 12, 2016 at 9:32 PM, Gilles Gouaillardet wrote: > This is quite unlikely, and fwiw, your test program works for me. > > i suggest you check your 3 TCP networks are usable, for example > > $ mpirun -np 2 -hostfile ~/hostfile -mca btl self,tcp -mca pml ob1 --mca > btl_tcp_if_include xxx ./mpitest > > in which xxx is a [list of] interface name : > eth0 > eth1 > ib0 > eth0,eth1 > eth0,ib0 > ... > eth0,eth1,ib0 > > and see where problem start occuring. > > btw, are your 3 interfaces in 3 different subnet ? is routing required > between two interfaces of the same type ? > > Cheers, > > Gilles > > On 4/13/2016 7:15 AM, dpchoudh . wrote: > > Hi all > > I have reported this issue before, but then had brushed it off as > something that was caused by my modifications to the source tree. It looks > like that is not the case. > > Just now, I did the following: > > 1. Cloned a fresh copy from master. > 2. Configured with the following flags, built and installed it in my > two-node "cluster". > --enable-debug --enable-debug-symbols --disable-dlopen > 3. Compiled the following program, mpitest.c with these flags: -g3 -Wall > -Wextra > 4. Ran it like this: > [durga@smallMPI ~]$ mpirun -np 2 -hostfile ~/hostfile -mca btl self,tcp > -mca pml ob1 ./mpitest > > With this, the code hangs at MPI_Barrier() on both nodes, after generating > the following output: > > Hello world from processor smallMPI, rank 0 out of 2 processors > Hello world from processor bigMPI, rank 1 out of 2 processors > smallMPI sent haha! > bigMPI received haha! > > Attaching to the hung process at one node gives the following backtrace: > > (gdb) bt > #0 0x7f55b0f41c3d in poll () from /lib64/libc.so.6 > #1 0x7f55b03ccde6 in poll_dispatch (base=0x70e7b0, tv=0x7ffd1bb551c0) > at poll.c:165 > #2 0x7f55b03c4a90 in opal_libevent2022_event_base_loop > (base=0x70e7b0, flags=2) at event.c:1630 > #3 0x7f55b02f0144 in opal_progress () at runtime/opal_progress.c:171 > #4 0x7f55b14b4d8b in opal_condition_wait (c=0x7f55b19fec40 > , m=0x7f55b19febc0 ) at > ../opal/threads/condition.h:76 > #5 0x7f55b14b531b in ompi_request_default_wait_all (count=2, > requests=0x7ffd1bb55370, statuses=0x7ffd1bb55340) at request/req_wait.c:287 > #6 0x7f55b157a225 in ompi_coll_base_sendrecv_zero (dest=1, stag=-16, > source=1, rtag=-16, comm=0x601280 ) > at base/coll_base_barrier.c:63 > #7 0x7f55b157a92a in ompi_coll_base_barrier_intra_two_procs > (comm=0x601280 , module=0x7c2630) at > base/coll_base_barrier.c:308 > #8 0x7f55b15aafec in ompi_coll_tuned_barrier_intra_dec_fixed > (comm=0x601280 , module=0x7c2630) at > coll_tuned_decision_fixed.c:196 > #9 0x7f55b14d36fd in PMPI_Barrier (comm=0x601280 > ) at pbarrier.c:63 > #10 0x00400b0b in main (argc=1, argv=0x7ffd1bb55658) at > mpitest.c:26 > (gdb) > > Thinking that this might be a bug in tuned collectives, since that is what > the stack shows, I ran the program like this (basically adding the ^tuned > part) > > [durga@smallMPI ~]$ mpirun -np 2 -hostfile ~/hostfile -mca btl self,tcp > -mca pml ob1 -mca coll ^tuned ./mpitest > > It still hangs, but now with a different stack trace: > (gdb) bt > #0 0x7f910d38ac3d in poll () from /lib64/libc.so.6 > #1 0x7f910c815de6 in poll_dispatch (base=0x1a317b0, > tv=0x7fff43ee3610) at poll.c:165 > #2 0x7f910c80da90 in opal_libevent2022_event_base_loop > (base=0x1a317b0, flags=2) at event.c:1630 > #3 0x7f910c739144 in opal_progress () at runtime/opal_progress.c:171 > #4 0x7f910db130f7 in opal_condition_wait (c=0x7f910de47c40 > , m=0x7f910de47bc0 ) > at ../../../../opal/threads/condition.h:76 > #5 0x7f910db132d8 in ompi_request_wait_completion (req=0x1b07680) at > ../../../../ompi/request/request.h:383
Re: [OMPI users] Possible bug in MPI_Barrier() ?
Thank you for your suggestion, Ralph. But it did not make any difference.
Let me say that my code is about a week stale. I just did a git pull and am
building it right now. The build takes quite a bit of time, so I avoid
doing that unless there is a reason. But what I am trying out is the most
basic functionality, so I'd think a week or so of lag would not make a
difference.
Does the stack trace suggest something to you? It seems that the send
hangs; but a 4 byte send should be sent eagerly.
Best regards
'Durga
1% of the executables have 99% of CPU privilege!
Userspace code! Unite!! Occupy the kernel!!!
On Sun, Apr 17, 2016 at 11:55 PM, Ralph Castain wrote:
> Try adding -mca oob_tcp_if_include eno1 to your cmd line and see if that
> makes a difference
>
> On Apr 17, 2016, at 8:43 PM, dpchoudh . wrote:
>
> Hello Gilles and all
>
> I am sorry to be bugging the developers, but this issue seems to be
> nagging me, and I am surprised it does not seem to affect anybody else. But
> then again, I am using the master branch, and most users are probably using
> a released version.
>
> This time I am using a totally different cluster. This has NO verbs
> capable interface; just 2 Ethernet (1 of which has no IP address and hence
> is unusable) plus 1 proprietary interface that currently supports only IP
> traffic. The two IP interfaces (Ethernet and proprietary) are on different
> IP subnets.
>
> My test program is as follows:
>
> #include
> #include
> #include "mpi.h"
> int main(int argc, char *argv[])
> {
> char host[128];
> int n;
> MPI_Init(&argc, &argv);
> MPI_Get_processor_name(host, &n);
> printf("Hello from %s\n", host);
> MPI_Comm_size(MPI_COMM_WORLD, &n);
> printf("The world has %d nodes\n", n);
> MPI_Comm_rank(MPI_COMM_WORLD, &n);
> printf("My rank is %d\n",n);
> //#if 0
> if (n == 0)
> {
> strcpy(host, "ha!");
> MPI_Send(host, strlen(host) + 1, MPI_CHAR, 1, 1, MPI_COMM_WORLD);
> printf("sent %s\n", host);
> }
> else
> {
> //int len = strlen(host) + 1;
> bzero(host, 128);
> MPI_Recv(host, 4, MPI_CHAR, 0, 1, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
> printf("Received %s from rank 0\n", host);
> }
> //#endif
> MPI_Finalize();
> return 0;
> }
>
> This program, when run between two nodes, hangs. The command was:
> [durga@b-1 ~]$ mpirun -np 2 -hostfile ~/hostfile -mca btl self,tcp -mca
> pml ob1 -mca btl_tcp_if_include eno1 ./mpitest
>
> And the hang is with the following output: (eno1 is one of the gigEth
> interfaces, that takes OOB traffic as well)
>
> Hello from b-1
> The world has 2 nodes
> My rank is 0
> Hello from b-2
> The world has 2 nodes
> My rank is 1
>
> Note that if I uncomment the #if 0 - #endif (i.e. comment out the
> MPI_Send()/MPI_Recv() part, the program runs to completion. Also note that
> the printfs following MPI_Send()/MPI_Recv() do not show up on console.
>
> Upon attaching gdb, the stack trace from the master node is as follows:
>
> Missing separate debuginfos, use: debuginfo-install
> glibc-2.17-78.el7.x86_64 libpciaccess-0.13.4-2.el7.x86_64
> (gdb) bt
> #0 0x7f72a533eb7d in poll () from /lib64/libc.so.6
> #1 0x7f72a4cb7146 in poll_dispatch (base=0xee33d0, tv=0x7fff81057b70)
> at poll.c:165
> #2 0x7f72a4caede0 in opal_libevent2022_event_base_loop (base=0xee33d0,
> flags=2) at event.c:1630
> #3 0x7f72a4c4e692 in opal_progress () at runtime/opal_progress.c:171
> #4 0x7f72a0d07ac1 in opal_condition_wait (
> c=0x7f72a5bb1e00 , m=0x7f72a5bb1d80
> )
> at ../../../../opal/threads/condition.h:76
> #5 0x7f72a0d07ca2 in ompi_request_wait_completion (req=0x113eb80)
> at ../../../../ompi/request/request.h:383
> #6 0x7f72a0d09cd5 in mca_pml_ob1_send (buf=0x7fff81057db0, count=4,
> datatype=0x601080 , dst=1, tag=1,
> sendmode=MCA_PML_BASE_SEND_STANDARD, comm=0x601280
> )
> at pml_ob1_isend.c:251
> #7 0x7f72a58d6be3 in PMPI_Send (buf=0x7fff81057db0, count=4,
> type=0x601080 , dest=1, tag=1,
> comm=0x601280 ) at psend.c:78
> #8 0x00400afa in main (argc=1, argv=0x7fff81057f18) at
> mpitest.c:19
> (gdb)
>
> And the backtrace on the non-master node is:
>
> (gdb) bt
> #0 0x7ff3b377e48d in nanosleep () from /lib64/libc.so.6
> #1 0x7ff3b37af014 in usleep () from /lib64/libc.so.6
> #2 0x7ff3b0c922de in OPAL_PMIX_PMIX120_PMIx_Fence (procs=0x0,
> nprocs=0,
> info=0x0, ninfo=0) at src/client/pmix_client_fence.c:100
> #3 0x7ff3b0c5f1a6 in pmix120_fence (procs=0x0, collect_data=0)
> at pmix120_client.c:258
> #4 0x7ff3b3cf8f4b in ompi_mpi_finalize ()
>
Re: [OMPI users] Possible bug in MPI_Barrier() ?
An update:
Building the latest from master did not make any difference; the code still
hangs with identical stack trace as before.
This should be a simple case to reproduce (positively or negatively). Would
somebody in the developer community mind giving it a quick try?
Thank you
Durga
1% of the executables have 99% of CPU privilege!
Userspace code! Unite!! Occupy the kernel!!!
On Mon, Apr 18, 2016 at 12:06 AM, dpchoudh . wrote:
> Thank you for your suggestion, Ralph. But it did not make any difference.
>
> Let me say that my code is about a week stale. I just did a git pull and
> am building it right now. The build takes quite a bit of time, so I avoid
> doing that unless there is a reason. But what I am trying out is the most
> basic functionality, so I'd think a week or so of lag would not make a
> difference.
>
> Does the stack trace suggest something to you? It seems that the send
> hangs; but a 4 byte send should be sent eagerly.
>
> Best regards
> 'Durga
>
> 1% of the executables have 99% of CPU privilege!
> Userspace code! Unite!! Occupy the kernel!!!
>
> On Sun, Apr 17, 2016 at 11:55 PM, Ralph Castain wrote:
>
>> Try adding -mca oob_tcp_if_include eno1 to your cmd line and see if that
>> makes a difference
>>
>> On Apr 17, 2016, at 8:43 PM, dpchoudh . wrote:
>>
>> Hello Gilles and all
>>
>> I am sorry to be bugging the developers, but this issue seems to be
>> nagging me, and I am surprised it does not seem to affect anybody else. But
>> then again, I am using the master branch, and most users are probably using
>> a released version.
>>
>> This time I am using a totally different cluster. This has NO verbs
>> capable interface; just 2 Ethernet (1 of which has no IP address and hence
>> is unusable) plus 1 proprietary interface that currently supports only IP
>> traffic. The two IP interfaces (Ethernet and proprietary) are on different
>> IP subnets.
>>
>> My test program is as follows:
>>
>> #include
>> #include
>> #include "mpi.h"
>> int main(int argc, char *argv[])
>> {
>> char host[128];
>> int n;
>> MPI_Init(&argc, &argv);
>> MPI_Get_processor_name(host, &n);
>> printf("Hello from %s\n", host);
>> MPI_Comm_size(MPI_COMM_WORLD, &n);
>> printf("The world has %d nodes\n", n);
>> MPI_Comm_rank(MPI_COMM_WORLD, &n);
>> printf("My rank is %d\n",n);
>> //#if 0
>> if (n == 0)
>> {
>> strcpy(host, "ha!");
>> MPI_Send(host, strlen(host) + 1, MPI_CHAR, 1, 1, MPI_COMM_WORLD);
>> printf("sent %s\n", host);
>> }
>> else
>> {
>> //int len = strlen(host) + 1;
>> bzero(host, 128);
>> MPI_Recv(host, 4, MPI_CHAR, 0, 1, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
>> printf("Received %s from rank 0\n", host);
>> }
>> //#endif
>> MPI_Finalize();
>> return 0;
>> }
>>
>> This program, when run between two nodes, hangs. The command was:
>> [durga@b-1 ~]$ mpirun -np 2 -hostfile ~/hostfile -mca btl self,tcp -mca
>> pml ob1 -mca btl_tcp_if_include eno1 ./mpitest
>>
>> And the hang is with the following output: (eno1 is one of the gigEth
>> interfaces, that takes OOB traffic as well)
>>
>> Hello from b-1
>> The world has 2 nodes
>> My rank is 0
>> Hello from b-2
>> The world has 2 nodes
>> My rank is 1
>>
>> Note that if I uncomment the #if 0 - #endif (i.e. comment out the
>> MPI_Send()/MPI_Recv() part, the program runs to completion. Also note that
>> the printfs following MPI_Send()/MPI_Recv() do not show up on console.
>>
>> Upon attaching gdb, the stack trace from the master node is as follows:
>>
>> Missing separate debuginfos, use: debuginfo-install
>> glibc-2.17-78.el7.x86_64 libpciaccess-0.13.4-2.el7.x86_64
>> (gdb) bt
>> #0 0x7f72a533eb7d in poll () from /lib64/libc.so.6
>> #1 0x7f72a4cb7146 in poll_dispatch (base=0xee33d0, tv=0x7fff81057b70)
>> at poll.c:165
>> #2 0x7f72a4caede0 in opal_libevent2022_event_base_loop
>> (base=0xee33d0,
>> flags=2) at event.c:1630
>> #3 0x7f72a4c4e692 in opal_progress () at runtime/opal_progress.c:171
>> #4 0x7f72a0d07ac1 in opal_condition_wait (
>> c=0x7f72a5bb1e00 , m=0x7f72a5bb1d80
>> )
>> at ../../../../opal/threads/condition.h:76
>> #5 0x7f72a0d07ca2 in ompi_request_wait_completion (req=0x113eb80)
>> at ../../../../ompi/request/request.h:383
>> #6 0x7f72a0d09cd5 in mca_pml_ob1_send (buf=0x7fff81057db0, count=4,
>> dat
Re: [OMPI users] Possible bug in MPI_Barrier() ?
Hello Gilles Thank you very much for your feedback. You are right that my original stack trace was on code that was several weeks behind, but updating it just now did not seem to make a difference: I am copying the stack from the latest code below: On the master node: (gdb) bt #0 0x7fc0524cbb7d in poll () from /lib64/libc.so.6 #1 0x7fc051e53116 in poll_dispatch (base=0x1aabbe0, tv=0x7fff29fcb240) at poll.c:165 #2 0x7fc051e4adb0 in opal_libevent2022_event_base_loop (base=0x1aabbe0, flags=2) at event.c:1630 #3 0x7fc051de9a00 in opal_progress () at runtime/opal_progress.c:171 #4 0x7fc04ce46b0b in opal_condition_wait (c=0x7fc052d3cde0 , m=0x7fc052d3cd60 ) at ../../../../opal/threads/condition.h:76 #5 0x7fc04ce46cec in ompi_request_wait_completion (req=0x1b7b580) at ../../../../ompi/request/request.h:383 #6 0x7fc04ce48d4f in mca_pml_ob1_send (buf=0x7fff29fcb480, count=4, datatype=0x601080 , dst=1, tag=1, sendmode=MCA_PML_BASE_SEND_STANDARD, comm=0x601280 ) at pml_ob1_isend.c:259 #7 0x7fc052a62d73 in PMPI_Send (buf=0x7fff29fcb480, count=4, type=0x601080 , dest=1, tag=1, comm=0x601280 ) at psend.c:78 #8 0x00400afa in main (argc=1, argv=0x7fff29fcb5e8) at mpitest.c:19 (gdb) And on the non-master node (gdb) bt #0 0x7fad2c32148d in nanosleep () from /lib64/libc.so.6 #1 0x7fad2c352014 in usleep () from /lib64/libc.so.6 #2 0x7fad296412de in OPAL_PMIX_PMIX120_PMIx_Fence (procs=0x0, nprocs=0, info=0x0, ninfo=0) at src/client/pmix_client_fence.c:100 #3 0x7fad2960e1a6 in pmix120_fence (procs=0x0, collect_data=0) at pmix120_client.c:258 #4 0x7fad2c89b2da in ompi_mpi_finalize () at runtime/ompi_mpi_finalize.c:242 #5 0x7fad2c8c5849 in PMPI_Finalize () at pfinalize.c:47 #6 0x00400958 in main (argc=1, argv=0x7fff163879c8) at mpitest.c:30 (gdb) And my configuration was done as follows: $ ./configure --enable-debug --enable-debug-symbols I double checked to ensure that there is not an older installation of OpenMPI that is getting mixed up with the master branch. sudo yum list installed | grep -i mpi shows nothing on both nodes, and pmap -p shows that all the libraries are coming from /usr/local/lib, which seems to be correct. I am also quite sure about the firewall issue (that there is none). I will try out your suggestion on installing from a tarball and see how it goes. Thanks Durga 1% of the executables have 99% of CPU privilege! Userspace code! Unite!! Occupy the kernel!!! On Mon, Apr 18, 2016 at 12:47 AM, Gilles Gouaillardet wrote: > here is your stack trace > > #6 0x7f72a0d09cd5 in mca_pml_ob1_send (buf=0x7fff81057db0, count=4, > datatype=0x601080 , dst=1, tag=1, > sendmode=MCA_PML_BASE_SEND_STANDARD, comm=0x601280 > ) > > at line 251 > > > that would be line 259 in current master, and this file was updated 21 > days ago > and that suggests your master is not quite up to date. > > even if the message is sent eagerly, the ob1 pml does use an internal > request it will wait for. > > btw, did you configure with --enable-mpi-thread-multiple ? > did you configure with --enable-mpirun-prefix-by-default ? > did you configure with --disable-dlopen ? > > at first, i d recommend you download a tarball from > https://www.open-mpi.org/nightly/master, > configure && make && make install > using a new install dir, and check if the issue is still here or not. > > there could be some side effects if some old modules were not removed > and/or if you are > not using the modules you expect. > /* when it hangs, you can pmap and check the path of the openmpi > libraries are the one you expect */ > > what if you do not send/recv but invoke MPI_Barrier multiple times ? > what if you send/recv a one byte message instead ? > did you double check there is no firewall running on your nodes ? > > Cheers, > > Gilles > > > > > > > On 4/18/2016 1:06 PM, dpchoudh . wrote: > > Thank you for your suggestion, Ralph. But it did not make any difference. > > Let me say that my code is about a week stale. I just did a git pull and > am building it right now. The build takes quite a bit of time, so I avoid > doing that unless there is a reason. But what I am trying out is the most > basic functionality, so I'd think a week or so of lag would not make a > difference. > > Does the stack trace suggest something to you? It seems that the send > hangs; but a 4 byte send should be sent eagerly. > > Best regards > 'Durga > > 1% of the executables have 99% of CPU privilege! > Userspace code! Unite!! Occupy the kernel!!! > > On Sun, Apr 17, 2016 at 11:55 PM, Ralph Castain wrote: > >> Try adding -mca oob_tcp_if_include eno1 to your cmd line and see if that >> makes a difference >> >>
Re: [OMPI users] Possible bug in MPI_Barrier() ?
Hello Gilles I did a sudo make uninstall followed by a sudo make install on both nodes. But that did not make a difference. I will try your tarball build suggestion a bit later. What I find a bit strange is that only I seem to be getting into this issue. What could I be doing wrong? Or am I discovering an obscure bug? Thanks Durga 1% of the executables have 99% of CPU privilege! Userspace code! Unite!! Occupy the kernel!!! On Mon, Apr 18, 2016 at 1:21 AM, Gilles Gouaillardet wrote: > so you might want to > rm -rf /usr/local/lib/openmpi > and run > make install > again, just to make sure old stuff does not get in the way > > Cheers, > > Gilles > > > On 4/18/2016 2:12 PM, dpchoudh . wrote: > > Hello Gilles > > Thank you very much for your feedback. You are right that my original > stack trace was on code that was several weeks behind, but updating it just > now did not seem to make a difference: I am copying the stack from the > latest code below: > > On the master node: > > (gdb) bt > #0 0x7fc0524cbb7d in poll () from /lib64/libc.so.6 > #1 0x7fc051e53116 in poll_dispatch (base=0x1aabbe0, > tv=0x7fff29fcb240) at poll.c:165 > #2 0x7fc051e4adb0 in opal_libevent2022_event_base_loop > (base=0x1aabbe0, flags=2) at event.c:1630 > #3 0x7fc051de9a00 in opal_progress () at runtime/opal_progress.c:171 > #4 0x7fc04ce46b0b in opal_condition_wait (c=0x7fc052d3cde0 > , > m=0x7fc052d3cd60 ) at > ../../../../opal/threads/condition.h:76 > #5 0x7fc04ce46cec in ompi_request_wait_completion (req=0x1b7b580) > at ../../../../ompi/request/request.h:383 > #6 0x7fc04ce48d4f in mca_pml_ob1_send (buf=0x7fff29fcb480, count=4, > datatype=0x601080 , dst=1, tag=1, > sendmode=MCA_PML_BASE_SEND_STANDARD, > comm=0x601280 ) at pml_ob1_isend.c:259 > #7 0x7fc052a62d73 in PMPI_Send (buf=0x7fff29fcb480, count=4, > type=0x601080 , dest=1, > tag=1, comm=0x601280 ) at psend.c:78 > #8 0x00400afa in main (argc=1, argv=0x7fff29fcb5e8) at > mpitest.c:19 > (gdb) > > And on the non-master node > > (gdb) bt > #0 0x7fad2c32148d in nanosleep () from /lib64/libc.so.6 > #1 0x7fad2c352014 in usleep () from /lib64/libc.so.6 > #2 0x7fad296412de in OPAL_PMIX_PMIX120_PMIx_Fence (procs=0x0, > nprocs=0, info=0x0, ninfo=0) > at src/client/pmix_client_fence.c:100 > #3 0x7fad2960e1a6 in pmix120_fence (procs=0x0, collect_data=0) at > pmix120_client.c:258 > #4 0x7fad2c89b2da in ompi_mpi_finalize () at > runtime/ompi_mpi_finalize.c:242 > #5 0x7fad2c8c5849 in PMPI_Finalize () at pfinalize.c:47 > #6 0x00400958 in main (argc=1, argv=0x7fff163879c8) at > mpitest.c:30 > (gdb) > > And my configuration was done as follows: > > $ ./configure --enable-debug --enable-debug-symbols > > I double checked to ensure that there is not an older installation of > OpenMPI that is getting mixed up with the master branch. > sudo yum list installed | grep -i mpi > shows nothing on both nodes, and pmap -p shows that all the > libraries are coming from /usr/local/lib, which seems to be correct. I am > also quite sure about the firewall issue (that there is none). I will try > out your suggestion on installing from a tarball and see how it goes. > > Thanks > Durga > > 1% of the executables have 99% of CPU privilege! > Userspace code! Unite!! Occupy the kernel!!! > > On Mon, Apr 18, 2016 at 12:47 AM, Gilles Gouaillardet < > gil...@rist.or.jp> wrote: > >> here is your stack trace >> >> #6 0x7f72a0d09cd5 in mca_pml_ob1_send (buf=0x7fff81057db0, count=4, >> datatype=0x601080 , dst=1, tag=1, >> sendmode=MCA_PML_BASE_SEND_STANDARD, comm=0x601280 >> ) >> >> at line 251 >> >> >> that would be line 259 in current master, and this file was updated 21 >> days ago >> and that suggests your master is not quite up to date. >> >> even if the message is sent eagerly, the ob1 pml does use an internal >> request it will wait for. >> >> btw, did you configure with --enable-mpi-thread-multiple ? >> did you configure with --enable-mpirun-prefix-by-default ? >> did you configure with --disable-dlopen ? >> >> at first, i d recommend you download a tarball from >> <https://www.open-mpi.org/nightly/master> >> https://www.open-mpi.org/nightly/master, >> configure && make && make install >> using a new install dir, and check if the issue is still here or not. >> >> there could be some side effects if some old modules were not removed >> and/or if you are >> not using the modules you expect. >> /* when it hangs, you can pmap and c
Re: [OMPI users] OMPI users] Possible bug in MPI_Barrier() ?
Dear developers Thank you all for jumping in to help. Here is what I have found so far: 1. Running Netpipe (NPmpi) between the two nodes (in either order) was successful, but following this test, my original code still hung. 2. Following Gilles's advice, I then added an MPI_Barrier() at the end of the code, just before MPI_Finalize(), and, to my surprise, the code ran to completion! 3. Then, I took out the barrier, leaving the code the way it was before, and it still ran to completion! 4. I tried several variations of call sequence, and all of them ran successfully. I can't explain why the runtime behavior seems to depend on the phase of the moon, but, although I cannot prove it, I have a gut feeling there is a bug somewhere in the development branch. I never run into this issue when running the release branch. (I sometimes work as MPI application developer, when I use the release branch, and sometime as MPI developer, when I use the master branch). Thank you all, again. Durga 1% of the executables have 99% of CPU privilege! Userspace code! Unite!! Occupy the kernel!!! On Mon, Apr 18, 2016 at 8:04 AM, George Bosilca wrote: > Durga, > > Can you run a simple netpipe over TCP using any of the two interfaces you > mentioned? > > George > On Apr 18, 2016 11:08 AM, "Gilles Gouaillardet" < > gilles.gouaillar...@gmail.com> wrote: > >> An other test is to swap the hostnames. >> If the single barrier test fails, this can hint to a firewall. >> >> Cheers, >> >> Gilles >> >> Gilles Gouaillardet wrote: >> sudo make uninstall >> will not remove modules that are no more built >> sudo rm -rf /usr/local/lib/openmpi >> is safe thought >> >> i confirm i did not see any issue on a system with two networks >> >> Cheers, >> >> Gilles >> >> On 4/18/2016 2:53 PM, dpchoudh . wrote: >> >> Hello Gilles >> >> I did a >> sudo make uninstall >> followed by a >> sudo make install >> on both nodes. But that did not make a difference. I will try your >> tarball build suggestion a bit later. >> >> What I find a bit strange is that only I seem to be getting into this >> issue. What could I be doing wrong? Or am I discovering an obscure bug? >> >> Thanks >> Durga >> >> 1% of the executables have 99% of CPU privilege! >> Userspace code! Unite!! Occupy the kernel!!! >> >> On Mon, Apr 18, 2016 at 1:21 AM, Gilles Gouaillardet >> wrote: >> >>> so you might want to >>> rm -rf /usr/local/lib/openmpi >>> and run >>> make install >>> again, just to make sure old stuff does not get in the way >>> >>> Cheers, >>> >>> Gilles >>> >>> >>> On 4/18/2016 2:12 PM, dpchoudh . wrote: >>> >>> Hello Gilles >>> >>> Thank you very much for your feedback. You are right that my original >>> stack trace was on code that was several weeks behind, but updating it just >>> now did not seem to make a difference: I am copying the stack from the >>> latest code below: >>> >>> On the master node: >>> >>> (gdb) bt >>> #0 0x7fc0524cbb7d in poll () from /lib64/libc.so.6 >>> #1 0x7fc051e53116 in poll_dispatch (base=0x1aabbe0, >>> tv=0x7fff29fcb240) at poll.c:165 >>> #2 0x7fc051e4adb0 in opal_libevent2022_event_base_loop >>> (base=0x1aabbe0, flags=2) at event.c:1630 >>> #3 0x7fc051de9a00 in opal_progress () at runtime/opal_progress.c:171 >>> #4 0x7fc04ce46b0b in opal_condition_wait (c=0x7fc052d3cde0 >>> , >>> m=0x7fc052d3cd60 ) at >>> ../../../../opal/threads/condition.h:76 >>> #5 0x7fc04ce46cec in ompi_request_wait_completion (req=0x1b7b580) >>> at ../../../../ompi/request/request.h:383 >>> #6 0x7fc04ce48d4f in mca_pml_ob1_send (buf=0x7fff29fcb480, count=4, >>> datatype=0x601080 , dst=1, tag=1, >>> sendmode=MCA_PML_BASE_SEND_STANDARD, >>> comm=0x601280 ) at pml_ob1_isend.c:259 >>> #7 0x7fc052a62d73 in PMPI_Send (buf=0x7fff29fcb480, count=4, >>> type=0x601080 , dest=1, >>> tag=1, comm=0x601280 ) at psend.c:78 >>> #8 0x00400afa in main (argc=1, argv=0x7fff29fcb5e8) at >>> mpitest.c:19 >>> (gdb) >>> >>> And on the non-master node >>> >>> (gdb) bt >>> #0 0x7fad2c32148d in nanosleep () from /lib64/libc.so.6 >>> #1 0x7fad2c352014 in usleep () from /lib64/libc.so.6 >>> #2 0x7fad296412de in OPAL_PMIX_PMIX120_PMIx_Fence (procs=0x0,
[OMPI users] make install warns about 'common symbols'
Hello all While doing a 'make install' with some additional code written by me, I get the following message: WARNING! Common symbols found: Doing a search on previous mails, I found the following thread that is pertinent: https://www.open-mpi.org/community/lists/devel/2015/04/17220.php However, I could not find out exactly what this warning/error is trying to tell me: is there a multiple definition of a global symbol? (that should be an error, not a warning, right?) Thanks in advance Durga 1% of the executables have 99% of CPU privilege! Userspace code! Unite!! Occupy the kernel!!!
Re: [OMPI users] make install warns about 'common symbols'
Thank you very much, Gilles. Declaring globals as static indeed fixed it, even without initialization (some are complicated structures, so initializing at declaration is not easy.) Best regards Durga 1% of the executables have 99% of CPU privilege! Userspace code! Unite!! Occupy the kernel!!! On Tue, Apr 19, 2016 at 10:58 PM, Gilles Gouaillardet wrote: > This is just a warning. > > there are some potential issues with uninitialized common symbols, but i > cannot remember the details. > > bottom line, they should be avoided : > - declare global variables as static if the scope is only one source file > - always initialize global variables > > Cheers, > > Gilles > > > On 4/20/2016 11:48 AM, dpchoudh . wrote: > > Hello all > > While doing a 'make install' with some additional code written by me, I > get the following message: > WARNING! Common symbols found: > Doing a search on previous mails, I found the following thread that is > pertinent: > https://www.open-mpi.org/community/lists/devel/2015/04/17220.php > > However, I could not find out exactly what this warning/error is trying to > tell me: is there a multiple definition of a global symbol? (that should be > an error, not a warning, right?) > > Thanks in advance > Durga > > 1% of the executables have 99% of CPU privilege! > Userspace code! Unite!! Occupy the kernel!!! > > > ___ > users mailing listus...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2016/04/28974.php > > > > ___ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2016/04/28975.php >
Re: [OMPI users] MPIRUN SEGMENTATION FAULT
Elio You should ask this question in the forum of the simulation program you are using. These failures have most likely nothing to do with MPI (or, at least, OpenMPI) so this is the wrong place for these questions. Here is a bit of suggestion: does your program run without MPI at all? (i.e. in a stand-alone mode or perhaps with a different SPMD model such as OpenMP). If so, try running it in that mode to see if it behaves any better. Even if it does not, the stack trace will be more insightful. With OMPI's process launcher getting mixed up with your code stack, the source of the crash can be harder to figure out. HTH, Durga 1% of the executables have 99% of CPU privilege! Userspace code! Unite!! Occupy the kernel!!! On Sat, Apr 23, 2016 at 4:10 PM, Elio Physics wrote: > Well, I changed the compiler from mpif90 to mpiifort with corresponding > flags -i8 -g and recompiled. i am not getting the segmentation fault > problem anymore and the program runs but later stops with no errors (except > that the Fermi energy was not found!) and with some strange empty files > that are produced something like: fortDgcQe3 fortechvF2 fortMaN6a1 > fortnxoYy1 fortvR5F8q. i still feel something is wrong.. Does anybody > know what are these files? > > > Regards > > > -- > *From:* users on behalf of Ralph Castain < > r...@open-mpi.org> > *Sent:* Saturday, April 23, 2016 1:38 PM > *To:* Open MPI Users > *Subject:* Re: [OMPI users] MPIRUN SEGMENTATION FAULT > > I don’t see any way this could be compilation related - I suspect there is > simply some error in the program (e.g., forgetting to initialize some > memory region). > > > On Apr 23, 2016, at 8:03 AM, Elio Physics wrote: > > Hello Andy, > > the program is not mine. I have got it from a group upon request. It might > be program related because I run other codes such as quantum espresso and > work perfectly fine although it is the cluster people who compiled it. > Since I have compiled the program I am having problems with, I am thinking > that it might be "compilation" related. This is why i wanted some experts' > opinion on this > > > > -- > *From:* users on behalf of Andy Riebs < > andy.ri...@hpe.com> > *Sent:* Saturday, April 23, 2016 12:49 PM > *To:* us...@open-mpi.org > *Subject:* Re: [OMPI users] MPIRUN SEGMENTATION FAULT > > The challenge for the MPI experts here (of which I am NOT one!) is that > the problem appears to be in your program; MPI is simply reporting that > your program failed. If you got the program from someone else, you will > need to solicit their help. If you wrote it, well, it is never a bad time > to learn to use gdb! > > Best regards > Andy > > On 04/23/2016 10:41 AM, Elio Physics wrote: > > I am not really an expert with gdb. What is the core file? and how to use > gdb? I have got three files as an output when the executable is used. One > is the actual output which stops and the other two are error files (from > which I knew about the segmentation fault). > > > thanks > > > -- > *From:* users on > behalf of Ralph Castain > *Sent:* Saturday, April 23, 2016 11:39 AM > *To:* Open MPI Users > *Subject:* Re: [OMPI users] MPIRUN SEGMENTATION FAULT > > valgrind isn’t going to help here - there are multiple reasons why your > application could be segfaulting. Take a look at the core file with gdb and > find out where it is failing. > > On Apr 22, 2016, at 10:20 PM, Elio Physics wrote: > > One more thing i forgot to mention in my previous e-mail. In the output > file I get the following message: > > > 2 total processes killed (some possibly by mpirun during cleanup) > > Thanks > > > > -- > *From:* users on behalf of Elio Physics < > elio-phys...@live.com> > *Sent:* Saturday, April 23, 2016 3:07 AM > *To:* Open MPI Users > *Subject:* Re: [OMPI users] MPIRUN SEGMENTATION FAULT > > I have used valgrind and this is what i got: > > valgrind mpirun ~/Elie/SPRKKR/bin/kkrscf6.3MPI Fe_SCF.inp > > scf-51551.jlborges.fisica.ufmg.br.out > ==8135== Memcheck, a memory error detector > ==8135== Copyright (C) 2002-2012, and GNU GPL'd, by Julian Seward et al. > ==8135== Using Valgrind-3.8.1 and LibVEX; rerun with -h for copyright info > ==8135== Command: mpirun /home/emoujaes/Elie/SPRKKR/bin/kkrscf6.3MPI > Fe_SCF.inp > ==8135== > -- > mpirun noticed that process rank 0 with PID 8147 on node > jlborges.fisica.ufmg.br exited on signal 11 (Segmentation fault). > -- > ==8135== > ==8135== HEAP SUMMARY: > ==8135== in use at exit: 485,683 bytes in 1,899 blocks > ==8135== total heap usage: 7,723 allocs, 5,824 frees, 12,185,660 bytes > allocated > ==8135== > ==8135== LEAK SUMMARY: > ==8135==definitely lost: 34,944 bytes in 34 blocks > ==8135==indirectly lost: 26,613 bytes in 58 blocks > ==8135=
Re: [OMPI users] track progress of a mpi gather
Hello I am not sure I am understanding your requirements correctly, but base on what I think it is, how about this: you do an MPI_Send() from all the non-root nodes to the root node and pack all the progress related data into this send. Use a special tag for this message to make it stand out from 'regular' sends. The root node does a non-blocking receive on this tag from all the nodes that it is expecting this message from. Would that work? Durga 1% of the executables have 99% of CPU privilege! Userspace code! Unite!! Occupy the kernel!!! On Sun, Apr 24, 2016 at 7:05 AM, MM wrote: > Hello, > > With a miniature case of 3 linux quadcore boxes, linked via 1Gbit > Ethernet, I have a UI that runs on 1 of the 3 boxes, and that is the root > of the communicator. > I have a 1-second-running function on up to 10 parameters, my parameter > space fits in the memory of the root, the space of it is N ~~ 1 million. > > I use broadcast/scatter/gather to collect the value of my function on each > of the 1million points, dividing 1million by the number of nodes (11: > rootnode has 1 core/thread assigned to the UI, 1 core/thread for its > evaluation of the function on its own part of the parameter space and 2 > other cores run non-root nodes, and the 2 other boxes all run non-root > nodes) > > the rootnode does: > 1. broadcast needed data > 2. scatter param space > 3. evaluate function locally > 4. gather results from this and all other nodes > > How would I go about having the non-root nodes send a sort of progress > status to the root node? that's used for plotting results on the root box > as soon as they are available? > > Rds, > > > > > > ___ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2016/04/29013.php >
Re: [OMPI users] track progress of a mpi gather
Hello Gilles That idea crossed my mind as well, but I was under the impression that MPI_THREAD_MULTIPLE is not very well supported on OpenMPI? I believe it is not supported on OpenIB, but the original poster seems to be using TCP. Does it work for TCP? Thanks Durga 1% of the executables have 99% of CPU privilege! Userspace code! Unite!! Occupy the kernel!!! On Sun, Apr 24, 2016 at 10:48 AM, Gilles Gouaillardet < gilles.gouaillar...@gmail.com> wrote: > an other option is to use MPI_THREAD_MULTIPLE, and MPI_Recv() on the > master task in a dedicated thread, and use a unique tag (or MPI_Comm_dup() > MPI_COMM_WORLD) to separate the traffic. > > If this is not the desired design, then the master task has to post > MPI_Irecv() and "poll" with MPI_Probe() / MPI_Test() and friends. > Note it is possible to use non blocking collective (MPI_Ibcast(), > MPI_Iscatter() and MPI_Igather()) and "both" collective and the progress > statuses > > Cheers, > > Gilles > > On Sunday, April 24, 2016, dpchoudh . wrote: > >> Hello >> >> >> I am not sure I am understanding your requirements correctly, but base on >> what I think it is, how about this: you do an MPI_Send() from all the >> non-root nodes to the root node and pack all the progress related data into >> this send. Use a special tag for this message to make it stand out from >> 'regular' sends. The root node does a non-blocking receive on this tag from >> all the nodes that it is expecting this message from. >> >> Would that work? >> >> Durga >> >> >> 1% of the executables have 99% of CPU privilege! >> Userspace code! Unite!! Occupy the kernel!!! >> >> On Sun, Apr 24, 2016 at 7:05 AM, MM wrote: >> >>> Hello, >>> >>> With a miniature case of 3 linux quadcore boxes, linked via 1Gbit >>> Ethernet, I have a UI that runs on 1 of the 3 boxes, and that is the root >>> of the communicator. >>> I have a 1-second-running function on up to 10 parameters, my parameter >>> space fits in the memory of the root, the space of it is N ~~ 1 million. >>> >>> I use broadcast/scatter/gather to collect the value of my function on >>> each of the 1million points, dividing 1million by the number of nodes (11: >>> rootnode has 1 core/thread assigned to the UI, 1 core/thread for its >>> evaluation of the function on its own part of the parameter space and 2 >>> other cores run non-root nodes, and the 2 other boxes all run non-root >>> nodes) >>> >>> the rootnode does: >>> 1. broadcast needed data >>> 2. scatter param space >>> 3. evaluate function locally >>> 4. gather results from this and all other nodes >>> >>> How would I go about having the non-root nodes send a sort of progress >>> status to the root node? that's used for plotting results on the root box >>> as soon as they are available? >>> >>> Rds, >>> >>> >>> >>> >>> >>> ___ >>> users mailing list >>> us...@open-mpi.org >>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>> Link to this post: >>> http://www.open-mpi.org/community/lists/users/2016/04/29013.php >>> >> >> > ___ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2016/04/29015.php >
[OMPI users] Cannot run a simple MPI program
Hello all Attached is a simple MPI program (a modified version of a similar program that was posted by another user). This program, when run on a single node machine, hangs most of the time, as follows: (in all cases, OS was CentOS 7) Scenario 1: OMPI v 1.10, single socket quad core machine, with Chelsio T3 card, link down, and GigE, link up mpirun -np 2 Backtrace of the two spawned processes as follows: (gdb) bt #0 0x7f6471647aba in mca_btl_vader_component_progress () at btl_vader_component.c:708 #1 0x7f6475c6722a in opal_progress () at runtime/opal_progress.c:187 #2 0x7f64767b7685 in opal_condition_wait (c=, m=) at ../opal/threads/condition.h:78 #3 ompi_request_default_wait_all (count=2, requests=0x7ffd1d921530, statuses=0x7ffd1d921540) at request/req_wait.c:281 #4 0x7f64709dd591 in ompi_coll_tuned_sendrecv_zero (stag=-16, rtag=-16, comm=, source=1, dest=1) at coll_tuned_barrier.c:78 #5 ompi_coll_tuned_barrier_intra_two_procs (comm=0x6022c0 , module=) at coll_tuned_barrier.c:324 #6 0x7f64767c92e6 in PMPI_Barrier (comm=0x6022c0 ) at pbarrier.c:70 #7 0x004010bd in main (argc=1, argv=0x7ffd1d9217d8) at mpi_hello_master_slave.c:115 (gdb) (gdb) bt #0 mca_pml_ob1_progress () at pml_ob1_progress.c:45 #1 0x7feeae7dc22a in opal_progress () at runtime/opal_progress.c:187 #2 0x7feea9e125c5 in opal_condition_wait (c=, m=) at ../../../../opal/threads/condition.h:78 #3 ompi_request_wait_completion (req=0xe55200) at ../../../../ompi/request/request.h:381 #4 mca_pml_ob1_recv (addr=, count=255, datatype=, src=, tag=, comm=, status=0x7fff4a618000) at pml_ob1_irecv.c:118 #5 0x7feeaf35068f in PMPI_Recv (buf=0x7fff4a618020, count=255, type=0x6020c0 , source=, tag=, comm=0x6022c0 , status=0x7fff4a618000) at precv.c:78 #6 0x00400d49 in slave () at mpi_hello_master_slave.c:67 #7 0x004010b3 in main (argc=1, argv=0x7fff4a6184d8) at mpi_hello_master_slave.c:113 (gdb) Scenario 2: Dual socket Hexcore machine with Qlogic IB, Chelsio iWARP and Fibre Channel, all link down, GigE, link up, OpenMPI compiled from master branch, crashes as follows: [durga@smallMPI Desktop]$ mpirun -np 2 ./mpi_hello_master_slave mpi_hello_master_slave:39570 terminated with signal 11 at PC=20 SP=7ffd438c00b8. Backtrace: mpi_hello_master_slave:39571 terminated with signal 11 at PC=20 SP=7ffee5903e08. Backtrace: --- Primary job terminated normally, but 1 process returned a non-zero exit code. Per user-direction, the job has been aborted. --- -- mpirun noticed that process rank 0 with PID 0 on node smallMPI exited on signal 11 (Segmentation fault). -- Scenario 3: Exactly same as scenario 2, but with command line more explicit as follows: [durga@smallMPI Desktop]$ mpirun -np 2 -mca btl self,sm ./mpi_hello_master_slave This hangs with the following backtrace: (gdb) bt #0 0x7ff6639f049d in nanosleep () from /lib64/libc.so.6 #1 0x7ff663a210d4 in usleep () from /lib64/libc.so.6 #2 0x7ff662f72796 in OPAL_PMIX_PMIX120_PMIx_Fence (procs=0x0, nprocs=0, info=0x0, ninfo=0) at src/client/pmix_client_fence.c:100 #3 0x7ff662f4f0bc in pmix120_fence (procs=0x0, collect_data=0) at pmix120_client.c:255 #4 0x7ff663f941af in ompi_mpi_init (argc=1, argv=0x7ffc18c9afd8, requested=0, provided=0x7ffc18c9adac) at runtime/ompi_mpi_init.c:813 #5 0x7ff663fc9c33 in PMPI_Init (argc=0x7ffc18c9addc, argv=0x7ffc18c9add0) at pinit.c:66 #6 0x0040101f in main (argc=1, argv=0x7ffc18c9afd8) at mpi_hello_master_slave.c:94 (gdb) q (gdb) bt #0 0x7f5af7646117 in sched_yield () from /lib64/libc.so.6 #1 0x7f5af7323875 in amsh_ep_connreq_wrap () from /lib64/libpsm_infinipath.so.1 #2 0x7f5af7324254 in amsh_ep_connect () from /lib64/libpsm_infinipath.so.1 #3 0x7f5af732d0df in psm_ep_connect () from /lib64/libpsm_infinipath.so.1 #4 0x7f5af7d94a69 in ompi_mtl_psm_add_procs (mtl=0x7f5af80f8500 , nprocs=2, procs=0xf53e60) at mtl_psm.c:312 #5 0x7f5af7df3630 in mca_pml_cm_add_procs (procs=0xf53e60, nprocs=2) at pml_cm.c:134 #6 0x7f5af7bcc0d1 in ompi_mpi_init (argc=1, argv=0x7ffc485a2f98, requested=0, provided=0x7ffc485a2d6c) at runtime/ompi_mpi_init.c:777 #7 0x7f5af7c01c33 in PMPI_Init (argc=0x7ffc485a2d9c, argv=0x7ffc485a2d90) at pinit.c:66 #8 0x0040101f in main (argc=1, argv=0x7ffc485a2f98) at mpi_hello_master_slave.c:94 This seems to suggest that it is trying PSM to connect even when the link was down and it was not mentioned in the command line. Is this behavior expected? Scenario 4: Exactly same as scenario 3, but with even more explicit command line: [durga@smallMPI Desktop]$ mpirun -np 2 -mca btl self,sm -mca pml ob1 ./mpi_hello_mas
Re: [OMPI users] Cannot run a simple MPI program
Hello Gilles Thank you for finding the bug; it was not there in the original code; I added it while trying to 'simplify' the code. With the bug fixed, the code now runs in the last scenario. But it still hangs with the following command line (even after updating to latest git tree, rebuilding and reinstalling): mpirun -np 2 -mca btl self,sm ./mpi_hello_master_slave and the stack is still as before: (gdb) bt #0 0x7f4e4bd60117 in sched_yield () from /lib64/libc.so.6 #1 0x7f4e4ba3d875 in amsh_ep_connreq_wrap () from /lib64/libpsm_infinipath.so.1 #2 0x7f4e4ba3e254 in amsh_ep_connect () from /lib64/libpsm_infinipath.so.1 #3 0x7f4e4ba470df in psm_ep_connect () from /lib64/libpsm_infinipath.so.1 #4 0x7f4e4c4c8975 in ompi_mtl_psm_add_procs (mtl=0x7f4e4c846500 , nprocs=2, procs=0x23bb420) at mtl_psm.c:312 #5 0x7f4e4c52ef6b in mca_pml_cm_add_procs (procs=0x23bb420, nprocs=2) at pml_cm.c:134 #6 0x7f4e4c2e7d0f in ompi_mpi_init (argc=1, argv=0x7fffe930f9b8, requested=0, provided=0x7fffe930f78c) at runtime/ompi_mpi_init.c:770 #7 0x7f4e4c324aff in PMPI_Init (argc=0x7fffe930f7bc, argv=0x7fffe930f7b0) at pinit.c:66 #8 0x0040101f in main (argc=1, argv=0x7fffe930f9b8) at mpi_hello_master_slave.c:94 As you can see, OMPI is trying the PSM link to communicate, even though the link is down and it is not mentioned in the arguments to mpirun. (There are not even multiple nodes mentioned in the arguments.) Is this the expected behaviour or is it a bug? Thanks Durga 1% of the executables have 99% of CPU privilege! Userspace code! Unite!! Occupy the kernel!!! On Sun, Apr 24, 2016 at 8:12 PM, Gilles Gouaillardet wrote: > two comments : > > - the program is incorrect : slave() should MPI_Recv(..., MPI_ANY_TAG, ...) > > - current master uses pmix114, and your traces mention pmix120 > so your master is out of sync, or pmix120 is an old module that was not > manually removed. > fwiw, once in a while, i > rm -rf /.../ompi_install_dir/lib/openmpi > to get rid of the removed modules > > Cheers, > > Gilles > > > On 4/25/2016 7:34 AM, dpchoudh . wrote: > > Hello all > > Attached is a simple MPI program (a modified version of a similar program > that was posted by another user). This program, when run on a single node > machine, hangs most of the time, as follows: (in all cases, OS was CentOS 7) > > Scenario 1: OMPI v 1.10, single socket quad core machine, with Chelsio T3 > card, link down, and GigE, link up > > mpirun -np 2 > Backtrace of the two spawned processes as follows: > > (gdb) bt > #0 0x7f6471647aba in mca_btl_vader_component_progress () at > btl_vader_component.c:708 > #1 0x7f6475c6722a in opal_progress () at runtime/opal_progress.c:187 > #2 0x7f64767b7685 in opal_condition_wait (c=, > m=) > at ../opal/threads/condition.h:78 > #3 ompi_request_default_wait_all (count=2, requests=0x7ffd1d921530, > statuses=0x7ffd1d921540) > at request/req_wait.c:281 > #4 0x7f64709dd591 in ompi_coll_tuned_sendrecv_zero (stag=-16, > rtag=-16, > comm=, source=1, dest=1) at coll_tuned_barrier.c:78 > #5 ompi_coll_tuned_barrier_intra_two_procs (comm=0x6022c0 > , > module=) at coll_tuned_barrier.c:324 > #6 0x7f64767c92e6 in PMPI_Barrier (comm=0x6022c0 > ) at pbarrier.c:70 > #7 0x004010bd in main (argc=1, argv=0x7ffd1d9217d8) at > mpi_hello_master_slave.c:115 > (gdb) > > > (gdb) bt > #0 mca_pml_ob1_progress () at pml_ob1_progress.c:45 > #1 0x7feeae7dc22a in opal_progress () at runtime/opal_progress.c:187 > #2 0x7feea9e125c5 in opal_condition_wait (c=, > m=) > at ../../../../opal/threads/condition.h:78 > #3 ompi_request_wait_completion (req=0xe55200) at > ../../../../ompi/request/request.h:381 > #4 mca_pml_ob1_recv (addr=, count=255, datatype= out>, > src=, tag=, comm=, > status=0x7fff4a618000) > at pml_ob1_irecv.c:118 > #5 0x7feeaf35068f in PMPI_Recv (buf=0x7fff4a618020, count=255, > type=0x6020c0 , source=, tag= out>, > comm=0x6022c0 , status=0x7fff4a618000) at > precv.c:78 > #6 0x00400d49 in slave () at mpi_hello_master_slave.c:67 > #7 0x004010b3 in main (argc=1, argv=0x7fff4a6184d8) at > mpi_hello_master_slave.c:113 > (gdb) > > > Scenario 2: > Dual socket Hexcore machine with Qlogic IB, Chelsio iWARP and Fibre > Channel, all link down, GigE, link up, OpenMPI compiled from master branch, > crashes as follows: > > [durga@smallMPI Desktop]$ mpirun -np 2 ./mpi_hello_master_slave > > mpi_hello_master_slave:39570 terminated with signal 11 at PC=20 > SP=7ffd438c00b8. Backtrace: > > mpi_he
Re: [OMPI users] Cannot run a simple MPI program
Hello George Adding --mca pml ob1 does make the program run. I just wanted to make sure that was the expected behaviour (as opposed to a bug in mpirun). Thanks Durga 1% of the executables have 99% of CPU privilege! Userspace code! Unite!! Occupy the kernel!!! On Sun, Apr 24, 2016 at 9:43 PM, George Bosilca wrote: > Add --mca pml ob1 to your mpirun command. > > George > > > On Sunday, April 24, 2016, dpchoudh . wrote: > >> Hello Gilles >> >> Thank you for finding the bug; it was not there in the original code; I >> added it while trying to 'simplify' the code. >> >> With the bug fixed, the code now runs in the last scenario. But it still >> hangs with the following command line (even after updating to latest git >> tree, rebuilding and reinstalling): >> >> mpirun -np 2 -mca btl self,sm ./mpi_hello_master_slave >> >> and the stack is still as before: >> >> (gdb) bt >> #0 0x7f4e4bd60117 in sched_yield () from /lib64/libc.so.6 >> #1 0x7f4e4ba3d875 in amsh_ep_connreq_wrap () from >> /lib64/libpsm_infinipath.so.1 >> #2 0x7f4e4ba3e254 in amsh_ep_connect () from >> /lib64/libpsm_infinipath.so.1 >> #3 0x7f4e4ba470df in psm_ep_connect () from >> /lib64/libpsm_infinipath.so.1 >> #4 0x7f4e4c4c8975 in ompi_mtl_psm_add_procs (mtl=0x7f4e4c846500 >> , nprocs=2, procs=0x23bb420) >> at mtl_psm.c:312 >> #5 0x7f4e4c52ef6b in mca_pml_cm_add_procs (procs=0x23bb420, >> nprocs=2) at pml_cm.c:134 >> #6 0x7f4e4c2e7d0f in ompi_mpi_init (argc=1, argv=0x7fffe930f9b8, >> requested=0, provided=0x7fffe930f78c) >> at runtime/ompi_mpi_init.c:770 >> #7 0x7f4e4c324aff in PMPI_Init (argc=0x7fffe930f7bc, >> argv=0x7fffe930f7b0) at pinit.c:66 >> #8 0x0040101f in main (argc=1, argv=0x7fffe930f9b8) at >> mpi_hello_master_slave.c:94 >> >> As you can see, OMPI is trying the PSM link to communicate, even though >> the link is down and it is not mentioned in the arguments to mpirun. (There >> are not even multiple nodes mentioned in the arguments.) >> >> Is this the expected behaviour or is it a bug? >> >> Thanks >> Durga >> >> 1% of the executables have 99% of CPU privilege! >> Userspace code! Unite!! Occupy the kernel!!! >> >> On Sun, Apr 24, 2016 at 8:12 PM, Gilles Gouaillardet >> wrote: >> >>> two comments : >>> >>> - the program is incorrect : slave() should MPI_Recv(..., MPI_ANY_TAG, >>> ...) >>> >>> - current master uses pmix114, and your traces mention pmix120 >>> so your master is out of sync, or pmix120 is an old module that was >>> not manually removed. >>> fwiw, once in a while, i >>> rm -rf /.../ompi_install_dir/lib/openmpi >>> to get rid of the removed modules >>> >>> Cheers, >>> >>> Gilles >>> >>> >>> On 4/25/2016 7:34 AM, dpchoudh . wrote: >>> >>> Hello all >>> >>> Attached is a simple MPI program (a modified version of a similar >>> program that was posted by another user). This program, when run on a >>> single node machine, hangs most of the time, as follows: (in all cases, OS >>> was CentOS 7) >>> >>> Scenario 1: OMPI v 1.10, single socket quad core machine, with Chelsio >>> T3 card, link down, and GigE, link up >>> >>> mpirun -np 2 >>> Backtrace of the two spawned processes as follows: >>> >>> (gdb) bt >>> #0 0x7f6471647aba in mca_btl_vader_component_progress () at >>> btl_vader_component.c:708 >>> #1 0x7f6475c6722a in opal_progress () at runtime/opal_progress.c:187 >>> #2 0x7f64767b7685 in opal_condition_wait (c=, >>> m=) >>> at ../opal/threads/condition.h:78 >>> #3 ompi_request_default_wait_all (count=2, requests=0x7ffd1d921530, >>> statuses=0x7ffd1d921540) >>> at request/req_wait.c:281 >>> #4 0x7f64709dd591 in ompi_coll_tuned_sendrecv_zero (stag=-16, >>> rtag=-16, >>> comm=, source=1, dest=1) at coll_tuned_barrier.c:78 >>> #5 ompi_coll_tuned_barrier_intra_two_procs (comm=0x6022c0 >>> , >>> module=) at coll_tuned_barrier.c:324 >>> #6 0x7f64767c92e6 in PMPI_Barrier (comm=0x6022c0 >>> ) at pbarrier.c:70 >>> #7 0x004010bd in main (argc=1, argv=0x7ffd1d9217d8) at >>> mpi_hello_master_slave.c:115 >>> (gdb) >>> >>> >>> (gdb) bt >>> #0 mca_pml_ob1_progress () at pml_ob1_progress.c
[OMPI users] Add release dates to release notes
Hello developers May I request that you add the release dates to the release notes (the NEWS file)? The reason I ask is that some one-of-a-kind hardware or obsolete hardware run very old versions of OpenMPI, and I am asked to maintain that version using a PC platform. I want to know what is the time range of PC hardware/OS release that I should choose that will still run that older release. Thank you Durga 1% of the executables have 99% of CPU privilege! Userspace code! Unite!! Occupy the kernel!!!
Re: [OMPI users] libmpi_cxx
Hello Gilles and all Sorry if this is a bit off topic, but I am curious as to why C++bindings were dropped? Any pointers would be appreciated. Best regards Durga $man why dump woman? man: too many arguments On Wed, Mar 28, 2018 at 11:43 PM, Gilles Gouaillardet wrote: > Arthur, > > Try to > configure --enable-mpi-xxx > > Note c++ bindings have been removed from the MPI standard long time ago, so > you might want to consider modernizing your app. > > Cheers, > > Gilles > > "Arthur H. Edwards" wrote: >>I have built openmpi 3.0 on an ubuntu 16.04 system I have used --with-cuda. >>There is no libmpi_cxx.so generated, yet the code I'm using requires it. >>There is a libmpi_cxx.so in the ubuntu installed version. Any insight, or >>instruction on how to configure so that the build generates this library >>would be greatly appreciated. >> >>Art Edwards >> >>-- >> Arthur H. Edwards >> edwards...@fastmail.fm >>___ >>users mailing list >>users@lists.open-mpi.org >>https://lists.open-mpi.org/mailman/listinfo/users > ___ > users mailing list > users@lists.open-mpi.org > https://lists.open-mpi.org/mailman/listinfo/users ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
Re: [OMPI users] running mpi program between my PC and an ARM-architektur raspberry
Sorry for a pedantic follow up: Is this (heterogeneous cluster support) something that is specified by the MPI standard (perhaps as an optional component)? Do people know if MPICH. MVAPICH, Intel MPI etc support it? (I do realize this is an OpenMPI forum) The reason I ask is that I have a mini Linux lab of sort that consists of Linux running on many architectures, both 32 and 64 bit and both LE and BE. Some have advanced fabrics, but all have garden variety Ethernet. I mostly use this for software porting work, but I'd love to set it up as a test bench for testing OpenMPI in a heterogeneous environment and report issues, if that is something that the developers want to achieve. Thanks Durga $man why dump woman? man: too many arguments On Mon, Apr 2, 2018 at 1:19 PM, Jeff Squyres (jsquyres) wrote: > On Mar 31, 2018, at 10:57 PM, peng jia wrote: >> >> I would like to run some MPI code with a cluster out of a normal laptop and >> a ARM-architektur raspberrypi, but unsuccessfully the system would not >> response, even though i install openmpi manuelly on both pc and raspberrypi. >> >> mypc@pc:~ $ mpiexec -H pc,raspberrypi001 hello_world >> >> i would like to ask, is it possible to run such a MPI porgram in a so >> heterogeneous cluster? if not possible, why? i am really really confuse >> about it, could you please teach me? > > Heterogeneous support is, at best, not well supported. I would consider it a > topic for an advanced user, and even so, our heterogeneous support inside > Open MPI is probably not well tested these days. I would not recommend > mixing machines with different data sizes and/or representations in a single > job. There are many issues that come up; the easiest to describe is: what > should MPI do when a message sent of type X is sent between two machines > where X is a different size? Should MPI truncate the data? Should it round? > Should it error? ...? There's no good answer. > > My advice: run your job exclusively on your Raspberry Pi's *or* your closer > of laptops, and you should probably be ok. > > -- > Jeff Squyres > jsquy...@cisco.com > > ___ > users mailing list > users@lists.open-mpi.org > https://lists.open-mpi.org/mailman/listinfo/users ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
Re: [OMPI users] problem
What Jeff is suggesting is probably valgrind. However, in my experience, which is much less than most OpenMPI developers, a simple code inspection often is adequate. Here are the steps: 1. If you don't already have it, build a debug version of your code. If you are using gcc, you'd use a -g to CFLAGS on your makefile for C programs (adding -g3, taking out any -O flags is better) 2. Have your shell generate a core dump when the crash happens. 3. Launch gdb with the debug image and core file I have had near 100% luck in detecting sources of SEGV-type crash using the steps above, but your mileage may vary. If you are not familiar with gdb, you may be able to enlist someone local who does. We learn from history that we never learn from history. On Thu, May 10, 2018 at 5:47 AM, Ankita m wrote: > ok...Thank you so much sir > > On Wed, May 9, 2018 at 11:13 PM, Jeff Squyres (jsquyres) > wrote: >> >> It looks like you're getting a segv when calling MPI_Comm_rank(). >> >> This is quite unusual -- MPI_Comm_rank() is just a local lookup / return >> of an integer. If MPI_Comm_rank() is seg faulting, it usually indicates >> that there's some other kind of memory error in the application, and this >> seg fault you're seeing is just a symptom -- it's not the real problem. It >> may have worked with Intel MPI by chance, or for some reason, Intel MPI has >> a different memory pattern than Open MPI and it didn't happen to trigger >> this exact problem. >> >> You might want to run your application through a memory-checking debugger. >> >> >> >> > On May 9, 2018, at 11:39 AM, Ankita m wrote: >> > >> > yes. Because previously i was using intel-mpi. That time the program was >> > running perfectly. Now when i use openmpi this shows this error >> > files...Though i am not quite sure. I just thought if the issue will be for >> > Openmpi then i could get some help here. >> > >> > On Wed, May 9, 2018 at 6:47 PM, Gilles Gouaillardet >> > wrote: >> > Ankita, >> > >> > Do you have any reason to suspect the root cause of the crash is Open >> > MPI ? >> > >> > Cheers, >> > >> > Gilles >> > >> > >> > On Wednesday, May 9, 2018, Ankita m wrote: >> > MPI "Hello World" program is also working >> > >> > please see this error file attached below. its of a different program >> > >> > On Wed, May 9, 2018 at 4:10 PM, John Hearns via users >> > wrote: >> > Ankita, looks like your program is not launching correctly. >> > I would try the following: >> > define two hosts in a machinefile. Use mpirun -np 2 machinefile date >> > Ie can you use mpirun just to run the command 'date' >> > >> > Secondly compile up and try to run an MPI 'Hello World' program >> > >> > >> > On 9 May 2018 at 12:28, Ankita m wrote: >> > I am using ompi -3.1.0 version in my program and compiler is mpicc >> > >> > its a parallel program which uses multiple nodes with 16 cores in each >> > node. >> > >> > but its not working and generates a error file . i Have attached the >> > error file below. >> > >> > can anyone please tell what is the issue actually >> > >> > ___ >> > users mailing list >> > users@lists.open-mpi.org >> > https://lists.open-mpi.org/mailman/listinfo/users >> > >> > >> > ___ >> > users mailing list >> > users@lists.open-mpi.org >> > https://lists.open-mpi.org/mailman/listinfo/users >> > >> > >> > ___ >> > users mailing list >> > users@lists.open-mpi.org >> > https://lists.open-mpi.org/mailman/listinfo/users >> > >> > ___ >> > users mailing list >> > users@lists.open-mpi.org >> > https://lists.open-mpi.org/mailman/listinfo/users >> >> >> -- >> Jeff Squyres >> jsquy...@cisco.com >> >> ___ >> users mailing list >> users@lists.open-mpi.org >> https://lists.open-mpi.org/mailman/listinfo/users > > > > ___ > users mailing list > users@lists.open-mpi.org > https://lists.open-mpi.org/mailman/listinfo/users ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
