Hi there I apologize for the lengthy delay in responding to this question with a resolution.
There may be multiple issues here. The segfault you show in one test would most likely be a symptom of a thread safety issue - I haven't been able to replicate that specific issue. I can only recommend running without threads. The issue with the number of comm_spawns you can perform stems from a fundamental operating system limit on the number of child processes you can have at any one time. Since you have no nodes defined, you are running all of your processes on your local node. In this situation, Open MPI "forks" a copy of its daemon (called "orted") on your node for each call to comm_spawn. This orted subsequently "forks" your application process. Thus, each call to comm_spawn results in *two* child processes on your node. Hence, you can quickly run into the limit on the number of child processes each user is allowed to have concurrently operating. Unfortunately, there is no way around this problem at this time. We are looking into at least providing a warning and aborting cleanly when this is encountered, but that "patch" would not be back-ported to the 1.1 code family. The only real solution to the problem you have encountered, therefore, is to spread your processes across other nodes. Is that possible? Ralph On 3/6/07 10:00 AM, "rozzen.vinc...@fr.thalesgroup.com" <rozzen.vinc...@fr.thalesgroup.com> wrote: > Hi Tim, I get back to you > > "What kind of system is it?" > =>The system is a "Debian Sarge". > "How many nodes are you running on?" > => There is no cluster configured, so I guess I work with no node > environnement. > "Have you been able to try a more recent version of Open MPI?" > =>Today, I tried with version 1.1.4, but the results are not better. > I tested 2 cases : > Test 1 : with the sames configuration options (./configure > --enable-mpi-threads --enable-progress-threads --with-threads=posix > --enable-smp-locks) > The program stopped on MPI_Init_thread in __lll_mutex_lock_wait () from > /lib/tls/libpthread.so.0 > > Test 2 : with the default configuration options (./configure > --prefix=/usr/local/Mpi/openmpi-1.1.4-noBproc-noThread) > The program stoped on the "node allocation" after the spawn n°31. > Maybe the problem comes from the lack of node definition? > Thanks for your help. > > Here below, the different log files of the 2 tests > > /******************************TEST 1*******************************/ > GNU gdb 6.3-debian > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and you are > welcome to change it and/or distribute copies of it under certain conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for details. > This GDB was configured as "i386-linux"...Using host libthread_db library > "/lib/tls/libthread_db.so.1". > > (gdb) run > Starting program: /home/workspace/test_spaw1/src/spawn > [Thread debugging using libthread_db enabled] > [New Thread 1076646560 (LWP 5178)] > main******************************* > main : Lancement MPI* > [New Thread 1085225904 (LWP 5181)] > [New Thread 1094495152 (LWP 5182)] > > Program received signal SIGINT, Interrupt. > [Switching to Thread 1076646560 (LWP 5178)] > 0x4018a436 in __lll_mutex_lock_wait () from /lib/tls/libpthread.so.0 > (gdb) where > #0 0x4018a436 in __lll_mutex_lock_wait () from /lib/tls/libpthread.so.0 > #1 0x40187893 in _L_mutex_lock_26 () from /lib/tls/libpthread.so.0 > #2 0xbffff508 in ?? () > #3 0x4000bcd0 in _dl_map_object_deps () from /lib/ld-linux.so.2 > #4 0x40b9f8cb in mca_btl_tcp_component_create_listen () from > /usr/local/Mpi/openmpi-1.1.4-noBproc/lib/openmpi/mca_btl_tcp.so > #5 0x40b9f8cb in mca_btl_tcp_component_create_listen () from > /usr/local/Mpi/openmpi-1.1.4-noBproc/lib/openmpi/mca_btl_tcp.so > #6 0x40b9eef4 in mca_btl_tcp_component_init () from > /usr/local/Mpi/openmpi-1.1.4-noBproc/lib/openmpi/mca_btl_tcp.so > #7 0x4008c652 in mca_btl_base_select () from > /usr/local/Mpi/CURRENT_MPI/lib/libmpi.so.0 > #8 0x40b8dd28 in mca_bml_r2_component_init () from > /usr/local/Mpi/openmpi-1.1.4-noBproc/lib/openmpi/mca_bml_r2.so > #9 0x4008bf54 in mca_bml_base_init () from > /usr/local/Mpi/CURRENT_MPI/lib/libmpi.so.0 > #10 0x40b7e5c9 in mca_pml_ob1_component_init () from > /usr/local/Mpi/openmpi-1.1.4-noBproc/lib/openmpi/mca_pml_ob1.so > #11 0x40094192 in mca_pml_base_select () from > /usr/local/Mpi/CURRENT_MPI/lib/libmpi.so.0 > #12 0x4005742c in ompi_mpi_init () from > /usr/local/Mpi/CURRENT_MPI/lib/libmpi.so.0 > #13 0x4007c182 in PMPI_Init_thread () from > /usr/local/Mpi/CURRENT_MPI/lib/libmpi.so.0 > #14 0x080489f3 in main (argc=1, argv=0xbffff8a4) at spawn6.c:33 > > > > /******************************TEST 2*******************************/ > > GNU gdb 6.3-debian > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and you are > welcome to change it and/or distribute copies of it under certain conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for details. > This GDB was configured as "i386-linux"...Using host libthread_db library > "/lib/tls/libthread_db.so.1". > > (gdb) run -np 1 --host myhost spawn6 > Starting program: /usr/local/Mpi/openmpi-1.1.4-noBproc-noThread/bin/mpirun -np > 1 --host myhost spawn6 > [Thread debugging using libthread_db enabled] > [New Thread 1076121728 (LWP 4022)] > main******************************* > main : Lancement MPI* > Exe : Lance > Exe: lRankExe = 1 lRankMain = 0 > 1 main***MPI_Comm_spawn return : 0 > 1 main***Rang main : 0 Rang exe : 1 > Exe : Lance > Exe: Fin. > > > Exe: lRankExe = 1 lRankMain = 0 > 2 main***MPI_Comm_spawn return : 0 > 2 main***Rang main : 0 Rang exe : 1 > Exe : Lance > Exe: Fin. > > ... > > Exe: lRankExe = 1 lRankMain = 0 > 30 main***MPI_Comm_spawn return : 0 > 30 main***Rang main : 0 Rang exe : 1 > Exe : Lance > Exe: Fin. > > Exe: lRankExe = 1 lRankMain = 0 > 31 main***MPI_Comm_spawn return : 0 > 31 main***Rang main : 0 Rang exe : 1 > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 1076121728 (LWP 4022)] > 0x4018833b in strlen () from /lib/tls/libc.so.6 > (gdb) where > #0 0x4018833b in strlen () from /lib/tls/libc.so.6 > #1 0x40297c5e in orte_gpr_replica_create_itag () from > /usr/local/Mpi/openmpi-1.1.4-noBproc-noThread/lib/openmpi/mca_gpr_replica.so > #2 0x4029d2df in orte_gpr_replica_put_fn () from > /usr/local/Mpi/openmpi-1.1.4-noBproc-noThread/lib/openmpi/mca_gpr_replica.so > #3 0x40297281 in orte_gpr_replica_put () from > /usr/local/Mpi/openmpi-1.1.4-noBproc-noThread/lib/openmpi/mca_gpr_replica.so > #4 0x40048287 in orte_ras_base_node_assign () from > /usr/local/Mpi/openmpi-1.1.4-noBproc-noThread/lib/liborte.so.0 > #5 0x400463e1 in orte_ras_base_allocate_nodes () from > /usr/local/Mpi/openmpi-1.1.4-noBproc-noThread/lib/liborte.so.0 > #6 0x402c2bb8 in orte_ras_hostfile_allocate () from > /usr/local/Mpi/openmpi-1.1.4-noBproc-noThread/lib/openmpi/mca_ras_hostfile.so > #7 0x400464e0 in orte_ras_base_allocate () from > /usr/local/Mpi/openmpi-1.1.4-noBproc-noThread/lib/liborte.so.0 > #8 0x402b063f in orte_rmgr_urm_allocate () from > /usr/local/Mpi/openmpi-1.1.4-noBproc-noThread/lib/openmpi/mca_rmgr_urm.so > #9 0x4004f277 in orte_rmgr_base_cmd_dispatch () from > /usr/local/Mpi/openmpi-1.1.4-noBproc-noThread/lib/liborte.so.0 > #10 0x402b10ae in orte_rmgr_urm_recv () from > /usr/local/Mpi/openmpi-1.1.4-noBproc-noThread/lib/openmpi/mca_rmgr_urm.so > #11 0x4004301e in mca_oob_recv_callback () from > /usr/local/Mpi/openmpi-1.1.4-noBproc-noThread/lib/liborte.so.0 > #12 0x4027a748 in mca_oob_tcp_msg_data () from > /usr/local/Mpi/openmpi-1.1.4-noBproc-noThread/lib/openmpi/mca_oob_tcp.so > #13 0x4027bb12 in mca_oob_tcp_peer_recv_handler () from > /usr/local/Mpi/openmpi-1.1.4-noBproc-noThread/lib/openmpi/mca_oob_tcp.so > #14 0x400703f9 in opal_event_loop () from > /usr/local/Mpi/openmpi-1.1.4-noBproc-noThread/lib/libopal.so.0 > #15 0x4006adfa in opal_progress () from > /usr/local/Mpi/openmpi-1.1.4-noBproc-noThread/lib/libopal.so.0 > #16 0x0804c7a1 in opal_condition_wait (c=0x804fbcc, m=0x804fba8) at > condition.h:81 > #17 0x0804a4c8 in orterun (argc=6, argv=0xbffff854) at orterun.c:427 > #18 0x08049dd6 in main (argc=6, argv=0xbffff854) at main.c:13 > (gdb) > -----Message d'origine----- > De : users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org]De la > part de Tim Prins > Envoyé : lundi 5 mars 2007 22:34 > À : Open MPI Users > Objet : Re: [OMPI users] MPI_Comm_Spawn > > > Never mind, I was just able to replicate it. I'll look into it. > > Tim > > On Mar 5, 2007, at 4:26 PM, Tim Prins wrote: > >> That is possible. Threading support is VERY lightly tested, but I >> doubt it is the problem since it always fails after 31 spawns. >> >> Again, I have tried with these configure options and the same version >> of Open MPI and have still have been able to replicate this (after >> letting it spawn over 500 times). Have you been able to try a more >> recent version of Open MPI? What kind of system is it? How many nodes >> are you running on? >> >> Tim >> >> On Mar 5, 2007, at 1:21 PM, rozzen.vinc...@fr.thalesgroup.com wrote: >> >>> >>> Maybe the problem comes from the configuration options. >>> The configuration options used are : >>> ./configure --enable-mpi-threads --enable-progress-threads --with- >>> threads=posix --enable-smp-locks >>> Could you give me your point of view about that please ? >>> Thanks >>> >>> -----Message d'origine----- >>> De : users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] >>> De la >>> part de Ralph H Castain >>> Envoyé : mardi 27 février 2007 16:26 >>> À : Open MPI Users <us...@open-mpi.org> >>> Objet : Re: [OMPI users] MPI_Comm_Spawn >>> >>> >>> Now that's interesting! There shouldn't be a limit, but to be >>> honest, I've >>> never tested that mode of operation - let me look into it and see. >>> It sounds >>> like there is some counter that is overflowing, but I'll look. >>> >>> Thanks >>> Ralph >>> >>> >>> On 2/27/07 8:15 AM, "rozzen.vinc...@fr.thalesgroup.com" >>> <rozzen.vinc...@fr.thalesgroup.com> wrote: >>> >>>> Do you know if there is a limit to the number of MPI_Comm_spawn we >>>> can use in >>>> order to launch a program? >>>> I want to start and stop a program several times (with the function >>>> MPI_Comm_spawn) but every time after 31 MPI_Comm_spawn, I get a >>>> "segmentation >>>> fault". >>>> Could you give me your point of you to solve this problem? >>>> Thanks >>>> >>>> /*file .c : spawned the file Exe*/ >>>> #include <stdio.h> >>>> #include <malloc.h> >>>> #include <unistd.h> >>>> #include "mpi.h" >>>> #include <pthread.h> >>>> #include <signal.h> >>>> #include <sys/time.h> >>>> #include <errno.h> >>>> #define EXE_TEST "/home/workspace/test_spaw1/src/ >>>> Exe" >>>> >>>> >>>> >>>> int main( int argc, char **argv ) { >>>> >>>> long *lpBufferMpi; >>>> MPI_Comm lIntercom; >>>> int lErrcode; >>>> MPI_Comm lCommunicateur; >>>> int lRangMain,lRangExe,lMessageEnvoi,lIter,NiveauThreadVoulu, >>>> NiveauThreadObtenu,lTailleBuffer; >>>> int *lpMessageEnvoi=&lMessageEnvoi; >>>> MPI_Status lStatus; /*status de reception*/ >>>> >>>> lIter=0; >>>> >>>> >>>> /* MPI environnement */ >>>> >>>> printf("main*******************************\n"); >>>> printf("main : Lancement MPI*\n"); >>>> >>>> NiveauThreadVoulu = MPI_THREAD_MULTIPLE; >>>> MPI_Init_thread( &argc, &argv, NiveauThreadVoulu, >>>> &NiveauThreadObtenu ); >>>> lpBufferMpi = calloc( 10000, sizeof(long)); >>>> MPI_Buffer_attach( (void*)lpBufferMpi, 10000 * sizeof(long) ); >>>> >>>> while (lIter<1000){ >>>> lIter ++; >>>> lIntercom=(MPI_Comm)-1 ; >>>> >>>> MPI_Comm_spawn( EXE_TEST, NULL, 1, MPI_INFO_NULL, >>>> 0, MPI_COMM_WORLD, &lIntercom, &lErrcode ); >>>> printf( "%i main***MPI_Comm_spawn return : %d\n",lIter, >>>> lErrcode ); >>>> >>>> if(lIntercom == (MPI_Comm)-1 ){ >>>> printf("%i Intercom null\n",lIter); >>>> return 0; >>>> } >>>> MPI_Intercomm_merge(lIntercom, 0,&lCommunicateur ); >>>> MPI_Comm_rank( lCommunicateur, &lRangMain); >>>> lRangExe=1-lRangMain; >>>> >>>> printf("%i main***Rang main : %i Rang exe : %i >>>> \n",lIter,(int)lRangMain,(int)lRangExe); >>>> sleep(2); >>>> >>>> } >>>> >>>> >>>> /* Arret de l'environnement MPI */ >>>> lTailleBuffer=10000* sizeof(long); >>>> MPI_Buffer_detach( (void*)lpBufferMpi, &lTailleBuffer ); >>>> MPI_Comm_free( &lCommunicateur ); >>>> MPI_Finalize( ); >>>> free( lpBufferMpi ); >>>> >>>> printf( "Main = End .\n" ); >>>> return 0; >>>> >>>> } >>>> / >>>> ******************************************************************** >>>> * >>>> ******** >>>> *******************/ >>>> Exe: >>>> #include <string.h> >>>> #include <stdlib.h> >>>> #include <stdio.h> >>>> #include <malloc.h> >>>> #include <unistd.h> /* pour sleep() */ >>>> #include <pthread.h> >>>> #include <semaphore.h> >>>> #include "mpi.h" >>>> >>>> int main( int argc, char **argv ) { >>>> /*1)pour communiaction MPI*/ >>>> MPI_Comm lCommunicateur; /*communicateur du process*/ >>>> MPI_Comm CommParent; /*Communiacteur parent à >>>> récupérer*/ >>>> int lRank; /*rang du communicateur du >>>> process*/ >>>> int lRangMain; /*rang du séquenceur si lancé en >>>> mode normal*/ >>>> int lTailleCommunicateur; /*taille du communicateur;*/ >>>> long *lpBufferMpi; /*buffer pour message*/ >>>> int lBufferSize; /*taille du buffer*/ >>>> >>>> /*2) pour les thread*/ >>>> int NiveauThreadVoulu, NiveauThreadObtenu; >>>> >>>> >>>> lCommunicateur = (MPI_Comm)-1; >>>> NiveauThreadVoulu = MPI_THREAD_MULTIPLE; >>>> int erreur = MPI_Init_thread( &argc, &argv, NiveauThreadVoulu, >>>> &NiveauThreadObtenu ); >>>> >>>> if (erreur!=0){ >>>> printf("erreur\n"); >>>> free( lpBufferMpi ); >>>> return -1; >>>> } >>>> >>>> /*2) Attachement à un buffer pour le message*/ >>>> lBufferSize=10000 * sizeof(long); >>>> lpBufferMpi = calloc( 10000, sizeof(long)); >>>> erreur = MPI_Buffer_attach( (void*)lpBufferMpi, lBufferSize ); >>>> >>>> if (erreur!=0){ >>>> printf("erreur\n"); >>>> free( lpBufferMpi ); >>>> return -1; >>>> } >>>> >>>> printf( "Exe : Lance \n" ); >>>> MPI_Comm_get_parent(&CommParent); >>>> MPI_Intercomm_merge( CommParent, 1, &lCommunicateur ); >>>> MPI_Comm_rank( lCommunicateur, &lRank ); >>>> MPI_Comm_size( lCommunicateur, &lTailleCommunicateur ); >>>> lRangMain =1-lRank; >>>> printf( "Exe: lRankExe = %d lRankMain = %d\n", lRank , >>>> lRangMain, >>>> lTailleCommunicateur); >>>> >>>> sleep(1); >>>> MPI_Buffer_detach( (void*)lpBufferMpi, &lBufferSize ); >>>> MPI_Comm_free( &lCommunicateur ); >>>> MPI_Finalize( ); >>>> free( lpBufferMpi ); >>>> printf( "Exe: Fin.\n\n\n" ); >>>> } >>>> >>>> >>>> / >>>> ******************************************************************** >>>> * >>>> ******** >>>> *******************/ >>>> result : >>>> main******************************* >>>> main : Lancement MPI* >>>> 1 main***MPI_Comm_spawn return : 0 >>>> Exe : Lance >>>> 1 main***Rang main : 0 Rang exe : 1 >>>> Exe: lRankExe = 1 lRankMain = 0 >>>> Exe: Fin. >>>> >>>> >>>> 2 main***MPI_Comm_spawn return : 0 >>>> Exe : Lance >>>> 2 main***Rang main : 0 Rang exe : 1 >>>> Exe: lRankExe = 1 lRankMain = 0 >>>> Exe: Fin. >>>> >>>> >>>> 3 main***MPI_Comm_spawn return : 0 >>>> Exe : Lance >>>> 3 main***Rang main : 0 Rang exe : 1 >>>> Exe: lRankExe = 1 lRankMain = 0 >>>> Exe: Fin. >>>> >>>> .... >>>> >>>> 30 main***MPI_Comm_spawn return : 0 >>>> Exe : Lance >>>> 30 main***Rang main : 0 Rang exe : 1 >>>> Exe: lRankExe = 1 lRankMain = 0 >>>> Exe: Fin. >>>> >>>> >>>> 31 main***MPI_Comm_spawn return : 0 >>>> Exe : Lance >>>> 31 main***Rang main : 0 Rang exe : 1 >>>> Exe: lRankExe = 1 lRankMain = 0 >>>> Erreur de segmentation >>>> >>>> >>>> >>>> _______________________________________________ >>>> users mailing list >>>> us...@open-mpi.org >>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>> >>> >>> >>> _______________________________________________ >>> users mailing list >>> us...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>> >>> _______________________________________________ >>> users mailing list >>> us...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/users >> >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users > > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
