Grzegorz: something occurred to me. When you start all these processes, how are you staggering their wireup? Are they flooding us, or are you time-shifting them a little?
On Jul 19, 2010, at 10:32 AM, Edgar Gabriel wrote: > Hm, so I am not sure how to approach this. First of all, the test case > works for me. I used up to 80 clients, and for both optimized and > non-optimized compilation. I ran the tests with trunk (not with 1.4 > series, but the communicator code is identical in both cases). Clearly, > the patch from Ralph is necessary to make it work. > > Additionally, I went through the communicator creation code for dynamic > communicators trying to find spots that could create problems. The only > place that I found the number 64 appear is the fortran-to-c mapping > arrays (e.g. for communicators), where the initial size of the table is > 64. I looked twice over the pointer-array code to see whether we could > have a problem their (since it is a key-piece of the cid allocation code > for communicators), but I am fairly confident that it is correct. > > Note, that we have other (non-dynamic tests), were comm_set is called > 100,000 times, and the code per se does not seem to have a problem due > to being called too often. So I am not sure what else to look at. > > Edgar > > > > On 7/13/2010 8:42 PM, Ralph Castain wrote: >> As far as I can tell, it appears the problem is somewhere in our >> communicator setup. The people knowledgeable on that area are going to look >> into it later this week. >> >> I'm creating a ticket to track the problem and will copy you on it. >> >> >> On Jul 13, 2010, at 6:57 AM, Ralph Castain wrote: >> >>> >>> On Jul 13, 2010, at 3:36 AM, Grzegorz Maj wrote: >>> >>>> Bad news.. >>>> I've tried the latest patch with and without the prior one, but it >>>> hasn't changed anything. I've also tried using the old code but with >>>> the OMPI_DPM_BASE_MAXJOBIDS constant changed to 80, but it also didn't >>>> help. >>>> While looking through the sources of openmpi-1.4.2 I couldn't find any >>>> call of the function ompi_dpm_base_mark_dyncomm. >>> >>> It isn't directly called - it shows in ompi_comm_set as >>> ompi_dpm.mark_dyncomm. You were definitely overrunning that array, but I >>> guess something else is also being hit. Have to look further... >>> >>> >>>> >>>> >>>> 2010/7/12 Ralph Castain <r...@open-mpi.org>: >>>>> Just so you don't have to wait for 1.4.3 release, here is the patch >>>>> (doesn't include the prior patch). >>>>> >>>>> >>>>> >>>>> >>>>> On Jul 12, 2010, at 12:13 PM, Grzegorz Maj wrote: >>>>> >>>>>> 2010/7/12 Ralph Castain <r...@open-mpi.org>: >>>>>>> Dug around a bit and found the problem!! >>>>>>> >>>>>>> I have no idea who or why this was done, but somebody set a limit of 64 >>>>>>> separate jobids in the dynamic init called by ompi_comm_set, which >>>>>>> builds the intercommunicator. Unfortunately, they hard-wired the array >>>>>>> size, but never check that size before adding to it. >>>>>>> >>>>>>> So after 64 calls to connect_accept, you are overwriting other areas of >>>>>>> the code. As you found, hitting 66 causes it to segfault. >>>>>>> >>>>>>> I'll fix this on the developer's trunk (I'll also add that original >>>>>>> patch to it). Rather than my searching this thread in detail, can you >>>>>>> remind me what version you are using so I can patch it too? >>>>>> >>>>>> I'm using 1.4.2 >>>>>> Thanks a lot and I'm looking forward for the patch. >>>>>> >>>>>>> >>>>>>> Thanks for your patience with this! >>>>>>> Ralph >>>>>>> >>>>>>> >>>>>>> On Jul 12, 2010, at 7:20 AM, Grzegorz Maj wrote: >>>>>>> >>>>>>>> 1024 is not the problem: changing it to 2048 hasn't change anything. >>>>>>>> Following your advice I've run my process using gdb. Unfortunately I >>>>>>>> didn't get anything more than: >>>>>>>> >>>>>>>> Program received signal SIGSEGV, Segmentation fault. >>>>>>>> [Switching to Thread 0xf7e4c6c0 (LWP 20246)] >>>>>>>> 0xf7f39905 in ompi_comm_set () from /home/gmaj/openmpi/lib/libmpi.so.0 >>>>>>>> >>>>>>>> (gdb) bt >>>>>>>> #0 0xf7f39905 in ompi_comm_set () from >>>>>>>> /home/gmaj/openmpi/lib/libmpi.so.0 >>>>>>>> #1 0xf7e3ba95 in connect_accept () from >>>>>>>> /home/gmaj/openmpi/lib/openmpi/mca_dpm_orte.so >>>>>>>> #2 0xf7f62013 in PMPI_Comm_connect () from >>>>>>>> /home/gmaj/openmpi/lib/libmpi.so.0 >>>>>>>> #3 0x080489ed in main (argc=825832753, argv=0x34393638) at client.c:43 >>>>>>>> >>>>>>>> What's more: when I've added a breakpoint on ompi_comm_set in 66th >>>>>>>> process and stepped a couple of instructions, one of the other >>>>>>>> processes crashed (as usualy on ompi_comm_set) earlier than 66th did. >>>>>>>> >>>>>>>> Finally I decided to recompile openmpi using -g flag for gcc. In this >>>>>>>> case the 66 processes issue has gone! I was running my applications >>>>>>>> exactly the same way as previously (even without recompilation) and >>>>>>>> I've run successfully over 130 processes. >>>>>>>> When switching back to the openmpi compilation without -g it again >>>>>>>> segfaults. >>>>>>>> >>>>>>>> Any ideas? I'm really confused. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> 2010/7/7 Ralph Castain <r...@open-mpi.org>: >>>>>>>>> I would guess the #files limit of 1024. However, if it behaves the >>>>>>>>> same way when spread across multiple machines, I would suspect it is >>>>>>>>> somewhere in your program itself. Given that the segfault is in your >>>>>>>>> process, can you use gdb to look at the core file and see where and >>>>>>>>> why it fails? >>>>>>>>> >>>>>>>>> On Jul 7, 2010, at 10:17 AM, Grzegorz Maj wrote: >>>>>>>>> >>>>>>>>>> 2010/7/7 Ralph Castain <r...@open-mpi.org>: >>>>>>>>>>> >>>>>>>>>>> On Jul 6, 2010, at 8:48 AM, Grzegorz Maj wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Ralph, >>>>>>>>>>>> sorry for the late response, but I couldn't find free time to play >>>>>>>>>>>> with this. Finally I've applied the patch you prepared. I've >>>>>>>>>>>> launched >>>>>>>>>>>> my processes in the way you've described and I think it's working >>>>>>>>>>>> as >>>>>>>>>>>> you expected. None of my processes runs the orted daemon and they >>>>>>>>>>>> can >>>>>>>>>>>> perform MPI operations. Unfortunately I'm still hitting the 65 >>>>>>>>>>>> processes issue :( >>>>>>>>>>>> Maybe I'm doing something wrong. >>>>>>>>>>>> I attach my source code. If anybody could have a look on this, I >>>>>>>>>>>> would >>>>>>>>>>>> be grateful. >>>>>>>>>>>> >>>>>>>>>>>> When I run that code with clients_count <= 65 everything works >>>>>>>>>>>> fine: >>>>>>>>>>>> all the processes create a common grid, exchange some information >>>>>>>>>>>> and >>>>>>>>>>>> disconnect. >>>>>>>>>>>> When I set clients_count > 65 the 66th process crashes on >>>>>>>>>>>> MPI_Comm_connect (segmentation fault). >>>>>>>>>>> >>>>>>>>>>> I didn't have time to check the code, but my guess is that you are >>>>>>>>>>> still hitting some kind of file descriptor or other limit. Check to >>>>>>>>>>> see what your limits are - usually "ulimit" will tell you. >>>>>>>>>> >>>>>>>>>> My limitations are: >>>>>>>>>> time(seconds) unlimited >>>>>>>>>> file(blocks) unlimited >>>>>>>>>> data(kb) unlimited >>>>>>>>>> stack(kb) 10240 >>>>>>>>>> coredump(blocks) 0 >>>>>>>>>> memory(kb) unlimited >>>>>>>>>> locked memory(kb) 64 >>>>>>>>>> process 200704 >>>>>>>>>> nofiles 1024 >>>>>>>>>> vmemory(kb) unlimited >>>>>>>>>> locks unlimited >>>>>>>>>> >>>>>>>>>> Which one do you think could be responsible for that? >>>>>>>>>> >>>>>>>>>> I was trying to run all the 66 processes on one machine or spread >>>>>>>>>> them >>>>>>>>>> across several machines and it always crashes the same way on the >>>>>>>>>> 66th >>>>>>>>>> process. >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Another thing I would like to know is if it's normal that any of my >>>>>>>>>>>> processes when calling MPI_Comm_connect or MPI_Comm_accept when the >>>>>>>>>>>> other side is not ready, is eating up a full CPU available. >>>>>>>>>>> >>>>>>>>>>> Yes - the waiting process is polling in a tight loop waiting for >>>>>>>>>>> the connection to be made. >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Any help would be appreciated, >>>>>>>>>>>> Grzegorz Maj >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> 2010/4/24 Ralph Castain <r...@open-mpi.org>: >>>>>>>>>>>>> Actually, OMPI is distributed with a daemon that does pretty much >>>>>>>>>>>>> what you >>>>>>>>>>>>> want. Checkout "man ompi-server". I originally wrote that code to >>>>>>>>>>>>> support >>>>>>>>>>>>> cross-application MPI publish/subscribe operations, but we can >>>>>>>>>>>>> utilize it >>>>>>>>>>>>> here too. Have to blame me for not making it more publicly known. >>>>>>>>>>>>> The attached patch upgrades ompi-server and modifies the >>>>>>>>>>>>> singleton startup >>>>>>>>>>>>> to provide your desired support. This solution works in the >>>>>>>>>>>>> following >>>>>>>>>>>>> manner: >>>>>>>>>>>>> 1. launch "ompi-server -report-uri <filename>". This starts a >>>>>>>>>>>>> persistent >>>>>>>>>>>>> daemon called "ompi-server" that acts as a rendezvous point for >>>>>>>>>>>>> independently started applications. The problem with starting >>>>>>>>>>>>> different >>>>>>>>>>>>> applications and wanting them to MPI connect/accept lies in the >>>>>>>>>>>>> need to have >>>>>>>>>>>>> the applications find each other. If they can't discover contact >>>>>>>>>>>>> info for >>>>>>>>>>>>> the other app, then they can't wire up their interconnects. The >>>>>>>>>>>>> "ompi-server" tool provides that rendezvous point. I don't like >>>>>>>>>>>>> that >>>>>>>>>>>>> comm_accept segfaulted - should have just error'd out. >>>>>>>>>>>>> 2. set OMPI_MCA_orte_server=file:<filename>" in the environment >>>>>>>>>>>>> where you >>>>>>>>>>>>> will start your processes. This will allow your singleton >>>>>>>>>>>>> processes to find >>>>>>>>>>>>> the ompi-server. I automatically also set the envar to connect >>>>>>>>>>>>> the MPI >>>>>>>>>>>>> publish/subscribe system for you. >>>>>>>>>>>>> 3. run your processes. As they think they are singletons, they >>>>>>>>>>>>> will detect >>>>>>>>>>>>> the presence of the above envar and automatically connect >>>>>>>>>>>>> themselves to the >>>>>>>>>>>>> "ompi-server" daemon. This provides each process with the ability >>>>>>>>>>>>> to perform >>>>>>>>>>>>> any MPI-2 operation. >>>>>>>>>>>>> I tested this on my machines and it worked, so hopefully it will >>>>>>>>>>>>> meet your >>>>>>>>>>>>> needs. You only need to run one "ompi-server" period, so long as >>>>>>>>>>>>> you locate >>>>>>>>>>>>> it where all of the processes can find the contact file and can >>>>>>>>>>>>> open a TCP >>>>>>>>>>>>> socket to the daemon. There is a way to knit multiple >>>>>>>>>>>>> ompi-servers into a >>>>>>>>>>>>> broader network (e.g., to connect processes that cannot directly >>>>>>>>>>>>> access a >>>>>>>>>>>>> server due to network segmentation), but it's a tad tricky - let >>>>>>>>>>>>> me know if >>>>>>>>>>>>> you require it and I'll try to help. >>>>>>>>>>>>> If you have trouble wiring them all into a single communicator, >>>>>>>>>>>>> you might >>>>>>>>>>>>> ask separately about that and see if one of our MPI experts can >>>>>>>>>>>>> provide >>>>>>>>>>>>> advice (I'm just the RTE grunt). >>>>>>>>>>>>> HTH - let me know how this works for you and I'll incorporate it >>>>>>>>>>>>> into future >>>>>>>>>>>>> OMPI releases. >>>>>>>>>>>>> Ralph >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Apr 24, 2010, at 1:49 AM, Krzysztof Zarzycki wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Hi Ralph, >>>>>>>>>>>>> I'm Krzysztof and I'm working with Grzegorz Maj on this our small >>>>>>>>>>>>> project/experiment. >>>>>>>>>>>>> We definitely would like to give your patch a try. But could you >>>>>>>>>>>>> please >>>>>>>>>>>>> explain your solution a little more? >>>>>>>>>>>>> You still would like to start one mpirun per mpi grid, and then >>>>>>>>>>>>> have >>>>>>>>>>>>> processes started by us to join the MPI comm? >>>>>>>>>>>>> It is a good solution of course. >>>>>>>>>>>>> But it would be especially preferable to have one daemon running >>>>>>>>>>>>> persistently on our "entry" machine that can handle several mpi >>>>>>>>>>>>> grid starts. >>>>>>>>>>>>> Can your patch help us this way too? >>>>>>>>>>>>> Thanks for your help! >>>>>>>>>>>>> Krzysztof >>>>>>>>>>>>> >>>>>>>>>>>>> On 24 April 2010 03:51, Ralph Castain <r...@open-mpi.org> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> In thinking about this, my proposed solution won't entirely fix >>>>>>>>>>>>>> the >>>>>>>>>>>>>> problem - you'll still wind up with all those daemons. I believe >>>>>>>>>>>>>> I can >>>>>>>>>>>>>> resolve that one as well, but it would require a patch. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Would you like me to send you something you could try? Might >>>>>>>>>>>>>> take a couple >>>>>>>>>>>>>> of iterations to get it right... >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Apr 23, 2010, at 12:12 PM, Ralph Castain wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hmmm....I -think- this will work, but I cannot guarantee it: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 1. launch one process (can just be a spinner) using mpirun that >>>>>>>>>>>>>>> includes >>>>>>>>>>>>>>> the following option: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> mpirun -report-uri file >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> where file is some filename that mpirun can create and insert >>>>>>>>>>>>>>> its >>>>>>>>>>>>>>> contact info into it. This can be a relative or absolute path. >>>>>>>>>>>>>>> This process >>>>>>>>>>>>>>> must remain alive throughout your application - doesn't matter >>>>>>>>>>>>>>> what it does. >>>>>>>>>>>>>>> It's purpose is solely to keep mpirun alive. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 2. set OMPI_MCA_dpm_orte_server=FILE:file in your environment, >>>>>>>>>>>>>>> where >>>>>>>>>>>>>>> "file" is the filename given above. This will tell your >>>>>>>>>>>>>>> processes how to >>>>>>>>>>>>>>> find mpirun, which is acting as a meeting place to handle the >>>>>>>>>>>>>>> connect/accept >>>>>>>>>>>>>>> operations >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Now run your processes, and have them connect/accept to each >>>>>>>>>>>>>>> other. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> The reason I cannot guarantee this will work is that these >>>>>>>>>>>>>>> processes >>>>>>>>>>>>>>> will all have the same rank && name since they all start as >>>>>>>>>>>>>>> singletons. >>>>>>>>>>>>>>> Hence, connect/accept is likely to fail. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> But it -might- work, so you might want to give it a try. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Apr 23, 2010, at 8:10 AM, Grzegorz Maj wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> To be more precise: by 'server process' I mean some process >>>>>>>>>>>>>>>> that I >>>>>>>>>>>>>>>> could run once on my system and it could help in creating those >>>>>>>>>>>>>>>> groups. >>>>>>>>>>>>>>>> My typical scenario is: >>>>>>>>>>>>>>>> 1. run N separate processes, each without mpirun >>>>>>>>>>>>>>>> 2. connect them into MPI group >>>>>>>>>>>>>>>> 3. do some job >>>>>>>>>>>>>>>> 4. exit all N processes >>>>>>>>>>>>>>>> 5. goto 1 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 2010/4/23 Grzegorz Maj <ma...@wp.pl>: >>>>>>>>>>>>>>>>> Thank you Ralph for your explanation. >>>>>>>>>>>>>>>>> And, apart from that descriptors' issue, is there any other >>>>>>>>>>>>>>>>> way to >>>>>>>>>>>>>>>>> solve my problem, i.e. to run separately a number of >>>>>>>>>>>>>>>>> processes, >>>>>>>>>>>>>>>>> without mpirun and then to collect them into an MPI intracomm >>>>>>>>>>>>>>>>> group? >>>>>>>>>>>>>>>>> If I for example would need to run some 'server process' >>>>>>>>>>>>>>>>> (even using >>>>>>>>>>>>>>>>> mpirun) for this task, that's OK. Any ideas? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>> Grzegorz Maj >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 2010/4/18 Ralph Castain <r...@open-mpi.org>: >>>>>>>>>>>>>>>>>> Okay, but here is the problem. If you don't use mpirun, and >>>>>>>>>>>>>>>>>> are not >>>>>>>>>>>>>>>>>> operating in an environment we support for "direct" launch >>>>>>>>>>>>>>>>>> (i.e., starting >>>>>>>>>>>>>>>>>> processes outside of mpirun), then every one of those >>>>>>>>>>>>>>>>>> processes thinks it is >>>>>>>>>>>>>>>>>> a singleton - yes? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> What you may not realize is that each singleton immediately >>>>>>>>>>>>>>>>>> fork/exec's an orted daemon that is configured to behave >>>>>>>>>>>>>>>>>> just like mpirun. >>>>>>>>>>>>>>>>>> This is required in order to support MPI-2 operations such as >>>>>>>>>>>>>>>>>> MPI_Comm_spawn, MPI_Comm_connect/accept, etc. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> So if you launch 64 processes that think they are >>>>>>>>>>>>>>>>>> singletons, then >>>>>>>>>>>>>>>>>> you have 64 copies of orted running as well. This eats up a >>>>>>>>>>>>>>>>>> lot of file >>>>>>>>>>>>>>>>>> descriptors, which is probably why you are hitting this 65 >>>>>>>>>>>>>>>>>> process limit - >>>>>>>>>>>>>>>>>> your system is probably running out of file descriptors. You >>>>>>>>>>>>>>>>>> might check you >>>>>>>>>>>>>>>>>> system limits and see if you can get them revised upward. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Apr 17, 2010, at 4:24 PM, Grzegorz Maj wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Yes, I know. The problem is that I need to use some special >>>>>>>>>>>>>>>>>>> way for >>>>>>>>>>>>>>>>>>> running my processes provided by the environment in which >>>>>>>>>>>>>>>>>>> I'm >>>>>>>>>>>>>>>>>>> working >>>>>>>>>>>>>>>>>>> and unfortunately I can't use mpirun. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> 2010/4/18 Ralph Castain <r...@open-mpi.org>: >>>>>>>>>>>>>>>>>>>> Guess I don't understand why you can't use mpirun - all it >>>>>>>>>>>>>>>>>>>> does is >>>>>>>>>>>>>>>>>>>> start things, provide a means to forward io, etc. It >>>>>>>>>>>>>>>>>>>> mainly sits there >>>>>>>>>>>>>>>>>>>> quietly without using any cpu unless required to support >>>>>>>>>>>>>>>>>>>> the job. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Sounds like it would solve your problem. Otherwise, I know >>>>>>>>>>>>>>>>>>>> of no >>>>>>>>>>>>>>>>>>>> way to get all these processes into comm_world. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On Apr 17, 2010, at 2:27 PM, Grzegorz Maj wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>>>>> I'd like to dynamically create a group of processes >>>>>>>>>>>>>>>>>>>>> communicating >>>>>>>>>>>>>>>>>>>>> via >>>>>>>>>>>>>>>>>>>>> MPI. Those processes need to be run without mpirun and >>>>>>>>>>>>>>>>>>>>> create >>>>>>>>>>>>>>>>>>>>> intracommunicator after the startup. Any ideas how to do >>>>>>>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>>>>>> efficiently? >>>>>>>>>>>>>>>>>>>>> I came up with a solution in which the processes are >>>>>>>>>>>>>>>>>>>>> connecting >>>>>>>>>>>>>>>>>>>>> one by >>>>>>>>>>>>>>>>>>>>> one using MPI_Comm_connect, but unfortunately all the >>>>>>>>>>>>>>>>>>>>> processes >>>>>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>> are already in the group need to call MPI_Comm_accept. >>>>>>>>>>>>>>>>>>>>> This means >>>>>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>> when the n-th process wants to connect I need to collect >>>>>>>>>>>>>>>>>>>>> all the >>>>>>>>>>>>>>>>>>>>> n-1 >>>>>>>>>>>>>>>>>>>>> processes on the MPI_Comm_accept call. After I run about >>>>>>>>>>>>>>>>>>>>> 40 >>>>>>>>>>>>>>>>>>>>> processes >>>>>>>>>>>>>>>>>>>>> every subsequent call takes more and more time, which I'd >>>>>>>>>>>>>>>>>>>>> like to >>>>>>>>>>>>>>>>>>>>> avoid. >>>>>>>>>>>>>>>>>>>>> Another problem in this solution is that when I try to >>>>>>>>>>>>>>>>>>>>> connect >>>>>>>>>>>>>>>>>>>>> 66-th >>>>>>>>>>>>>>>>>>>>> process the root of the existing group segfaults on >>>>>>>>>>>>>>>>>>>>> MPI_Comm_accept. >>>>>>>>>>>>>>>>>>>>> Maybe it's my bug, but it's weird as everything works >>>>>>>>>>>>>>>>>>>>> fine for at >>>>>>>>>>>>>>>>>>>>> most >>>>>>>>>>>>>>>>>>>>> 65 processes. Is there any limitation I don't know about? >>>>>>>>>>>>>>>>>>>>> My last question is about MPI_COMM_WORLD. When I run my >>>>>>>>>>>>>>>>>>>>> processes >>>>>>>>>>>>>>>>>>>>> without mpirun their MPI_COMM_WORLD is the same as >>>>>>>>>>>>>>>>>>>>> MPI_COMM_SELF. >>>>>>>>>>>>>>>>>>>>> Is >>>>>>>>>>>>>>>>>>>>> there any way to change MPI_COMM_WORLD and set it to the >>>>>>>>>>>>>>>>>>>>> intracommunicator that I've created? >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>>> Grzegorz Maj >>>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>>> users mailing list >>>>>>>>>>>>>>>>>>>>> us...@open-mpi.org >>>>>>>>>>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>> users mailing list >>>>>>>>>>>>>>>>>>>> us...@open-mpi.org >>>>>>>>>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>> users mailing list >>>>>>>>>>>>>>>>>>> us...@open-mpi.org >>>>>>>>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>> users mailing list >>>>>>>>>>>>>>>>>> us...@open-mpi.org >>>>>>>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>> users mailing list >>>>>>>>>>>>>>>> us...@open-mpi.org >>>>>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> users mailing list >>>>>>>>>>>>>> us...@open-mpi.org >>>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> users mailing list >>>>>>>>>>>>> us...@open-mpi.org >>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> users mailing list >>>>>>>>>>>>> us...@open-mpi.org >>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>>>>>> >>>>>>>>>>>> <client.c><server.c>_______________________________________________ >>>>>>>>>>>> users mailing list >>>>>>>>>>>> us...@open-mpi.org >>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> users mailing list >>>>>>>>>>> us...@open-mpi.org >>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> users mailing list >>>>>>>>>> us...@open-mpi.org >>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> users mailing list >>>>>>>>> us...@open-mpi.org >>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> users mailing list >>>>>>>> us...@open-mpi.org >>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> users mailing list >>>>>>> us...@open-mpi.org >>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>> >>>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> users mailing list >>>>>> us...@open-mpi.org >>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>> >>>>> >>>>> _______________________________________________ >>>>> users mailing list >>>>> us...@open-mpi.org >>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>> >>>> >>>> _______________________________________________ >>>> users mailing list >>>> us...@open-mpi.org >>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>> >> >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users > > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
