On Jul 21, 2010, at 7:44 AM, Philippe wrote: > Ralph, > > Sorry for the late reply -- I was away on vacation.
no problem at all! > > regarding your earlier question about how many processes where > involved when the memory was entirely allocated, it was only two, a > sender and a receiver. I'm still trying to pinpoint what can be > different between the standalone case and the "integrated" case. I > will try to find out what part of the code is allocating memory in a > loop. hmmm....that sounds like a bug in your program. let me know what you find > > > On Tue, Jul 20, 2010 at 12:51 AM, Ralph Castain <r...@open-mpi.org> wrote: >> Well, I finally managed to make this work without the required ompi-server >> rendezvous point. The fix is only in the devel trunk right now - I'll have >> to ask the release managers for 1.5 and 1.4 if they want it ported to those >> series. >> > > great -- i'll give it a try > >> On the notion of integrating OMPI to your launch environment: remember that >> we don't necessarily require that you use mpiexec for that purpose. If your >> launch environment provides just a little info in the environment of the >> launched procs, we can usually devise a method that allows the procs to >> perform an MPI_Init as a single job without all this work you are doing. >> > > I'm working on creating operators using MPI for the IBM product > "InfoSphere Streams". It has its own launching mechanism to start the > processes. However I can pass some information to the processes that > belong to the same job (Streams job -- which should neatly map to MPI > job). > >> Only difference is that your procs will all block in MPI_Init until they >> -all- have executed that function. If that isn't a problem, this would be a >> much more scalable and reliable method than doing it thru massive calls to >> MPI_Port_connect. >> > > in the general case, that would be a problem, but for my prototype, > this is acceptable. > > In general, each process is composed of operators, some may be MPI > related and some may not. But in my case, I know ahead of time which > processes will be part of the MPI job, so I can easily deal with the > fact that they would block on MPI_init (actually -- MPI_thread_init > since its using a lot of threads). We have talked in the past about creating a non-blocking MPI_Init as an extension to the standard. It would lock you to Open MPI, though... Regardless, at some point you would have to know how many processes are going to be part of the job so you can know when MPI_Init is complete. I would think you would require that info for the singleton wireup anyway - yes? Otherwise, how would you know when to quit running connect-accept? > > Is there a documentation or example I can use to see what information > I can pass to the processes to enable that? Is it just environment > variables? No real documentation - a lack I should probably fill. At the moment, we don't have a "generic" module for standalone launch, but I can create one as it is pretty trivial. I would then need you to pass each process envars telling it the total number of processes in the MPI job, its rank within that job, and a file where some rendezvous process (can be rank=0) has provided that port string. Armed with that info, I can wireup the job. Won't be as scalable as an mpirun-initiated startup, but will be much better than doing it from singletons. Or if you prefer, we could setup an "infosphere" module that we could customize for this system. Main thing here would be to provide us with some kind of regex (or access to a file containing the info) that describes the map of rank to node so we can construct the wireup communication pattern. Either way would work. The second is more scalable, but I don't know if you have (or can construct) the map info. > > Many thanks! > p. > >> >> On Jul 18, 2010, at 4:09 PM, Philippe wrote: >> >>> Ralph, >>> >>> thanks for investigating. >>> >>> I've applied the two patches you mentioned earlier and ran with the >>> ompi server. Although i was able to runn our standalone test, when I >>> integrated the changes to our code, the processes entered a crazy loop >>> and allocated all the memory available when calling MPI_Port_Connect. >>> I was not able to identify why it works standalone but not integrated >>> with our code. If I found why, I'll let your know. >>> >>> looking forward to your findings. We'll be happy to test any patches >>> if you have some! >>> >>> p. >>> >>> On Sat, Jul 17, 2010 at 9:47 PM, Ralph Castain <r...@open-mpi.org> wrote: >>>> Okay, I can reproduce this problem. Frankly, I don't think this ever >>>> worked with OMPI, and I'm not sure how the choice of BTL makes a >>>> difference. >>>> >>>> The program is crashing in the communicator definition, which involves a >>>> communication over our internal out-of-band messaging system. That system >>>> has zero connection to any BTL, so it should crash either way. >>>> >>>> Regardless, I will play with this a little as time allows. Thanks for the >>>> reproducer! >>>> >>>> >>>> On Jun 25, 2010, at 7:23 AM, Philippe wrote: >>>> >>>>> Hi, >>>>> >>>>> I'm trying to run a test program which consists of a server creating a >>>>> port using MPI_Open_port and N clients using MPI_Comm_connect to >>>>> connect to the server. >>>>> >>>>> I'm able to do so with 1 server and 2 clients, but with 1 server + 3 >>>>> clients, I get the following error message: >>>>> >>>>> [node003:32274] [[37084,0],0]:route_callback tried routing message >>>>> from [[37084,1],0] to [[40912,1],0]:102, can't find route >>>>> >>>>> This is only happening with the openib BTL. With tcp BTL it works >>>>> perfectly fine (ofud also works as a matter of fact...). This has been >>>>> tested on two completely different clusters, with identical results. >>>>> In either cases, the IB frabic works normally. >>>>> >>>>> Any help would be greatly appreciated! Several people in my team >>>>> looked at the problem. Google and the mailing list archive did not >>>>> provide any clue. I believe that from an MPI standpoint, my test >>>>> program is valid (and it works with TCP, which make me feel better >>>>> about the sequence of MPI calls) >>>>> >>>>> Regards, >>>>> Philippe. >>>>> >>>>> >>>>> >>>>> Background: >>>>> >>>>> I intend to use openMPI to transport data inside a much larger >>>>> application. Because of that, I cannot used mpiexec. Each process is >>>>> started by our own "job management" and use a name server to find >>>>> about each others. Once all the clients are connected, I would like >>>>> the server to do MPI_Recv to get the data from all the client. I dont >>>>> care about the order or which client are sending data, as long as I >>>>> can receive it with on call. Do do that, the clients and the server >>>>> are going through a series of Comm_accept/Conn_connect/Intercomm_merge >>>>> so that at the end, all the clients and the server are inside the same >>>>> intracomm. >>>>> >>>>> Steps: >>>>> >>>>> I have a sample program that show the issue. I tried to make it as >>>>> short as possible. It needs to be executed on a shared file system >>>>> like NFS because the server write the port info to a file that the >>>>> client will read. To reproduce the issue, the following steps should >>>>> be performed: >>>>> >>>>> 0. compile the test with "mpicc -o ben12 ben12.c" >>>>> 1. ssh to the machine that will be the server >>>>> 2. run ./ben12 3 1 >>>>> 3. ssh to the machine that will be the client #1 >>>>> 4. run ./ben12 3 0 >>>>> 5. repeat step 3-4 for client #2 and #3 >>>>> >>>>> the server accept the connection from client #1 and merge it in a new >>>>> intracomm. It then accept connection from client #2 and merge it. when >>>>> the client #3 arrives, the server accept the connection, but that >>>>> cause client #1 and #2 to die with the error above (see the complete >>>>> trace in the tarball). >>>>> >>>>> The exact steps are: >>>>> >>>>> - server open port >>>>> - server does accept >>>>> - client #1 does connect >>>>> - server and client #1 do merge >>>>> - server does accept >>>>> - client #2 does connect >>>>> - server, client #1 and client #2 do merge >>>>> - server does accept >>>>> - client #3 does connect >>>>> - server, client #1, client #2 and client #3 do merge >>>>> >>>>> >>>>> My infiniband network works normally with other test programs or >>>>> applications (MPI or others like Verbs). >>>>> >>>>> Info about my setup: >>>>> >>>>> openMPI version = 1.4.1 (I also tried 1.4.2, nightly snapshot of >>>>> 1.4.3, nightly snapshot of 1.5 --- all show the same error) >>>>> config.log in the tarball >>>>> "ompi_info --all" in the tarball >>>>> OFED version = 1.3 installed from RHEL 5.3 >>>>> Distro = RedHat Entreprise Linux 5.3 >>>>> Kernel = 2.6.18-128.4.1.el5 x86_64 >>>>> subnet manager = built-in SM from the cisco/topspin switch >>>>> output of ibv_devinfo included in the tarball (there are no "bad" >>>>> nodes) >>>>> "ulimit -l" says "unlimited" >>>>> >>>>> The tarball contains: >>>>> >>>>> - ben12.c: my test program showing the behavior >>>>> - config.log / config.out / make.out / make-install.out / >>>>> ifconfig.txt / ibv-devinfo.txt / ompi_info.txt >>>>> - trace-tcp.txt: output of the server and each client when it works >>>>> with TCP (I added "btl = tcp,self" in ~/.openmpi/mca-params.conf) >>>>> - trace-ib.txt: output of the server and each client when it fails >>>>> with IB (I added "btl = openib,self" in ~/.openmpi/mca-params.conf) >>>>> >>>>> I hope I provided enough info for somebody to reproduce the problem... >>>>> <ompi-output.tar.bz2>_______________________________________________ >>>>> users mailing list >>>>> us...@open-mpi.org >>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>> >>>> >>>> _______________________________________________ >>>> users mailing list >>>> us...@open-mpi.org >>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>> >>> >>> _______________________________________________ >>> users mailing list >>> us...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/users >> >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users >> > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
