Hi,
I would like to ask - are there any new solutions or investigations in this
problem?
Cheers,
Matus Dobrotka
2016-07-19 15:23 GMT+02:00 Gilles Gouaillardet <
gilles.gouaillar...@gmail.com>:
> my bad for the confusion,
>
> I misread you and miswrote my reply.
>
> I will investigate this again.
>
> strictly speaking, the clients can only start after the server first write
> the port info to a file.
> if you start the client right after the server start, they might use
> incorrect/outdated info and cause all the test hang.
>
> I will start reproducing the hang
>
> Cheers,
>
> Gilles
>
>
> On Tuesday, July 19, 2016, M. D. <matus.dobro...@gmail.com> wrote:
>
>> Yes I understand it, but I think, this is exactly that situation you are
>> talking about. In my opinion, the test is doing exactly what you said -
>> when a new player is willing to join, other players must invoke
>> MPI_Comm_accept().
>> All *other* players must invoke MPI_Comm_accept(). Only the last client
>> (in this case last player which wants to join) does not
>> invoke MPI_Comm_accept(), because this client invokes only
>> MPI_Comm_connect(). He is connecting to communicator, in which all other
>> players are already involved and therefore this last client doesn't have to
>> invoke MPI_Comm_accept().
>>
>> Am I still missing something in this my reflection?
>>
>> Matus
>>
>> 2016-07-19 10:55 GMT+02:00 Gilles Gouaillardet <gil...@rist.or.jp>:
>>
>>> here is what the client is doing
>>>
>>> printf("CLIENT: after merging, new comm: size=%d rank=%d\n", size,
>>> rank) ;
>>>
>>> for (i = rank ; i < num_clients ; i++)
>>> {
>>> /* client performs a collective accept */
>>> CHK(MPI_Comm_accept(server_port_name, MPI_INFO_NULL, 0,
>>> intracomm, &intercomm)) ;
>>>
>>> printf("CLIENT: connected to server on port\n") ;
>>> [...]
>>>
>>> }
>>>
>>> 2) has rank 1
>>>
>>> /* and 3) has rank 2) */
>>>
>>> so unless you run 2) with num_clients=2, MPI_Comm_accept() is never
>>> called, hence my analysis of the crash/hang
>>>
>>>
>>> I understand what you are trying to achieve, keep in mind
>>> MPI_Comm_accept() is a collective call, so when a new player
>>>
>>> is willing to join, other players must invoke MPI_Comm_accept().
>>>
>>> and it is up to you to make sure that happens
>>>
>>>
>>> Cheers,
>>>
>>>
>>> Gilles
>>>
>>> On 7/19/2016 5:48 PM, M. D. wrote:
>>>
>>>
>>>
>>> 2016-07-19 10:06 GMT+02:00 Gilles Gouaillardet <gil...@rist.or.jp>:
>>>
>>>> MPI_Comm_accept must be called by all the tasks of the local
>>>> communicator.
>>>>
>>> Yes, that's how I understand it. In the source code of the test, all the
>>> tasks call MPI_Comm_accept - server and also relevant clients.
>>>
>>>> so if you
>>>>
>>>> 1) mpirun -np 1 ./singleton_client_server 2 1
>>>>
>>>> 2) mpirun -np 1 ./singleton_client_server 2 0
>>>>
>>>> 3) mpirun -np 1 ./singleton_client_server 2 0
>>>>
>>>> then 3) starts after 2) has exited, so on 1), intracomm is made of 1)
>>>> and an exited task (2)
>>>>
>>> This is not true in my opinion - because of above mentioned fact that
>>> MPI_Comm_accept is called by all the tasks of the local communicator.
>>>
>>>> /*
>>>>
>>>> strictly speaking, there is a race condition, if 2) has exited, then
>>>> MPI_Comm_accept will crash when 1) informs 2) that 3) has joined.
>>>>
>>>> if 2) has not yet exited, then the test will hang because 2) does not
>>>> invoke MPI_Comm_accept
>>>>
>>>> */
>>>>
>>> Task 2) does not exit, because of blocking call of MPI_Comm_accept.
>>>
>>>>
>>>>
>>>
>>>> there are different ways of seeing things :
>>>>
>>>> 1) this is an incorrect usage of the test, the number of clients should
>>>> be the same everywhere
>>>>
>>>> 2) task 2) should not exit (because it did not call
>>>> MPI_Comm_disconnect()) and the test should hang when
>>>>
>>>> starting task 3) because task 2) does not call MPI_Comm_accept()
>>>>
>>>>
>>>> ad 1) I am sorry, but maybe I do not understand what you think - In my
>>> previous post I wrote that the number of clients is the same in every
>>> mpirun instance.
>>> ad 2) it is the same as above
>>>
>>>> i do not know how you want to spawn your tasks.
>>>>
>>>> if 2) and 3) do not need to communicate with each other (they only
>>>> communicate with 1)), then
>>>>
>>>> you can simply MPI_Comm_accept(MPI_COMM_WORLD) in 1)
>>>>
>>>> if 2 and 3) need to communicate with each other, it would be much
>>>> easier to MPI_Comm_spawn or MPI_Comm_spawn_multiple only once in 1),
>>>>
>>>> so there is only one inter communicator with all the tasks.
>>>>
>>> My aim is that all the tasks need to communicate with each other. I am
>>> implementing a distributed application - game with more players
>>> communicating with each other via MPI. It should work as follows - First
>>> player creates a game and waits for other players to connect to this game.
>>> On different computers (in the same network) the other players can join
>>> this game. When they are connected, they should be able to play this game
>>> together.
>>> I hope, it is clear what my idea is. If it is not, just ask me, please.
>>>
>>>>
>>>> The current test program is growing incrementally the intercomm, which
>>>> does require extra steps for synchronization.
>>>>
>>>>
>>>> Cheers,
>>>>
>>>>
>>>> Gilles
>>>>
>>> Cheers,
>>>
>>> Matus
>>>
>>>> On 7/19/2016 4:37 PM, M. D. wrote:
>>>>
>>>> Hi,
>>>> thank you for your interest in this topic.
>>>>
>>>> So, I normally run the test as follows:
>>>> Firstly, I run "server" (second parameter is 1):
>>>> *mpirun -np 1 ./singleton_client_server number_of_clients 1*
>>>>
>>>> Secondly, I run corresponding number of "clients" via following command:
>>>> *mpirun -np 1 ./singleton_client_server number_of_clients 0*
>>>>
>>>> So, for example with 3 clients I do:
>>>> mpirun -np 1 ./singleton_client_server 3 1
>>>> mpirun -np 1 ./singleton_client_server 3 0
>>>> mpirun -np 1 ./singleton_client_server 3 0
>>>> mpirun -np 1 ./singleton_client_server 3 0
>>>>
>>>> It means you are right - there should be the same number of clients in
>>>> each mpirun instance.
>>>>
>>>> The test does not involve MPI_Comm_disconnect(), but the problem in
>>>> the test is in the earlier position, because some of clients (in the most
>>>> cases actually the last client) cannot sometimes connect to the server and
>>>> therefore all clients with server are hanging (waiting for the connections
>>>> with the last client(s) ).
>>>>
>>>> So, the bahaviour of accept/connect method is a bit confusing for me.
>>>> If I understand you, according to your post - the problem is not in the
>>>> timeout value, isn't it?
>>>>
>>>> Cheers,
>>>>
>>>> Matus
>>>>
>>>> 2016-07-19 6:28 GMT+02:00 Gilles Gouaillardet <gil...@rist.or.jp>:
>>>>
>>>>> How do you run the test ?
>>>>>
>>>>> you should have the same number of clients in each mpirun instance,
>>>>> the following simple shell starts the test as i think it is supposed to
>>>>>
>>>>> note the test itself is arguable since MPI_Comm_disconnect() is never
>>>>> invoked
>>>>>
>>>>> (and you will observe some related dpm_base_disconnect_init errors)
>>>>>
>>>>>
>>>>> #!/bin/sh
>>>>>
>>>>> clients=3
>>>>>
>>>>> screen -d -m sh -c "mpirun -np 1 ./singleton_client_server
>>>>> $clients 1 2>&1 | tee /tmp/server.$clients"
>>>>> for i in $(seq $clients); do
>>>>>
>>>>> sleep 1
>>>>>
>>>>> screen -d -m sh -c "mpirun -np 1 ./singleton_client_server
>>>>> $clients 0 2>&1 | tee /tmp/client.$clients.$i"
>>>>> done
>>>>>
>>>>>
>>>>> Ralph,
>>>>>
>>>>>
>>>>> this test fails with master.
>>>>>
>>>>> when the "server" (second parameter is 1), MPI_Comm_accept() fails
>>>>> with a timeout.
>>>>>
>>>>> i ompi/dpm/dpm.c, there is a hard coded 60 seconds timeout
>>>>>
>>>>> OPAL_PMIX_EXCHANGE(rc, &info, &pdat, 60);
>>>>>
>>>>> but this is not the timeout that is triggered ...
>>>>>
>>>>> the eviction_cbfunc timeout function is invoked, and it has been set
>>>>> when opal_hotel_init() was invoked in orte/orted/pmix/pmix_server.c
>>>>>
>>>>>
>>>>> default timeout is 2 seconds, but in this case (user invokes
>>>>> MPI_Comm_accept), i guess the timeout should be infinite or 60 seconds
>>>>> (hard coded value described above)
>>>>>
>>>>> sadly, if i set a higher timeout value (mpirun --mca
>>>>> orte_pmix_server_max_wait 180 ...), MPI_Comm_accept() does not return when
>>>>> the client invokes MPI_Comm_connect()
>>>>>
>>>>>
>>>>> could you please have a look at this ?
>>>>>
>>>>>
>>>>> Cheers,
>>>>>
>>>>>
>>>>> Gilles
>>>>>
>>>>> On 7/15/2016 9:20 PM, M. D. wrote:
>>>>>
>>>>> Hello,
>>>>>
>>>>> I have a problem with basic client - server application. I tried to
>>>>> run C program from this website
>>>>> <https://github.com/hpc/cce-mpi-openmpi-1.7.1/blob/master/orte/test/mpi/singleton_client_server.c>
>>>>> https://github.com/hpc/cce-mpi-openmpi-1.7.1/blob/
>>>>> master/orte/test/mpi/singleton_client_server.c
>>>>> I saw this program mentioned in many discussions in your website, so I
>>>>> expected that it should work properly, but after more testing I found out
>>>>> that there is probably an error somewhere in connect/accept method. I have
>>>>> read many discussions and threads on your website, but I have not found
>>>>> similar problem that I am facing. It seems that nobody had similar problem
>>>>> like me. When I run this app with one server and more clients
>>>>> (3,4,5,6,...)
>>>>> sometimes the app hangs. It hangs when second or next client wants to
>>>>> connect to the server (it depends, sometimes third client hangs, sometimes
>>>>> fourth, sometimes second, and so on).
>>>>> So it means that app starts to hang where server waits for accept and
>>>>> client waits for connect. And it is not possible to continue, because this
>>>>> client cannot connect to the server. It is strange, because I observed
>>>>> this
>>>>> behaviour only in some cases... Sometimes it works without any problems,
>>>>> sometimes it does not work. The behaviour is unpredictable and not stable.
>>>>>
>>>>> I have installed openmpi 1.10.2 on my Fedora 19. I have the same
>>>>> problem with Java alternative of this application. It hangs also
>>>>> sometimes... I need this app in Java, but firstly it must work properly in
>>>>> C implementation. Because of this strange behaviour I assume that there
>>>>> can
>>>>> be an error maybe inside of openmpi implementation of connect/accept
>>>>> methods. I tried it also with another version of openmpi - 1.8.1. However,
>>>>> the problem did not disappear.
>>>>>
>>>>> Could you help me, what can cause the problem? Maybe I did not get
>>>>> something about openmpi (or connect/server) and the problem is with me...
>>>>> I
>>>>> will appreciate any your help, support, or interest about this topic.
>>>>>
>>>>> Best regards,
>>>>> Matus Dobrotka
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> users mailing listus...@open-mpi.org
>>>>> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>> Link to this post:
>>>>> http://www.open-mpi.org/community/lists/users/2016/07/29673.php
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> users mailing list
>>>>> us...@open-mpi.org
>>>>> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>> Link to this post:
>>>>> <http://www.open-mpi.org/community/lists/users/2016/07/29681.php>
>>>>> http://www.open-mpi.org/community/lists/users/2016/07/29681.php
>>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> users mailing listus...@open-mpi.org
>>>> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>
>>>> Link to this post:
>>>> http://www.open-mpi.org/community/lists/users/2016/07/29687.php
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> users mailing list
>>>> us...@open-mpi.org
>>>> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users
>>>> Link to this post: http://www.open-mpi.org/commun
>>>> ity/lists/users/2016/07/29688.php
>>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> users mailing listus...@open-mpi.org
>>> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users
>>>
>>> Link to this post:
>>> http://www.open-mpi.org/community/lists/users/2016/07/29689.php
>>>
>>>
>>>
>>> _______________________________________________
>>> users mailing list
>>> us...@open-mpi.org
>>> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users
>>> Link to this post: http://www.open-mpi.org/commun
>>> ity/lists/users/2016/07/29690.php
>>>
>>
>>
> _______________________________________________
> users mailing list
> us...@open-mpi.org
> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: http://www.open-mpi.org/commun
> ity/lists/users/2016/07/29693.php
>
_______________________________________________
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
