Yes, that's fine. Thx!
On Aug 24, 2010, at 9:02 AM, Philippe wrote:
> awesome, I'll give it a spin! with the parameters as below?
>
> p.
>
> On Tue, Aug 24, 2010 at 10:47 AM, Ralph Castain <r...@open-mpi.org> wrote:
>> I think I have this working now - try anything on or after r23647
>>
>>
>> On Aug 23, 2010, at 1:36 PM, Philippe wrote:
>>
>>> sure. I took a guess at ppn and nodes for the case where 2 processes
>>> are on the same node... I dont claim these are the right values ;-)
>>>
>>>
>>>
>>> c0301b10e1 ~/mpi> env|grep OMPI
>>> OMPI_MCA_orte_nodes=c0301b10e1
>>> OMPI_MCA_orte_rank=0
>>> OMPI_MCA_orte_ppn=2
>>> OMPI_MCA_orte_num_procs=2
>>> OMPI_MCA_oob_tcp_static_ports_v6=10000-11000
>>> OMPI_MCA_ess=generic
>>> OMPI_MCA_orte_jobid=9999
>>> OMPI_MCA_oob_tcp_static_ports=10000-11000
>>> c0301b10e1 ~/hpa/benchmark/mpi> ./ben1 1 1 1
>>> [c0301b10e1:22827] [[0,9999],0] assigned port 10001
>>> [c0301b10e1:22827] [[0,9999],0] accepting connections via event library
>>> minsize=1 maxsize=1 delay=1.000000
>>>
>>> <no more output after that>
>>>
>>>
>>> c0301b10e1 ~/mpi> env|grep OMPI
>>> OMPI_MCA_orte_nodes=c0301b10e1
>>> OMPI_MCA_orte_rank=1
>>> OMPI_MCA_orte_ppn=2
>>> OMPI_MCA_orte_num_procs=2
>>> OMPI_MCA_oob_tcp_static_ports_v6=10000-11000
>>> OMPI_MCA_ess=generic
>>> OMPI_MCA_orte_jobid=9999
>>> OMPI_MCA_oob_tcp_static_ports=10000-11000
>>> c0301b10e1 ~/hpa/benchmark/mpi> ./ben1 1 1 1
>>> [c0301b10e1:22830] [[0,9999],1] assigned port 10002
>>> [c0301b10e1:22830] [[0,9999],1] accepting connections via event library
>>> [c0301b10e1:22830] [[0,9999],1]-[[0,0],0] mca_oob_tcp_send_nb: tag 15 size
>>> 189
>>> [c0301b10e1:22830] [[0,9999],1]-[[0,0],0]
>>> mca_oob_tcp_peer_try_connect: connecting port 10002 to:
>>> 10.4.72.110:10000
>>> [c0301b10e1:22830] [[0,9999],1]-[[0,0],0]
>>> mca_oob_tcp_peer_complete_connect: connection failed: Connection
>>> refused (111) - retrying
>>> [c0301b10e1:22830] [[0,9999],1]-[[0,0],0]
>>> mca_oob_tcp_peer_try_connect: connecting port 10002 to:
>>> 10.4.72.110:10000
>>> [c0301b10e1:22830] [[0,9999],1]-[[0,0],0]
>>> mca_oob_tcp_peer_complete_connect: connection failed: Connection
>>> refused (111) - retrying
>>> [c0301b10e1:22830] [[0,9999],1]-[[0,0],0]
>>> mca_oob_tcp_peer_try_connect: connecting port 10002 to:
>>> 10.4.72.110:10000
>>> [c0301b10e1:22830] [[0,9999],1]-[[0,0],0]
>>> mca_oob_tcp_peer_complete_connect: connection failed: Connection
>>> refused (111) - retrying
>>>
>>> <repeats..>
>>>
>>>
>>> Thanks!
>>> p.
>>>
>>>
>>> On Mon, Aug 23, 2010 at 3:24 PM, Ralph Castain <r...@open-mpi.org> wrote:
>>>> Can you send me the values you are using for the relevant envars? That way
>>>> I can try to replicate here
>>>>
>>>>
>>>> On Aug 23, 2010, at 1:15 PM, Philippe wrote:
>>>>
>>>>> I took a look at the code but I'm afraid I dont see anything wrong.
>>>>>
>>>>> p.
>>>>>
>>>>> On Thu, Aug 19, 2010 at 2:32 PM, Ralph Castain <r...@open-mpi.org> wrote:
>>>>>> Yes, that is correct - we reserve the first port in the range for a
>>>>>> daemon,
>>>>>> should one exist.
>>>>>> The problem is clearly that get_node_rank is returning the wrong value
>>>>>> for
>>>>>> the second process (your rank=1). If you want to dig deeper, look at the
>>>>>> orte/mca/ess/generic code where it generates the nidmap and pidmap.
>>>>>> There is
>>>>>> a bug down there somewhere that gives the wrong answer when ppn > 1.
>>>>>>
>>>>>>
>>>>>> On Thu, Aug 19, 2010 at 12:12 PM, Philippe <phil...@mytoaster.net> wrote:
>>>>>>>
>>>>>>> Ralph,
>>>>>>>
>>>>>>> somewhere in ./orte/mca/oob/tcp/oob_tcp.c, there is this comment:
>>>>>>>
>>>>>>> orte_node_rank_t nrank;
>>>>>>> /* do I know my node_local_rank yet? */
>>>>>>> if (ORTE_NODE_RANK_INVALID != (nrank =
>>>>>>> orte_ess.get_node_rank(ORTE_PROC_MY_NAME)) &&
>>>>>>> (nrank+1) <
>>>>>>> opal_argv_count(mca_oob_tcp_component.tcp4_static_ports)) {
>>>>>>> /* any daemon takes the first entry, so we start
>>>>>>> with the second */
>>>>>>>
>>>>>>> which seems constant with process #0 listening on 10001. the question
>>>>>>> would be why process #1 attempt to connect to port 10000 then? or
>>>>>>> maybe totally unrelated :-)
>>>>>>>
>>>>>>> btw, if I trick process #1 to open the connection to 10001 by shifting
>>>>>>> the range, I now get this error and the process terminate immediately:
>>>>>>>
>>>>>>> [c0301b10e1:03919] [[0,9999],1]-[[0,0],0]
>>>>>>> mca_oob_tcp_peer_recv_connect_ack: received unexpected process
>>>>>>> identifier [[0,9999],0]
>>>>>>>
>>>>>>> good luck with the surgery and wishing you a prompt recovery!
>>>>>>>
>>>>>>> p.
>>>>>>>
>>>>>>> On Thu, Aug 19, 2010 at 2:02 PM, Ralph Castain <r...@open-mpi.org>
>>>>>>> wrote:
>>>>>>>> Something doesn't look right - here is what the algo attempts to do:
>>>>>>>> given a port range of 10000-12000, the lowest rank'd process on the
>>>>>>>> node
>>>>>>>> should open port 10000. The next lowest rank on the node will open
>>>>>>>> 10001,
>>>>>>>> etc.
>>>>>>>> So it looks to me like there is some confusion in the local rank algo.
>>>>>>>> I'll
>>>>>>>> have to look at the generic module - must be a bug in it somewhere.
>>>>>>>> This might take a couple of days as I have surgery tomorrow morning, so
>>>>>>>> please forgive the delay.
>>>>>>>>
>>>>>>>> On Thu, Aug 19, 2010 at 11:13 AM, Philippe <phil...@mytoaster.net>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Ralph,
>>>>>>>>>
>>>>>>>>> I'm able to use the generic module when the processes are on different
>>>>>>>>> machines.
>>>>>>>>>
>>>>>>>>> what would be the values of the EV when two processes are on the same
>>>>>>>>> machine (hopefully talking over SHM).
>>>>>>>>>
>>>>>>>>> i've played with combination of nodelist and ppn but no luck. I get
>>>>>>>>> errors
>>>>>>>>> like:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [c0301b10e1:03172] [[0,9999],1] -> [[0,0],0] (node: c0301b10e1)
>>>>>>>>> oob-tcp: Number of attempts to create TCP connection has been
>>>>>>>>> exceeded. Can not communicate with peer
>>>>>>>>> [c0301b10e1:03172] [[0,9999],1] ORTE_ERROR_LOG: Unreachable in file
>>>>>>>>> grpcomm_hier_module.c at line 303
>>>>>>>>> [c0301b10e1:03172] [[0,9999],1] ORTE_ERROR_LOG: Unreachable in file
>>>>>>>>> base/grpcomm_base_modex.c at line 470
>>>>>>>>> [c0301b10e1:03172] [[0,9999],1] ORTE_ERROR_LOG: Unreachable in file
>>>>>>>>> grpcomm_hier_module.c at line 484
>>>>>>>>>
>>>>>>>>> --------------------------------------------------------------------------
>>>>>>>>> It looks like MPI_INIT failed for some reason; your parallel process
>>>>>>>>> is
>>>>>>>>> likely to abort. There are many reasons that a parallel process can
>>>>>>>>> fail during MPI_INIT; some of which are due to configuration or
>>>>>>>>> environment
>>>>>>>>> problems. This failure appears to be an internal failure; here's some
>>>>>>>>> additional information (which may only be relevant to an Open MPI
>>>>>>>>> developer):
>>>>>>>>>
>>>>>>>>> orte_grpcomm_modex failed
>>>>>>>>> --> Returned "Unreachable" (-12) instead of "Success" (0)
>>>>>>>>>
>>>>>>>>> --------------------------------------------------------------------------
>>>>>>>>> *** The MPI_Init() function was called before MPI_INIT was invoked.
>>>>>>>>> *** This is disallowed by the MPI standard.
>>>>>>>>> *** Your MPI job will now abort.
>>>>>>>>> [c0301b10e1:3172] Abort before MPI_INIT completed successfully; not
>>>>>>>>> able to guarantee that all other processes were killed!
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> maybe a related question is how to assign the TCP port range and how
>>>>>>>>> is it used? when the processes are on different machines, I use the
>>>>>>>>> same range and that's ok as long as the range is free. but when the
>>>>>>>>> processes are on the same node, what value should the range be for
>>>>>>>>> each process? My range is 10000-12000 (for both processes) and I see
>>>>>>>>> that process with rank #0 listen on port 10001 while process with rank
>>>>>>>>> #1 try to establish a connect to port 10000.
>>>>>>>>>
>>>>>>>>> Thanks so much!
>>>>>>>>> p. still here... still trying... ;-)
>>>>>>>>>
>>>>>>>>> On Tue, Jul 27, 2010 at 12:58 AM, Ralph Castain <r...@open-mpi.org>
>>>>>>>>> wrote:
>>>>>>>>>> Use what hostname returns - don't worry about IP addresses as we'll
>>>>>>>>>> discover them.
>>>>>>>>>>
>>>>>>>>>> On Jul 26, 2010, at 10:45 PM, Philippe wrote:
>>>>>>>>>>
>>>>>>>>>>> Thanks a lot!
>>>>>>>>>>>
>>>>>>>>>>> now, for the ev "OMPI_MCA_orte_nodes", what do I put exactly? our
>>>>>>>>>>> nodes have a short/long name (it's rhel 5.x, so the command hostname
>>>>>>>>>>> returns the long name) and at least 2 IP addresses.
>>>>>>>>>>>
>>>>>>>>>>> p.
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Jul 27, 2010 at 12:06 AM, Ralph Castain <r...@open-mpi.org>
>>>>>>>>>>> wrote:
>>>>>>>>>>>> Okay, fixed in r23499. Thanks again...
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Jul 26, 2010, at 9:47 PM, Ralph Castain wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Doh - yes it should! I'll fix it right now.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Jul 26, 2010, at 9:28 PM, Philippe wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Ralph,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> i was able to test the generic module and it seems to be working.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> one question tho, the function orte_ess_generic_component_query
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>> "orte/mca/ess/generic/ess_generic_component.c" calls getenv with
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> argument "OMPI_MCA_enc", which seems to cause the module to fail
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> load. shouldnt it be "OMPI_MCA_ess" ?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> .....
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> /* only pick us if directed to do so */
>>>>>>>>>>>>>> if (NULL != (pick = getenv("OMPI_MCA_env")) &&
>>>>>>>>>>>>>> 0 == strcmp(pick, "generic")) {
>>>>>>>>>>>>>> *priority = 1000;
>>>>>>>>>>>>>> *module = (mca_base_module_t *)&orte_ess_generic_module;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> p.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Thu, Jul 22, 2010 at 5:53 PM, Ralph Castain
>>>>>>>>>>>>>> <r...@open-mpi.org>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> Dev trunk looks okay right now - I think you'll be fine using
>>>>>>>>>>>>>>> it.
>>>>>>>>>>>>>>> My new component -might- work with 1.5, but probably not with
>>>>>>>>>>>>>>> 1.4. I haven't
>>>>>>>>>>>>>>> checked either of them.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Anything at r23478 or above will have the new module. Let me
>>>>>>>>>>>>>>> know
>>>>>>>>>>>>>>> how it works for you. I haven't tested it myself, but am pretty
>>>>>>>>>>>>>>> sure it
>>>>>>>>>>>>>>> should work.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Jul 22, 2010, at 3:22 PM, Philippe wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Ralph,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thank you so much!!
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'll give it a try and let you know.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I know it's a tough question, but how stable is the dev trunk?
>>>>>>>>>>>>>>>> Can
>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>> just grab the latest and run, or am I better off taking your
>>>>>>>>>>>>>>>> changes
>>>>>>>>>>>>>>>> and copy them back in a stable release? (if so, which one? 1.4?
>>>>>>>>>>>>>>>> 1.5?)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> p.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Thu, Jul 22, 2010 at 3:50 PM, Ralph Castain
>>>>>>>>>>>>>>>> <r...@open-mpi.org>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> It was easier for me to just construct this module than to
>>>>>>>>>>>>>>>>> explain how to do so :-)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I will commit it this evening (couple of hours from now) as
>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>> is our standard practice. You'll need to use the developer's
>>>>>>>>>>>>>>>>> trunk, though,
>>>>>>>>>>>>>>>>> to use it.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Here are the envars you'll need to provide:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Each process needs to get the same following values:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> * OMPI_MCA_ess=generic
>>>>>>>>>>>>>>>>> * OMPI_MCA_orte_num_procs=<number of MPI procs>
>>>>>>>>>>>>>>>>> * OMPI_MCA_orte_nodes=<a comma-separated list of nodenames
>>>>>>>>>>>>>>>>> where
>>>>>>>>>>>>>>>>> MPI procs reside>
>>>>>>>>>>>>>>>>> * OMPI_MCA_orte_ppn=<number of procs/node>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Note that I have assumed this last value is a constant for
>>>>>>>>>>>>>>>>> simplicity. If that isn't the case, let me know - you could
>>>>>>>>>>>>>>>>> instead provide
>>>>>>>>>>>>>>>>> it as a comma-separated list of values with an entry for each
>>>>>>>>>>>>>>>>> node.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> In addition, you need to provide the following value that will
>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>> unique to each process:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> * OMPI_MCA_orte_rank=<MPI rank>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Finally, you have to provide a range of static TCP ports for
>>>>>>>>>>>>>>>>> use
>>>>>>>>>>>>>>>>> by the processes. Pick any range that you know will be
>>>>>>>>>>>>>>>>> available across all
>>>>>>>>>>>>>>>>> the nodes. You then need to ensure that each process sees the
>>>>>>>>>>>>>>>>> following
>>>>>>>>>>>>>>>>> envar:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> * OMPI_MCA_oob_tcp_static_ports=6000-6010 <== obviously,
>>>>>>>>>>>>>>>>> replace
>>>>>>>>>>>>>>>>> this with your range
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> You will need a port range that is at least equal to the ppn
>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>> the job (each proc on a node will take one of the provided
>>>>>>>>>>>>>>>>> ports).
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> That should do it. I compute everything else I need from those
>>>>>>>>>>>>>>>>> values.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Does that work for you?
>>>>>>>>>>>>>>>>> Ralph
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> users mailing list
>>>>>>>>>>> us...@open-mpi.org
>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> users mailing list
>>>>>>>>>> us...@open-mpi.org
>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> users mailing list
>>>>>>>>> us...@open-mpi.org
>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> users mailing list
>>>>>>>> us...@open-mpi.org
>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> users mailing list
>>>>>>> us...@open-mpi.org
>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>
>>>>>> _______________________________________________
>>>>>> users mailing list
>>>>>> us...@open-mpi.org
>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> users mailing list
>>>>> us...@open-mpi.org
>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>
>>>>
>>>> _______________________________________________
>>>> users mailing list
>>>> us...@open-mpi.org
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>
>>>
>>> _______________________________________________
>>> users mailing list
>>> us...@open-mpi.org
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>>
>> _______________________________________________
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>
> _______________________________________________
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
