On Nov 21, 2005, at 4:44 PM, Enrique Curchitser wrote:
OK, I'll take you up on the offer. I have 4 Power Mac G5's on a
private network connected through a GigE switch. Even for large
problems
the communications are slugish. This same code has shown to scale
to upwards of 128 processors on IBM SP's. So here is the output
to ompi_info --param btl tcp. Any help tunning this, or other
parameters would be greatly appreciated. Are there any particular
plugins for TCP?
The TCP BTL is really the only plugin for MPI TCP support in Open MPI.
MCA btl: parameter "btl_base_debug" (current value: "0")
If btl_base_debug is 1 standard debug is
output, if > 1 verbose debug is output
MCA btl: parameter "btl" (current value: <none>)
Default selection set of components for
the btl framework (<none> means "use all components that can be found")
MCA btl: parameter "btl_base_verbose" (current
value: "0")
Verbosity level for the btl framework (0 =
no verbosity)
These are pretty self-explanitory; they won't help with performance
tuning.
MCA btl: parameter "btl_tcp_if_include" (current
value: "en1")
MCA btl: parameter "btl_tcp_if_exclude" (current
value: "lo")
These will help you include and exclude various TCP interfaces on your
machine. So they're not really for tuning so much as selecting which
network you want to run over. They're pretty much only useful on
machines with multiple TCP networks.
MCA btl: parameter "btl_tcp_free_list_num" (current
value: "8")
MCA btl: parameter "btl_tcp_free_list_max" (current
value: "-1")
MCA btl: parameter "btl_tcp_free_list_inc" (current
value: "32")
This is a free list of buffers for copying non-contiguous user data
into on a send. That is, if you have an MPI_Datatype that represents
non-contiguous data, we'll first copy it into buffers from this
freelist and then send from there. We start with 8 initial buffers and
grow the list 32 buffers at a time (-1 mean no max number of buffers).
This probably shouldn't have too much to do with performance.
MCA btl: parameter "btl_tcp_sndbuf" (current value:
"131072")
MCA btl: parameter "btl_tcp_rcvbuf" (current value:
"131072")
This is the OS socket buffer sizing (i.e., how much internal buffer
space the OS has for each socket, in bytes). I have a dim recollection
that Linux has a max size for this, and if you set it above Linux's max
size, it's silently/implicitly downsized to the max. So increasing the
value arbitrarily doesn't help much.
MCA btl: parameter "btl_tcp_exclusivity" (current
value: "0")
This has to do with the priority of the tcp btl relative to the other
btl's. Right now, it's set to the lowest value so that tcp is always
chosen after all the rest (e.g., GM, IB, and shared memory all have
higher exclusivity values).
MCA btl: parameter "btl_tcp_eager_limit" (current
value: "65536")
This is the largest size that we'll send in the first fragment
(eagerly). If you have a fast network, you might try increasing this.
MCA btl: parameter "btl_tcp_min_send_size" (current
value: "65536")
MCA btl: parameter "btl_tcp_max_send_size" (current
value: "131072")
MCA btl: parameter "btl_tcp_min_rdma_size" (current
value: "131072")
MCA btl: parameter "btl_tcp_max_rdma_size" (current
value: "2147483647")
MCA btl: parameter "btl_tcp_flags" (current value:
"10")
I'm going to defer explanation of these params to Galen. Galen?
MCA btl: parameter "btl_tcp_priority" (current
value: "0")
This parameter is not actually used.
--
{+} Jeff Squyres
{+} The Open MPI Project
{+} http://www.open-mpi.org/
