In which ways can I check the failure of the ethernet connections?

2012/9/3  <users-requ...@open-mpi.org>:
> Send users mailing list submissions to
>         us...@open-mpi.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         http://www.open-mpi.org/mailman/listinfo.cgi/users
> or, via email, send a message with subject or body 'help' to
>         users-requ...@open-mpi.org
>
> You can reach the person managing the list at
>         users-ow...@open-mpi.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of users digest..."
>
>
> Today's Topics:
>
>    1. -hostfile ignored in 1.6.1 / SGE integration broken (Reuti)
>    2. Re: some mpi processes "disappear" on a cluster of        servers
>       (Ralph Castain)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Mon, 3 Sep 2012 23:12:14 +0200
> From: Reuti <re...@staff.uni-marburg.de>
> Subject: [OMPI users] -hostfile ignored in 1.6.1 / SGE integration
>         broken
> To: Open MPI Users <us...@open-mpi.org>
> Message-ID:
>         <b8136f9f-da01-4f04-a9f2-0f72d2b7a...@staff.uni-marburg.de>
> Content-Type: text/plain; charset=us-ascii
>
> Hi all,
>
> I just compiled Open MPI 1.6.1 and before digging any deeper: does anyone 
> else notice, that the command:
>
> $ mpiexec -n 4 -machinefile mymachines ./mpihello
>
> will ignore the argument "-machinefile mymachines" and use the file 
> "openmpi-default-hostfile" instead all the time?
>
> ==
>
> SGE issue
>
> I usually don't install new versions instantly, so I only noticed right now, 
> that in 1.4.5 I get a wrong allocation inside SGE (always one process less 
> than requested with `qsub -pe orted N ...`. This I tried only, as with 1.6.1 
> I get:
>
> --------------------------------------------------------------------------
> There are no nodes allocated to this job.
> --------------------------------------------------------------------------
>
> all the time.
>
> ==
>
> I configured with:
>
> ./configure --prefix=$HOME/local/... --enable-static --disable-shared 
> --with-sge
>
> and adjusted my PATHs accordingly (at least: I hope so).
>
> -- Reuti
>
>
> ------------------------------
>
> Message: 2
> Date: Mon, 3 Sep 2012 14:32:48 -0700
> From: Ralph Castain <r...@open-mpi.org>
> Subject: Re: [OMPI users] some mpi processes "disappear" on a cluster
>         of      servers
> To: Open MPI Users <us...@open-mpi.org>
> Message-ID: <c04139de-10b2-42b0-935d-40b104936...@open-mpi.org>
> Content-Type: text/plain; charset=us-ascii
>
> It looks to me like the network is losing connections - your error messages 
> all state "no route to host", which implies that the network interface 
> dropped out.
>
> On Sep 3, 2012, at 1:39 PM, Andrea Negri <negri.an...@gmail.com> wrote:
>
>> I have asked to my admin and he said that no log messages were present
>> in /var/log, apart my login on the compute node.
>> No killed processes, no full stack errors, the memory is ok, 1GB is
>> used and 2GB are free.
>> Actually I don't know what kind of problem coud be, does someone have
>> ideas? Or at least a suspect?
>>
>> Please, don't let me alone!
>>
>> Sorry for the trouble with the mail
>>
>> 2012/9/1  <users-requ...@open-mpi.org>:
>>> Send users mailing list submissions to
>>>        us...@open-mpi.org
>>>
>>> To subscribe or unsubscribe via the World Wide Web, visit
>>>        http://www.open-mpi.org/mailman/listinfo.cgi/users
>>> or, via email, send a message with subject or body 'help' to
>>>        users-requ...@open-mpi.org
>>>
>>> You can reach the person managing the list at
>>>        users-ow...@open-mpi.org
>>>
>>> When replying, please edit your Subject line so it is more specific
>>> than "Re: Contents of users digest..."
>>>
>>>
>>> Today's Topics:
>>>
>>>   1. Re: some mpi processes "disappear" on a cluster of        servers
>>>      (John Hearns)
>>>   2. Re: users Digest, Vol 2339, Issue 5 (Andrea Negri)
>>>
>>>
>>> ----------------------------------------------------------------------
>>>
>>> Message: 1
>>> Date: Sat, 1 Sep 2012 08:48:56 +0100
>>> From: John Hearns <hear...@googlemail.com>
>>> Subject: Re: [OMPI users] some mpi processes "disappear" on a cluster
>>>        of      servers
>>> To: Open MPI Users <us...@open-mpi.org>
>>> Message-ID:
>>>        <capqne2wo3bgefjiyfums6yquoduhjj75zjoeeyjrdq60fmz...@mail.gmail.com>
>>> Content-Type: text/plain; charset=ISO-8859-1
>>>
>>> Apologies, I have not taken the time to read your comprehensive diagnostics!
>>>
>>> As Gus says, this sounds like a memory problem.
>>> My suspicion would be the kernel Out Of Memory (OOM) killer.
>>> Log into those nodes (or ask your systems manager to do this). Look
>>> closely at /var/log/messages where there will be notifications when
>>> the OOM Killer kicks in and - well - kills large memory processes!
>>> Grep for "killed process" in /var/log/messages
>>>
>>> http://linux-mm.org/OOM_Killer
>>>
>>>
>>> ------------------------------
>>>
>>> Message: 2
>>> Date: Sat, 1 Sep 2012 11:50:59 +0200
>>> From: Andrea Negri <negri.an...@gmail.com>
>>> Subject: Re: [OMPI users] users Digest, Vol 2339, Issue 5
>>> To: us...@open-mpi.org
>>> Message-ID:
>>>        <CAPUxaiQ4RFqSK1kz7fM7U9XRxjQh8N+=98paqm2ikvr7bv-...@mail.gmail.com>
>>> Content-Type: text/plain; charset=ISO-8859-1
>>>
>>> Hi, Gus and John,
>>>
>>> my code (zeusmp2) is a F77 code ported in F95, and the very first time
>>> I have launched it, the processed disappears almost immediately.
>>> I checked the code with valgrind and some unallocated arrays were
>>> passed wrongly to some subroutines.
>>> After having corrected this bug, the code runs for a while and after
>>> occour all the stuff described in my first post.
>>> However, the code still performs a lot of main temporal cycle before
>>> "die" (I don't know if thies piece of information is useful).
>>>
>>> Now I'm going to check the memory usage, (I also have a lot of unused
>>> variables in this pretty large code, maybe I shoud comment them).
>>>
>>> uname -a returns
>>> Linux cloud 2.6.9-42.0.3.ELsmp #1 SMP Thu Oct 5 16:29:37 CDT 2006
>>> x86_64 x86_64 x86_64 GNU/Linux
>>>
>>> ulimit -a returns:
>>> core file size                        (blocks, -c) 0
>>> data seg size                       (kbytes, -d) unlimited
>>> file size                               (blocks, -f) unlimited
>>> pending signals                    (-i) 1024
>>> max locked memory             (kbytes, -l) 32
>>> max memory size                (kbytes, -m) unlimited
>>> open files                           (-n) 1024
>>> pipe size                            (512 bytes, -p) 8
>>> POSIX message queues     (bytes, -q) 819200
>>> stack size                           (kbytes, -s) 10240
>>> cpu time                             (seconds, -t) unlimited
>>> max user processes             (-u) 36864
>>> virtual memory                   (kbytes, -v) unlimited
>>> file locks                            (-x) unlimited
>>>
>>>
>>> I can log on the logins nodes, but unfortunately the command ls
>>> /var/log/messages return
>>> acpid               cron.4              messages.3         secure.4
>>> anaconda.log        cups                messages.4         spooler
>>> anaconda.syslog     dmesg               mpi_uninstall.log  spooler.1
>>> anaconda.xlog       gdm                 ppp                spooler.2
>>> audit               httpd               prelink.log        spooler.3
>>> boot.log            itac_uninstall.log  rpmpkgs            spooler.4
>>> boot.log.1          lastlog             rpmpkgs.1          vbox
>>> boot.log.2          mail                rpmpkgs.2          wtmp
>>> boot.log.3          maillog             rpmpkgs.3          wtmp.1
>>> boot.log.4          maillog.1           rpmpkgs.4          Xorg.0.log
>>> cmkl_install.log    maillog.2           samba              Xorg.0.log.old
>>> cmkl_uninstall.log  maillog.3           scrollkeeper.log   yum.log
>>> cron                maillog.4           secure             yum.log.1
>>> cron.1              messages            secure.1
>>> cron.2              messages.1          secure.2
>>> cron.3              messages.2          secure.3
>>>
>>> so, the log should be in some of these files (I don't have read
>>> permission on these files). I'll contact the admin for that.
>>>
>>> 2012/9/1  <users-requ...@open-mpi.org>:
>>>> Send users mailing list submissions to
>>>>        us...@open-mpi.org
>>>>
>>>> To subscribe or unsubscribe via the World Wide Web, visit
>>>>        http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>> or, via email, send a message with subject or body 'help' to
>>>>        users-requ...@open-mpi.org
>>>>
>>>> You can reach the person managing the list at
>>>>        users-ow...@open-mpi.org
>>>>
>>>> When replying, please edit your Subject line so it is more specific
>>>> than "Re: Contents of users digest..."
>>>>
>>>>
>>>> Today's Topics:
>>>>
>>>>   1. Re: some mpi processes "disappear" on a cluster of        servers
>>>>      (Gus Correa)
>>>>
>>>>
>>>> ----------------------------------------------------------------------
>>>>
>>>> Message: 1
>>>> Date: Fri, 31 Aug 2012 20:11:41 -0400
>>>> From: Gus Correa <g...@ldeo.columbia.edu>
>>>> Subject: Re: [OMPI users] some mpi processes "disappear" on a cluster
>>>>        of      servers
>>>> To: Open MPI Users <us...@open-mpi.org>
>>>> Message-ID: <504152bd.3000...@ldeo.columbia.edu>
>>>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>>>>
>>>> Hi Andrea
>>>>
>>>> I would guess this is a memory problem.
>>>> Do you know how much memory each node has?
>>>> Do you know the memory that
>>>> each MPI process in the CFD code requires?
>>>> If the program starts swapping/paging into disk, because of
>>>> low memory, those interesting things that you described can happen.
>>>>
>>>> I would login to the compute nodes and monitor the
>>>> amount of memory being used with "top" right after the program
>>>> starts to run.  If there is a pattern of which node tends to fail,
>>>> track login to that fail-prone node and monitor it.
>>>>
>>>> Alternatively, if you cluster is running Ganglia,
>>>> you can see the memory use graphically,
>>>> in the Ganglia web page in a web browser.
>>>>
>>>> If your cluster
>>>> doesn't allow direct user logins to compute nodes,
>>>> you can ask the system administrator to do this for you.
>>>>
>>>> It may well be that the code has a memory leak, or that
>>>> it has a memory request spike, which may not be caught by
>>>> OpenMPI.
>>>> [Jeff and Ralph will probably correct me soon for
>>>> saying this, and I know the OpenMPI safeguards against
>>>> process misbehavior are great, but ...]
>>>>
>>>> Anyway, we had several codes here that would make a node go south
>>>> because of either type of memory problem, and subsequently the
>>>> program would die, or the other processes in other nodes would
>>>> continue "running" [i.e. mostly waiting for MPI calls to the
>>>> dead node that would never return] as you described.
>>>>
>>>> If the problem is benign, i.e., if it is just that the
>>>> memory-per-processor is not large enough to run in 10 processors,
>>>> you can get around it by running in, say, 20 processors.
>>>>
>>>> Yet another issue that you may check is the stacksize in the
>>>> compute nodes.  Many codes require a large stacksize, i.e.,
>>>> they create large arrays in subroutines, etc, and
>>>> the default stacksize in standard Linux distributions
>>>> may not be as large as needed.
>>>> We use ulimited stacksize in our compute nodes.
>>>>
>>>> You can ask the system administrator to check this for you,
>>>> and perhaps change it in /etc/security/limits.conf to make it
>>>> unlimited or at least larger than the default.
>>>> The Linux shell command "ulimit -a" [bash] or
>>>> "limit" [tcsh] will tell what the limits are.
>>>>
>>>> I hope this helps,
>>>> Gus Correa
>>>>
>>>> On 08/31/2012 07:15 PM, Andrea Negri wrote:
>>>>> Hi, I have been in trouble for a year.
>>>>>
>>>>> I run a pure MPI (no openMP) Fortran fluid dynamical code on a cluster
>>>>> of server, and I obtain a strange behaviour by running the code on
>>>>> multiple nodes.
>>>>> The cluster is formed by 16 pc (1 pc is a node) with a dual core 
>>>>> processor.
>>>>> Basically, I'm able to run the code from the login node with the command:
>>>>> mpirun  --mca btl_base_verbose 100 --mca backtrace_base_verbose 100
>>>>> --mca memory_base_verbose 100 --mca sysinfo_base_verbose 100  -nolocal
>>>>> -hostfile ./host_file -n 10  ./zeusmp2.x>>  zmp_errors 2>&1
>>>>> by selecting one process per core (i.e. in this case I use 5 nodes)
>>>>>
>>>>> The code starts, and it runs correctely for some time.
>>>>> After that, an entire node (sometimes two) "disappears" and it cannot
>>>>> be reached with the ssh command, which returns: No route to host.
>>>>> Sometimes the node is still reachable, but the two processes that was
>>>>> running on the node are disappears.
>>>>> In addition, on the other nodes, the others processes are still running.
>>>>>
>>>>> If I have a look on the output and error file of mpirun, the following
>>>>> error is present: [btl_tcp_frag.c:215:mca_btl_tcp_frag_recv]
>>>>> mca_btl_tcp_frag_recv: readv failed: No route to host (113)
>>>>>
>>>>> PS: I'm not the admin of the cluster, I've installed the gcc and
>>>>> openmpi on my own because the complier aviable on that machine are 8
>>>>> years old.
>>>>>
>>>>>
>>>>> I post here some information, if you want other info, you have only to
>>>>> tell me which command I have to type on the bash and I will
>>>>> immediately reply.
>>>>>
>>>>>
>>>>> complier: gcc 4.7 (which was also used to compile openmpi)
>>>>> openmpi version: 1.6
>>>>>
>>>>> output of "ompi_info --all" from the node where I launch mpirun (which
>>>>> is also the login node of the cluster)
>>>>>
>>>>>   Package: Open MPI and...@cloud.bo.astro.it Distribution
>>>>>                 Open MPI: 1.6
>>>>>    Open MPI SVN revision: r26429
>>>>>    Open MPI release date: May 10, 2012
>>>>>                 Open RTE: 1.6
>>>>>    Open RTE SVN revision: r26429
>>>>>    Open RTE release date: May 10, 2012
>>>>>                     OPAL: 1.6
>>>>>        OPAL SVN revision: r26429
>>>>>        OPAL release date: May 10, 2012
>>>>>                  MPI API: 2.1
>>>>>             Ident string: 1.6
>>>>>            MCA backtrace: execinfo (MCA v2.0, API v2.0, Component v1.6)
>>>>>               MCA memory: linux (MCA v2.0, API v2.0, Component v1.6)
>>>>>            MCA paffinity: hwloc (MCA v2.0, API v2.0, Component v1.6)
>>>>>                MCA carto: auto_detect (MCA v2.0, API v2.0, Component v1.6)
>>>>>                MCA carto: file (MCA v2.0, API v2.0, Component v1.6)
>>>>>                MCA shmem: mmap (MCA v2.0, API v2.0, Component v1.6)
>>>>>                MCA shmem: posix (MCA v2.0, API v2.0, Component v1.6)
>>>>>                MCA shmem: sysv (MCA v2.0, API v2.0, Component v1.6)
>>>>>            MCA maffinity: first_use (MCA v2.0, API v2.0, Component v1.6)
>>>>>            MCA maffinity: hwloc (MCA v2.0, API v2.0, Component v1.6)
>>>>>                MCA timer: linux (MCA v2.0, API v2.0, Component v1.6)
>>>>>          MCA installdirs: env (MCA v2.0, API v2.0, Component v1.6)
>>>>>          MCA installdirs: config (MCA v2.0, API v2.0, Component v1.6)
>>>>>              MCA sysinfo: linux (MCA v2.0, API v2.0, Component v1.6)
>>>>>                MCA hwloc: hwloc132 (MCA v2.0, API v2.0, Component v1.6)
>>>>>                  MCA dpm: orte (MCA v2.0, API v2.0, Component v1.6)
>>>>>               MCA pubsub: orte (MCA v2.0, API v2.0, Component v1.6)
>>>>>            MCA allocator: basic (MCA v2.0, API v2.0, Component v1.6)
>>>>>            MCA allocator: bucket (MCA v2.0, API v2.0, Component v1.6)
>>>>>                 MCA coll: basic (MCA v2.0, API v2.0, Component v1.6)
>>>>>                 MCA coll: hierarch (MCA v2.0, API v2.0, Component v1.6)
>>>>>                 MCA coll: inter (MCA v2.0, API v2.0, Component v1.6)
>>>>>                 MCA coll: self (MCA v2.0, API v2.0, Component v1.6)
>>>>>                 MCA coll: sm (MCA v2.0, API v2.0, Component v1.6)
>>>>>                 MCA coll: sync (MCA v2.0, API v2.0, Component v1.6)
>>>>>                 MCA coll: tuned (MCA v2.0, API v2.0, Component v1.6)
>>>>>                   MCA io: romio (MCA v2.0, API v2.0, Component v1.6)
>>>>>                MCA mpool: fake (MCA v2.0, API v2.0, Component v1.6)
>>>>>                MCA mpool: rdma (MCA v2.0, API v2.0, Component v1.6)
>>>>>                MCA mpool: sm (MCA v2.0, API v2.0, Component v1.6)
>>>>>                  MCA pml: bfo (MCA v2.0, API v2.0, Component v1.6)
>>>>>                  MCA pml: csum (MCA v2.0, API v2.0, Component v1.6)
>>>>>                  MCA pml: ob1 (MCA v2.0, API v2.0, Component v1.6)
>>>>>                  MCA pml: v (MCA v2.0, API v2.0, Component v1.6)
>>>>>                  MCA bml: r2 (MCA v2.0, API v2.0, Component v1.6)
>>>>>               MCA rcache: vma (MCA v2.0, API v2.0, Component v1.6)
>>>>>                  MCA btl: self (MCA v2.0, API v2.0, Component v1.6)
>>>>>                  MCA btl: sm (MCA v2.0, API v2.0, Component v1.6)
>>>>>                  MCA btl: tcp (MCA v2.0, API v2.0, Component v1.6)
>>>>>                 MCA topo: unity (MCA v2.0, API v2.0, Component v1.6)
>>>>>                  MCA osc: pt2pt (MCA v2.0, API v2.0, Component v1.6)
>>>>>                  MCA osc: rdma (MCA v2.0, API v2.0, Component v1.6)
>>>>>                  MCA iof: hnp (MCA v2.0, API v2.0, Component v1.6)
>>>>>                  MCA iof: orted (MCA v2.0, API v2.0, Component v1.6)
>>>>>                  MCA iof: tool (MCA v2.0, API v2.0, Component v1.6)
>>>>>                  MCA oob: tcp (MCA v2.0, API v2.0, Component v1.6)
>>>>>                 MCA odls: default (MCA v2.0, API v2.0, Component v1.6)
>>>>>                  MCA ras: cm (MCA v2.0, API v2.0, Component v1.6)
>>>>>                  MCA ras: loadleveler (MCA v2.0, API v2.0, Component v1.6)
>>>>>                  MCA ras: slurm (MCA v2.0, API v2.0, Component v1.6)
>>>>>                MCA rmaps: load_balance (MCA v2.0, API v2.0, Component 
>>>>> v1.6)
>>>>>                MCA rmaps: rank_file (MCA v2.0, API v2.0, Component v1.6)
>>>>>                MCA rmaps: resilient (MCA v2.0, API v2.0, Component v1.6)
>>>>>                MCA rmaps: round_robin (MCA v2.0, API v2.0, Component v1.6)
>>>>>                MCA rmaps: seq (MCA v2.0, API v2.0, Component v1.6)
>>>>>                MCA rmaps: topo (MCA v2.0, API v2.0, Component v1.6)
>>>>>                  MCA rml: oob (MCA v2.0, API v2.0, Component v1.6)
>>>>>               MCA routed: binomial (MCA v2.0, API v2.0, Component v1.6)
>>>>>               MCA routed: cm (MCA v2.0, API v2.0, Component v1.6)
>>>>>               MCA routed: direct (MCA v2.0, API v2.0, Component v1.6)
>>>>>               MCA routed: linear (MCA v2.0, API v2.0, Component v1.6)
>>>>>               MCA routed: radix (MCA v2.0, API v2.0, Component v1.6)
>>>>>               MCA routed: slave (MCA v2.0, API v2.0, Component v1.6)
>>>>>                  MCA plm: rsh (MCA v2.0, API v2.0, Component v1.6)
>>>>>                  MCA plm: slurm (MCA v2.0, API v2.0, Component v1.6)
>>>>>                MCA filem: rsh (MCA v2.0, API v2.0, Component v1.6)
>>>>>               MCA errmgr: default (MCA v2.0, API v2.0, Component v1.6)
>>>>>                  MCA ess: env (MCA v2.0, API v2.0, Component v1.6)
>>>>>                  MCA ess: hnp (MCA v2.0, API v2.0, Component v1.6)
>>>>>                  MCA ess: singleton (MCA v2.0, API v2.0, Component v1.6)
>>>>>                  MCA ess: slave (MCA v2.0, API v2.0, Component v1.6)
>>>>>                  MCA ess: slurm (MCA v2.0, API v2.0, Component v1.6)
>>>>>                  MCA ess: slurmd (MCA v2.0, API v2.0, Component v1.6)
>>>>>                  MCA ess: tool (MCA v2.0, API v2.0, Component v1.6)
>>>>>              MCA grpcomm: bad (MCA v2.0, API v2.0, Component v1.6)
>>>>>              MCA grpcomm: basic (MCA v2.0, API v2.0, Component v1.6)
>>>>>              MCA grpcomm: hier (MCA v2.0, API v2.0, Component v1.6)
>>>>>             MCA notifier: command (MCA v2.0, API v1.0, Component v1.6)
>>>>>             MCA notifier: syslog (MCA v2.0, API v1.0, Component v1.6)
>>>>>                   Prefix: /home/andrea/library/openmpi/openmpi-1.6-gnu-4.7
>>>>>              Exec_prefix: /home/andrea/library/openmpi/openmpi-1.6-gnu-4.7
>>>>>                   Bindir: 
>>>>> /home/andrea/library/openmpi/openmpi-1.6-gnu-4.7/bin
>>>>>                  Sbindir: 
>>>>> /home/andrea/library/openmpi/openmpi-1.6-gnu-4.7/sbin
>>>>>                   Libdir: 
>>>>> /home/andrea/library/openmpi/openmpi-1.6-gnu-4.7/lib
>>>>>                   Incdir:
>>>>> /home/andrea/library/openmpi/openmpi-1.6-gnu-4.7/include
>>>>>                   Mandir:
>>>>> /home/andrea/library/openmpi/openmpi-1.6-gnu-4.7/share/man
>>>>>                Pkglibdir:
>>>>> /home/andrea/library/openmpi/openmpi-1.6-gnu-4.7/lib/openmpi
>>>>>               Libexecdir:
>>>>> /home/andrea/library/openmpi/openmpi-1.6-gnu-4.7/libexec
>>>>>              Datarootdir: 
>>>>> /home/andrea/library/openmpi/openmpi-1.6-gnu-4.7/share
>>>>>                  Datadir: 
>>>>> /home/andrea/library/openmpi/openmpi-1.6-gnu-4.7/share
>>>>>               Sysconfdir: 
>>>>> /home/andrea/library/openmpi/openmpi-1.6-gnu-4.7/etc
>>>>>           Sharedstatedir: 
>>>>> /home/andrea/library/openmpi/openmpi-1.6-gnu-4.7/com
>>>>>            Localstatedir: 
>>>>> /home/andrea/library/openmpi/openmpi-1.6-gnu-4.7/var
>>>>>                  Infodir:
>>>>> /home/andrea/library/openmpi/openmpi-1.6-gnu-4.7/share/info
>>>>>               Pkgdatadir:
>>>>> /home/andrea/library/openmpi/openmpi-1.6-gnu-4.7/share/openmpi
>>>>>                Pkglibdir:
>>>>> /home/andrea/library/openmpi/openmpi-1.6-gnu-4.7/lib/openmpi
>>>>>            Pkgincludedir:
>>>>> /home/andrea/library/openmpi/openmpi-1.6-gnu-4.7/include/openmpi
>>>>>  Configured architecture: x86_64-unknown-linux-gnu
>>>>>           Configure host: cloud.bo.astro.it
>>>>>            Configured by: andrea
>>>>>            Configured on: Tue Jul 31 10:53:46 CEST 2012
>>>>>           Configure host: cloud.bo.astro.it
>>>>>                 Built by: andrea
>>>>>                 Built on: Tue Jul 31 11:08:33 CEST 2012
>>>>>               Built host: cloud.bo.astro.it
>>>>>               C bindings: yes
>>>>>             C++ bindings: yes
>>>>>       Fortran77 bindings: yes (all)
>>>>>       Fortran90 bindings: yes
>>>>>  Fortran90 bindings size: medium
>>>>>               C compiler: /home/andrea/library/gcc/gcc-objects/bin/gcc
>>>>>      C compiler absolute:
>>>>>   C compiler family name: GNU
>>>>>       C compiler version: 4.7.1
>>>>>              C char size: 1
>>>>>              C bool size: 1
>>>>>             C short size: 2
>>>>>               C int size: 4
>>>>>              C long size: 8
>>>>>             C float size: 4
>>>>>            C double size: 8
>>>>>           C pointer size: 8
>>>>>             C char align: 1
>>>>>             C bool align: 1
>>>>>              C int align: 4
>>>>>            C float align: 4
>>>>>           C double align: 8
>>>>>             C++ compiler: /home/andrea/library/gcc/gcc-objects/bin/g++
>>>>>    C++ compiler absolute: none
>>>>>       Fortran77 compiler: 
>>>>> /home/andrea/library/gcc/gcc-objects/bin/gfortran
>>>>>   Fortran77 compiler abs:
>>>>>       Fortran90 compiler: 
>>>>> /home/andrea/library/gcc/gcc-objects/bin/gfortran
>>>>>   Fortran90 compiler abs:
>>>>>        Fort integer size: 4
>>>>>        Fort logical size: 4
>>>>>  Fort logical value true: 1
>>>>>       Fort have integer1: yes
>>>>>       Fort have integer2: yes
>>>>>       Fort have integer4: yes
>>>>>       Fort have integer8: yes
>>>>>      Fort have integer16: no
>>>>>          Fort have real4: yes
>>>>>          Fort have real8: yes
>>>>>         Fort have real16: no
>>>>>       Fort have complex8: yes
>>>>>      Fort have complex16: yes
>>>>>      Fort have complex32: no
>>>>>       Fort integer1 size: 1
>>>>>       Fort integer2 size: 2
>>>>>       Fort integer4 size: 4
>>>>>       Fort integer8 size: 8
>>>>>      Fort integer16 size: -1
>>>>>           Fort real size: 4
>>>>>          Fort real4 size: 4
>>>>>          Fort real8 size: 8
>>>>>         Fort real16 size: 16
>>>>>       Fort dbl prec size: 8
>>>>>           Fort cplx size: 8
>>>>>       Fort dbl cplx size: 16
>>>>>          Fort cplx8 size: 8
>>>>>         Fort cplx16 size: 16
>>>>>         Fort cplx32 size: 32
>>>>>       Fort integer align: 4
>>>>>      Fort integer1 align: 1
>>>>>      Fort integer2 align: 2
>>>>>      Fort integer4 align: 4
>>>>>      Fort integer8 align: 8
>>>>>     Fort integer16 align: -1
>>>>>          Fort real align: 4
>>>>>         Fort real4 align: 4
>>>>>         Fort real8 align: 8
>>>>>        Fort real16 align: 16
>>>>>      Fort dbl prec align: 8
>>>>>          Fort cplx align: 4
>>>>>      Fort dbl cplx align: 8
>>>>>         Fort cplx8 align: 4
>>>>>        Fort cplx16 align: 8
>>>>>        Fort cplx32 align: 16
>>>>>              C profiling: yes
>>>>>            C++ profiling: yes
>>>>>      Fortran77 profiling: yes
>>>>>      Fortran90 profiling: yes
>>>>>           C++ exceptions: no
>>>>>           Thread support: posix (MPI_THREAD_MULTIPLE: no, progress: no)
>>>>>            Sparse Groups: no
>>>>>             Build CFLAGS: -DNDEBUG -g -O2 -finline-functions
>>>>> -fno-strict-aliasing
>>>>>                           -pthread
>>>>>           Build CXXFLAGS: -O3 -DNDEBUG -finline-functions -pthread
>>>>>             Build FFLAGS:
>>>>>            Build FCFLAGS:
>>>>>            Build LDFLAGS: -Wl,--rpath
>>>>> -Wl,/home/andrea/library/gcc/gcc-objects/lib64
>>>>>               Build LIBS: -lrt -lnsl  -lutil -lm
>>>>>     Wrapper extra CFLAGS: -pthread
>>>>>   Wrapper extra CXXFLAGS: -pthread
>>>>>     Wrapper extra FFLAGS: -pthread
>>>>>    Wrapper extra FCFLAGS: -pthread
>>>>>    Wrapper extra LDFLAGS:
>>>>>       Wrapper extra LIBS: -ldl  -lm -lnuma -lrt -lnsl  -lutil -lm
>>>>>   Internal debug support: no
>>>>>   MPI interface warnings: yes
>>>>>      MPI parameter check: runtime
>>>>> Memory profiling support: no
>>>>> Memory debugging support: no
>>>>>          libltdl support: no
>>>>>    Heterogeneous support: no
>>>>>  mpirun default --prefix: yes
>>>>>          MPI I/O support: yes
>>>>>        MPI_WTIME support: gettimeofday
>>>>>      Symbol vis. support: yes
>>>>>    Host topology support: yes
>>>>>           MPI extensions: affinity example
>>>>>    FT Checkpoint support: no (checkpoint thread: no)
>>>>>      VampirTrace support: yes
>>>>>   MPI_MAX_PROCESSOR_NAME: 256
>>>>>     MPI_MAX_ERROR_STRING: 256
>>>>>      MPI_MAX_OBJECT_NAME: 64
>>>>>         MPI_MAX_INFO_KEY: 36
>>>>>         MPI_MAX_INFO_VAL: 256
>>>>>        MPI_MAX_PORT_NAME: 1024
>>>>>   MPI_MAX_DATAREP_STRING: 128
>>>>>                  MCA mca: parameter "mca_param_files" (current value:
>>>>>
>>>>> </home/andrea/.openmpi/mca-params.conf:/home/andrea/library/openmpi/openmpi-1.6-gnu-4.7/etc/openmpi-mca-params.conf>,
>>>>>                           data source: default value)
>>>>>                           Path for MCA configuration files containing
>>>>> default parameter
>>>>>                           values
>>>>>                  MCA mca: parameter "mca_base_param_file_prefix"
>>>>> (current value:<none>,
>>>>>                           data source: default value)
>>>>>                           Aggregate MCA parameter file sets
>>>>>                  MCA mca: parameter "mca_base_param_file_path" (current 
>>>>> value:
>>>>>
>>>>> </home/andrea/library/openmpi/openmpi-1.6-gnu-4.7/share/openmpi/amca-param-sets:/home/andrea/library/openmpi/openmpi-1.6-gnu-4.7/bin>,
>>>>>                           data source: default value)
>>>>>                           Aggregate MCA parameter Search path
>>>>>                  MCA mca: parameter "mca_base_param_file_path_force"
>>>>> (current value:
>>>>>                           <none>, data source: default value)
>>>>>                           Forced Aggregate MCA parameter Search path
>>>>>                  MCA mca: parameter "mca_component_path" (current value:
>>>>>
>>>>> </home/andrea/library/openmpi/openmpi-1.6-gnu-4.7/lib/openmpi:/home/andrea/.openmpi/components>,
>>>>>                           data source: default value)
>>>>>                           Path where to look for Open MPI and ORTE 
>>>>> components
>>>>>                  MCA mca: parameter "mca_component_show_load_errors"
>>>>> (current value:<1>,
>>>>>                           data source: default value)
>>>>>                           Whether to show errors for components that
>>>>> failed to load or
>>>>>                           not
>>>>>                  MCA mca: parameter "mca_component_disable_dlopen"
>>>>> (current value:<0>,
>>>>>                           data source: default value)
>>>>>                           Whether to attempt to disable opening
>>>>> dynamic components or not
>>>>>                  MCA mca: parameter "mca_verbose" (current value:
>>>>> <stderr>, data source:
>>>>>                           default value)
>>>>>                           Specifies where the default error output
>>>>> stream goes (this is
>>>>>                           separate from distinct help messages).  Accepts 
>>>>> a
>>>>>                           comma-delimited list of: stderr, stdout, syslog,
>>>>>                           syslogpri:<notice|info|debug>,
>>>>> syslogid:<str>  (where str is the
>>>>>                           prefix string for all syslog notices),
>>>>> file[:filename] (if
>>>>>                           filename is not specified, a default
>>>>> filename is used),
>>>>>                           fileappend (if not specified, the file is 
>>>>> opened for
>>>>>                           truncation), level[:N] (if specified,
>>>>> integer verbose level;
>>>>>                           otherwise, 0 is implied)
>>>>>                  MCA mpi: parameter "mpi_paffinity_alone" (current
>>>>> value:<0>, data
>>>>>                           source: default value, synonym of:
>>>>> opal_paffinity_alone)
>>>>>                           If nonzero, assume that this job is the only 
>>>>> (set of)
>>>>>                           process(es) running on each node and bind 
>>>>> processes to
>>>>>                           processors, starting with processor ID 0
>>>>>                  MCA mpi: parameter "mpi_param_check" (current value:
>>>>> <1>, data source:
>>>>>                           default value)
>>>>>                           Whether you want MPI API parameters checked
>>>>> at run-time or not.
>>>>>                            Possible values are 0 (no checking) and 1
>>>>> (perform checking at
>>>>>                           run-time)
>>>>>                  MCA mpi: parameter "mpi_yield_when_idle" (current
>>>>> value:<-1>, data
>>>>>                           source: default value)
>>>>>                           Yield the processor when waiting for MPI
>>>>> communication (for MPI
>>>>>                           processes, will default to 1 when
>>>>> oversubscribing nodes)
>>>>>                  MCA mpi: parameter "mpi_event_tick_rate" (current
>>>>> value:<-1>, data
>>>>>                           source: default value)
>>>>>                           How often to progress TCP communications (0
>>>>> = never, otherwise
>>>>>                           specified in microseconds)
>>>>>                  MCA mpi: parameter "mpi_show_handle_leaks" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           Whether MPI_FINALIZE shows all MPI handles
>>>>> that were not freed
>>>>>                           or not
>>>>>                  MCA mpi: parameter "mpi_no_free_handles" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           Whether to actually free MPI objects when
>>>>> their handles are
>>>>>                           freed
>>>>>                  MCA mpi: parameter "mpi_show_mpi_alloc_mem_leaks"
>>>>> (current value:<0>,
>>>>>                           data source: default value)
>>>>>                           If>0, MPI_FINALIZE will show up to this
>>>>> many instances of
>>>>>                           memory allocated by MPI_ALLOC_MEM that was
>>>>> not freed by
>>>>>                           MPI_FREE_MEM
>>>>>                  MCA mpi: parameter "mpi_show_mca_params" (current
>>>>> value:<none>, data
>>>>>                           source: default value)
>>>>>                           Whether to show all MCA parameter values
>>>>> during MPI_INIT or not
>>>>>                           (good for reproducability of MPI jobs for
>>>>> debug purposes).
>>>>>                           Accepted values are all, default, file, api,
>>>>> and enviro - or a
>>>>>                           comma delimited combination of them
>>>>>                  MCA mpi: parameter "mpi_show_mca_params_file"
>>>>> (current value:<none>,
>>>>>                           data source: default value)
>>>>>                           If mpi_show_mca_params is true, setting this
>>>>> string to a valid
>>>>>                           filename tells Open MPI to dump all the MCA
>>>>> parameter values
>>>>>                           into a file suitable for reading via the
>>>>> mca_param_files
>>>>>                           parameter (good for reproducability of MPI jobs)
>>>>>                  MCA mpi: parameter "mpi_keep_peer_hostnames" (current
>>>>> value:<1>, data
>>>>>                           source: default value)
>>>>>                           If nonzero, save the string hostnames of all
>>>>> MPI peer processes
>>>>>                           (mostly for error / debugging output
>>>>> messages).  This can add
>>>>>                           quite a bit of memory usage to each MPI process.
>>>>>                  MCA mpi: parameter "mpi_abort_delay" (current value:
>>>>> <0>, data source:
>>>>>                           default value)
>>>>>                           If nonzero, print out an identifying message
>>>>> when MPI_ABORT is
>>>>>                           invoked (hostname, PID of the process that
>>>>> called MPI_ABORT)
>>>>>                           and delay for that many seconds before
>>>>> exiting (a negative
>>>>>                           delay value means to never abort).  This
>>>>> allows attaching of a
>>>>>                           debugger before quitting the job.
>>>>>                  MCA mpi: parameter "mpi_abort_print_stack" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           If nonzero, print out a stack trace when
>>>>> MPI_ABORT is invoked
>>>>>                  MCA mpi: parameter "mpi_preconnect_mpi" (current
>>>>> value:<0>, data
>>>>>                           source: default value, synonyms: 
>>>>> mpi_preconnect_all)
>>>>>                           Whether to force MPI processes to fully
>>>>> wire-up the MPI
>>>>>                           connections between MPI processes during
>>>>> MPI_INIT (vs. making
>>>>>                           connections lazily -- upon the first MPI
>>>>> traffic between each
>>>>>                           process peer pair)
>>>>>                  MCA mpi: parameter "mpi_preconnect_all" (current
>>>>> value:<0>, data
>>>>>                           source: default value, deprecated, synonym of:
>>>>>                           mpi_preconnect_mpi)
>>>>>                           Whether to force MPI processes to fully
>>>>> wire-up the MPI
>>>>>                           connections between MPI processes during
>>>>> MPI_INIT (vs. making
>>>>>                           connections lazily -- upon the first MPI
>>>>> traffic between each
>>>>>                           process peer pair)
>>>>>                  MCA mpi: parameter "mpi_leave_pinned" (current value:
>>>>> <-1>, data source:
>>>>>                           default value)
>>>>>                           Whether to use the "leave pinned" protocol
>>>>> or not.  Enabling
>>>>>                           this setting can help bandwidth performance
>>>>> when repeatedly
>>>>>                           sending and receiving large messages with
>>>>> the same buffers over
>>>>>                           RDMA-based networks (0 = do not use "leave
>>>>> pinned" protocol, 1
>>>>>                           = use "leave pinned" protocol, -1 = allow
>>>>> network to choose at
>>>>>                           runtime).
>>>>>                  MCA mpi: parameter "mpi_leave_pinned_pipeline"
>>>>> (current value:<0>, data
>>>>>                           source: default value)
>>>>>                           Whether to use the "leave pinned pipeline"
>>>>> protocol or not.
>>>>>                  MCA mpi: parameter "mpi_warn_on_fork" (current value:
>>>>> <1>, data source:
>>>>>                           default value)
>>>>>                           If nonzero, issue a warning if program forks
>>>>> under conditions
>>>>>                           that could cause system errors
>>>>>                  MCA mpi: information "mpi_have_sparse_group_storage"
>>>>> (value:<0>, data
>>>>>                           source: default value)
>>>>>                           Whether this Open MPI installation supports
>>>>> storing of data in
>>>>>                           MPI groups in "sparse" formats (good for
>>>>> extremely large
>>>>>                           process count MPI jobs that create many
>>>>> communicators/groups)
>>>>>                  MCA mpi: parameter "mpi_use_sparse_group_storage"
>>>>> (current value:<0>,
>>>>>                           data source: default value)
>>>>>                           Whether to use "sparse" storage formats for
>>>>> MPI groups (only
>>>>>                           relevant if mpi_have_sparse_group_storage is 1)
>>>>>                  MCA mpi: parameter "mpi_notify_init_finalize"
>>>>> (current value:<1>, data
>>>>>                           source: default value)
>>>>>                           If nonzero, send two notifications during
>>>>> MPI_INIT: one near
>>>>>                           when MPI_INIT starts, and another right
>>>>> before MPI_INIT
>>>>>                           finishes, and send 2 notifications during
>>>>> MPI_FINALIZE: one
>>>>>                           right when MPI_FINALIZE starts, and another 
>>>>> near when
>>>>>                           MPI_FINALIZE finishes.
>>>>>                 MCA orte: parameter "orte_base_help_aggregate"
>>>>> (current value:<1>, data
>>>>>                           source: default value)
>>>>>                           If orte_base_help_aggregate is true,
>>>>> duplicate help messages
>>>>>                           will be aggregated rather than displayed
>>>>> individually.  This
>>>>>                           can be helpful for parallel jobs that
>>>>> experience multiple
>>>>>                           identical failures; rather than print out
>>>>> the same help/failure
>>>>>                           message N times, display it once with a
>>>>> count of how many
>>>>>                           processes sent the same message.
>>>>>                 MCA orte: parameter "orte_tmpdir_base" (current value:
>>>>> <none>, data
>>>>>                           source: default value)
>>>>>                           Base of the session directory tree
>>>>>                 MCA orte: parameter "orte_no_session_dirs" (current
>>>>> value:<none>, data
>>>>>                           source: default value)
>>>>>                           Prohibited locations for session directories 
>>>>> (multiple
>>>>>                           locations separated by ',', default=NULL)
>>>>>                 MCA orte: parameter "orte_send_profile" (current
>>>>> value:<0>, data source:
>>>>>                           default value)
>>>>>                           Send profile info in launch message
>>>>>                 MCA orte: parameter "orte_debug" (current value:<0>,
>>>>> data source:
>>>>>                           default value)
>>>>>                           Top-level ORTE debug switch (default verbosity: 
>>>>> 1)
>>>>>                 MCA orte: parameter "orte_debug_verbose" (current
>>>>> value:<-1>, data
>>>>>                           source: default value)
>>>>>                           Verbosity level for ORTE debug messages 
>>>>> (default: 1)
>>>>>                 MCA orte: parameter "orte_debug_daemons" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           Whether to debug the ORTE daemons or not
>>>>>                 MCA orte: parameter "orte_debug_daemons_file" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           Whether want stdout/stderr of daemons to go
>>>>> to a file or not
>>>>>                 MCA orte: parameter "orte_daemon_bootstrap" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           Bootstrap the connection to the HNP
>>>>>                 MCA orte: parameter "orte_leave_session_attached"
>>>>> (current value:<0>,
>>>>>                           data source: default value)
>>>>>                           Whether applications and/or daemons should
>>>>> leave their sessions
>>>>>                           attached so that any output can be received
>>>>> - this allows X
>>>>>                           forwarding without all the attendant debugging 
>>>>> output
>>>>>                 MCA orte: parameter "orte_output_debugger_proctable"
>>>>> (current value:<0>,
>>>>>                           data source: default value)
>>>>>                           Whether or not to output the debugger
>>>>> proctable after launch
>>>>>                           (default: false)
>>>>>                 MCA orte: parameter "orte_debugger_test_daemon"
>>>>> (current value:<none>,
>>>>>                           data source: default value)
>>>>>                           Name of the executable to be used to
>>>>> simulate a debugger
>>>>>                           colaunch (relative or absolute path)
>>>>>                 MCA orte: parameter "orte_debugger_test_attach"
>>>>> (current value:<0>, data
>>>>>                           source: default value)
>>>>>                           Test debugger colaunch after debugger attachment
>>>>>                 MCA orte: parameter "orte_debugger_check_rate"
>>>>> (current value:<0>, data
>>>>>                           source: default value)
>>>>>                           Set rate (in secs) for auto-detect of
>>>>> debugger attachment (0 =>
>>>>>                           do not check)
>>>>>                 MCA orte: parameter "orte_do_not_launch" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           Perform all necessary operations to prepare
>>>>> to launch the
>>>>>                           application, but do not actually launch it
>>>>>                 MCA orte: parameter "orte_daemon_spin" (current value:
>>>>> <0>, data source:
>>>>>                           default value)
>>>>>                           Have any orteds spin until we can connect a
>>>>> debugger to them
>>>>>                 MCA orte: parameter "orte_daemon_fail" (current value:
>>>>> <-1>, data source:
>>>>>                           default value)
>>>>>                           Have the specified orted fail after init for
>>>>> debugging purposes
>>>>>                 MCA orte: parameter "orte_daemon_fail_delay" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           Have the specified orted fail after
>>>>> specified number of seconds
>>>>>                           (default: 0 =>  no delay)
>>>>>                 MCA orte: parameter "orte_heartbeat_rate" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           Seconds between checks for daemon
>>>>> state-of-health (default: 0
>>>>>                           =>  do not check)
>>>>>                 MCA orte: parameter "orte_startup_timeout" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           Milliseconds/daemon to wait for startup
>>>>> before declaring
>>>>>                           failed_to_start (default: 0 =>  do not check)
>>>>>                 MCA orte: parameter "orte_timing" (current value:<0>,
>>>>> data source:
>>>>>                           default value)
>>>>>                           Request that critical timing loops be measured
>>>>>                 MCA orte: parameter "orte_timing_details" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           Request that detailed timing data by reported
>>>>>                 MCA orte: parameter "orte_base_user_debugger" (current
>>>>> value:<totalview
>>>>>                           @mpirun@ -a @mpirun_args@ : ddt -n @np@
>>>>> -start @executable@
>>>>>                           @executable_argv@ @single_app@ : fxp @mpirun@ -a
>>>>>                           @mpirun_args@>, data source: default value)
>>>>>                           Sequence of user-level debuggers to search
>>>>> for in orterun
>>>>>                 MCA orte: parameter "orte_abort_timeout" (current
>>>>> value:<1>, data
>>>>>                           source: default value)
>>>>>                           Max time to wait [in secs] before aborting
>>>>> an ORTE operation
>>>>>                           (default: 1sec)
>>>>>                 MCA orte: parameter "orte_timeout_step" (current
>>>>> value:<1000>, data
>>>>>                           source: default value)
>>>>>                           Time to wait [in usecs/proc] before aborting
>>>>> an ORTE operation
>>>>>                           (default: 1000 usec/proc)
>>>>>                 MCA orte: parameter "orte_default_hostfile" (current 
>>>>> value:
>>>>>
>>>>> </home/andrea/library/openmpi/openmpi-1.6-gnu-4.7/etc/openmpi-default-hostfile>,
>>>>>                           data source: default value)
>>>>>                           Name of the default hostfile (relative or
>>>>> absolute path, "none"
>>>>>                           to ignore environmental or default MCA param 
>>>>> setting)
>>>>>                 MCA orte: parameter "orte_rankfile" (current value:
>>>>> <none>, data source:
>>>>>                           default value, synonyms: rmaps_rank_file_path)
>>>>>                           Name of the rankfile to be used for mapping
>>>>> processes (relative
>>>>>                           or absolute path)
>>>>>                 MCA orte: parameter "orte_keep_fqdn_hostnames"
>>>>> (current value:<0>, data
>>>>>                           source: default value)
>>>>>                           Whether or not to keep FQDN hostnames [default: 
>>>>> no]
>>>>>                 MCA orte: parameter "orte_use_regexp" (current value:
>>>>> <0>, data source:
>>>>>                           default value)
>>>>>                           Whether or not to use regular expressions
>>>>> for launch [default:
>>>>>                           no]
>>>>>                 MCA orte: parameter "orte_tag_output" (current value:
>>>>> <0>, data source:
>>>>>                           default value)
>>>>>                           Tag all output with [job,rank] (default: false)
>>>>>                 MCA orte: parameter "orte_xml_output" (current value:
>>>>> <0>, data source:
>>>>>                           default value)
>>>>>                           Display all output in XML format (default: 
>>>>> false)
>>>>>                 MCA orte: parameter "orte_xml_file" (current value:
>>>>> <none>, data source:
>>>>>                           default value)
>>>>>                           Provide all output in XML format to the 
>>>>> specified file
>>>>>                 MCA orte: parameter "orte_timestamp_output" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           Timestamp all application process output
>>>>> (default: false)
>>>>>                 MCA orte: parameter "orte_output_filename" (current
>>>>> value:<none>, data
>>>>>                           source: default value)
>>>>>                           Redirect output from application processes
>>>>> into filename.rank
>>>>>                           [default: NULL]
>>>>>                 MCA orte: parameter "orte_show_resolved_nodenames"
>>>>> (current value:<0>,
>>>>>                           data source: default value)
>>>>>                           Display any node names that are resolved to
>>>>> a different name
>>>>>                           (default: false)
>>>>>                 MCA orte: parameter "orte_hetero_apps" (current value:
>>>>> <0>, data source:
>>>>>                           default value)
>>>>>                           Indicates that multiple app_contexts are
>>>>> being provided that
>>>>>                           are a mix of 32/64 bit binaries (default: false)
>>>>>                 MCA orte: parameter "orte_launch_agent" (current
>>>>> value:<orted>, data
>>>>>                           source: default value)
>>>>>                           Command used to start processes on remote
>>>>> nodes (default:
>>>>>                           orted)
>>>>>                 MCA orte: parameter "orte_allocation_required"
>>>>> (current value:<0>, data
>>>>>                           source: default value)
>>>>>                           Whether or not an allocation by a resource
>>>>> manager is required
>>>>>                           [default: no]
>>>>>                 MCA orte: parameter "orte_xterm" (current value:
>>>>> <none>, data source:
>>>>>                           default value)
>>>>>                           Create a new xterm window and display output
>>>>> from the specified
>>>>>                           ranks there [default: none]
>>>>>                 MCA orte: parameter "orte_forward_job_control"
>>>>> (current value:<0>, data
>>>>>                           source: default value)
>>>>>                           Forward SIGTSTP (after converting to
>>>>> SIGSTOP) and SIGCONT
>>>>>                           signals to the application procs [default: no]
>>>>>                 MCA orte: parameter "orte_rsh_agent" (current value:
>>>>> <ssh : rsh>, data
>>>>>                           source: default value, synonyms:
>>>>> pls_rsh_agent, plm_rsh_agent)
>>>>>                           The command used to launch executables on 
>>>>> remote nodes
>>>>>                           (typically either "ssh" or "rsh")
>>>>>                 MCA orte: parameter "orte_assume_same_shell" (current
>>>>> value:<1>, data
>>>>>                           source: default value, synonyms:
>>>>> plm_rsh_assume_same_shell)
>>>>>                           If set to 1, assume that the shell on the
>>>>> remote node is the
>>>>>                           same as the shell on the local node.
>>>>> Otherwise, probe for what
>>>>>                           the remote shell [default: 1]
>>>>>                 MCA orte: parameter "orte_report_launch_progress"
>>>>> (current value:<0>,
>>>>>                           data source: default value)
>>>>>                           Output a brief periodic report on launch
>>>>> progress [default: no]
>>>>>                 MCA orte: parameter "orte_num_boards" (current value:
>>>>> <1>, data source:
>>>>>                           default value)
>>>>>                           Number of processor boards/node (1-256) 
>>>>> [default: 1]
>>>>>                 MCA orte: parameter "orte_num_sockets" (current value:
>>>>> <0>, data source:
>>>>>                           default value)
>>>>>                           Number of sockets/board (1-256)
>>>>>                 MCA orte: parameter "orte_num_cores" (current value:
>>>>> <0>, data source:
>>>>>                           default value)
>>>>>                           Number of cores/socket (1-256)
>>>>>                 MCA orte: parameter "orte_cpu_set" (current value:
>>>>> <none>, data source:
>>>>>                           default value)
>>>>>                           Comma-separated list of ranges specifying 
>>>>> logical cpus
>>>>>                           allocated to this job [default: none]
>>>>>                 MCA orte: parameter "orte_process_binding" (current
>>>>> value:<none>, data
>>>>>                           source: default value)
>>>>>                           Policy for binding processes [none | core |
>>>>> socket | board]
>>>>>                           (supported qualifier: if-avail)
>>>>>                 MCA orte: parameter "orte_report_bindings" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           Report bindings
>>>>>                 MCA orte: parameter "orte_report_events" (current
>>>>> value:<none>, data
>>>>>                           source: default value)
>>>>>                           URI to which events are to be reported
>>>>> (default: NULL)]
>>>>>                 MCA opal: parameter "opal_net_private_ipv4" (current 
>>>>> value:
>>>>>
>>>>> <10.0.0.0/8;172.16.0.0/12;192.168.0.0/16;169.254.0.0/16>, data
>>>>>                           source: default value)
>>>>>                           Semicolon-delimited list of CIDR notation
>>>>> entries specifying
>>>>>                           what networks are considered "private"
>>>>> (default value based on
>>>>>                           RFC1918 and RFC3330)
>>>>>                 MCA opal: parameter "opal_signal" (current value:
>>>>> <6,7,8,11>, data
>>>>>                           source: default value)
>>>>>                           Comma-delimited list of integer signal
>>>>> numbers to Open MPI to
>>>>>                           attempt to intercept.  Upon receipt of the
>>>>> intercepted signal,
>>>>>                           Open MPI will display a stack trace and
>>>>> abort.  Open MPI will
>>>>>                           *not* replace signals if handlers are
>>>>> already installed by the
>>>>>                           time MPI_INIT is invoked.  Optionally append
>>>>> ":complain" to any
>>>>>                           signal number in the comma-delimited list to
>>>>> make Open MPI
>>>>>                           complain if it detects another signal
>>>>> handler (and therefore
>>>>>                           does not insert its own).
>>>>>                 MCA opal: parameter "opal_profile" (current value:
>>>>> <0>, data source:
>>>>>                           default value)
>>>>>                           Set to non-zero to profile component selections
>>>>>                 MCA opal: parameter "opal_profile_file" (current
>>>>> value:<none>, data
>>>>>                           source: default value)
>>>>>                           Name of the file containing the cluster 
>>>>> configuration
>>>>>                           information
>>>>>                 MCA opal: parameter "opal_paffinity_alone" (current
>>>>> value:<0>, data
>>>>>                           source: default value, synonyms: 
>>>>> mpi_paffinity_alone)
>>>>>                           If nonzero, assume that this job is the only 
>>>>> (set of)
>>>>>                           process(es) running on each node and bind 
>>>>> processes to
>>>>>                           processors, starting with processor ID 0
>>>>>                 MCA opal: parameter "opal_set_max_sys_limits" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           Set to non-zero to automatically set any
>>>>> system-imposed limits
>>>>>                           to the maximum allowed
>>>>>                 MCA opal: parameter "opal_event_include" (current
>>>>> value:<poll>, data
>>>>>                           source: default value)
>>>>>                           Comma-delimited list of libevent subsystems
>>>>> to use (epoll,
>>>>>                           poll, select -- available on your platform)
>>>>>            MCA backtrace: parameter "backtrace" (current value:
>>>>> <none>, data source:
>>>>>                           default value)
>>>>>                           Default selection set of components for the
>>>>> backtrace framework
>>>>>                           (<none>  means use all components that can be 
>>>>> found)
>>>>>            MCA backtrace: parameter "backtrace_base_verbose" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           Verbosity level for the backtrace framework
>>>>> (0 = no verbosity)
>>>>>            MCA backtrace: parameter "backtrace_execinfo_priority"
>>>>> (current value:<0>,
>>>>>                           data source: default value)
>>>>>           MCA memchecker: parameter "memchecker" (current value:
>>>>> <none>, data source:
>>>>>                           default value)
>>>>>                           Default selection set of components for the 
>>>>> memchecker
>>>>>                           framework (<none>  means use all components
>>>>> that can be found)
>>>>>               MCA memory: parameter "memory" (current value:<none>,
>>>>> data source: default
>>>>>                           value)
>>>>>                           Default selection set of components for the
>>>>> memory framework
>>>>>                           (<none>  means use all components that can be 
>>>>> found)
>>>>>               MCA memory: parameter "memory_base_verbose" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           Verbosity level for the memory framework (0
>>>>> = no verbosity)
>>>>>               MCA memory: information
>>>>> "memory_linux_ptmalloc2_available" (value:<1>,
>>>>>                           data source: default value)
>>>>>                           Whether ptmalloc2 support is included in
>>>>> Open MPI or not (1 =
>>>>>                           yes, 0 = no)
>>>>>               MCA memory: information
>>>>> "memory_linux_ummunotify_available" (value:<0>,
>>>>>                           data source: default value)
>>>>>                           Whether ummunotify support is included in
>>>>> Open MPI or not (1 =
>>>>>                           yes, 0 = no)
>>>>>               MCA memory: parameter "memory_linux_ptmalloc2_enable"
>>>>> (current value:<-1>,
>>>>>                           data source: default value)
>>>>>                           Whether to enable ptmalloc2 support or not
>>>>> (negative = try to
>>>>>                           enable, but continue even if support is not
>>>>> available, 0 = do
>>>>>                           not enable support, positive = try to enable
>>>>> and fail if
>>>>>                           support is not available)
>>>>>               MCA memory: parameter "memory_linux_ummunotify_enable"
>>>>> (current value:
>>>>>                           <-1>, data source: default value)
>>>>>                           Whether to enable ummunotify support or not
>>>>> (negative = try to
>>>>>                           enable, but continue even if support is not
>>>>> available, 0 = do
>>>>>                           not enable support, positive = try to enable
>>>>> and fail if
>>>>>                           support is not available)
>>>>>               MCA memory: parameter "memory_linux_disable" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           If this MCA parameter is set to 1 **VIA
>>>>> ENVIRONMENT VARIABLE
>>>>>                           ONLY*** (this MCA parameter *CANNOT* be set
>>>>> in a file or on the
>>>>>                           mpirun command line!), this component will
>>>>> be disabled and will
>>>>>                           not attempt to use either ummunotify or
>>>>> memory hook support
>>>>>               MCA memory: parameter "memory_linux_priority" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>            MCA paffinity: parameter "paffinity_base_verbose" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           Verbosity level of the paffinity framework
>>>>>            MCA paffinity: parameter "paffinity" (current value:
>>>>> <none>, data source:
>>>>>                           default value)
>>>>>                           Default selection set of components for the
>>>>> paffinity framework
>>>>>                           (<none>  means use all components that can be 
>>>>> found)
>>>>>            MCA paffinity: parameter "paffinity_hwloc_priority"
>>>>> (current value:<40>, data
>>>>>                           source: default value)
>>>>>                           Priority of the hwloc paffinity component
>>>>>                MCA carto: parameter "carto_base_verbose" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           Verbosity level of the carto framework
>>>>>                MCA carto: parameter "carto" (current value:<none>,
>>>>> data source: default
>>>>>                           value)
>>>>>                           Default selection set of components for the
>>>>> carto framework
>>>>>                           (<none>  means use all components that can be 
>>>>> found)
>>>>>                MCA carto: parameter "carto_auto_detect_priority"
>>>>> (current value:<11>,
>>>>>                           data source: default value)
>>>>>                           Priority of the auto_detect carto component
>>>>>                MCA carto: parameter "carto_file_path" (current value:
>>>>> <none>, data
>>>>>                           source: default value)
>>>>>                           The path to the cartography file
>>>>>                MCA carto: parameter "carto_file_priority" (current
>>>>> value:<10>, data
>>>>>                           source: default value)
>>>>>                           Priority of the file carto component
>>>>>                MCA shmem: parameter "shmem_base_verbose" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           Verbosity level of the shmem framework
>>>>>                MCA shmem: parameter "shmem" (current value:<none>,
>>>>> data source: default
>>>>>                           value)
>>>>>                           Default selection set of components for the
>>>>> shmem framework
>>>>>                           (<none>  means use all components that can be 
>>>>> found)
>>>>>                MCA shmem: parameter "shmem_mmap_enable_nfs_warning"
>>>>> (current value:<1>,
>>>>>                           data source: default value)
>>>>>                           Enable the warning emitted when Open MPI
>>>>> detects that its
>>>>>                           shared memory backing file is located on a
>>>>> network filesystem
>>>>>                           (1 = enabled, 0 = disabled).
>>>>>                MCA shmem: parameter "shmem_mmap_priority" (current
>>>>> value:<50>, data
>>>>>                           source: default value)
>>>>>                           Priority of the mmap shmem component
>>>>>                MCA shmem: parameter "shmem_mmap_relocate_backing_file"
>>>>> (current value:
>>>>>                           <0>, data source: default value)
>>>>>                           Whether to change the default placement of
>>>>> backing files or not
>>>>>                           (Negative = try to relocate backing files to
>>>>> an area rooted at
>>>>>                           the path specified by
>>>>>
>>>>> shmem_mmap_opal_shmem_mmap_backing_file_base_dir, but continue
>>>>>                           with the default path if the relocation
>>>>> fails, 0 = do not
>>>>>                           relocate, Positive = same as the negative
>>>>> option, but will fail
>>>>>                           if the relocation fails.
>>>>>                MCA shmem: parameter "shmem_mmap_backing_file_base_dir"
>>>>> (current value:
>>>>>                           </dev/shm>, data source: default value)
>>>>>                           Specifies where backing files will be created 
>>>>> when
>>>>>                           shmem_mmap_relocate_backing_file is in use.
>>>>>                MCA shmem: parameter "shmem_posix_priority" (current
>>>>> value:<40>, data
>>>>>                           source: default value)
>>>>>                           Priority of the posix shmem component
>>>>>                MCA shmem: parameter "shmem_sysv_priority" (current
>>>>> value:<30>, data
>>>>>                           source: default value)
>>>>>                           Priority of the sysv shmem component
>>>>>            MCA maffinity: parameter "maffinity_base_verbose" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           Verbosity level of the maffinity framework
>>>>>            MCA maffinity: parameter "maffinity" (current value:
>>>>> <none>, data source:
>>>>>                           default value)
>>>>>                           Default selection set of components for the
>>>>> maffinity framework
>>>>>                           (<none>  means use all components that can be 
>>>>> found)
>>>>>            MCA maffinity: parameter "maffinity_first_use_priority"
>>>>> (current value:<10>,
>>>>>                           data source: default value)
>>>>>                           Priority of the first_use maffinity component
>>>>>            MCA maffinity: parameter "maffinity_hwloc_priority"
>>>>> (current value:<40>, data
>>>>>                           source: default value)
>>>>>                           Priority of the hwloc maffinity component
>>>>>                MCA timer: parameter "timer" (current value:<none>,
>>>>> data source: default
>>>>>                           value)
>>>>>                           Default selection set of components for the
>>>>> timer framework
>>>>>                           (<none>  means use all components that can be 
>>>>> found)
>>>>>                MCA timer: parameter "timer_base_verbose" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           Verbosity level for the timer framework (0 =
>>>>> no verbosity)
>>>>>                MCA timer: parameter "timer_linux_priority" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>              MCA sysinfo: parameter "sysinfo" (current value:<none>,
>>>>> data source:
>>>>>                           default value)
>>>>>                           Default selection set of components for the
>>>>> sysinfo framework
>>>>>                           (<none>  means use all components that can be 
>>>>> found)
>>>>>              MCA sysinfo: parameter "sysinfo_base_verbose" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           Verbosity level for the sysinfo framework (0
>>>>> = no verbosity)
>>>>>              MCA sysinfo: parameter "sysinfo_linux_priority" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                MCA hwloc: parameter "hwloc_base_verbose" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           Verbosity level of the hwloc framework
>>>>>                MCA hwloc: parameter "hwloc_base_mem_alloc_policy"
>>>>> (current value:<none>,
>>>>>                           data source: default value)
>>>>>                           Policy that determines how general memory
>>>>> allocations are bound
>>>>>                           after MPI_INIT.  A value of "none" means
>>>>> that no memory policy
>>>>>                           is applied.  A value of "local_only" means
>>>>> that all memory
>>>>>                           allocations will be restricted to the local
>>>>> NUMA node where
>>>>>                           each process is placed.  Note that operating
>>>>> system paging
>>>>>                           policies are unaffected by this setting.
>>>>> For example, if
>>>>>                           "local_only" is used and local NUMA node
>>>>> memory is exhausted, a
>>>>>                           new memory allocation may cause paging.
>>>>>                MCA hwloc: parameter
>>>>> "hwloc_base_mem_bind_failure_action" (current value:
>>>>>                           <error>, data source: default value)
>>>>>                           What Open MPI will do if it explicitly tries
>>>>> to bind memory to
>>>>>                           a specific NUMA location, and fails.  Note
>>>>> that this is a
>>>>>                           different case than the general allocation
>>>>> policy described by
>>>>>                           hwloc_base_alloc_policy.  A value of "warn"
>>>>> means that Open MPI
>>>>>                           will warn the first time this happens, but
>>>>> allow the job to
>>>>>                           continue (possibly with degraded
>>>>> performance).  A value of
>>>>>                           "error" means that Open MPI will abort the
>>>>> job if this happens.
>>>>>                MCA hwloc: parameter "hwloc" (current value:<none>,
>>>>> data source: default
>>>>>                           value)
>>>>>                           Default selection set of components for the
>>>>> hwloc framework
>>>>>                           (<none>  means use all components that can be 
>>>>> found)
>>>>>                MCA hwloc: parameter "hwloc_hwloc132_priority" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                  MCA dpm: parameter "dpm" (current value:<none>, data
>>>>> source: default
>>>>>                           value)
>>>>>                           Default selection set of components for the
>>>>> dpm framework
>>>>>                           (<none>  means use all components that can be 
>>>>> found)
>>>>>                  MCA dpm: parameter "dpm_base_verbose" (current value:
>>>>> <0>, data source:
>>>>>                           default value)
>>>>>                           Verbosity level for the dpm framework (0 =
>>>>> no verbosity)
>>>>>                  MCA dpm: parameter "dpm_orte_priority" (current
>>>>> value:<0>, data source:
>>>>>                           default value)
>>>>>               MCA pubsub: parameter "pubsub" (current value:<none>,
>>>>> data source: default
>>>>>                           value)
>>>>>                           Default selection set of components for the
>>>>> pubsub framework
>>>>>                           (<none>  means use all components that can be 
>>>>> found)
>>>>>               MCA pubsub: parameter "pubsub_base_verbose" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           Verbosity level for the pubsub framework (0
>>>>> = no verbosity)
>>>>>               MCA pubsub: parameter "pubsub_orte_priority" (current
>>>>> value:<50>, data
>>>>>                           source: default value)
>>>>>                           Priority of the pubsub pmi component
>>>>>            MCA allocator: parameter "allocator" (current value:
>>>>> <none>, data source:
>>>>>                           default value)
>>>>>                           Default selection set of components for the
>>>>> allocator framework
>>>>>                           (<none>  means use all components that can be 
>>>>> found)
>>>>>            MCA allocator: parameter "allocator_base_verbose" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           Verbosity level for the allocator framework
>>>>> (0 = no verbosity)
>>>>>            MCA allocator: parameter "allocator_basic_priority"
>>>>> (current value:<0>, data
>>>>>                           source: default value)
>>>>>            MCA allocator: parameter "allocator_bucket_num_buckets"
>>>>> (current value:<30>,
>>>>>                           data source: default value)
>>>>>            MCA allocator: parameter "allocator_bucket_priority"
>>>>> (current value:<0>, data
>>>>>                           source: default value)
>>>>>                 MCA coll: parameter "coll" (current value:<none>,
>>>>> data source: default
>>>>>                           value)
>>>>>                           Default selection set of components for the
>>>>> coll framework
>>>>>                           (<none>  means use all components that can be 
>>>>> found)
>>>>>                 MCA coll: parameter "coll_base_verbose" (current
>>>>> value:<0>, data source:
>>>>>                           default value)
>>>>>                           Verbosity level for the coll framework (0 =
>>>>> no verbosity)
>>>>>                 MCA coll: parameter "coll_basic_priority" (current
>>>>> value:<10>, data
>>>>>                           source: default value)
>>>>>                           Priority of the basic coll component
>>>>>                 MCA coll: parameter "coll_basic_crossover" (current
>>>>> value:<4>, data
>>>>>                           source: default value)
>>>>>                           Minimum number of processes in a
>>>>> communicator before using the
>>>>>                           logarithmic algorithms
>>>>>                 MCA coll: parameter "coll_hierarch_priority" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           Priority of the hierarchical coll component
>>>>>                 MCA coll: parameter "coll_hierarch_verbose" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           Turn verbose message of the hierarchical
>>>>> coll component on/off
>>>>>                 MCA coll: parameter "coll_hierarch_use_rdma" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           Switch from the send btl list used to detect
>>>>> hierarchies to the
>>>>>                           rdma btl list
>>>>>                 MCA coll: parameter "coll_hierarch_ignore_sm" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           Ignore sm protocol when detecting
>>>>> hierarchies. Required to
>>>>>                           enable the usage of protocol specific
>>>>> collective operations
>>>>>                 MCA coll: parameter "coll_hierarch_detection_alg"
>>>>> (current value:<2>,
>>>>>                           data source: default value)
>>>>>                           Used to specify the algorithm for detecting
>>>>> Hierarchy.Choose
>>>>>                           between all or two levels of hierarchy
>>>>>                 MCA coll: parameter "coll_hierarch_bcast_alg" (current
>>>>> value:<4>, data
>>>>>                           source: default value)
>>>>>                           Used to specify the algorithm used for bcast
>>>>> operations.
>>>>>                 MCA coll: parameter "coll_hierarch_segment_size"
>>>>> (current value:<32768>,
>>>>>                           data source: default value)
>>>>>                           Used to specify the segment size for
>>>>> segmented algorithms.
>>>>>                 MCA coll: parameter "coll_inter_priority" (current
>>>>> value:<40>, data
>>>>>                           source: default value)
>>>>>                           Priority of the inter coll component
>>>>>                 MCA coll: parameter "coll_inter_verbose" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           Turn verbose message of the inter coll
>>>>> component on/off
>>>>>                 MCA coll: parameter "coll_self_priority" (current
>>>>> value:<75>, data
>>>>>                           source: default value)
>>>>>                 MCA coll: parameter "coll_sm_priority" (current value:
>>>>> <0>, data source:
>>>>>                           default value)
>>>>>                           Priority of the sm coll component
>>>>>                 MCA coll: parameter "coll_sm_control_size" (current
>>>>> value:<4096>, data
>>>>>                           source: default value)
>>>>>                           Length of the control data -- should usually
>>>>> be either the
>>>>>                           length of a cache line on most SMPs, or the
>>>>> size of a page on
>>>>>                           machines that support direct memory affinity
>>>>> page placement (in
>>>>>                           bytes)
>>>>>                 MCA coll: parameter "coll_sm_fragment_size" (current
>>>>> value:<8192>, data
>>>>>                           source: default value)
>>>>>                           Fragment size (in bytes) used for passing
>>>>> data through shared
>>>>>                           memory (will be rounded up to the nearest
>>>>> control_size size)
>>>>>                 MCA coll: parameter "coll_sm_comm_in_use_flags"
>>>>> (current value:<2>, data
>>>>>                           source: default value)
>>>>>                           Number of "in use" flags, used to mark a
>>>>> message passing area
>>>>>                           segment as currently being used or not (must
>>>>> be>= 2 and<=
>>>>>                           comm_num_segments)
>>>>>                 MCA coll: parameter "coll_sm_comm_num_segments"
>>>>> (current value:<8>, data
>>>>>                           source: default value)
>>>>>                           Number of segments in each communicator's
>>>>> shared memory message
>>>>>                           passing area (must be>= 2, and must be a 
>>>>> multiple of
>>>>>                           comm_in_use_flags)
>>>>>                 MCA coll: parameter "coll_sm_tree_degree" (current
>>>>> value:<4>, data
>>>>>                           source: default value)
>>>>>                           Degree of the tree for tree-based operations
>>>>> (must be =>  1 and
>>>>>                           <= min(control_size, 255))
>>>>>                 MCA coll: parameter "coll_sm_info_num_procs" (current
>>>>> value:<4>, data
>>>>>                           source: default value)
>>>>>                           Number of processes to use for the calculation 
>>>>> of the
>>>>>                           shared_mem_size MCA information parameter
>>>>> (must be =>  2)
>>>>>                 MCA coll: information "coll_sm_shared_mem_used_data"
>>>>> (value:<548864>,
>>>>>                           data source: default value)
>>>>>                           Amount of shared memory used, per
>>>>> communicator, in the shared
>>>>>                           memory data area for info_num_procs
>>>>> processes (in bytes)
>>>>>                 MCA coll: parameter "coll_sync_priority" (current
>>>>> value:<50>, data
>>>>>                           source: default value)
>>>>>                           Priority of the sync coll component; only 
>>>>> relevant if
>>>>>                           barrier_before or barrier_after is>  0
>>>>>                 MCA coll: parameter "coll_sync_barrier_before"
>>>>> (current value:<1000>,
>>>>>                           data source: default value)
>>>>>                           Do a synchronization before each Nth collective
>>>>>                 MCA coll: parameter "coll_sync_barrier_after" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           Do a synchronization after each Nth collective
>>>>>                 MCA coll: parameter "coll_tuned_priority" (current
>>>>> value:<30>, data
>>>>>                           source: default value)
>>>>>                           Priority of the tuned coll component
>>>>>                 MCA coll: parameter
>>>>> "coll_tuned_pre_allocate_memory_comm_size_limit"
>>>>>                           (current value:<32768>, data source: default 
>>>>> value)
>>>>>                           Size of communicator were we stop
>>>>> pre-allocating memory for the
>>>>>                           fixed internal buffer used for message
>>>>> requests etc that is
>>>>>                           hung off the communicator data segment. I.e.
>>>>> if you have a
>>>>>                           100'000 nodes you might not want to
>>>>> pre-allocate 200'000
>>>>>                           request handle slots per communicator instance!
>>>>>                 MCA coll: parameter "coll_tuned_init_tree_fanout"
>>>>> (current value:<4>,
>>>>>                           data source: default value)
>>>>>                           Inital fanout used in the tree topologies for 
>>>>> each
>>>>>                           communicator. This is only an initial guess,
>>>>> if a tuned
>>>>>                           collective needs a different fanout for an
>>>>> operation, it build
>>>>>                           it dynamically. This parameter is only for
>>>>> the first guess and
>>>>>                           might save a little time
>>>>>                 MCA coll: parameter "coll_tuned_init_chain_fanout"
>>>>> (current value:<4>,
>>>>>                           data source: default value)
>>>>>                           Inital fanout used in the chain (fanout
>>>>> followed by pipeline)
>>>>>                           topologies for each communicator. This is
>>>>> only an initial
>>>>>                           guess, if a tuned collective needs a
>>>>> different fanout for an
>>>>>                           operation, it build it dynamically. This
>>>>> parameter is only for
>>>>>                           the first guess and might save a little time
>>>>>                 MCA coll: parameter "coll_tuned_use_dynamic_rules"
>>>>> (current value:<0>,
>>>>>                           data source: default value)
>>>>>                           Switch used to decide if we use static
>>>>> (compiled/if statements)
>>>>>                           or dynamic (built at runtime) decision function 
>>>>> rules
>>>>>                   MCA io: parameter "io_base_freelist_initial_size"
>>>>> (current value:<16>,
>>>>>                           data source: default value)
>>>>>                           Initial MPI-2 IO request freelist size
>>>>>                   MCA io: parameter "io_base_freelist_max_size"
>>>>> (current value:<64>,
>>>>>                           data source: default value)
>>>>>                           Max size of the MPI-2 IO request freelist
>>>>>                   MCA io: parameter "io_base_freelist_increment"
>>>>> (current value:<16>,
>>>>>                           data source: default value)
>>>>>                           Increment size of the MPI-2 IO request freelist
>>>>>                   MCA io: parameter "io" (current value:<none>, data
>>>>> source: default
>>>>>                           value)
>>>>>                           Default selection set of components for the
>>>>> io framework
>>>>>                           (<none>  means use all components that can be 
>>>>> found)
>>>>>                   MCA io: parameter "io_base_verbose" (current value:
>>>>> <0>, data source:
>>>>>                           default value)
>>>>>                           Verbosity level for the io framework (0 = no
>>>>> verbosity)
>>>>>                   MCA io: parameter "io_romio_priority" (current
>>>>> value:<10>, data
>>>>>                           source: default value)
>>>>>                           Priority of the io romio component
>>>>>                   MCA io: parameter "io_romio_delete_priority"
>>>>> (current value:<10>, data
>>>>>                           source: default value)
>>>>>                           Delete priority of the io romio component
>>>>>                   MCA io: information "io_romio_version" (value:<from
>>>>> MPICH2 v1.3.1 with
>>>>>                           an additional patch from
>>>>> romio-ma...@mcs.anl.gov about an
>>>>>                           attribute issue>, data source: default value)
>>>>>                           Version of ROMIO
>>>>>                   MCA io: information "io_romio_user_configure_params"
>>>>> (value:<none>,
>>>>>                           data source: default value)
>>>>>                           User-specified command line parameters
>>>>> passed to ROMIO's
>>>>>                           configure script
>>>>>                   MCA io: information
>>>>> "io_romio_complete_configure_params" (value:<
>>>>>                           CFLAGS='-DNDEBUG -g -O2 -finline-functions
>>>>> -fno-strict-aliasing
>>>>>                           -pthread' CPPFLAGS='
>>>>>
>>>>> -I/home/andrea/library/openmpi/openmpi-1.6/opal/mca/hwloc/hwloc132/hwloc/include
>>>>>                             -I/usr/include/infiniband 
>>>>> -I/usr/include/infiniband'
>>>>>                           FFLAGS='' LDFLAGS='-Wl,--rpath
>>>>>                           -Wl,/home/andrea/library/gcc/gcc-objects/lib64 '
>>>>>                           --enable-shared --enable-static
>>>>>
>>>>> --prefix=/home/andrea/library/openmpi/openmpi-1.6-gnu-4.7
>>>>>                           --with-mpi=open_mpi --disable-aio>, data
>>>>> source: default value)
>>>>>                           Complete set of command line parameters
>>>>> passed to ROMIO's
>>>>>                           configure script
>>>>>                MCA mpool: parameter "mpool" (current value:<none>,
>>>>> data source: default
>>>>>                           value)
>>>>>                           Default selection set of components for the
>>>>> mpool framework
>>>>>                           (<none>  means use all components that can be 
>>>>> found)
>>>>>                MCA mpool: parameter "mpool_base_verbose" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           Verbosity level for the mpool framework (0 =
>>>>> no verbosity)
>>>>>                MCA mpool: parameter "mpool_fake_priority" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                MCA mpool: parameter "mpool_rdma_rcache_name" (current
>>>>> value:<vma>, data
>>>>>                           source: default value)
>>>>>                           The name of the registration cache the mpool
>>>>> should use
>>>>>                MCA mpool: parameter "mpool_rdma_rcache_size_limit"
>>>>> (current value:<0>,
>>>>>                           data source: default value)
>>>>>                           the maximum size of registration cache in
>>>>> bytes. 0 is unlimited
>>>>>                           (default 0)
>>>>>                MCA mpool: parameter "mpool_rdma_print_stats" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           print pool usage statistics at the end of the 
>>>>> run
>>>>>                MCA mpool: parameter "mpool_rdma_priority" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                MCA mpool: parameter "mpool_sm_allocator" (current
>>>>> value:<bucket>, data
>>>>>                           source: default value)
>>>>>                           Name of allocator component to use with sm mpool
>>>>>                MCA mpool: parameter "mpool_sm_min_size" (current
>>>>> value:<67108864>, data
>>>>>                           source: default value)
>>>>>                           Minimum size of the sm mpool shared memory file
>>>>>                MCA mpool: parameter "mpool_sm_verbose" (current value:
>>>>> <0>, data source:
>>>>>                           default value)
>>>>>                           Enable verbose output for mpool sm component
>>>>>                MCA mpool: parameter "mpool_sm_priority" (current
>>>>> value:<0>, data source:
>>>>>                           default value)
>>>>>                  MCA pml: parameter "pml_base_verbose" (current value:
>>>>> <0>, data source:
>>>>>                           default value)
>>>>>                           Verbosity level of the PML framework
>>>>>                  MCA pml: parameter "pml" (current value:<none>, data
>>>>> source: default
>>>>>                           value)
>>>>>                           Default selection set of components for the
>>>>> pml framework
>>>>>                           (<none>  means use all components that can be 
>>>>> found)
>>>>>                  MCA pml: parameter "pml_bfo_verbose" (current value:
>>>>> <0>, data source:
>>>>>                           default value)
>>>>>                  MCA pml: parameter "pml_bfo_free_list_num" (current
>>>>> value:<4>, data
>>>>>                           source: default value)
>>>>>                  MCA pml: parameter "pml_bfo_free_list_max" (current
>>>>> value:<-1>, data
>>>>>                           source: default value)
>>>>>                  MCA pml: parameter "pml_bfo_free_list_inc" (current
>>>>> value:<64>, data
>>>>>                           source: default value)
>>>>>                  MCA pml: parameter "pml_bfo_priority" (current value:
>>>>> <5>, data source:
>>>>>                           default value)
>>>>>                  MCA pml: parameter "pml_bfo_send_pipeline_depth"
>>>>> (current value:<3>,
>>>>>                           data source: default value)
>>>>>                  MCA pml: parameter "pml_bfo_recv_pipeline_depth"
>>>>> (current value:<4>,
>>>>>                           data source: default value)
>>>>>                  MCA pml: parameter "pml_bfo_rdma_put_retries_limit"
>>>>> (current value:<5>,
>>>>>                           data source: default value)
>>>>>                  MCA pml: parameter "pml_bfo_max_rdma_per_request"
>>>>> (current value:<4>,
>>>>>                           data source: default value)
>>>>>                  MCA pml: parameter "pml_bfo_max_send_per_range"
>>>>> (current value:<4>,
>>>>>                           data source: default value)
>>>>>                  MCA pml: parameter "pml_bfo_unexpected_limit"
>>>>> (current value:<128>,
>>>>>                           data source: default value)
>>>>>                  MCA pml: parameter "pml_bfo_allocator" (current
>>>>> value:<bucket>, data
>>>>>                           source: default value)
>>>>>                           Name of allocator component for unexpected 
>>>>> messages
>>>>>                  MCA pml: parameter "pml_cm_free_list_num" (current
>>>>> value:<4>, data
>>>>>                           source: default value)
>>>>>                           Initial size of request free lists
>>>>>                  MCA pml: parameter "pml_cm_free_list_max" (current
>>>>> value:<-1>, data
>>>>>                           source: default value)
>>>>>                           Maximum size of request free lists
>>>>>                  MCA pml: parameter "pml_cm_free_list_inc" (current
>>>>> value:<64>, data
>>>>>                           source: default value)
>>>>>                           Number of elements to add when growing
>>>>> request free lists
>>>>>                  MCA pml: parameter "pml_cm_priority" (current value:
>>>>> <10>, data source:
>>>>>                           default value)
>>>>>                           CM PML selection priority
>>>>>                  MCA pml: parameter "pml_csum_free_list_num" (current
>>>>> value:<4>, data
>>>>>                           source: default value)
>>>>>                  MCA pml: parameter "pml_csum_free_list_max" (current
>>>>> value:<-1>, data
>>>>>                           source: default value)
>>>>>                  MCA pml: parameter "pml_csum_free_list_inc" (current
>>>>> value:<64>, data
>>>>>                           source: default value)
>>>>>                  MCA pml: parameter "pml_csum_send_pipeline_depth"
>>>>> (current value:<3>,
>>>>>                           data source: default value)
>>>>>                  MCA pml: parameter "pml_csum_recv_pipeline_depth"
>>>>> (current value:<4>,
>>>>>                           data source: default value)
>>>>>                  MCA pml: parameter "pml_csum_rdma_put_retries_limit"
>>>>> (current value:
>>>>>                           <5>, data source: default value)
>>>>>                  MCA pml: parameter "pml_csum_max_rdma_per_request"
>>>>> (current value:<4>,
>>>>>                           data source: default value)
>>>>>                  MCA pml: parameter "pml_csum_max_send_per_range"
>>>>> (current value:<4>,
>>>>>                           data source: default value)
>>>>>                  MCA pml: parameter "pml_csum_unexpected_limit"
>>>>> (current value:<128>,
>>>>>                           data source: default value)
>>>>>                  MCA pml: parameter "pml_csum_allocator" (current
>>>>> value:<bucket>, data
>>>>>                           source: default value)
>>>>>                           Name of allocator component for unexpected 
>>>>> messages
>>>>>                  MCA pml: parameter "pml_csum_priority" (current
>>>>> value:<0>, data source:
>>>>>                           default value)
>>>>>                  MCA pml: parameter "pml_ob1_free_list_num" (current
>>>>> value:<4>, data
>>>>>                           source: default value)
>>>>>                  MCA pml: parameter "pml_ob1_free_list_max" (current
>>>>> value:<-1>, data
>>>>>                           source: default value)
>>>>>                  MCA pml: parameter "pml_ob1_free_list_inc" (current
>>>>> value:<64>, data
>>>>>                           source: default value)
>>>>>                  MCA pml: parameter "pml_ob1_priority" (current value:
>>>>> <20>, data source:
>>>>>                           default value)
>>>>>                  MCA pml: parameter "pml_ob1_send_pipeline_depth"
>>>>> (current value:<3>,
>>>>>                           data source: default value)
>>>>>                  MCA pml: parameter "pml_ob1_recv_pipeline_depth"
>>>>> (current value:<4>,
>>>>>                           data source: default value)
>>>>>                  MCA pml: parameter "pml_ob1_rdma_put_retries_limit"
>>>>> (current value:<5>,
>>>>>                           data source: default value)
>>>>>                  MCA pml: parameter "pml_ob1_max_rdma_per_request"
>>>>> (current value:<4>,
>>>>>                           data source: default value)
>>>>>                  MCA pml: parameter "pml_ob1_max_send_per_range"
>>>>> (current value:<4>,
>>>>>                           data source: default value)
>>>>>                  MCA pml: parameter "pml_ob1_unexpected_limit"
>>>>> (current value:<128>,
>>>>>                           data source: default value)
>>>>>                  MCA pml: parameter "pml_ob1_allocator" (current
>>>>> value:<bucket>, data
>>>>>                           source: default value)
>>>>>                           Name of allocator component for unexpected 
>>>>> messages
>>>>>                  MCA pml: parameter "pml_v_priority" (current value:
>>>>> <-1>, data source:
>>>>>                           default value)
>>>>>                  MCA pml: parameter "pml_v_output" (current value:
>>>>> <stderr>, data source:
>>>>>                           default value)
>>>>>                  MCA pml: parameter "pml_v_verbose" (current value:
>>>>> <0>, data source:
>>>>>                           default value)
>>>>>                  MCA bml: parameter "bml" (current value:<none>, data
>>>>> source: default
>>>>>                           value)
>>>>>                           Default selection set of components for the
>>>>> bml framework
>>>>>                           (<none>  means use all components that can be 
>>>>> found)
>>>>>                  MCA bml: parameter "bml_base_verbose" (current value:
>>>>> <0>, data source:
>>>>>                           default value)
>>>>>                           Verbosity level for the bml framework (0 =
>>>>> no verbosity)
>>>>>                  MCA bml: parameter "bml_r2_show_unreach_errors"
>>>>> (current value:<1>,
>>>>>                           data source: default value)
>>>>>                           Show error message when procs are unreachable
>>>>>                  MCA bml: parameter "bml_r2_priority" (current value:
>>>>> <0>, data source:
>>>>>                           default value)
>>>>>               MCA rcache: parameter "rcache" (current value:<none>,
>>>>> data source: default
>>>>>                           value)
>>>>>                           Default selection set of components for the
>>>>> rcache framework
>>>>>                           (<none>  means use all components that can be 
>>>>> found)
>>>>>               MCA rcache: parameter "rcache_base_verbose" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           Verbosity level for the rcache framework (0
>>>>> = no verbosity)
>>>>>               MCA rcache: parameter "rcache_vma_priority" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                  MCA btl: parameter "btl_base_verbose" (current value:
>>>>> <0>, data source:
>>>>>                           default value)
>>>>>                           Verbosity level of the BTL framework
>>>>>                  MCA btl: parameter "btl" (current value:<none>, data
>>>>> source: default
>>>>>                           value)
>>>>>                           Default selection set of components for the
>>>>> btl framework
>>>>>                           (<none>  means use all components that can be 
>>>>> found)
>>>>>                  MCA btl: parameter "btl_self_free_list_num" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           Number of fragments by default
>>>>>                  MCA btl: parameter "btl_self_free_list_max" (current
>>>>> value:<-1>, data
>>>>>                           source: default value)
>>>>>                           Maximum number of fragments
>>>>>                  MCA btl: parameter "btl_self_free_list_inc" (current
>>>>> value:<32>, data
>>>>>                           source: default value)
>>>>>                           Increment by this number of fragments
>>>>>                  MCA btl: parameter "btl_self_exclusivity" (current
>>>>> value:<65536>, data
>>>>>                           source: default value)
>>>>>                           BTL exclusivity (must be>= 0)
>>>>>                  MCA btl: parameter "btl_self_flags" (current value:
>>>>> <10>, data source:
>>>>>                           default value)
>>>>>                           BTL bit flags (general flags: SEND=1, PUT=2, 
>>>>> GET=4,
>>>>>                           SEND_INPLACE=8, RDMA_MATCHED=64,
>>>>> HETEROGENEOUS_RDMA=256; flags
>>>>>                           only used by the "dr" PML (ignored by others): 
>>>>> ACK=16,
>>>>>                           CHECKSUM=32, RDMA_COMPLETION=128; flags only
>>>>> used by the "bfo"
>>>>>                           PML (ignored by others): FAILOVER_SUPPORT=512)
>>>>>                  MCA btl: parameter "btl_self_rndv_eager_limit"
>>>>> (current value:<131072>,
>>>>>                           data source: default value)
>>>>>                           Size (in bytes) of "phase 1" fragment sent
>>>>> for all large
>>>>>                           messages (must be>= 0 and<= eager_limit)
>>>>>                  MCA btl: parameter "btl_self_eager_limit" (current
>>>>> value:<131072>, data
>>>>>                           source: default value)
>>>>>                           Maximum size (in bytes) of "short" messages
>>>>> (must be>= 1).
>>>>>                  MCA btl: parameter "btl_self_max_send_size" (current
>>>>> value:<262144>,
>>>>>                           data source: default value)
>>>>>                           Maximum size (in bytes) of a single "phase
>>>>> 2" fragment of a
>>>>>                           long message when using the pipeline
>>>>> protocol (must be>= 1)
>>>>>                  MCA btl: parameter
>>>>> "btl_self_rdma_pipeline_send_length" (current value:
>>>>>                           <2147483647>, data source: default value)
>>>>>                           Length of the "phase 2" portion of a large
>>>>> message (in bytes)
>>>>>                           when using the pipeline protocol.  This part
>>>>> of the message
>>>>>                           will be split into fragments of size
>>>>> max_send_size and sent
>>>>>                           using send/receive semantics (must be>= 0;
>>>>> only relevant when
>>>>>                           the PUT flag is set)
>>>>>                  MCA btl: parameter "btl_self_rdma_pipeline_frag_size"
>>>>> (current value:
>>>>>                           <2147483647>, data source: default value)
>>>>>                           Maximum size (in bytes) of a single "phase
>>>>> 3" fragment from a
>>>>>                           long message when using the pipeline
>>>>> protocol.  These fragments
>>>>>                           will be sent using RDMA semantics (must be
>>>>>> = 1; only relevant
>>>>>                           when the PUT flag is set)
>>>>>                  MCA btl: parameter "btl_self_min_rdma_pipeline_size"
>>>>> (current value:
>>>>>                           <0>, data source: default value)
>>>>>                           Messages smaller than this size (in bytes)
>>>>> will not use the
>>>>>                           RDMA pipeline protocol.  Instead, they will
>>>>> be split into
>>>>>                           fragments of max_send_size and sent using 
>>>>> send/receive
>>>>>                           semantics (must be>=0, and is automatically
>>>>> adjusted up to at
>>>>>                           least
>>>>> (eager_limit+btl_rdma_pipeline_send_length); only
>>>>>                           relevant when the PUT flag is set)
>>>>>                  MCA btl: parameter "btl_self_bandwidth" (current
>>>>> value:<100>, data
>>>>>                           source: default value)
>>>>>                           Approximate maximum bandwidth of
>>>>> interconnect(0 = auto-detect
>>>>>                           value at run-time [not supported in all BTL
>>>>> modules],>= 1 =
>>>>>                           bandwidth in Mbps)
>>>>>                  MCA btl: parameter "btl_self_latency" (current value:
>>>>> <0>, data source:
>>>>>                           default value)
>>>>>                           Approximate latency of interconnect (must be>= 
>>>>> 0)
>>>>>                  MCA btl: parameter "btl_self_priority" (current
>>>>> value:<0>, data source:
>>>>>                           default value)
>>>>>                  MCA btl: information "btl_sm_have_knem_support"
>>>>> (value:<0>, data
>>>>>                           source: default value)
>>>>>                           Whether this component supports the knem
>>>>> Linux kernel module or
>>>>>                           not
>>>>>                  MCA btl: parameter "btl_sm_use_knem" (current value:
>>>>> <0>, data source:
>>>>>                           default value)
>>>>>                           Whether knem support is desired or not
>>>>> (negative = try to
>>>>>                           enable knem support, but continue even if it
>>>>> is not available,
>>>>>                           0 = do not enable knem support, positive =
>>>>> try to enable knem
>>>>>                           support and fail if it is not available)
>>>>>                  MCA btl: parameter "btl_sm_knem_dma_min" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           Minimum message size (in bytes) to use the
>>>>> knem DMA mode;
>>>>>                           ignored if knem does not support DMA mode (0
>>>>> = do not use the
>>>>>                           knem DMA mode)
>>>>>                  MCA btl: parameter "btl_sm_knem_max_simultaneous"
>>>>> (current value:<0>,
>>>>>                           data source: default value)
>>>>>                           Max number of simultaneous ongoing knem
>>>>> operations to support
>>>>>                           (0 = do everything synchronously, which
>>>>> probably gives the best
>>>>>                           large message latency;>0 means to do all 
>>>>> operations
>>>>>                           asynchronously, which supports better
>>>>> overlap for simultaneous
>>>>>                           large message sends)
>>>>>                  MCA btl: parameter "btl_sm_free_list_num" (current
>>>>> value:<8>, data
>>>>>                           source: default value)
>>>>>                  MCA btl: parameter "btl_sm_free_list_max" (current
>>>>> value:<-1>, data
>>>>>                           source: default value)
>>>>>                  MCA btl: parameter "btl_sm_free_list_inc" (current
>>>>> value:<64>, data
>>>>>                           source: default value)
>>>>>                  MCA btl: parameter "btl_sm_max_procs" (current value:
>>>>> <-1>, data source:
>>>>>                           default value)
>>>>>                  MCA btl: parameter "btl_sm_mpool" (current value:
>>>>> <sm>, data source:
>>>>>                           default value)
>>>>>                  MCA btl: parameter "btl_sm_fifo_size" (current value:
>>>>> <4096>, data
>>>>>                           source: default value)
>>>>>                  MCA btl: parameter "btl_sm_num_fifos" (current value:
>>>>> <1>, data source:
>>>>>                           default value)
>>>>>                  MCA btl: parameter "btl_sm_fifo_lazy_free" (current
>>>>> value:<120>, data
>>>>>                           source: default value)
>>>>>                  MCA btl: parameter "btl_sm_sm_extra_procs" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                  MCA btl: parameter "btl_sm_exclusivity" (current
>>>>> value:<65535>, data
>>>>>                           source: default value)
>>>>>                           BTL exclusivity (must be>= 0)
>>>>>                  MCA btl: parameter "btl_sm_flags" (current value:
>>>>> <1>, data source:
>>>>>                           default value)
>>>>>                           BTL bit flags (general flags: SEND=1, PUT=2, 
>>>>> GET=4,
>>>>>                           SEND_INPLACE=8, RDMA_MATCHED=64,
>>>>> HETEROGENEOUS_RDMA=256; flags
>>>>>                           only used by the "dr" PML (ignored by others): 
>>>>> ACK=16,
>>>>>                           CHECKSUM=32, RDMA_COMPLETION=128; flags only
>>>>> used by the "bfo"
>>>>>                           PML (ignored by others): FAILOVER_SUPPORT=512)
>>>>>                  MCA btl: parameter "btl_sm_rndv_eager_limit" (current
>>>>> value:<4096>,
>>>>>                           data source: default value)
>>>>>                           Size (in bytes) of "phase 1" fragment sent
>>>>> for all large
>>>>>                           messages (must be>= 0 and<= eager_limit)
>>>>>                  MCA btl: parameter "btl_sm_eager_limit" (current
>>>>> value:<4096>, data
>>>>>                           source: default value)
>>>>>                           Maximum size (in bytes) of "short" messages
>>>>> (must be>= 1).
>>>>>                  MCA btl: parameter "btl_sm_max_send_size" (current
>>>>> value:<32768>, data
>>>>>                           source: default value)
>>>>>                           Maximum size (in bytes) of a single "phase
>>>>> 2" fragment of a
>>>>>                           long message when using the pipeline
>>>>> protocol (must be>= 1)
>>>>>                  MCA btl: parameter "btl_sm_bandwidth" (current value:
>>>>> <9000>, data
>>>>>                           source: default value)
>>>>>                           Approximate maximum bandwidth of
>>>>> interconnect(0 = auto-detect
>>>>>                           value at run-time [not supported in all BTL
>>>>> modules],>= 1 =
>>>>>                           bandwidth in Mbps)
>>>>>                  MCA btl: parameter "btl_sm_latency" (current value:
>>>>> <1>, data source:
>>>>>                           default value)
>>>>>                           Approximate latency of interconnect (must be>= 
>>>>> 0)
>>>>>                  MCA btl: parameter "btl_sm_priority" (current value:
>>>>> <0>, data source:
>>>>>                           default value)
>>>>>                  MCA btl: parameter "btl_tcp_links" (current value:
>>>>> <1>, data source:
>>>>>                           default value)
>>>>>                  MCA btl: parameter "btl_tcp_if_include" (current
>>>>> value:<none>, data
>>>>>                           source: default value)
>>>>>                           Comma-delimited list of devices or CIDR
>>>>> notation of networks to
>>>>>                           use for MPI communication (e.g., "eth0,eth1" or
>>>>>                           "192.168.0.0/16,10.1.4.0/24").  Mutually
>>>>> exclusive with
>>>>>                           btl_tcp_if_exclude.
>>>>>                  MCA btl: parameter "btl_tcp_if_exclude" (current
>>>>> value:<lo,sppp>, data
>>>>>                           source: default value)
>>>>>                           Comma-delimited list of devices or CIDR
>>>>> notation of networks to
>>>>>                           NOT use for MPI communication -- all devices
>>>>> not matching these
>>>>>                           specifications will be used (e.g., "eth0,eth1" 
>>>>> or
>>>>>                           "192.168.0.0/16,10.1.4.0/24").  Mutually
>>>>> exclusive with
>>>>>                           btl_tcp_if_include.
>>>>>                  MCA btl: parameter "btl_tcp_free_list_num" (current
>>>>> value:<8>, data
>>>>>                           source: default value)
>>>>>                  MCA btl: parameter "btl_tcp_free_list_max" (current
>>>>> value:<-1>, data
>>>>>                           source: default value)
>>>>>                  MCA btl: parameter "btl_tcp_free_list_inc" (current
>>>>> value:<32>, data
>>>>>                           source: default value)
>>>>>                  MCA btl: parameter "btl_tcp_sndbuf" (current value:
>>>>> <131072>, data
>>>>>                           source: default value)
>>>>>                  MCA btl: parameter "btl_tcp_rcvbuf" (current value:
>>>>> <131072>, data
>>>>>                           source: default value)
>>>>>                  MCA btl: parameter "btl_tcp_endpoint_cache" (current
>>>>> value:<30720>,
>>>>>                           data source: default value)
>>>>>                           The size of the internal cache for each TCP
>>>>> connection. This
>>>>>                           cache is used to reduce the number of
>>>>> syscalls, by replacing
>>>>>                           them with memcpy. Every read will read the
>>>>> expected data plus
>>>>>                           the amount of the endpoint_cache
>>>>>                  MCA btl: parameter "btl_tcp_use_nagle" (current
>>>>> value:<0>, data source:
>>>>>                           default value)
>>>>>                           Whether to use Nagle's algorithm or not (using 
>>>>> Nagle's
>>>>>                           algorithm may increase short message latency)
>>>>>                  MCA btl: parameter "btl_tcp_port_min_v4" (current
>>>>> value:<1024>, data
>>>>>                           source: default value)
>>>>>                           The minimum port where the TCP BTL will try
>>>>> to bind (default
>>>>>                           1024)
>>>>>                  MCA btl: parameter "btl_tcp_port_range_v4" (current
>>>>> value:<64511>, data
>>>>>                           source: default value)
>>>>>                           The number of ports where the TCP BTL will
>>>>> try to bind (default
>>>>>                           64511). This parameter together with the
>>>>> port min, define a
>>>>>                           range of ports where Open MPI will open sockets.
>>>>>                  MCA btl: parameter "btl_tcp_exclusivity" (current
>>>>> value:<100>, data
>>>>>                           source: default value)
>>>>>                           BTL exclusivity (must be>= 0)
>>>>>                  MCA btl: parameter "btl_tcp_flags" (current value:
>>>>> <314>, data source:
>>>>>                           default value)
>>>>>                           BTL bit flags (general flags: SEND=1, PUT=2, 
>>>>> GET=4,
>>>>>                           SEND_INPLACE=8, RDMA_MATCHED=64,
>>>>> HETEROGENEOUS_RDMA=256; flags
>>>>>                           only used by the "dr" PML (ignored by others): 
>>>>> ACK=16,
>>>>>                           CHECKSUM=32, RDMA_COMPLETION=128; flags only
>>>>> used by the "bfo"
>>>>>                           PML (ignored by others): FAILOVER_SUPPORT=512)
>>>>>                  MCA btl: parameter "btl_tcp_rndv_eager_limit"
>>>>> (current value:<65536>,
>>>>>                           data source: default value)
>>>>>                           Size (in bytes) of "phase 1" fragment sent
>>>>> for all large
>>>>>                           messages (must be>= 0 and<= eager_limit)
>>>>>                  MCA btl: parameter "btl_tcp_eager_limit" (current
>>>>> value:<65536>, data
>>>>>                           source: default value)
>>>>>                           Maximum size (in bytes) of "short" messages
>>>>> (must be>= 1).
>>>>>                  MCA btl: parameter "btl_tcp_max_send_size" (current
>>>>> value:<131072>,
>>>>>                           data source: default value)
>>>>>                           Maximum size (in bytes) of a single "phase
>>>>> 2" fragment of a
>>>>>                           long message when using the pipeline
>>>>> protocol (must be>= 1)
>>>>>                  MCA btl: parameter
>>>>> "btl_tcp_rdma_pipeline_send_length" (current value:
>>>>>                           <131072>, data source: default value)
>>>>>                           Length of the "phase 2" portion of a large
>>>>> message (in bytes)
>>>>>                           when using the pipeline protocol.  This part
>>>>> of the message
>>>>>                           will be split into fragments of size
>>>>> max_send_size and sent
>>>>>                           using send/receive semantics (must be>= 0;
>>>>> only relevant when
>>>>>                           the PUT flag is set)
>>>>>                  MCA btl: parameter "btl_tcp_rdma_pipeline_frag_size"
>>>>> (current value:
>>>>>                           <2147483647>, data source: default value)
>>>>>                           Maximum size (in bytes) of a single "phase
>>>>> 3" fragment from a
>>>>>                           long message when using the pipeline
>>>>> protocol.  These fragments
>>>>>                           will be sent using RDMA semantics (must be
>>>>>> = 1; only relevant
>>>>>                           when the PUT flag is set)
>>>>>                  MCA btl: parameter "btl_tcp_min_rdma_pipeline_size"
>>>>> (current value:<0>,
>>>>>                           data source: default value)
>>>>>                           Messages smaller than this size (in bytes)
>>>>> will not use the
>>>>>                           RDMA pipeline protocol.  Instead, they will
>>>>> be split into
>>>>>                           fragments of max_send_size and sent using 
>>>>> send/receive
>>>>>                           semantics (must be>=0, and is automatically
>>>>> adjusted up to at
>>>>>                           least
>>>>> (eager_limit+btl_rdma_pipeline_send_length); only
>>>>>                           relevant when the PUT flag is set)
>>>>>                  MCA btl: parameter "btl_tcp_bandwidth" (current
>>>>> value:<100>, data
>>>>>                           source: default value)
>>>>>                           Approximate maximum bandwidth of
>>>>> interconnect(0 = auto-detect
>>>>>                           value at run-time [not supported in all BTL
>>>>> modules],>= 1 =
>>>>>                           bandwidth in Mbps)
>>>>>                  MCA btl: parameter "btl_tcp_latency" (current value:
>>>>> <100>, data source:
>>>>>                           default value)
>>>>>                           Approximate latency of interconnect (must be>= 
>>>>> 0)
>>>>>                  MCA btl: parameter "btl_tcp_disable_family" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                  MCA btl: parameter "btl_tcp_if_seq" (current value:
>>>>> <none>, data source:
>>>>>                           default value)
>>>>>                           If specified, a comma-delimited list of TCP
>>>>> interfaces.
>>>>>                           Interfaces will be assigned, one to each MPI
>>>>> process, in a
>>>>>                           round-robin fashion on each server.  For
>>>>> example, if the list
>>>>>                           is "eth0,eth1" and four MPI processes are
>>>>> run on a single
>>>>>                           server, then local ranks 0 and 2 will use
>>>>> eth0 and local ranks
>>>>>                           1 and 3 will use eth1.
>>>>>                  MCA btl: parameter "btl_tcp_priority" (current value:
>>>>> <0>, data source:
>>>>>                           default value)
>>>>>                  MCA btl: parameter "btl_base_include" (current value:
>>>>> <none>, data
>>>>>                           source: default value)
>>>>>                  MCA btl: parameter "btl_base_exclude" (current value:
>>>>> <none>, data
>>>>>                           source: default value)
>>>>>                  MCA btl: parameter "btl_base_warn_component_unused"
>>>>> (current value:<1>,
>>>>>                           data source: default value)
>>>>>                           This parameter is used to turn on warning
>>>>> messages when certain
>>>>>                           NICs are not used
>>>>>                  MCA mtl: parameter "mtl" (current value:<none>, data
>>>>> source: default
>>>>>                           value)
>>>>>                           Default selection set of components for the
>>>>> mtl framework
>>>>>                           (<none>  means use all components that can be 
>>>>> found)
>>>>>                  MCA mtl: parameter "mtl_base_verbose" (current value:
>>>>> <0>, data source:
>>>>>                           default value)
>>>>>                           Verbosity level for the mtl framework (0 =
>>>>> no verbosity)
>>>>>                 MCA topo: parameter "topo" (current value:<none>,
>>>>> data source: default
>>>>>                           value)
>>>>>                           Default selection set of components for the
>>>>> topo framework
>>>>>                           (<none>  means use all components that can be 
>>>>> found)
>>>>>                 MCA topo: parameter "topo_base_verbose" (current
>>>>> value:<0>, data source:
>>>>>                           default value)
>>>>>                           Verbosity level for the topo framework (0 =
>>>>> no verbosity)
>>>>>                 MCA topo: parameter "topo_unity_priority" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                  MCA osc: parameter "osc" (current value:<none>, data
>>>>> source: default
>>>>>                           value)
>>>>>                           Default selection set of components for the
>>>>> osc framework
>>>>>                           (<none>  means use all components that can be 
>>>>> found)
>>>>>                  MCA osc: parameter "osc_base_verbose" (current value:
>>>>> <0>, data source:
>>>>>                           default value)
>>>>>                           Verbosity level for the osc framework (0 =
>>>>> no verbosity)
>>>>>                  MCA osc: parameter "osc_pt2pt_no_locks" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           Enable optimizations available only if
>>>>> MPI_LOCK is not used.
>>>>>                  MCA osc: parameter "osc_pt2pt_eager_limit" (current
>>>>> value:<16384>, data
>>>>>                           source: default value)
>>>>>                           Max size of eagerly sent data
>>>>>                  MCA osc: parameter "osc_pt2pt_priority" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                  MCA osc: parameter "osc_rdma_eager_send" (current
>>>>> value:<1>, data
>>>>>                           source: default value)
>>>>>                           Attempt to start data movement during
>>>>> communication call,
>>>>>                           instead of at synchrnoization time.  Info
>>>>> key of same name
>>>>>                           overrides this value.
>>>>>                  MCA osc: parameter "osc_rdma_use_buffers" (current
>>>>> value:<1>, data
>>>>>                           source: default value)
>>>>>                           Coalesce messages during an epoch to reduce 
>>>>> network
>>>>>                           utilization.  Info key of same name
>>>>> overrides this value.
>>>>>                  MCA osc: parameter "osc_rdma_use_rdma" (current
>>>>> value:<0>, data source:
>>>>>                           default value)
>>>>>                           Use real RDMA operations to transfer data.
>>>>> Info key of same
>>>>>                           name overrides this value.
>>>>>                  MCA osc: parameter "osc_rdma_rdma_completion_wait"
>>>>> (current value:<1>,
>>>>>                           data source: default value)
>>>>>                           Wait for all completion of rdma events before 
>>>>> sending
>>>>>                           acknowledgment.  Info key of same name
>>>>> overrides this value.
>>>>>                  MCA osc: parameter "osc_rdma_no_locks" (current
>>>>> value:<0>, data source:
>>>>>                           default value)
>>>>>                           Enable optimizations available only if
>>>>> MPI_LOCK is not used.
>>>>>                           Info key of same name overrides this value.
>>>>>                  MCA osc: parameter "osc_rdma_priority" (current
>>>>> value:<0>, data source:
>>>>>                           default value)
>>>>>                   MCA op: parameter "op_base_verbose" (current value:
>>>>> <0>, data source:
>>>>>                           default value)
>>>>>                           Verbosity level of the op framework
>>>>>                  MCA iof: parameter "iof" (current value:<none>, data
>>>>> source: default
>>>>>                           value)
>>>>>                           Default selection set of components for the
>>>>> iof framework
>>>>>                           (<none>  means use all components that can be 
>>>>> found)
>>>>>                  MCA iof: parameter "iof_base_verbose" (current value:
>>>>> <0>, data source:
>>>>>                           default value)
>>>>>                           Verbosity level for the iof framework (0 =
>>>>> no verbosity)
>>>>>                  MCA iof: parameter "iof_hnp_priority" (current value:
>>>>> <0>, data source:
>>>>>                           default value)
>>>>>                  MCA iof: parameter "iof_orted_priority" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                  MCA iof: parameter "iof_tool_priority" (current
>>>>> value:<0>, data source:
>>>>>                           default value)
>>>>>                  MCA oob: parameter "oob" (current value:<none>, data
>>>>> source: default
>>>>>                           value)
>>>>>                           Default selection set of components for the
>>>>> oob framework
>>>>>                           (<none>  means use all components that can be 
>>>>> found)
>>>>>                  MCA oob: parameter "oob_base_verbose" (current value:
>>>>> <0>, data source:
>>>>>                           default value)
>>>>>                           Verbosity level for the oob framework (0 =
>>>>> no verbosity)
>>>>>                  MCA oob: parameter "oob_tcp_verbose" (current value:
>>>>> <0>, data source:
>>>>>                           default value)
>>>>>                           Verbose level for the OOB tcp component
>>>>>                  MCA oob: parameter "oob_tcp_peer_limit" (current
>>>>> value:<-1>, data
>>>>>                           source: default value)
>>>>>                           Maximum number of peer connections to
>>>>> simultaneously maintain
>>>>>                           (-1 = infinite)
>>>>>                  MCA oob: parameter "oob_tcp_peer_retries" (current
>>>>> value:<60>, data
>>>>>                           source: default value)
>>>>>                           Number of times to try shutting down a
>>>>> connection before giving
>>>>>                           up
>>>>>                  MCA oob: parameter "oob_tcp_debug" (current value:
>>>>> <0>, data source:
>>>>>                           default value)
>>>>>                           Enable (1) / disable (0) debugging output
>>>>> for this component
>>>>>                  MCA oob: parameter "oob_tcp_sndbuf" (current value:
>>>>> <131072>, data
>>>>>                           source: default value)
>>>>>                           TCP socket send buffering size (in bytes)
>>>>>                  MCA oob: parameter "oob_tcp_rcvbuf" (current value:
>>>>> <131072>, data
>>>>>                           source: default value)
>>>>>                           TCP socket receive buffering size (in bytes)
>>>>>                  MCA oob: parameter "oob_tcp_if_include" (current
>>>>> value:<none>, data
>>>>>                           source: default value)
>>>>>                           Comma-delimited list of TCP interfaces to use
>>>>>                  MCA oob: parameter "oob_tcp_if_exclude" (current
>>>>> value:<none>, data
>>>>>                           source: default value)
>>>>>                           Comma-delimited list of TCP interfaces to 
>>>>> exclude
>>>>>                  MCA oob: parameter "oob_tcp_connect_sleep" (current
>>>>> value:<1>, data
>>>>>                           source: default value)
>>>>>                           Enable (1) / disable (0) random sleep for
>>>>> connection wireup.
>>>>>                  MCA oob: parameter "oob_tcp_listen_mode" (current
>>>>> value:<event>, data
>>>>>                           source: default value)
>>>>>                           Mode for HNP to accept incoming connections: 
>>>>> event,
>>>>>                           listen_thread.
>>>>>                  MCA oob: parameter "oob_tcp_listen_thread_max_queue"
>>>>> (current value:
>>>>>                           <10>, data source: default value)
>>>>>                           High water mark for queued accepted socket
>>>>> list size.  Used
>>>>>                           only when listen_mode is listen_thread.
>>>>>                  MCA oob: parameter "oob_tcp_listen_thread_wait_time"
>>>>> (current value:
>>>>>                           <10>, data source: default value)
>>>>>                           Time in milliseconds to wait before actively
>>>>> checking for new
>>>>>                           connections when listen_mode is listen_thread.
>>>>>                  MCA oob: parameter "oob_tcp_static_ports" (current
>>>>> value:<none>, data
>>>>>                           source: default value)
>>>>>                           Static ports for daemons and procs (IPv4)
>>>>>                  MCA oob: parameter "oob_tcp_dynamic_ports" (current
>>>>> value:<none>, data
>>>>>                           source: default value)
>>>>>                           Range of ports to be dynamically used by
>>>>> daemons and procs
>>>>>                           (IPv4)
>>>>>                  MCA oob: parameter "oob_tcp_disable_family" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           Disable IPv4 (4) or IPv6 (6)
>>>>>                  MCA oob: parameter "oob_tcp_priority" (current value:
>>>>> <0>, data source:
>>>>>                           default value)
>>>>>                 MCA odls: parameter "odls_base_sigkill_timeout"
>>>>> (current value:<1>, data
>>>>>                           source: default value)
>>>>>                           Time to wait for a process to die after
>>>>> issuing a kill signal
>>>>>                           to it
>>>>>                 MCA odls: parameter "odls" (current value:<none>,
>>>>> data source: default
>>>>>                           value)
>>>>>                           Default selection set of components for the
>>>>> odls framework
>>>>>                           (<none>  means use all components that can be 
>>>>> found)
>>>>>                 MCA odls: parameter "odls_base_verbose" (current
>>>>> value:<0>, data source:
>>>>>                           default value)
>>>>>                           Verbosity level for the odls framework (0 =
>>>>> no verbosity)
>>>>>                 MCA odls: parameter "odls_default_priority" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                  MCA ras: parameter "ras_base_display_alloc" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           Whether to display the allocation after it
>>>>> is determined
>>>>>                  MCA ras: parameter "ras_base_display_devel_alloc"
>>>>> (current value:<0>,
>>>>>                           data source: default value)
>>>>>                           Whether to display a developer-detail
>>>>> allocation after it is
>>>>>                           determined
>>>>>                  MCA ras: parameter "ras" (current value:<none>, data
>>>>> source: default
>>>>>                           value)
>>>>>                           Default selection set of components for the
>>>>> ras framework
>>>>>                           (<none>  means use all components that can be 
>>>>> found)
>>>>>                  MCA ras: parameter "ras_base_verbose" (current value:
>>>>> <0>, data source:
>>>>>                           default value)
>>>>>                           Verbosity level for the ras framework (0 =
>>>>> no verbosity)
>>>>>                  MCA ras: parameter "ras_cm_priority" (current value:
>>>>> <0>, data source:
>>>>>                           default value)
>>>>>                  MCA ras: parameter "ras_loadleveler_priority"
>>>>> (current value:<90>, data
>>>>>                           source: default value)
>>>>>                           Priority of the loadleveler ras component
>>>>>                  MCA ras: parameter "ras_slurm_priority" (current
>>>>> value:<75>, data
>>>>>                           source: default value)
>>>>>                           Priority of the slurm ras component
>>>>>                MCA rmaps: parameter "rmaps_rank_file_path" (current
>>>>> value:<none>, data
>>>>>                           source: default value, synonym of: 
>>>>> orte_rankfile)
>>>>>                           Name of the rankfile to be used for mapping
>>>>> processes (relative
>>>>>                           or absolute path)
>>>>>                MCA rmaps: parameter "rmaps_base_schedule_policy"
>>>>> (current value:<slot>,
>>>>>                           data source: default value)
>>>>>                           Scheduling Policy for RMAPS. [slot
>>>>> (alias:core) | socket |
>>>>>                           board | node]
>>>>>                MCA rmaps: parameter "rmaps_base_pernode" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           Launch one ppn as directed
>>>>>                MCA rmaps: parameter "rmaps_base_n_pernode" (current
>>>>> value:<-1>, data
>>>>>                           source: default value)
>>>>>                           Launch n procs/node
>>>>>                MCA rmaps: parameter "rmaps_base_n_perboard" (current
>>>>> value:<-1>, data
>>>>>                           source: default value)
>>>>>                           Launch n procs/board
>>>>>                MCA rmaps: parameter "rmaps_base_n_persocket" (current
>>>>> value:<-1>, data
>>>>>                           source: default value)
>>>>>                           Launch n procs/socket
>>>>>                MCA rmaps: parameter "rmaps_base_loadbalance" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           Balance total number of procs across all
>>>>> allocated nodes
>>>>>                MCA rmaps: parameter "rmaps_base_cpus_per_proc"
>>>>> (current value:<1>, data
>>>>>                           source: default value, synonyms:
>>>>> rmaps_base_cpus_per_rank)
>>>>>                           Number of cpus to use for each rank [1-2**15
>>>>> (default=1)]
>>>>>                MCA rmaps: parameter "rmaps_base_cpus_per_rank"
>>>>> (current value:<1>, data
>>>>>                           source: default value, synonym of:
>>>>> rmaps_base_cpus_per_proc)
>>>>>                           Number of cpus to use for each rank [1-2**15
>>>>> (default=1)]
>>>>>                MCA rmaps: parameter "rmaps_base_stride" (current
>>>>> value:<1>, data source:
>>>>>                           default value)
>>>>>                           When binding multiple cores to a rank, the
>>>>> step size to use
>>>>>                           between cores [1-2**15 (default: 1)]
>>>>>                MCA rmaps: parameter "rmaps_base_slot_list" (current
>>>>> value:<none>, data
>>>>>                           source: default value)
>>>>>                           List of processor IDs to bind MPI processes
>>>>> to (e.g., used in
>>>>>                           conjunction with rank files) [default=NULL]
>>>>>                MCA rmaps: parameter "rmaps_base_no_schedule_local"
>>>>> (current value:<0>,
>>>>>                           data source: default value)
>>>>>                           If false, allow scheduling MPI applications
>>>>> on the same node as
>>>>>                           mpirun (default).  If true, do not schedule any 
>>>>> MPI
>>>>>                           applications on the same node as mpirun
>>>>>                MCA rmaps: parameter "rmaps_base_no_oversubscribe"
>>>>> (current value:<0>,
>>>>>                           data source: default value)
>>>>>                           If true, then do not allow oversubscription
>>>>> of nodes - mpirun
>>>>>                           will return an error if there aren't enough
>>>>> nodes to launch all
>>>>>                           processes without oversubscribing
>>>>>                MCA rmaps: parameter "rmaps_base_display_map" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           Whether to display the process map after it
>>>>> is computed
>>>>>                MCA rmaps: parameter "rmaps_base_display_devel_map"
>>>>> (current value:<0>,
>>>>>                           data source: default value)
>>>>>                           Whether to display a developer-detail
>>>>> process map after it is
>>>>>                           computed
>>>>>                MCA rmaps: parameter "rmaps" (current value:<none>,
>>>>> data source: default
>>>>>                           value)
>>>>>                           Default selection set of components for the
>>>>> rmaps framework
>>>>>                           (<none>  means use all components that can be 
>>>>> found)
>>>>>                MCA rmaps: parameter "rmaps_base_verbose" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           Verbosity level for the rmaps framework (0 =
>>>>> no verbosity)
>>>>>                MCA rmaps: parameter "rmaps_load_balance_priority"
>>>>> (current value:<0>,
>>>>>                           data source: default value)
>>>>>                MCA rmaps: parameter "rmaps_rank_file_priority"
>>>>> (current value:<0>, data
>>>>>                           source: default value)
>>>>>                MCA rmaps: parameter "rmaps_resilient_fault_grp_file"
>>>>> (current value:
>>>>>                           <none>, data source: default value)
>>>>>                           Filename that contains a description of
>>>>> fault groups for this
>>>>>                           system
>>>>>                MCA rmaps: parameter "rmaps_resilient_priority"
>>>>> (current value:<0>, data
>>>>>                           source: default value)
>>>>>                MCA rmaps: parameter "rmaps_round_robin_priority"
>>>>> (current value:<0>,
>>>>>                           data source: default value)
>>>>>                MCA rmaps: parameter "rmaps_seq_priority" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                MCA rmaps: parameter "rmaps_topo_priority" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                  MCA rml: parameter "rml_wrapper" (current value:
>>>>> <none>, data source:
>>>>>                           default value)
>>>>>                           Use a Wrapper component around the selected
>>>>> RML component
>>>>>                  MCA rml: parameter "rml" (current value:<none>, data
>>>>> source: default
>>>>>                           value)
>>>>>                           Default selection set of components for the
>>>>> rml framework
>>>>>                           (<none>  means use all components that can be 
>>>>> found)
>>>>>                  MCA rml: parameter "rml_base_verbose" (current value:
>>>>> <0>, data source:
>>>>>                           default value)
>>>>>                           Verbosity level for the rml framework (0 =
>>>>> no verbosity)
>>>>>                  MCA rml: parameter "rml_oob_priority" (current value:
>>>>> <0>, data source:
>>>>>                           default value)
>>>>>               MCA routed: parameter "routed" (current value:<none>,
>>>>> data source: default
>>>>>                           value)
>>>>>                           Default selection set of components for the
>>>>> routed framework
>>>>>                           (<none>  means use all components that can be 
>>>>> found)
>>>>>               MCA routed: parameter "routed_base_verbose" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           Verbosity level for the routed framework (0
>>>>> = no verbosity)
>>>>>               MCA routed: parameter "routed_binomial_priority"
>>>>> (current value:<0>, data
>>>>>                           source: default value)
>>>>>               MCA routed: parameter "routed_cm_priority" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>               MCA routed: parameter "routed_direct_priority" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>               MCA routed: parameter "routed_linear_priority" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>               MCA routed: parameter "routed_radix_priority" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>               MCA routed: parameter "routed_slave_priority" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                  MCA plm: parameter "plm_rsh_agent" (current value:
>>>>> <ssh : rsh>, data
>>>>>                           source: default value, deprecated, synonym
>>>>> of: orte_rsh_agent)
>>>>>                           The command used to launch executables on 
>>>>> remote nodes
>>>>>                           (typically either "ssh" or "rsh")
>>>>>                  MCA plm: parameter "plm_rsh_assume_same_shell"
>>>>> (current value:<1>, data
>>>>>                           source: default value, deprecated, synonym of:
>>>>>                           orte_assume_same_shell)
>>>>>                           If set to 1, assume that the shell on the
>>>>> remote node is the
>>>>>                           same as the shell on the local node.
>>>>> Otherwise, probe for what
>>>>>                           the remote shell [default: 1]
>>>>>                  MCA plm: parameter "plm" (current value:<none>, data
>>>>> source: default
>>>>>                           value)
>>>>>                           Default selection set of components for the
>>>>> plm framework
>>>>>                           (<none>  means use all components that can be 
>>>>> found)
>>>>>                  MCA plm: parameter "plm_base_verbose" (current value:
>>>>> <0>, data source:
>>>>>                           default value)
>>>>>                           Verbosity level for the plm framework (0 =
>>>>> no verbosity)
>>>>>                  MCA plm: parameter "plm_rsh_num_concurrent" (current
>>>>> value:<128>, data
>>>>>                           source: default value)
>>>>>                           How many plm_rsh_agent instances to invoke
>>>>> concurrently (must
>>>>>                           be>  0)
>>>>>                  MCA plm: parameter "plm_rsh_force_rsh" (current
>>>>> value:<0>, data source:
>>>>>                           default value)
>>>>>                           Force the launcher to always use rsh
>>>>>                  MCA plm: parameter "plm_rsh_disable_qrsh" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           Disable the launcher to use qrsh when under
>>>>> the SGE parallel
>>>>>                           environment
>>>>>                  MCA plm: parameter "plm_rsh_daemonize_qrsh" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           Daemonize the orted under the SGE parallel 
>>>>> environment
>>>>>                  MCA plm: parameter "plm_rsh_disable_llspawn" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           Disable the use of llspawn when under the 
>>>>> LoadLeveler
>>>>>                           environment
>>>>>                  MCA plm: parameter "plm_rsh_daemonize_llspawn"
>>>>> (current value:<0>, data
>>>>>                           source: default value)
>>>>>                           Daemonize the orted when under the
>>>>> LoadLeveler environment
>>>>>                  MCA plm: parameter "plm_rsh_priority" (current value:
>>>>> <10>, data source:
>>>>>                           default value)
>>>>>                           Priority of the rsh plm component
>>>>>                  MCA plm: parameter "plm_rsh_delay" (current value:
>>>>> <1>, data source:
>>>>>                           default value)
>>>>>                           Delay (in seconds) between invocations of
>>>>> the remote agent, but
>>>>>                           only used when the "debug" MCA parameter is
>>>>> true, or the
>>>>>                           top-level MCA debugging is enabled
>>>>> (otherwise this value is
>>>>>                           ignored)
>>>>>                  MCA plm: parameter "plm_rsh_tree_spawn" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           If set to 1, launch via a tree-based topology
>>>>>                  MCA plm: parameter "plm_slurm_args" (current value:
>>>>> <none>, data source:
>>>>>                           default value)
>>>>>                           Custom arguments to srun
>>>>>                  MCA plm: parameter "plm_slurm_priority" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                MCA filem: parameter "filem" (current value:<none>,
>>>>> data source: default
>>>>>                           value)
>>>>>                           Which Filem component to use (empty = 
>>>>> auto-select)
>>>>>                MCA filem: parameter "filem_base_verbose" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           Verbosity level for the filem framework (0 =
>>>>> no verbosity)
>>>>>                MCA filem: parameter "filem_rsh_priority" (current
>>>>> value:<20>, data
>>>>>                           source: default value)
>>>>>                           Priority of the FILEM rsh component
>>>>>                MCA filem: parameter "filem_rsh_verbose" (current
>>>>> value:<0>, data source:
>>>>>                           default value)
>>>>>                           Verbose level for the FILEM rsh component
>>>>>                MCA filem: parameter "filem_rsh_rcp" (current value:
>>>>> <scp>, data source:
>>>>>                           default value)
>>>>>                           The rsh cp command for the FILEM rsh component
>>>>>                MCA filem: parameter "filem_rsh_cp" (current value:
>>>>> <cp>, data source:
>>>>>                           default value)
>>>>>                           The Unix cp command for the FILEM rsh component
>>>>>                MCA filem: parameter "filem_rsh_rsh" (current value:
>>>>> <ssh>, data source:
>>>>>                           default value)
>>>>>                           The remote shell command for the FILEM rsh 
>>>>> component
>>>>>                MCA filem: parameter "filem_rsh_max_incomming" (current
>>>>> value:<10>, data
>>>>>                           source: default value)
>>>>>                           Maximum number of incomming connections (0 = 
>>>>> any)
>>>>>                MCA filem: parameter "filem_rsh_max_outgoing" (current
>>>>> value:<10>, data
>>>>>                           source: default value)
>>>>>                           Maximum number of out going connections (0 = 
>>>>> any)
>>>>>               MCA errmgr: parameter "errmgr" (current value:<none>,
>>>>> data source: default
>>>>>                           value)
>>>>>                           Default selection set of components for the
>>>>> errmgr framework
>>>>>                           (<none>  means use all components that can be 
>>>>> found)
>>>>>               MCA errmgr: parameter "errmgr_base_verbose" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           Verbosity level for the errmgr framework (0
>>>>> = no verbosity)
>>>>>               MCA errmgr: parameter "errmgr_default_priority" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                  MCA ess: parameter "ess" (current value:<none>, data
>>>>> source: default
>>>>>                           value)
>>>>>                           Default selection set of components for the
>>>>> ess framework
>>>>>                           (<none>  means use all components that can be 
>>>>> found)
>>>>>                  MCA ess: parameter "ess_base_verbose" (current value:
>>>>> <0>, data source:
>>>>>                           default value)
>>>>>                           Verbosity level for the ess framework (0 =
>>>>> no verbosity)
>>>>>                  MCA ess: parameter "ess_env_priority" (current value:
>>>>> <0>, data source:
>>>>>                           default value)
>>>>>                  MCA ess: parameter "ess_hnp_priority" (current value:
>>>>> <0>, data source:
>>>>>                           default value)
>>>>>                  MCA ess: parameter "ess_singleton_priority" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                  MCA ess: parameter "ess_slave_priority" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                  MCA ess: parameter "ess_slurm_priority" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                  MCA ess: parameter "ess_slurmd_priority" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                  MCA ess: parameter "ess_tool_priority" (current
>>>>> value:<0>, data source:
>>>>>                           default value)
>>>>>              MCA grpcomm: parameter "grpcomm" (current value:<none>,
>>>>> data source:
>>>>>                           default value)
>>>>>                           Default selection set of components for the
>>>>> grpcomm framework
>>>>>                           (<none>  means use all components that can be 
>>>>> found)
>>>>>              MCA grpcomm: parameter "grpcomm_base_verbose" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           Verbosity level for the grpcomm framework (0
>>>>> = no verbosity)
>>>>>              MCA grpcomm: parameter "grpcomm_bad_priority" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>              MCA grpcomm: parameter "grpcomm_basic_priority" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>              MCA grpcomm: parameter "grpcomm_hier_priority" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>             MCA notifier: parameter "notifier_threshold_severity"
>>>>> (current value:
>>>>>                           <critical>, data source: default value)
>>>>>                           Report all events at or above this severity
>>>>> [default: critical]
>>>>>             MCA notifier: parameter "notifier" (current value:<none>,
>>>>> data source:
>>>>>                           default value)
>>>>>                           Default selection set of components for the
>>>>> notifier framework
>>>>>                           (<none>  means use all components that can be 
>>>>> found)
>>>>>             MCA notifier: parameter "notifier_base_verbose" (current
>>>>> value:<0>, data
>>>>>                           source: default value)
>>>>>                           Verbosity level for the notifier framework
>>>>> (0 = no verbosity)
>>>>>             MCA notifier: parameter "notifier_command_cmd" (current
>>>>> value:</sbin/initlog
>>>>>                           -f $s -n "Open MPI" -s "$S: $m (errorcode:
>>>>> $e)">, data source:
>>>>>                           default value)
>>>>>                           Command to execute, with substitution.  $s =
>>>>> integer severity;
>>>>>                           $S = string severity; $e = integer error
>>>>> code; $m = string
>>>>>                           message
>>>>>             MCA notifier: parameter "notifier_command_timeout"
>>>>> (current value:<30>, data
>>>>>                           source: default value)
>>>>>                           Timeout (in seconds) of the command
>>>>>             MCA notifier: parameter "notifier_command_priority"
>>>>> (current value:<10>,
>>>>>                           data source: default value)
>>>>>                           Priority of this component
>>>>>             MCA notifier: parameter "notifier_syslog_priority"
>>>>> (current value:<0>, data
>>>>>                           source: default value)
>>>>>
>>>>> ====================================================================================================
>>>>>
>>>>> output of cat /proc/cpuinfo
>>>>> processor     : 0
>>>>> vendor_id     : AuthenticAMD
>>>>> cpu family    : 15
>>>>> model         : 75
>>>>> model name    : AMD Athlon(tm) 64 X2 Dual Core Processor 3800+
>>>>> stepping      : 2
>>>>> cpu MHz               : 1002.094
>>>>> cache size    : 512 KB
>>>>> physical id   : 0
>>>>> siblings      : 2
>>>>> core id               : 0
>>>>> cpu cores     : 2
>>>>> fpu           : yes
>>>>> fpu_exception : yes
>>>>> cpuid level   : 1
>>>>> wp            : yes
>>>>> flags         : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
>>>>> cmov
>>>>> pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext lm 3dnowext
>>>>> 3dnow pni cx16
>>>>> bogomips      : 2003.90
>>>>> TLB size      : 1088 4K pages
>>>>> clflush size  : 64
>>>>> cache_alignment       : 64
>>>>> address sizes : 40 bits physical, 48 bits virtual
>>>>> power management: ts fid vid ttp [4] [5]
>>>>>
>>>>> processor     : 1
>>>>> vendor_id     : AuthenticAMD
>>>>> cpu family    : 15
>>>>> model         : 75
>>>>> model name    : AMD Athlon(tm) 64 X2 Dual Core Processor 3800+
>>>>> stepping      : 2
>>>>> cpu MHz               : 1002.094
>>>>> cache size    : 512 KB
>>>>> physical id   : 0
>>>>> siblings      : 2
>>>>> core id               : 1
>>>>> cpu cores     : 2
>>>>> fpu           : yes
>>>>> fpu_exception : yes
>>>>> cpuid level   : 1
>>>>> wp            : yes
>>>>> flags         : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
>>>>> cmov
>>>>> pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext lm 3dnowext
>>>>> 3dnow pni cx16
>>>>> bogomips      : 2003.90
>>>>> TLB size      : 1088 4K pages
>>>>> clflush size  : 64
>>>>> cache_alignment       : 64
>>>>> address sizes : 40 bits physical, 48 bits virtual
>>>>> power management: ts fid vid ttp [4] [5]
>>>>>
>>>>>
>>>>> ====================================================================================================
>>>>> output of ifconfig -a from a compute node
>>>>> eth0      Link encap:Ethernet  HWaddr 00:18:F3:3F:84:A1
>>>>>           inet addr:192.168.0.2  Bcast:192.168.0.255  Mask:255.255.255.0
>>>>>           inet6 addr: fe80::218:f3ff:fe3f:84a1/64 Scope:Link
>>>>>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>>>>>           RX packets:2006 errors:0 dropped:0 overruns:0 frame:0
>>>>>           TX packets:2064 errors:0 dropped:0 overruns:0 carrier:0
>>>>>           collisions:0 txqueuelen:1000
>>>>>           RX bytes:242685 (236.9 KiB)  TX bytes:0 (0.0 b)
>>>>>           Interrupt:11 Base address:0x8000
>>>>>
>>>>> lo        Link encap:Local Loopback
>>>>>           inet addr:127.0.0.1  Mask:255.0.0.0
>>>>>           inet6 addr: ::1/128 Scope:Host
>>>>>           UP LOOPBACK RUNNING  MTU:16436  Metric:1
>>>>>           RX packets:60 errors:0 dropped:0 overruns:0 frame:0
>>>>>           TX packets:60 errors:0 dropped:0 overruns:0 carrier:0
>>>>>           collisions:0 txqueuelen:0
>>>>>           RX bytes:4440 (4.3 KiB)  TX bytes:4440 (4.3 KiB)
>>>>>
>>>>> sit0      Link encap:IPv6-in-IPv4
>>>>>           NOARP  MTU:1480  Metric:1
>>>>>           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>>>>>           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
>>>>>           collisions:0 txqueuelen:0
>>>>>           RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
>>>>>
>>>>>
>>>>> ====================================================================================================
>>>>> output of ifconfig -a from the login node, where I run mpirun
>>>>>
>>>>> eth0      Link encap:Ethernet  HWaddr 00:18:F3:51:B3:6E
>>>>>           inet addr:192.168.0.1  Bcast:192.168.0.255  Mask:255.255.255.0
>>>>>           inet6 addr: fe80::218:f3ff:fe51:b36e/64 Scope:Link
>>>>>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>>>>>           RX packets:7180758 errors:0 dropped:0 overruns:0 frame:0
>>>>>           TX packets:4989496 errors:0 dropped:0 overruns:0 carrier:0
>>>>>           collisions:0 txqueuelen:1000
>>>>>           RX bytes:6045614452 (5.6 GiB)  TX bytes:0 (0.0 b)
>>>>>           Interrupt:201 Base address:0xe000
>>>>>
>>>>> eth1      Link encap:Ethernet  HWaddr 00:01:02:13:AA:3C
>>>>>           inet addr:137.204.66.188  Bcast:137.204.66.255  
>>>>> Mask:255.255.255.0
>>>>>           inet6 addr: fe80::201:2ff:fe13:aa3c/64 Scope:Link
>>>>>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>>>>>           RX packets:4750212 errors:0 dropped:0 overruns:0 frame:0
>>>>>           TX packets:405027 errors:0 dropped:0 overruns:0 carrier:0
>>>>>           collisions:0 txqueuelen:1000
>>>>>           RX bytes:629146679 (600.0 MiB)  TX bytes:332118265 (316.7 MiB)
>>>>>           Interrupt:177 Base address:0x9c00
>>>>>
>>>>> lo        Link encap:Local Loopback
>>>>>           inet addr:127.0.0.1  Mask:255.0.0.0
>>>>>           inet6 addr: ::1/128 Scope:Host
>>>>>           UP LOOPBACK RUNNING  MTU:16436  Metric:1
>>>>>           RX packets:288455 errors:0 dropped:0 overruns:0 frame:0
>>>>>           TX packets:288455 errors:0 dropped:0 overruns:0 carrier:0
>>>>>           collisions:0 txqueuelen:0
>>>>>           RX bytes:35908038 (34.2 MiB)  TX bytes:35908038 (34.2 MiB)
>>>>>
>>>>> sit0      Link encap:IPv6-in-IPv4
>>>>>           NOARP  MTU:1480  Metric:1
>>>>>           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>>>>>           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
>>>>>           collisions:0 txqueuelen:0
>>>>>           RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
>>>>>
>>>>> ====================================================================================================
>>>>> output of mpirun --bynode --hostfile my_hostfile.txt --tag-output
>>>>> ompi_info -v ompi full --parsable
>>>>>
>>>>> [1,0]<stdout>:package:Open MPI and...@cloud.bo.astro.it Distribution
>>>>> [1,0]<stdout>:ompi:version:full:1.6
>>>>> [1,0]<stdout>:ompi:version:svn:r26429
>>>>> [1,0]<stdout>:ompi:version:release_date:May 10, 2012
>>>>> [1,0]<stdout>:orte:version:full:1.6
>>>>> [1,0]<stdout>:orte:version:svn:r26429
>>>>> [1,0]<stdout>:orte:version:release_date:May 10, 2012
>>>>> [1,0]<stdout>:opal:version:full:1.6
>>>>> [1,0]<stdout>:opal:version:svn:r26429
>>>>> [1,0]<stdout>:opal:version:release_date:May 10, 2012
>>>>> [1,0]<stdout>:mpi-api:version:full:2.1
>>>>> [1,0]<stdout>:ident:1.6
>>>>> [1,6]<stdout>:package:Open MPI and...@cloud.bo.astro.it Distribution
>>>>> [1,6]<stdout>:ompi:version:full:1.6
>>>>> [1,6]<stdout>:ompi:version:svn:r26429
>>>>> [1,6]<stdout>:ompi:version:release_date:May 10, 2012
>>>>> [1,6]<stdout>:orte:version:full:1.6
>>>>> [1,6]<stdout>:orte:version:svn:r26429
>>>>> [1,6]<stdout>:orte:version:release_date:May 10, 2012
>>>>> [1,6]<stdout>:opal:version:full:1.6
>>>>> [1,6]<stdout>:opal:version:svn:r26429
>>>>> [1,6]<stdout>:opal:version:release_date:May 10, 2012
>>>>> [1,6]<stdout>:mpi-api:version:full:2.1
>>>>> [1,6]<stdout>:ident:1.6
>>>>> [1,9]<stdout>:package:Open MPI and...@cloud.bo.astro.it Distribution
>>>>> [1,10]<stdout>:package:Open MPI and...@cloud.bo.astro.it Distribution
>>>>> [1,3]<stdout>:package:Open MPI and...@cloud.bo.astro.it Distribution
>>>>> [1,3]<stdout>:ompi:version:full:1.6
>>>>> [1,3]<stdout>:ompi:version:svn:r26429
>>>>> [1,3]<stdout>:ompi:version:release_date:May 10, 2012
>>>>> [1,3]<stdout>:orte:version:full:1.6
>>>>> [1,3]<stdout>:orte:version:svn:r26429
>>>>> [1,3]<stdout>:orte:version:release_date:May 10, 2012
>>>>> [1,3]<stdout>:opal:version:full:1.6
>>>>> [1,3]<stdout>:opal:version:svn:r26429
>>>>> [1,3]<stdout>:opal:version:release_date:May 10, 2012
>>>>> [1,3]<stdout>:mpi-api:version:full:2.1
>>>>> [1,3]<stdout>:ident:1.6
>>>>> [1,4]<stdout>:package:Open MPI and...@cloud.bo.astro.it Distribution
>>>>> [1,4]<stdout>:ompi:version:full:1.6
>>>>> [1,4]<stdout>:ompi:version:svn:r26429
>>>>> [1,4]<stdout>:ompi:version:release_date:May 10, 2012
>>>>> [1,4]<stdout>:orte:version:full:1.6
>>>>> [1,4]<stdout>:orte:version:svn:r26429
>>>>> [1,4]<stdout>:orte:version:release_date:May 10, 2012
>>>>> [1,4]<stdout>:opal:version:full:1.6
>>>>> [1,9]<stdout>:ompi:version:full:1.6
>>>>> [1,4]<stdout>:opal:version:svn:r26429
>>>>> [1,4]<stdout>:opal:version:release_date:May 10, 2012
>>>>> [1,4]<stdout>:mpi-api:version:full:2.1
>>>>> [1,4]<stdout>:ident:1.6
>>>>> [1,9]<stdout>:ompi:version:svn:r26429
>>>>> [1,10]<stdout>:ompi:version:full:1.6
>>>>> [1,9]<stdout>:ompi:version:release_date:May 10, 2012
>>>>> [1,10]<stdout>:ompi:version:svn:r26429
>>>>> [1,9]<stdout>:orte:version:full:1.6
>>>>> [1,10]<stdout>:ompi:version:release_date:May 10, 2012
>>>>> [1,9]<stdout>:orte:version:svn:r26429
>>>>> [1,10]<stdout>:orte:version:full:1.6
>>>>> [1,10]<stdout>:orte:version:svn:r26429
>>>>> [1,9]<stdout>:orte:version:release_date:May 10, 2012
>>>>> [1,10]<stdout>:orte:version:release_date:May 10, 2012
>>>>> [1,9]<stdout>:opal:version:full:1.6
>>>>> [1,10]<stdout>:opal:version:full:1.6
>>>>> [1,9]<stdout>:opal:version:svn:r26429
>>>>> [1,10]<stdout>:opal:version:svn:r26429
>>>>> [1,9]<stdout>:opal:version:release_date:May 10, 2012
>>>>> [1,10]<stdout>:opal:version:release_date:May 10, 2012
>>>>> [1,9]<stdout>:mpi-api:version:full:2.1
>>>>> [1,9]<stdout>:ident:1.6
>>>>> [1,10]<stdout>:mpi-api:version:full:2.1
>>>>> [1,10]<stdout>:ident:1.6
>>>>> [1,2]<stdout>:package:Open MPI and...@cloud.bo.astro.it Distribution
>>>>> [1,2]<stdout>:ompi:version:full:1.6
>>>>> [1,2]<stdout>:ompi:version:svn:r26429
>>>>> [1,2]<stdout>:ompi:version:release_date:May 10, 2012
>>>>> [1,2]<stdout>:orte:version:full:1.6
>>>>> [1,2]<stdout>:orte:version:svn:r26429
>>>>> [1,2]<stdout>:orte:version:release_date:May 10, 2012
>>>>> [1,2]<stdout>:opal:version:full:1.6
>>>>> [1,2]<stdout>:opal:version:svn:r26429
>>>>> [1,2]<stdout>:opal:version:release_date:May 10, 2012
>>>>> [1,2]<stdout>:mpi-api:version:full:2.1
>>>>> [1,2]<stdout>:ident:1.6
>>>>> [1,8]<stdout>:package:Open MPI and...@cloud.bo.astro.it Distribution
>>>>> [1,8]<stdout>:ompi:version:full:1.6
>>>>> [1,8]<stdout>:ompi:version:svn:r26429
>>>>> [1,8]<stdout>:ompi:version:release_date:May 10, 2012
>>>>> [1,8]<stdout>:orte:version:full:1.6
>>>>> [1,8]<stdout>:orte:version:svn:r26429
>>>>> [1,8]<stdout>:orte:version:release_date:May 10, 2012
>>>>> [1,8]<stdout>:opal:version:full:1.6
>>>>> [1,8]<stdout>:opal:version:svn:r26429
>>>>> [1,8]<stdout>:opal:version:release_date:May 10, 2012
>>>>> [1,8]<stdout>:mpi-api:version:full:2.1
>>>>> [1,8]<stdout>:ident:1.6
>>>>> [1,11]<stdout>:package:Open MPI and...@cloud.bo.astro.it Distribution
>>>>> [1,11]<stdout>:ompi:version:full:1.6
>>>>> [1,11]<stdout>:ompi:version:svn:r26429
>>>>> [1,11]<stdout>:ompi:version:release_date:May 10, 2012
>>>>> [1,11]<stdout>:orte:version:full:1.6
>>>>> [1,11]<stdout>:orte:version:svn:r26429
>>>>> [1,11]<stdout>:orte:version:release_date:May 10, 2012
>>>>> [1,11]<stdout>:opal:version:full:1.6
>>>>> [1,11]<stdout>:opal:version:svn:r26429
>>>>> [1,11]<stdout>:opal:version:release_date:May 10, 2012
>>>>> [1,11]<stdout>:mpi-api:version:full:2.1
>>>>> [1,11]<stdout>:ident:1.6
>>>>> [1,5]<stdout>:package:Open MPI and...@cloud.bo.astro.it Distribution
>>>>> [1,5]<stdout>:ompi:version:full:1.6
>>>>> [1,5]<stdout>:ompi:version:svn:r26429
>>>>> [1,5]<stdout>:ompi:version:release_date:May 10, 2012
>>>>> [1,5]<stdout>:orte:version:full:1.6
>>>>> [1,5]<stdout>:orte:version:svn:r26429
>>>>> [1,5]<stdout>:orte:version:release_date:May 10, 2012
>>>>> [1,5]<stdout>:opal:version:full:1.6
>>>>> [1,5]<stdout>:opal:version:svn:r26429
>>>>> [1,5]<stdout>:opal:version:release_date:May 10, 2012
>>>>> [1,5]<stdout>:mpi-api:version:full:2.1
>>>>> [1,5]<stdout>:ident:1.6
>>>>> [1,1]<stdout>:package:Open MPI and...@cloud.bo.astro.it Distribution
>>>>> [1,7]<stdout>:package:Open MPI and...@cloud.bo.astro.it Distribution
>>>>> [1,7]<stdout>:ompi:version:full:1.6
>>>>> [1,7]<stdout>:ompi:version:svn:r26429
>>>>> [1,7]<stdout>:ompi:version:release_date:May 10, 2012
>>>>> [1,7]<stdout>:orte:version:full:1.6
>>>>> [1,7]<stdout>:orte:version:svn:r26429
>>>>> [1,7]<stdout>:orte:version:release_date:May 10, 2012
>>>>> [1,7]<stdout>:opal:version:full:1.6
>>>>> [1,7]<stdout>:opal:version:svn:r26429
>>>>> [1,7]<stdout>:opal:version:release_date:May 10, 2012
>>>>> [1,7]<stdout>:mpi-api:version:full:2.1
>>>>> [1,7]<stdout>:ident:1.6
>>>>> [1,1]<stdout>:ompi:version:full:1.6
>>>>> [1,1]<stdout>:ompi:version:svn:r26429
>>>>> [1,1]<stdout>:ompi:version:release_date:May 10, 2012
>>>>> [1,1]<stdout>:orte:version:full:1.6
>>>>> [1,1]<stdout>:orte:version:svn:r26429
>>>>> [1,1]<stdout>:orte:version:release_date:May 10, 2012
>>>>> [1,1]<stdout>:opal:version:full:1.6
>>>>> [1,1]<stdout>:opal:version:svn:r26429
>>>>> [1,1]<stdout>:opal:version:release_date:May 10, 2012
>>>>> [1,1]<stdout>:mpi-api:version:full:2.1
>>>>> [1,1]<stdout>:ident:1.6
>>>>> _______________________________________________
>>>>> users mailing list
>>>>> us...@open-mpi.org
>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>
>>>>
>>>>
>>>> ------------------------------
>>>>
>>>> _______________________________________________
>>>> users mailing list
>>>> us...@open-mpi.org
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>
>>>> End of users Digest, Vol 2339, Issue 5
>>>> **************************************
>>>
>>>
>>> ------------------------------
>>>
>>> _______________________________________________
>>> users mailing list
>>> us...@open-mpi.org
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>
>>> End of users Digest, Vol 2340, Issue 1
>>> **************************************
>> _______________________________________________
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
>
>
> ------------------------------
>
> _______________________________________________
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
> End of users Digest, Vol 2342, Issue 3
> **************************************

Reply via email to