Jorge, I am not that familiar with UCX, but I hope that will help:
The changes I mentioned were introduced by https://github.com/open-mpi/ompi/pull/8549 I suspect mpirun --mca pml_ucx_tls any --mca pml_ucx_devices --mca pml ucx ... will do what you expect Cheers, Gilles On Mon, Sep 13, 2021 at 9:05 PM Jorge D'Elia via users < users@lists.open-mpi.org> wrote: > Dear Gilles, > > Despite my last answer (see below), I am noticing that > some tests with a coarray fortran code on a laptop show a > performance drop of the order of 20% using the 4.1.1 version > (with --mca pml ucx disabled), versus the 4.1.0 one > (with --mca pml ucx enabled). > > I would like to experiment with pml/ucx framework using the 4.1.0 > version on that laptop. Then, please, how do I manually re-enable > those providers? (e.g. perhaps, is it during the construction > stage?) or where can I find out how to do it? Thanks in advance. > > Regards. > Jorge. > > ----- Mensaje original ----- > > De: "Open MPI Users" <users@lists.open-mpi.org> > > Para: "Open MPI Users" <users@lists.open-mpi.org> > > CC: "Jorge D'Elia" > > Enviado: Sábado, 29 de Mayo 2021 7:18:23 > > Asunto: Re: [OMPI users] (Fedora 34, x86_64-pc-linux-gnu, > openmpi-4.1.1.tar.gz): PML ucx cannot be selected > > > > Dear Gilles, > > > > Ahhh ... now the new behavior is better understood. > > The intention of using pml/ucx was simply for preliminary > > testing, and does not merit re-enabling these providers in > > this notebook. > > > > Thank you very much for the clarification. > > > > Regards, > > Jorge. > > > > ----- Mensaje original ----- > >> De: "Gilles Gouaillardet" > >> Para: "Jorge D'Elia" , "Open MPI Users" <users@lists.open-mpi.org> > >> Enviado: Viernes, 28 de Mayo 2021 23:35:37 > >> Asunto: Re: [OMPI users] (Fedora 34, x86_64-pc-linux-gnu, > openmpi-4.1.1.tar.gz): > >> PML ucx cannot be selected > >> > >> Jorge, > >> > >> pml/ucx used to be selected when no fast interconnect were detected > >> (since ucx provides driver for both TCP and shared memory). > >> These providers are now disabled by default, so unless your machine > >> has a supported fast interconnect (such as Infiniband), > >> pml/ucx cannot be used out of the box anymore. > >> > >> if you really want to use pml/ucx on your notebook, you need to > >> manually re-enable these providers. > >> > >> That being said, your best choice here is really not to force any pml, > >> and let Open MPI use pml/ob1 > >> (that has support for both TCP and shared memory) > >> > >> Cheers, > >> > >> Gilles > >> > >> On Sat, May 29, 2021 at 11:19 AM Jorge D'Elia via users > >> <users@lists.open-mpi.org> wrote: > >>> > >>> Hi, > >>> > >>> We routinely build OpenMPI on x86_64-pc-linux-gnu machines from > >>> the sources using gcc and usually everything works fine. > >>> > >>> In one case we recently installed Fedora 34 from scratch on an > >>> ASUS G53SX notebook (Intel Core i7-2630QM CPU 2.00GHz ×4 cores, > >>> without any IB device). Next we build OpenMPI using the file > >>> openmpi-4.1.1.tar.gz and the GCC 12.0.0 20210524 (experimental) > >>> compiler. > >>> > >>> However, when trying to experiment OpenMPI using UCX > >>> with a simple test, we get the runtime errors: > >>> > >>> No components were able to be opened in the btl framework. > >>> PML ucx cannot be selected > >>> > >>> while the test worked fine until Fedora 33 on the same > >>> machine using the same OpenMPI configuration. > >>> > >>> We attach below some info about a simple test run. > >>> > >>> Please, any clues where to check or maybe something is missing? > >>> Thanks in advance. > >>> > >>> Regards > >>> Jorge. > >>> > >>> -- > >>> $ cat /proc/version > >>> Linux version 5.12.7-300.fc34.x86_64 > >>> (mockbu...@bkernel01.iad2.fedoraproject.org) (gcc (GCC) 11.1.1 > 20210428 (Red > >>> Hat 11.1.1-1), GNU ld version 2.35.1-41.fc34) #1 SMP Wed May 26 > 12:58:58 UTC > >>> 2021 > >>> > >>> $ mpifort --version > >>> GNU Fortran (GCC) 12.0.0 20210524 (experimental) > >>> Copyright (C) 2021 Free Software Foundation, Inc. > >>> > >>> $ which mpifort > >>> /usr/beta/openmpi/bin/mpifort > >>> > >>> $ mpifort -o hello_usempi_f08.exe hello_usempi_f08.f90 > >>> > >>> $ mpirun --mca orte_base_help_aggregate 0 --mca btl self,vader,tcp > --map-by node > >>> --report-bindings --machinefile ~/machi-openmpi.dat --np 2 > >>> hello_usempi_f08.exe > >>> [verne:200650] MCW rank 0 bound to socket 0[core 0[hwt 0]]: [B/././.] > >>> [verne:200650] MCW rank 1 bound to socket 0[core 1[hwt 0]]: [./B/./.] > >>> Hello, world, I am 0 of 2: Open MPI v4.1.1, package: Open MPI > bigpack@verne > >>> Distribution, ident: 4.1.1, repo rev: v4.1.1, Apr 24, 2021 > >>> Hello, world, I am 1 of 2: Open MPI v4.1.1, package: Open MPI > bigpack@verne > >>> Distribution, ident: 4.1.1, repo rev: v4.1.1, Apr 24, 2021 > >>> > >>> $ mpirun --mca orte_base_help_aggregate 0 --mca pml ucx --mca btl > >>> ^self,vader,tcp --map-by node --report-bindings --machinefile > >>> ~/machi-openmpi.dat --np 2 hello_usempi_f08.exe > >>> [verne:200772] MCW rank 0 bound to socket 0[core 0[hwt 0]]: [B/././.] > >>> [verne:200772] MCW rank 1 bound to socket 0[core 1[hwt 0]]: [./B/./.] > >>> > -------------------------------------------------------------------------- > >>> No components were able to be opened in the btl framework. > >>> > >>> This typically means that either no components of this type were > >>> installed, or none of the installed components can be loaded. > >>> Sometimes this means that shared libraries required by these > >>> components are unable to be found/loaded. > >>> > >>> Host: verne > >>> Framework: btl > >>> > -------------------------------------------------------------------------- > >>> > -------------------------------------------------------------------------- > >>> No components were able to be opened in the btl framework. > >>> > >>> This typically means that either no components of this type were > >>> installed, or none of the installed components can be loaded. > >>> Sometimes this means that shared libraries required by these > >>> components are unable to be found/loaded. > >>> > >>> Host: verne > >>> Framework: btl > >>> > -------------------------------------------------------------------------- > >>> > -------------------------------------------------------------------------- > >>> No components were able to be opened in the pml framework. > >>> > >>> This typically means that either no components of this type were > >>> installed, or none of the installed components can be loaded. > >>> Sometimes this means that shared libraries required by these > >>> components are unable to be found/loaded. > >>> > >>> Host: verne > >>> Framework: pml > >>> > -------------------------------------------------------------------------- > >>> [verne:200777] PML ucx cannot be selected > >>> > -------------------------------------------------------------------------- > >>> No components were able to be opened in the pml framework. > >>> > >>> This typically means that either no components of this type were > >>> installed, or none of the installed components can be loaded. > >>> Sometimes this means that shared libraries required by these > >>> components are unable to be found/loaded. > >>> > >>> Host: verne > >>> Framework: pml > >>> > -------------------------------------------------------------------------- > >>> [verne:200772] PMIX ERROR: UNREACHABLE in file > >>> > ../../../../../../../opal/mca/pmix/pmix3x/pmix/src/server/pmix_server.c at > line > >>> 2198 > >>> > >>> > >>> $ ompi_info | grep ucx > >>> Configure command line: '--enable-ipv6' '--enable-sparse-groups' > >>> '--enable-mpi-ext' '--enable-mpi-cxx' '--enable-oshmem' > >>> '--with-libevent=internal' '--with-ucx' '--with-pmix=internal' > >>> '--without-libfabric' '--prefix=/usr/beta/openmpi' > >>> MCA osc: ucx (MCA v2.1.0, API v3.0.0, Component > v4.1.1) > >>> MCA pml: ucx (MCA v2.1.0, API v2.0.0, Component > >>> v4.1.1) > >>> > >>> $ ompi_info --param all all --level 9 | grep ucx > >>> MCA osc: ucx (MCA v2.1.0, API v3.0.0, Component > v4.1.1) > >>> MCA pml: ucx (MCA v2.1.0, API v2.0.0, Component > v4.1.1) > >>> MCA osc ucx: > --------------------------------------------------- > >>> MCA osc ucx: parameter "osc_ucx_priority" (current value: > "60", data source: > >>> default, level: 3 user/all, type: unsigned_int) > >>> Priority of the osc/ucx component (default: > 60) > >>> MCA osc ucx: parameter "osc_ucx_verbose" (current value: > "0", data source: > >>> default, level: 3 user/all, type: int, synonym of: > opal_common_ucx_verbose) > >>> MCA osc ucx: parameter "osc_ucx_progress_iterations" > (current value: "100", data > >>> source: default, level: 3 user/all, type: int, synonym of: > >>> opal_common_ucx_progress_iterations) > >>> MCA osc ucx: parameter "osc_ucx_opal_mem_hooks" (current > value: "false", data > >>> source: default, level: 3 user/all, type: bool, synonym > of: > >>> opal_common_ucx_opal_mem_hooks) > >>> MCA osc ucx: parameter "osc_ucx_tls" (current value: > >>> "rc_verbs,ud_verbs,rc_mlx5,dc_mlx5,cuda_ipc,rocm_ipc", > data source: default, > >>> level: 3 user/all, type: string, synonym of: > opal_common_ucx_tls) > >>> MCA osc ucx: parameter "osc_ucx_devices" (current value: > "mlx*", data source: > >>> default, level: 3 user/all, type: string, synonym of: > opal_common_ucx_devices) > >>> MCA pml ucx: > --------------------------------------------------- > >>> MCA pml ucx: parameter "pml_ucx_priority" (current value: > "51", data source: > >>> default, level: 3 user/all, type: int) > >>> MCA pml ucx: parameter "pml_ucx_num_disconnect" (current > value: "1", data > >>> source: default, level: 3 user/all, type: int) > >>> MCA pml ucx: parameter "pml_ucx_verbose" (current value: > "0", data source: > >>> default, level: 3 user/all, type: int, synonym of: > opal_common_ucx_verbose) > >>> MCA pml ucx: parameter "pml_ucx_progress_iterations" > (current value: "100", data > >>> source: default, level: 3 user/all, type: int, synonym of: > >>> opal_common_ucx_progress_iterations) > >>> MCA pml ucx: parameter "pml_ucx_opal_mem_hooks" (current > value: "false", data > >>> source: default, level: 3 user/all, type: bool, synonym > of: > >>> opal_common_ucx_opal_mem_hooks) > >>> MCA pml ucx: parameter "pml_ucx_tls" (current value: > >>> "rc_verbs,ud_verbs,rc_mlx5,dc_mlx5,cuda_ipc,rocm_ipc", > data source: default, > >>> level: 3 user/all, type: string, synonym of: > opal_common_ucx_tls) > >>> MCA pml ucx: parameter "pml_ucx_devices" (current value: > "mlx*", data source: > >>> default, level: 3 user/all, type: string, synonym of: > opal_common_ucx_devices) > >>> > >>> $ ompi_info > >>> Package: Open MPI bigpack@verne Distribution > >>> Open MPI: 4.1.1 > >>> Open MPI repo revision: v4.1.1 > >>> Open MPI release date: Apr 24, 2021 > >>> Open RTE: 4.1.1 > >>> Open RTE repo revision: v4.1.1 > >>> Open RTE release date: Apr 24, 2021 > >>> OPAL: 4.1.1 > >>> OPAL repo revision: v4.1.1 > >>> OPAL release date: Apr 24, 2021 > >>> MPI API: 3.1.0 > >>> Ident string: 4.1.1 > >>> Prefix: /usr/beta/openmpi > >>> Configured architecture: x86_64-pc-linux-gnu > >>> Configure host: verne > >>> Configured by: bigpack > >>> Configured on: Tue May 25 17:16:38 UTC 2021 > >>> Configure host: verne > >>> Configure command line: '--enable-ipv6' '--enable-sparse-groups' > >>> '--enable-mpi-ext' '--enable-mpi-cxx' > >>> '--enable-oshmem' '--with-libevent=internal' > >>> '--with-ucx' '--with-pmix=internal' > >>> '--without-libfabric' > '--prefix=/usr/beta/openmpi' > >>> Built by: bigpack > >>> Built on: Tue 25 May 17:57:46 UTC 2021 > >>> Built host: verne > >>> C bindings: yes > >>> C++ bindings: yes > >>> Fort mpif.h: yes (all) > >>> Fort use mpi: yes (full: ignore TKR) > >>> Fort use mpi size: deprecated-ompi-info-value > >>> Fort use mpi_f08: yes > >>> Fort mpi_f08 compliance: The mpi_f08 module is available, but due to > >>> limitations in the gfortran compiler and/or > Open > >>> MPI, does not support the following: array > >>> subsections, direct passthru (where > possible) to > >>> underlying Open MPI's C functionality > >>> Fort mpi_f08 subarrays: no > >>> Java bindings: no > >>> Wrapper compiler rpath: runpath > >>> C compiler: gcc > >>> C compiler absolute: /usr/beta/gcc-trunk/bin/gcc > >>> C compiler family name: GNU > >>> C compiler version: 12.0.0 > >>> C++ compiler: g++ > >>> C++ compiler absolute: /usr/beta/gcc-trunk/bin/g++ > >>> Fort compiler: gfortran > >>> Fort compiler abs: /usr/beta/gcc-trunk/bin/gfortran > >>> Fort ignore TKR: yes (!GCC$ ATTRIBUTES NO_ARG_CHECK ::) > >>> Fort 08 assumed shape: yes > >>> Fort optional args: yes > >>> Fort INTERFACE: yes > >>> Fort ISO_FORTRAN_ENV: yes > >>> Fort STORAGE_SIZE: yes > >>> Fort BIND(C) (all): yes > >>> Fort ISO_C_BINDING: yes > >>> Fort SUBROUTINE BIND(C): yes > >>> Fort TYPE,BIND(C): yes > >>> Fort T,BIND(C,name="a"): yes > >>> Fort PRIVATE: yes > >>> Fort PROTECTED: yes > >>> Fort ABSTRACT: yes > >>> Fort ASYNCHRONOUS: yes > >>> Fort PROCEDURE: yes > >>> Fort USE...ONLY: yes > >>> Fort C_FUNLOC: yes > >>> Fort f08 using wrappers: yes > >>> Fort MPI_SIZEOF: yes > >>> C profiling: yes > >>> C++ profiling: yes > >>> Fort mpif.h profiling: yes > >>> Fort use mpi profiling: yes > >>> Fort use mpi_f08 prof: yes > >>> C++ exceptions: no > >>> Thread support: posix (MPI_THREAD_MULTIPLE: yes, OPAL > support: yes, > >>> OMPI progress: no, ORTE progress: yes, Event > lib: > >>> yes) > >>> Sparse Groups: yes > >>> Internal debug support: no > >>> MPI interface warnings: yes > >>> MPI parameter check: runtime > >>> Memory profiling support: no > >>> Memory debugging support: no > >>> dl support: yes > >>> Heterogeneous support: no > >>> mpirun default --prefix: no > >>> MPI_WTIME support: native > >>> Symbol vis. support: yes > >>> Host topology support: yes > >>> IPv6 support: yes > >>> MPI1 compatibility: no > >>> MPI extensions: affinity, cuda, pcollreq > >>> FT Checkpoint support: no (checkpoint thread: no) > >>> C/R Enabled Debugging: no > >>> MPI_MAX_PROCESSOR_NAME: 256 > >>> MPI_MAX_ERROR_STRING: 256 > >>> MPI_MAX_OBJECT_NAME: 64 > >>> MPI_MAX_INFO_KEY: 36 > >>> MPI_MAX_INFO_VAL: 256 > >>> MPI_MAX_PORT_NAME: 1024 > >>> MPI_MAX_DATAREP_STRING: 128 > >>> MCA allocator: basic (MCA v2.1.0, API v2.0.0, Component > v4.1.1) > >>> MCA allocator: bucket (MCA v2.1.0, API v2.0.0, Component > v4.1.1) > >>> MCA backtrace: execinfo (MCA v2.1.0, API v2.0.0, Component > v4.1.1) > >>> MCA btl: self (MCA v2.1.0, API v3.1.0, Component > v4.1.1) > >>> MCA btl: openib (MCA v2.1.0, API v3.1.0, Component > v4.1.1) > >>> MCA btl: tcp (MCA v2.1.0, API v3.1.0, Component > v4.1.1) > >>> MCA btl: vader (MCA v2.1.0, API v3.1.0, Component > v4.1.1) > >>> MCA compress: bzip (MCA v2.1.0, API v2.0.0, Component > v4.1.1) > >>> MCA compress: gzip (MCA v2.1.0, API v2.0.0, Component > v4.1.1) > >>> MCA crs: none (MCA v2.1.0, API v2.0.0, Component > v4.1.1) > >>> MCA dl: dlopen (MCA v2.1.0, API v1.0.0, Component > v4.1.1) > >>> MCA event: libevent2022 (MCA v2.1.0, API v2.0.0, > Component > >>> v4.1.1) > >>> MCA hwloc: external (MCA v2.1.0, API v2.0.0, Component > v4.1.1) > >>> MCA if: linux_ipv6 (MCA v2.1.0, API v2.0.0, Component > >>> v4.1.1) > >>> MCA if: posix_ipv4 (MCA v2.1.0, API v2.0.0, Component > >>> v4.1.1) > >>> MCA installdirs: env (MCA v2.1.0, API v2.0.0, Component > v4.1.1) > >>> MCA installdirs: config (MCA v2.1.0, API v2.0.0, Component > v4.1.1) > >>> MCA memory: patcher (MCA v2.1.0, API v2.0.0, Component > v4.1.1) > >>> MCA mpool: hugepage (MCA v2.1.0, API v3.0.0, Component > v4.1.1) > >>> MCA patcher: overwrite (MCA v2.1.0, API v1.0.0, Component > >>> v4.1.1) > >>> MCA pmix: isolated (MCA v2.1.0, API v2.0.0, Component > v4.1.1) > >>> MCA pmix: flux (MCA v2.1.0, API v2.0.0, Component > v4.1.1) > >>> MCA pmix: pmix3x (MCA v2.1.0, API v2.0.0, Component > v4.1.1) > >>> MCA pstat: linux (MCA v2.1.0, API v2.0.0, Component > v4.1.1) > >>> MCA rcache: grdma (MCA v2.1.0, API v3.3.0, Component > v4.1.1) > >>> MCA reachable: weighted (MCA v2.1.0, API v2.0.0, Component > v4.1.1) > >>> MCA shmem: mmap (MCA v2.1.0, API v2.0.0, Component > v4.1.1) > >>> MCA shmem: posix (MCA v2.1.0, API v2.0.0, Component > v4.1.1) > >>> MCA shmem: sysv (MCA v2.1.0, API v2.0.0, Component > v4.1.1) > >>> MCA timer: linux (MCA v2.1.0, API v2.0.0, Component > v4.1.1) > >>> MCA errmgr: default_app (MCA v2.1.0, API v3.0.0, > Component > >>> v4.1.1) > >>> MCA errmgr: default_hnp (MCA v2.1.0, API v3.0.0, > Component > >>> v4.1.1) > >>> MCA errmgr: default_orted (MCA v2.1.0, API v3.0.0, > Component > >>> v4.1.1) > >>> MCA errmgr: default_tool (MCA v2.1.0, API v3.0.0, > Component > >>> v4.1.1) > >>> MCA ess: env (MCA v2.1.0, API v3.0.0, Component > v4.1.1) > >>> MCA ess: hnp (MCA v2.1.0, API v3.0.0, Component > v4.1.1) > >>> MCA ess: pmi (MCA v2.1.0, API v3.0.0, Component > v4.1.1) > >>> MCA ess: singleton (MCA v2.1.0, API v3.0.0, Component > >>> v4.1.1) > >>> MCA ess: tool (MCA v2.1.0, API v3.0.0, Component > v4.1.1) > >>> MCA ess: slurm (MCA v2.1.0, API v3.0.0, Component > v4.1.1) > >>> MCA filem: raw (MCA v2.1.0, API v2.0.0, Component > v4.1.1) > >>> MCA grpcomm: direct (MCA v2.1.0, API v3.0.0, Component > v4.1.1) > >>> MCA iof: hnp (MCA v2.1.0, API v2.0.0, Component > v4.1.1) > >>> MCA iof: orted (MCA v2.1.0, API v2.0.0, Component > v4.1.1) > >>> MCA iof: tool (MCA v2.1.0, API v2.0.0, Component > v4.1.1) > >>> MCA odls: default (MCA v2.1.0, API v2.0.0, Component > v4.1.1) > >>> MCA odls: pspawn (MCA v2.1.0, API v2.0.0, Component > v4.1.1) > >>> MCA oob: tcp (MCA v2.1.0, API v2.0.0, Component > v4.1.1) > >>> MCA plm: isolated (MCA v2.1.0, API v2.0.0, Component > v4.1.1) > >>> MCA plm: rsh (MCA v2.1.0, API v2.0.0, Component > v4.1.1) > >>> MCA plm: slurm (MCA v2.1.0, API v2.0.0, Component > v4.1.1) > >>> MCA ras: simulator (MCA v2.1.0, API v2.0.0, Component > >>> v4.1.1) > >>> MCA ras: slurm (MCA v2.1.0, API v2.0.0, Component > v4.1.1) > >>> MCA regx: fwd (MCA v2.1.0, API v1.0.0, Component > v4.1.1) > >>> MCA regx: naive (MCA v2.1.0, API v1.0.0, Component > v4.1.1) > >>> MCA regx: reverse (MCA v2.1.0, API v1.0.0, Component > v4.1.1) > >>> MCA rmaps: mindist (MCA v2.1.0, API v2.0.0, Component > v4.1.1) > >>> MCA rmaps: ppr (MCA v2.1.0, API v2.0.0, Component > v4.1.1) > >>> MCA rmaps: rank_file (MCA v2.1.0, API v2.0.0, Component > >>> v4.1.1) > >>> MCA rmaps: resilient (MCA v2.1.0, API v2.0.0, Component > >>> v4.1.1) > >>> MCA rmaps: round_robin (MCA v2.1.0, API v2.0.0, > Component > >>> v4.1.1) > >>> MCA rmaps: seq (MCA v2.1.0, API v2.0.0, Component > v4.1.1) > >>> MCA rml: oob (MCA v2.1.0, API v3.0.0, Component > v4.1.1) > >>> MCA routed: binomial (MCA v2.1.0, API v3.0.0, Component > v4.1.1) > >>> MCA routed: direct (MCA v2.1.0, API v3.0.0, Component > v4.1.1) > >>> MCA routed: radix (MCA v2.1.0, API v3.0.0, Component > v4.1.1) > >>> MCA rtc: hwloc (MCA v2.1.0, API v1.0.0, Component > v4.1.1) > >>> MCA schizo: flux (MCA v2.1.0, API v1.0.0, Component > v4.1.1) > >>> MCA schizo: ompi (MCA v2.1.0, API v1.0.0, Component > v4.1.1) > >>> MCA schizo: orte (MCA v2.1.0, API v1.0.0, Component > v4.1.1) > >>> MCA schizo: jsm (MCA v2.1.0, API v1.0.0, Component > v4.1.1) > >>> MCA schizo: slurm (MCA v2.1.0, API v1.0.0, Component > v4.1.1) > >>> MCA state: app (MCA v2.1.0, API v1.0.0, Component > v4.1.1) > >>> MCA state: hnp (MCA v2.1.0, API v1.0.0, Component > v4.1.1) > >>> MCA state: novm (MCA v2.1.0, API v1.0.0, Component > v4.1.1) > >>> MCA state: orted (MCA v2.1.0, API v1.0.0, Component > v4.1.1) > >>> MCA state: tool (MCA v2.1.0, API v1.0.0, Component > v4.1.1) > >>> MCA bml: r2 (MCA v2.1.0, API v2.0.0, Component v4.1.1) > >>> MCA coll: adapt (MCA v2.1.0, API v2.0.0, Component > v4.1.1) > >>> MCA coll: basic (MCA v2.1.0, API v2.0.0, Component > v4.1.1) > >>> MCA coll: han (MCA v2.1.0, API v2.0.0, Component > v4.1.1) > >>> MCA coll: inter (MCA v2.1.0, API v2.0.0, Component > v4.1.1) > >>> MCA coll: libnbc (MCA v2.1.0, API v2.0.0, Component > v4.1.1) > >>> MCA coll: self (MCA v2.1.0, API v2.0.0, Component > v4.1.1) > >>> MCA coll: sm (MCA v2.1.0, API v2.0.0, Component v4.1.1) > >>> MCA coll: sync (MCA v2.1.0, API v2.0.0, Component > v4.1.1) > >>> MCA coll: tuned (MCA v2.1.0, API v2.0.0, Component > v4.1.1) > >>> MCA coll: monitoring (MCA v2.1.0, API v2.0.0, Component > >>> v4.1.1) > >>> MCA fbtl: posix (MCA v2.1.0, API v2.0.0, Component > v4.1.1) > >>> MCA fcoll: dynamic (MCA v2.1.0, API v2.0.0, Component > v4.1.1) > >>> MCA fcoll: dynamic_gen2 (MCA v2.1.0, API v2.0.0, > Component > >>> v4.1.1) > >>> MCA fcoll: individual (MCA v2.1.0, API v2.0.0, Component > >>> v4.1.1) > >>> MCA fcoll: two_phase (MCA v2.1.0, API v2.0.0, Component > >>> v4.1.1) > >>> MCA fcoll: vulcan (MCA v2.1.0, API v2.0.0, Component > v4.1.1) > >>> MCA fs: ufs (MCA v2.1.0, API v2.0.0, Component > v4.1.1) > >>> MCA io: ompio (MCA v2.1.0, API v2.0.0, Component > v4.1.1) > >>> MCA io: romio321 (MCA v2.1.0, API v2.0.0, Component > v4.1.1) > >>> MCA op: avx (MCA v2.1.0, API v1.0.0, Component > v4.1.1) > >>> MCA osc: sm (MCA v2.1.0, API v3.0.0, Component v4.1.1) > >>> MCA osc: monitoring (MCA v2.1.0, API v3.0.0, Component > >>> v4.1.1) > >>> MCA osc: pt2pt (MCA v2.1.0, API v3.0.0, Component > v4.1.1) > >>> MCA osc: rdma (MCA v2.1.0, API v3.0.0, Component > v4.1.1) > >>> MCA osc: ucx (MCA v2.1.0, API v3.0.0, Component > v4.1.1) > >>> MCA pml: v (MCA v2.1.0, API v2.0.0, Component v4.1.1) > >>> MCA pml: cm (MCA v2.1.0, API v2.0.0, Component v4.1.1) > >>> MCA pml: monitoring (MCA v2.1.0, API v2.0.0, Component > >>> v4.1.1) > >>> MCA pml: ob1 (MCA v2.1.0, API v2.0.0, Component > v4.1.1) > >>> MCA pml: ucx (MCA v2.1.0, API v2.0.0, Component > v4.1.1) > >>> MCA rte: orte (MCA v2.1.0, API v2.0.0, Component > v4.1.1) > >>> MCA sharedfp: individual (MCA v2.1.0, API v2.0.0, Component > >>> v4.1.1) > >>> MCA sharedfp: lockedfile (MCA v2.1.0, API v2.0.0, Component > >>> v4.1.1) > >>> MCA sharedfp: sm (MCA v2.1.0, API v2.0.0, Component v4.1.1) > >>> MCA topo: basic (MCA v2.1.0, API v2.2.0, Component > v4.1.1) > >>> MCA topo: treematch (MCA v2.1.0, API v2.2.0, Component > >>> v4.1.1) > >>> MCA vprotocol: pessimist (MCA v2.1.0, API v2.0.0, Component > >>> v4.1.1) > >>> > > > > #end >
