Hi Gilles,

thank you very much for identifying the reason for the problem
and fixing it.

Have a nice weekend

Siegmar

Am 29.04.2016 um 03:38 schrieb Gilles Gouaillardet:
Siegmar,

in pmix_bfrop_pack_app,
app->argc
must be replaced with
app[i].argc

I will PR to pmix, ompi and ompi-release when I am back at work on Monday

Cheers,

Gilles

On Thursday, April 28, 2016, Gilles Gouaillardet <gil...@rist.or.jp
<mailto:gil...@rist.or.jp>> wrote:

    Siegmar,


    can you please also post the source of spawn_slave ?


    Cheers,

    Gilles


    On 4/28/2016 1:17 AM, Siegmar Gross wrote:
    Hi Gilles,

    it is not necessary to have a heterogeneous environment to reproduce
    the error as you can see below. All machines are 64 bit.

    tyr spawn 119 ompi_info | grep -e "OPAL repo revision" -e "C compiler
    absolute"
          OPAL repo revision: v2.x-dev-1290-gbd0e4e1
         C compiler absolute: /usr/local/gcc-5.1.0/bin/gcc
    tyr spawn 120 uname -a
    SunOS tyr.informatik.hs-fulda.de <http://tyr.informatik.hs-fulda.de>
    5.10 Generic_150400-11 sun4u sparc SUNW,A70 Solaris
    tyr spawn 121 mpiexec -np 1 --host tyr,tyr,tyr,tyr spawn_multiple_master

    Parent process 0 running on tyr.informatik.hs-fulda.de
    <http://tyr.informatik.hs-fulda.de>
      I create 3 slave processes.

    [tyr.informatik.hs-fulda.de:27286
    <http://tyr.informatik.hs-fulda.de:27286>] PMIX ERROR: UNPACK-PAST-END
    in file
    
../../../../../../openmpi-v2.x-dev-1290-gbd0e4e1/opal/mca/pmix/pmix112/pmix/src/server/pmix_server_ops.c
    at line 829
    [tyr.informatik.hs-fulda.de:27286
    <http://tyr.informatik.hs-fulda.de:27286>] PMIX ERROR: UNPACK-PAST-END
    in file
    
../../../../../../openmpi-v2.x-dev-1290-gbd0e4e1/opal/mca/pmix/pmix112/pmix/src/server/pmix_server.c
    at line 2176
    [tyr:27288] *** An error occurred in MPI_Comm_spawn_multiple
    [tyr:27288] *** reported by process [3434086401,0]
    [tyr:27288] *** on communicator MPI_COMM_WORLD
    [tyr:27288] *** MPI_ERR_SPAWN: could not spawn processes
    [tyr:27288] *** MPI_ERRORS_ARE_FATAL (processes in this communicator
    will now abort,
    [tyr:27288] ***    and potentially your MPI job)
    tyr spawn 122






    sunpc1 fd1026 105 ompi_info | grep -e "OPAL repo revision" -e "C
    compiler absolute"
          OPAL repo revision: v2.x-dev-1290-gbd0e4e1
         C compiler absolute: /usr/local/gcc-5.1.0/bin/gcc
    sunpc1 fd1026 106 uname -a
    SunOS sunpc1 5.10 Generic_147441-21 i86pc i386 i86pc Solaris
    sunpc1 fd1026 107 mpiexec -np 1 --host sunpc1,sunpc1,sunpc1,sunpc1
    spawn_multiple_master

    Parent process 0 running on sunpc1
      I create 3 slave processes.

    [sunpc1:00368] PMIX ERROR: UNPACK-PAST-END in file
    
../../../../../../openmpi-v2.x-dev-1290-gbd0e4e1/opal/mca/pmix/pmix112/pmix/src/server/pmix_server_ops.c
    at line 829
    [sunpc1:00368] PMIX ERROR: UNPACK-PAST-END in file
    
../../../../../../openmpi-v2.x-dev-1290-gbd0e4e1/opal/mca/pmix/pmix112/pmix/src/server/pmix_server.c
    at line 2176
    [sunpc1:370] *** An error occurred in MPI_Comm_spawn_multiple
    [sunpc1:370] *** reported by process [43909121,0]
    [sunpc1:370] *** on communicator MPI_COMM_WORLD
    [sunpc1:370] *** MPI_ERR_SPAWN: could not spawn processes
    [sunpc1:370] *** MPI_ERRORS_ARE_FATAL (processes in this communicator
    will now abort,
    [sunpc1:370] ***    and potentially your MPI job)
    sunpc1 fd1026 108





    linpc1 fd1026 105 ompi_info | grep -e "OPAL repo revision" -e "C
    compiler absolute"
          OPAL repo revision: v2.x-dev-1290-gbd0e4e1
         C compiler absolute: /usr/local/gcc-5.1.0/bin/gcc
    linpc1 fd1026 106 uname -a
    Linux linpc1 3.1.10-1.29-desktop #1 SMP PREEMPT Fri May 31 20:10:04 UTC
    2013 (2529847) x86_64 x86_64 x86_64 GNU/Linux
    linpc1 fd1026 107 mpiexec -np 1 --host linpc1,linpc1,linpc1,linpc1
    spawn_multiple_master

    Parent process 0 running on linpc1
      I create 3 slave processes.

    [linpc1:21502] PMIX ERROR: UNPACK-PAST-END in file
    
../../../../../../openmpi-v2.x-dev-1290-gbd0e4e1/opal/mca/pmix/pmix112/pmix/src/server/pmix_server_ops.c
    at line 829
    [linpc1:21502] PMIX ERROR: UNPACK-PAST-END in file
    
../../../../../../openmpi-v2.x-dev-1290-gbd0e4e1/opal/mca/pmix/pmix112/pmix/src/server/pmix_server.c
    at line 2176
    [linpc1:21507] *** An error occurred in MPI_Comm_spawn_multiple
    [linpc1:21507] *** reported by process [1005518849,0]
    [linpc1:21507] *** on communicator MPI_COMM_WORLD
    [linpc1:21507] *** MPI_ERR_SPAWN: could not spawn processes
    [linpc1:21507] *** MPI_ERRORS_ARE_FATAL (processes in this communicator
    will now abort,
    [linpc1:21507] ***    and potentially your MPI job)
    linpc1 fd1026 108


    I used the following configure command.

    ../openmpi-v2.x-dev-1290-gbd0e4e1/configure \
      --prefix=/usr/local/openmpi-2.0.0_64_gcc \
      --libdir=/usr/local/openmpi-2.0.0_64_gcc/lib64 \
      --with-jdk-bindir=/usr/local/jdk1.8.0/bin \
      --with-jdk-headers=/usr/local/jdk1.8.0/include \
      JAVA_HOME=/usr/local/jdk1.8.0 \
      LDFLAGS="-m64" CC="gcc" CXX="g++" FC="gfortran" \
      CFLAGS="-m64" CXXFLAGS="-m64" FCFLAGS="-m64" \
      CPP="cpp" CXXCPP="cpp" \
      --enable-mpi-cxx \
      --enable-cxx-exceptions \
      --enable-mpi-java \
      --enable-heterogeneous \
      --enable-mpi-thread-multiple \
      --with-hwloc=internal \
      --without-verbs \
      --with-wrapper-cflags="-std=c11 -m64" \
      --with-wrapper-cxxflags="-m64" \
      --with-wrapper-fcflags="-m64" \
      --enable-debug \
      |& tee log.configure.$SYSTEM_ENV.$MACHINE_ENV.64_gcc


    Kind regards

    Siegmar



    Am 27.04.2016 um 13:21 schrieb Gilles Gouaillardet:
    Siegmar,

    please add this to your CFLAGS for the time being.

    configure tries to detect which flags must be added for C99 support,
    and it seems
    the test is not working for Solaris 10 and Oracle compilers.
    this is no more a widely used environment, and I am not sure I can find
    the
    time to fix this
    in a near future.


    regarding the runtime issue, can you please describe your 4 hosts (os,
    endianness and bitness)

    Cheers,

    Gilles

    On Wednesday, April 27, 2016, Siegmar Gross
    <siegmar.gr...@informatik.hs-fulda.de
    <javascript:_e(%7B%7D,'cvml','siegmar.gr...@informatik.hs-fulda.de');>
    <javascript:_e(%7B%7D,'cvml','siegmar.gr...@informatik.hs-fulda.de');>>
    wrote:

        Hi Gilles,

        adding "-std=c99" to CFLAGS solves the problem with the missing
    library.
        Shall I add it permanently to my configure command or will you add it,
        so that I will not run into problems if you need the C11 standard
    later?

        "spawn_multiple_master" breaks with the same error that I reported
        yesterday for my gcc-version of Open MPI. Hopefully you can solve the
        problem as well.


        Kind regards and thank you very much for your help

        Siegmar


        Am 27.04.2016 um 08:05 schrieb Gilles Gouaillardet:

            Siegmar,


            here is the error :

            configure:17969: cc -o conftest -m64 -D_REENTRANT -g  -g
            -I/export2/src/openmpi-2.0.0/openmpi-v2.x-dev-1290-gbd0e4e1

    
-I/export2/src/openmpi-2.0.0/openmpi-v2.x-dev-1290-gbd0e4e1-SunOS.sparc.64_cc

    -I/export2/src/openmpi-2.0.0/openmpi-v2.x-dev-1290-gbd0e4e1/opal/include

    
-I/export2/src/openmpi-2.0.0/openmpi-v2.x-dev-1290-gbd0e4e1-SunOS.sparc.64_cc/opal/include
            -D_REENTRANT

    
-I/export2/src/openmpi-2.0.0/openmpi-v2.x-dev-1290-gbd0e4e1/opal/mca/hwloc/hwloc1112/hwloc/include

    
-I/export2/src/openmpi-2.0.0/openmpi-v2.x-dev-1290-gbd0e4e1-SunOS.sparc.64_cc/opal/mca/hwloc/hwloc1112/hwloc/include

    
-I/export2/src/openmpi-2.0.0/openmpi-v2.x-dev-1290-gbd0e4e1/opal/mca/event/libevent2022/libevent

    
-I/export2/src/openmpi-2.0.0/openmpi-v2.x-dev-1290-gbd0e4e1/opal/mca/event/libevent2022/libevent/include

    
-I/export2/src/openmpi-2.0.0/openmpi-v2.x-dev-1290-gbd0e4e1-SunOS.sparc.64_cc/opal/mca/event/libevent2022/libevent/include
            -m64 conftest.c  >&5
            "/usr/include/stdbool.h", line 42: #error: "Use of <stdbool.h> is
            valid only
            in a c99 compilation environment."


            i cannot reproduce this on solaris 11 with oracle studio 5.3
    compiler,
            and i
            do not have solaris 10 yet.

            could you please re-configure with '-std=c99' appended to your
    CFLAGS
            and see
            if it helps ?


            Cheers,


            Gilles


            On 4/26/2016 7:57 PM, Siegmar Gross wrote:

                Hi Gilles and Ralph,

                I was able to sort out my mess. In my last email I compared
    the
                files from "SunOS_sparc/openmpi-2.0.0_64_gcc/lib64/openmpi"
    from
                the attachment of my email to Ralph with the files from
                "SunOS_sparc/openmpi-2.0.0_64_cc/lib64/openmpi" from my
    current
                file system. That's the reason while I have had different
                timestamps. The other problem was that Ralph didn't recognize
                that "mca_pmix_pmix112.so" wasn't built on Solaris with the
                Sun C compiler. I've removed most of the files from the
    attachment
                of my email so that it is easier to see the relevant files.
    Below
                I try to give you more information that may be relevant to
    track
                down the problem. I still get an error running one of my small
                test programs, when I use my gcc-version of Open MPI.
                "mca_pmix_pmix112.so" is a 64 bits library.

                Linux_x86_64/openmpi-2.0.0_64_cc/lib64/openmpi:
                ...
                -rwxr-xr-x 1 root root  261327 Apr 19 16:46 mca_plm_slurm.so
                -rwxr-xr-x 1 root root    1002 Apr 19 16:45
    mca_pmix_pmix112.la <http://mca_pmix_pmix112.la>
                <http://mca_pmix_pmix112.la> <http://mca_pmix_pmix112.la>
                -rwxr-xr-x 1 root root 3906526 Apr 19 16:45
    mca_pmix_pmix112.so
                -rwxr-xr-x 1 root root     966 Apr 19 16:51 mca_pml_cm.la
    <http://mca_pml_cm.la>
                <http://mca_pml_cm.la> <http://mca_pml_cm.la>
                -rwxr-xr-x 1 root root 1574265 Apr 19 16:51 mca_pml_cm.so
                ...

                Linux_x86_64/openmpi-2.0.0_64_gcc/lib64/openmpi:
                ...
                -rwxr-xr-x 1 root root   70371 Apr 19 16:43 mca_plm_slurm.so
                -rwxr-xr-x 1 root root    1008 Apr 19 16:42
    mca_pmix_pmix112.la <http://mca_pmix_pmix112.la>
                <http://mca_pmix_pmix112.la> <http://mca_pmix_pmix112.la>
                -rwxr-xr-x 1 root root 1029005 Apr 19 16:42
    mca_pmix_pmix112.so
                -rwxr-xr-x 1 root root     972 Apr 19 16:46 mca_pml_cm.la
    <http://mca_pml_cm.la>
                <http://mca_pml_cm.la> <http://mca_pml_cm.la>
                -rwxr-xr-x 1 root root  284858 Apr 19 16:46 mca_pml_cm.so
                ...

                SunOS_sparc/openmpi-2.0.0_64_cc/lib64/openmpi:
                ...
                -rwxr-xr-x 1 root root  319816 Apr 19 19:58 mca_plm_rsh.so
                -rwxr-xr-x 1 root root     970 Apr 19 20:00 mca_pml_cm.la
    <http://mca_pml_cm.la>
                <http://mca_pml_cm.la> <http://mca_pml_cm.la>
                -rwxr-xr-x 1 root root 1507440 Apr 19 20:00 mca_pml_cm.so
                ...

                SunOS_sparc/openmpi-2.0.0_64_gcc/lib64/openmpi:
                ...
                -rwxr-xr-x 1 root root  153280 Apr 19 19:49 mca_plm_rsh.so
                -rwxr-xr-x 1 root root    1007 Apr 19 19:47
    mca_pmix_pmix112.la <http://mca_pmix_pmix112.la>
                <http://mca_pmix_pmix112.la> <http://mca_pmix_pmix112.la>
                -rwxr-xr-x 1 root root 1400512 Apr 19 19:47
    mca_pmix_pmix112.so
                -rwxr-xr-x 1 root root     971 Apr 19 19:52 mca_pml_cm.la
    <http://mca_pml_cm.la>
                <http://mca_pml_cm.la> <http://mca_pml_cm.la>
                -rwxr-xr-x 1 root root  342440 Apr 19 19:52 mca_pml_cm.so
                ...

                SunOS_x86_64/openmpi-2.0.0_64_cc/lib64/openmpi:
                ...
                -rwxr-xr-x 1 root root  300096 Apr 19 17:18 mca_plm_rsh.so
                -rwxr-xr-x 1 root root     970 Apr 19 17:23 mca_pml_cm.la
    <http://mca_pml_cm.la>
                <http://mca_pml_cm.la> <http://mca_pml_cm.la>
                -rwxr-xr-x 1 root root 1458816 Apr 19 17:23 mca_pml_cm.so
                ...

                SunOS_x86_64/openmpi-2.0.0_64_gcc/lib64/openmpi:
                ...
                -rwxr-xr-x 1 root root  133096 Apr 19 17:42 mca_plm_rsh.so
                -rwxr-xr-x 1 root root    1007 Apr 19 17:41
    mca_pmix_pmix112.la <http://mca_pmix_pmix112.la>
                <http://mca_pmix_pmix112.la> <http://mca_pmix_pmix112.la>
                -rwxr-xr-x 1 root root 1320240 Apr 19 17:41
    mca_pmix_pmix112.so
                -rwxr-xr-x 1 root root     971 Apr 19 17:46 mca_pml_cm.la
    <http://mca_pml_cm.la>
                <http://mca_pml_cm.la> <http://mca_pml_cm.la>
                -rwxr-xr-x 1 root root  419848 Apr 19 17:46 mca_pml_cm.so
                ...


                Yesterday I've installed openmpi-v2.x-dev-1290-gbd0e4e1 so
    that we
                have a current version for the investigation of the
    problem. Once
                more mca_pmix_pmix112.so isn't available on Solaris if I
    use the
                Sun C compiler.

                "config.log" for gcc-5.1.0 shows the following.

                ...
                configure:127799: /bin/bash

    '../../../../../../openmpi-v2.x-dev-1290-gbd0e4e1/opal/mca/pmix/pmix112/
                pmix/configure' succeeded for opal/mca/pmix/pmix112/pmix
                configure:127916: checking if MCA component pmix:pmix112
    can compile
                configure:127918: result: yes
                configure:5637: --- MCA component pmix:external (m4
    configuration
                macro)
                configure:128523: checking for MCA component pmix:external
    compile
                mode
                configure:128529: result: dso
                configure:129054: checking if MCA component pmix:external
    can compile
                configure:129056: result: no
                ...
                config.status:3897: creating opal/mca/pmix/Makefile
                config.status:3897: creating opal/mca/pmix/s1/Makefile
                config.status:3897: creating opal/mca/pmix/cray/Makefile
                config.status:3897: creating opal/mca/pmix/s2/Makefile
                config.status:3897: creating opal/mca/pmix/pmix112/Makefile
                config.status:3897: creating opal/mca/pmix/external/Makefile
                ...
                MCA_BUILD_opal_pmix_cray_DSO_FALSE='#'
                MCA_BUILD_opal_pmix_cray_DSO_TRUE=''
                MCA_BUILD_opal_pmix_external_DSO_FALSE='#'
                MCA_BUILD_opal_pmix_external_DSO_TRUE=''
                MCA_BUILD_opal_pmix_pmix112_DSO_FALSE='#'
                MCA_BUILD_opal_pmix_pmix112_DSO_TRUE=''
                MCA_BUILD_opal_pmix_s1_DSO_FALSE='#'
                MCA_BUILD_opal_pmix_s1_DSO_TRUE=''
                MCA_BUILD_opal_pmix_s2_DSO_FALSE='#'
                MCA_BUILD_opal_pmix_s2_DSO_TRUE=''
                ...
                MCA_opal_FRAMEWORKS='common  allocator backtrace btl dl
    event hwloc if
                installdirs memchecker memcpy memory mpool pmix pstat
    rcache sec
                shmem timer'
                MCA_opal_FRAMEWORKS_SUBDIRS='mca/common  mca/allocator
                mca/backtrace mca/btl
                mca/dl mca/event mca/hwloc mca/if mca/installdirs
    mca/memchecker
                mca/memcpy
                mca/memory mca/mpool mca/pmix mca/pstat mca/rcache mca/sec
                mca/shmem mca/timer'

    MCA_opal_FRAMEWORK_COMPONENT_ALL_SUBDIRS='$(MCA_opal_common_ALL_SUBDIRS)
                $(MCA_opal_allocator_ALL_SUBDIRS)
    $(MCA_opal_backtrace_ALL_SUBDIRS)
                $(MCA_opal_btl_ALL_SUBDIRS) $(MCA_opal_dl_ALL_SUBDIRS)
                $(MCA_opal_event_ALL_SUBDIRS) $(MCA_opal_hwloc_ALL_SUBDIRS)
                $(MCA_opal_if_ALL_SUBDIRS) $(MCA_opal_installdirs_ALL_SUBDIRS)
                $(MCA_opal_memchecker_ALL_SUBDIRS)
    $(MCA_opal_memcpy_ALL_SUBDIRS)
                $(MCA_opal_memory_ALL_SUBDIRS) $(MCA_opal_mpool_ALL_SUBDIRS)
                $(MCA_opal_pmix_ALL_SUBDIRS) $(MCA_opal_pstat_ALL_SUBDIRS)
                $(MCA_opal_rcache_ALL_SUBDIRS) $(MCA_opal_sec_ALL_SUBDIRS)
                $(MCA_opal_shmem_ALL_SUBDIRS) $(MCA_opal_timer_ALL_SUBDIRS)'

    MCA_opal_FRAMEWORK_COMPONENT_DSO_SUBDIRS='$(MCA_opal_common_DSO_SUBDIRS)
                $(MCA_opal_allocator_DSO_SUBDIRS)
    $(MCA_opal_backtrace_DSO_SUBDIRS)
                $(MCA_opal_btl_DSO_SUBDIRS) $(MCA_opal_dl_DSO_SUBDIRS)
                $(MCA_opal_event_DSO_SUBDIRS) $(MCA_opal_hwloc_DSO_SUBDIRS)
                $(MCA_opal_if_DSO_SUBDIRS) $(MCA_opal_installdirs_DSO_SUBDIRS)
                $(MCA_opal_memchecker_DSO_SUBDIRS)
    $(MCA_opal_memcpy_DSO_SUBDIRS)
                $(MCA_opal_memory_DSO_SUBDIRS) $(MCA_opal_mpool_DSO_SUBDIRS)
                $(MCA_opal_pmix_DSO_SUBDIRS) $(MCA_opal_pstat_DSO_SUBDIRS)
                $(MCA_opal_rcache_DSO_SUBDIRS) $(MCA_opal_sec_DSO_SUBDIRS)
                $(MCA_opal_shmem_DSO_SUBDIRS) $(MCA_opal_timer_DSO_SUBDIRS)'

    
MCA_opal_FRAMEWORK_COMPONENT_STATIC_SUBDIRS='$(MCA_opal_common_STATIC_SUBDIRS)
                 $(MCA_opal_allocator_STATIC_SUBDIRS)
                $(MCA_opal_backtrace_STATIC_SUBDIRS)
                $(MCA_opal_btl_STATIC_SUBDIRS) $(MCA_opal_dl_STATIC_SUBDIRS)
                $(MCA_opal_event_STATIC_SUBDIRS)
    $(MCA_opal_hwloc_STATIC_SUBDIRS)
                $(MCA_opal_if_STATIC_SUBDIRS)
    $(MCA_opal_installdirs_STATIC_SUBDIRS)
                $(MCA_opal_memchecker_STATIC_SUBDIRS)
                $(MCA_opal_memcpy_STATIC_SUBDIRS)
                $(MCA_opal_memory_STATIC_SUBDIRS)
    $(MCA_opal_mpool_STATIC_SUBDIRS)
                $(MCA_opal_pmix_STATIC_SUBDIRS)
    $(MCA_opal_pstat_STATIC_SUBDIRS)
                $(MCA_opal_rcache_STATIC_SUBDIRS)
    $(MCA_opal_sec_STATIC_SUBDIRS)
                $(MCA_opal_shmem_STATIC_SUBDIRS)
    $(MCA_opal_timer_STATIC_SUBDIRS)'
                MCA_opal_FRAMEWORK_LIBS=' $(MCA_opal_common_STATIC_LTLIBS)
                mca/allocator/libmca_allocator.la
    <http://libmca_allocator.la> <http://libmca_allocator.la>
    <http://libmca_allocator.la>
                $(MCA_opal_allocator_STATIC_LTLIBS)
                mca/backtrace/libmca_backtrace.la
    <http://libmca_backtrace.la> <http://libmca_backtrace.la>
    <http://libmca_backtrace.la>
                $(MCA_opal_backtrace_STATIC_LTLIBS)
                mca/btl/libmca_btl.la <http://libmca_btl.la>
    <http://libmca_btl.la> <http://libmca_btl.la>
                $(MCA_opal_btl_STATIC_LTLIBS) mca/dl/libmca_dl.la
    <http://libmca_dl.la>
                <http://libmca_dl.la> <http://libmca_dl.la>
                $(MCA_opal_dl_STATIC_LTLIBS) mca/event/libmca_event.la
    <http://libmca_event.la>
                <http://libmca_event.la> <http://libmca_event.la>
                $(MCA_opal_event_STATIC_LTLIBS) mca/hwloc/libmca_hwloc.la
    <http://libmca_hwloc.la>
                <http://libmca_hwloc.la> <http://libmca_hwloc.la>
                $(MCA_opal_hwloc_STATIC_LTLIBS) mca/if/libmca_if.la
    <http://libmca_if.la>
                <http://libmca_if.la> <http://libmca_if.la>
                $(MCA_opal_if_STATIC_LTLIBS)
    mca/installdirs/libmca_installdirs.la <http://libmca_installdirs.la>
                <http://libmca_installdirs.la> <http://libmca_installdirs.la>
                $(MCA_opal_installdirs_STATIC_LTLIBS)
                mca/memchecker/libmca_memchecker.la
    <http://libmca_memchecker.la> <http://libmca_memchecker.la>
    <http://libmca_memchecker.la>
                $(MCA_opal_memchecker_STATIC_LTLIBS)
    mca/memcpy/libmca_memcpy.la <http://libmca_memcpy.la>
                <http://libmca_memcpy.la> <http://libmca_memcpy.la>
                $(MCA_opal_memcpy_STATIC_LTLIBS)
    mca/memory/libmca_memory.la <http://libmca_memory.la>
                <http://libmca_memory.la> <http://libmca_memory.la>
                $(MCA_opal_memory_STATIC_LTLIBS) mca/mpool/libmca_mpool.la
    <http://libmca_mpool.la>
                <http://libmca_mpool.la> <http://libmca_mpool.la>
                $(MCA_opal_mpool_STATIC_LTLIBS) mca/pmix/libmca_pmix.la
    <http://libmca_pmix.la>
                <http://libmca_pmix.la> <http://libmca_pmix.la>
                $(MCA_opal_pmix_STATIC_LTLIBS) mca/pstat/libmca_pstat.la
    <http://libmca_pstat.la>
                <http://libmca_pstat.la> <http://libmca_pstat.la>
                $(MCA_opal_pstat_STATIC_LTLIBS) mca/rcache/libmca_rcache.la
    <http://libmca_rcache.la>
                <http://libmca_rcache.la> <http://libmca_rcache.la>
                $(MCA_opal_rcache_STATIC_LTLIBS) mca/sec/libmca_sec.la
    <http://libmca_sec.la>
                <http://libmca_sec.la> <http://libmca_sec.la>
                $(MCA_opal_sec_STATIC_LTLIBS) mca/shmem/libmca_shmem.la
    <http://libmca_shmem.la>
                <http://libmca_shmem.la> <http://libmca_shmem.la>
                $(MCA_opal_shmem_STATIC_LTLIBS) mca/timer/libmca_timer.la
    <http://libmca_timer.la>
                <http://libmca_timer.la> <http://libmca_timer.la>
                $(MCA_opal_timer_STATIC_LTLIBS)'
                ...
                MCA_opal_pmix_ALL_COMPONENTS=' s1 cray s2 pmix112 external'
                MCA_opal_pmix_ALL_SUBDIRS=' mca/pmix/s1 mca/pmix/cray
    mca/pmix/s2
                mca/pmix/pmix112 mca/pmix/external'
                MCA_opal_pmix_DSO_COMPONENTS=' pmix112'
                MCA_opal_pmix_DSO_SUBDIRS=' mca/pmix/pmix112'
                MCA_opal_pmix_STATIC_COMPONENTS=''
                MCA_opal_pmix_STATIC_LTLIBS=''
                MCA_opal_pmix_STATIC_SUBDIRS=''
                ...
                opal_pmix_ext_CPPFLAGS=''
                opal_pmix_ext_LDFLAGS=''
                opal_pmix_ext_LIBS=''

    
opal_pmix_pmix112_CPPFLAGS='-I$(OPAL_TOP_BUILDDIR)/opal/mca/pmix/pmix112/pmix/include/pmix
                -I$(OPAL_TOP_BUILDDIR)/opal/mca/pmix/pmix112/pmix/include
                -I$(OPAL_TOP_BUILDDIR)/opal/mca/pmix/pmix112/pmix
                -I$(OPAL_TOP_SRCDIR)/opal/mca/pmix/pmix112/pmix'

    
opal_pmix_pmix112_LIBS='$(OPAL_TOP_BUILDDIR)/opal/mca/pmix/pmix112/pmix/libpmix.la
    <http://libpmix.la>
                <http://libpmix.la> <http://libpmix.la>'

                ...



                "config.log" for Sun C 5.13 shows the following.

                ...
                configure:127803: /bin/bash

    '../../../../../../openmpi-v2.x-dev-1290-gbd0e4e1/opal/mca/pmix/pmix112/
                pmix/configure' *failed* for opal/mca/pmix/pmix112/pmix
                configure:128379: checking if MCA component pmix:pmix112
    can compile
                configure:128381: result: no
                configure:5637: --- MCA component pmix:external (m4
    configuration
                macro)
                configure:128523: checking for MCA component pmix:external
    compile
                mode
                configure:128529: result: dso
                configure:129054: checking if MCA component pmix:external
    can compile
                configure:129056: result: no
                ...
                config.status:3887: creating opal/mca/pmix/Makefile
                config.status:3887: creating opal/mca/pmix/s1/Makefile
                config.status:3887: creating opal/mca/pmix/cray/Makefile
                config.status:3887: creating opal/mca/pmix/s2/Makefile
                config.status:3887: creating opal/mca/pmix/pmix112/Makefile
                config.status:3887: creating opal/mca/pmix/external/Makefile
                ...
                MCA_BUILD_opal_pmix_cray_DSO_FALSE='#'
                MCA_BUILD_opal_pmix_cray_DSO_TRUE=''
                MCA_BUILD_opal_pmix_external_DSO_FALSE='#'
                MCA_BUILD_opal_pmix_external_DSO_TRUE=''
                MCA_BUILD_opal_pmix_pmix112_DSO_FALSE='#'
                MCA_BUILD_opal_pmix_pmix112_DSO_TRUE=''
                MCA_BUILD_opal_pmix_s1_DSO_FALSE='#'
                MCA_BUILD_opal_pmix_s1_DSO_TRUE=''
                MCA_BUILD_opal_pmix_s2_DSO_FALSE='#'
                MCA_BUILD_opal_pmix_s2_DSO_TRUE=''
                ...
                MCA_opal_FRAMEWORKS='common  allocator backtrace btl dl
    event hwloc if
                installdirs memchecker memcpy memory mpool pmix pstat
    rcache sec
                shmem timer'
                MCA_opal_FRAMEWORKS_SUBDIRS='mca/common  mca/allocator
                mca/backtrace mca/btl
                mca/dl mca/event mca/hwloc mca/if mca/installdirs
    mca/memchecker
                mca/memcpy
                mca/memory mca/mpool mca/pmix mca/pstat mca/rcache mca/sec
                mca/shmem mca/timer'

    MCA_opal_FRAMEWORK_COMPONENT_ALL_SUBDIRS='$(MCA_opal_common_ALL_SUBDIRS)
                $(MCA_opal_allocator_ALL_SUBDIRS)
    $(MCA_opal_backtrace_ALL_SUBDIRS)
                $(MCA_opal_btl_ALL_SUBDIRS) $(MCA_opal_dl_ALL_SUBDIRS)
                $(MCA_opal_event_ALL_SUBDIRS) $(MCA_opal_hwloc_ALL_SUBDIRS)
                $(MCA_opal_if_ALL_SUBDIRS) $(MCA_opal_installdirs_ALL_SUBDIRS)
                $(MCA_opal_memchecker_ALL_SUBDIRS)
    $(MCA_opal_memcpy_ALL_SUBDIRS)
                $(MCA_opal_memory_ALL_SUBDIRS) $(MCA_opal_mpool_ALL_SUBDIRS)
                $(MCA_opal_pmix_ALL_SUBDIRS) $(MCA_opal_pstat_ALL_SUBDIRS)
                $(MCA_opal_rcache_ALL_SUBDIRS) $(MCA_opal_sec_ALL_SUBDIRS)
                $(MCA_opal_shmem_ALL_SUBDIRS) $(MCA_opal_timer_ALL_SUBDIRS)'

    MCA_opal_FRAMEWORK_COMPONENT_DSO_SUBDIRS='$(MCA_opal_common_DSO_SUBDIRS)
                $(MCA_opal_allocator_DSO_SUBDIRS)
    $(MCA_opal_backtrace_DSO_SUBDIRS)
                $(MCA_opal_btl_DSO_SUBDIRS) $(MCA_opal_dl_DSO_SUBDIRS)
                $(MCA_opal_event_DSO_SUBDIRS) $(MCA_opal_hwloc_DSO_SUBDIRS)
                $(MCA_opal_if_DSO_SUBDIRS) $(MCA_opal_installdirs_DSO_SUBDIRS)
                $(MCA_opal_memchecker_DSO_SUBDIRS)
    $(MCA_opal_memcpy_DSO_SUBDIRS)
                $(MCA_opal_memory_DSO_SUBDIRS) $(MCA_opal_mpool_DSO_SUBDIRS)
                $(MCA_opal_pmix_DSO_SUBDIRS) $(MCA_opal_pstat_DSO_SUBDIRS)
                $(MCA_opal_rcache_DSO_SUBDIRS) $(MCA_opal_sec_DSO_SUBDIRS)
                $(MCA_opal_shmem_DSO_SUBDIRS) $(MCA_opal_timer_DSO_SUBDIRS)'

    
MCA_opal_FRAMEWORK_COMPONENT_STATIC_SUBDIRS='$(MCA_opal_common_STATIC_SUBDIRS)
                 $(MCA_opal_allocator_STATIC_SUBDIRS)
                $(MCA_opal_backtrace_STATIC_SUBDIRS)
                $(MCA_opal_btl_STATIC_SUBDIRS) $(MCA_opal_dl_STATIC_SUBDIRS)
                $(MCA_opal_event_STATIC_SUBDIRS)
    $(MCA_opal_hwloc_STATIC_SUBDIRS)
                $(MCA_opal_if_STATIC_SUBDIRS)
    $(MCA_opal_installdirs_STATIC_SUBDIRS)
                $(MCA_opal_memchecker_STATIC_SUBDIRS)
                $(MCA_opal_memcpy_STATIC_SUBDIRS)
                $(MCA_opal_memory_STATIC_SUBDIRS)
    $(MCA_opal_mpool_STATIC_SUBDIRS)
                $(MCA_opal_pmix_STATIC_SUBDIRS)
    $(MCA_opal_pstat_STATIC_SUBDIRS)
                $(MCA_opal_rcache_STATIC_SUBDIRS)
    $(MCA_opal_sec_STATIC_SUBDIRS)
                $(MCA_opal_shmem_STATIC_SUBDIRS)
    $(MCA_opal_timer_STATIC_SUBDIRS)'
                MCA_opal_FRAMEWORK_LIBS=' $(MCA_opal_common_STATIC_LTLIBS)
                mca/allocator/libmca_allocator.la
    <http://libmca_allocator.la> <http://libmca_allocator.la>
    <http://libmca_allocator.la>
                $(MCA_opal_allocator_STATIC_LTLIBS)
                mca/backtrace/libmca_backtrace.la
    <http://libmca_backtrace.la> <http://libmca_backtrace.la>
    <http://libmca_backtrace.la>
                $(MCA_opal_backtrace_STATIC_LTLIBS)
                mca/btl/libmca_btl.la <http://libmca_btl.la>
    <http://libmca_btl.la> <http://libmca_btl.la>
                $(MCA_opal_btl_STATIC_LTLIBS) mca/dl/libmca_dl.la
    <http://libmca_dl.la>
                <http://libmca_dl.la> <http://libmca_dl.la>
                $(MCA_opal_dl_STATIC_LTLIBS) mca/event/libmca_event.la
    <http://libmca_event.la>
                <http://libmca_event.la> <http://libmca_event.la>
                $(MCA_opal_event_STATIC_LTLIBS) mca/hwloc/libmca_hwloc.la
    <http://libmca_hwloc.la>
                <http://libmca_hwloc.la> <http://libmca_hwloc.la>
                $(MCA_opal_hwloc_STATIC_LTLIBS) mca/if/libmca_if.la
    <http://libmca_if.la>
                <http://libmca_if.la> <http://libmca_if.la>
                $(MCA_opal_if_STATIC_LTLIBS)
    mca/installdirs/libmca_installdirs.la <http://libmca_installdirs.la>
                <http://libmca_installdirs.la> <http://libmca_installdirs.la>
                $(MCA_opal_installdirs_STATIC_LTLIBS)
                mca/memchecker/libmca_memchecker.la
    <http://libmca_memchecker.la> <http://libmca_memchecker.la>
    <http://libmca_memchecker.la>
                $(MCA_opal_memchecker_STATIC_LTLIBS)
    mca/memcpy/libmca_memcpy.la <http://libmca_memcpy.la>
                <http://libmca_memcpy.la> <http://libmca_memcpy.la>
                $(MCA_opal_memcpy_STATIC_LTLIBS)
    mca/memory/libmca_memory.la <http://libmca_memory.la>
                <http://libmca_memory.la> <http://libmca_memory.la>
                $(MCA_opal_memory_STATIC_LTLIBS) mca/mpool/libmca_mpool.la
    <http://libmca_mpool.la>
                <http://libmca_mpool.la> <http://libmca_mpool.la>
                $(MCA_opal_mpool_STATIC_LTLIBS) mca/pmix/libmca_pmix.la
    <http://libmca_pmix.la>
                <http://libmca_pmix.la> <http://libmca_pmix.la>
                $(MCA_opal_pmix_STATIC_LTLIBS) mca/pstat/libmca_pstat.la
    <http://libmca_pstat.la>
                <http://libmca_pstat.la> <http://libmca_pstat.la>
                $(MCA_opal_pstat_STATIC_LTLIBS) mca/rcache/libmca_rcache.la
    <http://libmca_rcache.la>
                <http://libmca_rcache.la> <http://libmca_rcache.la>
                $(MCA_opal_rcache_STATIC_LTLIBS) mca/sec/libmca_sec.la
    <http://libmca_sec.la>
                <http://libmca_sec.la> <http://libmca_sec.la>
                $(MCA_opal_sec_STATIC_LTLIBS) mca/shmem/libmca_shmem.la
    <http://libmca_shmem.la>
                <http://libmca_shmem.la> <http://libmca_shmem.la>
                $(MCA_opal_shmem_STATIC_LTLIBS) mca/timer/libmca_timer.la
    <http://libmca_timer.la>
                <http://libmca_timer.la> <http://libmca_timer.la>
                $(MCA_opal_timer_STATIC_LTLIBS)'
                ...
                MCA_opal_pmix_ALL_COMPONENTS=' s1 cray s2 pmix112 external'
                MCA_opal_pmix_ALL_SUBDIRS=' mca/pmix/s1 mca/pmix/cray
    mca/pmix/s2
                mca/pmix/pmix112 mca/pmix/external'
                MCA_opal_pmix_DSO_COMPONENTS=''
                MCA_opal_pmix_DSO_SUBDIRS=''
                MCA_opal_pmix_STATIC_COMPONENTS=''
                MCA_opal_pmix_STATIC_LTLIBS=''
                MCA_opal_pmix_STATIC_SUBDIRS=''
                ...
                opal_pmix_ext_CPPFLAGS=''
                opal_pmix_ext_LDFLAGS=''
                opal_pmix_ext_LIBS=''
                opal_pmix_pmix112_CPPFLAGS=''
                opal_pmix_pmix112_LIBS=''
                ...




                I've attached the config.log files for pmix.

                tyr openmpi-2.0.0 142 tar zvft pmix_config.log.tar.gz
                -rw-r--r-- root/root    136291 2016-04-25 08:05:34

    
openmpi-v2.x-dev-1290-gbd0e4e1-SunOS.sparc.64_cc/opal/mca/pmix/pmix112/pmix/config.log
                -rw-r--r-- root/root    528808 2016-04-25 08:07:54

    
openmpi-v2.x-dev-1290-gbd0e4e1-SunOS.sparc.64_gcc/opal/mca/pmix/pmix112/pmix/config.log
                tyr openmpi-2.0.0 143



                I've also attached the output for the broken execution of
                "spawn_multiple_master" for my gcc-version of Open MPI.
                "spawn_master" works as expected with my gcc-version of
    Open MPI.

                Hopefully you can fix the problem.


                Kind regards and thank you very much for your help

                Siegmar



                Am 23.04.2016 um 21:34 schrieb Siegmar Gross:

                    Hi Gilles,

                    I don't know what happened, but the files are not
    available now
                    and they were definitely available when I answered the
    email from
                    Ralph. The files also have a different timestamp now.
    This is an
                    extract from my email to Ralph for Solaris Sparc.

                    -rwxr-xr-x 1 root root     977 Apr 19 19:49
    mca_plm_rsh.la <http://mca_plm_rsh.la>
                    <http://mca_plm_rsh.la> <http://mca_plm_rsh.la>
                    -rwxr-xr-x 1 root root  153280 Apr 19 19:49 mca_plm_rsh.so
                    -rwxr-xr-x 1 root root    1007 Apr 19 19:47
                    mca_pmix_pmix112.la <http://mca_pmix_pmix112.la>
    <http://mca_pmix_pmix112.la> <http://mca_pmix_pmix112.la>
                    -rwxr-xr-x 1 root root 1400512 Apr 19 19:47
    mca_pmix_pmix112.so
                    -rwxr-xr-x 1 root root     971 Apr 19 19:52
    mca_pml_cm.la <http://mca_pml_cm.la>
                    <http://mca_pml_cm.la> <http://mca_pml_cm.la>
                    -rwxr-xr-x 1 root root  342440 Apr 19 19:52 mca_pml_cm.so

                    Now I have the following output for these files.

                    -rwxr-xr-x 1 root root     976 Apr 19 19:58
    mca_plm_rsh.la <http://mca_plm_rsh.la>
                    <http://mca_plm_rsh.la> <http://mca_plm_rsh.la>
                    -rwxr-xr-x 1 root root  319816 Apr 19 19:58 mca_plm_rsh.so
                    -rwxr-xr-x 1 root root     970 Apr 19 20:00
    mca_pml_cm.la <http://mca_pml_cm.la>
                    <http://mca_pml_cm.la> <http://mca_pml_cm.la>
                    -rwxr-xr-x 1 root root 1507440 Apr 19 20:00 mca_pml_cm.so

                    I'll try to find out what happened next week when I'm
    back in
                    my office.


                    Kind regards

                    Siegmar





                    Am 23.04.16 um 02:12 schrieb Gilles Gouaillardet:

                        Siegmar,

                        I will try to reproduce this on my solaris11 x86_64 vm

                        In the mean time, can you please double check
                        mca_pmix_pmix_pmix112.so
                        is a 64 bits library ?
                        (E.g, confirm "-m64" was correctly passed to pmix)

                        Cheers,

                        Gilles

                        On Friday, April 22, 2016, Siegmar Gross
                        <siegmar.gr...@informatik.hs-fulda.de
    <javascript:_e(%7B%7D,'cvml','siegmar.gr...@informatik.hs-fulda.de');>
                        <mailto:siegmar.gr...@informatik.hs-fulda.de>
    <javascript:_e(%7B%7D,'cvml','siegmar.gr...@informatik.hs-fulda.de');>>
    wrote:

                            Hi Ralph,

                            I've already used "-enable-debug". "SYSTEM_ENV" is
                        "SunOS" or
                            "Linux" and "MACHINE_ENV" is "sparc" or "x86_84".

                            mkdir

    openmpi-v2.x-dev-1280-gc110ae8-${SYSTEM_ENV}.${MACHINE_ENV}.64_gcc
                            cd

    openmpi-v2.x-dev-1280-gc110ae8-${SYSTEM_ENV}.${MACHINE_ENV}.64_gcc

                            ../openmpi-v2.x-dev-1280-gc110ae8/configure \
                              --prefix=/usr/local/openmpi-2.0.0_64_gcc \
                              --libdir=/usr/local/openmpi-2.0.0_64_gcc/lib64 \
                              --with-jdk-bindir=/usr/local/jdk1.8.0/bin \
                              --with-jdk-headers=/usr/local/jdk1.8.0/include \
                              JAVA_HOME=/usr/local/jdk1.8.0 \
                              LDFLAGS="-m64" CC="gcc" CXX="g++"
    FC="gfortran" \
                              CFLAGS="-m64" CXXFLAGS="-m64" FCFLAGS="-m64" \
                              CPP="cpp" CXXCPP="cpp" \
                              --enable-mpi-cxx \
                              --enable-cxx-exceptions \
                              --enable-mpi-java \
                              --enable-heterogeneous \
                              --enable-mpi-thread-multiple \
                              --with-hwloc=internal \
                              --without-verbs \
                              --with-wrapper-cflags="-std=c11 -m64" \
                              --with-wrapper-cxxflags="-m64" \
                              --with-wrapper-fcflags="-m64" \
                              --enable-debug \
                              |& tee
    log.configure.$SYSTEM_ENV.$MACHINE_ENV.64_gcc


                            mkdir

    openmpi-v2.x-dev-1280-gc110ae8-${SYSTEM_ENV}.${MACHINE_ENV}.64_cc
                            cd

    openmpi-v2.x-dev-1280-gc110ae8-${SYSTEM_ENV}.${MACHINE_ENV}.64_cc

                            ../openmpi-v2.x-dev-1280-gc110ae8/configure \
                              --prefix=/usr/local/openmpi-2.0.0_64_cc \
                              --libdir=/usr/local/openmpi-2.0.0_64_cc/lib64 \
                              --with-jdk-bindir=/usr/local/jdk1.8.0/bin \
                              --with-jdk-headers=/usr/local/jdk1.8.0/include \
                              JAVA_HOME=/usr/local/jdk1.8.0 \
                              LDFLAGS="-m64" CC="cc" CXX="CC" FC="f95" \
                              CFLAGS="-m64" CXXFLAGS="-m64 -library=stlport4"
                        FCFLAGS="-m64" \
                              CPP="cpp" CXXCPP="cpp" \
                              --enable-mpi-cxx \
                              --enable-cxx-exceptions \
                              --enable-mpi-java \
                              --enable-heterogeneous \
                              --enable-mpi-thread-multiple \
                              --with-hwloc=internal \
                              --without-verbs \
                              --with-wrapper-cflags="-m64" \
                              --with-wrapper-cxxflags="-m64
    -library=stlport4" \
                              --with-wrapper-fcflags="-m64" \
                              --with-wrapper-ldflags="" \
                              --enable-debug \
                              |& tee
    log.configure.$SYSTEM_ENV.$MACHINE_ENV.64_cc


                            Kind regards

                            Siegmar

                            Am 21.04.2016 um 18:18 schrieb Ralph Castain:

                                Can you please rebuild OMPI with
    -enable-debug in
                        the configure
                                cmd? It will let us see more error output


                                    On Apr 21, 2016, at 8:52 AM, Siegmar Gross
                                    <siegmar.gr...@informatik.hs-fulda.de>
    <javascript:_e(%7B%7D,'cvml','siegmar.gr...@informatik.hs-fulda.de');>
    wrote:

                                    Hi Ralph,

                                    I don't see any additional information.

                                    tyr hello_1 108 mpiexec -np 4 --host
                                    tyr,sunpc1,linpc1,ruester -mca
                                    mca_base_component_show_load_errors 1
    hello_1_mpi
                                    [tyr.informatik.hs-fulda.de:06211
    <http://tyr.informatik.hs-fulda.de:06211>
                        <http://tyr.informatik.hs-fulda.de:06211>
    <http://tyr.informatik.hs-fulda.de:06211>

    <http://tyr.informatik.hs-fulda.de:06211>
    <http://tyr.informatik.hs-fulda.de:06211>]
                        [[48741,0],0]
                                    ORTE_ERROR_LOG: Not found in file


    
../../../../../openmpi-v2.x-dev-1280-gc110ae8/orte/mca/ess/hnp/ess_hnp_module.c

                                    at line 638


    --------------------------------------------------------------------------
                                    It looks like orte_init failed for some
                        reason; your
                                    parallel process is
                                    likely to abort.  There are many
    reasons that
                        a parallel
                                    process can
                                    fail during orte_init; some of which
    are due
                        to configuration or
                                    environment problems.  This failure
    appears to
                        be an
                                    internal failure;
                                    here's some additional information
    (which may
                        only be
                                    relevant to an
                                    Open MPI developer):

                                     opal_pmix_base_select failed
                                     --> Returned value Not found (-13)
    instead of
                        ORTE_SUCCESS


    --------------------------------------------------------------------------


                                    tyr hello_1 109 mpiexec -np 4 --host
                                    tyr,sunpc1,linpc1,ruester -mca
                                    mca_base_component_show_load_errors 1 -mca
                        pmix_base_verbose
                                    10 -mca pmix_server_verbose 5 hello_1_mpi
                                    [tyr.informatik.hs-fulda.de:06212
    <http://tyr.informatik.hs-fulda.de:06212>
                        <http://tyr.informatik.hs-fulda.de:06212>
    <http://tyr.informatik.hs-fulda.de:06212>

    <http://tyr.informatik.hs-fulda.de:06212>
    <http://tyr.informatik.hs-fulda.de:06212>]
                        mca: base:
                                    components_register: registering framework
                        pmix components
                                    [tyr.informatik.hs-fulda.de:06212
    <http://tyr.informatik.hs-fulda.de:06212>
                        <http://tyr.informatik.hs-fulda.de:06212>
    <http://tyr.informatik.hs-fulda.de:06212>

    <http://tyr.informatik.hs-fulda.de:06212>
    <http://tyr.informatik.hs-fulda.de:06212>]
                        mca: base:
                                    components_open: opening pmix components
                                    [tyr.informatik.hs-fulda.de:06212
    <http://tyr.informatik.hs-fulda.de:06212>
                        <http://tyr.informatik.hs-fulda.de:06212>
    <http://tyr.informatik.hs-fulda.de:06212>

    <http://tyr.informatik.hs-fulda.de:06212>
    <http://tyr.informatik.hs-fulda.de:06212>]
                        mca:base:select:
                                    Auto-selecting pmix components
                                    [tyr.informatik.hs-fulda.de:06212
    <http://tyr.informatik.hs-fulda.de:06212>
                        <http://tyr.informatik.hs-fulda.de:06212>
    <http://tyr.informatik.hs-fulda.de:06212>

    <http://tyr.informatik.hs-fulda.de:06212>
    <http://tyr.informatik.hs-fulda.de:06212>]
                        mca:base:select:(
                                    pmix) No component selected!
                                    [tyr.informatik.hs-fulda.de:06212
    <http://tyr.informatik.hs-fulda.de:06212>
                        <http://tyr.informatik.hs-fulda.de:06212>
    <http://tyr.informatik.hs-fulda.de:06212>

    <http://tyr.informatik.hs-fulda.de:06212>
    <http://tyr.informatik.hs-fulda.de:06212>]
                        [[48738,0],0]
                                    ORTE_ERROR_LOG: Not found in file


    
../../../../../openmpi-v2.x-dev-1280-gc110ae8/orte/mca/ess/hnp/ess_hnp_module.c

                                    at line 638


    --------------------------------------------------------------------------
                                    It looks like orte_init failed for some
                        reason; your
                                    parallel process is
                                    likely to abort.  There are many
    reasons that
                        a parallel
                                    process can
                                    fail during orte_init; some of which
    are due
                        to configuration or
                                    environment problems.  This failure
    appears to
                        be an
                                    internal failure;
                                    here's some additional information
    (which may
                        only be
                                    relevant to an
                                    Open MPI developer):

                                     opal_pmix_base_select failed
                                     --> Returned value Not found (-13)
    instead of
                        ORTE_SUCCESS


    --------------------------------------------------------------------------
                                    tyr hello_1 110


                                    Kind regards

                                    Siegmar


                                    Am 21.04.2016 um 17:24 schrieb Ralph
    Castain:

                                        Hmmm…it looks like you built the right
                        components, but
                                        they are not being picked up. Can
    you run
                        your mpiexec
                                        command again, adding “-mca
                                        mca_base_component_show_load_errors
    1” to
                        the cmd line?


                                            On Apr 21, 2016, at 8:16 AM,
    Siegmar Gross

    <siegmar.gr...@informatik.hs-fulda.de>
    <javascript:_e(%7B%7D,'cvml','siegmar.gr...@informatik.hs-fulda.de');>
                        wrote:

                                            Hi Ralph,

                                            I have attached ompi_info
    output for
                        both compilers
                                            from my
                                            sparc machine and the listings for
                        both compilers
                                            from the
                                            <prefix>/lib/openmpi directories.
                        Hopefully that
                                            helps to
                                            find the problem.

                                            hermes tmp 3 tar zvft
                        openmpi-2.x_info.tar.gz
                                            -rw-r--r-- root/root     10969
                        2016-04-21 17:06
                                            ompi_info_SunOS_sparc_cc.txt
                                            -rw-r--r-- root/root     11044
                        2016-04-21 17:06
                                            ompi_info_SunOS_sparc_gcc.txt
                                            -rw-r--r-- root/root     71252
                        2016-04-21 17:02
                                            lib64_openmpi.txt
                                            hermes tmp 4


                                            Kind regards and thank you very
    much
                        once more for
                                            your help

                                            Siegmar


                                            Am 21.04.2016 um 15:54 schrieb
    Ralph
                        Castain:

                                                Odd - it would appear that
    none of
                        the pmix
                                                components built? Can you send
                                                along the output from
    ompi_info?
                        Or just send a
                                                listing of the files in the
                                                <prefix>/lib/openmpi
    directory?


                                                    On Apr 21, 2016, at
    1:27 AM,
                        Siegmar Gross

                        <siegmar.gr...@informatik.hs-fulda.de
    <javascript:_e(%7B%7D,'cvml','siegmar.gr...@informatik.hs-fulda.de');>

                        <mailto:siegmar.gr...@informatik.hs-fulda.de>
    <javascript:_e(%7B%7D,'cvml','siegmar.gr...@informatik.hs-fulda.de');>>
                                                    wrote:

                                                    Hi Ralph,

                                                    Am 21.04.2016 um 00:18
    schrieb
                        Ralph Castain:

                                                        Could you please rerun
                        these test and
                                                        add “-mca
    pmix_base_verbose 10
                                                        -mca
    pmix_server_verbose
                        5” to your cmd
                                                        line? I need to see
    why the
                                                        pmix components
    failed.



                                                    tyr spawn 111 mpiexec
    -np 1 --host

    tyr,sunpc1,linpc1,ruester -mca
                                                    pmix_base_verbose 10 -mca
                                                    pmix_server_verbose 5
                        spawn_multiple_master

    [tyr.informatik.hs-fulda.de <http://tyr.informatik.hs-fulda.de>
                        <http://tyr.informatik.hs-fulda.de>
    <http://tyr.informatik.hs-fulda.de>

                        <http://tyr.informatik.hs-fulda.de>
    <http://tyr.informatik.hs-fulda.de>

                        <http://tyr.informatik.hs-fulda.de/>
    <http://tyr.informatik.hs-fulda.de/>:26652] mca:
                                                    base: components_register:
                        registering
                                                    framework pmix components

    [tyr.informatik.hs-fulda.de <http://tyr.informatik.hs-fulda.de>
                        <http://tyr.informatik.hs-fulda.de>
    <http://tyr.informatik.hs-fulda.de>

                        <http://tyr.informatik.hs-fulda.de>
    <http://tyr.informatik.hs-fulda.de>

                        <http://tyr.informatik.hs-fulda.de/>
    <http://tyr.informatik.hs-fulda.de/>:26652] mca:
                                                    base: components_open:
    opening
                        pmix components

    [tyr.informatik.hs-fulda.de <http://tyr.informatik.hs-fulda.de>
                        <http://tyr.informatik.hs-fulda.de>
    <http://tyr.informatik.hs-fulda.de>

                        <http://tyr.informatik.hs-fulda.de>
    <http://tyr.informatik.hs-fulda.de>

                        <http://tyr.informatik.hs-fulda.de/>
    <http://tyr.informatik.hs-fulda.de/>:26652]
                                                    mca:base:select:
                        Auto-selecting pmix components

    [tyr.informatik.hs-fulda.de <http://tyr.informatik.hs-fulda.de>
                        <http://tyr.informatik.hs-fulda.de>
    <http://tyr.informatik.hs-fulda.de>

                        <http://tyr.informatik.hs-fulda.de>
    <http://tyr.informatik.hs-fulda.de>

                        <http://tyr.informatik.hs-fulda.de/>
    <http://tyr.informatik.hs-fulda.de/>:26652]
                                                    mca:base:select:( pmix) No
                        component selected!

    [tyr.informatik.hs-fulda.de <http://tyr.informatik.hs-fulda.de>
                        <http://tyr.informatik.hs-fulda.de>
    <http://tyr.informatik.hs-fulda.de>

                        <http://tyr.informatik.hs-fulda.de>
    <http://tyr.informatik.hs-fulda.de>

                        <http://tyr.informatik.hs-fulda.de/>
    <http://tyr.informatik.hs-fulda.de/>:26652]
                                                    [[52794,0],0]
    ORTE_ERROR_LOG:
                        Not found in file


    
../../../../../openmpi-v2.x-dev-1280-gc110ae8/orte/mca/ess/hnp/ess_hnp_module.c

                                                    at line 638


    --------------------------------------------------------------------------
                                                    It looks like orte_init
    failed
                        for some
                                                    reason; your parallel
    process is
                                                    likely to abort.  There
    are
                        many reasons
                                                    that a parallel process
    can
                                                    fail during orte_init;
    some of
                        which are due
                                                    to configuration or
                                                    environment problems.
    This
                        failure appears
                                                    to be an internal failure;
                                                    here's some additional
                        information (which
                                                    may only be relevant to an
                                                    Open MPI developer):

                                                    opal_pmix_base_select
    failed
                                                    --> Returned value Not
    found
                        (-13) instead
                                                    of ORTE_SUCCESS


    --------------------------------------------------------------------------
                                                    tyr spawn 112




                                                    tyr hello_1 116 mpiexec
    -np 1
                        --host

    tyr,sunpc1,linpc1,ruester -mca
                                                    pmix_base_verbose 10 -mca
                                                    pmix_server_verbose 5
    hello_1_mpi

    [tyr.informatik.hs-fulda.de <http://tyr.informatik.hs-fulda.de>
                        <http://tyr.informatik.hs-fulda.de>
    <http://tyr.informatik.hs-fulda.de>

                        <http://tyr.informatik.hs-fulda.de>
    <http://tyr.informatik.hs-fulda.de>

                        <http://tyr.informatik.hs-fulda.de/>
    <http://tyr.informatik.hs-fulda.de/>:27261] mca:
                                                    base: components_register:
                        registering
                                                    framework pmix components

    [tyr.informatik.hs-fulda.de <http://tyr.informatik.hs-fulda.de>
                        <http://tyr.informatik.hs-fulda.de>
    <http://tyr.informatik.hs-fulda.de>

                        <http://tyr.informatik.hs-fulda.de>
    <http://tyr.informatik.hs-fulda.de>

                        <http://tyr.informatik.hs-fulda.de/>
    <http://tyr.informatik.hs-fulda.de/>:27261] mca:
                                                    base: components_open:
    opening
                        pmix components

    [tyr.informatik.hs-fulda.de <http://tyr.informatik.hs-fulda.de>
                        <http://tyr.informatik.hs-fulda.de>
    <http://tyr.informatik.hs-fulda.de>

                        <http://tyr.informatik.hs-fulda.de>
    <http://tyr.informatik.hs-fulda.de>

                        <http://tyr.informatik.hs-fulda.de/>
    <http://tyr.informatik.hs-fulda.de/>:27261]
                                                    mca:base:select:
                        Auto-selecting pmix components

    [tyr.informatik.hs-fulda.de <http://tyr.informatik.hs-fulda.de>
                        <http://tyr.informatik.hs-fulda.de>
    <http://tyr.informatik.hs-fulda.de>

                        <http://tyr.informatik.hs-fulda.de>
    <http://tyr.informatik.hs-fulda.de>

                        <http://tyr.informatik.hs-fulda.de/>
    <http://tyr.informatik.hs-fulda.de/>:27261]
                                                    mca:base:select:( pmix) No
                        component selected!

    [tyr.informatik.hs-fulda.de <http://tyr.informatik.hs-fulda.de>
                        <http://tyr.informatik.hs-fulda.de>
    <http://tyr.informatik.hs-fulda.de>

                        <http://tyr.informatik.hs-fulda.de>
    <http://tyr.informatik.hs-fulda.de>

                        <http://tyr.informatik.hs-fulda.de/>
    <http://tyr.informatik.hs-fulda.de/>:27261]
                                                    [[52315,0],0]
    ORTE_ERROR_LOG:
                        Not found in file


    
../../../../../openmpi-v2.x-dev-1280-gc110ae8/orte/mca/ess/hnp/ess_hnp_module.c

                                                    at line 638


    --------------------------------------------------------------------------
                                                    It looks like orte_init
    failed
                        for some
                                                    reason; your parallel
    process is
                                                    likely to abort.  There
    are
                        many reasons
                                                    that a parallel process
    can
                                                    fail during orte_init;
    some of
                        which are due
                                                    to configuration or
                                                    environment problems.
    This
                        failure appears
                                                    to be an internal failure;
                                                    here's some additional
                        information (which
                                                    may only be relevant to an
                                                    Open MPI developer):

                                                    opal_pmix_base_select
    failed
                                                    --> Returned value Not
    found
                        (-13) instead
                                                    of ORTE_SUCCESS


    --------------------------------------------------------------------------
                                                    tyr hello_1 117



                                                    Thank you very much for
    your help.


                                                    Kind regards

                                                    Siegmar




                                                        Thanks
                                                        Ralph

                                                            On Apr 20,
    2016, at
                        10:12 AM,
                                                            Siegmar Gross

                        <siegmar.gr...@informatik.hs-fulda.de
    <javascript:_e(%7B%7D,'cvml','siegmar.gr...@informatik.hs-fulda.de');>

                        <mailto:siegmar.gr...@informatik.hs-fulda.de>
    <javascript:_e(%7B%7D,'cvml','siegmar.gr...@informatik.hs-fulda.de');>>
                                                            wrote:

                                                            Hi,

                                                            I have built

                        openmpi-v2.x-dev-1280-gc110ae8 on my
                                                            machines
                                                            (Solaris 10 Sparc,
                        Solaris 10
                                                            x86_64, and
    openSUSE Linux
                                                            12.1 x86_64) with
                        gcc-5.1.0 and Sun
                                                            C 5.13.
    Unfortunately
                        I get
                                                            runtime errors for
                        some programs.


                                                            Sun C 5.13:
                                                            ===========

                                                            For all my test
                        programs I get the
                                                            same error on
    Solaris
                        Sparc and
                                                            Solaris x86_64,
    while
                        the programs
                                                            work fine on
    Linux.

                                                            tyr hello_1 115
                        mpiexec -np 2
                                                            hello_1_mpi

                        [tyr.informatik.hs-fulda.de
    <http://tyr.informatik.hs-fulda.de>
                        <http://tyr.informatik.hs-fulda.de>
    <http://tyr.informatik.hs-fulda.de>

                        <http://tyr.informatik.hs-fulda.de>
    <http://tyr.informatik.hs-fulda.de>

                        <http://tyr.informatik.hs-fulda.de>
    <http://tyr.informatik.hs-fulda.de>:22373]
                                                            [[61763,0],0]
                        ORTE_ERROR_LOG: Not
                                                            found in file


    
../../../../../openmpi-v2.x-dev-1280-gc110ae8/orte/mca/ess/hnp/ess_hnp_module.c

                                                            at line 638


    --------------------------------------------------------------------------
                                                            It looks like
                        orte_init failed for
                                                            some reason; your
                        parallel process is
                                                            likely to abort.
                        There are many
                                                            reasons that a
                        parallel process can
                                                            fail during
    orte_init;
                        some of which
                                                            are due to
                        configuration or
                                                            environment
    problems.
                        This failure
                                                            appears to be an
                        internal failure;
                                                            here's some
    additional
                        information
                                                            (which may only be
                        relevant to an
                                                            Open MPI
    developer):


    opal_pmix_base_select
                        failed
                                                            --> Returned
    value Not
                        found (-13)
                                                            instead of
    ORTE_SUCCESS


    --------------------------------------------------------------------------
                                                            tyr hello_1 116




                                                            GCC-5.1.0:
                                                            ==========

                                                            tyr spawn 121
    mpiexec
                        -np 1 --host

    tyr,sunpc1,linpc1,ruester

    spawn_multiple_master

                                                            Parent process 0
                        running on

                        tyr.informatik.hs-fulda.de
    <http://tyr.informatik.hs-fulda.de> <http://tyr.informatik.hs-fulda.de>
    <http://tyr.informatik.hs-fulda.de>

                        <http://tyr.informatik.hs-fulda.de>
    <http://tyr.informatik.hs-fulda.de>

                        <http://tyr.informatik.hs-fulda.de>
    <http://tyr.informatik.hs-fulda.de>
                                                            I create 3 slave
                        processes.


                        [tyr.informatik.hs-fulda.de
    <http://tyr.informatik.hs-fulda.de>
                        <http://tyr.informatik.hs-fulda.de>
    <http://tyr.informatik.hs-fulda.de>

                        <http://tyr.informatik.hs-fulda.de>
    <http://tyr.informatik.hs-fulda.de>

                        <http://tyr.informatik.hs-fulda.de>
    <http://tyr.informatik.hs-fulda.de>:25366]
                                                            PMIX ERROR:
                        UNPACK-PAST-END in file


    
../../../../../../openmpi-v2.x-dev-1280-gc110ae8/opal/mca/pmix/pmix112/pmix/src/server/pmix_server_ops.c


                                                            at line 829

                        [tyr.informatik.hs-fulda.de
    <http://tyr.informatik.hs-fulda.de>
                        <http://tyr.informatik.hs-fulda.de>
    <http://tyr.informatik.hs-fulda.de>

                        <http://tyr.informatik.hs-fulda.de>
    <http://tyr.informatik.hs-fulda.de>

                        <http://tyr.informatik.hs-fulda.de>
    <http://tyr.informatik.hs-fulda.de>:25366]
                                                            PMIX ERROR:
                        UNPACK-PAST-END in file


    
../../../../../../openmpi-v2.x-dev-1280-gc110ae8/opal/mca/pmix/pmix112/pmix/src/server/pmix_server.c


                                                            at line 2176
                                                            [tyr:25377] *** An
                        error occurred in

    MPI_Comm_spawn_multiple
                                                            [tyr:25377] ***
                        reported by process
                                                            [3308257281,0]
                                                            [tyr:25377] *** on
                        communicator
                                                            MPI_COMM_WORLD
                                                            [tyr:25377] ***
                        MPI_ERR_SPAWN: could
                                                            not spawn
    processes
                                                            [tyr:25377] ***
                        MPI_ERRORS_ARE_FATAL
                                                            (processes in this
                        communicator will
                                                            now abort,
                                                            [tyr:25377]
    ***    and
                        potentially
                                                            your MPI job)
                                                            tyr spawn 122


                                                            I would be
    grateful if
                        somebody can
                                                            fix the problems.
                        Thank you very
                                                            much for any
    help in
                        advance.


                                                            Kind regards

                                                            Siegmar


    
<hello_1_mpi.c><spawn_multiple_master.c>_______________________________________________


                                                            users mailing list
                                                            us...@open-mpi.org
                        <mailto:us...@open-mpi.org>
                                                            Subscription:

                        http://www.open-mpi.org/mailman/listinfo.cgi/users
                                                            Link to this post:


    <http://www.open-mpi.org/community/lists/users/2016/04/28983.php>



_______________________________________________
users mailing list
us...@open-mpi.org
Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2016/04/29047.php

Reply via email to