Ralph, attached please find a file with the results of
$ mpirun -d singularity exec ./mpi_test.img /usr/bin/ring
I pressed Ctrl-C at about line 200 of the output file.
I hope there is something useful in it.
My nodefile looks like this
nyx6219
nyx6219
nyx6145
nyx6145
nyx6191
nyx6191
nyx6155
nyx6155
nyx6213
nyx6213
nyx6223
nyx6223
nyx6233
nyx6233
nyx6127
nyx6127
where nyx6219 is the host from which I run mpirun. I can cycle all of
the remaining pairs of nodes such that
$ mpirun -np 4 singularity exec ./mpi_test.img /usr/bin/ring
will run successfully with every pair of processors in the list, which
leads me to believe it is not some sort of base communication issue
(but I could be wrong).
I built the host system's OMPI from 2.x-devel-7762c21
That inside the container is from
Singularity.mpi_test.img> git log
commit 5c64c0bc3bfd2b60c8c38a10482f8387cff1d879
Merge: af7e2cc bb2481a
Author: Gilles Gouaillardet <ggouaillar...@users.noreply.github.com>
Date: Mon Feb 20 11:29:35 2017 +0900
The one inside the container was built with only --PREFIX specified.
That outside had
--disable-dlopen --enable-shared --with-tm --with-verbs
Both inside and outside the container are CentOS 7, both built with
gcc 4.8.5 as shipped.
-- bennet
On Mon, Feb 20, 2017 at 1:10 PM, r...@open-mpi.org <r...@open-mpi.org> wrote:
> If you can send us some more info on how it breaks, that would be helpful.
> I’ll file it as an issue so we can track things
>
> Thanks
> Ralph
>
>
>> On Feb 20, 2017, at 9:13 AM, Bennet Fauber <ben...@umich.edu> wrote:
>>
>> I got mixed results when bringing a container that doesn't have the IB
>> and Torque libraries compiled into the OMPI inside the container to a
>> cluster where it does.
>>
>> The short summary is that mutlinode communication seems unreliable. I
>> can mostly get up to 8 procs, two-per-node, to run, but beyond that
>> not. In a couple of cases, a particular node seemed able to cause a
>> problem. I am going to try again making the configure line inside the
>> container the same as outside, but I have to chase down the IB and
>> Torque to do so.
>>
>> If you're interested in how it breaks, I can send you some more
>> information. If there are diagnostics you would like, I can try to
>> provide those. I will be gone starting Thu for a week.
>>
>> -- bennet
>>
>>
>>
>>
>> On Fri, Feb 17, 2017 at 11:20 PM, r...@open-mpi.org <r...@open-mpi.org>
>> wrote:
>>> I -think- that is correct, but you may need the verbs library as well - I
>>> honestly don’t remember if the configury checks for functions in the
>>> library or not. If so, then you’ll need that wherever you build OMPI, but
>>> everything else is accurate
>>>
>>> Good luck - and let us know how it goes!
>>> Ralph
>>>
>>>> On Feb 17, 2017, at 4:34 PM, Bennet Fauber <ben...@umich.edu> wrote:
>>>>
>>>> Ralph.
>>>>
>>>> I will be building from the Master branch at github.com for testing
>>>> purposes. We are not 'supporting' Singularity container creation, but
>>>> we do hope to be able to offer some guidance, so I think we can
>>>> finesse the PMIx version, yes?
>>>>
>>>> That is good to know about the verbs headers being the only thing
>>>> needed; thanks for that detail. Sometimes the library also needs to
>>>> be present.
>>>>
>>>> Also very good to know that the host mpirun will start processes, as
>>>> we are using cgroups, and if the processes get started by a
>>>> non-tm-supporting MPI, they will be outside the proper cgroup.
>>>>
>>>> So, just to recap, if I install from the current master at
>>>> http://github.com/open-mpi/ompi.git on the host system and within the
>>>> container, I copy the verbs headers into the container, then configure
>>>> and build OMPI within the container and ignore TM support, I should be
>>>> able to copy the container to the cluster and run it with verbs and
>>>> the system OMPI using tm.
>>>>
>>>> If a user were to build without the verbs support, it would still run,
>>>> but it would fall back to non-verbs communication, so it would just be
>>>> commensurately slower.
>>>>
>>>> Let me know if I've garbled things. Otherwise, wish me luck, and have
>>>> a good weekend!
>>>>
>>>> Thanks, -- bennet
>>>>
>>>>
>>>>
>>>> On Fri, Feb 17, 2017 at 7:24 PM, r...@open-mpi.org <r...@open-mpi.org>
>>>> wrote:
>>>>> The embedded Singularity support hasn’t made it into the OMPI 2.x release
>>>>> series yet, though OMPI will still work within a Singularity container
>>>>> anyway.
>>>>>
>>>>> Compatibility across the container boundary is always a problem, as your
>>>>> examples illustrate. If the system is using one OMPI version and the
>>>>> container is using another, then the only concern is compatibility across
>>>>> the container boundary of the process-to-ORTE daemon communication. In
>>>>> the OMPI 2.x series and beyond, this is done with PMIx. OMPI v2.0 is
>>>>> based on PMIx v1.x, and so will OMPI v2.1. Thus, there is no
>>>>> compatibility issue there. However, that statement is _not_ true for OMPI
>>>>> v1.10 and earlier series.
>>>>>
>>>>> Future OMPI versions will utilize PMIx v2 and above, which include a
>>>>> cross-version compatibility layer. Thus, you shouldn’t have any issues
>>>>> mixing and matching OMPI versions from this regard.
>>>>>
>>>>> However, your second example is a perfect illustration of where
>>>>> containerization can break down. If you build your container on a system
>>>>> that doesn’t have (for example) tm and verbs installed on it, then those
>>>>> OMPI components will not be built. The tm component won’t matter as the
>>>>> system version of mpirun will be executing, and it presumably knows how
>>>>> to interact with Torque.
>>>>>
>>>>> However, if you run that container on a system that has verbs, your
>>>>> application won’t be able to utilize the verbs support because those
>>>>> components were never compiled. Note that the converse is not true: if
>>>>> you build your container on a system that has verbs installed, you can
>>>>> then run it on a system that doesn’t have verbs support and those
>>>>> components will dynamically disqualify themselves.
>>>>>
>>>>> Remember, you only need the verbs headers to be installed - you don’t
>>>>> have to build on a machine that actually has a verbs-supporting NIC
>>>>> installed (this is how the distributions get around the problem). Thus,
>>>>> it isn’t hard to avoid this portability problem - you just need to think
>>>>> ahead a bit.
>>>>>
>>>>> HTH
>>>>> Ralph
>>>>>
>>>>>> On Feb 17, 2017, at 3:49 PM, Bennet Fauber <ben...@umich.edu> wrote:
>>>>>>
>>>>>> I am wishing to follow the instructions on the Singularity web site,
>>>>>>
>>>>>> http://singularity.lbl.gov/docs-hpc
>>>>>>
>>>>>> to test Singularity and OMPI on our cluster. My previously normal
>>>>>> configure for the 1.x series looked like this.
>>>>>>
>>>>>> ./configure --prefix=/usr/local \
>>>>>> --mandir=${PREFIX}/share/man \
>>>>>> --with-tm --with-verbs \
>>>>>> --disable-dlopen --enable-shared
>>>>>> CC=gcc CXX=g++ FC=gfortran
>>>>>>
>>>>>> I have a couple of wonderments.
>>>>>>
>>>>>> First, I presume it will be best to have the same version of OMPI
>>>>>> inside the container as out, but how sensitive will it be to minor
>>>>>> versions? All 2.1.x version should be fine, but not mix 2.1.x outside
>>>>>> with 2.2.x inside or vice-versa (might be backward compatible but not
>>>>>> forward)?
>>>>>>
>>>>>> Second, if someone builds OMPI inside their container on an external
>>>>>> system, without tm and verbs, then brings the container to our system,
>>>>>> will the tm and verbs be handled by the calling mpirun from the host
>>>>>> system, and the OMPI inside the container won't care? Will not having
>>>>>> those inside the container cause them to be suppressed outside?
>>>>>>
>>>>>> Thanks in advance, -- bennet
>>>>>> _______________________________________________
>>>>>> users mailing list
>>>>>> users@lists.open-mpi.org
>>>>>> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
>>>>>
>>>>> _______________________________________________
>>>>> users mailing list
>>>>> users@lists.open-mpi.org
>>>>> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
>>>> _______________________________________________
>>>> users mailing list
>>>> users@lists.open-mpi.org
>>>> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
>>>
>>> _______________________________________________
>>> users mailing list
>>> users@lists.open-mpi.org
>>> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
>> _______________________________________________
>> users mailing list
>> users@lists.open-mpi.org
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
>
> _______________________________________________
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
[bennet@nyx6219 singularity_testing]$ mpirun -d singularity exec ./mpi_test.img
/usr/bin/ring
[nyx6219.arc-ts.umich.edu:07616] procdir: /tmp/ompi.nyx6219.28050/pid.7616/0/0
[nyx6219.arc-ts.umich.edu:07616] jobdir: /tmp/ompi.nyx6219.28050/pid.7616/0
[nyx6219.arc-ts.umich.edu:07616] top: /tmp/ompi.nyx6219.28050/pid.7616
[nyx6219.arc-ts.umich.edu:07616] top: /tmp/ompi.nyx6219.28050
[nyx6219.arc-ts.umich.edu:07616] tmp: /tmp
[nyx6219.arc-ts.umich.edu:07616] sess_dir_cleanup: job session dir does not
exist
[nyx6219.arc-ts.umich.edu:07616] procdir: /tmp/ompi.nyx6219.28050/pid.7616/0/0
[nyx6219.arc-ts.umich.edu:07616] jobdir: /tmp/ompi.nyx6219.28050/pid.7616/0
[nyx6219.arc-ts.umich.edu:07616] top: /tmp/ompi.nyx6219.28050/pid.7616
[nyx6219.arc-ts.umich.edu:07616] top: /tmp/ompi.nyx6219.28050
[nyx6219.arc-ts.umich.edu:07616] tmp: /tmp
[nyx6127.arc-ts.umich.edu:10607] procdir: /tmp/ompi.nyx6127.28050/jf.59907/0/1
[nyx6127.arc-ts.umich.edu:10607] jobdir: /tmp/ompi.nyx6127.28050/jf.59907/0
[nyx6127.arc-ts.umich.edu:10607] top: /tmp/ompi.nyx6127.28050/jf.59907
[nyx6127.arc-ts.umich.edu:10607] top: /tmp/ompi.nyx6127.28050
[nyx6127.arc-ts.umich.edu:10607] tmp: /tmp
[nyx6127.arc-ts.umich.edu:10607] sess_dir_cleanup: job session dir does not
exist
[nyx6127.arc-ts.umich.edu:10607] procdir: /tmp/ompi.nyx6127.28050/jf.59907/0/1
[nyx6127.arc-ts.umich.edu:10607] jobdir: /tmp/ompi.nyx6127.28050/jf.59907/0
[nyx6127.arc-ts.umich.e[nyx6223.arc-ts.umich.edu:11585] procdir:
/tmp/ompi.nyx6223.28050/jf.59907/0/6
[nyx6223.arc-ts.umich.edu:11585] jobdir: /tmp/ompi.nyx6223.28050/jf.59907/0
[nyx6223.arc-ts.umich.edu:11585] top: /tmp/ompi.nyx6223.28050/jf.59907
[nyx6223.arc-ts.umich.edu:1158du:10607] top: /tmp/ompi.nyx6127.28050/jf.59907
[nyx6127.arc-ts.umich.edu:10607] top: /tmp/ompi.nyx6127.28050
[nyx6127.arc-ts.umich.edu:10607] tmp: /tmp
[nyx6155.arc-ts.umich.edu:11158] procdir: /tmp/ompi.nyx6155.28050/jf.59907/0/3
[nyx6155.arc-ts.umich.edu:11158] jobdir: /tmp/ompi.nyx6155.28050/jf.59907/0
[nyx6155.arc-ts.umich.edu:11158] top: /tmp/ompi.nyx6155.28050/jf.59907
[nyx6155.arc-ts.umich.edu:11155] top: /tmp/ompi.nyx6223.28050
[nyx6223.arc-ts.umich.edu:11585] tmp: /tmp
[nyx6223.arc-ts.umich.edu:11585] sess_dir_cleanup: job session dir does not
exist
[nyx6223.arc-ts.umich.edu:11585] procdir: /tmp/ompi.nyx6223.28050/jf.59907/0/6
[nyx6223.arc-ts.umic8] top: /tmp/ompi.nyx6155.28050
[nyx6155.arc-ts.umich.edu:11158] tmp: /tmp
[nyx6155.arc-ts.umich.edu:11158] sess_dir_cleanup: job session dir does not
exist
[nyx6155.arc-ts.umich.edu:11158] procdir: /tmp/ompi.nyx6155.28050/jf.59907/0/3
[nyx6155.arc-ts.umich.edu:11585] jobdir: /tmp/ompi.nyx6223.28050/jf.59907/0
[nyx6223.arc-ts.umich.edu:11585] top: /tmp/ompi.nyx6223.28050/jf.59907
[nyx6223.arc-ts.umich.edu:11585] top: /tmp/ompi.nyx6223.28050
[nyx6223.arc-ts.umich.edu:11585] tmp: /tmp
h.edu:11158] jobdir: /tmp/ompi.nyx6155.28050/jf.59907/0
[nyx6155.arc-ts.umich.edu:11158] top: /tmp/ompi.nyx6155.28050/jf.59907
[nyx6155.arc-ts.umich.edu:11158] top: /tmp/ompi.nyx6155.28050
[nyx6155.arc-ts.umich.edu:11158] tmp: /tmp
[nyx6233.arc-ts.umich.edu:11521] procdir: /tmp/ompi.nyx6233.28050/jf.59907/0/7
[nyx6233.arc-ts.umich.edu:11521] jobdir: /tmp/ompi.nyx6233.28050/jf.59907/0
[nyx6233.arc-ts.umich.edu:11521] top: /tmp/ompi.nyx6233.28050/jf.59907
[nyx6233.arc-ts.umich.edu:1152[nyx6145.arc-ts.umich.edu:15451] procdir:
/tmp/ompi.nyx6145.28050/jf.59907/0/2
[nyx6145.arc-ts.umich.edu:15451] jobdir: /tmp/ompi.nyx6145.28050/jf.59907/0
[nyx6145.arc-ts.umich.edu:15451] top: /tmp/ompi.nyx6145.28050/jf.59907
[nyx6145.arc-ts.umich.edu:15451] top: /tmp/ompi.nyx6233.28050
[nyx6233.arc-ts.umich.edu:11521] tmp: /tmp
[nyx6233.arc-ts.umich.edu:11521] sess_dir_cleanup: job session dir does not
exist
[nyx6233.arc-ts.umich.edu:11521] procdir: /tmp/ompi.nyx6233.28050/jf.59907/0/7
[nyx6233.arc-ts.umic[nyx6213.arc-ts.umich.edu:29106] procdir:
/tmp/ompi.nyx6213.28050/jf.59907/0/5
[nyx6213.arc-ts.umich.edu:29106] jobdir: /tmp/ompi.nyx6213.28050/jf.59907/0
[nyx6213.arc-ts.umich.edu:29106] top: /tmp/ompi.nyx6213.28050/jf.59907
[nyx6213.arc-ts.umich.edu:29101] top: /tmp/ompi.nyx6145.28050
[nyx6145.arc-ts.umich.edu:15451] tmp: /tmp
[nyx6145.arc-ts.umich.edu:15451] sess_dir_cleanup: job session dir does not
exist
[nyx6145.arc-ts.umich.edu:15451] procdir: /tmp/ompi.nyx6145.28050/jf.59907/0/2
[nyx6145.arc-ts.umich.edu:11521] jobdir: /tmp/ompi.nyx6233.28050/jf.59907/0
[nyx6233.arc-ts.umich.edu:11521] top: /tmp/ompi.nyx6233.28050/jf.59907
[nyx6233.arc-ts.umich.edu:11521] top: /tmp/ompi.nyx6233.28050
[nyx6233.arc-ts.umich.edu:11521] tmp: /tmp
[nyx6191.arc-ts.umich.edu:22287] procdir: /tmp/ompi.nyx6191.28050/jf.59907/0/4
[nyx6191.arc-ts.umich.edu:22287] jobdir: /tmp/ompi.nyx6191.28050/jf.59907/0
[nyx6191.arc-ts.umich.edu:22287] top: /tmp/ompi.nyx6191.28050/jf.59907
[nyx6191.arc-ts.umich.edu:22286] top: /tmp/ompi.nyx6213.28050
[nyx6213.arc-ts.umich.edu:29106] tmp: /tmp
[nyx6213.arc-ts.umich.edu:29106] sess_dir_cleanup: job session dir does not
exist
[nyx6213.arc-ts.umich.edu:29106] procdir: /tmp/ompi.nyx6213.28050/jf.59907/0/5
[nyx6213.arc-ts.umich.edu:15451] jobdir: /tmp/ompi.nyx6145.28050/jf.59907/0
[nyx6145.arc-ts.umich.edu:15451] top: /tmp/ompi.nyx6145.28050/jf.59907
[nyx6145.arc-ts.umich.edu:15451] top: /tmp/ompi.nyx6145.28050
[nyx6145.arc-ts.umich.edu:15451] tmp: /tmp
7] top: /tmp/ompi.nyx6191.28050
[nyx6191.arc-ts.umich.edu:22287] tmp: /tmp
[nyx6191.arc-ts.umich.edu:22287] sess_dir_cleanup: job session dir does not
exist
[nyx6191.arc-ts.umich.edu:22287] procdir: /tmp/ompi.nyx6191.28050/jf.59907/0/4
[nyx6191.arc-ts.umich.edu:29106] jobdir: /tmp/ompi.nyx6213.28050/jf.59907/0
[nyx6213.arc-ts.umich.edu:29106] top: /tmp/ompi.nyx6213.28050/jf.59907
[nyx6213.arc-ts.umich.edu:29106] top: /tmp/ompi.nyx6213.28050
[nyx6213.arc-ts.umich.edu:29106] tmp: /tmp
h.edu:22287] jobdir: /tmp/ompi.nyx6191.28050/jf.59907/0
[nyx6191.arc-ts.umich.edu:22287] top: /tmp/ompi.nyx6191.28050/jf.59907
[nyx6191.arc-ts.umich.edu:22287] top: /tmp/ompi.nyx6191.28050
[nyx6191.arc-ts.umich.edu:22287] tmp: /tmp
[nyx6219.arc-ts.umich.edu:07616] [[59907,0],0] Releasing job data for [INVALID]
[nyx6219.arc-ts.umich.edu:07628] procdir: /tmp/ompi.nyx6219.28050/pid.7616/1/0
[nyx6219.arc-ts.umich.edu:07628] jobdir: /tmp/ompi.nyx6219.28050/pid.7616/1
[nyx6219.arc-ts.umich.edu:07628] top: /tmp/ompi.nyx6219.28050/pid.7616
[nyx6219.arc-ts.umich.edu:07628] top: /tmp/ompi.nyx6219.28050
[nyx6219.arc-ts.umich.edu:07628] tmp: /tmp
[nyx6219.arc-ts.umich.edu:07629] procdir: /tmp/ompi.nyx6219.28050/pid.7616/1/1
[nyx6219.arc-ts.umich.edu:07629] jobdir: /tmp/ompi.nyx6219.28050/pid.7616/1
[nyx6219.arc-ts.umich.edu:07629] top: /tmp/ompi.nyx6219.28050/pid.7616
[nyx6219.arc-ts.umich.edu:07629] top: /tmp/ompi.nyx6219.28050
[nyx6219.arc-ts.umich.edu:07629] tmp: /tmp
[nyx6223.arc-ts.umich.edu:11596] procdir: /tmp/ompi.nyx6223.28050/jf.59907/1/12
[nyx6223.arc-ts.umich.edu:11596] jobdir: /tmp/ompi.nyx6223.28050/jf.59907/1
[nyx6223.arc-ts.umich.edu:11596] top: /tmp/ompi.nyx6223.28050/jf.59907
[nyx6223.arc-ts.umich.edu:11596] top: /tmp/ompi.nyx6223.28050
[nyx6223.arc-ts.umich.edu:11596] tmp: /tmp
[nyx6223.arc-ts.umich.edu:11597] procdir: /tmp/ompi.nyx6223.28050/jf.59907/1/13
[nyx6223.arc-ts.umich.edu:11597] jobdir: /tmp/ompi.nyx6223.28050/jf.59907/1
[nyx6223.arc-ts.umich.edu:11597] top: /tmp/ompi.nyx6223.28050/jf.59907
[nyx6223.arc-ts.umich.edu:11597] top: /tmp/ompi.nyx6223.28050
[nyx6223.arc-ts.umich.edu:11597] tmp: /tmp
[nyx6127.arc-ts.umich.edu:10620] procdir: /tmp/ompi.nyx6127.28050/jf.59907/1/2
[nyx6127.arc-ts.umich.edu:10620] jobdir: /tmp/ompi.nyx6127.28050/jf.59907/1
[nyx6127.arc-ts.umich.edu:10620] top: /tmp/ompi.nyx6127.28050/jf.59907
[nyx6127.arc-ts.umich.edu:10620] top: /tmp/ompi.nyx6127.28050
[nyx6127.arc-ts.umich.edu:10620] tmp: /tmp
[nyx6127.arc-ts.umich.edu:10621] procdir: /tmp/ompi.nyx6127.28050/jf.59907/1/3
[nyx6127.arc-ts.umich.edu:10621] jobdir: /tmp/ompi.nyx6127.28050/jf.59907/1
[nyx6127.arc-ts.umich.edu:10621] top: /tmp/ompi.nyx6127.28050/jf.59907
[nyx6127.arc-ts.umich.edu:10621] top: /tmp/ompi.nyx6127.28050
[nyx6127.arc-ts.umich.edu:10621] tmp: /tmp
[nyx6155.arc-ts.umich.edu:11169] procdir: /tmp/ompi.nyx6155.28050/jf.59907/1/6
[nyx6155.arc-ts.umich.edu:11169] jobdir: /tmp/ompi.nyx6155.28050/jf.59907/1
[nyx6155.arc-ts.umich.edu:11169] top: /tmp/ompi.nyx6155.28050/jf.59907
[nyx6155.arc-ts.umich.edu:11169] top: /tmp/ompi.nyx6155.28050
[nyx6155.arc-ts.umich.edu:11169] tmp: /tmp
[nyx6213.arc-ts.umich.edu:29117] procdir: /tmp/ompi.nyx6213.28050/jf.59907/1/10
[nyx6213.arc-ts.umich.edu:29117] jobdir: /tmp/ompi.nyx6213.28050/jf.59907/1
[nyx6213.arc-ts.umich.edu:29117] top: /tmp/ompi.nyx6213.28050/jf.59907
[nyx6213.arc-ts.umich.edu:29117] top: /tmp/ompi.nyx6213.28050
[nyx6213.arc-ts.umich.edu:29117] tmp: /tmp
[nyx6155.arc-ts.umich.edu:11170] procdir: /tmp/ompi.nyx6155.28050/jf.59907/1/7
[nyx6155.arc-ts.umich.edu:11170] jobdir: /tmp/ompi.nyx6155.28050/jf.59907/1
[nyx6155.arc-ts.umich.edu:11170] top: /tmp/ompi.nyx6155.28050/jf.59907
[nyx6155.arc-ts.umich.edu:11170] top: /tmp/ompi.nyx6155.28050
[nyx6155.arc-ts.umich.edu:11170] tmp: /tmp
[nyx6213.arc-ts.umich.edu:29118] procdir: /tmp/ompi.nyx6213.28050/jf.59907/1/11
[nyx6213.arc-ts.umich.edu:29118] jobdir: /tmp/ompi.nyx6213.28050/jf.59907/1
[nyx6213.arc-ts.umich.edu:29118] top: /tmp/ompi.nyx6213.28050/jf.59907
[nyx6213.arc-ts.umich.edu:29118] top: /tmp/ompi.nyx6213.28050
[nyx6213.arc-ts.umich.edu:29118] tmp: /tmp
[nyx6191.arc-ts.umich.edu:22299] procdir: /tmp/ompi.nyx6191.28050/jf.59907/1/9
[nyx6191.arc-ts.umich.edu:22298] procdir: /tmp/ompi.nyx6191.28050/jf.59907/1/8
[nyx6191.arc-ts.umich.edu:22298] jobdir: /tmp/ompi.nyx6191.28050/jf.59907/1
[nyx6191.arc-ts.umich.edu:22298] top: /tmp/ompi.nyx6191.28050/jf.59907
[nyx6191.arc-ts.umich.edu:22298] top: /tmp/ompi.nyx6191.28050
[nyx6191.arc-ts.umich.edu:22298] tmp: /tmp
[nyx6191.arc-ts.umich.edu:22299] jobdir: /tmp/ompi.nyx6191.28050/jf.59907/1
[nyx6191.arc-ts.umich.edu:22299] top: /tmp/ompi.nyx6191.28050/jf.59907
[nyx6191.arc-ts.umich.edu:22299] top: /tmp/ompi.nyx6191.28050
[nyx6191.arc-ts.umich.edu:22299] tmp: /tmp
MPIR_being_debugged = 0
MPIR_debug_state = 1
MPIR_partial_attach_ok = 1
MPIR_i_am_starter = 0
MPIR_forward_output = 0
MPIR_proctable_size = 16
MPIR_proctable:
(i, host, exe, pid) = (0, nyx6219, /usr/bin/singularity, 7628)
(i, host, exe, pid) = (1, nyx6219, /usr/bin/singularity, 7629)
(i, host, exe, pid) = (2, nyx6127, /usr/bin/singularity, 10620)
(i, host, exe, pid) = (3, nyx6127, /usr/bin/singularity, 10621)
(i, host, exe, pid) = (4, nyx6145, /usr/bin/singularity, 15462)
(i, host, exe, pid) = (5, nyx6145, /usr/bin/singularity, 15463)
(i, host, exe, pid) = (6, nyx6155, /usr/bin/singularity, 11169)
(i, host, exe, pid) = (7, nyx6155, /usr/bin/singularity, 11170)
(i, host, exe, pid) = (8, nyx6191, /usr/bin/singularity, 22298)
(i, host, exe, pid) = (9, nyx6191, /usr/bin/singularity, 22299)
(i, host, exe, pid) = (10, nyx6213, /usr/bin/singularity, 29117)
(i, host, exe, pid) = (11, nyx6213, /usr/bin/singularity, 29118)
(i, host, exe, pid) = (12, nyx6223, /usr/bin/singularity, 11596)
(i, host, exe, pid) = (13, nyx6223, /usr/bin/singularity, 11597)
(i, host, exe, pid) = (14, nyx6233, /usr/bin/singularity, 11532)
(i, host, exe, pid) = (15, nyx6233, /usr/bin/singularity, 11533)
MPIR_executable_path: NULL
MPIR_server_arguments: NULL
[nyx6145.arc-ts.umich.edu:15462] procdir: /tmp/ompi.nyx6145.28050/jf.59907/1/4
[nyx6145.arc-ts.umich.edu:15462] jobdir: /tmp/ompi.nyx6145.28050/jf.59907/1
[nyx6145.arc-ts.umich.edu:15462] top: /tmp/ompi.nyx6145.28050/jf.59907
[nyx6145.arc-ts.umich.edu:15462] top: /tmp/ompi.nyx6145.28050
[nyx6145.arc-ts.umich.edu:15462] tmp: /tmp
[nyx6145.arc-ts.umich.edu:15463] procdir: /tmp/ompi.nyx6145.28050/jf.59907/1/5
[nyx6145.arc-ts.umich.edu:15463] jobdir: /tmp/ompi.nyx6145.28050/jf.59907/1
[nyx6145.arc-ts.umich.edu:15463] top: /tmp/ompi.nyx6145.28050/jf.59907
[nyx6145.arc-ts.umich.edu:15463] top: /tmp/ompi.nyx6145.28050
[nyx6145.arc-ts.umich.edu:15463] tmp: /tmp
[nyx6233.arc-ts.umich.edu:11532] procdir: /tmp/ompi.nyx6233.28050/jf.59907/1/14
[nyx6233.arc-ts.umich.edu:11532] jobdir: /tmp/ompi.nyx6233.28050/jf.59907/1
[nyx6233.arc-ts.umich.edu:11532] top: /tmp/ompi.nyx6233.28050/jf.59907
[nyx6233.arc-ts.umich.edu:11532] top: /tmp/ompi.nyx6233.28050
[nyx6233.arc-ts.umich.edu:11532] tmp: /tmp
[nyx6233.arc-ts.umich.edu:11533] procdir: /tmp/ompi.nyx6233.28050/jf.59907/1/15
[nyx6233.arc-ts.umich.edu:11533] jobdir: /tmp/ompi.nyx6233.28050/jf.59907/1
[nyx6233.arc-ts.umich.edu:11533] top: /tmp/ompi.nyx6233.28050/jf.59907
[nyx6233.arc-ts.umich.edu:11533] top: /tmp/ompi.nyx6233.28050
[nyx6233.arc-ts.umich.edu:11533] tmp: /tmp
Process 0 sending 10 to 1, tag 201 (16 processes in ring)
Process 0 sent to 1
Process 0 decremented value: 9
[nyx6127][[59907,1],3][btl_tcp_endpoint.c:796:mca_btl_tcp_endpoint_complete_connect]
connect() to 169.254.95.120 failed: Connection refused (111)
^C
[nyx6219.arc-ts.umich.edu:07616] sess_dir_finalize: proc session dir does not
exist
[nyx6219.arc-ts.umich.edu:07616] sess_dir_finalize: job session dir not empty -
leaving
[nyx6219.arc-ts.umich.edu:07616] sess_dir_finalize: jobfam session dir not
empty - leaving
[nyx6219.arc-ts.umich.edu:07616] sess_dir_finalize: top session dir not empty -
leaving
[nyx6219.arc-ts.umich.edu:07616] sess_dir_finalize: proc session dir does not
exist
[nyx6219.arc-ts.umich.edu:07616] sess_dir_finalize: job session dir not empty -
leaving
[nyx6219.arc-ts.umich.edu:07616] sess_dir_finalize: jobfam session dir not
empty - leaving
[nyx6219.arc-ts.umich.edu:07616] sess_dir_finalize: top session dir not empty -
leaving
[nyx6219.arc-ts.umich.edu:07616] sess_dir_finalize: proc session dir does not
exist
[nyx6219.arc-ts.umich.edu:07616] sess_dir_finalize: job session dir not empty -
leaving
[nyx6127.arc-ts.umich.edu:10607] sess_dir_finalize: proc session dir does not
exist
[nyx6127.arc-ts.umich.edu:10607] sess_dir_finalize: job session dir not empty -
leaving
[nyx6127.arc-ts.umich.edu:10607] sess_dir_finalize: jobfam session dir not
empty - leaving
[nyx6127.arc-ts.umich.edu:10607] sess_d[nyx6219.arc-ts.umich.edu:07616]
sess_dir_finalize: jobfam session dir not empty - leaving
ir_finalize[nyx6219.arc-ts.umich.edu:07616] sess_dir_finalize: top session dir
not empty - leaving
: top session dir not empty[nyx6145.arc-ts.umich.edu:15451] sess_dir_finalize:
proc session dir does not exist
[nyx6145.arc-ts.umich.edu:15451] sess_dir_finalize: job session dir not empty -
leaving
[nyx6145.arc-ts.umich.edu:15451] sess_dir_finalize: jobfam session dir not
empty - l[nyx6223.arc-ts.umich.edu:11585] sess_dir_finalize: proc session dir
does not exist
[nyx6223.arc-ts.umich.edu:11585] sess_dir_finalize: job session dir not empty -
leaving
[nyx6223.arc-ts.umich.edu:11585] sess_dir_fina[nyx6219.arc-ts.umich.edu:07616]
sess_dir_finalize: proc session dir does not exist
liz[nyx6219.arc-ts.umich.edu:07616] sess_dir_finalize: job session dir not
empty - leaving
e: jo[nyx6219.arc-ts.umich.edu:07616] sess_dir_finalize: jobfam session dir not
empty - leaving
bf[nyx6219.arc-ts.umich.edu:07616] sess_dir_finalize: top session dir not empty
- leaving
am session dir not empty - l[nyx6191.arc-ts.umich.edu:22287] sess_dir_finalize:
proc session dir does not exist
[nyx6191.arc-ts.umich.edu:22287] sess_dir_finalize: job session dir not empty -
leaving
[nyx6191.arc-ts.umich.edu:22287] sess_dir_finalize: jobfam session dir not
empty - l[nyx6213.arc-ts.umich.edu:29106] sess_dir_finalize: proc session dir
does not exist
[nyx6213.arc-ts.umich.edu:29106] sess_dir_finalize: job session dir not empty -
leaving
[nyx6213.arc-ts.umich.edu:29106] sess_dir_finalize: jobfam session dir not
empty - l - leaving
[nyx6127.arc-ts.umich.edu:10607] sess_dir_finalize: proc session dir does not
exist
[nyx6127.arc-ts.umich.edu:10607] sess_dir_finalize: job session dir not empty -
leaving
[nyx6127.arc-ts.umich.edu:10607] sess_dir_finalize: jobfam session dir noeaving
[nyx6145.arc-ts.umich.edu:15451] sess_dir_finalize: top session dir not empty -
leaving
[nyx6145.arc-ts.umich.edu:15451] sess_dir_finalize: proc session dir does not
exist
[nyx6145.arc-ts.umich.edu:15451] sess_dir_finalize: job session dir not
empty[nyx6155.arc-ts.umich.edu:11158] sess_dir_finalize: proc session dir does
not exist
[nyx6155.arc-ts.umich.edu:11158] sess_dir_finalize: job session dir not empty -
leaving
[nyx6155.arc-ts.umich.edu:11158] sess_dir_finalize: jobfam session dir not
empty - leaving[nyx6219.arc-ts.umich.edu:07616] Job UNKNOWN has launched
[nyx[nyx6219.arc-ts.umich.edu:07616] [[59907,0],0] Releasing job data for
[59907,1]
6223.arc-ts.umich.edu:11585] sess_dir_finalize: top session dir not empty -
leaving
[nyx6223.arc-ts.umich.edu:11585] sess_dir_finalize: proc session dir does not
exist
[nyx6223.arc-ts.umich.edu:11585] sess_dir_finalize: job session dir not
empty[nyx6233.arc-ts.umich.edu:11521] sess_dir_finalize: proc session dir does
not exist
[nyx6233.arc-ts.umich.edu:11521] sess_dir_finalize: job session dir not empty -
leaving
[nyx6233.arc-ts.umich.edu:11521] sess_dir_finalize: jobfam session dir not
empty - leaving
[nyx6191.arc-ts.umich.edu:22287] sess_dir_finalize: top session dir not empty -
leaving
[nyx6191.arc-ts.umich.edu:22287] sess_dir_finalize: proc session dir does not
exist
[nyx6191.arc-ts.umich.edu:22287] sess_dir_finalize: job session dir not
emptyeaving
[nyx6213.arc-ts.umich.edu:29106] sess_dir_finalize: top session dir not empty -
leaving
[nyx6213.arc-ts.umich.edu:29106] sess_dir_finalize: proc session dir does not
exist
[nyx6213.arc-ts.umich.edu:29106] sess_dir_finalize: job session dir not emptyt
empty - leaving
[nyx6127.arc-ts.umich.edu:10607] sess_dir_finalize: top session dir not empty -
leaving
[nyx6127.arc-ts.umich.edu:10607] sess_dir_finalize: proc session dir does not
exist
[nyx6127.arc-ts.umich.edu:10607] sess_dir_finalize: job session di - leaving
[nyx6145.arc-ts.umich.edu:15451] sess_dir_finalize: jobfam session dir not
empty - leaving
[nyx6145.arc-ts.umich.edu:15451] sess_dir_finalize: top session dir not empty -
leaving
[nyx6145.arc-ts.umich.edu:15451] sess_dir_finalize: proc session deaving
[nyx6155.arc-ts.umich.edu:11158] sess_dir_finalize: top session dir not empty -
leaving
[nyx6155.arc-ts.umich.ed[nyx6219.arc-ts.umich.edu:07616] sess_dir_finalize:
proc session dir does not exist
u:1[nyx6219.arc-ts.umich.edu:07616] sess_dir_finalize: job session dir not
empty - leaving
11[nyx6219.arc-ts.umich.edu:07616] sess_dir_finalize: jobfam session dir not
empty - leaving
58][nyx6219.arc-ts.umich.edu:07616] sess_dir_finalize: top session dir not
empty - leaving
sess_dir_finalize: proc session dir does not exist
[nyx6155.arc-ts.umich.edu:11158][nyx6219.arc-ts.umich.edu:07616]
sess_dir_cleanup: job session dir does not exist
s[nyx6219.arc-ts.umich.edu:07616] [[59907,0],0] Releasing job data for
[59907,0]
ess_dir_finalize: job session dir not empty - leaving
[nyx6223.arc-ts.umich.edu:11585] sess_dir_finalize: jobfam session dir not
empty - leaving
[nyx6223.arc-ts.umich.edu:11585] sess_dir_finalize: top session dir not empty -
leaving
[nyx6223.arc-ts.umich.edu:11585] seexiting with status 1
ss_dir_finalize: proc session deaving
[nyx6233.arc-ts.umich.edu:11521] sess_dir_finalize: top session dir not empty -
leaving
[nyx6233.arc-ts.umich.edu:11521] sess_dir_finalize: proc session dir does not
exist
[nyx6233.arc-ts.umich.edu:11521] sess_dir_finalize: job session dir not empty -
leaving
[nyx6191.arc-ts.umich.edu:22287] sess_dir_finalize: jobfam session dir not
empty - leaving
[nyx6191.arc-ts.umich.edu:22287] sess_dir_finalize: top session dir not empty -
leaving
[nyx6191.arc-ts.umich.edu:22287] sess_dir_finalize: proc session d - leaving
[nyx6213.arc-ts.umich.edu:29106] sess_dir_finalize: jobfam session dir not
empty - leaving
[nyx6213.arc-ts.umich.edu:29106] sess_dir_finalize: top session dir not empty -
leaving
[nyx6213.arc-ts.umich.edu:29106] sess_dir_finalize: proc session dr not empty -
leaving
[nyx6127.arc-ts.umich.edu:10607] sess_dir_finalize: jobfam session dir not
empty - leaving
[nyx6127.arc-ts.umich.edu:10607] sess_dir_finalize: top session dir not empty -
leaving
[nyx6127.arc-ts.umich.edu:10607] sess_dir_finalize: proir does not exist
[nyx6145.arc-ts.umich.edu:15451] sess_dir_finalize: job session dir not empty -
leaving
[nyx6145.arc-ts.umich.edu:15451] sess_dir_finalize: jobfam session dir not
empty - leaving
[nyx6145.arc-ts.umich.edu:15451] sess_dir_finalize: top ses - leaving
[nyx6155.arc-ts.umich.edu:11158] sess_dir_finalize: jobfam session dir not
empty - leaving
[nyx6155.arc-ts.umich.edu:11158] sess_dir_finalize: top session dir not empty -
leaving
[nyx6155.arc-ts.umich.edu:11158] sess_dir_finalize: proc session dir does not
exist
[nyx6223.arc-ts.umich.edu:11585] sess_dir_finalize: job session dir not empty -
leaving
[nyx6223.arc-ts.umich.edu:11585] sess_dir_finalize: jobfam session dir not
empty - leaving
[nyx6223.arc-ts.umich.edu:11585] sess_dir_finalize: top ses - leaving
[nyx6233.arc-ts.umich.edu:11521] sess_dir_finalize: jobfam session dir not
empty - leaving
[nyx6233.arc-ts.umich.edu:11521] sess_dir_finalize: top session dir not empty -
leaving
[nyx6233.arc-ts.umich.edu:11521] sess_dir_finalize: proc session dir does not
exist
[nyx6191.arc-ts.umich.edu:22287] sess_dir_finalize: job session dir not empty -
leaving
[nyx6191.arc-ts.umich.edu:22287] sess_dir_finalize: jobfam session dir not
empty - leaving
[nyx6191.arc-ts.umich.edu:22287] sess_dir_finalize: top sesir does not exist
[nyx6213.arc-ts.umich.edu:29106] sess_dir_finalize: job session dir not empty -
leaving
[nyx6213.arc-ts.umich.edu:29106] sess_dir_finalize: jobfam session dir not
empty - leaving
[nyx6213.arc-ts.umich.edu:29106] sess_dir_finalize: top sesc session dir does
not exist
[nyx6127.arc-ts.umich.edu:10607] sess_dir_finalize: job session dir not empty -
leaving
[nyx6127.arc-ts.umich.edu:10607] sess_dir_finalize: jobfam session dir not
empty - leaving
[nyx6127.arc-ts.umich.edu:10607] sess_dir_finalision dir not empty - leaving
[nyx6145.arc-ts.umich.edu:15451] sess_dir_finalize: proc session dir does not
exist
[nyx6145.arc-ts.umich.edu:15451] sess_dir_finalize: job session dir not empty -
leaving
[nyx6145.arc-ts.umich.edu:15451] sess_dir_finalize: jobir does not exist
[nyx6155.arc-ts.umich.edu:11158] sess_dir_finalize: job session dir not empty -
leaving
[nyx6155.arc-ts.umich.edu:11158] sess_dir_finalize: jobfam session dir not
empty - leaving
[nyx6155.arc-ts.umich.edu:11158] sess_dir_finalize: top session dir not empty -
leaving
[nyx6223.arc-ts.umich.edu:11585] sess_dir_finalize: proc session dir does not
exist
[nyx6223.arc-ts.umich.edu:11585] sess_dir_finalize: job session dir not empty -
leaving
[nyx6223.arc-ts.umich.edu:11585] sess_dir_finalize: jobir does not exist
[nyx6233.arc-ts.umich.edu:11521] sess_dir_finalize: job session dir not empty -
leaving
[nyx6233.arc-ts.umich.edu:11521] sess_dir_finalize: jobfam session dir not
empty - leaving
[nyx6233.arc-ts.umich.edu:11521] sess_dir_finalize: top session dir not empty -
leaving
[nyx6191.arc-ts.umich.edu:22287] sess_dir_finalize: proc session dir does not
exist
[nyx6191.arc-ts.umich.edu:22287] sess_dir_finalize: job session dir not empty -
leaving
[nyx6191.arc-ts.umich.edu:22287] sess_dir_finalize: jobsion dir not empty -
leaving
[nyx6213.arc-ts.umich.edu:29106] sess_dir_finalize: proc session dir does not
exist
[nyx6213.arc-ts.umich.edu:29106] sess_dir_finalize: job session dir not empty -
leaving
[nyx6213.arc-ts.umich.edu:29106] sess_dir_finalize: jobze: top session dir not
empty - leaving
[nyx6127.arc-ts.umich.edu:10607] sess_dir_finalize: proc session dir does not
exist
[nyx6127.arc-ts.umich.edu:10607] sess_dir_finalize: job session dir not empty -
leaving
[nyx6127.arc-ts.umich.edu:10607] sess_dir_fifam session dir not empty - leaving
[nyx6145.arc-ts.umich.edu:15451] sess_dir_finalize: top session dir not empty -
leaving
[nyx6145.arc-ts.umich.edu:15451] sess_dir_finalize: proc session dir does not
exist
[nyx6145.arc-ts.umich.edu:15451] sess_dir_finalision dir not empty - leaving
[nyx6155.arc-ts.umich.edu:11158] sess_dir_finalize: proc session dir does not
exist
[nyx6155.arc-ts.umich.edu:11158] sess_dir_finalize: job session dir not empty -
leaving
[nyx6155.arc-ts.umich.edu:11158] sess_dir_finalize: jobfam session dir not
empty - leaving
[nyx6223.arc-ts.umich.edu:11585] sess_dir_finalize: top session dir not empty -
leaving
[nyx6223.arc-ts.umich.edu:11585] sess_dir_finalize: proc session dir does not
exist
[nyx6223.arc-ts.umich.edu:11585] sess_dir_finalision dir not empty - leaving
[nyx6233.arc-ts.umich.edu:11521] sess_dir_finalize: proc session dir does not
exist
[nyx6233.arc-ts.umich.edu:11521] sess_dir_finalize: job session dir not empty -
leaving
[nyx6233.arc-ts.umich.edu:11521] sess_dir_finalize: jobfam session dir not
empty - leaving
[nyx6191.arc-ts.umich.edu:22287] sess_dir_finalize: top session dir not empty -
leaving
[nyx6191.arc-ts.umich.edu:22287] sess_dir_finalize: proc session dir does not
exist
[nyx6191.arc-ts.umich.edu:22287] sess_dir_finalifam session dir not empty -
leaving
[nyx6213.arc-ts.umich.edu:29106] sess_dir_finalize: top session dir not empty -
leaving
[nyx6213.arc-ts.umich.edu:29106] sess_dir_finalize: proc session dir does not
exist
[nyx6213.arc-ts.umich.edu:29106] sess_dir_finalinalize: jobfam session dir not
empty - leaving
[nyx6127.arc-ts.umich.edu:10607] sess_dir_finalize: top session dir not empty -
leaving
[nyx6127.arc-ts.umich.edu:10607] sess_dir_cleanup: job session dir does not
exist
exiting with status 0
ze: job session dir not empty - leaving
[nyx6145.arc-ts.umich.edu:15451] sess_dir_finalize: jobfam session dir not
empty - leaving
[nyx6145.arc-ts.umich.edu:15451] sess_dir_finalize: top session dir not empty -
leaving
[nyx6145.arc-ts.umich.edu:15451] sessfam session dir not empty - leaving
[nyx6155.arc-ts.umich.edu:11158] sess_dir_finalize: top session dir not empty -
leaving
[nyx6155.arc-ts.umich.edu:11158] sess_dir_finalize: proc session dir does not
exist
[nyx6155.arc-ts.umich.edu:11158] sess_dir_finalize: job session dir not empty -
leaving
[nyx6223.arc-ts.umich.edu:11585] sess_dir_finalize: jobfam session dir not
empty - leaving
[nyx6223.arc-ts.umich.edu:11585] sess_dir_finalize: top session dir not empty -
leaving
[nyx6223.arc-ts.umich.edu:11585] sessfam session dir not empty - leaving
[nyx6233.arc-ts.umich.edu:11521] sess_dir_finalize: top session dir not empty -
leaving
[nyx6233.arc-ts.umich.edu:11521] sess_dir_finalize: proc session dir does not
exist
[nyx6233.arc-ts.umich.edu:11521] sess_dir_finalize: job session dir not empty -
leaving
[nyx6191.arc-ts.umich.edu:22287] sess_dir_finalize: jobfam session dir not
empty - leaving
[nyx6191.arc-ts.umich.edu:22287] sess_dir_finalize: top session dir not empty -
leaving
[nyx6191.arc-ts.umich.edu:22287] sessze: job session dir not empty - leaving
[nyx6213.arc-ts.umich.edu:29106] sess_dir_finalize: jobfam session dir not
empty - leaving
[nyx6213.arc-ts.umich.edu:29106] sess_dir_finalize: top session dir not empty -
leaving
[nyx6213.arc-ts.umich.edu:29106] sess_dir_cleanup: job session dir does not
exist
exiting with status 0
ze: j[bennet@nyx6219 singularity_testing]$ ob session dir not empty - leaving
[nyx6155.arc-ts.umich.edu:11158] sess_dir_finalize: jobfam session dir not
empty - leaving
[nyx6155.arc-ts.umich.edu:11158] sess_dir_finalize: top session dir not empty -
leaving
[nyx6155.arc-ts.umich.edu:11158] sess_dir_cleanup: job session dir does not
exist
exiting with status 0
ze: job session dir not empty - leaving
[nyx6233.arc-ts.umich.edu:11521] sess_dir_finalize: jobfam session dir not
empty - leaving
[nyx6233.arc-ts.umich.edu:11521] sess_dir_finalize: top session dir not empty -
leaving
[nyx6233.arc-ts.umich.edu:11521] sess_dir_cleanup: job session dir does not
exist
exiting with status 0
_dir_cleanup: job session dir does not exist
exiting with status 0
_dir_cleanup: job session dir does not exist
exiting with status 0
_dir_cleanup: job session dir does not exist
exiting with status 0
_______________________________________________
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
