Greg,
If Open MPI was built with UCX, your jobs will likely use UCX (and the
shared memory provider) even if running on a single node.
You can
mpirun --mca pml ob1 --mca btl self,sm ...
if you want to avoid using UCX.
What is a typical mpirun command line used under the hood by your "make
test"?
Though the warning might be ignored, SIGILL is definitely an issue.
I encourage you to have your app dump a core in order to figure out where
this is coming from
Cheers,
Gilles
On Tue, Apr 16, 2024 at 5:20 AM Greg Samonds via users <
users@lists.open-mpi.org> wrote:
> Hello,
>
>
>
> We’re running into issues with jobs failing in a non-deterministic way
> when running multiple jobs concurrently within a “make test” framework.
>
>
>
> Make test is launched from within a shell script running inside a Podman
> container, and we’re typically running with “-j 20” and “-np 4” (20 jobs
> concurrently with 4 procs each). We’ve also tried reducing the number of
> jobs to no avail. Each time the battery of test cases is run, about 2 to 4
> different jobs out of around 200 fail with the following errors:
>
>
>
>
> *[podman-ci-rocky-8.8:03528] MCW rank 1 is not bound (or bound to all
> available processors) [podman-ci-rocky-8.8:03540] MCW rank 3 is not bound
> (or bound to all available processors) [podman-ci-rocky-8.8:03519] MCW rank
> 0 is not bound (or bound to all available processors)
> [podman-ci-rocky-8.8:03533] MCW rank 2 is not bound (or bound to all
> available processors) *
>
> *Program received signal SIGILL: Illegal instruction.*
>
> Some info about our setup:
>
>- Ampere Altra 80 core ARM machine
>- Open MPI 4.1.7a1 from HPC-X v2.18
>- Rocky Linux 8.6 host, Rocky Linux 8.8 container
>- Podman 4.4.1
>- This machine has a Mellanox Connect X-6 Lx NIC, however we’re
>avoiding the Mellanox software stack by running in a container, and these
>are single node jobs only
>
>
>
> We tried passing “—bind-to none” to the running jobs, and while this
> seemed to reduce the number of failing jobs on average, it didn’t eliminate
> the issue.
>
>
>
> We also encounter the following warning:
>
>
>
> *[1712927028.412063] [**podman-ci-rocky-8:3519 :0]sock.c:514
> UCX WARN unable to read somaxconn value from /proc/sys/net/core/somaxconn
> file*
>
>
>
> …however as far as I can tell this is probably unrelated and occurs
> because the associated file isn’t accessible inside the container, and
> after checking the UCX source I can see that SOMAXCONN is picked up from
> the system headers anyway.
>
>
>
> If anyone has hints about how to workaround this issue we’d greatly
> appreciate it!
>
>
>
> Thanks,
>
> Greg
>
