Re: [OMPI users] [OMPI devel] There are not enough slots available in the system to satisfy the 2, slots that were requested by the application

2022-11-14 Thread Jeff Squyres (jsquyres) via users 
Yes, somehow I'm not seeing all the output that I expect to see.  Can you 
ensure that if you're copy-and-pasting from the email, that it's actually using 
"dash dash" in front of "mca" and "machinefile" (vs. a copy-and-pasted "em 
dash")?

--
Jeff Squyres
jsquy...@cisco.com

From: users  on behalf of Gilles Gouaillardet 
via users 
Sent: Sunday, November 13, 2022 9:18 PM
To: Open MPI Users 
Cc: Gilles Gouaillardet 
Subject: Re: [OMPI users] [OMPI devel] There are not enough slots available in 
the system to satisfy the 2, slots that were requested by the application

There is a typo in your command line.
You should use --mca (minus minus) instead of -mca

Also, you can try --machinefile instead of -machinefile

Cheers,

Gilles

There are not enough slots available in the system to satisfy the 2
slots that were requested by the application:

  –mca

On Mon, Nov 14, 2022 at 11:04 AM timesir via users 
mailto:users@lists.open-mpi.org>> wrote:

(py3.9) ➜  /share  mpirun -n 2 -machinefile hosts –mca rmaps_base_verbose 100 
--mca ras_base_verbose 100  which mpirun
[computer01:04570] mca: base: component_find: searching NULL for ras components
[computer01:04570] mca: base: find_dyn_components: checking NULL for ras 
components
[computer01:04570] pmix:mca: base: components_register: registering framework 
ras components
[computer01:04570] pmix:mca: base: components_register: found loaded component 
simulator
[computer01:04570] pmix:mca: base: components_register: component simulator 
register function successful
[computer01:04570] pmix:mca: base: components_register: found loaded component 
pbs
[computer01:04570] pmix:mca: base: components_register: component pbs register 
function successful
[computer01:04570] pmix:mca: base: components_register: found loaded component 
slurm
[computer01:04570] pmix:mca: base: components_register: component slurm 
register function successful
[computer01:04570] mca: base: components_open: opening ras components
[computer01:04570] mca: base: components_open: found loaded component simulator
[computer01:04570] mca: base: components_open: found loaded component pbs
[computer01:04570] mca: base: components_open: component pbs open function 
successful
[computer01:04570] mca: base: components_open: found loaded component slurm
[computer01:04570] mca: base: components_open: component slurm open function 
successful
[computer01:04570] mca:base:select: Auto-selecting ras components
[computer01:04570] mca:base:select:(  ras) Querying component [simulator]
[computer01:04570] mca:base:select:(  ras) Querying component [pbs]
[computer01:04570] mca:base:select:(  ras) Querying component [slurm]
[computer01:04570] mca:base:select:(  ras) No component selected!

==   ALLOCATED NODES   ==   
   [10/1444]
computer01: slots=1 max_slots=0 slots_inuse=0 state=UP
Flags: DAEMON_LAUNCHED:LOCATION_VERIFIED:SLOTS_GIVEN
aliases: 192.168.180.48
192.168.60.203: slots=1 max_slots=0 slots_inuse=0 
state=UNKNOWN
Flags: SLOTS_GIVEN
aliases: NONE
=

==   ALLOCATED NODES   ==
computer01: slots=1 max_slots=0 slots_inuse=0 state=UP
Flags: DAEMON_LAUNCHED:LOCATION_VERIFIED:SLOTS_GIVEN
aliases: 192.168.180.48
hepslustretest03: slots=1 max_slots=0 slots_inuse=0 state=UP
Flags: DAEMON_LAUNCHED:LOCATION_VERIFIED:SLOTS_GIVEN
aliases: 192.168.60.203,172.17.180.203,172.168.10.23,172.168.10.143
=
--
There are not enough slots available in the system to satisfy the 2
slots that were requested by the application:

  –mca

Either request fewer procs for your application, or make more slots
available for use.

A "slot" is the PRRTE term for an allocatable unit where we can
launch a process.  The number of slots available are defined by the
environment in which PRRTE processes are run:

  1. Hostfile, via "slots=N" clauses (N defaults to number of
 processor cores if not provided)
  2. The --host command line parameter, via a ":N" suffix on the
 hostname (N defaults to 1 if not provided)
  3. Resource manager (e.g., SLURM, PBS/Torque, LSF, etc.)
  4. If none of a hostfile, the --host command line parameter, or an
 RM is present, PRRTE defaults to the number of processor cores

In all the above cases, if you want PRRTE to default to the number
of hardware threads instead of the number of processor cores, use the
--use-hwthread-cpus option.

Alternatively, you can use the --map-by :OVERSUBSCRIBE option to ignore the
number of available slots when deciding the number of processes to
launch.
---

Re: [OMPI users] Tracing of openmpi internal functions

2022-11-14 Thread Jeff Squyres (jsquyres) via users 
Open MPI uses plug-in modules for its implementations of the MPI collective 
algorithms.  From that perspective, once you understand that infrastructure, 
it's exactly the same regardless of whether the MPI job is using intra-node or 
inter-node collectives.

We don't have much in the way of detailed internal function call tracing inside 
Open MPI itself, due to performance considerations.  You might want to look 
into flamegraphs, or something similar...?

--
Jeff Squyres
jsquy...@cisco.com

From: users  on behalf of arun c via users 

Sent: Saturday, November 12, 2022 9:46 AM
To: users@lists.open-mpi.org 
Cc: arun c 
Subject: [OMPI users] Tracing of openmpi internal functions

Hi All,

I am new to openmpi and trying to learn the internals (source code
level) of data transfer during collective operations. At first, I will
limit it to intra-node (between cpu cores, and sockets) to minimize
the scope of learning.

What are the best options (Looking for only free and open methods) for
tracing the openmpi code? (say I want to execute alltoall collective
and trace all the function calls and event callbacks that happened
inside the libmpi.so on all the cores)

Linux kernel has something called ftrace, it gives a neat call graph
of all the internal functions inside the kernel with time, is
something similar available?

--Arun

Re: [OMPI users] users Digest, Vol 4818, Issue 1

2022-11-14 Thread timesir via users 
 unable to replicate your error:


$ mpirun --version

mpirun (Open MPI) 5.0.0rc9


Report bugs tohttps://www.open-mpi.org/community/help/

$ cat hostfile

mpi002 slots=1

mpi005 slots=1

$ mpirun -n 2 --machinefile hostfile hostname

mpi002

mpi005

Can you try running with "--mca rmaps_base_verbose 100" so that we can get some 
debugging output and see why the slots aren't working for you?  Show the full output, 
like I did above (e.g., cat the hostfile, and then mpirun with the MCA param and all the 
output).  Thanks!

--
Jeff Squyres
jsquy...@cisco.com<mailto:jsquy...@cisco.com>

From: devel<mailto:devel-boun...@lists.open-mpi.org>  on 
behalf of mrlong via devel<mailto:de...@lists.open-mpi.org>
Sent: Monday, November 7, 2022 3:37 AM
To:de...@lists.open-mpi.org<mailto:de...@lists.open-mpi.org>  
<mailto:de...@lists.open-mpi.org>; Open MPI 
Users<mailto:users@lists.open-mpi.org>
Cc: mrlong<mailto:mrlong...@gmail.com>
Subject: [OMPI devel] There are not enough slots available in the system to 
satisfy the 2, slots that were requested by the application


Two machines, each with 64 cores. The contents of the hosts file are:

192.168.180.48 slots=1
192.168.60.203 slots=1

Why do you get the following error when running with openmpi 5.0.0rc9?

(py3.9) [user@machine01 share]0.5692263713929891nbsp; mpirun -n 2 --machinefile 
hosts hostname
--
There are not enough slots available in the system to satisfy the 2
slots that were requested by the application:

   hostname

Either request fewer procs for your application, or make more slots
available for use.

A "slot" is the PRRTE term for an allocatable unit where we can
launch a process.  The number of slots available are defined by the
environment in which PRRTE processes are run:

   1. Hostfile, via "slots=N" clauses (N defaults to number of
  processor cores if not provided)
   2. The --host command line parameter, via a ":N" suffix on the
  hostname (N defaults to 1 if not provided)
   3. Resource manager (e.g., SLURM, PBS/Torque, LSF, etc.)
   4. If none of a hostfile, the --host command line parameter, or an
  RM is present, PRRTE defaults to the number of processor cores

In all the above cases, if you want PRRTE to default to the number
of hardware threads instead of the number of processor cores, use the
--use-hwthread-cpus option.

Alternatively, you can use the --map-by :OVERSUBSCRIBE option to ignore the
number of available slots when deciding the number of processes to
launch.

-- next part --
An HTML attachment was scrubbed...
URL:<https://lists.open-mpi.org/mailman/private/users/attachments/20221114/2c75fc85/attachment.html>

--

Message: 2
Date: Mon, 14 Nov 2022 18:04:06 +
From: "Jeff Squyres (jsquyres)"
To:"users@lists.open-mpi.org"  
Cc: arun c
Subject: Re: [OMPI users] Tracing of openmpi internal functions
Message-ID:



Content-Type: text/plain; charset="us-ascii"

Open MPI uses plug-in modules for its implementations of the MPI collective 
algorithms.  From that perspective, once you understand that infrastructure, 
it's exactly the same regardless of whether the MPI job is using intra-node or 
inter-node collectives.

We don't have much in the way of detailed internal function call tracing inside 
Open MPI itself, due to performance considerations.  You might want to look 
into flamegraphs, or something similar...?

--
Jeff Squyres
jsquy...@cisco.com

From: users  on behalf of arun c via 
users
Sent: Saturday, November 12, 2022 9:46 AM
To:users@lists.open-mpi.org  
Cc: arun c
Subject: [OMPI users] Tracing of openmpi internal functions

Hi All,

I am new to openmpi and trying to learn the internals (source code
level) of data transfer during collective operations. At first, I will
limit it to intra-node (between cpu cores, and sockets) to minimize
the scope of learning.

What are the best options (Looking for only free and open methods) for
tracing the openmpi code? (say I want to execute alltoall collective
and trace all the function calls and event callbacks that happened
inside the libmpi.so on all the cores)

Linux kernel has something called ftrace, it gives a neat call graph
of all the internal functions inside the kernel with time, is
something similar available?

--Arun
-- next part --
An HTML attachment was scrubbed...
URL:<https://lists.open-mpi.org/mailman/private/users/attachments/20221114/0c9d0e69/attachment.html>

--

Subject: Digest Footer

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

--

End of users Digest, Vol 4818, Issue 1
**

Re: [OMPI users] users Digest, Vol 4818, Issue 1

2022-11-14 Thread Gilles Gouaillardet via users 
ce we're not talking 
about the internals of Open MPI itself), so I'll reply just on the users list.

For what it's worth, I'm unable to replicate your error:


$ mpirun --version

mpirun (Open MPI) 5.0.0rc9


Report bugs tohttps://www.open-mpi.org/community/help/

$ cat hostfile

mpi002 slots=1

mpi005 slots=1

$ mpirun -n 2 --machinefile hostfile hostname

mpi002

mpi005

Can you try running with "--mca rmaps_base_verbose 100" so that we can get some 
debugging output and see why the slots aren't working for you?  Show the full output, 
like I did above (e.g., cat the hostfile, and then mpirun with the MCA param and all the 
output).  Thanks!

--
Jeff Squyres
jsquy...@cisco.com<mailto:jsquy...@cisco.com>

From: devel<mailto:devel-boun...@lists.open-mpi.org>  on 
behalf of mrlong via devel<mailto:de...@lists.open-mpi.org>
Sent: Monday, November 7, 2022 3:37 AM
To:de...@lists.open-mpi.org<mailto:de...@lists.open-mpi.org>  
<mailto:de...@lists.open-mpi.org>; Open MPI 
Users<mailto:users@lists.open-mpi.org>
Cc: mrlong<mailto:mrlong...@gmail.com>
Subject: [OMPI devel] There are not enough slots available in the system to 
satisfy the 2, slots that were requested by the application


Two machines, each with 64 cores. The contents of the hosts file are:

192.168.180.48 slots=1
192.168.60.203 slots=1

Why do you get the following error when running with openmpi 5.0.0rc9?

(py3.9) [user@machine01 share]0.5692263713929891nbsp; mpirun -n 2 --machinefile 
hosts hostname
--
There are not enough slots available in the system to satisfy the 2
slots that were requested by the application:

   hostname

Either request fewer procs for your application, or make more slots
available for use.

A "slot" is the PRRTE term for an allocatable unit where we can
launch a process.  The number of slots available are defined by the
environment in which PRRTE processes are run:

   1. Hostfile, via "slots=N" clauses (N defaults to number of
  processor cores if not provided)
   2. The --host command line parameter, via a ":N" suffix on the
  hostname (N defaults to 1 if not provided)
   3. Resource manager (e.g., SLURM, PBS/Torque, LSF, etc.)
   4. If none of a hostfile, the --host command line parameter, or an
  RM is present, PRRTE defaults to the number of processor cores

In all the above cases, if you want PRRTE to default to the number
of hardware threads instead of the number of processor cores, use the
--use-hwthread-cpus option.

Alternatively, you can use the --map-by :OVERSUBSCRIBE option to ignore the
number of available slots when deciding the number of processes to
launch.

-- next part --
An HTML attachment was scrubbed...
URL:<https://lists.open-mpi.org/mailman/private/users/attachments/20221114/2c75fc85/attachment.html>

--

Message: 2
Date: Mon, 14 Nov 2022 18:04:06 +
From: "Jeff Squyres (jsquyres)"
To:"users@lists.open-mpi.org"  
Cc: arun c
Subject: Re: [OMPI users] Tracing of openmpi internal functions
Message-ID:



Content-Type: text/plain; charset="us-ascii"

Open MPI uses plug-in modules for its implementations of the MPI collective 
algorithms.  From that perspective, once you understand that infrastructure, 
it's exactly the same regardless of whether the MPI job is using intra-node or 
inter-node collectives.

We don't have much in the way of detailed internal function call tracing inside 
Open MPI itself, due to performance considerations.  You might want to look 
into flamegraphs, or something similar...?

--
Jeff Squyres
jsquy...@cisco.com

From: users  on behalf of arun c via 
users
Sent: Saturday, November 12, 2022 9:46 AM
To:users@lists.open-mpi.org  
Cc: arun c
Subject: [OMPI users] Tracing of openmpi internal functions

Hi All,

I am new to openmpi and trying to learn the internals (source code
level) of data transfer during collective operations. At first, I will
limit it to intra-node (between cpu cores, and sockets) to minimize
the scope of learning.

What are the best options (Looking for only free and open methods) for
tracing the openmpi code? (say I want to execute alltoall collective
and trace all the function calls and event callbacks that happened
inside the libmpi.so on all the cores)

Linux kernel has something called ftrace, it gives a neat call graph
of all the internal functions inside the kernel with time, is
something similar available?

--Arun
-- next part --
An HTML attachment was scrubbed...
URL:<https://lists.open-mpi.org/mailman/private/users/attachments/20221114/0c9d0e69/attachment.html>

--

Subject: Digest Footer

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

--

End of users Digest, Vol 4818, Issue 1
**