Re: [OMPI users] Binding to thread 0
Hi, @Gilles Awesome. --bind-to hwthread does the trick. I thought binding and mapping would not be read if using --rankfile. @Jeff I am running on a shared cluster. Some apps do use HT for achieving better performance. Nothing we could do with BIOS settings. Regards On 08/09/2023 17:53, Jeff Squyres (jsquyres) wrote: In addition to what Gilles mentioned, I'm curious: is there a reason you have hardware threads enabled? You could disable them in the BIOS, and then each of your MPI processes can use the full core, not just a single hardware thread. *From:* users on behalf of Luis Cebamanos via users *Sent:* Friday, September 8, 2023 7:10 AM *To:* Ralph Castain via users *Cc:* Luis Cebamanos *Subject:* [OMPI users] Binding to thread 0 Hello, Up to now, I have been using numerous ways of binding with wrappers (numactl, taskset) whenever I wanted to play with core placing. Another way I have been using is via -rankfile, however I notice that some ranks jump from thread 0 to thread 1 on SMT chips. I can control this with numactl for instance, but it would be great to see similar behaviour when using -rankfile. Is there a way to pack all ranks to one of the threads of each core (preferibly to thread 0) so I can nicely see all ranks with htop on either left or right of the screen? The command I am using is pretty simple: mpirun -np $MPIRANKS --rankfile ./myrankfile and ./myrankfile looks like rank 33=argon slot=33 rank 34=argon slot=34 rank 35=argon slot=35 rank 36=argon slot=36 Thanks!
Re: [OMPI users] Binding to thread 0
@Gilles @Jeff Sorry, I think I replied too quickly. This is what I see if using --bind-to hwthread This is not what I was after. I only want to use thread 0 of a core ie (cores 0-7), so "cores 192-199" should not have any activity. If I do --bind-to core, the activity jumps from "core 0" to "core 192", and I want to avoid that. Any other suggestion? Regards L On 08/09/2023 17:53, Jeff Squyres (jsquyres) wrote: In addition to what Gilles mentioned, I'm curious: is there a reason you have hardware threads enabled? You could disable them in the BIOS, and then each of your MPI processes can use the full core, not just a single hardware thread. *From:* users on behalf of Luis Cebamanos via users *Sent:* Friday, September 8, 2023 7:10 AM *To:* Ralph Castain via users *Cc:* Luis Cebamanos *Subject:* [OMPI users] Binding to thread 0 Hello, Up to now, I have been using numerous ways of binding with wrappers (numactl, taskset) whenever I wanted to play with core placing. Another way I have been using is via -rankfile, however I notice that some ranks jump from thread 0 to thread 1 on SMT chips. I can control this with numactl for instance, but it would be great to see similar behaviour when using -rankfile. Is there a way to pack all ranks to one of the threads of each core (preferibly to thread 0) so I can nicely see all ranks with htop on either left or right of the screen? The command I am using is pretty simple: mpirun -np $MPIRANKS --rankfile ./myrankfile and ./myrankfile looks like rank 33=argon slot=33 rank 34=argon slot=34 rank 35=argon slot=35 rank 36=argon slot=36 Thanks!
Re: [OMPI users] Binding to thread 0
Isn't this a case for --map-by core --bind-to hwthread? Because you want to map each process by core but bind the the first hwthread.From the looks of it your process is both binding and mapping by hwthread now. -NathanOn Sep 11, 2023, at 10:20 AM, Luis Cebamanos via users wrote:@Gilles @Jeff Sorry, I think I replied too quickly. This is what I see if using --bind-to hwthread This is not what I was after. I only want to use thread 0 of a core ie (cores 0-7), so "cores 192-199" should not have any activity. If I do --bind-to core, the activity jumps from "core 0" to "core 192", and I want to avoid that. Any other suggestion? Regards L On 08/09/2023 17:53, Jeff Squyres (jsquyres) wrote:In addition to what Gilles mentioned, I'm curious: is there a reason you have hardware threads enabled? You could disable them in the BIOS, and then each of your MPI processes can use the full core, not just a single hardware thread.From: users on behalf of Luis Cebamanos via users Sent: Friday, September 8, 2023 7:10 AM To: Ralph Castain via users Cc: Luis Cebamanos Subject: [OMPI users] Binding to thread 0 Hello,Up to now, I have been using numerous ways of binding with wrappers (numactl, taskset) whenever I wanted to play with core placing. Another way I have been using is via -rankfile, however I notice that some ranks jump from thread 0 to thread 1 on SMT chips. I can control this with numactl for instance, but it would be great to see similar behaviour when using -rankfile. Is there a way to pack all ranks to one of the threads of each core (preferibly to thread 0) so I can nicely see all ranks with htop on either left or right of the screen?The command I am using is pretty simple:mpirun -np $MPIRANKS --rankfile ./myrankfile and ./myrankfile looks likerank 33=argon slot=33 rank 34=argon slot=34 rank 35=argon slot=35 rank 36=argon slot=36Thanks!
Re: [OMPI users] Binding to thread 0
HIi Nathan, Isn't this a case for --map-by core --bind-to hwthread? Because you want to map each process by core but bind the the first hwthread. It is, but indicating a rankfile I can map to certain core numbers. I cannot use both --map-by and --rankfile: Conflicting directives for mapping policy are causing the policy to be redefined: New policy: RANK_FILE Prior policy: BYCORE Regards, L From the looks of it your process is both binding and mapping by hwthread now. -Nathan On Sep 11, 2023, at 10:20 AM, Luis Cebamanos via users wrote: @Gilles @Jeff Sorry, I think I replied too quickly. This is what I see if using --bind-to hwthread This is not what I was after. I only want to use thread 0 of a core ie (cores 0-7), so "cores 192-199" should not have any activity. If I do --bind-to core, the activity jumps from "core 0" to "core 192", and I want to avoid that. Any other suggestion? Regards L On 08/09/2023 17:53, Jeff Squyres (jsquyres) wrote: In addition to what Gilles mentioned, I'm curious: is there a reason you have hardware threads enabled? You could disable them in the BIOS, and then each of your MPI processes can use the full core, not just a single hardware thread. *From:* users on behalf of Luis Cebamanos via users *Sent:* Friday, September 8, 2023 7:10 AM *To:* Ralph Castain via users *Cc:* Luis Cebamanos *Subject:* [OMPI users] Binding to thread 0 Hello, Up to now, I have been using numerous ways of binding with wrappers (numactl, taskset) whenever I wanted to play with core placing. Another way I have been using is via -rankfile, however I notice that some ranks jump from thread 0 to thread 1 on SMT chips. I can control this with numactl for instance, but it would be great to see similar behaviour when using -rankfile. Is there a way to pack all ranks to one of the threads of each core (preferibly to thread 0) so I can nicely see all ranks with htop on either left or right of the screen? The command I am using is pretty simple: mpirun -np $MPIRANKS --rankfile ./myrankfile and ./myrankfile looks like rank 33=argon slot=33 rank 34=argon slot=34 rank 35=argon slot=35 rank 36=argon slot=36 Thanks!
[OMPI users] syntax of the mpirun command [SEC=OFFICIAL]
What is the syntax of the mpirun command running an application built with OpenMPI 4.1.4 to place the first N MPI processes with a density of n PEs per a Cascade Lake node (n/2 per socket, each node has 48 cores) and the next M (N+1, N+2, ..., N+M) MPI processes after that with a density of m MPI processes per node (m/2 PEs per socket)? Thank you.
