Brian, i have no doubt this was enough to get rid of the warning messages.
out of curiosity, are you now able to experience performances close to native runs ? if i understand correctly, the linux kernel allocates memory on the closest NUMA domain (e.g. socket if i oversimplify), and since MPI tasks are bound by orted/mpirun before they are execv'ed, i have some hard time understanding how not binding MPI tasks to memory can have a significant impact on performances as long as they are bound on cores. Cheers, Gilles On Sat, Dec 23, 2017 at 7:27 AM, Brian Dobbins <bdobb...@gmail.com> wrote: > > Hi Ralph, > > Well, this gets chalked up to user error - the default AMI images come > without the NUMA-dev libraries, so OpenMPI didn't get built with it (and in > my haste, I hadn't checked). Oops. Things seem to be working correctly > now. > > Thanks again for your help, > - Brian > > > On Fri, Dec 22, 2017 at 2:14 PM, r...@open-mpi.org <r...@open-mpi.org> wrote: >> >> I honestly don’t know - will have to defer to Brian, who is likely out for >> at least the extended weekend. I’ll point this one to him when he returns. >> >> >> On Dec 22, 2017, at 1:08 PM, Brian Dobbins <bdobb...@gmail.com> wrote: >> >> >> Hi Ralph, >> >> OK, that certainly makes sense - so the next question is, what prevents >> binding memory to be local to particular cores? Is this possible in a >> virtualized environment like AWS HVM instances? >> >> And does this apply only to dynamic allocations within an instance, or >> static as well? I'm pretty unfamiliar with how the hypervisor (KVM-based, I >> believe) maps out 'real' hardware, including memory, to particular >> instances. We've seen some parts of the code (bandwidth heavy) run ~10x >> faster on bare-metal hardware, though, presumably from memory locality, so >> it certainly has a big impact. >> >> Thanks again, and merry Christmas! >> - Brian >> >> >> On Fri, Dec 22, 2017 at 1:53 PM, r...@open-mpi.org <r...@open-mpi.org> >> wrote: >>> >>> Actually, that message is telling you that binding to core is available, >>> but that we cannot bind memory to be local to that core. You can verify the >>> binding pattern by adding --report-bindings to your cmd line. >>> >>> >>> On Dec 22, 2017, at 11:58 AM, Brian Dobbins <bdobb...@gmail.com> wrote: >>> >>> >>> Hi all, >>> >>> We're testing a model on AWS using C4/C5 nodes and some of our timers, >>> in a part of the code with no communication, show really poor performance >>> compared to native runs. We think this is because we're not binding to a >>> core properly and thus not caching, and a quick 'mpirun --bind-to core >>> hostname' does suggest issues with this on AWS: >>> >>> [bdobbins@head run]$ mpirun --bind-to core hostname >>> >>> -------------------------------------------------------------------------- >>> WARNING: a request was made to bind a process. While the system >>> supports binding the process itself, at least one node does NOT >>> support binding memory to the process location. >>> >>> Node: head >>> >>> Open MPI uses the "hwloc" library to perform process and memory >>> binding. This error message means that hwloc has indicated that >>> processor binding support is not available on this machine. >>> >>> (It also happens on compute nodes, and with real executables.) >>> >>> Does anyone know how to enforce binding to cores on AWS instances? Any >>> insight would be great. >>> >>> Thanks, >>> - Brian >>> _______________________________________________ >>> users mailing list >>> users@lists.open-mpi.org >>> https://lists.open-mpi.org/mailman/listinfo/users >>> >>> >>> >>> _______________________________________________ >>> users mailing list >>> users@lists.open-mpi.org >>> https://lists.open-mpi.org/mailman/listinfo/users >> >> >> _______________________________________________ >> users mailing list >> users@lists.open-mpi.org >> https://lists.open-mpi.org/mailman/listinfo/users >> >> >> >> _______________________________________________ >> users mailing list >> users@lists.open-mpi.org >> https://lists.open-mpi.org/mailman/listinfo/users > > > > _______________________________________________ > users mailing list > users@lists.open-mpi.org > https://lists.open-mpi.org/mailman/listinfo/users _______________________________________________ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
