Jumping in a little late… Today, EC2 instances don’t expose all the required information for memory pinning to work, which is why you see the warning. The action-less error message is obviously a bit annoying (although it makes sense in the general case), but we haven’t had the time to work out the right balance between warning the user that Open MPI can’t bind memory (which is what many users expect when they bind to core) and annoying the user.
I’m sure that having nearly twice as much L3 cache will improve performance on HPC applications. Depending on what your existing cluster looks like, it may also have more memory bandwidth compared to C4 instances. And, of course, there’s the fact that C4 is 10 Gbps ethernet, vs. whatever you’re using in your existing cluster. Finally, you didn’t mention which version of Open MPI you’re using. If you’re using a version before 2.1.2, there were some antiquated choices for TCP buffer tuning parameters that were causing some fairly severe performance problems on EC2 instances in the multi-node parallel case. Brian On Dec 22, 2017, at 7:14 PM, Brian Dobbins <bdobb...@gmail.com<mailto:bdobb...@gmail.com>> wrote: Hi Gilles, You're right, we no longer get warnings... and the performance disparity still exists, though to be clear it's only in select parts of the code - others run as we'd expect. This is probably why I initially guessed it was a process/memory affinity issue - the one timer I looked at is in a memory-intensive part of the code. Now I'm wondering if we're still getting issues binding (I need to do a comparison with a local system), or if it could be due to the cache size differences - the AWS C4 instances have 25MB/socket, and we have 45MB/socket. If we fit in cache on our system, and don't on theirs, that could account for things. Testing that is next up on my list, actually. Cheers, - Brian On Fri, Dec 22, 2017 at 7:55 PM, Gilles Gouaillardet <gilles.gouaillar...@gmail.com<mailto:gilles.gouaillar...@gmail.com>> wrote: Brian, i have no doubt this was enough to get rid of the warning messages. out of curiosity, are you now able to experience performances close to native runs ? if i understand correctly, the linux kernel allocates memory on the closest NUMA domain (e.g. socket if i oversimplify), and since MPI tasks are bound by orted/mpirun before they are execv'ed, i have some hard time understanding how not binding MPI tasks to memory can have a significant impact on performances as long as they are bound on cores. Cheers, Gilles On Sat, Dec 23, 2017 at 7:27 AM, Brian Dobbins <bdobb...@gmail.com<mailto:bdobb...@gmail.com>> wrote: > > Hi Ralph, > > Well, this gets chalked up to user error - the default AMI images come > without the NUMA-dev libraries, so OpenMPI didn't get built with it (and in > my haste, I hadn't checked). Oops. Things seem to be working correctly > now. > > Thanks again for your help, > - Brian > > > On Fri, Dec 22, 2017 at 2:14 PM, r...@open-mpi.org<mailto:r...@open-mpi.org> > <r...@open-mpi.org<mailto:r...@open-mpi.org>> wrote: >> >> I honestly don’t know - will have to defer to Brian, who is likely out for >> at least the extended weekend. I’ll point this one to him when he returns. >> >> >> On Dec 22, 2017, at 1:08 PM, Brian Dobbins >> <bdobb...@gmail.com<mailto:bdobb...@gmail.com>> wrote: >> >> >> Hi Ralph, >> >> OK, that certainly makes sense - so the next question is, what prevents >> binding memory to be local to particular cores? Is this possible in a >> virtualized environment like AWS HVM instances? >> >> And does this apply only to dynamic allocations within an instance, or >> static as well? I'm pretty unfamiliar with how the hypervisor (KVM-based, I >> believe) maps out 'real' hardware, including memory, to particular >> instances. We've seen some parts of the code (bandwidth heavy) run ~10x >> faster on bare-metal hardware, though, presumably from memory locality, so >> it certainly has a big impact. >> >> Thanks again, and merry Christmas! >> - Brian >> >> >> On Fri, Dec 22, 2017 at 1:53 PM, r...@open-mpi.org<mailto:r...@open-mpi.org> >> <r...@open-mpi.org<mailto:r...@open-mpi.org>> >> wrote: >>> >>> Actually, that message is telling you that binding to core is available, >>> but that we cannot bind memory to be local to that core. You can verify the >>> binding pattern by adding --report-bindings to your cmd line. >>> >>> >>> On Dec 22, 2017, at 11:58 AM, Brian Dobbins >>> <bdobb...@gmail.com<mailto:bdobb...@gmail.com>> wrote: >>> >>> >>> Hi all, >>> >>> We're testing a model on AWS using C4/C5 nodes and some of our timers, >>> in a part of the code with no communication, show really poor performance >>> compared to native runs. We think this is because we're not binding to a >>> core properly and thus not caching, and a quick 'mpirun --bind-to core >>> hostname' does suggest issues with this on AWS: >>> >>> [bdobbins@head run]$ mpirun --bind-to core hostname >>> >>> -------------------------------------------------------------------------- >>> WARNING: a request was made to bind a process. While the system >>> supports binding the process itself, at least one node does NOT >>> support binding memory to the process location. >>> >>> Node: head >>> >>> Open MPI uses the "hwloc" library to perform process and memory >>> binding. This error message means that hwloc has indicated that >>> processor binding support is not available on this machine. >>> >>> (It also happens on compute nodes, and with real executables.) >>> >>> Does anyone know how to enforce binding to cores on AWS instances? Any >>> insight would be great. >>> >>> Thanks, >>> - Brian >>> _______________________________________________ >>> users mailing list >>> users@lists.open-mpi.org<mailto:users@lists.open-mpi.org> >>> https://lists.open-mpi.org/mailman/listinfo/users >>> >>> >>> >>> _______________________________________________ >>> users mailing list >>> users@lists.open-mpi.org<mailto:users@lists.open-mpi.org> >>> https://lists.open-mpi.org/mailman/listinfo/users >> >> >> _______________________________________________ >> users mailing list >> users@lists.open-mpi.org<mailto:users@lists.open-mpi.org> >> https://lists.open-mpi.org/mailman/listinfo/users >> >> >> >> _______________________________________________ >> users mailing list >> users@lists.open-mpi.org<mailto:users@lists.open-mpi.org> >> https://lists.open-mpi.org/mailman/listinfo/users > > > > _______________________________________________ > users mailing list > users@lists.open-mpi.org<mailto:users@lists.open-mpi.org> > https://lists.open-mpi.org/mailman/listinfo/users _______________________________________________ users mailing list users@lists.open-mpi.org<mailto:users@lists.open-mpi.org> https://lists.open-mpi.org/mailman/listinfo/users _______________________________________________ users mailing list users@lists.open-mpi.org<mailto:users@lists.open-mpi.org> https://lists.open-mpi.org/mailman/listinfo/users
_______________________________________________ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
