Hi Mike, It sure is possible to tune for a particular code. Especially if one aims at getting best performance numbers.
Thats one thing; however when a communication library (MXM) imposes limits that might be conflicting with limits of some applications , its another. Maintaining stacks of MPI's and compilers is already quite a task; making one MPI build for this app and another MPI build for that app is sort of, not really practical, nor is educating users to set ulimits and mca parameters per each job. -- Grigory Shamov Westgrid/ComputeCanada Site Lead University of Manitoba E2-588 EITC Building, (204) 474-9625 From: users <users-boun...@open-mpi.org<mailto:users-boun...@open-mpi.org>> on behalf of Mike Dubman <mi...@dev.mellanox.co.il<mailto:mi...@dev.mellanox.co.il>> Reply-To: Open MPI Users <us...@open-mpi.org<mailto:us...@open-mpi.org>> List-Post: users@lists.open-mpi.org Date: Tuesday, 29 September, 2015 11:10 AM To: Open MPI Users <us...@open-mpi.org<mailto:us...@open-mpi.org>> Subject: Re: [OMPI users] Using OpenMPI (1.8, 1.10) with Mellanox MXM, ulimits ? unfortunately, there is no one size fits all here. mxm provides best performance for IB. different application may require different OMPI, mxm, OS tunables and requires some performance engineering. On Mon, Sep 28, 2015 at 9:49 PM, Grigory Shamov <grigory.sha...@umanitoba.ca<mailto:grigory.sha...@umanitoba.ca>> wrote: Hi Nathan, Hi Mike, Thanks for the quick replies! My problem is I don't know what are my applications. I mean, I know them, but we are a general purpose cluster, running in production for quite a while, and there are everybody, from quantum chemists to machine learners to bioinformatists. SO a system-wide change might harm some of them; and doing per-app benchmarking/tuning looks a bit daunting. The default behaviour our users are used to was to have unlimited values for all memory limits. We have set it so a few years ago, as a response for some user complaints that applications won't start (we set the ulimits in Torque). Is it known (I know every application is different ) how much costs, performance-wise, to have MXM with good ulimits vs unlimited ulimits, vs not using MXM at all? -- Grigory Shamov Westgrid/ComputeCanada Site Lead University of Manitoba E2-588 EITC Building, (204) 474-9625<tel:%28204%29%20474-9625> On 15-09-28 12:58 PM, "users on behalf of Nathan Hjelm" <users-boun...@open-mpi.org<mailto:users-boun...@open-mpi.org> on behalf of hje...@lanl.gov<mailto:hje...@lanl.gov>> wrote: > >I would like to add that you may want to play with the value and see >what works for your applications. Most applications should be using >malloc or similar functions to allocate large memory regions in the heap >and not on the stack. > >-Nathan > >On Mon, Sep 28, 2015 at 08:01:09PM +0300, Mike Dubman wrote: >> Hello Grigory, >> We observed ~10% performance degradation with heap size set to >>unlimited >> for CFD applications. >> You can measure your application performance with default and >>unlimited >> "limits" and select the best setting. >> Kind Regards. >> M >> On Mon, Sep 28, 2015 at 7:36 PM, Grigory Shamov >> <grigory.sha...@umanitoba.ca<mailto:grigory.sha...@umanitoba.ca>> wrote: >> >> Hi All, >> >> We have built OpenMPI (1.8.8., 1.10.0) against Mellanox OFED 2.4 >>and >> corresponding MXM. When it runs now, it gives the following >>warning, per >> process: >> >> [1443457390.911053] [myhist:5891 :0] mxm.c:185 MXM WARN >>The >> 'ulimit -s' on the system is set to 'unlimited'. This may have >>negative >> performance implications. Please set the heap size to the default >>value >> (10240) >> >> We have ulimits for heap (as well as most of the other limits) set >> unlimited because of applications that might possibly need a lot >>of RAM. >> >> The question is if we should do as MXM wants, or ignore it? Has >>anyone >> an >> experience running recent OpenMPI with MXM enabled, and what kind >>of >> ulimits do you have? Any suggestions/comments appreciated, thanks! >> >> -- >> Grigory Shamov >> >> Westgrid/ComputeCanada Site Lead >> University of Manitoba >> E2-588 EITC Building, >> (204) 474-9625 >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org<mailto:us...@open-mpi.org> >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >> Link to this post: >> http://www.open-mpi.org/community/lists/users/2015/09/27697.php >> >> -- >> Kind Regards, >> M. > >> _______________________________________________ >> users mailing list >> us...@open-mpi.org<mailto:us...@open-mpi.org> >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >> Link to this post: >>http://www.open-mpi.org/community/lists/users/2015/09/27698.php > _______________________________________________ users mailing list us...@open-mpi.org<mailto:us...@open-mpi.org> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users Link to this post: http://www.open-mpi.org/community/lists/users/2015/09/27701.php -- Kind Regards, M.
