Hi Nathan, Hi Mike, Thanks for the quick replies!
My problem is I don't know what are my applications. I mean, I know them, but we are a general purpose cluster, running in production for quite a while, and there are everybody, from quantum chemists to machine learners to bioinformatists. SO a system-wide change might harm some of them; and doing per-app benchmarking/tuning looks a bit daunting. The default behaviour our users are used to was to have unlimited values for all memory limits. We have set it so a few years ago, as a response for some user complaints that applications won't start (we set the ulimits in Torque). Is it known (I know every application is different ) how much costs, performance-wise, to have MXM with good ulimits vs unlimited ulimits, vs not using MXM at all? -- Grigory Shamov Westgrid/ComputeCanada Site Lead University of Manitoba E2-588 EITC Building, (204) 474-9625 On 15-09-28 12:58 PM, "users on behalf of Nathan Hjelm" <users-boun...@open-mpi.org on behalf of hje...@lanl.gov> wrote: > >I would like to add that you may want to play with the value and see >what works for your applications. Most applications should be using >malloc or similar functions to allocate large memory regions in the heap >and not on the stack. > >-Nathan > >On Mon, Sep 28, 2015 at 08:01:09PM +0300, Mike Dubman wrote: >> Hello Grigory, >> We observed ~10% performance degradation with heap size set to >>unlimited >> for CFD applications. >> You can measure your application performance with default and >>unlimited >> "limits" and select the best setting. >> Kind Regards. >> M >> On Mon, Sep 28, 2015 at 7:36 PM, Grigory Shamov >> <grigory.sha...@umanitoba.ca> wrote: >> >> Hi All, >> >> We have built OpenMPI (1.8.8., 1.10.0) against Mellanox OFED 2.4 >>and >> corresponding MXM. When it runs now, it gives the following >>warning, per >> process: >> >> [1443457390.911053] [myhist:5891 :0] mxm.c:185 MXM WARN >>The >> 'ulimit -s' on the system is set to 'unlimited'. This may have >>negative >> performance implications. Please set the heap size to the default >>value >> (10240) >> >> We have ulimits for heap (as well as most of the other limits) set >> unlimited because of applications that might possibly need a lot >>of RAM. >> >> The question is if we should do as MXM wants, or ignore it? Has >>anyone >> an >> experience running recent OpenMPI with MXM enabled, and what kind >>of >> ulimits do you have? Any suggestions/comments appreciated, thanks! >> >> -- >> Grigory Shamov >> >> Westgrid/ComputeCanada Site Lead >> University of Manitoba >> E2-588 EITC Building, >> (204) 474-9625 >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >> Link to this post: >> http://www.open-mpi.org/community/lists/users/2015/09/27697.php >> >> -- >> Kind Regards, >> M. > >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >> Link to this post: >>http://www.open-mpi.org/community/lists/users/2015/09/27698.php >
