On 9 March 2012 14:50, Stuart Barkley <[email protected]> wrote: > We are running into an awkward problem with one of our users jobs. > > We have h_vmem set as a consumable resource and have it set to the > physical memory (minus a small amount) on the systems. These are > diskless systems and have no swap defined. This works well for most > of our user jobs in preventing over allocation of physical memory. > > Our user job is using mmap(2) to map a large file into memory (it is > actually using the R bigmemory library). The entire file size gets > counted as virtual memory use even though only a small portion of the > file is actually accessed. This seems correct behavior since the > backing file is virtual memory. > > The problem is that in reality the job is only reads selected portions > of the file and using just a small amount of physical memory. > However, the jobs must request a large h_vmem in order to run. This > prevents other jobs from using the available memory on the node. > > A further complication is that this user is expecting this file to > eventually grow larger than physical memory and then the node h_vmem > will not be large enough for the job to even start. > > Has anyone had to deal with this issue before? > > Is there a better consumable to use instead of h_vmem? h_vmem is > seems to be the only resource which actually applies an enforced > memory limit to jobs. > > I have suggested that our use revisit the use of bigmemory for this > specific case and instead just directly read the necessary information > from the file. This should address both the immediate problem and the > problem as the file size grows. > > We do have other users who are also using mmap() for more practical > reasons so I expect that this problem may occur again for other users. > Never dealt with it as such but have noticed that a lot of our users codes use far less memory than virtual memory/address space even when there is little contention for memory which makes ?_vmem less than ideal. Disk space is relatively cheap. ?_rss seems close in definition to what you want but is unenforced. So the net of that is what you observed h_vmem constrains the wrong thing for most purposes and the more useful built in resources are unenforced.
We're currently running Scientific Linux 5 where the kernel doesn't support it but we're thinking of using the cgroups memory resource contoller to control memory allocation for the whole job. Grid engine doesn't have built in support for that AFAIK but the various forks are adding hwloc support in order to allocate cpus (which can use cgroups AIUI) and we were hoping to piggyback the memory control on top of that using one (or more)of the many fine hooks grid engine provides. The cgroups memory resource controller also has issues with Lustre which we use but we hope they'll be fixed by the time we upgrade. The above is entirely theoretical at this time. William > Thanks for any further suggestions, > Stuart Barkley > -- > I've never been lost; I was once bewildered for three days, but never lost! > -- Daniel Boone > _______________________________________________ > users mailing list > [email protected] > https://gridengine.org/mailman/listinfo/users > > _______________________________________________ users mailing list [email protected] https://gridengine.org/mailman/listinfo/users
