In the message dated: Tue, 28 Aug 2012 02:41:51 -0000,
The pithy ruminations from "Edward Ned Harvey (lopser)" on
<Re: [lopsa-tech] linux and swapping> were:
=> > From: tech-boun...@lists.lopsa.org [mailto:tech-boun...@lists.lopsa.org]
=> > On Behalf Of Andrew Hume
=> >
=> > err, that is the whole point.
=> > SwapTotal - SwapFree reflects the high water mark problem.
=> > for example, if we are swapping, and process P1 has 20GB of swap,
=> > then SwapTotal - SwapFree = 20GB. if i start a new process P2 which uses
=>
=> OMG, holy crap. I don't care how big your processes are, or how much memory
you have in your sy
=> stem, you should never be swapping active memory. You can always solve this
problem by either a
=> dding more memory, or using a memory mapped file in your process.
I do so love people who proclaim absolutes, like:
You can always add more memory
Sure, you can always add more memory...Unless, of course, you can't...in
cases like: the chassis is maxed out, or you don't have the funds, or
the process that's swapping is an instantaneous issue and you cannot add
more RAM on the fly. In addition, sometimes it is not always sensible
from a business perspective to add more memory. For example, if the
process that's swapping is infrequent and well-understood (think of
something like quarterly inventory processing that is brief, and which
can be scheduled so that the high memory use doesn't have a significant
impact on other operations) and doesn't warrent adding more RAM that
would be unnecessary 99% of the time.
Sure, you can always use a memory mapped file...unless, of course,
you can't because...you're not developing the application, or because
the file operations occur in a 3rd-party component, or because the
development overhead to use a memory mapped file exceeds the cost of
having a process swap.
At $WORK I manage systems for a research group that writes software. Software
is a means to an end--our goal is algorithm development for medical imaging.
We're not in industry, but research, so results are more important than
efficiency (though speed of results is a factor, and clearly a process that
uses swap space will be slow).
While our machines are fairly beefy (typically 64GB of RAM), we
sometimes have processes that use more--maybe just 65GB, or maybe someone
is trying an existing analysis with a higher resolution dataset so that
the process that was previously well-behaved now needs 80GB, or maybe
someone is trying to find optimal parameters, and one job of 50,000
jobs needs enough RAM that it uses swap. Under these circumstances,
the priority is to let the processes complete, even if they use swap
and are 100x slower. Sure, I provide feedback to the reseachers--and
they may make coding changes or parameter choices that improve memory use.
However, their goal is to do scientific research, not necessarily to write
the most memory-efficient code. If a process uses swap rarely (usually due
to parameter selection or dataset type), then it's not worth the overhead
to restructure software to improve these edge cases, at the expense of
development time that could be put toward algorithm design. In addition,
we rely heavily on outside toolkits for most file operations. While these
are open source, and while we do contribute back to these projects, it
is completely out of our scope, and counter-productive to our scientific
mission, to put significant effort into re-writing basic I/O operations
to use memory-mapped files, even if that's the 'right way' to do things.
I strongly agree with your outlook that "you should never be swapping active
memory", but there are too many cases in the real world where it does happen
to make absolute statements about solutions.
Mark
_______________________________________________
Tech mailing list
Tech@lists.lopsa.org
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
