Greetings.... stuff inline...
> - SPARC, 64 GB memory > - UFS, PxFS file systems > > Our application is writing some logs to disk (4 GB / hour), flushing some > mmapped files from time to time (4 GB each 15 min), but is not doing much > disk I/O. > > Once our application is started and "warm", it doesn't allocate any further > memory. At this point, we have 3-4 GB of free memory (vmstat) and nothing > paged out to disk (swap -l). > Well, something is certainly consuming memory, because you indicate this is a 64GB system, and you show 3-4GB free. Who/what is consuming 60GB of RAM? > Since those memory requests are not coming from our application, I assume > that those 5 GB (3 GB less free memory plus 2 GB paged-out data) are used for > the file system cache. I always thought the fs cache would never grow any > more once memory gets short, so it should never cause paging activity (since > the cache list is part of the free list). Reading Solaris Internals, I just > learned that there's not only a cache list, but also a segmap cache. As I > understand this, the segmap cache may very well grow up to 12% of main memory > and may even cause application pages to be paged out, correct? So, this might > be what's happening here. Can I somehow monitor the segmap cache (since it is > kernel memory, is it reported as "Kernel" in ::memstat?)? > Thinks of UFS as having an L1 and L2 cache (like processors). segmap is the L1 cache, when segmap fills up, pages get pushed out the cache list (the "L2" cache), where they can be reclaimed back into segmap if they are referenced again via read/write before. The 12% of memory being consumed by segmap is not what's hurting you here (at least, I would be very surprised if it is). > My idea is now to set segmap_percent=1 to decrease the max size of the segmap > cache and this way avoid having pages paged out due to growing fs cache. In a > testrun with this configuration, my free memory doesn't fall below 3.5 GB any > more and nothing is being paged out -- saving me 4.5 GB of memory! > Does this machine really have 64GB of RAM (as indicated above)? > > Since we don't do much disk I/O, I would assume that we don't gain much from > the segmap cache anyway, so I would like to configure it to 1%. File system > pages will still be cached in the cache list as long as memory is available, > right? With the advantage, that the cache list is basically "free" memory and > would never cause other pages to be paged out. > Generally, yes. > I'm not sure, but as I understand it the segmap cache is still used during > read and write operations, right? So, every time we write a file, we always > write into the segmap cache. If this cache is small (let's say: 1% = 640 MB), > we might be slowed done when writing more than 640 MB all at once. However, > if we would only write 64 MB every minute, pages from the segmap cache would > migrate to the cache list and make room for more pages in the segmap cache, > so next time we write 64 MB, would there again be enough space in the segmap > cache for the write operation? > Generally, yes, assuming the writes are not to files with O_SYNC/O_DSYNC, in which case every write must go through the cache anyway. > Also, just to be sure: memory mapped files are never read or written through > the segmap cache, so shrinking that cache has no effect on memory mapped > files, right? > That is correct. mmap()'d files are not cached in segmap. Something is missing here, or the 64GB value is wrong. You need to figure out who/what is consuming 60GB of RAM. Use ' echo "::memstat" | mdb -k' for a high-order profile. Use "prstat -s rss" to look at processes physical memory use. Thanks, /jim > Are there any other side effects when decreasing the segmap cache to 1%? Or > is this the recommended way to decrease unnecessary memory usage for file > system caching? > > Thanks much for your help, > Nick. > > > This message posted from opensolaris.org > _______________________________________________ > perf-discuss mailing list > perf-discuss@opensolaris.org > _______________________________________________ perf-discuss mailing list perf-discuss@opensolaris.org
