On 17.09.2010 10:14, Andriy Gapon wrote:
I've been investigating interaction between zfs and uma for a while. You might remember that there is a noticeable fragmentation in zfs uma zones when uma use is not enabled for actual data/metadata buffers. I also noticed that when uma use is enabled for data/metadata buffers (zio.use_uma=1) amount of memory reserved in free items of zfs uma zones becomes really huge. And this is despite the fact that the vast majority of the data/metadata zone have items with sizes that are multiples of page size. This couldn't really be because of fragmentation. Further checks show that the free items are accumulated in per-cpu cache buckets. uz_count for those buckets starts with 1, but over time, during bursts of activity, it grows up to maximum of 128. Problem with those buckets is that they are not drained on low memory conditions and uz_count never goes down. So, after a while, I observe about 300 free items (on a mere two core system) cached in 4 per-cpu buckets for a single zone with 128KB item size. That's 30MB right there. For all data and metadata zones the number goes as high as 500MB on my machine with 4GB physical RAM. This seems like a bit too much to me. Although keeping free items around improves performance, it does consume memory too. And the fact that that memory is not freed on lowmem condition makes the situation worse.
Interesting. We may run into related issues with excessive mbuf (cluster) caching in the per-cpu buckets as well. Having a general solutions for that is appreciated. Maybe the size of the free per-cpu buckets should be specified when setting up the UMA zone. Of certain frequently re-used elements we may want to cache more, other less. -- Andre _______________________________________________ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
