on 21/09/2010 19:16 Alan Cox said the following: > Actually, I think that there is a middle ground between "per-cpu caches" and > "directly from the VM" that we are missing. When I've looked at the default > configuration of ZFS (without the extra UMA zones enabled), there is an > incredible amount of churn on the kmem map caused by the implementation of > uma_large_malloc() and uma_large_free() going directly to the kmem map. Not > only are the obvious things happening, like allocating and freeing kernel > virtual addresses and underlying physical pages on every call, but also > system-wide TLB shootdowns and sometimes superpage demotions are occurring. > > I have some trouble believing that the large allocations being performed by > ZFS > really need per-CPU caching, but I can certainly believe that they could > benefit > from not going directly to the kmem map on every uma_large_malloc() and > uma_large_free(). In other words, I think it would make a lot of sense to > have > a thin layer between UMA and the kmem map that caches allocated but unused > ranges of pages.
Alan, thank you very much for the testing and analysis. These are very good points. So, for the reference, here are two patches that I came up with: 1. original patch that attempts to implement Solaris-like behavior but doesn't go all the way to disabling per-CPU caches: http://people.freebsd.org/~avg/uma-1.diff 2. patch that attempts to implement Jeff's three suggestions; I've tested per-CPU cache size adaptive behavior, works well, but haven't tested per-CPU cache draining yet: http://people.freebsd.org/~avg/uma-2.diff -- Andriy Gapon _______________________________________________ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
