On Fri, Jun 28, 2013 at 12:26:44AM +0300, Alexander Motin wrote: > Hi. > > While doing some profiles of GEOM/CAM IOPS scalability, on some test > patterns I've noticed serious congestion with spinning on global > pbuf_mtx mutex inside getpbuf() and relpbuf(). Since that code is > already very simple, I've tried to optimize probably the only thing > possible there: switch bswlist from TAILQ to SLIST. As I can see, > b_freelist field of struct buf is really used as TAILQ in some other > places, so I've just added another SLIST_ENTRY field. And result > appeared to be surprising -- I can no longer reproduce the issue at all. > May be it was just unlucky synchronization of specific test, but I've > seen in on two different systems and rechecked results with/without > patch three times. This is too unbelievable. Could it be, e.g. some cache line conflicts which cause the trashing, in fact ? Does it help if you add void *b_pad before b_freelist instead of adding b_freeslist ?
> > The present patch is here: > http://people.freebsd.org/~mav/buf_slist.patch > > The question is how to do it better? What is the KPI/KBI policy for > struct buf? I could replace b_freelist by a union and keep KBI, but > partially break KPI. Or I could add another field, probably breaking > KBI, but keeping KPI. Or I could do something handmade with no breakage. > Or this change is just a bad idea? The same question about using union for b_freelist/b_freeslist, does the effect of magically fixing the contention still there if b_freeslist is on the same offset as the b_freelist ? There are no K{B,P}I policy for struct buf in HEAD, just change it as it fits.
pgpGt3DQKL6vB.pgp
Description: PGP signature
