Recently I was tasked to find a way to scale up our MYSQL server, running
MYSQL3.22.15 on FreeBSD3.3. I've been testing a hardware RAID solution,
and found that with 6 disks in a RAID5 configuration, the system was only
perhaps 30% faster than when running on a single disk. [The 6 disks in the
RAID5 are the same model as the single-disk test I was comparing against.]
Experimentation determined that pthreads was the problem. FreeBSD's
implementation of pthreads using a select() loop, and select() always says
that disk I/O is ready to proceed, and disk I/O never return EWOULDBLOCK.
Essentially, pthreads was serializing the MYSQL read() requests, and if the
dataset exceeds memory size, performance becomes entirely seek bound.
I've implemented a rough fix, which is to rfork() processes which I label
"iothreads" to handle the disk I/O. The parent process running pthreads
has a socketpair() to each of the iothreads. The iothreads wait for
requests on the socketpair, and since socketpairs can block, pthreads can
handle them efficiently. This essentially allows me to turn blocking disk
I/O calls into non-blocking calls. Having multiple pending seeks turns out
to be a huge win for MYSQL, allowing it to scale much better as disks are
added to the RAID5 array.
Unfortunately, I'm concerned about using this code in production, because
it needs a fair amount of cleanup to handle signals and administrative
functions correctly. For this reason and others, I'm starting a project to
move it into the pthreads library itself. Before I embark on that effort,
I have a couple questions:
1) Does this seem like a reasonable approach? [It _works_, and well. But
it tastes strongly of hack.]
2) Does anyone have suggestions for a solution that will be cleaner and
won't take man-months to implement? [Which is the redeeming quality of
what I've got - it took me two days to zero in on a very workable
solution.]
3) Is anyone working on something I might leverage off of in this area?
[For instance, a select()-based interface to async I/O? Or support for
blocking disk I/O in select() and read()?]
4) Is there anyone willing to commit to testing my modified pthreads
library against MYSQL? [I'll be stress testing it quite heavily, of
course. It would probably also be testable against Squid with async I/O
and multithreaded Apache 2.0.]
Thanks,
scott
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-hackers" in the body of the message
