On Thu, Apr 14, 2016 at 04:24:45PM -0600, Warner Losh wrote: > On Thu, Apr 14, 2016 at 4:15 PM, Shawn Webb <shawn.w...@hardenedbsd.org> > wrote: > > > On Thu, Apr 14, 2016 at 04:04:27PM -0600, Warner Losh wrote: > > > On Thu, Apr 14, 2016 at 3:54 PM, Dmitry Morozovsky <ma...@rinet.ru> > > wrote: > > > > > > > Warner, > > > > > > > > On Thu, 14 Apr 2016, Warner Losh wrote: > > > > > > > > > Author: imp > > > > > Date: Thu Apr 14 21:47:58 2016 > > > > > New Revision: 298002 > > > > > URL: https://svnweb.freebsd.org/changeset/base/298002 > > > > > > > > > > Log: > > > > > New CAM I/O scheduler for FreeBSD. The default I/O scheduler is the > > > > same > > > > > > > > [snip] > > > > > > > > First, thanks so much for this quite a non-trivial work! > > > > What are the ways to enable this instead of deafult, and what ar the > > > > benefits > > > > and drawbacks? > > > > > > > > > You add CAM_NETFLIX_IOSCHED to your kernel config to enable it. Hmmm, > > > looking at the diff, perhaps I should add that to LINT. > > > > > > In production, we use it for three things. First, our scheduler keeps a > > lot > > > more > > > statistics than the default one. These statistics are useful for us > > knowing > > > when > > > a system is saturated and needs to shed load. Second, we favor reads over > > > writes because our workload, as you might imagine, is a read mostly work > > > load. > > > Finally, in some systems, we throttle the write throughput to the SSDs. > > The > > > SSDs > > > we buy can do 300MB/s write while serving 400MB/s read, but only for > > short > > > periods of time (long enough to do 10-20GB of traffic). After that, write > > > performance > > > drops, and read performance goes out the window. Experiments have shown > > that > > > if we limit the write speed to no more than 30MB/s or so, then the > > garbage > > > collection the drive is doing won't adversely affect the read latency / > > > performance. > > > > Going on a tangent here, but related: > > > > As someone who is just barely stepping into the world of benchmarks and > > performance metrics, can you shed some light as to how you gained those > > metrics? I'd be extremely interested to learn. > > > > These numbers were derived through an iterative process. All our systems > report > a large number of statistics while they are running. The disk performance > numbers > come from gstat(8) which ultimately derives them from devstat(9). When we > enabled > serving customer traffic while refreshing content, we noticed a large > number of > reports from our playback clients indicating problems with the server > during this > time period. I looked at the graphs to see what was going on. Once I found > the problem, > I was able to see that as the write load varied, the latency numbers for > the reads > would vary substantially as well. I added code to the I/O scheduler so I > could rate > limit the write speed to the SSDs. After running through a number of > different > machines over a number of nights of filling and serving, I was able to find > the right > number. If I set it to 30MB, the 20 machines I tested didn't have any > reports above > background level of problems. When I set it to 35MB/s there was a couple of > those > machines that had problems. when I set it to 40MB/s there were a couple > more. When > I set it to 80MB/s, almost all had problems. Being conservative, I set it > to the highest > number that showed no ill effect on the clients. I was able to see large > jumps in read > latency as low as 25MB/s though. Sadly, this is with Netflix internal > tools, but one > could do the same research with gstat and scripting. One could also use > dtrace > to study the latency patterns to a much finer degree of fidelity than gstat > offers.
Awesome. Thank you very much for the informative response. I didn't even know tools like gstat and devstat existed. I think Michael W. Lucas' new book covers some of the DTrace side of things. I'll be taking a look at that, too. "Learn something new everyday" certainly applies here, and thanks for the input. :) Thanks again, -- Shawn Webb HardenedBSD GPG Key ID: 0x6A84658F52456EEE GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89 3D9E 6A84 658F 5245 6EEE
signature.asc
Description: PGP signature
