On Tue, 13 Oct 2009, Ivan Voras wrote:
Thomas Backman wrote:
I'm copying this over from the freebsd-performance list, as I'm looking for
a few more opinions - not on the problems *I* am having, but rather to
check whether the problem is universal or not, and if not, find a possible
common factor. In other words: I want to hear about your experiences, *good
or bad*!
Here's the original thread (not from the beginning, though):
http://lists.freebsd.org/pipermail/freebsd-performance/2009-October/003843.html
Long story short, my version: when the disk is stressed hard enough,
console IO becomes COMPLETELY unbearable. 10+ seconds to switch between
windows in screen(1), running (or even typing) simple commands, etc. This
happens both via SSH and the serial console.
Hmm, this looks familiar - I've noticed it before on the physical (VGA)
console and I notice it all the time under VMWare. It sort of looks like
disk IO really blocks network IO in this case - I use the VMs over ssh.
Real hardware and virtual hardware have vastly different performance
properties, so I'd be careful not to assume that the issue described by the
original reporter and the issue you're experiencing are the same. In our
kernel, low level network protocols will essentially always take precedence
over disk I/O activity. So on face value "disk IO really blocks network IO"
is highly unlikely.
There are two much more likely possibilities: (1) poor VM implementation
causes the virtual CPU to be suspended behind synchronous host OS I/O or (2)
the network stack is running fine but the interactive user application is
getting I/O or locks scheduled behind a bulk process.
A useful diagnostic here is to compare the behavior of three kinds of network
latency tests:
(1) ping from the host OS to the guest OS
(2) netperf TCP_RR from the host OS to the guest OS
(3) ssh interactive latency
If (1) is highly variable during I/O, it's almost certainly a property of the
VM technology you're using, and there's nought to be done about it in the
guest OS.
If (2) but not (1) is highly variable, it may well be a scheduling issue,
although under high memory pressure you couldn't rule out paging out of
netserver pages/etc causing latency.
If (3) but not (1) or (2) is highly variable, it's most likely an I/O
scheduling issue, perhaps caused by priority inversion on lockmgr locks on a
vnode, disk I/O scheduling leading to starvation, etc.
Robert
_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
