Date: Tue, 13 Oct 2009 14:14:15 +0100 (BST) From: Robert Watson <rwatson@FreeBSD.org> To: Ivan Voras <ivoras@freebsd.org> Cc: freebsd-stable@freebsd.org Subject: Re: Extreme console latency during disk IO (8.0-RC1, previous releases also affected according to others) Message-ID: <alpine.BSF.2.00.0910131406340.26071@fledge.watson.org> In-Reply-To: <hb1qs0$qjd$1@ger.gmane.org> References: <E316139E-FFCF-432F-8DCE-62B120C38E55@exscape.org> <CC16B639-7A75-4016-A8A8-5C59E9CD5E95@exscape.org> <hb1qs0$qjd$1@ger.gmane.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, 13 Oct 2009, Ivan Voras wrote: > Thomas Backman wrote: >> I'm copying this over from the freebsd-performance list, as I'm looking for >> a few more opinions - not on the problems *I* am having, but rather to >> check whether the problem is universal or not, and if not, find a possible >> common factor. In other words: I want to hear about your experiences, *good >> or bad*! >> >> Here's the original thread (not from the beginning, though): >> http://lists.freebsd.org/pipermail/freebsd-performance/2009-October/003843.html >> >> Long story short, my version: when the disk is stressed hard enough, >> console IO becomes COMPLETELY unbearable. 10+ seconds to switch between >> windows in screen(1), running (or even typing) simple commands, etc. This >> happens both via SSH and the serial console. > > Hmm, this looks familiar - I've noticed it before on the physical (VGA) > console and I notice it all the time under VMWare. It sort of looks like > disk IO really blocks network IO in this case - I use the VMs over ssh. Real hardware and virtual hardware have vastly different performance properties, so I'd be careful not to assume that the issue described by the original reporter and the issue you're experiencing are the same. In our kernel, low level network protocols will essentially always take precedence over disk I/O activity. So on face value "disk IO really blocks network IO" is highly unlikely. There are two much more likely possibilities: (1) poor VM implementation causes the virtual CPU to be suspended behind synchronous host OS I/O or (2) the network stack is running fine but the interactive user application is getting I/O or locks scheduled behind a bulk process. A useful diagnostic here is to compare the behavior of three kinds of network latency tests: (1) ping from the host OS to the guest OS (2) netperf TCP_RR from the host OS to the guest OS (3) ssh interactive latency If (1) is highly variable during I/O, it's almost certainly a property of the VM technology you're using, and there's nought to be done about it in the guest OS. If (2) but not (1) is highly variable, it may well be a scheduling issue, although under high memory pressure you couldn't rule out paging out of netserver pages/etc causing latency. If (3) but not (1) or (2) is highly variable, it's most likely an I/O scheduling issue, perhaps caused by priority inversion on lockmgr locks on a vnode, disk I/O scheduling leading to starvation, etc. Robert
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?alpine.BSF.2.00.0910131406340.26071>