Date: Wed, 12 Oct 2005 17:17:12 -0400 (EDT) From: Andrew Gallatin <gallatin@cs.duke.edu> To: Robert Watson <rwatson@FreeBSD.org> Cc: net@FreeBSD.org Subject: Re: Call for performance evaluation: net.isr.direct (fwd) Message-ID: <17229.32088.696346.868182@grasshopper.cs.duke.edu> In-Reply-To: <20051012212915.E66014@fledge.watson.org> References: <20051008143854.B84936@fledge.watson.org> <17229.29164.891534.200216@grasshopper.cs.duke.edu> <20051012212915.E66014@fledge.watson.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Robert Watson writes:
>
> On Wed, 12 Oct 2005, Andrew Gallatin wrote:
>
> > Speaking of net.isr, is there any reason why if_simloop() calls
> > netisr_queue() rather than netisr_dispatch()?
>
> Yes -- it's basically to prevent recursion for loopback traffic, which can
> result in both lock orders and general concerns regarding reentrance. To
> be specific: if you send a packet on a loopback TCP socket, it gets
> processes asynchronously in the netisr rather than immediately walking
> back into the TCP code again. Right now WITNESS would warn about this,
> but there were also quite bad things that could happen before we did the
> locking work -- for example, when connections are torn down. It also
> avoids Really Deep Stacks.
Right now, at least, it seems to work OK. I haven't tried witness,
but a non-debug kernel shows a big speedup from enabling it. Do
you think there is a chance that it could be made to work in FreeBSD?
For what its worth, our TCP RR loopback latency on a dual-core AMD64
3800+ is roughly 2x as much as the solaris and linux latency. Heck,
we're even slower than Darwin. And I didn't think it was *possible*
to be slower than Darwin:
---------------------------------------------------------------------
Host OS 2p/0K Pipe AF UDP RPC/ TCP RPC/ TCP
ctxsw UNIX UDP TCP conn
--------- ------------- ----- ----- ---- ----- ----- ----- ----- ----
rome FreeBSD 7.0-C 7.220 15.7 18.1 36.1
rome.sw.m Darwin 8.2.0 1.250 6.531 19.9 33.7 48.4 33.0 48.0 63.
rome SunOS 5.11 5.140 12.8 15.4 22.0 17.0 43.1 141.
rome Linux 2.6.9-1 3.760 9.780 11.7 17.0 20.5 18.9 23.2 32.
Using net.isr.direct over the loopback brings the latency down
to 25us.
vmstat 1 with direct disabled:
procs memory page disk faults cpu
r b w avm fre flt re pi po fr sr ad0 in sy cs us sy id
2 0 0 33216 457016 0 0 0 0 0 0 4 53593 107075 214030 5 68 27
2 0 0 33216 457016 0 0 0 0 0 0 0 53694 107279 214356 5 72 24
<...>
And with it enabled:
2 0 0 33796 456620 0 0 0 0 0 0 0 106 153987 150570 2 59 39
2 0 0 33796 456620 0 0 0 0 0 0 0 107 154100 150945 3 61 36
Drew
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?17229.32088.696346.868182>