Date: Sun, 11 Mar 2007 08:11:03 +0100 (BST) From: Robert Watson <rwatson@FreeBSD.org> To: Keith Arner <vornum@gmail.com> Cc: Dave Baukus <david.baukus@us.fujitsu.com>, net@freebsd.org Subject: Re: netisr_direct Message-ID: <20070311073249.R20646@fledge.watson.org> In-Reply-To: <8e552a500703102157p1845926au65bb3adaf81c01c0@mail.gmail.com> References: <45C0CA5D.5090903@incunabulum.net> <45E6BEE0.2050307@FreeBSD.org> <45E6C22D.7060200@freebsd.org> <45E6D70C.10104@FreeBSD.org> <45EEB086.3050409@FreeBSD.org> <45F03269.7050705@FreeBSD.org> <45F08F1D.5080708@us.fujitsu.com> <20070310035135.B30274@fledge.watson.org> <8e552a500703102157p1845926au65bb3adaf81c01c0@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sun, 11 Mar 2007, Keith Arner wrote:
> On 3/9/07, Robert Watson <rwatson@freebsd.org> wrote:
>
>> It also introduces parallelism in the in-bound network layer processing
>> path by allowing processing to occur in more than one thread at a time.
>> However, you can see
>
> From the experimentation I've done, it seems that for TCP loads at least,
> the input path tends to bottleneck on INP_INFO_WLOCK(&tcbinfo), essentially
> reducing the input path to a single thread.
Yes -- right now the in-bound TCP path is essentially serialized because of
the tcbinfo lock. The reason for this is that the tcbinfo lock doesn't just
protect the inpcb chains during lookup, but also effectively acts as a
reference to prevent the inpcb from being freed during input processing.
There are several ways we could start to reduce contention on that lock:
(1) Introduce true references on inpcbs, which would prevent the inpcb from
being freed even though the tcbinfo lock has been released. This would
require reworking the TCP inpcb close/free model a bit, among other
things.
(2) Move from using a mutex supporting only exclusive locking on tcbinfo to a
lock class supporting shared locks, and use shared locks in situations
where exclusion isn't required (i.e., in cases where the connection may
close, requiring the global lists to be modified).
(3) Move towards greater granularity of locking for the tcbinfo: instead of
a single mutex, move to more than one locks, so that different connections
processed simultaneously are likely to involve different locks. For
listen sockets, we would have to have a special case, such as a single
lock to serialize simultaneous lock acquisitions of multiple chain locks
(for example).
I think all of these are reasonable approaches. Mohan and I have had plans to
do (1) for at least a year or two now, but have not gotten to it yet. I
experimented with (2), with the help of Attilio, a few months ago and ran into
several issues (especially relating to recursion). I've not yet explored (3);
for web servers with many connections, it is likely quite a good approach, but
since it doesn't preclude "unlucky" hashing assignments of connections to
chains that could lead to high contention. It might be quite easy to
implement, however, which would be a big win :-).
>> reduced parallelism in some environments as the ithread no longer run
>> concurrently with the netisr. I would not use net.isr.direct in versions
>> of FreeBSD before 6.1, but it should be pretty safe for 6.1 and later.
>
> Admitedly, the experimentation I've done has been with 6.0, rather than 6.1.
Looking back from the CVS log, I see that the issue I was concerned about was
fixed in 6.x before 6.0, rather than after, which was my fear. I'm not aware
of specific bugs that might cause problems with direct dispatch in earlier 6.x
releases, off hand.
> Have there been any changes in 6.1 that address the locking in the TCP input
> path? I'm very interested in being able to run highly parallel TCP loads.
I'd have to look in detail at the logs to answer that -- I'm not aware of any
significant changes. There are probably some minor ones to reduce contention
in various ways, but I think the more important changes are all in 7.x only
still. In 7.x, the pcbinfo lock is held significantly less in socket-side
paths such as send/receive, which should improve contention issues.
Robert N M Watson
Computer Laboratory
University of Cambridge
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20070311073249.R20646>