Date: Wed, 21 Jun 2006 17:45:41 -0400 From: John Baldwin <jhb@freebsd.org> To: freebsd-current@freebsd.org Cc: Paul Allen <nospam@ugcs.caltech.edu>, Robert Watson <rwatson@freebsd.org>, current@freebsd.org Subject: Re: FILEDESC_LOCK() implementation Message-ID: <200606211745.42525.jhb@freebsd.org> In-Reply-To: <20060621214346.G8526@fledge.watson.org> References: <20060612054115.GA42379@xor.obsecurity.org> <20060621201927.GJ28128@groat.ugcs.caltech.edu> <20060621214346.G8526@fledge.watson.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wednesday 21 June 2006 16:46, Robert Watson wrote: > On Wed, 21 Jun 2006, Paul Allen wrote: > > > From Robert Watson <rwatson@freebsd.org>, Wed, Jun 21, 2006 at 07:46:33PM +0100: > >> I would optimize very carefully here, the trade-offs are tricky, and we > >> may find that by making locking more complex, we cause cache problems, > >> increase lock hold periods, etc, even if we decrease contention. I've > >> wondered a bit about a model where we loan fd's to threads to optimize > >> repeated access to the same fd by the same thread, but this mostly makes > >> sense in the context of a 1:1 model rather than an m:n model. > > I apologize for not understanding all of the uses of the FILEDESC lock > > but, isn't the more obvious partitioning per-cpu: each cpu may allocate > > from a range of fd, which cpu cache used depends on where the thread > > happens to be running. When closing a fd, it is returned to the local > > (possibly different cpu cache). A watermark is used to generate an IPI > > message to rebalance the caches as needed. > > The issue is actually a bit different than that. We in effect already do > the above using UMA. > > The problem is this: when you have threads in the same process, file > descriptor lookup is performed against a common file descriptor array. That > array is protected by a lock, the filedesc lock. When lots of threads > simultaneously perform file descriptor operations, they contend on the file > descriptor array lock. So if you have 30 threads all doing I/O, they are > constantly looking up file descriptors and bumping into each other. This is > particularly noticeable for network workloads, where many operations are > very fast, and so they occur in significant quantity. The M:N threading > library actually handles this quite well by bounding the number of threads > trying to acquire the lock to the number of processors, but with libthr you > get pretty bad performance. This contention problem also affects MySQL, > etc. > > You can imagine a number of ways to work on this, but it's a tricky problem > that has to be looked at carefully. Are the lookup operations using a shared lock so that only things like open and close would actually contend? -- John Baldwin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200606211745.42525.jhb>