Date: Sun, 7 May 2006 17:41:53 -0400 From: Kris Kennaway <kris@obsecurity.org> To: Kris Kennaway <kris@obsecurity.org> Cc: performance@FreeBSD.org, Robert Watson <rwatson@FreeBSD.org>, current@FreeBSD.org Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) Message-ID: <20060507214153.GA5275@xor.obsecurity.org> In-Reply-To: <20060507210426.GA4422@xor.obsecurity.org> References: <20060506150622.C17611@fledge.watson.org> <20060506221908.GB51268@xor.obsecurity.org> <20060507210426.GA4422@xor.obsecurity.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--C7zPtVaVf+AK4Oqc
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On Sun, May 07, 2006 at 05:04:26PM -0400, Kris Kennaway wrote:
> > 477 23472709 2810986 8 5671248 1900047 kern/ke=
rn_synch.c:220 (process lock)
> >=20
> > The top 10 heavily contended mutexes are very different (but note the
> > number of mutex acquisitions, column 3, is about the same).
> >=20
> > There is not much contention on unp_global_mtx any longer, but there
> > is a lot more on some of the other mutexes, especially the process
> > lock via msleep(). Off-hand I don't know what is the cause of this
> > bottleneck (note: libthr is used as threading library and libpthread
> > is not ported to sparc64).
> i.e. the same number of msleep calls in both cases, but the patch
> shuffles them from sbwait onto everything else.
Oops, they're the same number because that was the size of my ktr
buffer, duh :) Nevertheless it's a sample from the end of the same
workload, so the distribution should be comparable.
> sigwait +27%
This appears to be the only relevant msleep wait channel that is using
the process lock.
static int
kern_sigtimedwait(struct thread *td, sigset_t waitset, ksiginfo_t *ksi,
struct timespec *timeout)
{
=2E..
td->td_sigmask =3D savedmask;
SIGSETNAND(td->td_sigmask, waitset);
signotify(td);
error =3D msleep(&ps, &p->p_mtx, PPAUSE|PCATCH, "sigwait", hz);
i.e. several threads in the process are all sleeping for 1 tick and
then doing a thundering herd on the same proc lock when they wake up.
To confirm this I added a KTR tracepoint before the msleep call. It's
indeed being called from the various threads in mysqld. It's hard to
say for sure how many CPUs are contending at once, but plotting a
histogram of the time distribution of the calls indicates that several
CPUs are frequently executing this code in close proximity.
Kris
--C7zPtVaVf+AK4Oqc
Content-Type: application/pgp-signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (FreeBSD)
iD8DBQFEXmmhWry0BWjoQKURAl4gAKCHsE1VGXgIJP7jm0HRUQ5p0VpS5ACcDzwX
FRuZa8snPICD00VfJdFPym4=
=0uuO
-----END PGP SIGNATURE-----
--C7zPtVaVf+AK4Oqc--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20060507214153.GA5275>