From owner-freebsd-current@FreeBSD.ORG Sun May 7 21:41:55 2006 Return-Path: X-Original-To: current@FreeBSD.org Delivered-To: freebsd-current@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A147E16A404; Sun, 7 May 2006 21:41:55 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.FreeBSD.org (Postfix) with ESMTP id 59BDA43D45; Sun, 7 May 2006 21:41:55 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id 442C31A3C19; Sun, 7 May 2006 14:41:55 -0700 (PDT) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id 3814C515DC; Sun, 7 May 2006 17:41:54 -0400 (EDT) Date: Sun, 7 May 2006 17:41:53 -0400 From: Kris Kennaway To: Kris Kennaway Message-ID: <20060507214153.GA5275@xor.obsecurity.org> References: <20060506150622.C17611@fledge.watson.org> <20060506221908.GB51268@xor.obsecurity.org> <20060507210426.GA4422@xor.obsecurity.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="C7zPtVaVf+AK4Oqc" Content-Disposition: inline In-Reply-To: <20060507210426.GA4422@xor.obsecurity.org> User-Agent: Mutt/1.4.2.1i Cc: performance@FreeBSD.org, Robert Watson , current@FreeBSD.org Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 May 2006 21:41:55 -0000 --C7zPtVaVf+AK4Oqc Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, May 07, 2006 at 05:04:26PM -0400, Kris Kennaway wrote: > > 477 23472709 2810986 8 5671248 1900047 kern/ke= rn_synch.c:220 (process lock) > >=20 > > The top 10 heavily contended mutexes are very different (but note the > > number of mutex acquisitions, column 3, is about the same). > >=20 > > There is not much contention on unp_global_mtx any longer, but there > > is a lot more on some of the other mutexes, especially the process > > lock via msleep(). Off-hand I don't know what is the cause of this > > bottleneck (note: libthr is used as threading library and libpthread > > is not ported to sparc64). > i.e. the same number of msleep calls in both cases, but the patch > shuffles them from sbwait onto everything else. Oops, they're the same number because that was the size of my ktr buffer, duh :) Nevertheless it's a sample from the end of the same workload, so the distribution should be comparable. > sigwait +27% This appears to be the only relevant msleep wait channel that is using the process lock. static int kern_sigtimedwait(struct thread *td, sigset_t waitset, ksiginfo_t *ksi, struct timespec *timeout) { =2E.. td->td_sigmask =3D savedmask; SIGSETNAND(td->td_sigmask, waitset); signotify(td); error =3D msleep(&ps, &p->p_mtx, PPAUSE|PCATCH, "sigwait", hz); i.e. several threads in the process are all sleeping for 1 tick and then doing a thundering herd on the same proc lock when they wake up. To confirm this I added a KTR tracepoint before the msleep call. It's indeed being called from the various threads in mysqld. It's hard to say for sure how many CPUs are contending at once, but plotting a histogram of the time distribution of the calls indicates that several CPUs are frequently executing this code in close proximity. Kris --C7zPtVaVf+AK4Oqc Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (FreeBSD) iD8DBQFEXmmhWry0BWjoQKURAl4gAKCHsE1VGXgIJP7jm0HRUQ5p0VpS5ACcDzwX FRuZa8snPICD00VfJdFPym4= =0uuO -----END PGP SIGNATURE----- --C7zPtVaVf+AK4Oqc--