From owner-freebsd-current@FreeBSD.ORG Mon Apr 17 07:27:52 2006 Return-Path: X-Original-To: current@FreeBSD.org Delivered-To: freebsd-current@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0A8C216A401; Mon, 17 Apr 2006 07:27:52 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.FreeBSD.org (Postfix) with ESMTP id BD98A43D45; Mon, 17 Apr 2006 07:27:51 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id A3B231A4E34; Mon, 17 Apr 2006 00:27:51 -0700 (PDT) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id 86E8051664; Mon, 17 Apr 2006 03:27:50 -0400 (EDT) Date: Mon, 17 Apr 2006 03:27:50 -0400 From: Kris Kennaway To: Kris Kennaway Message-ID: <20060417072750.GA62515@xor.obsecurity.org> References: <20060417051440.GB60956@xor.obsecurity.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="DocE+STaALJfprDB" Content-Disposition: inline In-Reply-To: <20060417051440.GB60956@xor.obsecurity.org> User-Agent: Mutt/1.4.2.1i Cc: smp@FreeBSD.org, current@FreeBSD.org Subject: Re: Anomalous performance increase from mutex profiling X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Apr 2006 07:27:52 -0000 --DocE+STaALJfprDB Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Apr 17, 2006 at 01:14:40AM -0400, Kris Kennaway wrote: > * Our best guess is that mutex profiling is doing something that > reduces contention on this very heavily contended mutex (unp), but I'd > like to know what is happening precisely so I can maybe make use of > it. >=20 > Can anyone think of what may be happening that I've missed? I think it is just doing effectively the same thing as my exponential spin backoff patch, namely introducing delays with the effect of reducing common memory accesses. When I turn the maximum spin backoff limit *way* up (from 1600 to 51200) I get performance that slightly exceeds what I see from mutex profiling alone (adding mutex profiling again on top of this gives a small further increase, but only a few % and so probably achievable by further increasing the backoff limit). A limit of 51200 is not an appropriate default since it penalizes the common case of light to moderate contention. The point is that here basically all 12 CPUs are spinning on a single lock (kern/uipc_usrreq.c:308), so it's massively over-contended and all we can do is mitigate the effects of this. On this system, the maximum supersmack performance (3700 queries/sec) comes when there are only 6 clients, so (as jasone eloquently put it) with 10 clients the difference between 2300 queries/sec (with absurdly high backoff limits or mutex profiling) and 1450/sec (with reasonable backoff limits) is the difference between "slow" and "ass slow". Fortunately rwatson is working on breaking up this lock, so that should help a lot here :-) Kris --DocE+STaALJfprDB Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (FreeBSD) iD8DBQFEQ0N1Wry0BWjoQKURAt3WAJ9Q3kipclq/L5pCLHeQUeYX+Z1DVwCffq4X Z0quoa5vJNHbZcITo1rZiGU= =W46k -----END PGP SIGNATURE----- --DocE+STaALJfprDB--