From owner-freebsd-performance@FreeBSD.ORG Sun May 29 15:43:55 2005 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D3C3E16A41C for ; Sun, 29 May 2005 15:43:55 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [204.156.12.53]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9C66E43D49 for ; Sun, 29 May 2005 15:43:55 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by cyrus.watson.org (Postfix) with ESMTP id 3A3DE46B2A; Sun, 29 May 2005 11:43:55 -0400 (EDT) Date: Sun, 29 May 2005 16:44:14 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: "Matthew D. Fuller" In-Reply-To: <20050528221027.GB47448@over-yonder.net> Message-ID: <20050529164207.A52379@fledge.watson.org> References: <4297E2C4.1030505@roq.com> <20050528221027.GB47448@over-yonder.net> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Michael VInce , freebsd-performance@freebsd.org Subject: Re: High usage of mbufs X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 May 2005 15:43:55 -0000 On Sat, 28 May 2005, Matthew D. Fuller wrote: > On Sat, May 28, 2005 at 01:17:24PM +1000 I heard the voice of > Michael VInce, and lo! it spake thus: >> On one of my web servers I have a really high usage of mbuf clusters in >> use on a web server that does about 3million hits a day. >> 4294914731/262144 mbuf clusters in use (current/max) > > No it doesn't (have really high usage, that is). It just has messed up > statistics, which is standard on SMP machines; see the archives. I have two sets of patches that correct this problem: - One uses atomic operations to synchronize access to the mbuf-related counters instead of performing non-atomic updates, which results in races. - One uses per-cpu statistics in the mbuf allocator, and critical sections to synchronize updates. Both encounter a several percent performance hit in packet-intensive workloads, so I've been reluctant to commit either. However, most of the information required for mbuf statistics is already gathered by UMA, which is now used as the back-end for the mbuf allocator. As such, the better answer is to use the UMA statistics and then see if we still need the mbuf layer stats. I haven't had a chance to investigate this in detail yet, but I'd like to get something in before 6.0. Robert N M Watson