From owner-freebsd-stable@FreeBSD.ORG  Tue Oct 16 00:42:48 2007
Return-Path: <owner-freebsd-stable@FreeBSD.ORG>
Delivered-To: freebsd-stable@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id DFAEE16A41B;
	Tue, 16 Oct 2007 00:42:48 +0000 (UTC)
	(envelope-from kris@FreeBSD.org)
Received: from weak.local (hub.freebsd.org [IPv6:2001:4f8:fff6::36])
	by mx1.freebsd.org (Postfix) with ESMTP id 4E91C13C458;
	Tue, 16 Oct 2007 00:42:47 +0000 (UTC)
	(envelope-from kris@FreeBSD.org)
Message-ID: <47140906.2020107@FreeBSD.org>
Date: Tue, 16 Oct 2007 02:42:46 +0200
From: Kris Kennaway <kris@FreeBSD.org>
User-Agent: Thunderbird 2.0.0.6 (Macintosh/20070728)
MIME-Version: 1.0
To: Alexey Popov <lol@chistydom.ru>
References: <47137D36.1020305@chistydom.ru>
In-Reply-To: <47137D36.1020305@chistydom.ru>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Cc: freebsd-hackers@freebsd.org, freebsd-stable@freebsd.org
Subject: Re: amrd disk performance drop after running under high load
X-BeenThere: freebsd-stable@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-stable>, 
	<mailto:freebsd-stable-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-stable>
List-Post: <mailto:freebsd-stable@freebsd.org>
List-Help: <mailto:freebsd-stable-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-stable>,
	<mailto:freebsd-stable-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 16 Oct 2007 00:42:49 -0000

Alexey Popov wrote:

> After some time of running under high load disk performance become 
> expremely poor. At that periods 'systat -vm 1' shows something like  this:

What does "high load" mean?  You need to explain the system workload more.

> Disks amrd0
> KB/t  85.39
> tps       5
> MB/s   0.38
> % busy   99

> Apart of all, I tried to make mutex profiling and here's the results 
> (sorted by the total number of acquisitions):
> 
> Bad case:
> 
>  102 223514 273977 0 14689 1651568 /usr/src/sys/vm/uma_core.c:2349 (512)
>  950 263099 273968 0 15004 14427 /usr/src/sys/vm/uma_core.c:2450 (512)
>  108 150422 175840 0 10978 22988519 /usr/src/sys/vm/uma_core.c:1888 (mbuf)

 > Here you can see that high UMA activity happens in periods of low disk
 > performance. But I'm not sure whether this is a root of the problem, not
 > a consequence.

The extremely high contention there does seem to say you have a mbuf 
starvation problem and not a disk problem.  I don't know why this would 
be happening off-hand.

Can you also provide more details about the system hardware and 
configuration?

Kris