From owner-freebsd-scsi@FreeBSD.ORG  Thu Nov 13 06:15:28 2003
Return-Path: <owner-freebsd-scsi@FreeBSD.ORG>
Delivered-To: freebsd-scsi@freebsd.org
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id 2796E16A4CE
	for <freebsd-scsi@freebsd.org>; Thu, 13 Nov 2003 06:15:28 -0800 (PST)
Received: from smtp.helinet.de (smtp.helinet.de [212.37.38.7])
	by mx1.FreeBSD.org (Postfix) with ESMTP id C94C243FCB
	for <freebsd-scsi@freebsd.org>; Thu, 13 Nov 2003 06:15:25 -0800 (PST)
	(envelope-from braukmann@tse-online.de)
Date: Thu, 13 Nov 2003 15:15:10 +0100
From: Andreas Braukmann <braukmann@tse-online.de>
To: Martin Blapp <mb@imp.ch>
Message-ID: <2147483647.1068736510@[192.168.111.140]>
In-Reply-To: <20031113142735.S13503@cvs.imp.ch>
References: <20031112172306.J4572@pooker.samsco.home>
 	<20031113084355.P13503@cvs.imp.ch>
 	<20031113132327.GF13029@canolog.ninthwonder.com>
 <20031113142735.S13503@cvs.imp.ch>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
X-Virus-Scanned: by amavisd-new-20030314-p2 (Debian) at helinet.de
cc: freebsd-scsi@freebsd.org
cc: Allen Briggs <briggs@wasabisystems.com>
Subject: Re: Very bad FreeBSD SCSI RAID5 write speed performance
X-BeenThere: freebsd-scsi@freebsd.org
X-Mailman-Version: 2.1.1
Precedence: list
List-Id: SCSI subsystem <freebsd-scsi.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-scsi>,
	<mailto:freebsd-scsi-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-scsi>
List-Post: <mailto:freebsd-scsi@freebsd.org>
List-Help: <mailto:freebsd-scsi-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-scsi>,
	<mailto:freebsd-scsi-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Thu, 13 Nov 2003 14:15:28 -0000

--On Donnerstag, 13. November 2003 14:29 Uhr +0100 Martin Blapp 
<mb@imp.ch> wrote:
>> On Thu, Nov 13, 2003 at 06:38:53AM -0500, Gary Stanley wrote:
>> > root@64:[/tmp/blah]>dd if=/dev/zero of=/tmp/blah/blah
>> > 89472+0 records in
>> > 89471+0 records out
>> > 45809152 bytes transferred in 8.546312 secs (5360108 bytes/sec)

ServerWorks Chipset, PIII 1 GHz, Adaptec 5400S, Raid-5 over four
spindles (Fujitsu U160, 10kUPM disks, write cache off) on one
channel, -current from end of september:

roach:[/home/ab] # dd if=/dev/zero of=./test bs=128k count=20000
20000+0 records in
20000+0 records out
2621440000 bytes transferred in 66.062844 secs (39681004 bytes/sec)

Another system: Tyan K8S, Adaptec 2200S, RAID-5 over five spindles
(Fujitsu U320, 10kUPM disks, write-cache off) across both channels,
-current from end of september (i386):

opti#  dd if=/dev/zero of=./test bs=128k count=20000
20000+0 records in
20000+0 records out
2621440000 bytes transferred in 57.648042 secs (45473184 bytes/sec)


> Exactly what we get here: ~ 6MB per second with RAID5 and the IPS
> driver. The Adaptec 3200s is similar.

I'll update the Opteron box to a recent kernel and will check
again.


> A single disk has writes to the 53MB to the raw slice. I don't
> know why 5 equal disks on 2 busses are that slow.

What about the write cache settings on the disks and on the
raid-controllers? Both of my controllers are equipped with
battery backup modules for the controller cache and are set
to "write behind caching".
Turning off the controller's write cache leads to raid-5
write performance less than 10 MByte/s.

-Andreas