From owner-freebsd-performance@FreeBSD.ORG Wed May 4 13:06:58 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5256E16A4D1 for ; Wed, 4 May 2005 13:06:58 +0000 (GMT) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [64.7.153.18]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4604F43D64 for ; Wed, 4 May 2005 13:06:57 +0000 (GMT) (envelope-from mike@sentex.net) Received: from pumice5.sentex.ca (pumice5.sentex.ca [64.7.153.20]) by smarthost1.sentex.ca (8.13.3/8.13.3) with ESMTP id j44BvP8j067432 for ; Wed, 4 May 2005 07:57:25 -0400 (EDT) (envelope-from mike@sentex.net) Received: from lava.sentex.ca (pyroxene.sentex.ca [199.212.134.18]) by pumice5.sentex.ca (8.13.3/8.13.1) with ESMTP id j44BvTVa009478; Wed, 4 May 2005 07:57:29 -0400 (EDT) (envelope-from mike@sentex.net) Received: from simian.sentex.net (simeon.sentex.ca [192.168.43.27]) by lava.sentex.ca (8.13.3/8.12.11) with ESMTP id j44BvFlT001611; Wed, 4 May 2005 07:57:16 -0400 (EDT) (envelope-from mike@sentex.net) Message-Id: <6.2.1.2.0.20050502222250.02da34f8@64.7.153.2> X-Mailer: QUALCOMM Windows Eudora Version 6.2.1.2 Date: Wed, 04 May 2005 07:57:15 -0400 To: "Poul-Henning Kamp" From: Mike Tancsa In-Reply-To: <20925.1115069492@critter.freebsd.dk> References: <20925.1115069492@critter.freebsd.dk> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Virus-Scanned: by amavisd-new X-Scanned-By: MIMEDefang 2.51 on 64.7.153.20 cc: freebsd-performance@freebsd.org Subject: Re: Very low disk performance on 5.x (some tweaking and more results) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 May 2005 13:06:58 -0000 At 05:31 PM 02/05/2005, Poul-Henning Kamp wrote: >In message <6.2.1.2.0.20050502163426.02c4e678@64.7.153.2>, Mike Tancsa writes: > > >Using an amrc controller in RAID5, it doesnt make a difference really on > >the dd stuff - read or write-- perhaps 2-5MB difference on the faster > >side. Raw reads and writes whether I use da0 or da0s1d, the speed is > >pretty well the same as before... HOWEVER, one measurement is really odd on > >bonnie-- the random seeks .... and postmark shows a *big* difference as > well. > >This is very likely the alignment thing. OK, some further tests, trying to control for this. I am not sure what values to actually fiddle with via bsdlable as this is the entire disk so I will just vary mounting da0 vs da0s1d. I also cvsup'd to current as of yesterday (FreeBSD nfs.sentex.ca 6.0-CURRENT FreeBSD 6.0-CURRENT #0: Mon May 2 15:03:53 EDT 2005). I will also try with Scott's changes to the driver which went in after. Also, the newfs params does seem to make a very big difference and vfs.read_max as well. [nfs]# fdisk da0 ******* Working on device /dev/da0 ******* parameters extracted from in-core disklabel are: cylinders=91182 heads=255 sectors/track=63 (16065 blks/cyl) Figures below won't work with BIOS for partitions not in cyl 1 parameters to be used for BIOS calculations are: cylinders=91182 heads=255 sectors/track=63 (16065 blks/cyl) Media sector size is 512 Warning: BIOS sector numbering starts with sector 1 Information from DOS bootblock is: The data for partition 1 is: sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD) start 63, size 1464838767 (715253 Meg), flag 80 (active) beg: cyl 0/ head 1/ sector 1; end: cyl 1023/ head 254/ sector 63 The data for partition 2 is: The data for partition 3 is: The data for partition 4 is: [nfs]# bsdlabel da0s1 # /dev/da0s1: 8 partitions: # size offset fstype [fsize bsize bps/cpg] c: 1464838767 0 unused 0 0 # "raw" part, don't edit d: 1464838767 0 4.2BSD 2048 16384 28552 [nfs]# newfs -b 32768 -f 4096 /dev/da0 [nfs]# bsdlabel da0 bsdlabel: /dev/da0: no valid label found [nfs]# even though I can mount it just fine [nfs]# mount /dev/da0 /mnt [nfs]# df Filesystem 1K-blocks Used Avail Capacity Mounted on /dev/ad0s1a 1012974 356028 575910 38% / devfs 1 1 0 100% /dev /dev/ad0s1e 1012974 580870 351068 62% /tmp /dev/ad0s1d 15231278 5848016 8164760 42% /usr /dev/ad0s1f 19564270 192176 17806954 1% /var /dev/da0 720910352 8 663237516 0% /mnt [nfs]# -------Sequential Output-------- ---Sequential Input-- --Random-- -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks--- Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU da0s1 4000 61175 53.3 61832 16.3 31829 8.7 63008 60.4 79551 11.2 224.2 0.7 da0s1 v 4000 61278 54.3 61938 16.4 31828 8.8 73566 71.7 81778 12.6 216.5 0.7 da0 4000 62625 54.5 62154 15.7 31907 8.3 63297 60.8 80139 11.0 226.2 0.7 da0 v 4000 62315 54.3 61759 15.4 32094 8.3 74209 72.3 84470 12.6 223.5 0.7 da0 N 4000 75239 62.7 75771 15.5 33596 8.4 81909 78.6 98453 13.3 214.8 0.8 da0 vN 4000 72786 61.7 74939 15.3 33672 8.4 82944 79.8 108106 14.9 214.7 0.8 v = sysctl -w vfs.read_max=16 N = newfs -b 32768 -f 4096 /dev/da0 [nfs]# postmark PostMark v1.5 : 3/27/01 pm>set location /mnt pm>set size 300 100000 pm>set transactions 400000 pm>run Creating files...Done Performing transactions..........Done Deleting files...Done Time: 1418 seconds total 1418 seconds of transactions (282 per second) Files: 200107 created (141 per second) Creation alone: 500 files (500 per second) Mixed with transactions: 199607 files (140 per second) 199905 read (140 per second) 199384 appended (140 per second) 200107 deleted (141 per second) Deletion alone: 889 files (889 per second) Mixed with transactions: 199218 files (140 per second) Data: 12715.55 megabytes read (8.97 megabytes per second) 12728.92 megabytes written (8.98 megabytes per second) pm>quit [nfs]# create a junk file [nfs]# time dd if=/dev/urandom of=/mnt/big-4g-rand bs=1024k count=4000 4000+0 records in 4000+0 records out 4194304000 bytes transferred in 124.694915 secs (33636528 bytes/sec) 0.022u 116.020s 2:04.70 93.0% 21+2910k 124+32000io 0pf+0w wait 2 min for disks to totally flush [nfs]# time dd if=/dev/zero of=/mnt/big-4g-zero bs=1024k count=4000 4000+0 records in 4000+0 records out 4194304000 bytes transferred in 66.463634 secs (63106751 bytes/sec) 0.000u 11.265s 1:06.50 16.9% 21+2926k 126+32000io 0pf+0w sysctl -w vfs.read_max=16 (clear out cache by reading via md5 4G of other files) and then wait a bit [nfs]# time dd if=/mnt/big-4g-rand of=/dev/null bs=1024k count=4000 4000+0 records in 4000+0 records out 4194304000 bytes transferred in 59.178470 secs (70875506 bytes/sec) 0.000u 7.677s 0:59.18 12.9% 20+2784k 32128+0io 0pf+0w [nfs]# postmark PostMark v1.5 : 3/27/01 pm>set locatime /mnt Eh? pm>set location /mnt pm>set size 300 100000 pm>set transactions 400000 pm>run Creating files...Done Performing transactions..........Done Deleting files...Done Time: 1369 seconds total 1369 seconds of transactions (292 per second) Files: 200107 created (146 per second) Creation alone: 500 files (500 per second) Mixed with transactions: 199607 files (145 per second) 199905 read (146 per second) 199384 appended (145 per second) 200107 deleted (146 per second) Deletion alone: 889 files (889 per second) Mixed with transactions: 199218 files (145 per second) Data: 12715.55 megabytes read (9.29 megabytes per second) 12728.92 megabytes written (9.30 megabytes per second) pm> With vfs.read_max=8 [nfs]# postmark PostMark v1.5 : 3/27/01 pm>set location /mnt pm>set size 300 100000 pm>set transactions 400000 pm>run Creating files...Done Performing transactions..........Done Deleting files...Done Time: 1013 seconds total 1013 seconds of transactions (394 per second) Files: 200107 created (197 per second) Creation alone: 500 files (500 per second) Mixed with transactions: 199607 files (197 per second) 199905 read (197 per second) 199384 appended (196 per second) 200107 deleted (197 per second) Deletion alone: 889 files (889 per second) Mixed with transactions: 199218 files (196 per second) Data: 12715.55 megabytes read (12.55 megabytes per second) 12728.92 megabytes written (12.57 megabytes per second) pm> pm>quit [nfs]# --------------------------------------------------------------------- newfs -b 32768 -f 4096 /dev/da0 [nfs]# postmark PostMark v1.5 : 3/27/01 pm>set location /mnt pm>set transactions 400000 pm>set size 300 100000 pm>run Creating files...Done Performing transactions..........Done Deleting files...Done Time: 542 seconds total 541 seconds of transactions (739 per second) Files: 200107 created (369 per second) Creation alone: 500 files (500 per second) Mixed with transactions: 199607 files (368 per second) 199905 read (369 per second) 199384 appended (368 per second) 200107 deleted (369 per second) Deletion alone: 889 files (889 per second) Mixed with transactions: 199218 files (368 per second) Data: 12715.55 megabytes read (23.46 megabytes per second) 12728.92 megabytes written (23.49 megabytes per second) pm> [nfs]# sysctl -w vfs.read_max=16 vfs.read_max: 8 -> 16 pm>set location /mnt pm>set transactions 400000 pm>set size 300 100000 pm>run Creating files...Done Performing transactions..........Done Deleting files...Done Time: 542 seconds total 541 seconds of transactions (739 per second) Files: 200107 created (369 per second) Creation alone: 500 files (500 per second) Mixed with transactions: 199607 files (368 per second) 199905 read (369 per second) 199384 appended (368 per second) 200107 deleted (369 per second) Deletion alone: 889 files (889 per second) Mixed with transactions: 199218 files (368 per second) Data: 12715.55 megabytes read (23.46 megabytes per second) 12728.92 megabytes written (23.49 megabytes per second) pm> [nfs]# time dd if=/mnt/big-2g-rand count=2000 bs=1024k of=/dev/null 2000+0 records in 2000+0 records out 2097152000 bytes transferred in 19.298799 secs (108667489 bytes/sec) 0.000u 3.603s 0:19.32 18.6% 21+2965k 16019+0io 0pf+0w [nfs]# time dd if=/mnt/big-2g-rand2 count=2000 bs=1024k of=/dev/null 2000+0 records in 2000+0 records out 2097152000 bytes transferred in 20.188331 secs (103879414 bytes/sec) 0.015u 3.515s 0:20.24 17.3% 20+2865k 16020+0io 3pf+0w [nfs]# sysctl -w vfs.read_max=8 vfs.read_max: 16 -> 8 [nfs]# time dd if=/mnt/big-2g-rand count=2000 bs=1024k of=/dev/null 2000+0 records in 2000+0 records out 2097152000 bytes transferred in 21.421867 secs (97897722 bytes/sec) 0.007u 3.482s 0:21.43 16.2% 21+2873k 16018+0io 0pf+0w [nfs]# time dd if=/mnt/big-2g-rand2 count=2000 bs=1024k of=/dev/null 2000+0 records in 2000+0 records out 2097152000 bytes transferred in 23.306603 secs (89981024 bytes/sec) 0.000u 3.590s 0:23.37 15.3% 19+2687k 16020+0io 3pf+0w [nfs]#