Skip site navigation (1)Skip section navigation (2)
Date:      Wed, 04 May 2005 07:57:15 -0400
From:      Mike Tancsa <mike@sentex.net>
To:        "Poul-Henning Kamp" <phk@phk.freebsd.dk>
Cc:        freebsd-performance@freebsd.org
Subject:   Re: Very low disk performance on 5.x (some tweaking and more results)
Message-ID:  <6.2.1.2.0.20050502222250.02da34f8@64.7.153.2>
In-Reply-To: <20925.1115069492@critter.freebsd.dk>
References:  <Your message of "Mon, 02 May 2005 17:19:13 EDT." <6.2.1.2.0.20050502163426.02c4e678@64.7.153.2> <20925.1115069492@critter.freebsd.dk>

next in thread | previous in thread | raw e-mail | index | archive | help
At 05:31 PM 02/05/2005, Poul-Henning Kamp wrote:
>In message <6.2.1.2.0.20050502163426.02c4e678@64.7.153.2>, Mike Tancsa writes:
>
> >Using an amrc controller in RAID5, it doesnt make a difference really on
> >the dd stuff - read or write-- perhaps 2-5MB difference on the faster
> >side.  Raw reads and writes whether I use da0 or da0s1d, the speed is
> >pretty well the same as before... HOWEVER, one measurement is really odd on
> >bonnie-- the random seeks .... and postmark shows a *big* difference as 
> well.
>
>This is very likely the alignment thing.

OK, some further tests, trying to control for this.  I am not sure what 
values to actually fiddle with via bsdlable as this is the entire disk so I 
will just vary mounting da0 vs da0s1d.   I also cvsup'd to current as of 
yesterday (FreeBSD nfs.sentex.ca 6.0-CURRENT FreeBSD 6.0-CURRENT #0: Mon 
May  2 15:03:53 EDT 2005).  I will also try with Scott's changes to the 
driver which went in after.

Also, the newfs params does seem to make a very big difference and 
vfs.read_max as well.


[nfs]# fdisk da0
******* Working on device /dev/da0 *******
parameters extracted from in-core disklabel are:
cylinders=91182 heads=255 sectors/track=63 (16065 blks/cyl)

Figures below won't work with BIOS for partitions not in cyl 1
parameters to be used for BIOS calculations are:
cylinders=91182 heads=255 sectors/track=63 (16065 blks/cyl)

Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
     start 63, size 1464838767 (715253 Meg), flag 80 (active)
         beg: cyl 0/ head 1/ sector 1;
         end: cyl 1023/ head 254/ sector 63
The data for partition 2 is:
<UNUSED>
The data for partition 3 is:
<UNUSED>
The data for partition 4 is:
<UNUSED>
[nfs]# bsdlabel da0s1
# /dev/da0s1:
8 partitions:
#        size   offset    fstype   [fsize bsize bps/cpg]
   c: 1464838767        0    unused        0     0         # "raw" part, 
don't edit
   d: 1464838767        0    4.2BSD     2048 16384 28552
[nfs]# newfs -b 32768 -f 4096 /dev/da0


[nfs]# bsdlabel da0
bsdlabel: /dev/da0: no valid label found
[nfs]#
even though I can mount it just fine

[nfs]# mount /dev/da0 /mnt
[nfs]# df
Filesystem  1K-blocks    Used     Avail Capacity  Mounted on
/dev/ad0s1a   1012974  356028    575910    38%    /
devfs               1       1         0   100%    /dev
/dev/ad0s1e   1012974  580870    351068    62%    /tmp
/dev/ad0s1d  15231278 5848016   8164760    42%    /usr
/dev/ad0s1f  19564270  192176  17806954     1%    /var
/dev/da0    720910352       8 663237516     0%    /mnt
[nfs]#


               -------Sequential Output-------- ---Sequential Input-- 
--Random--
               -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- 
--Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
da0s1      4000 61175 53.3 61832 16.3 31829  8.7 63008 60.4  79551 11.2 
224.2  0.7
da0s1 v    4000 61278 54.3 61938 16.4 31828  8.8 73566 71.7  81778 12.6 
216.5  0.7
da0        4000 62625 54.5 62154 15.7 31907  8.3 63297 60.8  80139 11.0 
226.2  0.7
da0 v      4000 62315 54.3 61759 15.4 32094  8.3 74209 72.3  84470 12.6 
223.5  0.7
da0 N      4000 75239 62.7 75771 15.5 33596  8.4 81909 78.6  98453 13.3 
214.8  0.8
da0 vN     4000 72786 61.7 74939 15.3 33672  8.4 82944 79.8 108106 14.9 
214.7  0.8
v = sysctl -w vfs.read_max=16
N = newfs -b 32768 -f 4096 /dev/da0

[nfs]# postmark
PostMark v1.5 : 3/27/01
pm>set location /mnt
pm>set size 300 100000
pm>set transactions 400000
pm>run
Creating files...Done
Performing transactions..........Done
Deleting files...Done
Time:
         1418 seconds total
         1418 seconds of transactions (282 per second)

Files:
         200107 created (141 per second)
                 Creation alone: 500 files (500 per second)
                 Mixed with transactions: 199607 files (140 per second)
         199905 read (140 per second)
         199384 appended (140 per second)
         200107 deleted (141 per second)
                 Deletion alone: 889 files (889 per second)
                 Mixed with transactions: 199218 files (140 per second)

Data:
         12715.55 megabytes read (8.97 megabytes per second)
         12728.92 megabytes written (8.98 megabytes per second)
pm>quit
[nfs]#

create a junk file
[nfs]# time dd if=/dev/urandom of=/mnt/big-4g-rand bs=1024k count=4000
4000+0 records in
4000+0 records out
4194304000 bytes transferred in 124.694915 secs (33636528 bytes/sec)
0.022u 116.020s 2:04.70 93.0%   21+2910k 124+32000io 0pf+0w

wait 2 min for disks to totally flush


[nfs]# time dd if=/dev/zero of=/mnt/big-4g-zero bs=1024k count=4000
4000+0 records in
4000+0 records out
4194304000 bytes transferred in 66.463634 secs (63106751 bytes/sec)
0.000u 11.265s 1:06.50 16.9%    21+2926k 126+32000io 0pf+0w


sysctl -w vfs.read_max=16

(clear out cache by reading via md5 4G of other files) and then wait a bit

[nfs]# time dd if=/mnt/big-4g-rand of=/dev/null bs=1024k count=4000
4000+0 records in
4000+0 records out
4194304000 bytes transferred in 59.178470 secs (70875506 bytes/sec)
0.000u 7.677s 0:59.18 12.9%     20+2784k 32128+0io 0pf+0w


[nfs]# postmark
PostMark v1.5 : 3/27/01
pm>set locatime /mnt
Eh?
pm>set location /mnt
pm>set size 300 100000
pm>set transactions 400000
pm>run
Creating files...Done
Performing transactions..........Done
Deleting files...Done
Time:
         1369 seconds total
         1369 seconds of transactions (292 per second)

Files:
         200107 created (146 per second)
                 Creation alone: 500 files (500 per second)
                 Mixed with transactions: 199607 files (145 per second)
         199905 read (146 per second)
         199384 appended (145 per second)
         200107 deleted (146 per second)
                 Deletion alone: 889 files (889 per second)
                 Mixed with transactions: 199218 files (145 per second)

Data:
         12715.55 megabytes read (9.29 megabytes per second)
         12728.92 megabytes written (9.30 megabytes per second)
pm>


With vfs.read_max=8


[nfs]# postmark
PostMark v1.5 : 3/27/01
pm>set location /mnt
pm>set size 300 100000
pm>set transactions 400000
pm>run
Creating files...Done
Performing transactions..........Done
Deleting files...Done
Time:
         1013 seconds total
         1013 seconds of transactions (394 per second)

Files:
         200107 created (197 per second)
                 Creation alone: 500 files (500 per second)
                 Mixed with transactions: 199607 files (197 per second)
         199905 read (197 per second)
         199384 appended (196 per second)
         200107 deleted (197 per second)
                 Deletion alone: 889 files (889 per second)
                 Mixed with transactions: 199218 files (196 per second)

Data:
         12715.55 megabytes read (12.55 megabytes per second)
         12728.92 megabytes written (12.57 megabytes per second)
pm>
pm>quit
[nfs]#

---------------------------------------------------------------------
newfs -b 32768 -f 4096 /dev/da0

[nfs]# postmark
PostMark v1.5 : 3/27/01
pm>set location /mnt
pm>set transactions 400000
pm>set size 300 100000
pm>run
Creating files...Done
Performing transactions..........Done
Deleting files...Done
Time:
         542 seconds total
         541 seconds of transactions (739 per second)

Files:
         200107 created (369 per second)
                 Creation alone: 500 files (500 per second)
                 Mixed with transactions: 199607 files (368 per second)
         199905 read (369 per second)
         199384 appended (368 per second)
         200107 deleted (369 per second)
                 Deletion alone: 889 files (889 per second)
                 Mixed with transactions: 199218 files (368 per second)

Data:
         12715.55 megabytes read (23.46 megabytes per second)
         12728.92 megabytes written (23.49 megabytes per second)
pm>
[nfs]# sysctl -w vfs.read_max=16
vfs.read_max: 8 -> 16

pm>set location /mnt
pm>set transactions 400000
pm>set size 300 100000
pm>run
Creating files...Done
Performing transactions..........Done
Deleting files...Done
Time:
         542 seconds total
         541 seconds of transactions (739 per second)

Files:
         200107 created (369 per second)
                 Creation alone: 500 files (500 per second)
                 Mixed with transactions: 199607 files (368 per second)
         199905 read (369 per second)
         199384 appended (368 per second)
         200107 deleted (369 per second)
                 Deletion alone: 889 files (889 per second)
                 Mixed with transactions: 199218 files (368 per second)

Data:
         12715.55 megabytes read (23.46 megabytes per second)
         12728.92 megabytes written (23.49 megabytes per second)
pm>




[nfs]# time dd if=/mnt/big-2g-rand count=2000 bs=1024k of=/dev/null
2000+0 records in
2000+0 records out
2097152000 bytes transferred in 19.298799 secs (108667489 bytes/sec)
0.000u 3.603s 0:19.32 18.6%     21+2965k 16019+0io 0pf+0w
[nfs]# time dd if=/mnt/big-2g-rand2 count=2000 bs=1024k of=/dev/null
2000+0 records in
2000+0 records out
2097152000 bytes transferred in 20.188331 secs (103879414 bytes/sec)
0.015u 3.515s 0:20.24 17.3%     20+2865k 16020+0io 3pf+0w
[nfs]# sysctl -w vfs.read_max=8

vfs.read_max: 16 -> 8
[nfs]# time dd if=/mnt/big-2g-rand count=2000 bs=1024k of=/dev/null
2000+0 records in
2000+0 records out
2097152000 bytes transferred in 21.421867 secs (97897722 bytes/sec)
0.007u 3.482s 0:21.43 16.2%     21+2873k 16018+0io 0pf+0w
[nfs]# time dd if=/mnt/big-2g-rand2 count=2000 bs=1024k of=/dev/null
2000+0 records in
2000+0 records out
2097152000 bytes transferred in 23.306603 secs (89981024 bytes/sec)
0.000u 3.590s 0:23.37 15.3%     19+2687k 16020+0io 3pf+0w
[nfs]#  



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?6.2.1.2.0.20050502222250.02da34f8>