Date: Fri, 27 Oct 2006 18:03:33 -0500 From: Steve Peterson <stevep-hv@zpfe.com> To: performance@freebsd.org Subject: gvinum raid5 performance seems slow Message-ID: <6.2.3.4.0.20061027180329.020bed68@localhost>
next in thread | raw e-mail | index | archive | help
I recently set up a media server for home use and decided to try the
gvinum raid5 support to learn about it and see how it performs. It
seems slower than I'd expect -- a little under 6MB/second, with about
50 IOs/drive/second -- and I'm trying to understand why. Any
assistance/pointers would be appreciated.
The disk system consists of 4 Seagate NL35 SATA ST3250623NS drives
connected to a Promise TX4300 (PDC40719) controller, organized as a
RAID5 volume via gvinum using this configuration:
drive drive01 device /dev/ad10
drive drive02 device /dev/ad6
drive drive03 device /dev/ad4
drive drive04 device /dev/ad8
volume vol1
plex org raid5 256k
sd length 200001m drive drive01
sd length 200001m drive drive02
sd length 200001m drive drive03
sd length 200001m drive drive04
dd reports the following performance on a 1G file write to the RAID5
hosted volume:
$ time dd if=/dev/zero of=big.file bs=1m count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 179.717742 secs (5834571 bytes/sec)
179.76 real 0.02 user 16.60 sys
By comparison, creating the same file on the system disk (an old ATA
ST380021A connected via a SIS 730 on the motherboard):
time dd if=/dev/zero of=big.file bs=1m count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 28.264056 secs (37099275 bytes/sec)
28.32 real 0.01 user 19.13 sys
and vmstat reports about 280-300 IOs/second to that drive.
The CPU is pretty weak -- an Athlon 750. Is that the source of my
problem? If you look at the vmstat output below the machine is busy
but not pegged.
$ vmstat -n 5 1
procs memory page disks
faults cpu
r b w avm fre flt re pi po fr sr ad0 ad4 ad6 ad8
ad10 in sy cs us sy id
1 0 0 96700 134124 19 3 0 0 65 12 0 0 0 0 0
1375 239 621 1 1 98
0 0 0 96700 134124 2 0 0 0 0 0 0 0 0 0 0
1305 180 445 2 0 98
0 0 0 96700 134124 0 0 0 0 0 0 0 0 0 0 0
1302 201 449 1 1 98
0 0 0 96700 134124 0 0 0 0 0 0 0 0 0 0 0
1353 165 545 1 1 98
1 0 0 99088 129828 311 0 0 0 1328 0 0 14 24 23 13
1448 310 1286 0 18 82
0 1 0 99088 120228 0 0 0 0 5088 0 0 56 48 48 56
1686 187 2794 0 64 36
0 1 0 99088 114212 0 0 0 0 4992 0 0 54 50 52 48
1659 184 2694 2 54 44
0 1 0 99088 107044 0 0 0 0 5088 0 0 50 51 57 50
1670 188 2777 2 56 42
0 1 0 99088 101668 0 0 0 0 5184 0 0 51 58 52 51
1678 190 2801 1 56 43
0 1 0 99088 95140 0 0 0 0 4896 0 1 48 52 48 52
1662 186 2665 1 57 42
0 1 0 99088 88356 0 0 0 0 4896 0 0 50 47 50 53
1648 184 2641 1 54 45
0 1 0 99088 81316 0 0 0 0 4800 0 0 52 48 48 53
1620 184 2602 0 57 43
0 1 0 99088 73892 0 0 0 0 4992 0 0 54 50 49 46
1685 191 2698 1 62 37
0 1 0 99088 68260 0 0 0 0 4992 0 0 48 50 54 51
1661 184 2663 1 56 43
0 1 0 99088 61032 0 0 0 0 5088 0 0 51 53 53 51
1656 186 2735 1 57 42
0 1 0 99088 54436 0 0 0 0 4768 0 0 48 50 50 47
1637 184 2580 0 56 44
0 1 0 99088 48292 0 0 0 0 5120 0 2 50 55 49 53
1647 186 2710 1 62 38
0 1 0 99088 42020 0 0 0 0 4928 0 0 53 47 49 55
1657 192 2671 1 58 41
0 1 0 99088 35492 0 0 0 0 4768 0 0 50 48 47 49
1639 181 2544 0 59 41
0 1 0 99088 28836 0 0 0 0 4996 0 2 54 50 52 48
1654 186 2689 0 57 43
0 1 0 99088 21540 0 0 0 0 4704 0 0 47 47 49 49
1648 184 2580 2 58 40
0 1 0 99088 14756 0 0 0 0 4928 0 5 49 51 54 48
1638 184 2645 1 58 41
0 1 0 99088 10276 0 0 0 0 4864 0 0 48 49 48 49
1679 196 2662 1 55 45
0 1 0 99088 10152 0 0 0 0 4800 1937 0 47 51 50 48
1651 181 2637 1 58 42
1 1 0 99088 14560 0 6746 0 0 4948
9308 0 46 37 40 49 1616 176 2396 1 57 42
0 1 0 99088 15336 0 9815 0 0 5328
11178 0 61 64 59 67 1612 188 2741 0 62 38
0 1 0 99088 15532 0 6188 0 0 5536
7452 0 49 49 57 38 1618 190 2526 1 61 38
1 0 0 99088 12012 0 3054 0 0 5728
3726 0 69 63 67 61 1637 189 2859 2 58 41
0 1 0 99088 15532 0 12692 0 0 5700
14904 1 55 49 53 57 1640 183 2677 0 58 42
0 1 0 99088 9388 0 0 0 0 5696 0 0 65 62 61 66
1600 184 2699 1 61 38
0 1 0 99088 14828 0 2925 0 0 5328
5565 0 64 68 57 65 1602 194 2758 1 53 46
0 1 0 99088 13100 0 2718 0 0 5904
3726 0 50 52 53 55 1594 184 2556 1 59 40
0 1 0 99088 14508 0 13368 0 0 5984
14904 0 64 66 61 70 1642 184 2901 2 63 36
0 1 0 99088 13608 0 0 0 0 5888 1855 0 65 71 71 65
1630 177 2849 1 63 36
0 1 0 99088 13932 0 10265 0 0 5728
11178 0 62 64 63 51 1633 176 2742 1 60 39
configuration from last boot:
Sep 24 12:11:35 archive kernel: Copyright (c) 1992-2006 The FreeBSD Project.
Sep 24 12:11:35 archive kernel: Copyright (c) 1979, 1980, 1983, 1986,
1988, 1989, 1991, 1992, 1993, 1994
Sep 24 12:11:35 archive kernel: The Regents of the University of
California. All rights reserved.
Sep 24 12:11:35 archive kernel: FreeBSD 6.1-RELEASE-p7 #5: Wed Sep 20
14:03:56 CDT 2006
Sep 24 12:11:35 archive kernel:
root@archive.zpfe.com:/usr/obj/usr/src/sys/ARCHIVE
Sep 24 12:11:35 archive kernel: Timecounter "i8254" frequency 1193182
Hz quality 0
Sep 24 12:11:35 archive kernel: CPU: AMD Athlon(tm) Processor
(746.67-MHz 686-class CPU)
Sep 24 12:11:35 archive kernel: Origin = "AuthenticAMD" Id =
0x644 Stepping = 4
Sep 24 12:11:35 archive kernel:
Features=0x183f9ff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR>
Sep 24 12:11:35 archive kernel: AMD
Features=0xc0440800<SYSCALL,<b18>,MMX+,3DNow+,3DNow>
Sep 24 12:11:35 archive kernel: real memory = 251592704 (239 MB)
Sep 24 12:11:35 archive kernel: avail memory = 236695552 (225 MB)
Sep 24 12:11:35 archive kernel: kbd1 at kbdmux0
Sep 24 12:11:35 archive kernel: acpi0: <AMIINT SiS730SX> on motherboard
Sep 24 12:11:35 archive kernel: acpi0: Power Button (fixed)
Sep 24 12:11:35 archive kernel: Timecounter "ACPI-fast" frequency
3579545 Hz quality 1000
Sep 24 12:11:35 archive kernel: acpi_timer0: <24-bit timer at
3.579545MHz> port 0x5008-0x500b on acpi0
Sep 24 12:11:35 archive kernel: cpu0: <ACPI CPU> on acpi0
Sep 24 12:11:35 archive kernel: acpi_throttle0: <ACPI CPU Throttling> on cpu0
Sep 24 12:11:35 archive kernel: acpi_button0: <Power Button> on acpi0
Sep 24 12:11:35 archive kernel: pcib0: <ACPI Host-PCI bridge> port
0xcf8-0xcff on acpi0
Sep 24 12:11:35 archive kernel: pci0: <ACPI PCI bus> on pcib0
Sep 24 12:11:35 archive kernel: agp0: <SiS 730 host to AGP bridge>
mem 0xd0000000-0xd3ffffff at device 0.0 on pci0
Sep 24 12:11:35 archive kernel: atapci0: <SiS 730 UDMA100 controller>
port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xff00-0xff0f at device 0.1 on pci0
Sep 24 12:11:35 archive kernel: ata0: <ATA channel 0> on atapci0
Sep 24 12:11:35 archive kernel: ata1: <ATA channel 1> on atapci0
Sep 24 12:11:35 archive kernel: isab0: <PCI-ISA bridge> at device 1.0 on pci0
Sep 24 12:11:35 archive kernel: isa0: <ISA bus> on isab0
Sep 24 12:11:35 archive kernel: sis0: <SiS 900 10/100BaseTX> port
0xd400-0xd4ff mem 0xcfff8000-0xcfff8fff irq 11 at device 1.1 on pci0
Sep 24 12:11:35 archive kernel: miibus0: <MII bus> on sis0
Sep 24 12:11:35 archive kernel: ukphy0: <Generic IEEE 802.3u media
interface> on miibus0
Sep 24 12:11:35 archive kernel: ukphy0: 10baseT, 10baseT-FDX,
100baseTX, 100baseTX-FDX, auto
Sep 24 12:11:35 archive kernel: sis0: Ethernet address: 00:0a:e6:14:e3:53
Sep 24 12:11:35 archive kernel: ohci0: <SiS 5571 USB controller> mem
0xcfff9000-0xcfff9fff irq 10 at device 1.2 on pci0
Sep 24 12:11:35 archive kernel: ohci0: [GIANT-LOCKED]
Sep 24 12:11:35 archive kernel: usb0: OHCI version 1.0, legacy support
Sep 24 12:11:35 archive kernel: usb0: <SiS 5571 USB controller> on ohci0
Sep 24 12:11:35 archive kernel: usb0: USB revision 1.0
Sep 24 12:11:35 archive kernel: uhub0: SiS OHCI root hub, class 9/0,
rev 1.00/1.00, addr 1
Sep 24 12:11:35 archive kernel: uhub0: 3 ports with 3 removable, self powered
Sep 24 12:11:35 archive kernel: ohci1: <SiS 5571 USB controller> mem
0xcfffa000-0xcfffafff irq 10 at device 1.3 on pci0
Sep 24 12:11:35 archive kernel: ohci1: [GIANT-LOCKED]
Sep 24 12:11:35 archive kernel: usb1: OHCI version 1.0, legacy support
Sep 24 12:11:35 archive kernel: usb1: <SiS 5571 USB controller> on ohci1
Sep 24 12:11:35 archive kernel: usb1: USB revision 1.0
Sep 24 12:11:35 archive kernel: uhub1: SiS OHCI root hub, class 9/0,
rev 1.00/1.00, addr 1
Sep 24 12:11:35 archive kernel: uhub1: 3 ports with 3 removable, self powered
Sep 24 12:11:35 archive kernel: pcib1: <PCI-PCI bridge> at device 2.0 on pci0
Sep 24 12:11:35 archive kernel: pci1: <PCI bus> on pcib1
Sep 24 12:11:35 archive kernel: pci1: <display, VGA> at device 0.0
(no driver attached)
Sep 24 12:11:35 archive kernel: atapci1: <Promise PDC40719 SATA300
controller> port 0xdc00-0xdc7f,0xd800-0xd8ff mem
0xcfffb000-0xcfffbfff,0xcffc0000-0xcffdffff
irq 11 at device 11.0 on pci0
Sep 24 12:11:35 archive kernel: ata2: <ATA channel 0> on atapci1
Sep 24 12:11:35 archive kernel: ata3: <ATA channel 1> on atapci1
Sep 24 12:11:35 archive kernel: ata4: <ATA channel 2> on atapci1
Sep 24 12:11:35 archive kernel: ata5: <ATA channel 3> on atapci1
Sep 24 12:11:35 archive kernel: acpi_button1: <Sleep Button> on acpi0
Sep 24 12:11:35 archive kernel: atkbdc0: <Keyboard controller
(i8042)> port 0x60,0x64 irq 1 on acpi0
Sep 24 12:11:35 archive kernel: atkbd0: <AT Keyboard> irq 1 on atkbdc0
Sep 24 12:11:35 archive kernel: atkbd0: <AT Keyboard> irq 1 on atkbdc0
Sep 24 12:11:35 archive kernel: kbd0 at atkbd0
Sep 24 12:11:35 archive kernel: atkbd0: [GIANT-LOCKED]
Sep 24 12:11:35 archive kernel: pmtimer0 on isa0
Sep 24 12:11:35 archive kernel: orm0: <ISA Option ROMs> at iomem
0xc0000-0xcbfff,0xcc000-0xd57ff,0xd5800-0xdffff on isa0
Sep 24 12:11:35 archive kernel: ppc0: parallel port not found.
Sep 24 12:11:35 archive kernel: sc0: <System console> at flags 0x100 on isa0
Sep 24 12:11:35 archive kernel: sc0: VGA <16 virtual consoles, flags=0x300>
Sep 24 12:11:35 archive kernel: sio0: configured irq 4 not in bitmap
of probed irqs 0
Sep 24 12:11:35 archive kernel: sio0: port may not be enabled
Sep 24 12:11:35 archive kernel: sio0 at port 0x3f8-0x3ff irq 4 flags
0x10 on isa0
Sep 24 12:11:35 archive kernel: sio0: type 8250 or not responding
Sep 24 12:11:35 archive kernel: sio1: configured irq 3 not in bitmap
of probed irqs 0
Sep 24 12:11:35 archive kernel: sio1: port may not be enabled
Sep 24 12:11:35 archive kernel: vga0: <Generic ISA VGA> at port
0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Sep 24 12:11:35 archive kernel: Timecounter "TSC" frequency 746674414
Hz quality 800
Sep 24 12:11:35 archive kernel: Timecounters tick every 1.000 msec
Sep 24 12:11:35 archive kernel: ad0: 76319MB <Seagate ST380021A 3.19>
at ata0-master UDMA100
Sep 24 12:11:35 archive kernel: acd0: CDROM <CD-524E/3.0C> at ata1-master PIO3
Sep 24 12:11:35 archive kernel: ad4: 238475MB <Seagate ST3250623NS
5.01> at ata2-master SATA150
Sep 24 12:11:35 archive kernel: ad6: 238475MB <Seagate ST3250623NS
5.01> at ata3-master SATA150
Sep 24 12:11:35 archive kernel: ad8: 238475MB <Seagate ST3250623NS
5.01> at ata4-master SATA150
Sep 24 12:11:35 archive kernel: ad10: 238475MB <Seagate ST3250623NS
5.01> at ata5-master SATA150
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?6.2.3.4.0.20061027180329.020bed68>