Date: Thu, 06 May 2010 16:11:47 +0400 From: Boris Samorodov <bsam@ipt.ru> To: Michal <michal@ionic.co.uk> Cc: freebsd-stable@freebsd.org Subject: Re: iscsi, zfs, RAIS = Cheap SAN...maybe Message-ID: <73575852@bb.ipt.ru> In-Reply-To: <4BE2874D.10909@ionic.co.uk> (michal@ionic.co.uk's message of "Thu, 06 May 2010 10:09:33 %2B0100") References: <4BE2874D.10909@ionic.co.uk>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi!
On Thu, 06 May 2010 10:09:33 +0100 Michal wrote:
> Many thanks...even if you just read this beast of an e-mail
This is not an answer to your questions but my tiny experience
in this area.
I created two USB 2G storages with FreeBSD current i386: node001
and node002. Each system has one 250G hard disk. Both nodes
export their disks via iscsi:
-----
node001% sudo istgtcontrol list
lun0 storage "/dev/ad0" 250059350016
DONE LIST command
-----
node002% sudo istgtcontrol list
lun0 storage "/dev/ad4" 250059350016
DONE LIST command
-----
I used my workstation as a controller. And since it's an i386
(FreeBSD-CURRENT) I didn't try zfs but created a gmirror:
-----
da0 at iscsi0 bus 0 scbus7 target 0 lun 0
da0: <FreeBSD iSCSI DISK 0001> Fixed Direct Access SCSI-5 device
da0: 238475MB (488397168 512 byte sectors: 255H 63S/T 30401C)
da1 at iscsi0 bus 0 scbus7 target 1 lun 0
da1: <FreeBSD iSCSI DISK 0001> Fixed Direct Access SCSI-5 device
da1: 238475MB (488397168 512 byte sectors: 255H 63S/T 30401C)
GEOM_MIRROR: Device mirror/storage0 launched (2/2).
-----
bsam% gmirror status
Name Status Components
mirror/storage0 COMPLETE da0
da1
bsam% mount -p | grep storage
/dev/mirror/storage0 /storage ufs rw 2 2
-----
While using 100Mb/s network connection I've got 7-8 MB/s writes
(after some sysctl tuning, 3-4 MB/s without). And switching to
1Gb/s network connection (Intel Gigabit Ethernet Controllers)
gave me something like 20 Mb/s writes (no tuning):
-----
bsam% dd if=/dev/zero of=/storage/file.2G bs=1M count=2000
2000+0 records in
2000+0 records out
2097152000 bytes transferred in 108.479201 secs (19332296 bytes/sec)
bsam% netstat -w10
input (Total) output
packets errs idrops bytes packets errs bytes colls
21 0 0 1658 9 0 582 0
44 0 0 5425 35 0 3014 0
70470 0 0 5640905 114909 0 184072616 0
157670 0 0 11966753 257062 0 390774311 0
175056 0 0 11951631 285343 0 390099162 0
156502 0 0 11867216 255072 0 387223400 0
174907 0 0 11947580 285064 0 390607312 0
155607 0 0 11800874 253530 0 385202710 0
173359 0 0 11835884 282798 0 386524530 0
157604 0 0 11922283 256966 0 389324902 0
174224 0 0 11919850 284396 0 390212650 0
173677 0 0 11853356 283110 0 386944770 0
174934 0 0 11932229 285361 0 389785504 0
69574 0 0 4584810 113056 0 148735429 0
282 0 0 22196 374 0 452062 0
20 0 0 2054 14 0 2473 0
-----
node001% netstat -w10
input (Total) output
packets errs idrops bytes packets errs bytes colls
12 0 0 1008 1 0 178 0
17 0 0 1626 2 0 292 0
60498 0 0 88413959 37258 0 2616474 0
139627 0 0 204095754 86076 0 6064930 0
139188 0 0 203450154 85816 0 6047488 0
138486 0 0 202411672 85363 0 6000534 0
139258 0 0 203561033 85848 0 6049360 0
137636 0 0 201181818 84866 0 5981524 0
138262 0 0 202064896 85169 0 6004090 0
139297 0 0 203633800 85829 0 6031458 0
138884 0 0 203011098 85495 0 6025390 0
138280 0 0 202089488 85166 0 6004198 0
139272 0 0 203583962 85828 0 6047758 0
58465 0 0 85346106 36192 0 2545632 0
193 0 0 236228 132 0 10360 0
-----
node002% netstat -w10
input (Total) output
packets errs idrops bytes packets errs bytes colls
12 0 0 1008 1 0 178 0
16 0 0 1566 2 0 292 0
73703 0 0 107730061 44956 0 3134984 0
139420 0 0 203862166 85058 0 5899360 0
139295 0 0 203674472 85024 0 5896468 0
138129 0 0 201945040 84345 0 5850658 0
139068 0 0 203351261 84873 0 5886130 0
138214 0 0 202097218 84293 0 5846170 0
137860 0 0 201540738 84120 0 5835754 0
138751 0 0 202889156 84668 0 5871694 0
139260 0 0 203626362 84907 0 5888854 0
138468 0 0 202435390 84534 0 5863900 0
139210 0 0 203557426 84865 0 5885938 0
45288 0 0 66099374 27684 0 1924360 0
193 0 0 236228 136 0 10624 0
11 0 0 1574 3 0 518 0
-----
Well, there were no tuning at all since I didn't have much spare time
to exteriment.
Here is some more info about hard and soft.
Controller:
-----
bsam% uname -a
FreeBSD bsam.sem.ipt.ru 9.0-CURRENT FreeBSD 9.0-CURRENT #2 r207498: Sun May 2 16:19:56 MSD 2010 bsam@bsam.sem.ipt.ru:/home/bsam/FreeBSD/base/head/obj/storage/src/sys/BB i386
age0@pci0:2:0:0: class=0x020000 card=0x82261043 chip=0x10481969 rev=0xb0 hdr=0x00
vendor = 'Attansic (Now owned by Atheros)'
device = 'Gigabit Ethernet 10/100/1000 Base-T Controller (Atheros L1)'
class = network
subclass = ethernet
cap 01[40] = powerspec 2 supports D0 D3 current D0
cap 05[48] = MSI supports 1 message, 64 bit enabled with 1 message
cap 10[58] = PCI-Express 1 endpoint max data 128(128) link x1(x1)
-----
Node001:
-----
node001% uname -a
FreeBSD node001.sem.ipt.ru 9.0-CURRENT FreeBSD 9.0-CURRENT #13 r206496: Mon Apr 12 18:28:45 MSD 2010 bsam@bsam.sem.ipt.ru:/m/home/bsam/FreeBSD/base/head/obj/m/home/bsam/FreeBSD/base/head/src/sys/BB i386
em0@pci0:2:1:0: class=0x020000 card=0x002e8086 chip=0x100e8086 rev=0x02 hdr=0x00
vendor = 'Intel Corporation'
device = 'Gigabit Ethernet Controller (82540EM)'
class = network
subclass = ethernet
cap 01[dc] = powerspec 2 supports D0 D3 current D0
cap 07[e4] = PCI-X supports 2048 burst read, 1 split transaction
cap 05[f0] = MSI supports 1 message, 64 bit
-----
Node002:
-----
node002% uname -a
FreeBSD node002.sem.ipt.ru 9.0-CURRENT FreeBSD 9.0-CURRENT #13 r206496: Mon Apr 12 18:28:45 MSD 2010 bsam@bsam.sem.ipt.ru:/m/home/bsam/FreeBSD/base/head/obj/m/home/bsam/FreeBSD/base/head/src/sys/BB i386
em0@pci0:0:10:0: class=0x020000 card=0x13768086 chip=0x107c8086 rev=0x05 hdr=0x00
vendor = 'Intel Corporation'
device = 'Gigabit Ethernet Controller (Copper) rev 5 (82541PI)'
class = network
subclass = ethernet
cap 01[dc] = powerspec 2 supports D0 D3 current D0
cap 07[e4] = PCI-X supports 2048 burst read, 1 split transaction
-----
--
WBR, Boris Samorodov (bsam)
Research Engineer, http://www.ipt.ru Telephone & Internet SP
FreeBSD Committer, http://www.FreeBSD.org The Power To Serve
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?73575852>