Date: Thu, 06 May 2010 10:09:33 +0100 From: Michal <michal@ionic.co.uk> To: freebsd-stable@freebsd.org Subject: iscsi, zfs, RAIS = Cheap SAN...maybe Message-ID: <4BE2874D.10909@ionic.co.uk>
next in thread | raw e-mail | index | archive | help
*I apologise about the length of this e-mail, I tried to cover all details* I am following up on a previous post which is here http://unix.derkeiler.com/Mailing-Lists/FreeBSD/stable/2010-03/msg00392.html The sum up my end goal is this; "To have a SAN type system where I have multiple servers that contain multiple disks. I can lose a server and due to RAID1 across the servers, the data will still be on the network, IO will increase due to their being multiple servers to read from simultaneously as opposed to one NAS box and lastly to be able to add new servers to the system to increase the storage (to the end users the amount of available space increases). Stage one is to create a two server system that can take a failure of a server. Second stage is to get better IO from two servers then from one NAS box. Last stage is to have all that and the ability to easily add more storage" I have created 3 (not great spec just some spear) servers, two of which have 2 HDD's each and I will call storedevice1 and storedevice2 these are my devices that will hold the data. My 3rd server is my controller which controls the devices. Each server has two hard drives which using iscsi-target I export as data0 and data1 and in the controller I use the iscsi-initiator to connect to these 4 HDD's. Here is the config files storedevice1: #cat /usr/local/etc/iscsi/targets # extents file start length #extent0 /tmp/iscsi-target0 0 100MB extent0 /data0/data 0 28GB extent1 /data1/data 0 28GB # target flags storage netmask target0 rw extent0 192.168.2.0/24 target1 rw extent1 192.168.2.0/24 # ls -lh /data0/ total 2195442 drwxrwxr-x 2 root operator 512B Apr 20 17:49 .snap -rw-r--r-- 1 root wheel 28G May 5 21:37 data # ls -lh /data1/ total 2195442 drwxrwxr-x 2 root operator 512B Apr 22 13:27 .snap -rw-r--r-- 1 root wheel 28G May 5 21:37 data storedevice2: #cat /usr/local/etc/iscsi/targets # extents file start length extent2 /data0/data 0 28GB extent3 /data1/data 0 28GB # target flags storage netmask target2 rw extent2 192.168.2.0/24 target3 rw extent3 192.168.2.0/24 # ls -lh /data0/ total 2191250 drwxrwxr-x 2 root operator 512B Apr 22 15:09 .snap -rw-r--r-- 1 root wheel 28G May 5 21:37 data # ls -lh /data1/ total 2191250 drwxrwxr-x 2 root operator 512B Apr 22 17:40 .snap -rw-r--r-- 1 root wheel 28G May 5 21:37 data which gives me 4 extents and 4 targets accross both. /dataX/data is a file which I think it needs to be (???) On my controller I have; OffSanCtrl1# cat /etc/iscsi.conf offsan0 { TargetName = iqn.1994-04.org.netbsd.iscsi-target:target0 TargetAddress = 192.168.2.160:3260,1 } offsan1 { TargetName = iqn.1994-04.org.netbsd.iscsi-target:target1 TargetAddress = 192.168.2.160:3260,1 } offsan2 { TargetName = iqn.1994-04.org.netbsd.iscsi-target:target2 TargetAddress = 192.168.2.161:3260,1 offsan3 { TargetName = iqn.1994-04.org.netbsd.iscsi-target:target3 TargetAddress = 192.168.2.161:3260,1 } which is my initiator and connects to my 4 targets Up to this point I think I am doing everything correctly. I then setup a zpool on the controller OffSanCtrl1# zpool status pool: store0 state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM store0 ONLINE 0 0 0 mirror ONLINE 0 0 0 da1s1d ONLINE 0 0 0 da3s1d ONLINE 0 0 0 mirror ONLINE 0 0 0 da2s1d ONLINE 0 0 0 da4s1d ONLINE 0 0 0 errors: No known data errors with da1s1d and da2s1d being from storedevice1 and da3s1d and da4s1d from storedevice2 so if I am correct this should be a a sort of RAID10 (anything that could be done better please tell me). I now set-up zfs on this zpool (again, I think I'm doing this the right way) OffSanCtrl1# zfs list NAME USED AVAIL REFER MOUNTPOINT store0 4.12G 50.5G 19K /store0 store0/users 4.12G 50.5G 4.12G /store0/users Lastly, I need to be able to allow network users/servers to connect to this. My choices I think are iscsi and samba as I have *nix and windows machines, so I'll try iscsi. From the controller, create a target from the zfs mount point I have created. If I am correct, a user should be able to connect to the target from the controller, write data which will actually be writing data across both storedevice's OffSanCtrl1# cat /usr/local/etc/iscsi/targets # extents file start length extent0 /store0/users/data 0 55G # target flags storage netmask target9 rw extent0 192.168.2.0/24 It didn't work until I created a file called data under /store0/users/ so I guess this relies on a file of some sort..maybe?? I think this bit I've done wrong. I test this from a windows7 machine, open up the iscsi-initiator, connect to the controller, it connects and I can see the drive so I format it and mount it. I can then write data to the drive and read data from it. So...it works...to a degree...though I think what is actually happening is it's writing data to the controller somewhere and NOT to the storedevices...however I'm getting a bit lost. I get quite good write speeds at first and it slowly craws down to about 5MB/s...but with small files and to begind with, with big files it is very fast and impressive. What is causing the slow down, I am not sure...possibly RAM...cache...I'm not using amazing servers as storedevice1/2 so it could possibly be that. But I know I have lots of things to do...so I'm asking for any advice and assistance from the community. P.S Please note at this point I am not looking at ZIL or things like that, I simply want a working system to test the theory and build a proposal with. I know better servers will give me better IO, but should not affect testing redundency, failovers etc etc which is what I need to work out. Many thanks...even if you just read this beast of an e-mail
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4BE2874D.10909>