Skip site navigation (1)Skip section navigation (2)
Date:      Sat, 30 Oct 2010 11:23:46 -0700
From:      Jeremy Chadwick <freebsd@jdc.parodius.com>
To:        Peter =?iso-8859-1?Q?Ankerst=E5l?= <peter@pean.org>
Cc:        freebsd-fs@freebsd.org
Subject:   Re: Raid + zfs performace.
Message-ID:  <20101030182346.GA61519@icarus.home.lan>
In-Reply-To: <86693036-9351-4303-BADA-C99F7A4C375C@pean.org>
References:  <D2954020-C3A0-46EC-8C64-EB57EA1E9B21@pean.org> <AANLkTinQWchAPtcqcO3mDt9gKK5tCsHo8khyiD69M4BV@mail.gmail.com> <86693036-9351-4303-BADA-C99F7A4C375C@pean.org>

next in thread | previous in thread | raw e-mail | index | archive | help
On Sat, Oct 30, 2010 at 07:56:56PM +0200, Peter Ankerstål wrote:
> 
> 
> On 30 okt 2010, at 19.39, Sean wrote:
> 
> >> I have a question about raid and zfs. I Have a hardware-raid running.
> >> A mirror thats the only storage in my zfs pool. Im going to
> >> add another mirror to the machine and my question is, what is the best option performace-wise?
> > 
> > The best performance option is to get rid of the hardware-raid, and
> > present each disc to ZFS in a JBOD fashion.
> 
> Ok. RIght now thats not an option. I have a da0 device thats a hardware-raid mirror. and it is currently
> the only device in the only pool on the machine.
> 
> > 
> >> Is it to add the other mirror to the same pool or create another separate pool for that mirror?
> >> Btw. Today my disk are quite saturated r/w wise.
> > 
> > RAID functionality only exists within the context of a single pool.
> > You don't create a new pool and then try to mirror the two pools. You
> > add the storage to an existing pool, unless you have a reason to start
> > a new pool. When I already have a mirror, I like to add new mirror
> > sets. It's the equivalent of a RAID 10 scenario.
> 
> Yes I know.
> I thought maybe because the existing pool is kind of r/w saturated it should be better
> to create a new independent pool for the new drives. In that way the heavy activity 
> would not "spread" to the new drives.
> 
> Now you presented me with a third option. So you think I should skip to create
> a new hardware-raid mirror and instead use two single drives and add these as
> a mirror to the existing pool? How will zfs handle howswap of these drives?
> I've seen a few crashes due to ata-detach in other systems.

As stated previously: ideally you should turn off all RAID capabilities
on the controller itself and just use JBOD + let FreeBSD deal with the
RAID'ing (in this case, ZFS).

I realise you can't do this for the current hardware RAID'd "da0"
mirror.  I also understand the reason you might have done this.  I too
would trust a hardware RAID controller to handle booting from a mirror
more than the FreeBSD bootstraps.  (I choose not to deal with things
like gptzfsboot, zfsloader, etc. etc. -- these should all just become
part of the standard bootstraps and stop requiring people to jump
through hoops to achieve such capabilities)

As for hot-swapping of drives: there are well-established procedures for
this sort of thing.  I blogged about it[1] not too long ago, with
real-world server hardware.  I think it will address your concerns,
since I spent the time to document *everything*, including all output.

What confuses me about your statement is that you're talking about
"ata-detach" when your drives are appearing as daX, which implies use of
CAM.  Are you talking about hot-swapping failing on a completely
different system which was using ata(4) without AHCI?  If so, I'm not
surprised.  You need AHCI to achieve this.

Finally, you should be aware that there are two AHCI "methods"
implemented in FreeBSD: ataahci.ko and ahci.ko.  ataahci is the older of
the two and uses ata(4).  ahci.ko is the newer of the two and is
significantly better, since it makes use of CAM, and thus provides
performance improvements and handles hot-swapping a lot better (IMHO).
ahci.ko "trumps" ataahci.ko (meaning if your kernel includes ataahci and
you do "load ahci" from loader or loader.conf, ahci.ko will be used
instead).  Be aware that ahci.ko disks appear as adaX (this is not a
typo).

[1]: http://koitsu.wordpress.com/2010/07/22/freebsd-and-zfs-hot-swapping-sata-disks-with-ahci/

-- 
| Jeremy Chadwick                                   jdc@parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.              PGP: 4BD6C0CB |




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20101030182346.GA61519>