Date: Thu, 2 Sep 2010 21:08:41 -0700 From: Jeremy Chadwick <freebsd@jdc.parodius.com> To: Zaphod Beeblebrox <zbeeble@gmail.com> Cc: Michal <michal@sharescope.co.uk>, Joshua Boyd <boydjd@jbip.net>, freebsd-stable@freebsd.org Subject: Re: Extending your zfs pool with multiple devices Message-ID: <20100903040841.GA59175@icarus.home.lan> In-Reply-To: <AANLkTi=38yK0eb6L27X_J6fO4qHHc2qxZz-hD-6wwNqx@mail.gmail.com> References: <4C7FA50D.4000409@sharescope.co.uk> <AANLkTimaqp6Zu23ADA%2BhcU8Gr8nZtzecRwPLGJaRaVpj@mail.gmail.com> <AANLkTi=38yK0eb6L27X_J6fO4qHHc2qxZz-hD-6wwNqx@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, Sep 02, 2010 at 04:56:04PM -0400, Zaphod Beeblebrox wrote:
> [regarding getting more disks in a machine]
>
> On Thu, Sep 2, 2010 at 12:47 PM, Joshua Boyd <boydjd@jbip.net> wrote:
> > You need an HP SAS expander card in the new box, and an HBA in your primary
> > box with external ports to hook it into.
> >
> > Then the drives in the other box will show up as local drives on your
> > primary box.
> >
> > You don't even need an operating system on the second box, it just needs
> > enough hardware in it to supply power to the SAS expander.
>
> An inexpensive option are SATA port replicators. Think SATA switch or
> hub. 1:4 is common and cheap.
>
> I have a motherboard with intel ICH10 chipset. It commonly provides 6
> ports. This chipset is happy to configure port replicators. Meaning
> you can put 24 drives on this motherboard.
>
> ...
>
> With 1.5T disks, I find that the 4 to 1 multipliers have a small
> effect on speed. The 4 drives I have on the multipler are saturated
> at 100% a little bit more than the drives directly connected.
> Essentially you have 3 gigabit for 4 drives instead of 3 gigabit for 1
> drive.
1:4 SATA replicators impose a bottleneck on the overall bandwidth
available between the replicator and the disks attached, as you stated.
Diagram:
ICH10
|||___ (SATA300) Port 0, Disk 0
||____ (SATA300) Port 1, Disk 1
|_____ (SATA300) Port 2, eSATA Replicator
||||________ (SATA300) Port 0, Disk 2
|||_________ (SATA300) Port 1, Disk 3
||__________ (SATA300) Port 2, Disk 4
|___________ (SATA300) Port 3, Disk 5
If Disks 2 through 5 are decent disks (pushing 100MB/sec), essentially
you have 100*4 = 400MB/sec worth of bandwidth being shoved across a
300MB/sec link. That's making the assumption the disks attached are
magnetic and not SSD, and not taking into consideration protocol
overhead.
Given the evolutionary rate of hard disks and SSDs, replicators are (in
my opinion) not a viable solution mid or long-term.
Even Silicon Image's products at this point are starting to force a 1:2
ratio on the replicators, probably to address the bottleneck issue:
http://www.siliconimage.com/products/product.aspx?pid=154
http://www.siliconimage.com/products/product.aspx?pid=155
A better choice is a SATA multilane HBA, which are usually PCIe-based
with a single connector on the back of the HBA which splits out to
multiple disks (usually 4, but sometimes more).
An ideal choice is ane Areca ARC-1300 series SAS-based PCIe x4 multilane
adapters, which provides SATA300 to each individual disk and uses PCIe
x4 (which can handle about 1GByte/sec in each direction, so 2GByte/sec
total)...
http://www.areca.com.tw/products/sasnoneraid.htm
...but there doesn't appear to be driver support for FreeBSD for this
series of controller (arcmsr(4) doesn't mention the ARC-1300 series). I
also don't know what Areca means on their site when they say
"BSD/FreeBSD (will be available with 6Gb/s Host Adapter"), given that
none of the ARC-1300 series cards are SATA600.
If people are more focused on total number of devices (disks) that are
available, then they should probably be looking at dropping a pretty
penny on a low-end filer. Otherwise, consider replacing the actual hard
disks themselves with drives of a higher capacity.
</soapbox>
--
| Jeremy Chadwick jdc@parodius.com |
| Parodius Networking http://www.parodius.com/ |
| UNIX Systems Administrator Mountain View, CA, USA |
| Making life hard for others since 1977. PGP: 4BD6C0CB |
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20100903040841.GA59175>