Skip site navigation (1)Skip section navigation (2)
Date:      Mon, 24 Dec 2018 18:13:47 -0500
From:      Zaphod Beeblebrox <zbeeble@gmail.com>
To:        freebsd-fs <freebsd-fs@freebsd.org>
Subject:   Re: Suggestion for hardware for ZFS fileserver
Message-ID:  <CACpH0Md5y%2BSFTHbRL=OzP9joG60gKStOkoK3GrZqTYHO97k_FA@mail.gmail.com>
In-Reply-To: <YQBPR01MB038868AC3D6BAC5C6FB40C9CDDBB0@YQBPR01MB0388.CANPRD01.PROD.OUTLOOK.COM>
References:  <CAEW%2BogZnWC07OCSuzO7E4TeYGr1E9BARKSKEh9ELCL9Zc4YY3w@mail.gmail.com> <C839431D-628C-4C73-8285-2360FE6FFE88@gmail.com> <CAEW%2BogYWKPL5jLW2H_UWEsCOiz=8fzFcSJ9S5k8k7FXMQjywsw@mail.gmail.com> <4f816be7-79e0-cacb-9502-5fbbe343cfc9@denninger.net> <3160F105-85C1-4CB4-AAD5-D16CF5D6143D@ifm.liu.se> <YQBPR01MB038805DBCCE94383219306E1DDB80@YQBPR01MB0388.CANPRD01.PROD.OUTLOOK.COM> <20181223113031.00005150@Leidinger.net> <YQBPR01MB038868AC3D6BAC5C6FB40C9CDDBB0@YQBPR01MB0388.CANPRD01.PROD.OUTLOOK.COM>

next in thread | previous in thread | raw e-mail | index | archive | help
[ regarding ZFS hardware thread ]

There's another type of server --- the "ghetto" or home storage serer.  For
this server, I like to optimize for not loosing data, not for uptime.

Going back a few years, there were consumer motherboards with 10 or 12 SATA
onboard.  Mostly, this was at the change of technologies ... so you had
some of one kind of port and some of another.  Used SAS HBAs are another
option ... but they have a caviat: many SATA drives will eventually reject
them under load.  Good SATA drives won't (but again, we're talking a ghetto
system).  If you're taking WD reds (and not, say, seagate barracudas) ...
these work well.  On the seagates, however, I've had drives repeatedly fail
... only to go on working fine in a workstation with a SATA controller.

Another source of ghetto ports are 4 port SATA controllers (~$50).  Cheaper
than SAS, but taking more PCIe ports.  I've had a 16 drive system running
on 10 motherboard ports and 2x 4 port SATA PCIe.  Watch out for the 8 port
ones --- the only 8 port SATA I've come accross are using a SATA expander
for the last 5 ports !?!.  Not a good value.

The motherboards with 10 or so ports tend to be high end boards, so a
ghetto system will want to get those on eBay used.  ASUS ROG and ASUS TUF
boards of the 4-ish generation (Intel) or the bulldozer (AMD) are
configured this way.  Having 10 + 8 gives you enough to have 2 SSD drives
for cache/log or 2 boot drives.

Get as much RAM as you can stuff in.  32G is often that number.

You'll get a good GigE on the motherboard, but 10GE will need to be an
addin card.  You'll likely be getting short on slots.

The next iteration of this hardware is the only one that will have a chance
at ECC RAM.  The cheapest threadripper motherboards with the basic
threadripper CPU will be in this range used soon ... giving you upto 128G
of ECC ram and many, many more PCIe lanes.  Might make me reverse my use of
SAS controllers and push me to all NAS drives.  To be clear, with SATA
controllers, cheap SATA drives seem to last fine.  But the SAS controllers
seem to spar with the cheap drive electronics (the drives smart test fine,
but the electronics disconnect --- and if the SAS controller is swapped for
a SATA one, this goes away).

Last point.  RAID-Z2 at a minimum.  I could even see the argument for Z3.
My current array is 16x 4T drvies in to 8 disk Z2 plexes.  Of that, one
plex is all WD Red on a SAS controller ... and the other (older) plex is
still largely cheap drives on SATA.  Right now, drives below 4T are
artificially expensive.  Drives right up to 10T are about the same price
per G (at least here in Canada).

This server (obviously) has many single points of failure.  Watch Netflix
when this happens :).  It doesn't fail often (I haven't had more than a few
hours downtime in 10-or-so-years) and the basic system can be had very
cheaply.  You don't need a recent CPU to fill GigE.  I'm about to upgrade
mine to 10GE ... but I realize that I may only get something less than 10GE
from it ... some fraction... but more than GigE nonetheless...



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CACpH0Md5y%2BSFTHbRL=OzP9joG60gKStOkoK3GrZqTYHO97k_FA>