Date: Mon, 16 Jun 2014 00:53:40 -0400 From: Rich <rercola@pha.jhu.edu> To: Kevin Day <toasty@dragondata.com> Cc: freebsd-fs <freebsd-fs@freebsd.org>, Dennis Glatting <dg@pki2.com> Subject: Re: [Fwd: Re: Large ZFS arrays?] Message-ID: <CAOeNLupd8viLbobGwROA1tg2mQ0X4ghcEn=d91J73rJbgMGtQw@mail.gmail.com> In-Reply-To: <F071839B-ED6C-4515-B7C1-D7327CEF12B7@dragondata.com> References: <1402846984.4722.363.camel@btw.pki2.com> <F071839B-ED6C-4515-B7C1-D7327CEF12B7@dragondata.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sun, Jun 15, 2014 at 12:00 PM, Kevin Day <toasty@dragondata.com> wrote: > > On Jun 15, 2014, at 10:43 AM, Dennis Glatting <dg@pki2.com> wrote: >> >> Total. I am looking at three pieces in total: >> >> * Two 1PT storage "blocks" providing load sharing and >> mirroring for failover. >> >> * One 5PB storage block for on-line archives (3-5 years). >> >> The 1PB nodes will divided into something that makes sense, such as >> multiple SuperMicro 847 chassis with 3TB disks providing some number of >> volumes. Division is a function of application, such as a 100TB RAIDz2 >> volumes for bulk storage whereas smaller 8TB volumes for active data, >> such as iSCSI, databases, and home directories. >> >> Thanks. > > 2) You definitely want a JBOD controller instead of a smart RAID controll= er. The LSI 9207 works pretty well, but when you exceed 192 drives it compl= ains on boot up of running out of heap space and makes you press a key to c= ontinue, which then works fine. There is a very recently released firmware = update for the card that seems to fix this, but we haven't completed testin= g yet. You'll also want to increase hw.mps.max_chains. The driver warns you= when you need to, but you need to reboot to change this, and you're probab= ly only going to discover this under heavy load. You should be able to disable the boot ROM on the HBAs to avoid this problem? I had this happen too in various configs, but only when the controllers were configured to add the devices attached as boot options... - Rich
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOeNLupd8viLbobGwROA1tg2mQ0X4ghcEn=d91J73rJbgMGtQw>