From owner-freebsd-fs@FreeBSD.ORG Mon Jun 16 04:53:41 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 13725A03 for ; Mon, 16 Jun 2014 04:53:41 +0000 (UTC) Received: from mail-ie0-x22d.google.com (mail-ie0-x22d.google.com [IPv6:2607:f8b0:4001:c03::22d]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D705D2488 for ; Mon, 16 Jun 2014 04:53:40 +0000 (UTC) Received: by mail-ie0-f173.google.com with SMTP id y20so4578428ier.18 for ; Sun, 15 Jun 2014 21:53:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type:content-transfer-encoding; bh=V7oNtmqOoQhJlpEgKIs3d/iMNcPw0N4h5gTmcnZ7s38=; b=bpgY6yak1E/oWITg2uLw3vE1Y4DSrT50OuMiETjBXgYSVweppQKbJu7KR4hEp+bIaK pjBl/D0cMCmTpk2+C0dGX60xEJ2gSNOIiUZBCrD4ZO6bWxXwPsAPWKpG7IS0QNZqxa+O EioahZQzlZDKDcbtEewgchvCdshSTeUXsqgi4+0+55Vwz9N2b86Xff+Lfg038Nywvb7+ 0yaKMsIAUf45eC4Qh04DcCPJaXAey70bQ1gONb83VwM/Gq+0JIOAPLGlFesdjUHWNuHy pH+klpOZs0BFIsYocHuIbweoDGhowoAL7RnXrmctpCx8LAqxqkrqeYtnrED9EgDvDml1 Z8qw== MIME-Version: 1.0 X-Received: by 10.50.1.6 with SMTP id 6mr22928986igi.36.1402894420353; Sun, 15 Jun 2014 21:53:40 -0700 (PDT) Sender: rincebrain@gmail.com Received: by 10.64.225.73 with HTTP; Sun, 15 Jun 2014 21:53:40 -0700 (PDT) In-Reply-To: References: <1402846984.4722.363.camel@btw.pki2.com> Date: Mon, 16 Jun 2014 00:53:40 -0400 X-Google-Sender-Auth: MWtHiL8fVLtZXGxyufys19aYHn8 Message-ID: Subject: Re: [Fwd: Re: Large ZFS arrays?] From: Rich To: Kevin Day Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs , Dennis Glatting X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Jun 2014 04:53:41 -0000 On Sun, Jun 15, 2014 at 12:00 PM, Kevin Day wrote: > > On Jun 15, 2014, at 10:43 AM, Dennis Glatting wrote: >> >> Total. I am looking at three pieces in total: >> >> * Two 1PT storage "blocks" providing load sharing and >> mirroring for failover. >> >> * One 5PB storage block for on-line archives (3-5 years). >> >> The 1PB nodes will divided into something that makes sense, such as >> multiple SuperMicro 847 chassis with 3TB disks providing some number of >> volumes. Division is a function of application, such as a 100TB RAIDz2 >> volumes for bulk storage whereas smaller 8TB volumes for active data, >> such as iSCSI, databases, and home directories. >> >> Thanks. > > 2) You definitely want a JBOD controller instead of a smart RAID controll= er. The LSI 9207 works pretty well, but when you exceed 192 drives it compl= ains on boot up of running out of heap space and makes you press a key to c= ontinue, which then works fine. There is a very recently released firmware = update for the card that seems to fix this, but we haven't completed testin= g yet. You'll also want to increase hw.mps.max_chains. The driver warns you= when you need to, but you need to reboot to change this, and you're probab= ly only going to discover this under heavy load. You should be able to disable the boot ROM on the HBAs to avoid this problem? I had this happen too in various configs, but only when the controllers were configured to add the devices attached as boot options... - Rich