From owner-freebsd-questions@FreeBSD.ORG Thu May 31 18:24:48 2012 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 8AD27106566B for ; Thu, 31 May 2012 18:24:48 +0000 (UTC) (envelope-from dg@pki2.com) Received: from btw.pki2.com (btw.pki2.com [IPv6:2001:470:a:6fd::2]) by mx1.freebsd.org (Postfix) with ESMTP id 3EB688FC08 for ; Thu, 31 May 2012 18:24:48 +0000 (UTC) Received: from btw.pki2.com (btw.pki2.com [192.168.23.1]) by btw.pki2.com (8.14.5/8.14.5) with ESMTP id q4VIOfCR050095 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Thu, 31 May 2012 11:24:43 -0700 (PDT) (envelope-from dg@pki2.com) Date: Thu, 31 May 2012 11:24:41 -0700 (PDT) From: Dennis Glatting X-X-Sender: dennisg@btw.pki2.com To: Oscar Hodgson In-Reply-To: Message-ID: References: User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-yoursite-MailScanner-Information: Dennis Glatting X-yoursite-MailScanner-ID: q4VIOfCR050095 X-yoursite-MailScanner: Found to be clean X-MailScanner-From: dg@pki2.com Cc: freebsd-questions@freebsd.org Subject: Re: Anyone using freebsd ZFS for large storage servers? X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 May 2012 18:24:48 -0000 On Thu, 31 May 2012, Oscar Hodgson wrote: > The subject is pretty much the question. Perhaps there's a better > place to be asking this question ... > > We have (very briefly) discussed the possibility of using FreeBSD > pizza boxes as a storage heads direct attached to external JBOD arrays > with ZFS. In perusing the list, I haven't stumbled across indications > of people actually doing this. External JBODs would be running 24 to > 48TB each, roughly. There would be a couple of units. The pizza > boxes would be used for computational tasks, and nominally would have > 8 cores and 96G+ RAM. > > Obvious questions are hardware compatibility and stability. I've set > up small FreeBSD 9 machines with ZFS roots and simple mirrors for > other tasks here, and those have been successful so far. > > Observations would be appreciated. > mc: real memory = 120259084288 (114688 MB) FreeBSD/SMP: Multiprocessor System Detected: 64 CPUs FreeBSD/SMP: 4 package(s) x 16 core(s) mc> zpool list NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT disk-1 14.5T 4.95T 9.55T 34% 1.00x ONLINE - disk-2 270G 297M 270G 0% 1.00x ONLINE - disk-1, RAIDz1, uses Hitachi 4TB drives. iirc: real memory = 68719476736 (65536 MB) FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs FreeBSD/SMP: 2 package(s) x 16 core(s) iirc> zpool list NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT disk-1 18.1T 6.70T 11.4T 36% 1.00x ONLINE - disk-2 5.44T 3.05G 5.43T 0% 1.00x ONLINE - disk-1, RAIDz1, uses a bunch of 2TB drives I have another storage server named bd3 that has a RAIDz2 array of 2.5T drives (11 of them, IIRC) but it is presently powered down for maintenance. btw: real memory = 25769803776 (24576 MB) FreeBSD/SMP: Multiprocessor System Detected: 12 CPUs FreeBSD/SMP: 1 package(s) x 6 core(s) x 2 SMT threads btw> zpool list NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT disk-1 9.06T 97.3G 8.97T 1% 1.00x ONLINE - disk-2 9.06T 5.13T 3.93T 56% 1.00x ONLINE - Those are smaller RAIDz1 arrays of 1TB and 2TB drives, IIRC. I also have three other systems, over clocked to 4GHz with 16GB of RAM and presently powered off, each with 3 or 4 2TB disks RAIDz1. None of these systems have external arrays. The storage systems use common technologies, such as NFS, to export their space but their primary mission is manipulating (sort-of) big data and crypto attacks, though one is being converted to a Hadoop node for experimentation. I have only had four issues over the past year and a half: 1) It is important to keep your ZFS patches up to date and the firmware in you controllers up to date. Failure to do this results in a => :( 2) Under heavy I/O my systems freeze for a few seconds. I haven't looked into why but they are completely unresponsive. Note I am also using compressed volumes (gzip), which puts a substantual load on the kernel. 3) I have had a number of disk failures -- not too many and not too few. These are merely an annoyance with no loss of data. 4) In two systems I use OCZ Revo drives. After several months of operating they go Tango Uniform, requiring a system boot where they return from the dead. None of my other SSD technologies exhibit the same problem.