From owner-freebsd-fs@FreeBSD.ORG Mon Feb 6 17:14:40 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AE4F11065673 for ; Mon, 6 Feb 2012 17:14:40 +0000 (UTC) (envelope-from michael.aronsen@gmail.com) Received: from mail-wi0-f182.google.com (mail-wi0-f182.google.com [209.85.212.182]) by mx1.freebsd.org (Postfix) with ESMTP id 39AAD8FC14 for ; Mon, 6 Feb 2012 17:14:39 +0000 (UTC) Received: by wibhn14 with SMTP id hn14so6965291wib.13 for ; Mon, 06 Feb 2012 09:14:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=subject:mime-version:content-type:from:in-reply-to:date:cc :message-id:references:to:x-mailer; bh=M1ueNhsWgI5btNp2XuVsGRTYwYj/oo30cNkaujBopjk=; b=soJ1aXDaOl4l8sPLyg1VuA1t6coULA+dCWZQ9L6+W86YF6pEXQ9Rhe1dKTg+YzyWOZ +1P/7YAyXXv/Y1SUAmq7xGO4wkPg5USjpTb2GlL7K33ZnBXPtdJfC4HWqjEQD1XjOEEq EKzFGf4+MW0X34nldY5oVVxpbVxoKdHP/BZ9g= Received: by 10.180.86.198 with SMTP id r6mr14367227wiz.2.1328546971480; Mon, 06 Feb 2012 08:49:31 -0800 (PST) Received: from mbp.lan ([2.107.247.158]) by mx.google.com with ESMTPS id ex2sm48042252wib.1.2012.02.06.08.49.30 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 06 Feb 2012 08:49:31 -0800 (PST) Mime-Version: 1.0 (Apple Message framework v1251.1) From: Michael Aronsen In-Reply-To: <20120206162206.GA541@icarus.home.lan> Date: Mon, 6 Feb 2012 17:49:39 +0100 Message-Id: References: <4F2FF72B.6000509@pean.org> <20120206162206.GA541@icarus.home.lan> To: Jeremy Chadwick X-Mailer: Apple Mail (2.1251.1) Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-fs@freebsd.org Subject: Re: HPC and zfs. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 17:14:40 -0000 Hi, On Feb 6, 2012, at 17:22 , Jeremy Chadwick wrote: > - What single motherboard supports up to 192GB of RAM Get an HP DL580/585 - they support 2TB/1TB RAM. > - How you plan on getting roughly 410 hard disks (or 422 assuming > an additional 12 SSDs) hooked up to a single machine Use LSI SAS92XX 4 (x4) port external controllers, and SuperMicro = SC847E26-RJBOD1 disk shelves. Each disk shelf needs 2 ports on the LSI controller, which means you get = 90 disks per LSI card. The DL580/585's have 11 PCIe slots, so you'd end up with 990 disks per = server using this setup. >=20 > If you are considering investing the time and especially money (the = cost > here is almost unfathomable, IMO) into this, I strongly recommend you > consider an actual hardware filer (e.g. NetApp). Your performance and > reliability will be much greater, plus you will get overall better > support from NetApp in the case something goes wrong. In the case you > run into problems with FreeBSD (and I can assure you in this kind of > setup you will) with this kind of extensive setup, you will be at the > mercy of developers' time/schedules with absolutely no guarantee that > your problem will be solved. You definitely want a support contract. > Thus, go NetApp. We have NetApp's at our University for home storage, but I would = struggle to recommend them for HPC storage. A dedicated HPC filesystem such as Lustre or FhGFS = (http://www.fhgfs.com/cms/) will almost certainly give you better = performance as they're purpose made. We use FhGFS in a rather small setup (44 TB usable space and ~200 HPC = nodes), but they do have installations with 700TB+. The setup consists of 2 metadata nodes and 4 storage nodes, all = supermicro servers with 24 WD Velociraptor 600 GB 10K RPM disks. This setup gives us 4.8GB/sec write and 4.3GB/sec read speeds, all for a = lot less than a comparable NetApp solution (we paid around =8030.000). It now has support for mirroring on a per folder level for resilience. Currently it only runs on Linux but i'm considering a FreeBSD port to = get ZFS for volume management and now that OFED is in FreeBSD 9, = Infinifband is possible. I'd highly recommend a parallel filesystem, unfortunately not many, if = any, are available on FreeBSD at this time. Regards, Michael