From owner-freebsd-current@FreeBSD.ORG Sat Jan 24 16:45:08 2009 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E20BE106564A for ; Sat, 24 Jan 2009 16:45:08 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from wa-out-1112.google.com (wa-out-1112.google.com [209.85.146.183]) by mx1.freebsd.org (Postfix) with ESMTP id B6E228FC13 for ; Sat, 24 Jan 2009 16:45:08 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: by wa-out-1112.google.com with SMTP id k34so204158wah.27 for ; Sat, 24 Jan 2009 08:45:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=wBI5rCUbH2ckoggdfZKn4NsbhWfkrdUKMZtc5/iIzEI=; b=SIF0DvD+3eREe8my8FOc4Ams63CdpDpN/8P27nrGY02iXAMct4Wxel4vBf+WgK4Lw1 jYP7jPPnHqO2lWZoX8hYzlnackY5ZQjGaUDvpPkvp6mc08Ybhqi3OVZ0iYuFkwzBbBuy 4ls3oY+90UEHV9jDS/ECG9K7MeuQoJlsOXVEQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=lxtqtp/di7GQtkh+A3qFkQj2vk2wSZYiEtKP1/PKFeougSRv1RfUdbuexHvyRPC/zW 0cIAr213E2o+pQqS9O0ySVZpfMktmfKfTtFGX3dXhQLq8N9VooxPfxk5DUQxpz7QUbsM v+yAcmSklxSugaFFHw/YbXohVjitSDToTc0dk= MIME-Version: 1.0 Received: by 10.114.25.19 with SMTP id 19mr3270252way.89.1232813870132; Sat, 24 Jan 2009 08:17:50 -0800 (PST) In-Reply-To: <01N4NEOEB7LY00EQWX@tmk.com> References: <01N4NEOEB7LY00EQWX@tmk.com> Date: Sat, 24 Jan 2009 08:17:50 -0800 Message-ID: From: Freddie Cash To: Terry Kennedy Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: freebsd-current@freebsd.org Subject: Re: Help me select hardware and software options for very large server X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Jan 2009 16:45:09 -0000 On Fri, Jan 23, 2009 at 6:30 PM, Terry Kennedy wrote: > I'm in the process of planning the hardware and software for the second > generation of my RAIDzilla file servers (see http://www.tmk.com/raidzilla > for the current generation, in production for 4+ years). > > I expect that what I'm planning is probably "off the scale" in terms of > processing and storage capacity, and I'd like to find out and address any > issues before spending lots of money. Here's what I'm thinking of: > > o Chassis - CI Design SR316 (same model as current chassis, except i2c link > between RAID controller and front panel > o Motherboard - Intel S5000PSLSATAR > o CPU - 2x Intel Xeon E5450 BX80574E5450P > p Remote management - Intel Remote Management Module 2 - AXXRM2 > o Memory - 16GB - 8x Kingston KVR667D2D4F5/2GI > o RAID controller - 3Ware 9650SE-16ML w/ BBU-MODULE-04 > o Drives - 16x 2TB drives [not mentioning manufacturer yet] > o Cables - 4x multi-lane SATA cables > o DVD-ROM drive > o Auxiliary slot fan next to BBU card > o Adaptec AHA-39160 (for Quantum Superloader 3 tape drive) > > So much for the hardware. On the software front: > > o FreeBSD 8.x? > o amd64 architecture > o MBR+UFS2 for operating system partitions (hard partition in controller) > o GPT+ZFS for data partitions > o Multiple 8TB data partitions (separate 8TB controller partitions or one > big partition divided with GPT?) We did something similar for our off-site, automated backups box. Hardware (2 identical boxes): - Chenbro 5U chassis with 24 hot-swap drive bays - 1350 W 4-way redundant PSU - Tyan h2000M motherboard - 2x Opteron 2200-series CPUs (dual-core) @ 2.8 GHz - 8 GB DDR2-800 SDRAM - 3Ware 9650SE-12ML PCIe RAID controller - 3Ware 9550SX-12ML PCI-X RAID controller - 12x Seagate Barracude 7200.11 500 GB SATA drives - 12x WD 500 GB SATA drives - Intel Pro/1000MT 4-port gigabit NIC One box has 2x 2 GB CompactFlash in IDE adapters, the other has 2x 2 GB USB flash drives. Software: - 64-bit FreeBSD 7.1-RELEASE (started with 7.1-STABLE from August 08) - UFS for / partition, using gmirror across the two CF/USB drives - ZFS for everything else - uses rsync and ssh to backup 83 remote servers everynight, creating ZFS snapshots every night - uses rsync to transfer "snapshots" between the two servers The drives on each of the RAID controllers are configured as "Single Disk Array", so they appear as 24 separate drives to the OS, but still benefit from the controller's disk cache, management interface, and so on (as compared to JBOD where it acts like nothing more than a SATA controller). The drives on one box are configured as 1 large 24-drive raidz2 in ZFS (this box also has 12x 400 GB drives). The drives on the other box are configured as 2 separate 11-drive raidz2 arrays, with 2 hot spares. The usable space on box boxes is 9 TB. I'm experimenting with different ways to allocated the drives in ZFS, to get the best performance. The recommendations of the Solaris folks seems to be to use raidz arrays of fewer than 10 disks. Depending on the cost, for the next storage box, I may use 3x 3Ware 9550SX RAID controllers and use 3x raidz2 arrays of 8 disks each, to see how that compares. Other than a bit of kernel tuning back in August/September, these boxes have been running nice and smooth. Just waiting for either the release of FreeBSD 8.0 or an MFC of ZFS v13 to 7-STABLE to get support for auto-rebuild using hot spares. Also waiting for a pair of CF-to-SATA controllers to come in so that I can remove the USB flash drives from the one box. They're just too unreliable in my testing. The CF-to-IDE adapters work really well, but there's only a single IDE controller on the motherboard, so it's doing gmirror across master and slave devices on the same controller, which isn't the fastest way of doing things. -- Freddie Cash fjwcash@gmail.com