From owner-freebsd-questions@FreeBSD.ORG Sat Jan 25 19:12:08 2014 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 82473B82 for ; Sat, 25 Jan 2014 19:12:08 +0000 (UTC) Received: from mail-we0-x22a.google.com (mail-we0-x22a.google.com [IPv6:2a00:1450:400c:c03::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 17F931A83 for ; Sat, 25 Jan 2014 19:12:07 +0000 (UTC) Received: by mail-we0-f170.google.com with SMTP id u57so3933159wes.15 for ; Sat, 25 Jan 2014 11:12:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject :content-type:content-transfer-encoding; bh=51pc8M5XGr8svfWZ6R8DuXdkY2uOrfH9fL17dhLiO4M=; b=wki9KC9hzapk5P2uMPVzsis6ahfUl8cQgbs3sxIZ5oj+MnYiCk50XbevG20htw11Yh GoI4xENL45b+nC0Qc3vi28ZPB55Rbsot9UWHFfv4M/fxjwiRNpCysxooT1W8e2a/1aLj Y/qb3NpGv4zVu00Bjgypu6/BXdclIH/P4jjsMQH31W+PUbS6h3PrrvV5EhITAGQd5dG0 zyq8TanLOZVyIPQ8QJRVkFEa9EyTlQJnTmAHmN50ReVrhXymDMZq98+ru2+bcKj7SaWL izfw6HhoYvr4sznfjayWlinhkgn/+ivqjwFpYFBtN+VU/i+pdlzp7AjsGLQ9ebz8kytg i1qg== X-Received: by 10.180.207.10 with SMTP id ls10mr1773062wic.4.1390677126481; Sat, 25 Jan 2014 11:12:06 -0800 (PST) Received: from x220.optiplex-networks.com (81-178-2-118.dsl.pipex.com. [81.178.2.118]) by mx.google.com with ESMTPSA id f9sm16317405wib.3.2014.01.25.11.12.05 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sat, 25 Jan 2014 11:12:05 -0800 (PST) Message-ID: <52E40C82.7050302@gmail.com> Date: Sat, 25 Jan 2014 19:12:02 +0000 From: Kaya Saman User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 MIME-Version: 1.0 To: freebsd-questions Subject: ZFS confusion Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Jan 2014 19:12:08 -0000 Hi, I'm really confused about something so I hope someone can help me clear the fog up.... basically I'm about to setup a ZFS RAIDZ3 pool and having discovered this site: https://calomel.org/zfs_raid_speed_capacity.html as a reference for disk quantity got totally confused. Though in addition have checked out these sites too: https://blogs.oracle.com/ahl/entry/triple_parity_raid_z http://www.zfsbuild.com/2010/06/03/howto-create-raidz2-pool/ http://www.zfsbuild.com/2010/05/26/zfs-raid-levels/ http://www.linux.org/threads/zettabyte-file-system-zfs.4619/ Implementing a test ZFS pool on my old FreeBSD 8.3 box using dd derived vdevs coupled with reading the man page for zpool found that raidz3 needs a minimum of 4 disks to work. However, according to the above mentioned site for triple parity one should use 5 disks in 2+3 format. My confusion is this: does the 2+3 mean 2 disks in the pool with 3 hot spares or does it mean 5 disks in the pool? As in: zpool create raidz3 disk1 disk2 disk3 disk4 disk5 In addition to my testing I was looking at ease of expansion... ie. growing the pool, so is doing something like this: zpool create raidz3 disk1 disk2 disk3 disk4 Then when I needed to expand just do: zpool add raidz3 disk5 disk6 disk7 disk8 which gets: pool: testpool state: ONLINE status: The pool is formatted using a legacy on-disk format. The pool can still be used, but some features are unavailable. action: Upgrade the pool using 'zpool upgrade'. Once this is done, the pool will no longer be accessible on software that does not support feature flags. scan: none requested config: NAME STATE READ WRITE CKSUM testpool ONLINE 0 0 0 raidz3-0 ONLINE 0 0 0 /tmp/disk1 ONLINE 0 0 0 /tmp/disk2 ONLINE 0 0 0 /tmp/disk3 ONLINE 0 0 0 /tmp/disk4 ONLINE 0 0 0 raidz3-1 ONLINE 0 0 0 /tmp/disk5 ONLINE 0 0 0 /tmp/disk6 ONLINE 0 0 0 /tmp/disk7 ONLINE 0 0 0 /tmp/disk8 ONLINE 0 0 0 ---------- The same as this: ---------- pool: testpool state: ONLINE status: The pool is formatted using a legacy on-disk format. The pool can still be used, but some features are unavailable. action: Upgrade the pool using 'zpool upgrade'. Once this is done, the pool will no longer be accessible on software that does not support feature flags. scan: none requested config: NAME STATE READ WRITE CKSUM testpool ONLINE 0 0 0 raidz3-0 ONLINE 0 0 0 /tmp/disk1 ONLINE 0 0 0 /tmp/disk2 ONLINE 0 0 0 /tmp/disk3 ONLINE 0 0 0 /tmp/disk4 ONLINE 0 0 0 /tmp/disk5 ONLINE 0 0 0 /tmp/disk6 ONLINE 0 0 0 /tmp/disk7 ONLINE 0 0 0 /tmp/disk8 ONLINE 0 0 0 ?? Of course using the 1st method there is extra meta data involved but not too much especially with TB drives. Having created a zfs filesystem on top of both setups, the fs will grow over the 1st scenario to utilize disks 5 through 8 added later; while of course with the second setup the filesystem is already created over all 8 disks. In a real situation however, the above would certainly be 5 disks at a time to gain the triple parity, with ZIL and L2ARC on SSD's and hot swap spares. The reason am asking the above is that I've got a new enclosure with up to 26 disk capacity and need to create a stable environment and make best use of the space. So another words, maximum redundancy with max capacity allowed per method: which would be raidz1..3 and of course raidz3 offers the best redundancy but yet has much more capacity then a raid1+0 setup. My intention was to grab 5 disks to start with then expand as necessary plus 2 SSD's for ZIL+L2ARC using (raid0 mirroring and raid1 mirroring consecutively) and then 3x hot swap spares and use lz4 compression on the filesystem. With FreeBSD 10.0 as base OS... my current 8.3 must be EOL now though on a different box so no matter :-) Hopefully someone can help me understanding the above. Many thanks. Regards, Kaya