From owner-freebsd-questions@FreeBSD.ORG Mon Jan 27 12:12:07 2014 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8F35F744 for ; Mon, 27 Jan 2014 12:12:07 +0000 (UTC) Received: from smtp.fagskolen.gjovik.no (smtp.fagskolen.gjovik.no [IPv6:2001:700:1100:1:200:ff:fe00:b]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id DFF1A1210 for ; Mon, 27 Jan 2014 12:12:06 +0000 (UTC) Received: from mail.fig.ol.no (localhost [127.0.0.1]) by mail.fig.ol.no (8.14.7/8.14.7) with ESMTP id s0RCC26K023583 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 27 Jan 2014 13:12:02 +0100 (CET) (envelope-from trond@fagskolen.gjovik.no) Received: from localhost (trond@localhost) by mail.fig.ol.no (8.14.7/8.14.7/Submit) with ESMTP id s0RCC2nk023580; Mon, 27 Jan 2014 13:12:02 +0100 (CET) (envelope-from trond@fagskolen.gjovik.no) X-Authentication-Warning: mail.fig.ol.no: trond owned process doing -bs Date: Mon, 27 Jan 2014 13:12:02 +0100 (CET) From: =?ISO-8859-1?Q?Trond_Endrest=F8l?= Sender: Trond.Endrestol@fagskolen.gjovik.no To: Kaya Saman Subject: Re: ZFS confusion In-Reply-To: <52E6463C.6090805@gmail.com> Message-ID: References: <52E40C82.7050302@gmail.com> <52E62DFF.3010600@gmail.com> <52E6463C.6090805@gmail.com> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) Organization: Fagskolen Innlandet OpenPGP: url=http://fig.ol.no/~trond/trond.key MIME-Version: 1.0 X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=unavailable version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on mail.fig.ol.no Content-Type: TEXT/PLAIN; charset=ISO-8859-1 Content-Transfer-Encoding: 8BIT X-Content-Filtered-By: Mailman/MimeDel 2.1.17 Cc: freebsd-questions X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Jan 2014 12:12:07 -0000 On Mon, 27 Jan 2014 11:42-0000, Kaya Saman wrote: > Many thanks everyone (Trond, Dennis, Steve)!! > > So RAIDz2 or 3 is going to be preferred per the advice given. > > Now I just need to figure out how to make that work best with my current 5 > block of disks.... perhaps wait for a while then add some more disks in the > mix then create the raidz(x) platform?? > > It would be really good if raidz could be expandable, ie by adding extra 'new' > disks into the same vdev. It's there! Try: zpool attach [new_member2 ...] > Anyhow, luckily I have multiple backups of everything in any case; just flying > around on loose disks though :-( > > > Regards, > > Kaya > > On 01/27/2014 11:20 AM, Trond Endrestøl wrote: > > On Mon, 27 Jan 2014 09:59-0000, Kaya Saman wrote: > > > > > Many thanks Trond for the response.... > > > > > > I was getting worried for a sec there :-) > > > > > > On 01/27/2014 09:08 AM, Trond Endrestøl wrote: > > > > On Sat, 25 Jan 2014 19:12-0000, Kaya Saman wrote: > > > > > > > > [...] > > > > No one's answered this, so I'll just give you my 2 cents. > > > > > > > > Triple parity means you're using storage capacity equivalent of three > > > > drives for parity alone. If you use five drives in total, this gives > > > > you 2 drives worth of real data and 3 drives worth of parity. In other > > > > words, you should really consider using a lot more drives when using > > > > triple parity, say nine drives. > > > So, basically if I have (just as silly example); 5x drives of 100MB > > > capacity, > > > this would equal 500MB in total. > > > > > > Meaning that only 200MB would be usable..... > > Also remember a couple of things: > > > > ZFS uses dynamically stripe widths, not fixed ones as HW RAID. A 37K > > file will occupy 37K, rounded up to the nearest integer of disk > > blocks, plus the space needed for parity and metadata. > > > > Without a dedicated ZIL, ZFS will also use some of the available space > > for housekeeping, e.g. ZIL. > > > > You shouldn't fill a pool in production completely with data, leaving > > no room for any COW operations. > > > > > Having just done a quick experiment: > > > > > > zpool create test_pool raidz3 /tmp/disk1 /tmp/disk2 /tmp/disk3 /tmp/disk4 > > > /tmp/disk5 > > > > > > I get: > > > > > > test_pool 476M 304K 476M 0% 1.00x ONLINE - > > > > > > then writing a 300M file to pool: > > > > > > dd if=/dev/zero of=/test_pool/file bs=300M count=1 > > > dd: /test_pool/file: No space left on device > > > > > > test_pool 476M 396M 79.8M 83% 1.00x ONLINE - > > > > > > du -sch /test_pool/file > > > 157M /test_pool/file > > > 157M total > > > > > > So I guess, now I have to ask; when building a disk pool, what's the best > > > option to go for between capacity and redundancy? > > If you value your data and your data grows, as it usually does, go for > > both. I hope you're adding some form of backup to the mix, as ZFS is > > no substitute for doing proper backups to independent media. > > > > > In my case I gradually want to expand my storage when I need it, so to > > > maximize capacity RAID0 would be the easiest. This doesn't give me any > > > redundancy however, unless I use RAID1+0 but then I loose physical > > > capacity > > > due to the mirrors. > > > > > > > > > Would it be better to create a raidz1 pool then just add raidz1 pools to > > > the > > > master as time goes by? > > > > > > As in: > > > > > > zpool create test_pool raidz1 /tmp/disk1 /tmp/disk2 /tmp/disk3 /tmp/disk4 > > > /tmp/disk5 > > > > > > > > > zpool add test_pool raidz1 /tmp/disk6 /tmp/disk7 /tmp/disk8 /tmp/disk9 > > > /tmp/disk10 > > > > > > > > > dd if=/dev/zero of=/test_pool/file bs=600M count=1 1+0 > > > records in > > > 1+0 records out > > > 629145600 bytes transferred in 190.069986 secs (3310073 bytes/sec) > > > > > > > > > Then running a 'zpool list' gets: > > > > > > test_pool 952M 751M 201M 78% 1.00x ONLINE - > > > > > > > > > So virtually out of: > > > > > > > > > NAME STATE READ WRITE CKSUM > > > test_pool ONLINE 0 0 0 > > > raidz1-0 ONLINE 0 0 0 > > > /tmp/disk1 ONLINE 0 0 0 > > > /tmp/disk2 ONLINE 0 0 0 > > > /tmp/disk3 ONLINE 0 0 0 > > > /tmp/disk4 ONLINE 0 0 0 > > > /tmp/disk5 ONLINE 0 0 0 > > > raidz1-1 ONLINE 0 0 0 > > > /tmp/disk6 ONLINE 0 0 0 > > > /tmp/disk7 ONLINE 0 0 0 > > > /tmp/disk8 ONLINE 0 0 0 > > > /tmp/disk9 ONLINE 0 0 0 > > > /tmp/disk10 ONLINE 0 0 0 > > > > > > errors: No known data errors > > > > > > > > > I have 8 disks usable with 2x disks for parity.... if my understanding is > > > correct. > > Correct. One "parity drive" evenly (or close to evenly) spread over > > the members in each raidz1 vdev. > > > > I just realized having multiple zraidN vdevs reduces the time needed > > for resilvering. Just don't make each vdev too small, i.e. with too > > few drives. > > > > As other's have told you, consider raidz2 or raidz3, as drives from > > the same batch and with equal wear and tear, might fail more or less > > simultaneously. -- +-------------------------------+------------------------------------+ | Vennlig hilsen, | Best regards, | | Trond Endrestøl, | Trond Endrestøl, | | IT-ansvarlig, | System administrator, | | Fagskolen Innlandet, | Gjøvik Technical College, Norway, | | tlf. mob. 952 62 567, | Cellular...: +47 952 62 567, | | sentralbord 61 14 54 00. | Switchboard: +47 61 14 54 00. | +-------------------------------+------------------------------------+