From owner-freebsd-fs@FreeBSD.ORG Mon Jul 7 11:06:58 2008 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id ACD18106567F for ; Mon, 7 Jul 2008 11:06:58 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 98B7E8FC0C for ; Mon, 7 Jul 2008 11:06:58 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id m67B6wUW062028 for ; Mon, 7 Jul 2008 11:06:58 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id m67B6wMk062024 for freebsd-fs@FreeBSD.org; Mon, 7 Jul 2008 11:06:58 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 7 Jul 2008 11:06:58 GMT Message-Id: <200807071106.m67B6wMk062024@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-fs@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Jul 2008 11:06:58 -0000 Current FreeBSD problem reports Critical problems Serious problems S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/93942 fs [vfs] [patch] panic: ufs_dirbad: bad dir (patch from D o kern/112658 fs [smbfs] [patch] smbfs and caching problems (resolves b o kern/114676 fs [ufs] snapshot creation panics: snapacct_ufs2: bad blo o kern/116170 fs [panic] Kernel panic when mounting /tmp o bin/121072 fs [smbfs] mount_smbfs(8) cannot normally convert the cha o bin/122172 fs [fs]: amd(8) automount daemon dies on 6.3-STABLE i386, o kern/122888 fs [zfs] zfs hang w/ prefetch on, zil off while running t 7 problems total. Non-critical problems S Tracker Resp. Description -------------------------------------------------------------------------------- o bin/113049 fs [patch] [request] make quot(8) use getopt(3) and show o bin/113838 fs [patch] [request] mount(8): add support for relative p o bin/114468 fs [patch] [request] add -d option to umount(8) to detach o kern/114847 fs [ntfs] [patch] [request] dirmask support for NTFS ala o kern/114955 fs [cd9660] [patch] [request] support for mask,dirmask,ui o bin/118249 fs mv(1): moving a directory changes its mtime o kern/124621 fs [ext3] Cannot mount ext2fs partition 7 problems total. From owner-freebsd-fs@FreeBSD.ORG Mon Jul 7 15:47:38 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 157BE1065681 for ; Mon, 7 Jul 2008 15:47:38 +0000 (UTC) (envelope-from brooks@lor.one-eyed-alien.net) Received: from lor.one-eyed-alien.net (cl-162.ewr-01.us.sixxs.net [IPv6:2001:4830:1200:a1::2]) by mx1.freebsd.org (Postfix) with ESMTP id 672EF8FC21 for ; Mon, 7 Jul 2008 15:47:37 +0000 (UTC) (envelope-from brooks@lor.one-eyed-alien.net) Received: from lor.one-eyed-alien.net (localhost [127.0.0.1]) by lor.one-eyed-alien.net (8.14.2/8.14.2) with ESMTP id m67Fm5PW057844; Mon, 7 Jul 2008 10:48:05 -0500 (CDT) (envelope-from brooks@lor.one-eyed-alien.net) Received: (from brooks@localhost) by lor.one-eyed-alien.net (8.14.2/8.14.2/Submit) id m67Fm5dK057843; Mon, 7 Jul 2008 10:48:05 -0500 (CDT) (envelope-from brooks) Date: Mon, 7 Jul 2008 10:48:05 -0500 From: Brooks Davis To: Carlos Luna Message-ID: <20080707154805.GA57420@lor.one-eyed-alien.net> References: <11c17ec30807050158t24c88491pe4407e01f6687d72@mail.gmail.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="W/nzBZO5zC0uMSeA" Content-Disposition: inline In-Reply-To: <11c17ec30807050158t24c88491pe4407e01f6687d72@mail.gmail.com> User-Agent: Mutt/1.5.17 (2007-11-01) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-3.0 (lor.one-eyed-alien.net [127.0.0.1]); Mon, 07 Jul 2008 10:48:05 -0500 (CDT) Cc: freebsd-fs@freebsd.org Subject: Re: Filesystem is not clean - run fsck X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Jul 2008 15:47:38 -0000 --W/nzBZO5zC0uMSeA Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Jul 05, 2008 at 10:58:33AM +0200, Carlos Luna wrote: > Hi I'd used freenas about 5 years without any problem. Now I can?t mount = my > raid volume and in his sourceforge forums seems they cant help me. Hope t= his > list is the right list for my issue. >=20 > When I try to fsck,I get: > casa:/dev# fsck -t ufs -y /dev/pst0s1 > ** /dev/pst0s1 > ** Last Mounted on /mnt/raid > ** Phase 1 - Check Blocks and Sizes > -4439300862985009506 BAD I=3D86 > 3443570138036206556 BAD I=3D86 > -7476842757969057647 BAD I=3D86 > -8078484667502176485 BAD I=3D86 > 2249916482063805839 BAD I=3D86 > -3291681609520367063 BAD I=3D86 > 7780434385339928353 BAD I=3D86 > -4372486048108189431 BAD I=3D86 > 8774078035736727371 BAD I=3D86 > -2035310265760485777 BAD I=3D86 > 6848295312539782814 BAD I=3D86 > EXCESSIVE BAD BLKS I=3D86 > CONTINUE? yes >=20 > ... > .... >=20 > UNKNOWN FILE TYPE I=3D7254140 > CLEAR? yes >=20 > UNKNOWN FILE TYPE I=3D7254141 > CLEAR? yes >=20 > UNKNOWN FILE TYPE I=3D7254142 > CLEAR? yes >=20 > UNKNOWN FILE TYPE I=3D7254143 > CLEAR? yes >=20 > fsck_ufs: cannot alloc 3037795832 bytes for inoinfo > I have a lot of info there, 1 TB. I will appreciate any help. It looks like you have a somewhat large file system, apparently with a lot of small files on it. The message indicates that you need to be able to allocate over 3GB of address space to handle this. As such you will need a 64-bit machine, ideally with 4GB or more RAM and probably with a large swap partition. In theory it should be possible to write a constrained memory use version of fsck, but to my knowledge no one has done so and I suspect it would be a time consuming development effort. -- Brooks --W/nzBZO5zC0uMSeA Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (FreeBSD) iD8DBQFIcjq0XY6L6fI4GtQRAunhAJ4zpVxurAB/+CV99qRfUv0P/grWMQCg2A3A lATWmGGl9VKCDBlpx6OMbCM= =IFI2 -----END PGP SIGNATURE----- --W/nzBZO5zC0uMSeA-- From owner-freebsd-fs@FreeBSD.ORG Mon Jul 7 17:40:25 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E299710656BE for ; Mon, 7 Jul 2008 17:40:25 +0000 (UTC) (envelope-from hartzell@alerce.com) Received: from merlin.alerce.com (merlin.alerce.com [64.62.142.94]) by mx1.freebsd.org (Postfix) with ESMTP id 40E038FC16 for ; Mon, 7 Jul 2008 17:40:25 +0000 (UTC) (envelope-from hartzell@alerce.com) Received: from merlin.alerce.com (localhost [127.0.0.1]) by merlin.alerce.com (Postfix) with ESMTP id 88B5633C62 for ; Mon, 7 Jul 2008 10:19:13 -0700 (PDT) Received: from postfix.alerce.com (w092.z064001164.sjc-ca.dsl.cnc.net [64.1.164.92]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by merlin.alerce.com (Postfix) with ESMTP id 2BAFC33C5B for ; Mon, 7 Jul 2008 10:19:13 -0700 (PDT) Received: by postfix.alerce.com (Postfix, from userid 501) id 9C84C465668; Mon, 7 Jul 2008 10:18:52 -0700 (PDT) From: George Hartzell MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <18546.20476.590665.29995@almost.alerce.com> Date: Mon, 7 Jul 2008 10:18:52 -0700 To: freebsd-fs@freebsd.org X-Mailer: VM 7.19 under Emacs 22.1.50.1 X-Virus-Scanned: ClamAV using ClamSMTP Subject: using zfs and unionfs together, does zfs need to be extended? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: hartzell@alerce.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Jul 2008 17:40:26 -0000 I'd like to be able to set up a large-ish number of very similar jails, with a minimum of fuss and take advantage of zfs' cool features. I'd like to use unionfs to do this, but zfs' lack of whiteout support seems to make it impossible. [jump to the bottom if you want to skip the setup and get to the questions] It seems like the most popular way to set up jails these days uses read-only nullfs mounts of a base system and symbolic links into a read-write nullfs mount for each jail's specific stuff (etc, /usr/local, etc...). These approaches are well described in: http://erdgeist.org/arts/software/ezjail http://www.freebsd.org/doc/en/books/handbook/jails-application.html and they work fine with zfs based storage. It's also possible to use unionfs to layer jail-specific storage over a base system. While this approach gives more per-jail flexibility and avoids having to relocate various directories in the base system, various unionfs problems seem to have pushed it out of favor. The ongoing work of daichi@freebsd.org et al. that fixes various problems with unionfs, http://people.freebsd.org/~daichi/unionfs/ makes it look as if this approach might be now be safe, using something like: mount -t unionfs -o below,noatime /usr/jails/base /usr/jails/www The obvious zfs analog to this: mount -t unionfs -o below,noatime /tank/jails/base /tank/jails/www fails with: mount_unionfs: /tank/jails/www: Operation not supported A bit of digging suggests that the mount fails when the unionfs code checks to see if /tank/jails/www supports whiteouts. The fact that this check doesn't occur if the uniondir is read-only provides a way to superficially check if whiteouts are the only problem, this: mount -t unionfs -o ro,below,noatime /tank/jails/base /tank/jails/www does indeed seem to lead to a working [albeit read-only] union mount. One can work around the problem by creating a ZFS volume, building a UFS filesystem on it, and then using that as the uniondir, e.g.: zfs create -V 5G tank/jail/vol1 newfs /dev/zvol/tank/jail/vol1 mkdir /usr/jail/zvol-www mount /dev/zvol/tank/jail/vol1 /usr/jail/zvol-www/ mount -t unionfs -o below,noatime /tank/jail/base/ /usr/jail/zvol-www The upper layer is still [presumably, I haven't tested these yet] snapshot-able, send-able, etc.... but this approach leaves me with a bunch of UFS filesystems that need care and feeding (fsck, etc...). So finally, the question: How hard would it be to add whiteout support to our ZFS? Is it "just" a matter of understanding the places in the UFS code that do whiteout things, locating the analogous places in the ZFS tree and doing similar things (it seems to be a "simple" matter of creating/destroying a whiteout vnode when necessary and checking for it when appropriate) or is there something fundamentally harder about it? Has anyone already done it? If it were doable/done cleanly, might it get committed? Thanks, g. From owner-freebsd-fs@FreeBSD.ORG Mon Jul 7 17:46:20 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2AF091065683 for ; Mon, 7 Jul 2008 17:46:20 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 766318FC1D; Mon, 7 Jul 2008 17:46:19 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <4872566C.6000206@FreeBSD.org> Date: Mon, 07 Jul 2008 19:46:20 +0200 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.14 (Macintosh/20080421) MIME-Version: 1.0 To: hartzell@alerce.com References: <18546.20476.590665.29995@almost.alerce.com> In-Reply-To: <18546.20476.590665.29995@almost.alerce.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: using zfs and unionfs together, does zfs need to be extended? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Jul 2008 17:46:20 -0000 George Hartzell wrote: > I'd like to be able to set up a large-ish number of very similar > jails, with a minimum of fuss and take advantage of zfs' cool > features. I'd like to use unionfs to do this, but zfs' lack of > whiteout support seems to make it impossible. [jump to the bottom if > you want to skip the setup and get to the questions] > > It seems like the most popular way to set up jails these days uses > read-only nullfs mounts of a base system and symbolic links into a > read-write nullfs mount for each jail's specific stuff (etc, > /usr/local, etc...). The "ZFS way" is to just clone your jail filesystem into each jail instance. Kris From owner-freebsd-fs@FreeBSD.ORG Mon Jul 7 17:59:25 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5B811106567B; Mon, 7 Jul 2008 17:59:25 +0000 (UTC) (envelope-from hartzell@alerce.com) Received: from merlin.alerce.com (merlin.alerce.com [64.62.142.94]) by mx1.freebsd.org (Postfix) with ESMTP id 3FDCC8FC17; Mon, 7 Jul 2008 17:59:25 +0000 (UTC) (envelope-from hartzell@alerce.com) Received: from merlin.alerce.com (localhost [127.0.0.1]) by merlin.alerce.com (Postfix) with ESMTP id 7F03833C62; Mon, 7 Jul 2008 10:59:44 -0700 (PDT) Received: from postfix.alerce.com (w092.z064001164.sjc-ca.dsl.cnc.net [64.1.164.92]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by merlin.alerce.com (Postfix) with ESMTP id 4F01B33C5B; Mon, 7 Jul 2008 10:59:44 -0700 (PDT) Received: by postfix.alerce.com (Postfix, from userid 501) id 3E8934656E1; Mon, 7 Jul 2008 10:59:24 -0700 (PDT) From: George Hartzell MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <18546.22908.193997.709865@almost.alerce.com> Date: Mon, 7 Jul 2008 10:59:24 -0700 To: Kris Kennaway In-Reply-To: <4872566C.6000206@FreeBSD.org> References: <18546.20476.590665.29995@almost.alerce.com> <4872566C.6000206@FreeBSD.org> X-Mailer: VM 7.19 under Emacs 22.1.50.1 X-Virus-Scanned: ClamAV using ClamSMTP Cc: freebsd-fs@freebsd.org Subject: Re: using zfs and unionfs together, does zfs need to be extended? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: hartzell@alerce.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Jul 2008 17:59:25 -0000 Kris Kennaway writes: > George Hartzell wrote: > > I'd like to be able to set up a large-ish number of very similar > > jails, with a minimum of fuss and take advantage of zfs' cool > > features. I'd like to use unionfs to do this, but zfs' lack of > > whiteout support seems to make it impossible. [jump to the bottom if > > you want to skip the setup and get to the questions] > > > > It seems like the most popular way to set up jails these days uses > > read-only nullfs mounts of a base system and symbolic links into a > > read-write nullfs mount for each jail's specific stuff (etc, > > /usr/local, etc...). > > The "ZFS way" is to just clone your jail filesystem into each jail instance. Both the nullfs approach used by ezjail and described in the handbook and the unionfs approach make updates *much* easier. A change/update to the jail base is automatically visible in all of the jails. As I understand a zfs clones (and a quick test backs this up), they're copies of the original filesystem, based on a snapshot. Once they're cloned they no longer "see" updates to the base system. I'm not even sure that you get the space savings, I just did a zfs snapshot and then a zfs clone and du -sH on the two filesystems reports the same size. That seems odd though (with all the copy on write stuff available), but.... g. From owner-freebsd-fs@FreeBSD.ORG Mon Jul 7 18:10:34 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 58F2D1065673 for ; Mon, 7 Jul 2008 18:10:34 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id E91DA8FC22; Mon, 7 Jul 2008 18:10:32 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <48725C19.5040408@FreeBSD.org> Date: Mon, 07 Jul 2008 20:10:33 +0200 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.14 (Macintosh/20080421) MIME-Version: 1.0 To: hartzell@alerce.com References: <18546.20476.590665.29995@almost.alerce.com> <4872566C.6000206@FreeBSD.org> <18546.22908.193997.709865@almost.alerce.com> In-Reply-To: <18546.22908.193997.709865@almost.alerce.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: using zfs and unionfs together, does zfs need to be extended? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Jul 2008 18:10:34 -0000 George Hartzell wrote: > Kris Kennaway writes: > > George Hartzell wrote: > > > I'd like to be able to set up a large-ish number of very similar > > > jails, with a minimum of fuss and take advantage of zfs' cool > > > features. I'd like to use unionfs to do this, but zfs' lack of > > > whiteout support seems to make it impossible. [jump to the bottom if > > > you want to skip the setup and get to the questions] > > > > > > It seems like the most popular way to set up jails these days uses > > > read-only nullfs mounts of a base system and symbolic links into a > > > read-write nullfs mount for each jail's specific stuff (etc, > > > /usr/local, etc...). > > > > The "ZFS way" is to just clone your jail filesystem into each jail instance. > > Both the nullfs approach used by ezjail and described in the handbook > and the unionfs approach make updates *much* easier. A change/update > to the jail base is automatically visible in all of the jails. > > As I understand a zfs clones (and a quick test backs this up), they're > copies of the original filesystem, based on a snapshot. Once they're > cloned they no longer "see" updates to the base system. That's right. Keep in mind that depending on what you are changing, it can be dangerous to modify files that are in use. Anyway if you require this model then nullfs or unionfs is indeed required. > I'm not even sure that you get the space savings, I just did a zfs > snapshot and then a zfs clone and du -sH on the two filesystems > reports the same size. That seems odd though (with all the copy on > write stuff available), but.... They are copy-on-write, so if you interpret the data correctly you will see that they aren't using additional space until you write to them :) Kris From owner-freebsd-fs@FreeBSD.ORG Tue Jul 8 03:25:13 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 167B8106564A for ; Tue, 8 Jul 2008 03:25:13 +0000 (UTC) (envelope-from john@kozubik.com) Received: from kozubik.com (kozubik.com [69.43.165.2]) by mx1.freebsd.org (Postfix) with ESMTP id F0B008FC14 for ; Tue, 8 Jul 2008 03:25:12 +0000 (UTC) (envelope-from john@kozubik.com) Received: from kozubik.com (localhost [127.0.0.1]) by kozubik.com (8.12.11/8.12.11) with ESMTP id m683nP4b079302; Mon, 7 Jul 2008 20:49:25 -0700 (PDT) (envelope-from john@kozubik.com) Received: from localhost (john@localhost) by kozubik.com (8.12.11/8.12.11/Submit) with ESMTP id m683nMjg079299; Mon, 7 Jul 2008 20:49:23 -0700 (PDT) (envelope-from john@kozubik.com) Date: Mon, 7 Jul 2008 20:49:22 -0700 (PDT) From: John Kozubik To: Wes Morgan In-Reply-To: Message-ID: <20080707203314.T1807@kozubik.com> References: <20080701213006.37D675B4B@mail.bitblocks.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Cc: freebsd-fs@freebsd.org, ticso@cicely.de Subject: Re: It's 2008. 1 TB disk drives cost $160. Quotas are 32-bit. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jul 2008 03:25:13 -0000 On Tue, 1 Jul 2008, Wes Morgan wrote: > > It is not the same but can serve a similer purpose if each > > user gets his own filesystem (and yes, I am aware of the > > rebooting issue with zfs with thousands of filesystems). He > > wanted support for 2TB+ quota on ufs by July 20. If that > > doesn't happen at least he can limp along with this. > > On a totally spurrious note, I'd love to know the storage environment > where a 1 TB quota on a multi-user system is meaningful. If I truly need > that much space as a user, and I hit your quota limit, I'll probably be a > very, very unhappy user! No, you'd be a paying customer. The environment is rsync.net. Users pay a monthly fee for X GB of storage. Some users require more than 2200 GB. It makes me very happy to run a modern enterprise with basic unix tools and methodologies. It's nice to imagine that all manner of normal folks out in the world, in 2008, are being served by the same logic and philosophies that evolved in the days of true multi-user shared unix systems. That is, if those core functions still worked. ----- John Kozubik - john@kozubik.com - http://www.kozubik.com From owner-freebsd-fs@FreeBSD.ORG Tue Jul 8 03:28:39 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D6BAF106567A for ; Tue, 8 Jul 2008 03:28:39 +0000 (UTC) (envelope-from john@kozubik.com) Received: from kozubik.com (kozubik.com [69.43.165.2]) by mx1.freebsd.org (Postfix) with ESMTP id BEF678FC16 for ; Tue, 8 Jul 2008 03:28:39 +0000 (UTC) (envelope-from john@kozubik.com) Received: from kozubik.com (localhost [127.0.0.1]) by kozubik.com (8.12.11/8.12.11) with ESMTP id m683r29G079345; Mon, 7 Jul 2008 20:53:02 -0700 (PDT) (envelope-from john@kozubik.com) Received: from localhost (john@localhost) by kozubik.com (8.12.11/8.12.11/Submit) with ESMTP id m683r1E7079342; Mon, 7 Jul 2008 20:53:01 -0700 (PDT) (envelope-from john@kozubik.com) Date: Mon, 7 Jul 2008 20:53:01 -0700 (PDT) From: John Kozubik To: Bakul Shah In-Reply-To: <20080701175932.0B76F5B4B@mail.bitblocks.com> Message-ID: <20080707204943.D1807@kozubik.com> References: <20080701175932.0B76F5B4B@mail.bitblocks.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Cc: freebsd-fs@freebsd.org Subject: Re: It's 2008. 1 TB disk drives cost $160. Quotas are 32-bit. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jul 2008 03:28:39 -0000 On Tue, 1 Jul 2008, Bakul Shah wrote: > To bring this back on topic, perhaps John Kozubik can just > use the zfs since it already has quota support? For example, > > # zfs create z/foo > # zfs quota=10M z/foo > dd < /dev/zero bs=1M count=20 > /z/foo/xx > dd: stdout: Disc quota exceeded > 11+0 records in > 10+0 records out > 10485760 bytes transferred in 4.718700 secs (2222171 bytes/sec) > # zfs set quota=10T z/foo > # zfs get quota z/foo > NAME PROPERTY VALUE SOURCE > z/foo quota 10T local Thanks - I appreciate this, and am continually impressed by the zfs work being done on FreeBSD. However, ZFS on FreeBSD is still experimental, and given the environment that I am deploying in (see previous post) it is impossible to consider it. ----- John Kozubik - john@kozubik.com - http://www.kozubik.com From owner-freebsd-fs@FreeBSD.ORG Tue Jul 8 06:42:06 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A9A051065692 for ; Tue, 8 Jul 2008 06:42:06 +0000 (UTC) (envelope-from mike503@gmail.com) Received: from py-out-1112.google.com (py-out-1112.google.com [64.233.166.177]) by mx1.freebsd.org (Postfix) with ESMTP id 6907A8FC23 for ; Tue, 8 Jul 2008 06:42:06 +0000 (UTC) (envelope-from mike503@gmail.com) Received: by py-out-1112.google.com with SMTP id p76so1114141pyb.10 for ; Mon, 07 Jul 2008 23:42:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:mime-version:content-type:content-transfer-encoding :content-disposition; bh=VEMAj/DjO1G9PenkPggpuk4NAx8CtANGjtmOUSzL/vE=; b=hSzXyUfsgp1AnAiWDFKoO35Ql9y5CT2/rRMlEsnFy8em1RIOxJw9z0duUWweWUdVSd 6NBNNkJgZzxI/RAN/OJHlChzv9cRSYARsMiIYGZHiDQNg24BWgvdY6K6N25nFYdgwOta LLsf8W5UAyI5nI402ZW+z5+PEwSXijVnbSDhk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:mime-version:content-type :content-transfer-encoding:content-disposition; b=S02h6NQGWSoGmpA2BUFTsB2IOVLqHZxEbzFDwrcycuIT8Q50JntcwaUEmWyHVRFTPR hre12mAP/4j3vO2khO5dTgTDadjFA4AWSAR92zTUiEppbyNySwRDU1/+Id4SfgNejaVA IpbyrrQCzewDIuIT9OYlyG7DvNSaA5x33gRFM= Received: by 10.142.11.2 with SMTP id 2mr1660206wfk.29.1215497754114; Mon, 07 Jul 2008 23:15:54 -0700 (PDT) Received: by 10.142.102.11 with HTTP; Mon, 7 Jul 2008 23:15:54 -0700 (PDT) Message-ID: Date: Mon, 7 Jul 2008 23:15:54 -0700 From: mike To: freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Content-Disposition: inline Subject: Thinking of using ZFS/FBSD for a backup system X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jul 2008 06:42:06 -0000 I administer a handful of servers - both for work and for my side business. Right now I am rsyncing /home each server back each night from the server to my own machine at home. I'd like to add in snapshots, so I want to sanity check here - I wouldn't be doing much more than: - creating a separate zfs filesystem for each server - creating a nightly snapshot after the rsync finishes Since I cannot change the filesystems on the remote machines now (and all run Linux anyway), this essentially gives me the ability to have daily snapshots of each machine at my fingertips should I need it - correct? Just wanted to sanity check here before investing some money and time into this solution... Also if anyone wants to reply to me off list with hardware that works well for FBSD 7 + ZFS I'd be grateful :) Thanks in advance. From owner-freebsd-fs@FreeBSD.ORG Tue Jul 8 08:23:48 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B3AC51065671 for ; Tue, 8 Jul 2008 08:23:48 +0000 (UTC) (envelope-from phoemix@harmless.hu) Received: from marvin.harmless.hu (marvin.harmless.hu [195.56.55.204]) by mx1.freebsd.org (Postfix) with ESMTP id 6FFB48FC15 for ; Tue, 8 Jul 2008 08:23:48 +0000 (UTC) (envelope-from phoemix@harmless.hu) Received: from fw.publishing.hu ([82.131.181.62] helo=twoflower.in.publishing.hu) by marvin.harmless.hu with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.69 (FreeBSD)) (envelope-from ) id 1KG8E2-0001xX-Fb; Tue, 08 Jul 2008 10:07:02 +0200 Date: Tue, 8 Jul 2008 10:07:01 +0200 From: CZUCZY Gergely To: mike Message-ID: <20080708100701.57031cda@twoflower.in.publishing.hu> In-Reply-To: References: Organization: Harmless Digital X-Mailer: Claws Mail 3.5.0 (GTK+ 2.12.11; i386-portbld-freebsd6.3) Mime-Version: 1.0 Content-Type: multipart/signed; boundary="Sig_/bR7m6HPa7YP/lF_wrVExvqe"; protocol="application/pgp-signature"; micalg=PGP-SHA1 Cc: freebsd-fs@freebsd.org Subject: Re: Thinking of using ZFS/FBSD for a backup system X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jul 2008 08:23:48 -0000 --Sig_/bR7m6HPa7YP/lF_wrVExvqe Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable I've already made a backup system exactly in the scheme you've just describ= ed, in order to replace dirvish. It worked quite well, but the ZFS port was so experimental that we couldn't go on. No matter how much i've tried to finetune ZFS it kept in randomly rebooting= in every 1-2-3 weeks, and a few backup cycles were lost. Regardless of this, the system worked quite well. If ZFS were stable, this easily could be our backup system. ZFS is great, awesome, but a bit unrelia= ble on FreeBSD, still needs some work. On Mon, 7 Jul 2008 23:15:54 -0700 mike wrote: > I administer a handful of servers - both for work and for my side > business. Right now I am rsyncing /home each server back each night > from the server to my own machine at home. >=20 > I'd like to add in snapshots, so I want to sanity check here - I > wouldn't be doing much more than: >=20 > - creating a separate zfs filesystem for each server > - creating a nightly snapshot after the rsync finishes >=20 > Since I cannot change the filesystems on the remote machines now (and > all run Linux anyway), this essentially gives me the ability to have > daily snapshots of each machine at my fingertips should I need it - > correct? >=20 > Just wanted to sanity check here before investing some money and time > into this solution... >=20 > Also if anyone wants to reply to me off list with hardware that works > well for FBSD 7 + ZFS I'd be grateful :) >=20 > Thanks in advance. > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" --=20 =C3=9Cdv=C3=B6lettel, Czuczy Gergely Harmless Digital Bt mailto: gergely.czuczy@harmless.hu Tel: +36-30-9702963 --Sig_/bR7m6HPa7YP/lF_wrVExvqe Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.3 (FreeBSD) iD8DBQFIcyAlzrC0WyuMkpsRAqhyAJ9Yoa9QR9/MXlrHYQ3B2IDHP/wC1ACff3Gu fi2GC29BnGTddlbZGevOKmY= =idKx -----END PGP SIGNATURE----- --Sig_/bR7m6HPa7YP/lF_wrVExvqe-- From owner-freebsd-fs@FreeBSD.ORG Tue Jul 8 08:31:28 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EE4141065677 for ; Tue, 8 Jul 2008 08:31:28 +0000 (UTC) (envelope-from mike503@gmail.com) Received: from yw-out-2324.google.com (yw-out-2324.google.com [74.125.46.29]) by mx1.freebsd.org (Postfix) with ESMTP id A51998FC18 for ; Tue, 8 Jul 2008 08:31:28 +0000 (UTC) (envelope-from mike503@gmail.com) Received: by yw-out-2324.google.com with SMTP id 9so1033608ywe.13 for ; Tue, 08 Jul 2008 01:31:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=95k2k/KMZp3ufjvmPg0Q1CrgLH5LnrCNJJ9fPfDMblE=; b=kQ9DLZYUoYQIYlmB12I/zCUkAM+GUehRCaYp/Z05PEgZdtRHG901PESSW8Mqj7wPZv F6ORWf7WZdBbXPjBxOs5DR2e/l1xle6AoMZcMX9gmcIZP0VXjE4vtf8STkjy/z76uMPa wqLKC18U9iJiE6rsKUr2adlH4p7Ib3O5HUixs= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=dL1lG8yBQd/pJZ+f0vySf14tPiLo6thEZUtauBr398d8qge2Sry6K9uEptsYtu26zg t9FkS3R9Hh8UlZpJHxc1Mu7MgDvX7I1wYos3fV3vhoZXYNsXmXqBGZ6v4YgRYCf4Py62 d6d92d+psqf1ZeT9uJU+j2hHgCcvPlmsMvWtM= Received: by 10.142.212.21 with SMTP id k21mr1698049wfg.266.1215505885461; Tue, 08 Jul 2008 01:31:25 -0700 (PDT) Received: by 10.142.102.11 with HTTP; Tue, 8 Jul 2008 01:31:25 -0700 (PDT) Message-ID: Date: Tue, 8 Jul 2008 01:31:25 -0700 From: mike To: "CZUCZY Gergely" In-Reply-To: <20080708100701.57031cda@twoflower.in.publishing.hu> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <20080708100701.57031cda@twoflower.in.publishing.hu> Cc: freebsd-fs@freebsd.org Subject: Re: Thinking of using ZFS/FBSD for a backup system X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jul 2008 08:31:29 -0000 On 7/8/08, CZUCZY Gergely wrote: > Regardless of this, the system worked quite well. If ZFS were stable, this > easily could be our backup system. ZFS is great, awesome, but a bit unreliable > on FreeBSD, still needs some work. Really? I thought ZFS for basic things was not too bad in FBSD now. By basic I mean simple filesystem creation, snapshots and normal devices. Not some crazy SAN LUNs and weird volume management stuff. I would really love to use FBSD as opposed to a Solaris derivative, since I know nothing about them and I'd have to dedicate a machine for it at home. Hrm. I wonder if I could just get by running a Solaris derivative inside of a VM in VMware or something. From owner-freebsd-fs@FreeBSD.ORG Tue Jul 8 08:34:41 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D396C1065671 for ; Tue, 8 Jul 2008 08:34:41 +0000 (UTC) (envelope-from phoemix@harmless.hu) Received: from marvin.harmless.hu (marvin.harmless.hu [195.56.55.204]) by mx1.freebsd.org (Postfix) with ESMTP id 8D62C8FC20 for ; Tue, 8 Jul 2008 08:34:41 +0000 (UTC) (envelope-from phoemix@harmless.hu) Received: from fw.publishing.hu ([82.131.181.62] helo=twoflower.in.publishing.hu) by marvin.harmless.hu with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.69 (FreeBSD)) (envelope-from ) id 1KG8em-0002Xe-92; Tue, 08 Jul 2008 10:34:40 +0200 Date: Tue, 8 Jul 2008 10:34:37 +0200 From: CZUCZY Gergely To: mike Message-ID: <20080708103437.553af3fb@twoflower.in.publishing.hu> In-Reply-To: References: <20080708100701.57031cda@twoflower.in.publishing.hu> Organization: Harmless Digital X-Mailer: Claws Mail 3.5.0 (GTK+ 2.12.11; i386-portbld-freebsd6.3) Mime-Version: 1.0 Content-Type: multipart/signed; boundary="Sig_/pYI3C4R6H+uUIwRHhliBjAt"; protocol="application/pgp-signature"; micalg=PGP-SHA1 Cc: freebsd-fs@freebsd.org Subject: Re: Thinking of using ZFS/FBSD for a backup system X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jul 2008 08:34:41 -0000 --Sig_/pYI3C4R6H+uUIwRHhliBjAt Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable This is the kmem_size overstep issue I'm mostly talking about. No matter how much you tune your system, the chance of a kernel panic due to kmem_size is= too small remains. And the time will come, and you will have a random reboot du= ring the backup procedure. It ofcourse happens when ZFS is in use :) So, in my humble opinion you're better off with (open)solaris for now. There were some posts on a @freebsd mailing list about making zfs more stab= le on amd64 by some VM patching, i don't quite remember the details... On Tue, 8 Jul 2008 01:31:25 -0700 mike wrote: > On 7/8/08, CZUCZY Gergely wrote: >=20 > > Regardless of this, the system worked quite well. If ZFS were stable, t= his > > easily could be our backup system. ZFS is great, awesome, but a bit > > unreliable on FreeBSD, still needs some work. >=20 > Really? I thought ZFS for basic things was not too bad in FBSD now. >=20 > By basic I mean simple filesystem creation, snapshots and normal > devices. Not some crazy SAN LUNs and weird volume management stuff. >=20 > I would really love to use FBSD as opposed to a Solaris derivative, > since I know nothing about them and I'd have to dedicate a machine for > it at home. Hrm. I wonder if I could just get by running a Solaris > derivative inside of a VM in VMware or something. --=20 =C3=9Cdv=C3=B6lettel, Czuczy Gergely Harmless Digital Bt mailto: gergely.czuczy@harmless.hu Tel: +36-30-9702963 --Sig_/pYI3C4R6H+uUIwRHhliBjAt Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.3 (FreeBSD) iD8DBQFIcyafzrC0WyuMkpsRAq0TAJwNboDQuuD3cUMuPcvuS2CM+78UewCdH3ae cc63u92IltmcusXsZafH8VQ= =112C -----END PGP SIGNATURE----- --Sig_/pYI3C4R6H+uUIwRHhliBjAt-- From owner-freebsd-fs@FreeBSD.ORG Tue Jul 8 16:22:20 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D9DF71065672 for ; Tue, 8 Jul 2008 16:22:20 +0000 (UTC) (envelope-from mike503@gmail.com) Received: from wf-out-1314.google.com (wf-out-1314.google.com [209.85.200.173]) by mx1.freebsd.org (Postfix) with ESMTP id AD56C8FC20 for ; Tue, 8 Jul 2008 16:22:20 +0000 (UTC) (envelope-from mike503@gmail.com) Received: by wf-out-1314.google.com with SMTP id 24so2401831wfg.7 for ; Tue, 08 Jul 2008 09:22:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=y6ownart9yjijdPsgAOrM7nGJHko/uto2HdDZAO0a04=; b=cK8MqjQ88H5TZOEkOtcRiqF+M5aunJUvG12X9GjqWiiCm7XlvZYyvmmRruyg2Gx7b+ HmulU+8gF1gNa92RDpYki9ogFUHdaLp5vCwL8ej5ICYLbJ584CixYg6dcxsW5ek3BcER sYitgJBliyPC1lSnKfLf8xG8JFPLNk3Dx3rLk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=d1UG4hB1zzJ3pVhWUD7jS1eYyyVI9aMwF0Ob0cH4ROPXDJ4Sy+0BXw5UHXSZzxCa8g DXZSo/TXGXPy74LcXspQmDwtyt+j+T2vxcYQMfGg9Z7+CmucoRLFqrFsCL9aNu25O0OL S3Xj/MUYsFoHL+4RQpAEqdJ6gZI3cP5gSaaF0= Received: by 10.142.132.2 with SMTP id f2mr1859732wfd.287.1215534140134; Tue, 08 Jul 2008 09:22:20 -0700 (PDT) Received: by 10.142.102.11 with HTTP; Tue, 8 Jul 2008 09:22:20 -0700 (PDT) Message-ID: Date: Tue, 8 Jul 2008 09:22:20 -0700 From: mike To: "CZUCZY Gergely" In-Reply-To: <20080708100701.57031cda@twoflower.in.publishing.hu> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <20080708100701.57031cda@twoflower.in.publishing.hu> Cc: freebsd-fs@freebsd.org Subject: Re: Thinking of using ZFS/FBSD for a backup system X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jul 2008 16:22:20 -0000 On 7/8/08, CZUCZY Gergely wrote: > Regardless of this, the system worked quite well. If ZFS were stable, this > easily could be our backup system. ZFS is great, awesome, but a bit unreliable > on FreeBSD, still needs some work. I forgot to ask - what are you doing now instead of FBSD+ZFS? From owner-freebsd-fs@FreeBSD.ORG Tue Jul 8 19:13:57 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 99538106567D for ; Tue, 8 Jul 2008 19:13:57 +0000 (UTC) (envelope-from phoemix@harmless.hu) Received: from marvin.harmless.hu (marvin.harmless.hu [195.56.55.204]) by mx1.freebsd.org (Postfix) with ESMTP id 538D68FC14 for ; Tue, 8 Jul 2008 19:13:57 +0000 (UTC) (envelope-from phoemix@harmless.hu) Received: from pool-0123.adsl.interware.hu ([213.178.100.123] helo=mort.in.publishing.hu) by marvin.harmless.hu with esmtpa (Exim 4.69 (FreeBSD)) (envelope-from ) id 1KGIdP-000Gt8-HN; Tue, 08 Jul 2008 21:13:55 +0200 Date: Tue, 8 Jul 2008 21:13:51 +0200 From: CZUCZY Gergely To: mike Message-ID: <20080708211351.069a2bc5@mort.in.publishing.hu> In-Reply-To: References: <20080708100701.57031cda@twoflower.in.publishing.hu> Organization: Harmless Digital Bt X-Mailer: Claws Mail 3.3.1 (GTK+ 2.12.1; i386-portbld-freebsd6.3) Mime-Version: 1.0 Content-Type: multipart/signed; boundary="Sig_/d2G74obkWQeffRuh7Pgdoub"; protocol="application/pgp-signature"; micalg=PGP-SHA1 Cc: freebsd-fs@freebsd.org Subject: Re: Thinking of using ZFS/FBSD for a backup system X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jul 2008 19:13:57 -0000 --Sig_/d2G74obkWQeffRuh7Pgdoub Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable We're still doing what we did. FBSD+ZFS _would_ have been the replacement. We're sucking with linux+dirvish. it's slow, dirvish is kinda retarded, it has many flaws, but it's stable. Well, in a way, you don't lose data like you would do with ZFS :) On Tue, 8 Jul 2008 09:22:20 -0700 mike wrote: > On 7/8/08, CZUCZY Gergely wrote: >=20 > > Regardless of this, the system worked quite well. If ZFS were > > stable, this easily could be our backup system. ZFS is great, > > awesome, but a bit unreliable on FreeBSD, still needs some work. >=20 > I forgot to ask - what are you doing now instead of FBSD+ZFS? --=20 Sincerely, Gergely CZUCZY, Harmless Digital mailto: gergely.czuczy@harmless.hu Legacy software is software that works. --Sig_/d2G74obkWQeffRuh7Pgdoub Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQFIc7xyzrC0WyuMkpsRAsv7AJ99Dumy7TyLGFhkKocy+hPC+Hp4gwCeO6OG zj9aOqbNY0GBAizR6N8aus8= =udZG -----END PGP SIGNATURE----- --Sig_/d2G74obkWQeffRuh7Pgdoub-- From owner-freebsd-fs@FreeBSD.ORG Tue Jul 8 19:50:15 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5B39B1065679 for ; Tue, 8 Jul 2008 19:50:15 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id B7E1B8FC15; Tue, 8 Jul 2008 19:50:13 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <4873C4FA.2020004@FreeBSD.org> Date: Tue, 08 Jul 2008 21:50:18 +0200 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.14 (Macintosh/20080421) MIME-Version: 1.0 To: mike References: <20080708100701.57031cda@twoflower.in.publishing.hu> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: Thinking of using ZFS/FBSD for a backup system X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jul 2008 19:50:15 -0000 mike wrote: > On 7/8/08, CZUCZY Gergely wrote: > >> Regardless of this, the system worked quite well. If ZFS were stable, this >> easily could be our backup system. ZFS is great, awesome, but a bit unreliable >> on FreeBSD, still needs some work. > > Really? I thought ZFS for basic things was not too bad in FBSD now. > > By basic I mean simple filesystem creation, snapshots and normal > devices. Not some crazy SAN LUNs and weird volume management stuff. > > I would really love to use FBSD as opposed to a Solaris derivative, > since I know nothing about them and I'd have to dedicate a machine for > it at home. Hrm. I wonder if I could just get by running a Solaris > derivative inside of a VM in VMware or something. ZFS needs careful memory tuning, but really, it's otherwise stable and it can be done. (ports-i386:~>sysctl hw.ncpu hw.ncpu: 4 (ports-i386:~)> sysctl hw.physmem hw.physmem: 4275478528 (ports-i386:~)> uname -a FreeBSD pointyhat.freebsd.org 8.0-CURRENT FreeBSD 8.0-CURRENT #31: Wed Jun 25 19:40:40 UTC 2008 kris@pointyhat.freebsd.org:/usr/src/sys.cvs/amd64/compile/POINTYHAT amd64 (ports-i386:~)> cat /boot/loader.conf vfs.zfs.prefetch_disable=1 vm.kmem_size=1572864000 This machine is highly disk loaded, with 1.08TB of disk, a load average usually between 8-30, currently hosting 94 ZFS filesystems, 898 snapshots, and making heavy use of ZFS features like cloning, incremental snapshot send/receive, etc. The disk workload is highly vnode-intensive, involving concurrent rsyncs over trees containing hundreds of thousands of files, busy NFS exports to about 40 clients, cvs updates, etc, constantly cycling through millions of vnodes. It works just fine. Kris From owner-freebsd-fs@FreeBSD.ORG Tue Jul 8 19:41:41 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4D1141065672 for ; Tue, 8 Jul 2008 19:41:41 +0000 (UTC) (envelope-from juri_mian@yahoo.com) Received: from n76.bullet.mail.sp1.yahoo.com (n76.bullet.mail.sp1.yahoo.com [98.136.44.48]) by mx1.freebsd.org (Postfix) with SMTP id 1907E8FC22 for ; Tue, 8 Jul 2008 19:41:41 +0000 (UTC) (envelope-from juri_mian@yahoo.com) Received: from [216.252.122.219] by n76.bullet.mail.sp1.yahoo.com with NNFMP; 08 Jul 2008 19:28:29 -0000 Received: from [69.147.65.163] by t4.bullet.sp1.yahoo.com with NNFMP; 08 Jul 2008 19:28:29 -0000 Received: from [127.0.0.1] by omp408.mail.sp1.yahoo.com with NNFMP; 08 Jul 2008 19:28:29 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 583350.46187.bm@omp408.mail.sp1.yahoo.com Received: (qmail 50828 invoked by uid 60001); 8 Jul 2008 19:28:29 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Received:X-Mailer:Date:From:Reply-To:Subject:To:MIME-Version:Content-Type:Message-ID; b=4i0xyHWXYP+27Dh8DyUbiuIjqA1XpmCVMqgJpRmRMBpq8MHERsGfwHH5xXn0vrtqVSUTB3wf8D03V6LCQQP/LLK7qabydAAMhI+GuHenUJ9HM5D80JwNTVdbC0/q/g7gazNYCiNqOz/M8T3R+KtvL5snMFUI/IjuzqnUZIuav84=; Received: from [71.63.232.32] by web45615.mail.sp1.yahoo.com via HTTP; Tue, 08 Jul 2008 12:28:28 PDT X-Mailer: YahooMailWebService/0.7.199 Date: Tue, 8 Jul 2008 12:28:28 -0700 (PDT) From: Juri Mianovich To: freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Message-ID: <400492.46414.qm@web45615.mail.sp1.yahoo.com> X-Mailman-Approved-At: Tue, 08 Jul 2008 19:57:00 +0000 Subject: 24 TB UFS2 reality check ? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: juri_mian@yahoo.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jul 2008 19:41:41 -0000 I am about to attach 24 1 TB drives to a 3ware 9650SE-24 raid card and attach it to a FreeBSD 6.3-RELEASE system. I am going to newfs that raw disk and turn it into one giant 24 TB UFS2 filesystem: newfs -i 65536 -U /dev/da1 I intend to enable quotas on this system BUT I do not intend to set any >2TB quotas for any one particular user. Questions: - anything else I should know ? Any danger ? Other than decreasing inode density, like I am with '-i 65535' are there any other settings I should be considering ? I will set kern.maxdsiz="2572000000" ... which I hope will be enough for fsck. - I have been (sort of) following the recent thread about >2TB quotas - let's say I have a user with _no quota set_ but they amass more than 2 TB of files - is that still a problem ? Or is it only a problem if I actually set a quota for them of >2TB ? Will repquota report correctly, even though they don't have a quota set ? Thanks. From owner-freebsd-fs@FreeBSD.ORG Tue Jul 8 20:01:51 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EAF681065670 for ; Tue, 8 Jul 2008 20:01:51 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outP.internet-mail-service.net (outp.internet-mail-service.net [216.240.47.239]) by mx1.freebsd.org (Postfix) with ESMTP id D59848FC17 for ; Tue, 8 Jul 2008 20:01:51 +0000 (UTC) (envelope-from julian@elischer.org) Received: from idiom.com (mx0.idiom.com [216.240.32.160]) by out.internet-mail-service.net (Postfix) with ESMTP id ABE5F2374; Tue, 8 Jul 2008 13:01:51 -0700 (PDT) Received: from julian-mac.elischer.org (localhost [127.0.0.1]) by idiom.com (Postfix) with ESMTP id 773492D601D; Tue, 8 Jul 2008 13:01:51 -0700 (PDT) Message-ID: <4873C7AE.50809@elischer.org> Date: Tue, 08 Jul 2008 13:01:50 -0700 From: Julian Elischer User-Agent: Thunderbird 2.0.0.14 (Macintosh/20080421) MIME-Version: 1.0 To: juri_mian@yahoo.com References: <400492.46414.qm@web45615.mail.sp1.yahoo.com> In-Reply-To: <400492.46414.qm@web45615.mail.sp1.yahoo.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: 24 TB UFS2 reality check ? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jul 2008 20:01:52 -0000 Juri Mianovich wrote: > I am about to attach 24 1 TB drives to a 3ware 9650SE-24 raid card and attach it to a FreeBSD 6.3-RELEASE system. > > I am going to newfs that raw disk and turn it into one giant 24 TB UFS2 filesystem: > > newfs -i 65536 -U /dev/da1 > > I intend to enable quotas on this system BUT I do not intend to set > any >2TB quotas for any one particular user. > > Questions: > > - anything else I should know ? Any danger ? Other than decreasing inode density, like I am with '-i 65535' are there any other settings I should be considering ? I will set kern.maxdsiz="2572000000" ... which I hope will be enough for fsck. > > - I have been (sort of) following the recent thread about >2TB quotas - let's say I have a user with _no quota set_ but they amass more than 2 TB of files - is that still a problem ? Or is it only a problem if I actually set a quota for them of >2TB ? Will repquota report correctly, even though they don't have a quota set ? > Thanks. > You had better have a lot of memory available ot your processes to be able to fsck this baby.. (it'd better be an amd64).. I don't remember the exact numbers but for 16k blocksize, it was something like 200MB ram for each 100GB of filesystem when populated with 60KB files.. (don't trust those numbers, do some testing (and let us know :-) ) From owner-freebsd-fs@FreeBSD.ORG Tue Jul 8 20:13:35 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 86E9F106567F for ; Tue, 8 Jul 2008 20:13:35 +0000 (UTC) (envelope-from phoemix@harmless.hu) Received: from marvin.harmless.hu (marvin.harmless.hu [195.56.55.204]) by mx1.freebsd.org (Postfix) with ESMTP id 2863A8FC1A for ; Tue, 8 Jul 2008 20:13:34 +0000 (UTC) (envelope-from phoemix@harmless.hu) Received: from pool-0123.adsl.interware.hu ([213.178.100.123] helo=mort.in.publishing.hu) by marvin.harmless.hu with esmtpa (Exim 4.69 (FreeBSD)) (envelope-from ) id 1KGJZ7-000IFb-7z; Tue, 08 Jul 2008 22:13:33 +0200 Date: Tue, 8 Jul 2008 22:13:27 +0200 From: CZUCZY Gergely To: Kris Kennaway Message-ID: <20080708221327.5c1d0e92@mort.in.publishing.hu> In-Reply-To: <4873C4FA.2020004@FreeBSD.org> References: <20080708100701.57031cda@twoflower.in.publishing.hu> <4873C4FA.2020004@FreeBSD.org> Organization: Harmless Digital Bt X-Mailer: Claws Mail 3.3.1 (GTK+ 2.12.1; i386-portbld-freebsd6.3) Mime-Version: 1.0 Content-Type: multipart/signed; boundary="Sig_/KaEqPRBplu5.ZRh+=81Qmdz"; protocol="application/pgp-signature"; micalg=PGP-SHA1 Cc: freebsd-fs@freebsd.org Subject: Re: Thinking of using ZFS/FBSD for a backup system X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jul 2008 20:13:35 -0000 --Sig_/KaEqPRBplu5.ZRh+=81Qmdz Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Yes Kris, but you've forgot something quite important. What you've just showed is -CURRENT, and how does that thumb-rule is about branches and (semi-)production systems?=20 My faint memories say something like "don't never ever even think of running -CURRENT on a production box", in a polite way. ZFS can be stable on -CURRENT but it's till -CURRENT, with its issues as a production system. So, the last we can go about a backup box is -STABLE, but i also wouldn't prefer that one, if I can. -RELEASE and patches for production, to be safe. Give us a stable ZFS in -RELEASE and -STABLE and we will be statisfied and happy. -CURRENT is still not a way for production boxes, that's asking for trouble. I've finetuned ZFS as much as I could, I've read every little tiny bit of hint/information/whatever that was available and I couldn't get rid of those kmem_size panics in -RELEASE and -STABLE. On Tue, 08 Jul 2008 21:50:18 +0200 Kris Kennaway wrote: > mike wrote: > > On 7/8/08, CZUCZY Gergely wrote: > >=20 > >> Regardless of this, the system worked quite well. If ZFS were > >> stable, this easily could be our backup system. ZFS is great, > >> awesome, but a bit unreliable on FreeBSD, still needs some work. > >=20 > > Really? I thought ZFS for basic things was not too bad in FBSD now. > >=20 > > By basic I mean simple filesystem creation, snapshots and normal > > devices. Not some crazy SAN LUNs and weird volume management stuff. > >=20 > > I would really love to use FBSD as opposed to a Solaris derivative, > > since I know nothing about them and I'd have to dedicate a machine > > for it at home. Hrm. I wonder if I could just get by running a > > Solaris derivative inside of a VM in VMware or something. >=20 > ZFS needs careful memory tuning, but really, it's otherwise stable > and it can be done. >=20 > (ports-i386:~>sysctl hw.ncpu > hw.ncpu: 4 > (ports-i386:~)> sysctl hw.physmem > hw.physmem: 4275478528 > (ports-i386:~)> uname -a > FreeBSD pointyhat.freebsd.org 8.0-CURRENT FreeBSD 8.0-CURRENT #31: > Wed Jun 25 19:40:40 UTC 2008=20 > kris@pointyhat.freebsd.org:/usr/src/sys.cvs/amd64/compile/POINTYHAT > amd64 (ports-i386:~)> cat /boot/loader.conf > vfs.zfs.prefetch_disable=3D1 > vm.kmem_size=3D1572864000 >=20 > This machine is highly disk loaded, with 1.08TB of disk, a load > average usually between 8-30, currently hosting 94 ZFS filesystems, > 898 snapshots, and making heavy use of ZFS features like cloning,=20 > incremental snapshot send/receive, etc. The disk workload is highly=20 > vnode-intensive, involving concurrent rsyncs over trees containing=20 > hundreds of thousands of files, busy NFS exports to about 40 clients,=20 > cvs updates, etc, constantly cycling through millions of vnodes. >=20 > It works just fine. >=20 > Kris --=20 Sincerely, Gergely CZUCZY, Harmless Digital mailto: gergely.czuczy@harmless.hu Legacy software is software that works. --Sig_/KaEqPRBplu5.ZRh+=81Qmdz Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQFIc8pqzrC0WyuMkpsRAj4MAKCm0u7GAlrNHmURNtDT/e1nOkyItQCfa+h0 yKN1cUk+HTkBtRULHij0TR0= =Iuzq -----END PGP SIGNATURE----- --Sig_/KaEqPRBplu5.ZRh+=81Qmdz-- From owner-freebsd-fs@FreeBSD.ORG Tue Jul 8 20:34:48 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E3CB31065673 for ; Tue, 8 Jul 2008 20:34:48 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 1305E8FC1B; Tue, 8 Jul 2008 20:34:47 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <4873CF6C.7000205@FreeBSD.org> Date: Tue, 08 Jul 2008 22:34:52 +0200 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.14 (Macintosh/20080421) MIME-Version: 1.0 To: CZUCZY Gergely References: <20080708100701.57031cda@twoflower.in.publishing.hu> <4873C4FA.2020004@FreeBSD.org> <20080708221327.5c1d0e92@mort.in.publishing.hu> In-Reply-To: <20080708221327.5c1d0e92@mort.in.publishing.hu> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: Thinking of using ZFS/FBSD for a backup system X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jul 2008 20:34:49 -0000 CZUCZY Gergely wrote: > Yes Kris, but you've forgot something quite important. > What you've just showed is -CURRENT, and how does that thumb-rule is > about branches and (semi-)production systems? > My faint memories say something like "don't never ever even think of > running -CURRENT on a production box", in a polite way. > ZFS can be stable on -CURRENT but it's till -CURRENT, with its issues > as a production system. So, the last we can go about a backup box is > -STABLE, but i also wouldn't prefer that one, if I can. -RELEASE and > patches for production, to be safe. > > Give us a stable ZFS in -RELEASE and -STABLE and we will be statisfied > and happy. -CURRENT is still not a way for production boxes, that's > asking for trouble. It's not relevant that I am running -CURRENT, there have been no changes in ZFS that are not also in -STABLE (and only one bug fix since 7.0-RELEASE, I think -- that was important, but it fixes mmap corruption, not a panic). I run -CURRENT to help debug it, but I am neither making use of ZFS fixes, nor encountering ZFS bugs. > I've finetuned ZFS as much as I could, I've read every little tiny bit > of hint/information/whatever that was available and I couldn't get rid > of those kmem_size panics in -RELEASE and -STABLE. Well, it's still almost certainly because you aren't setting kmem_size high enough. As you saw, that is the only thing I tuned (disabling prefetch is just for performance in my environment). If you can't set it high enough because you don't have enough RAM, that means your system does't have enough RAM to run ZFS, not that ZFS is unstable. Kris From owner-freebsd-fs@FreeBSD.ORG Tue Jul 8 20:40:24 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C61A81065670 for ; Tue, 8 Jul 2008 20:40:24 +0000 (UTC) (envelope-from roberto@keltia.freenix.fr) Received: from keltia.freenix.fr (keltia.freenix.org [IPv6:2001:660:330f:f820:213:72ff:fe15:f44]) by mx1.freebsd.org (Postfix) with ESMTP id 6E6618FC1A for ; Tue, 8 Jul 2008 20:40:24 +0000 (UTC) (envelope-from roberto@keltia.freenix.fr) Received: from localhost (localhost [127.0.0.1]) by keltia.freenix.fr (Postfix/TLS) with ESMTP id F0D31398A0 for ; Tue, 8 Jul 2008 22:40:21 +0200 (CEST) X-Virus-Scanned: amavisd-new at keltia.freenix.fr Received: from keltia.freenix.fr ([127.0.0.1]) by localhost (keltia.freenix.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Hu9pu6Go3qAi for ; Tue, 8 Jul 2008 22:40:21 +0200 (CEST) Received: by keltia.freenix.fr (Postfix/TLS, from userid 101) id 9E08B39889; Tue, 8 Jul 2008 22:40:21 +0200 (CEST) Date: Tue, 8 Jul 2008 22:40:21 +0200 From: Ollivier Robert To: freebsd-fs@freebsd.org Message-ID: <20080708204021.GA97977@keltia.freenix.fr> References: <20080708100701.57031cda@twoflower.in.publishing.hu> <4873C4FA.2020004@FreeBSD.org> <20080708221327.5c1d0e92@mort.in.publishing.hu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20080708221327.5c1d0e92@mort.in.publishing.hu> X-Operating-System: MacOS X / Macbook Pro - FreeBSD 7 / Dell D820 SMP User-Agent: Mutt/1.5.17 (2007-11-01) Subject: Re: Thinking of using ZFS/FBSD for a backup system X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jul 2008 20:40:24 -0000 According to CZUCZY Gergely: > Yes Kris, but you've forgot something quite important. > What you've just showed is -CURRENT, and how does that thumb-rule is > about branches and (semi-)production systems? There have been no significant change in CURRENT WRT ZFS compared to 7. -- Ollivier ROBERT -=- FreeBSD: The Power to Serve! -=- roberto@keltia.freenix.fr Darwin sidhe.keltia.net Version 9.2.0: Tue Feb 5 16:13:22 PST 2008; i386 From owner-freebsd-fs@FreeBSD.ORG Tue Jul 8 20:42:28 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6D4E71065674 for ; Tue, 8 Jul 2008 20:42:28 +0000 (UTC) (envelope-from roberto@keltia.freenix.fr) Received: from keltia.freenix.fr (keltia.freenix.org [IPv6:2001:660:330f:f820:213:72ff:fe15:f44]) by mx1.freebsd.org (Postfix) with ESMTP id 181878FC18 for ; Tue, 8 Jul 2008 20:42:28 +0000 (UTC) (envelope-from roberto@keltia.freenix.fr) Received: from localhost (localhost [127.0.0.1]) by keltia.freenix.fr (Postfix/TLS) with ESMTP id 3B06D398A0 for ; Tue, 8 Jul 2008 22:42:27 +0200 (CEST) X-Virus-Scanned: amavisd-new at keltia.freenix.fr Received: from keltia.freenix.fr ([127.0.0.1]) by localhost (keltia.freenix.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id wS3YOk0KKo+F for ; Tue, 8 Jul 2008 22:42:26 +0200 (CEST) Received: by keltia.freenix.fr (Postfix/TLS, from userid 101) id DC4D539889; Tue, 8 Jul 2008 22:42:26 +0200 (CEST) Date: Tue, 8 Jul 2008 22:42:26 +0200 From: Ollivier Robert To: freebsd-fs@freebsd.org Message-ID: <20080708204226.GB97977@keltia.freenix.fr> References: <20080708100701.57031cda@twoflower.in.publishing.hu> <4873C4FA.2020004@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4873C4FA.2020004@FreeBSD.org> X-Operating-System: MacOS X / Macbook Pro - FreeBSD 7 / Dell D820 SMP User-Agent: Mutt/1.5.17 (2007-11-01) Subject: Re: Thinking of using ZFS/FBSD for a backup system X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jul 2008 20:42:28 -0000 According to Kris Kennaway: > (ports-i386:~)> cat /boot/loader.conf > vfs.zfs.prefetch_disable=1 > vm.kmem_size=1572864000 Hvae you tried w/o the prefetch_disable tunable? -- Ollivier ROBERT -=- FreeBSD: The Power to Serve! -=- roberto@keltia.freenix.fr Darwin sidhe.keltia.net Version 9.2.0: Tue Feb 5 16:13:22 PST 2008; i386 From owner-freebsd-fs@FreeBSD.ORG Tue Jul 8 20:54:54 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B6E751065671; Tue, 8 Jul 2008 20:54:54 +0000 (UTC) (envelope-from phoemix@harmless.hu) Received: from marvin.harmless.hu (marvin.harmless.hu [195.56.55.204]) by mx1.freebsd.org (Postfix) with ESMTP id 6F8EA8FC19; Tue, 8 Jul 2008 20:54:54 +0000 (UTC) (envelope-from phoemix@harmless.hu) Received: from pool-0123.adsl.interware.hu ([213.178.100.123] helo=mort.in.publishing.hu) by marvin.harmless.hu with esmtpa (Exim 4.69 (FreeBSD)) (envelope-from ) id 1KGKD7-000J8L-6j; Tue, 08 Jul 2008 22:54:53 +0200 Date: Tue, 8 Jul 2008 22:54:49 +0200 From: CZUCZY Gergely To: Kris Kennaway Message-ID: <20080708225449.1070252d@mort.in.publishing.hu> In-Reply-To: <4873CF6C.7000205@FreeBSD.org> References: <20080708100701.57031cda@twoflower.in.publishing.hu> <4873C4FA.2020004@FreeBSD.org> <20080708221327.5c1d0e92@mort.in.publishing.hu> <4873CF6C.7000205@FreeBSD.org> Organization: Harmless Digital Bt X-Mailer: Claws Mail 3.3.1 (GTK+ 2.12.1; i386-portbld-freebsd6.3) Mime-Version: 1.0 Content-Type: multipart/signed; boundary="Sig_/abNulJsU+urI/gQ7PKIQqRh"; protocol="application/pgp-signature"; micalg=PGP-SHA1 Cc: freebsd-fs@freebsd.org Subject: Re: Thinking of using ZFS/FBSD for a backup system X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jul 2008 20:54:54 -0000 --Sig_/abNulJsU+urI/gQ7PKIQqRh Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable On Tue, 08 Jul 2008 22:34:52 +0200 Kris Kennaway wrote: > > I've finetuned ZFS as much as I could, I've read every little tiny > > bit of hint/information/whatever that was available and I couldn't > > get rid of those kmem_size panics in -RELEASE and -STABLE. >=20 > Well, it's still almost certainly because you aren't setting > kmem_size high enough. As you saw, that is the only thing I tuned > (disabling prefetch is just for performance in my environment). >=20 > If you can't set it high enough because you don't have enough RAM, > that means your system does't have enough RAM to run ZFS, not that > ZFS is unstable. I've had a box with 2GB of memory for it, and around 5-6 filesystems. I've set kmem_size as large as it was allowed, not a bit smaller. Where's the guide showing how much memory should I have for a setup? How can "enough memory" be determined for a setup, without having panics? >=20 > Kris >=20 --=20 Sincerely, Gergely CZUCZY, Harmless Digital mailto: gergely.czuczy@harmless.hu Legacy software is software that works. --Sig_/abNulJsU+urI/gQ7PKIQqRh Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQFIc9QbzrC0WyuMkpsRAjp9AJwOJy2wFJQBNmIbZ+Tz0Wrl/UhjDgCgidoR AsR0Sct76jVPpbgsAoyjiaM= =3uD9 -----END PGP SIGNATURE----- --Sig_/abNulJsU+urI/gQ7PKIQqRh-- From owner-freebsd-fs@FreeBSD.ORG Tue Jul 8 20:47:03 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 432FD1065674 for ; Tue, 8 Jul 2008 20:47:03 +0000 (UTC) (envelope-from juri_mian@yahoo.com) Received: from n76.bullet.mail.sp1.yahoo.com (n76.bullet.mail.sp1.yahoo.com [98.136.44.48]) by mx1.freebsd.org (Postfix) with SMTP id 27D018FC13 for ; Tue, 8 Jul 2008 20:47:03 +0000 (UTC) (envelope-from juri_mian@yahoo.com) Received: from [216.252.122.216] by n76.bullet.mail.sp1.yahoo.com with NNFMP; 08 Jul 2008 20:46:49 -0000 Received: from [69.147.65.168] by t1.bullet.sp1.yahoo.com with NNFMP; 08 Jul 2008 20:46:49 -0000 Received: from [127.0.0.1] by omp503.mail.sp1.yahoo.com with NNFMP; 08 Jul 2008 20:46:49 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 143946.29843.bm@omp503.mail.sp1.yahoo.com Received: (qmail 10275 invoked by uid 60001); 8 Jul 2008 20:46:49 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Received:X-Mailer:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Message-ID; b=XHLVwIKP00R7G7SeYYXaSUt7dCW1Rmnia3x8mh8DO+ZmpHjaHJDTayEahoogByhABuV/YiEqPfXQDUxAEQ2bu26s5cFxdPCbzwj/u7pZpx1ZoD2ogpi2+hsIqgSoYjeEztDHv52YC5fPXlU7J0j9ApKybLK3IcAP1ijAnP6gNwg=; Received: from [71.63.232.32] by web45611.mail.sp1.yahoo.com via HTTP; Tue, 08 Jul 2008 13:46:48 PDT X-Mailer: YahooMailWebService/0.7.199 Date: Tue, 8 Jul 2008 13:46:48 -0700 (PDT) From: Juri Mianovich To: Julian Elischer In-Reply-To: <4873C7AE.50809@elischer.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Message-ID: <958164.4787.qm@web45611.mail.sp1.yahoo.com> X-Mailman-Approved-At: Tue, 08 Jul 2008 20:59:11 +0000 Cc: freebsd-fs@freebsd.org Subject: Re: 24 TB UFS2 reality check ? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: juri_mian@yahoo.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jul 2008 20:47:03 -0000 --- On Tue, 7/8/08, Julian Elischer wrote: > You had better have a lot of memory available ot your > processes to be > able to fsck this baby.. (it'd better be an amd64).. > I don't remember the exact numbers but for 16k > blocksize, > it was something like 200MB ram for each 100GB of > filesystem when > populated with 60KB files.. > (don't trust those numbers, do some testing (and let us > know :-) ) Thank you very much. I currently have a similar system with: /dev/da1 8.0T 1.3T 6.1T 16% /users which was created with 'newfs -i 32768 -U /dev/da1' ... and I can successfully fsck it with my: kern.maxdsiz="2572000000" setting. So perhaps a filesystem 3x that size should be '-i 131072' to maintain the same ability to fsck ? None of these systems are 64-bit - they are all running i386 w/4 GB of ram. If I stuck with '-i 65536' (instead of going all the way to 131072) and things got sticky, I could always temporarily reboot with a maxdsiz closer to 3 GB, right ? From owner-freebsd-fs@FreeBSD.ORG Tue Jul 8 21:10:11 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 529C71065677 for ; Tue, 8 Jul 2008 21:10:11 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id B6E798FC1B; Tue, 8 Jul 2008 21:10:08 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <4873D7B5.2060901@FreeBSD.org> Date: Tue, 08 Jul 2008 23:10:13 +0200 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.14 (Macintosh/20080421) MIME-Version: 1.0 To: Ollivier Robert References: <20080708100701.57031cda@twoflower.in.publishing.hu> <4873C4FA.2020004@FreeBSD.org> <20080708204226.GB97977@keltia.freenix.fr> In-Reply-To: <20080708204226.GB97977@keltia.freenix.fr> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: Thinking of using ZFS/FBSD for a backup system X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jul 2008 21:10:11 -0000 Ollivier Robert wrote: > According to Kris Kennaway: >> (ports-i386:~)> cat /boot/loader.conf >> vfs.zfs.prefetch_disable=1 >> vm.kmem_size=1572864000 > > Hvae you tried w/o the prefetch_disable tunable? I have not done careful measurements, but casual observation suggests that on my workloads I get better performance without prefetch. I have a pair of mirrored disks on an amr, and my workload is quite random-access so prefetching just introduces latencies and wastes already-saturated disk bandwidth. With more disks or a different workload I would expect different performance characteristics. Kris From owner-freebsd-fs@FreeBSD.ORG Tue Jul 8 21:12:28 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 35778106567F for ; Tue, 8 Jul 2008 21:12:28 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outF.internet-mail-service.net (outf.internet-mail-service.net [216.240.47.229]) by mx1.freebsd.org (Postfix) with ESMTP id 1FFDB8FC42 for ; Tue, 8 Jul 2008 21:12:28 +0000 (UTC) (envelope-from julian@elischer.org) Received: from idiom.com (mx0.idiom.com [216.240.32.160]) by out.internet-mail-service.net (Postfix) with ESMTP id D34CC2382; Tue, 8 Jul 2008 14:12:27 -0700 (PDT) Received: from julian-mac.elischer.org (localhost [127.0.0.1]) by idiom.com (Postfix) with ESMTP id 8EAC42D600F; Tue, 8 Jul 2008 14:12:27 -0700 (PDT) Message-ID: <4873D83A.2080803@elischer.org> Date: Tue, 08 Jul 2008 14:12:26 -0700 From: Julian Elischer User-Agent: Thunderbird 2.0.0.14 (Macintosh/20080421) MIME-Version: 1.0 To: juri_mian@yahoo.com References: <958164.4787.qm@web45611.mail.sp1.yahoo.com> In-Reply-To: <958164.4787.qm@web45611.mail.sp1.yahoo.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: 24 TB UFS2 reality check ? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jul 2008 21:12:28 -0000 Juri Mianovich wrote: > > > --- On Tue, 7/8/08, Julian Elischer wrote: > > >> You had better have a lot of memory available ot your >> processes to be >> able to fsck this baby.. (it'd better be an amd64).. >> I don't remember the exact numbers but for 16k >> blocksize, >> it was something like 200MB ram for each 100GB of >> filesystem when >> populated with 60KB files.. >> (don't trust those numbers, do some testing (and let us >> know :-) ) > > > Thank you very much. > > I currently have a similar system with: > > /dev/da1 8.0T 1.3T 6.1T 16% /users > > which was created with 'newfs -i 32768 -U /dev/da1' ... and I can > successfully fsck it with my: > > kern.maxdsiz="2572000000" > > setting. > > So perhaps a filesystem 3x that size should be '-i 131072' to > maintain the same ability to fsck ? > > None of these systems are 64-bit - they are all running i386 w/4 GB > of ram. > > If I stuck with '-i 65536' (instead of going all the way to 131072) > and things got sticky, I could always temporarily reboot with a > maxdsiz closer to 3 GB, right ? You will have to do tests to see how big the virtual size of the fsck process gets per TB of disk. you should be able to see it with top. Notice that your 8TB filesystem is only 16% full. The memory will increase to some exgtent when you have more files.. try filling up your 8TB system with files and doing it again. > > From owner-freebsd-fs@FreeBSD.ORG Tue Jul 8 21:26:18 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7C1A5106567B for ; Tue, 8 Jul 2008 21:26:18 +0000 (UTC) (envelope-from bakul@bitblocks.com) Received: from mail.bitblocks.com (mail.bitblocks.com [64.142.15.60]) by mx1.freebsd.org (Postfix) with ESMTP id 63A998FC19 for ; Tue, 8 Jul 2008 21:26:18 +0000 (UTC) (envelope-from bakul@bitblocks.com) Received: from bitblocks.com (localhost.bitblocks.com [127.0.0.1]) by mail.bitblocks.com (Postfix) with ESMTP id C606C5B75; Tue, 8 Jul 2008 14:26:17 -0700 (PDT) To: juri_mian@yahoo.com In-reply-to: Your message of "Tue, 08 Jul 2008 14:17:57 PDT." <336596.22193.qm@web45608.mail.sp1.yahoo.com> Date: Tue, 08 Jul 2008 14:26:17 -0700 From: Bakul Shah Message-Id: <20080708212617.C606C5B75@mail.bitblocks.com> Cc: freebsd-fs@freebsd.org, Julian Elischer Subject: Re: 24 TB UFS2 reality check ? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jul 2008 21:26:18 -0000 On Tue, 08 Jul 2008 14:17:57 PDT Juri Mianovich wrote: > > I vaguely recall it was more like 700MB of memory per > > Terabyte on a 50% filled UFS2. Things may have improved > > in the three years since I did that. I don't recall the time > > to fsck but it was pretty bad! That was the main reason I > > switched from UFS2. > > Why does fsck need to reserve all that memory in advance and hold it the enti > re fsck ? Is it necessary by definition, or could it be written to not requi > re that ? May be it can but why bother. It just feels wrong to have to check the entire FS state after a crash -- it doesn't scale. From owner-freebsd-fs@FreeBSD.ORG Tue Jul 8 21:32:05 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 66FAC106564A for ; Tue, 8 Jul 2008 21:32:05 +0000 (UTC) (envelope-from bakul@bitblocks.com) Received: from mail.bitblocks.com (mail.bitblocks.com [64.142.15.60]) by mx1.freebsd.org (Postfix) with ESMTP id 4E8BE8FC13 for ; Tue, 8 Jul 2008 21:32:05 +0000 (UTC) (envelope-from bakul@bitblocks.com) Received: from bitblocks.com (localhost.bitblocks.com [127.0.0.1]) by mail.bitblocks.com (Postfix) with ESMTP id 39FC75B46; Tue, 8 Jul 2008 14:13:44 -0700 (PDT) To: Julian Elischer In-reply-to: Your message of "Tue, 08 Jul 2008 13:01:50 PDT." <4873C7AE.50809@elischer.org> Date: Tue, 08 Jul 2008 14:13:44 -0700 From: Bakul Shah Message-Id: <20080708211344.39FC75B46@mail.bitblocks.com> Cc: freebsd-fs@freebsd.org, juri_mian@yahoo.com Subject: Re: 24 TB UFS2 reality check ? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jul 2008 21:32:05 -0000 On Tue, 08 Jul 2008 13:01:50 PDT Julian Elischer wrote: > Juri Mianovich wrote: > > I am about to attach 24 1 TB drives to a 3ware 9650SE-24 raid card > and attach it to a FreeBSD 6.3-RELEASE system. > > > I am going to newfs that raw disk and turn it into one giant 24 TB > UFS2 filesystem: I think Jan is asking for trouble.... At the very least he should consider mirroring or RAID5ing. > You had better have a lot of memory available ot your processes to be > able to fsck this baby.. (it'd better be an amd64).. > I don't remember the exact numbers but for 16k blocksize, > it was something like 200MB ram for each 100GB of filesystem when > populated with 60KB files.. > (don't trust those numbers, do some testing (and let us know :-) ) I vaguely recall it was more like 700MB of memory per Terabyte on a 50% filled UFS2. Things may have improved in the three years since I did that. I don't recall the time to fsck but it was pretty bad! That was the main reason I switched from UFS2. From owner-freebsd-fs@FreeBSD.ORG Tue Jul 8 21:44:42 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A05391065671 for ; Tue, 8 Jul 2008 21:44:42 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outV.internet-mail-service.net (outv.internet-mail-service.net [216.240.47.245]) by mx1.freebsd.org (Postfix) with ESMTP id 7BA3E8FC13 for ; Tue, 8 Jul 2008 21:44:42 +0000 (UTC) (envelope-from julian@elischer.org) Received: from idiom.com (mx0.idiom.com [216.240.32.160]) by out.internet-mail-service.net (Postfix) with ESMTP id 6568123FA; Tue, 8 Jul 2008 14:44:42 -0700 (PDT) Received: from julian-mac.elischer.org (localhost [127.0.0.1]) by idiom.com (Postfix) with ESMTP id DC1932D6027; Tue, 8 Jul 2008 14:44:41 -0700 (PDT) Message-ID: <4873DFC9.6000006@elischer.org> Date: Tue, 08 Jul 2008 14:44:41 -0700 From: Julian Elischer User-Agent: Thunderbird 2.0.0.14 (Macintosh/20080421) MIME-Version: 1.0 To: juri_mian@yahoo.com References: <336596.22193.qm@web45608.mail.sp1.yahoo.com> In-Reply-To: <336596.22193.qm@web45608.mail.sp1.yahoo.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: 24 TB UFS2 reality check ? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jul 2008 21:44:42 -0000 Juri Mianovich wrote: > > > --- On Tue, 7/8/08, Bakul Shah wrote: > > >> I vaguely recall it was more like 700MB of memory per >> Terabyte on a 50% filled UFS2. Things may have improved >> in the three years since I did that. I don't recall >> the time >> to fsck but it was pretty bad! That was the main reason I >> switched from UFS2. > > > Why does fsck need to reserve all that memory in advance and hold it the entire fsck ? Is it necessary by definition, or could it be written to not require that ? > > > it doesn't reserve it .. that's how much data it builds up From owner-freebsd-fs@FreeBSD.ORG Tue Jul 8 21:17:58 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 54437106567B for ; Tue, 8 Jul 2008 21:17:58 +0000 (UTC) (envelope-from juri_mian@yahoo.com) Received: from n54.bullet.mail.sp1.yahoo.com (n54.bullet.mail.sp1.yahoo.com [98.136.44.32]) by mx1.freebsd.org (Postfix) with SMTP id 1DAD48FC17 for ; Tue, 8 Jul 2008 21:17:58 +0000 (UTC) (envelope-from juri_mian@yahoo.com) Received: from [216.252.122.218] by n54.bullet.mail.sp1.yahoo.com with NNFMP; 08 Jul 2008 21:17:57 -0000 Received: from [69.147.65.153] by t3.bullet.sp1.yahoo.com with NNFMP; 08 Jul 2008 21:17:57 -0000 Received: from [127.0.0.1] by omp401.mail.sp1.yahoo.com with NNFMP; 08 Jul 2008 21:17:57 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 537140.90166.bm@omp401.mail.sp1.yahoo.com Received: (qmail 24848 invoked by uid 60001); 8 Jul 2008 21:17:57 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Received:X-Mailer:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Message-ID; b=HvySe/LqkimwdhBGYyOtd80Q+xNaogMxOIfGIDfwi+qwT9uhiu1uGOmO/L85K+En4PiBtx6Q6dBGgi+JvHF05wicTS3pbPwYE2GwOCneQMKd1j2xkv96HSwTkCe5uGyE5J3F01RIjC9J8xcra50gfw/F/pJXzGh0j40V7wXCEC0=; Received: from [71.63.232.32] by web45608.mail.sp1.yahoo.com via HTTP; Tue, 08 Jul 2008 14:17:57 PDT X-Mailer: YahooMailWebService/0.7.199 Date: Tue, 8 Jul 2008 14:17:57 -0700 (PDT) From: Juri Mianovich To: Julian Elischer , Bakul Shah In-Reply-To: <20080708211344.39FC75B46@mail.bitblocks.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Message-ID: <336596.22193.qm@web45608.mail.sp1.yahoo.com> X-Mailman-Approved-At: Tue, 08 Jul 2008 21:45:08 +0000 Cc: freebsd-fs@freebsd.org Subject: Re: 24 TB UFS2 reality check ? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: juri_mian@yahoo.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jul 2008 21:17:58 -0000 --- On Tue, 7/8/08, Bakul Shah wrote: > I vaguely recall it was more like 700MB of memory per > Terabyte on a 50% filled UFS2. Things may have improved > in the three years since I did that. I don't recall > the time > to fsck but it was pretty bad! That was the main reason I > switched from UFS2. Why does fsck need to reserve all that memory in advance and hold it the entire fsck ? Is it necessary by definition, or could it be written to not require that ? From owner-freebsd-fs@FreeBSD.ORG Tue Jul 8 21:47:22 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 760BF106564A; Tue, 8 Jul 2008 21:47:22 +0000 (UTC) (envelope-from randy@psg.com) Received: from rip.psg.com (rip.psg.com [IPv6:2001:418:1::39]) by mx1.freebsd.org (Postfix) with ESMTP id 574F28FC12; Tue, 8 Jul 2008 21:47:22 +0000 (UTC) (envelope-from randy@psg.com) Received: from 50.216.138.210.bn.2iij.net ([210.138.216.50] helo=rmac.psg.com) by rip.psg.com with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.69 (FreeBSD)) (envelope-from ) id 1KGL1t-000JEa-V5; Tue, 08 Jul 2008 21:47:22 +0000 Message-ID: <4873E068.8060305@psg.com> Date: Wed, 09 Jul 2008 06:47:20 +0900 From: Randy Bush User-Agent: Thunderbird 2.0.0.14 (Macintosh/20080421) MIME-Version: 1.0 To: Kris Kennaway References: <20080708100701.57031cda@twoflower.in.publishing.hu> <4873C4FA.2020004@FreeBSD.org> In-Reply-To: <4873C4FA.2020004@FreeBSD.org> X-Enigmail-Version: 0.95.6 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: Thinking of using ZFS/FBSD for a backup system X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jul 2008 21:47:22 -0000 light to medium load am64 w 8g hw.ncpu: 2 vm.kmem_size=600M vm.kmem_size_max=600M vfs.zfs.prefetch_disable=1 been trouble free since boot. wondering if it is time to try on i386. anyone with serious experience with i386 zfs? randy From owner-freebsd-fs@FreeBSD.ORG Tue Jul 8 21:48:32 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1AE9B1065684; Tue, 8 Jul 2008 21:48:32 +0000 (UTC) (envelope-from randy@psg.com) Received: from rip.psg.com (rip.psg.com [IPv6:2001:418:1::39]) by mx1.freebsd.org (Postfix) with ESMTP id 016898FC26; Tue, 8 Jul 2008 21:48:32 +0000 (UTC) (envelope-from randy@psg.com) Received: from 50.216.138.210.bn.2iij.net ([210.138.216.50] helo=rmac.psg.com) by rip.psg.com with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.69 (FreeBSD)) (envelope-from ) id 1KGL31-000JFB-Qr; Tue, 08 Jul 2008 21:48:31 +0000 Message-ID: <4873E0AE.6090305@psg.com> Date: Wed, 09 Jul 2008 06:48:30 +0900 From: Randy Bush User-Agent: Thunderbird 2.0.0.14 (Macintosh/20080421) MIME-Version: 1.0 To: Kris Kennaway References: <20080708100701.57031cda@twoflower.in.publishing.hu> <4873C4FA.2020004@FreeBSD.org> <4873E068.8060305@psg.com> In-Reply-To: <4873E068.8060305@psg.com> X-Enigmail-Version: 0.95.6 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: Thinking of using ZFS/FBSD for a backup system X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jul 2008 21:48:32 -0000 > light to medium load am64 w 8g . first cuppa. it's 4g randy From owner-freebsd-fs@FreeBSD.ORG Tue Jul 8 22:03:53 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2AA511065679 for ; Tue, 8 Jul 2008 22:03:53 +0000 (UTC) (envelope-from speedtoys.racing@gmail.com) Received: from yx-out-2324.google.com (yx-out-2324.google.com [74.125.44.29]) by mx1.freebsd.org (Postfix) with ESMTP id D920C8FC16 for ; Tue, 8 Jul 2008 22:03:52 +0000 (UTC) (envelope-from speedtoys.racing@gmail.com) Received: by yx-out-2324.google.com with SMTP id 8so612244yxb.13 for ; Tue, 08 Jul 2008 15:03:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=rYl0TlHk27ssMxDaz2bafxhdjA9xH7FFA9kgjyaOiXU=; b=DaBVosfhb4qjhDAuWq6sU8aAX4qGFMR+ul1n0LXZLGFpIMByv/CStG6pqRlwjtxDFJ TAqy5pymZYlyR2WUV/TemzQ6nlCQ6csaAlgnKxJrMngJd3pWGiTL/sjdzG0W5Rmov0nm TB5u0dOLOYvhDD4SWhDUSNkCcZNjTa20UpNcw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=S5eYtIK+WQ3shQzBmHwts0YyiIkoie7b+FdvgG+6M/ZNctXeMkCnQql6NypeRsIw7Q 4ch9REZypRttFxVfTxK2ZHX3MbYMgcufKZxxboBYmh+TVEWeasUJiZxo67wES/TmkMJr gCuM5X14i9M2SX397AT/+pZnKzHQeNaw1snQs= Received: by 10.114.235.8 with SMTP id i8mr8492497wah.194.1215552893583; Tue, 08 Jul 2008 14:34:53 -0700 (PDT) Received: by 10.114.102.10 with HTTP; Tue, 8 Jul 2008 14:34:53 -0700 (PDT) Message-ID: Date: Tue, 8 Jul 2008 14:34:53 -0700 From: "Jeff Mohler" To: juri_mian@yahoo.com In-Reply-To: <400492.46414.qm@web45615.mail.sp1.yahoo.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <400492.46414.qm@web45615.mail.sp1.yahoo.com> Cc: freebsd-fs@freebsd.org Subject: Re: 24 TB UFS2 reality check ? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jul 2008 22:03:53 -0000 Wow..the odds of hitting an uncorrectable bit error are actually pretty HIGH in that configuration,. Good luck. You may get to find out why Enterprise arrays are expensive. On Tue, Jul 8, 2008 at 12:28 PM, Juri Mianovich wrote: > I am about to attach 24 1 TB drives to a 3ware 9650SE-24 raid card and attach it to a FreeBSD 6.3-RELEASE system. > > I am going to newfs that raw disk and turn it into one giant 24 TB UFS2 filesystem: > > newfs -i 65536 -U /dev/da1 > > I intend to enable quotas on this system BUT I do not intend to set any >2TB quotas for any one particular user. > > Questions: > > - anything else I should know ? Any danger ? Other than decreasing inode density, like I am with '-i 65535' are there any other settings I should be considering ? I will set kern.maxdsiz="2572000000" ... which I hope will be enough for fsck. > > - I have been (sort of) following the recent thread about >2TB quotas - let's say I have a user with _no quota set_ but they amass more than 2 TB of files - is that still a problem ? Or is it only a problem if I actually set a quota for them of >2TB ? Will repquota report correctly, even though they don't have a quota set ? > > Thanks. > > > > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@FreeBSD.ORG Tue Jul 8 22:31:20 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 09B371065679 for ; Tue, 8 Jul 2008 22:31:20 +0000 (UTC) (envelope-from mike503@gmail.com) Received: from wf-out-1314.google.com (wf-out-1314.google.com [209.85.200.169]) by mx1.freebsd.org (Postfix) with ESMTP id CEDC08FC1E for ; Tue, 8 Jul 2008 22:31:19 +0000 (UTC) (envelope-from mike503@gmail.com) Received: by wf-out-1314.google.com with SMTP id 24so2503690wfg.7 for ; Tue, 08 Jul 2008 15:31:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=du78Z5CcBlaClZ+4/xfkkEEtqDG5YU7ELdBmq2+5348=; b=OWp4juyZFulrqcWtzVkIqpLoyrSgRUZFhalFq0bsRUsM5hDgOwTwf4DHNeWOsgIjDZ cG54KNW2ZKXvZUdUWo7X2dqEzfH3avFWEAmXU78EwAQXHU2mMjYD4H7aEKO+syP2FE5A S+G2OAvKNTlSR343M6d9vZbz4lX0Xrm336s4E= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=IQS/aRi+uxgCDB9DYMfuKnfFl7+B7WyPppmZoo8RPi/mWUnOMAt7QFOLi7rByqpDjO dkg4RLaa8oVojJxXA673DKzpeJ5nu+HAzK4pQ32bUEiSvrllX1+X/Di12Keu+qgnlVaM YNgbnprHHD6MeP0KKgZu39RSgix0FO3N2h1MM= Received: by 10.142.211.1 with SMTP id j1mr2007229wfg.203.1215556279564; Tue, 08 Jul 2008 15:31:19 -0700 (PDT) Received: by 10.142.102.11 with HTTP; Tue, 8 Jul 2008 15:31:19 -0700 (PDT) Message-ID: Date: Tue, 8 Jul 2008 15:31:19 -0700 From: mike To: freebsd-fs@freebsd.org In-Reply-To: <4873E0AE.6090305@psg.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <20080708100701.57031cda@twoflower.in.publishing.hu> <4873C4FA.2020004@FreeBSD.org> <4873E068.8060305@psg.com> <4873E0AE.6090305@psg.com> Subject: Re: Thinking of using ZFS/FBSD for a backup system X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jul 2008 22:31:20 -0000 On 7/8/08, Randy Bush wrote: > . first cuppa. it's 4g okay, so it sounds like from multiple people this could be doable and stable enough: dual or quad-core current gen processor (amd64) 4g ram 4 or 6x 1tb disks freebsd 7.0-release these tweaks to /boot/loader.conf: vm.kmem_size=600M vm.kmem_size_max=600M vfs.zfs.prefetch_disable=1 right? I am fine with sacrificing a little bit of speed for stability. This will only run a few rsyncs per day, and then a single snapshot per day per filesystem (with maybe 15 filesystems maximum right now) From owner-freebsd-fs@FreeBSD.ORG Tue Jul 8 22:40:28 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 29964106567B for ; Tue, 8 Jul 2008 22:40:28 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 682F98FC19; Tue, 8 Jul 2008 22:40:27 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <4873ECE0.5090706@FreeBSD.org> Date: Wed, 09 Jul 2008 00:40:32 +0200 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.14 (Macintosh/20080421) MIME-Version: 1.0 To: mike References: <20080708100701.57031cda@twoflower.in.publishing.hu> <4873C4FA.2020004@FreeBSD.org> <4873E068.8060305@psg.com> <4873E0AE.6090305@psg.com> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: Thinking of using ZFS/FBSD for a backup system X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jul 2008 22:40:28 -0000 mike wrote: > On 7/8/08, Randy Bush wrote: > >> . first cuppa. it's 4g > > okay, so it sounds like from multiple people this could be doable and > stable enough: > > dual or quad-core current gen processor (amd64) > 4g ram > 4 or 6x 1tb disks > freebsd 7.0-release > > these tweaks to /boot/loader.conf: > > vm.kmem_size=600M > vm.kmem_size_max=600M > vfs.zfs.prefetch_disable=1 > > right? Might be enough kmem, if not, or if you want better performance from increased caching, increase as high as 1500M Kris From owner-freebsd-fs@FreeBSD.ORG Tue Jul 8 22:41:07 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 839161065673; Tue, 8 Jul 2008 22:41:07 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 66B538FC15; Tue, 8 Jul 2008 22:41:06 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <4873ED07.8060105@FreeBSD.org> Date: Wed, 09 Jul 2008 00:41:11 +0200 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.14 (Macintosh/20080421) MIME-Version: 1.0 To: Kris Kennaway References: <20080708100701.57031cda@twoflower.in.publishing.hu> <4873C4FA.2020004@FreeBSD.org> <4873E068.8060305@psg.com> <4873E0AE.6090305@psg.com> <4873ECE0.5090706@FreeBSD.org> In-Reply-To: <4873ECE0.5090706@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: Thinking of using ZFS/FBSD for a backup system X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jul 2008 22:41:07 -0000 Kris Kennaway wrote: > mike wrote: >> On 7/8/08, Randy Bush wrote: >> >>> . first cuppa. it's 4g >> >> okay, so it sounds like from multiple people this could be doable and >> stable enough: >> >> dual or quad-core current gen processor (amd64) >> 4g ram >> 4 or 6x 1tb disks >> freebsd 7.0-release >> >> these tweaks to /boot/loader.conf: >> >> vm.kmem_size=600M >> vm.kmem_size_max=600M >> vfs.zfs.prefetch_disable=1 >> >> right? > > Might be enough kmem, if not, or if you want better performance from > increased caching, increase as high as 1500M > > Kris > > Also test with/without prefetch to see which is faster on your workload. Kris From owner-freebsd-fs@FreeBSD.ORG Tue Jul 8 22:47:26 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5EBED106566B; Tue, 8 Jul 2008 22:47:26 +0000 (UTC) (envelope-from randy@psg.com) Received: from rip.psg.com (rip.psg.com [IPv6:2001:418:1::39]) by mx1.freebsd.org (Postfix) with ESMTP id 42B448FC24; Tue, 8 Jul 2008 22:47:26 +0000 (UTC) (envelope-from randy@psg.com) Received: from 50.216.138.210.bn.2iij.net ([210.138.216.50] helo=rmac.psg.com) by rip.psg.com with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.69 (FreeBSD)) (envelope-from ) id 1KGLy1-000JOf-W6; Tue, 08 Jul 2008 22:47:26 +0000 Message-ID: <4873EE7C.6030207@psg.com> Date: Wed, 09 Jul 2008 07:47:24 +0900 From: Randy Bush User-Agent: Thunderbird 2.0.0.14 (Macintosh/20080421) MIME-Version: 1.0 To: Kris Kennaway References: <20080708100701.57031cda@twoflower.in.publishing.hu> <4873C4FA.2020004@FreeBSD.org> <4873E068.8060305@psg.com> <4873E0AE.6090305@psg.com> <4873ECE0.5090706@FreeBSD.org> In-Reply-To: <4873ECE0.5090706@FreeBSD.org> X-Enigmail-Version: 0.95.6 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: Thinking of using ZFS/FBSD for a backup system X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jul 2008 22:47:26 -0000 >> vm.kmem_size=600M >> vm.kmem_size_max=600M >> vfs.zfs.prefetch_disable=1 > Might be enough kmem and where is the 'nuf-a-mometer? randy From owner-freebsd-fs@FreeBSD.ORG Tue Jul 8 23:14:45 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 13F841065673 for ; Tue, 8 Jul 2008 23:14:45 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 4ADC78FC12; Tue, 8 Jul 2008 23:14:44 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <4873F4E9.3040203@FreeBSD.org> Date: Wed, 09 Jul 2008 01:14:49 +0200 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.14 (Macintosh/20080421) MIME-Version: 1.0 To: CZUCZY Gergely References: <20080708100701.57031cda@twoflower.in.publishing.hu> <4873C4FA.2020004@FreeBSD.org> <20080708221327.5c1d0e92@mort.in.publishing.hu> <4873CF6C.7000205@FreeBSD.org> <20080708225449.1070252d@mort.in.publishing.hu> In-Reply-To: <20080708225449.1070252d@mort.in.publishing.hu> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: Thinking of using ZFS/FBSD for a backup system X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jul 2008 23:14:45 -0000 CZUCZY Gergely wrote: > On Tue, 08 Jul 2008 22:34:52 +0200 > Kris Kennaway wrote: >>> I've finetuned ZFS as much as I could, I've read every little tiny >>> bit of hint/information/whatever that was available and I couldn't >>> get rid of those kmem_size panics in -RELEASE and -STABLE. >> Well, it's still almost certainly because you aren't setting >> kmem_size high enough. As you saw, that is the only thing I tuned >> (disabling prefetch is just for performance in my environment). >> >> If you can't set it high enough because you don't have enough RAM, >> that means your system does't have enough RAM to run ZFS, not that >> ZFS is unstable. > I've had a box with 2GB of memory for it, and around 5-6 filesystems. > I've set kmem_size as large as it was allowed, not a bit smaller. > > Where's the guide showing how much memory should I have for a setup? > How can "enough memory" be determined for a setup, without having > panics? I don't know; empirically my setup is an upper bound. How large was "as large as it was allowed" for you? Kris From owner-freebsd-fs@FreeBSD.ORG Tue Jul 8 23:21:31 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 96DE61065687 for ; Tue, 8 Jul 2008 23:21:31 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 9E50F8FC17; Tue, 8 Jul 2008 23:21:30 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <4873F67F.6050202@FreeBSD.org> Date: Wed, 09 Jul 2008 01:21:35 +0200 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.14 (Macintosh/20080421) MIME-Version: 1.0 To: Randy Bush References: <20080708100701.57031cda@twoflower.in.publishing.hu> <4873C4FA.2020004@FreeBSD.org> <4873E068.8060305@psg.com> <4873E0AE.6090305@psg.com> <4873ECE0.5090706@FreeBSD.org> <4873EE7C.6030207@psg.com> In-Reply-To: <4873EE7C.6030207@psg.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: Thinking of using ZFS/FBSD for a backup system X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jul 2008 23:21:31 -0000 Randy Bush wrote: >>> vm.kmem_size=600M >>> vm.kmem_size_max=600M >>> vfs.zfs.prefetch_disable=1 >> Might be enough kmem > > and where is the 'nuf-a-mometer? The point at which it no longer panics from memory exhaustion ;) I've put no work into finding out exactly where this is, because my servers have gigabytes of memory and it is a performance optimization for me to set kmem as high as possible, which was previously just over 1500MB on amd64 but has now been increased in 8.0. Kris From owner-freebsd-fs@FreeBSD.ORG Wed Jul 9 02:25:54 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3BB961065671 for ; Wed, 9 Jul 2008 02:25:54 +0000 (UTC) (envelope-from speedtoys.racing@gmail.com) Received: from py-out-1112.google.com (py-out-1112.google.com [64.233.166.177]) by mx1.freebsd.org (Postfix) with ESMTP id E63988FC0A for ; Wed, 9 Jul 2008 02:25:53 +0000 (UTC) (envelope-from speedtoys.racing@gmail.com) Received: by py-out-1112.google.com with SMTP id p76so1380213pyb.10 for ; Tue, 08 Jul 2008 19:25:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=1HMgAQTxO+fCY3K5PmT3jWCiCXmiBhriD2Np+RQc7eM=; b=RT2UQ+8/6FNKoiNrw9AN9dwWv8hFj+ZMPOklRtluSkK5GQ+BHsUFnKNJE0hMEq3n3N leWvYuRqgm8OtZXZw0OOoEOj20SHMk4jvz8x4+RLnX0gXkOZnblmBNmKDa0zN3pU0911 jWsIu2t+GFUbv/Tq4Kw4bZ6iVBhs2gTCotp4g= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=arJWK+sYDApL/AxB6Lzvmjrbhjny5vrGVPhN84yAZkS/S6RBcNSUcJjqGUbB2DhtLF ++dpspR3F0hhrzNSKsUtBLJtRER9CcthwBwHLhaMNDmYbhUTCEojo8jjmRWaIc1BaI3q AZkcw1ZZeFhTwl5IAGT47+vQuqz0Lz4IRh604= Received: by 10.114.133.1 with SMTP id g1mr1788882wad.149.1215570352736; Tue, 08 Jul 2008 19:25:52 -0700 (PDT) Received: by 10.114.102.10 with HTTP; Tue, 8 Jul 2008 19:25:52 -0700 (PDT) Message-ID: Date: Tue, 8 Jul 2008 19:25:52 -0700 From: "Jeff Mohler" To: juri_mian@yahoo.com In-Reply-To: <96359.64292.qm@web45601.mail.sp1.yahoo.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <96359.64292.qm@web45601.mail.sp1.yahoo.com> Cc: freebsd-fs@freebsd.org Subject: Re: 24 TB UFS2 reality check ? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Jul 2008 02:25:54 -0000 One drive has a what..maybe a 1 per 1.0 E15 bits transferred uBER, and you have 24x that of one drive, as each drive it it's statistical crap shoot. Each drive may NEVER hit uBER for you, but one may do it tomorrow. Plus, you have commodity firmware levels on those drives and commodity BER mechanisms, so you COULD argue you have another 2x liability WRT losing it all without HEFTY raid, at least 5+1. Cuz..if you have RAID, and you lose 1 drive, you have to touch a lot of bits to recover that drive, which drives you quickly in to the mathematical region of yet another BER, then you have a double drive failure. Just saying, good luck And even back in 2002, uBER was only a point shorter, and that math was a LOT harder to hit on the much smaller drives/arrays. Let say you go bling for high end SATA drives, you only get about E16 at best, which still isnt good math with a 24TB FS...theoretically. And, not to mention the cubic inches of RAM required to manage it once you start to really fill it, much less fsck it. On Tue, Jul 8, 2008 at 7:04 PM, Juri Mianovich wrote: > > > > --- On Tue, 7/8/08, Jeff Mohler wrote: > > >> Wow..the odds of hitting an uncorrectable bit error are >> actually >> pretty HIGH in that configuration,. >> >> Good luck. >> >> You may get to find out why Enterprise arrays are >> expensive. > > > Can you elaborate ? Why do you say that ? > > Did people say the same thing circa 2002 when the first 1-2TB configurations were being put together with off the shelf parts, or is there something special in particular about >20 TB that I don't understand? > > Thanks. > > > > > From owner-freebsd-fs@FreeBSD.ORG Wed Jul 9 02:04:47 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 905D8106567E for ; Wed, 9 Jul 2008 02:04:47 +0000 (UTC) (envelope-from juri_mian@yahoo.com) Received: from n61.bullet.mail.sp1.yahoo.com (n61.bullet.mail.sp1.yahoo.com [98.136.44.37]) by mx1.freebsd.org (Postfix) with SMTP id 7370E8FC23 for ; Wed, 9 Jul 2008 02:04:47 +0000 (UTC) (envelope-from juri_mian@yahoo.com) Received: from [216.252.122.219] by n61.bullet.mail.sp1.yahoo.com with NNFMP; 09 Jul 2008 02:04:47 -0000 Received: from [69.147.84.109] by t4.bullet.sp1.yahoo.com with NNFMP; 09 Jul 2008 02:04:47 -0000 Received: from [127.0.0.1] by omp207.mail.sp1.yahoo.com with NNFMP; 09 Jul 2008 02:04:47 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 277561.4546.bm@omp207.mail.sp1.yahoo.com Received: (qmail 64671 invoked by uid 60001); 9 Jul 2008 02:04:47 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Received:X-Mailer:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Message-ID; b=IRAdVW0SBWaIp+RuqTTxWJ68y/JyvFEtYzI5FlSLOzxPb7t5ihSMVvUXD6KcDAfACGscLIxzoWtJAmgKfm/twqvKRMfVX4oqF2jW2c4qRuGST0ZkG9/gH8zARXdnYHC/rU02UUxVoO2DA5ZzRJlVPN1wEVb0YuKV2mbTEca/aKQ=; Received: from [71.63.232.32] by web45601.mail.sp1.yahoo.com via HTTP; Tue, 08 Jul 2008 19:04:46 PDT X-Mailer: YahooMailWebService/0.7.199 Date: Tue, 8 Jul 2008 19:04:46 -0700 (PDT) From: Juri Mianovich To: Jeff Mohler In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Message-ID: <96359.64292.qm@web45601.mail.sp1.yahoo.com> X-Mailman-Approved-At: Wed, 09 Jul 2008 02:28:11 +0000 Cc: freebsd-fs@freebsd.org Subject: Re: 24 TB UFS2 reality check ? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: juri_mian@yahoo.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Jul 2008 02:04:47 -0000 --- On Tue, 7/8/08, Jeff Mohler wrote: > Wow..the odds of hitting an uncorrectable bit error are > actually > pretty HIGH in that configuration,. > > Good luck. > > You may get to find out why Enterprise arrays are > expensive. Can you elaborate ? Why do you say that ? Did people say the same thing circa 2002 when the first 1-2TB configurations were being put together with off the shelf parts, or is there something special in particular about >20 TB that I don't understand? Thanks. From owner-freebsd-fs@FreeBSD.ORG Wed Jul 9 02:37:02 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 988051065673 for ; Wed, 9 Jul 2008 02:37:02 +0000 (UTC) (envelope-from marck@rinet.ru) Received: from woozle.rinet.ru (woozle.rinet.ru [195.54.192.68]) by mx1.freebsd.org (Postfix) with ESMTP id 115248FC15 for ; Wed, 9 Jul 2008 02:37:01 +0000 (UTC) (envelope-from marck@rinet.ru) Received: from localhost (localhost [127.0.0.1]) by woozle.rinet.ru (8.14.2/8.14.2) with ESMTP id m692QUP9051224; Wed, 9 Jul 2008 06:26:30 +0400 (MSD) (envelope-from marck@rinet.ru) Date: Wed, 9 Jul 2008 06:26:30 +0400 (MSD) From: Dmitry Morozovsky To: Kris Kennaway In-Reply-To: <4873C4FA.2020004@FreeBSD.org> Message-ID: <20080709062533.J58331@woozle.rinet.ru> References: <20080708100701.57031cda@twoflower.in.publishing.hu> <4873C4FA.2020004@FreeBSD.org> X-NCC-RegID: ru.rinet X-OpenPGP-Key-ID: 6B691B03 MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.0 (woozle.rinet.ru [0.0.0.0]); Wed, 09 Jul 2008 06:26:30 +0400 (MSD) Cc: freebsd-fs@freebsd.org Subject: Re: Thinking of using ZFS/FBSD for a backup system X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Jul 2008 02:37:02 -0000 On Tue, 8 Jul 2008, Kris Kennaway wrote: KK> (ports-i386:~)> uname -a KK> FreeBSD pointyhat.freebsd.org 8.0-CURRENT FreeBSD 8.0-CURRENT #31: Wed Jun KK> 25 19:40:40 UTC 2008 Wow! I did't realize you switched package building infrastructure to ZFS. Nice and promising! Sincerely, D.Marck [DM5020, MCK-RIPE, DM3-RIPN] [ FreeBSD committer: marck@FreeBSD.org ] ------------------------------------------------------------------------ *** Dmitry Morozovsky --- D.Marck --- Wild Woozle --- marck@rinet.ru *** ------------------------------------------------------------------------ From owner-freebsd-fs@FreeBSD.ORG Wed Jul 9 05:44:26 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A80321065671; Wed, 9 Jul 2008 05:44:26 +0000 (UTC) (envelope-from phoemix@harmless.hu) Received: from marvin.harmless.hu (marvin.harmless.hu [195.56.55.204]) by mx1.freebsd.org (Postfix) with ESMTP id 601278FC14; Wed, 9 Jul 2008 05:44:26 +0000 (UTC) (envelope-from phoemix@harmless.hu) Received: from pool-0123.adsl.interware.hu ([213.178.100.123] helo=mort.in.publishing.hu) by marvin.harmless.hu with esmtpa (Exim 4.69 (FreeBSD)) (envelope-from ) id 1KGSTY-0004h8-Uc; Wed, 09 Jul 2008 07:44:25 +0200 Date: Wed, 9 Jul 2008 07:44:20 +0200 From: CZUCZY Gergely To: Kris Kennaway Message-ID: <20080709074420.24df3be4@mort.in.publishing.hu> In-Reply-To: <4873F4E9.3040203@FreeBSD.org> References: <20080708100701.57031cda@twoflower.in.publishing.hu> <4873C4FA.2020004@FreeBSD.org> <20080708221327.5c1d0e92@mort.in.publishing.hu> <4873CF6C.7000205@FreeBSD.org> <20080708225449.1070252d@mort.in.publishing.hu> <4873F4E9.3040203@FreeBSD.org> Organization: Harmless Digital Bt X-Mailer: Claws Mail 3.3.1 (GTK+ 2.12.1; i386-portbld-freebsd6.3) Mime-Version: 1.0 Content-Type: multipart/signed; boundary="Sig_/.snRiPQzg1N3p=+iAohpBAk"; protocol="application/pgp-signature"; micalg=PGP-SHA1 Cc: freebsd-fs@freebsd.org Subject: Re: Thinking of using ZFS/FBSD for a backup system X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Jul 2008 05:44:26 -0000 --Sig_/.snRiPQzg1N3p=+iAohpBAk Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable On Wed, 09 Jul 2008 01:14:49 +0200 Kris Kennaway wrote: > CZUCZY Gergely wrote: > I don't know; empirically my setup is an upper bound. How large was > "as large as it was allowed" for you? Well, we cannot buy "upper bounds" all over, just because some developer is unable to figure out things. I think you can't expect FreeBSD users to spend as much money as possible, just because the devs can't tell how much is enough... It seems more like a twilight zone then a stable feature now ;) It was exactly as much as an amd64 installation would allow with 2GB of physical memory. We've dismissed the setup around february, and I don't have the configs anymore. It was an amd64 setup with 2GB of physical memory. >=20 > Kris --=20 Sincerely, Gergely CZUCZY, Harmless Digital mailto: gergely.czuczy@harmless.hu Legacy software is software that works. --Sig_/.snRiPQzg1N3p=+iAohpBAk Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQFIdFA3zrC0WyuMkpsRAgmZAKCNZW0PJMOGPzrOGxmIZVRyp3wDRwCeKEHl eZz2z5m7Wpmwavl62EbX09I= =jLRH -----END PGP SIGNATURE----- --Sig_/.snRiPQzg1N3p=+iAohpBAk-- From owner-freebsd-fs@FreeBSD.ORG Wed Jul 9 05:56:46 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 37F661065679; Wed, 9 Jul 2008 05:56:46 +0000 (UTC) (envelope-from jdc@parodius.com) Received: from mx01.sc1.parodius.com (mx01.sc1.parodius.com [72.20.106.3]) by mx1.freebsd.org (Postfix) with ESMTP id 221508FC17; Wed, 9 Jul 2008 05:56:45 +0000 (UTC) (envelope-from jdc@parodius.com) Received: by mx01.sc1.parodius.com (Postfix, from userid 1000) id 847061CC081; Tue, 8 Jul 2008 22:56:45 -0700 (PDT) Date: Tue, 8 Jul 2008 22:56:45 -0700 From: Jeremy Chadwick To: CZUCZY Gergely Message-ID: <20080709055645.GA40076@eos.sc1.parodius.com> References: <20080708100701.57031cda@twoflower.in.publishing.hu> <4873C4FA.2020004@FreeBSD.org> <20080708221327.5c1d0e92@mort.in.publishing.hu> <4873CF6C.7000205@FreeBSD.org> <20080708225449.1070252d@mort.in.publishing.hu> <4873F4E9.3040203@FreeBSD.org> <20080709074420.24df3be4@mort.in.publishing.hu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20080709074420.24df3be4@mort.in.publishing.hu> User-Agent: Mutt/1.5.18 (2008-05-17) Cc: freebsd-fs@freebsd.org, Kris Kennaway Subject: Re: Thinking of using ZFS/FBSD for a backup system X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Jul 2008 05:56:46 -0000 On Wed, Jul 09, 2008 at 07:44:20AM +0200, CZUCZY Gergely wrote: > On Wed, 09 Jul 2008 01:14:49 +0200 > Kris Kennaway wrote: > > > CZUCZY Gergely wrote: > > I don't know; empirically my setup is an upper bound. How large was > > "as large as it was allowed" for you? > Well, we cannot buy "upper bounds" all over, just because some > developer is unable to figure out things. I think you can't expect > FreeBSD users to spend as much money as possible, just because the devs > can't tell how much is enough... > It seems more like a twilight zone then a stable feature now ;) > > It was exactly as much as an amd64 installation would allow with 2GB of > physical memory. We've dismissed the setup around february, and I don't > have the configs anymore. It was an amd64 setup with 2GB of physical > memory. The bottom line here is that i386 and amd64 both have a kmem_size limit of 2GB. You can throw 32GB of RAM into an amd64 box, but FreeBSD will only utilise up to 2GB of that for kmem. That is purely a FreeBSD limitation, and is being dealt with in HEAD by Alan Cox. I believe he has a patch, or it may have been committed -- I don't follow HEAD. I can point people to a mailing list URL, if needed. This is one of the limitations Gergely is referring to. Since ZFS is incredibly memory-hungry, you're forced to tune ZFS to try and get it to "play nice" with that 2GB limit on STABLE/RELEASE systems. You also need to keep in mind that you can't just set kmem_size and kmem_size_max to 2048M, because the kernel needs memory for other things. The tuning parameters I use on my 2GB amd64 and 4GB amd64 boxes are: vm.kmem_size="1536M" vm.kmem_size_max="1536M" vfs.zfs.arc_min="16M" vfs.zfs.arc_max="64M" If you set kmem_size and kmem_size_max any higher than that, the machine will panic on boot, stating (indirectly) that there isn't enough memory available for the kernel to allocate for other things. Until I added the arc_min and arc_max setting, I could occasionally panic the machines under very heavy load (heavy zpool I/O), caused by kmem exhaustion. Since adding the arc_* tunings, I've tried very hard to crash the machines, and I cannot. But there's absolutely no guarantee those tuning parameters above will ensure FreeBSD won't panic due to kmem exhaustion. I believe this is the point Gergely is making about the "stability" of the whole thing. Now, with regards to prefetch_disable, folks can disable that if they want. I disable it on my above systems because for what they do, the overall performance appears better with prefetching disabled. I hope this helps shed some light here... -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Wed Jul 9 10:18:49 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F25001065671 for ; Wed, 9 Jul 2008 10:18:49 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id D3F268FC17; Wed, 9 Jul 2008 10:18:48 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <48749087.4070802@FreeBSD.org> Date: Wed, 09 Jul 2008 12:18:47 +0200 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.14 (Macintosh/20080421) MIME-Version: 1.0 To: Dmitry Morozovsky References: <20080708100701.57031cda@twoflower.in.publishing.hu> <4873C4FA.2020004@FreeBSD.org> <20080709062533.J58331@woozle.rinet.ru> In-Reply-To: <20080709062533.J58331@woozle.rinet.ru> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: Thinking of using ZFS/FBSD for a backup system X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Jul 2008 10:18:50 -0000 Dmitry Morozovsky wrote: > On Tue, 8 Jul 2008, Kris Kennaway wrote: > > KK> (ports-i386:~)> uname -a > KK> FreeBSD pointyhat.freebsd.org 8.0-CURRENT FreeBSD 8.0-CURRENT #31: Wed Jun > KK> 25 19:40:40 UTC 2008 > > Wow! I did't realize you switched package building infrastructure to ZFS. > > Nice and promising! Yeah, the server has been using ZFS since some time last year, but recently I went much further and made it make use of (i.e. require) ZFS features like snapshots and clones. Kris From owner-freebsd-fs@FreeBSD.ORG Wed Jul 9 10:26:09 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EB4DB1065678; Wed, 9 Jul 2008 10:26:09 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id CD06E8FC1B; Wed, 9 Jul 2008 10:26:08 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <4874923F.8080303@FreeBSD.org> Date: Wed, 09 Jul 2008 12:26:07 +0200 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.14 (Macintosh/20080421) MIME-Version: 1.0 To: Jeremy Chadwick References: <20080708100701.57031cda@twoflower.in.publishing.hu> <4873C4FA.2020004@FreeBSD.org> <20080708221327.5c1d0e92@mort.in.publishing.hu> <4873CF6C.7000205@FreeBSD.org> <20080708225449.1070252d@mort.in.publishing.hu> <4873F4E9.3040203@FreeBSD.org> <20080709074420.24df3be4@mort.in.publishing.hu> <20080709055645.GA40076@eos.sc1.parodius.com> In-Reply-To: <20080709055645.GA40076@eos.sc1.parodius.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: Thinking of using ZFS/FBSD for a backup system X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Jul 2008 10:26:10 -0000 Jeremy Chadwick wrote: > On Wed, Jul 09, 2008 at 07:44:20AM +0200, CZUCZY Gergely wrote: >> On Wed, 09 Jul 2008 01:14:49 +0200 >> Kris Kennaway wrote: >> >>> CZUCZY Gergely wrote: >>> I don't know; empirically my setup is an upper bound. How large was >>> "as large as it was allowed" for you? >> Well, we cannot buy "upper bounds" all over, just because some >> developer is unable to figure out things. I think you can't expect >> FreeBSD users to spend as much money as possible, just because the devs >> can't tell how much is enough... >> It seems more like a twilight zone then a stable feature now ;) >> >> It was exactly as much as an amd64 installation would allow with 2GB of >> physical memory. We've dismissed the setup around february, and I don't >> have the configs anymore. It was an amd64 setup with 2GB of physical >> memory. > > The bottom line here is that i386 and amd64 both have a kmem_size limit > of 2GB. No, it's the limit on KVA (address space), not kmem_size. On i386 there is only 1GB of KVA by default, so it's even harder to fit ZFS in. I thought you could tune it higher than 2GB if you liked, although this comes out of address space available to user programs (4GB total for user + kernel). > You can throw 32GB of RAM into an amd64 box, but FreeBSD will > only utilise up to 2GB of that for kmem. That is purely a FreeBSD > limitation, and is being dealt with in HEAD by Alan Cox. I believe he > has a patch, or it may have been committed -- I don't follow HEAD. I > can point people to a mailing list URL, if needed. > > This is one of the limitations Gergely is referring to. No it's not, since he has only 2GB of physical memory. > Since ZFS is incredibly memory-hungry, you're forced to tune ZFS to try > and get it to "play nice" with that 2GB limit on STABLE/RELEASE systems. > You also need to keep in mind that you can't just set kmem_size and > kmem_size_max to 2048M, because the kernel needs memory for other > things. > > The tuning parameters I use on my 2GB amd64 and 4GB amd64 boxes are: > > vm.kmem_size="1536M" > vm.kmem_size_max="1536M" > vfs.zfs.arc_min="16M" > vfs.zfs.arc_max="64M" > > If you set kmem_size and kmem_size_max any higher than that, the machine > will panic on boot, stating (indirectly) that there isn't enough memory > available for the kernel to allocate for other things. Yes, I said this earlier :) > Until I added the arc_min and arc_max setting, I could occasionally > panic the machines under very heavy load (heavy zpool I/O), caused by > kmem exhaustion. Since adding the arc_* tunings, I've tried very hard > to crash the machines, and I cannot. Good to hear. > But there's absolutely no guarantee those tuning parameters above will > ensure FreeBSD won't panic due to kmem exhaustion. I believe this is > the point Gergely is making about the "stability" of the whole thing. Not having the resources to run a very memory-intensive filesystem does not make it "unstable", it makes it "too memory intensive". Kris From owner-freebsd-fs@FreeBSD.ORG Wed Jul 9 10:37:37 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 00591106566C; Wed, 9 Jul 2008 10:37:36 +0000 (UTC) (envelope-from phoemix@harmless.hu) Received: from marvin.harmless.hu (marvin.harmless.hu [195.56.55.204]) by mx1.freebsd.org (Postfix) with ESMTP id A96248FC23; Wed, 9 Jul 2008 10:37:36 +0000 (UTC) (envelope-from phoemix@harmless.hu) Received: from fw.publishing.hu ([82.131.181.62] helo=twoflower.in.publishing.hu) by marvin.harmless.hu with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.69 (FreeBSD)) (envelope-from ) id 1KGX3G-000BJJ-V0; Wed, 09 Jul 2008 12:37:35 +0200 Date: Wed, 9 Jul 2008 12:37:29 +0200 From: CZUCZY Gergely To: Jeremy Chadwick Message-ID: <20080709123729.60d2431a@twoflower.in.publishing.hu> In-Reply-To: <20080709055645.GA40076@eos.sc1.parodius.com> References: <20080708100701.57031cda@twoflower.in.publishing.hu> <4873C4FA.2020004@FreeBSD.org> <20080708221327.5c1d0e92@mort.in.publishing.hu> <4873CF6C.7000205@FreeBSD.org> <20080708225449.1070252d@mort.in.publishing.hu> <4873F4E9.3040203@FreeBSD.org> <20080709074420.24df3be4@mort.in.publishing.hu> <20080709055645.GA40076@eos.sc1.parodius.com> Organization: Harmless Digital X-Mailer: Claws Mail 3.5.0 (GTK+ 2.12.11; i386-portbld-freebsd6.3) Mime-Version: 1.0 Content-Type: multipart/signed; boundary="Sig_/R3qcmpkSNJ2xBKy29soX7Ea"; protocol="application/pgp-signature"; micalg=PGP-SHA1 Sender: Czuczy Gergely Cc: freebsd-fs@freebsd.org, Kris Kennaway Subject: Re: Thinking of using ZFS/FBSD for a backup system X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Jul 2008 10:37:37 -0000 --Sig_/R3qcmpkSNJ2xBKy29soX7Ea Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Thank you for your thought, Jeremy. Yes, I was trying to refer to these thi= ngs. Putting more memory into the system then ZFS's need doesn't prove anything. There's no _proven_ _garantee_ that it won't panic, it just makes it more difficult (lowers the possibilty) to panic the system. As you said tuning ARC to a lower and kmem_size{,_max} to a higher value ma= kes it less likely to panic, but this won't garantee anything, just makes panic= ing bigger. "Stable ZFS" would mean, that these circumstances are cleared, and there's a proven garantee (either mathematically) that it's _unable_ to panic due to = this memory allocation issue. It's still there, but with a bigger amount of memory it's less likely to ha= ppen. And I haven't tried prefetch_disable back then. So i've got no experiences = on the effects of prefetch_disable. On Tue, 8 Jul 2008 22:56:45 -0700 Jeremy Chadwick wrote: > On Wed, Jul 09, 2008 at 07:44:20AM +0200, CZUCZY Gergely wrote: > > On Wed, 09 Jul 2008 01:14:49 +0200 > > Kris Kennaway wrote: > >=20 > > > CZUCZY Gergely wrote: > > > I don't know; empirically my setup is an upper bound. How large was > > > "as large as it was allowed" for you? > > Well, we cannot buy "upper bounds" all over, just because some > > developer is unable to figure out things. I think you can't expect > > FreeBSD users to spend as much money as possible, just because the devs > > can't tell how much is enough... > > It seems more like a twilight zone then a stable feature now ;) > >=20 > > It was exactly as much as an amd64 installation would allow with 2GB of > > physical memory. We've dismissed the setup around february, and I don't > > have the configs anymore. It was an amd64 setup with 2GB of physical > > memory. >=20 > The bottom line here is that i386 and amd64 both have a kmem_size limit > of 2GB. You can throw 32GB of RAM into an amd64 box, but FreeBSD will > only utilise up to 2GB of that for kmem. That is purely a FreeBSD > limitation, and is being dealt with in HEAD by Alan Cox. I believe he > has a patch, or it may have been committed -- I don't follow HEAD. I > can point people to a mailing list URL, if needed. >=20 > This is one of the limitations Gergely is referring to. >=20 > Since ZFS is incredibly memory-hungry, you're forced to tune ZFS to try > and get it to "play nice" with that 2GB limit on STABLE/RELEASE systems. > You also need to keep in mind that you can't just set kmem_size and > kmem_size_max to 2048M, because the kernel needs memory for other > things. >=20 > The tuning parameters I use on my 2GB amd64 and 4GB amd64 boxes are: >=20 > vm.kmem_size=3D"1536M" > vm.kmem_size_max=3D"1536M" > vfs.zfs.arc_min=3D"16M" > vfs.zfs.arc_max=3D"64M" >=20 > If you set kmem_size and kmem_size_max any higher than that, the machine > will panic on boot, stating (indirectly) that there isn't enough memory > available for the kernel to allocate for other things. >=20 > Until I added the arc_min and arc_max setting, I could occasionally > panic the machines under very heavy load (heavy zpool I/O), caused by > kmem exhaustion. Since adding the arc_* tunings, I've tried very hard > to crash the machines, and I cannot. >=20 > But there's absolutely no guarantee those tuning parameters above will > ensure FreeBSD won't panic due to kmem exhaustion. I believe this is > the point Gergely is making about the "stability" of the whole thing. >=20 > Now, with regards to prefetch_disable, folks can disable that if they > want. I disable it on my above systems because for what they do, the > overall performance appears better with prefetching disabled. >=20 > I hope this helps shed some light here... >=20 --=20 =C3=9Cdv=C3=B6lettel, Czuczy Gergely Harmless Digital Bt mailto: gergely.czuczy@harmless.hu Tel: +36-30-9702963 --Sig_/R3qcmpkSNJ2xBKy29soX7Ea Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.3 (FreeBSD) iD8DBQFIdJTrzrC0WyuMkpsRAoY2AJwM1zl32WGmIa4hh0vWga7X6ZrgUwCcDdtO Orwh0RNP8VsPdN2HlfwAqdw= =C73I -----END PGP SIGNATURE----- --Sig_/R3qcmpkSNJ2xBKy29soX7Ea-- From owner-freebsd-fs@FreeBSD.ORG Wed Jul 9 10:53:04 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4D19D106564A; Wed, 9 Jul 2008 10:53:04 +0000 (UTC) (envelope-from marck@rinet.ru) Received: from woozle.rinet.ru (woozle.rinet.ru [195.54.192.68]) by mx1.freebsd.org (Postfix) with ESMTP id C9DA68FC21; Wed, 9 Jul 2008 10:53:03 +0000 (UTC) (envelope-from marck@rinet.ru) Received: from localhost (localhost [127.0.0.1]) by woozle.rinet.ru (8.14.2/8.14.2) with ESMTP id m69Ar2Px063290; Wed, 9 Jul 2008 14:53:02 +0400 (MSD) (envelope-from marck@rinet.ru) Date: Wed, 9 Jul 2008 14:53:02 +0400 (MSD) From: Dmitry Morozovsky To: Kris Kennaway In-Reply-To: <48749087.4070802@FreeBSD.org> Message-ID: <20080709145010.Q58331@woozle.rinet.ru> References: <20080708100701.57031cda@twoflower.in.publishing.hu> <4873C4FA.2020004@FreeBSD.org> <20080709062533.J58331@woozle.rinet.ru> <48749087.4070802@FreeBSD.org> X-NCC-RegID: ru.rinet X-OpenPGP-Key-ID: 6B691B03 MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.0 (woozle.rinet.ru [0.0.0.0]); Wed, 09 Jul 2008 14:53:02 +0400 (MSD) Cc: freebsd-fs@freebsd.org Subject: Re: Thinking of using ZFS/FBSD for a backup system X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Jul 2008 10:53:04 -0000 On Wed, 9 Jul 2008, Kris Kennaway wrote: KK> > KK> (ports-i386:~)> uname -a KK> > KK> FreeBSD pointyhat.freebsd.org 8.0-CURRENT FreeBSD 8.0-CURRENT #31: Wed KK> > Jun KK> > KK> 25 19:40:40 UTC 2008 KK> > KK> > Wow! I did't realize you switched package building infrastructure to ZFS. KK> > KK> > Nice and promising! KK> KK> Yeah, the server has been using ZFS since some time last year, but recently KK> I went much further and made it make use of (i.e. require) ZFS features like KK> snapshots and clones. Is it documented somewhere? Hmm, well. marck@wizzle:/usr/ports> grep -Ril zfs Tools/ Tools/portbuild/scripts/claim-chroot Tools/portbuild/scripts/clean-chroot Tools/portbuild/scripts/cleanup-chroots ;-) Any pitfalls while using this? Thanks! Sincerely, D.Marck [DM5020, MCK-RIPE, DM3-RIPN] [ FreeBSD committer: marck@FreeBSD.org ] ------------------------------------------------------------------------ *** Dmitry Morozovsky --- D.Marck --- Wild Woozle --- marck@rinet.ru *** ------------------------------------------------------------------------ From owner-freebsd-fs@FreeBSD.ORG Wed Jul 9 11:17:10 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 105FA1065684; Wed, 9 Jul 2008 11:17:10 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 103F78FC12; Wed, 9 Jul 2008 11:17:05 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <48749E2E.40308@FreeBSD.org> Date: Wed, 09 Jul 2008 13:17:02 +0200 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.14 (Macintosh/20080421) MIME-Version: 1.0 To: CZUCZY Gergely References: <20080708100701.57031cda@twoflower.in.publishing.hu> <4873C4FA.2020004@FreeBSD.org> <20080708221327.5c1d0e92@mort.in.publishing.hu> <4873CF6C.7000205@FreeBSD.org> <20080708225449.1070252d@mort.in.publishing.hu> <4873F4E9.3040203@FreeBSD.org> <20080709074420.24df3be4@mort.in.publishing.hu> <20080709055645.GA40076@eos.sc1.parodius.com> <20080709123729.60d2431a@twoflower.in.publishing.hu> In-Reply-To: <20080709123729.60d2431a@twoflower.in.publishing.hu> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, Jeremy Chadwick Subject: Re: Thinking of using ZFS/FBSD for a backup system X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Jul 2008 11:17:10 -0000 CZUCZY Gergely wrote: > "Stable ZFS" would mean, that these circumstances are cleared, and there's a > proven garantee (either mathematically) that it's _unable_ to panic due to this > memory allocation issue. I suppose you can choose to use this definition if you like, but it must be kind of terrifying to live in a world where all but the most trivial of programs are "unstable" and MIGHT CRASH AT ANY MOMENT OH GOD NO. While technically true, I don't think it's a functionally useful definition to equate "stable" with "proven to be perfect", so I won't continue to debate the point. ZFS is what it is, several of us have shown that it is possible to tune memory parameters to make it fit into a FreeBSD kernel, and users can either take that for what it's worth, or decide that ZFS is not for them. Kris From owner-freebsd-fs@FreeBSD.ORG Wed Jul 9 11:50:54 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9B787106567A; Wed, 9 Jul 2008 11:50:54 +0000 (UTC) (envelope-from randy@psg.com) Received: from rip.psg.com (rip.psg.com [IPv6:2001:418:1::39]) by mx1.freebsd.org (Postfix) with ESMTP id 79C298FC1D; Wed, 9 Jul 2008 11:50:54 +0000 (UTC) (envelope-from randy@psg.com) Received: from 50.216.138.210.bn.2iij.net ([210.138.216.50] helo=rmac.psg.com) by rip.psg.com with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.69 (FreeBSD)) (envelope-from ) id 1KGYCD-000LSt-Mo; Wed, 09 Jul 2008 11:50:53 +0000 Message-ID: <4874A61C.5060805@psg.com> Date: Wed, 09 Jul 2008 20:50:52 +0900 From: Randy Bush User-Agent: Thunderbird 2.0.0.14 (Macintosh/20080421) MIME-Version: 1.0 To: CZUCZY Gergely References: <20080708100701.57031cda@twoflower.in.publishing.hu> <4873C4FA.2020004@FreeBSD.org> <20080708221327.5c1d0e92@mort.in.publishing.hu> <4873CF6C.7000205@FreeBSD.org> <20080708225449.1070252d@mort.in.publishing.hu> <4873F4E9.3040203@FreeBSD.org> <20080709074420.24df3be4@mort.in.publishing.hu> <20080709055645.GA40076@eos.sc1.parodius.com> <20080709123729.60d2431a@twoflower.in.publishing.hu> In-Reply-To: <20080709123729.60d2431a@twoflower.in.publishing.hu> X-Enigmail-Version: 0.95.6 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, Jeremy Chadwick , Kris Kennaway Subject: Re: Thinking of using ZFS/FBSD for a backup system X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Jul 2008 11:50:54 -0000 > There's no _proven_ _garantee_ that it won't panic go back to computability 301. the complexity of the systems we are all talking about are many many orders of magnitude beyond anything for which we can [dis]prove simple termination. so wind the rhetoric down a notch, please if you want the only guaranteed state people here can give you, unplug your machine. otherwise, you'll just have to accept experience, empirical testing, vetted advice, and wisdom. randy From owner-freebsd-fs@FreeBSD.ORG Wed Jul 9 13:34:17 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 17F2F1065670 for ; Wed, 9 Jul 2008 13:34:17 +0000 (UTC) (envelope-from lopez.on.the.lists@yellowspace.net) Received: from mail.yellowspace.net (mail.yellowspace.net [80.190.200.164]) by mx1.freebsd.org (Postfix) with ESMTP id A51508FC1A for ; Wed, 9 Jul 2008 13:34:16 +0000 (UTC) (envelope-from lopez.on.the.lists@yellowspace.net) Received: from five.intranet ([88.217.71.135]) (AUTH: LOGIN lopez.on.the.lists@yellowspace.net) by mail.yellowspace.net with esmtp; Wed, 09 Jul 2008 15:34:14 +0200 id 00330909.000000004874BE56.00008961 Message-Id: <743AA903-03CB-4F43-A30B-9D06C58A4EAC@yellowspace.net> From: Lorenzo Perone To: mike In-Reply-To: Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v924) Date: Wed, 9 Jul 2008 15:34:14 +0200 References: X-Mailer: Apple Mail (2.924) Cc: freebsd-fs@freebsd.org Subject: Re: Thinking of using ZFS/FBSD for a backup system X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Jul 2008 13:34:17 -0000 On 08.07.2008, at 08:15, mike wrote: > Just wanted to sanity check here before investing some money and time > into this solution... > > Also if anyone wants to reply to me off list with hardware that works > well for FBSD 7 + ZFS I'd be grateful :) On 09.07.2008, at 13:17, Kris Kennaway wrote: > ZFS is what it is, several of us have shown that it is possible to > tune memory parameters to make it fit into a FreeBSD kernel, and > users can either take that for what it's worth, or decide that ZFS > is not for them. > > Kris I know I'm definitively NOT going to make any friends here this way ;) BUT: At the present time, my impression is that if you need/want to put anything business-critical on a ZFS pool: go opensolaris. I got to know it on a SUN box, at least the bit to run and compile a few things, and it was definitively a pain compared to the good old FreeBSD hier(7). But for as far as zfs goes, I can sleep at night. Not that I wouldn't prefer the other way around, I'm one of those whom others have to stop before he starts installing FreeBSD on everything that has a CPU and runs as a server ;) I'm sure zfs on FreeBSD will be at least as stable (and, easilly, even better performing) as soon as pjb has the time to share his newest patches. I'll be one of the first csup'ping on RELENG_7 as soon as it's done. It's a way too sexy filesystem not to do so, and that's the reason why this kind of threads pop up regularly... As for my current experience, I have a box that backs up two offices every night and snapshots several filesystems, which is now running stably a zfs pool since about 2 months (which is almost a record for that box). It's a 7.0-STABLE #8 Sat Apr 26 10:10:53 CEST 2008, amd64 with 2 GB of RAM and the following in /boot/loader.conf: vm.kmem_size=900M vm.kmem_size_max=900M vfs.zfs.arc_max=300M vfs.zfs.prefetch_disable=1 However I'm happy that it is down the corridor, so if the whole thing gets stuck again, or panics, I can still watch it while it reboots on the console... BTW I can't recommend to rely on ufs snapshots while waiting for zfs to become stable - I tried and it almost destroyed that filesystem after a few runs (besides, it takes ages to finish a snapshot, compared to zfs). Regards, (and long live FreeBSD, it's half my life atm) Lorenzo From owner-freebsd-fs@FreeBSD.ORG Wed Jul 9 14:24:01 2008 Return-Path: Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F19321065693; Wed, 9 Jul 2008 14:24:01 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (unknown [IPv6:2a01:170:102f::2]) by mx1.freebsd.org (Postfix) with ESMTP id 69A768FC30; Wed, 9 Jul 2008 14:24:01 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (localhost [127.0.0.1]) by lurza.secnetix.de (8.14.1/8.14.1) with ESMTP id m69ENtj0075768; Wed, 9 Jul 2008 16:23:59 +0200 (CEST) (envelope-from oliver.fromme@secnetix.de) Received: (from olli@localhost) by lurza.secnetix.de (8.14.1/8.14.1/Submit) id m69ENtJM075767; Wed, 9 Jul 2008 16:23:55 +0200 (CEST) (envelope-from olli) Date: Wed, 9 Jul 2008 16:23:55 +0200 (CEST) Message-Id: <200807091423.m69ENtJM075767@lurza.secnetix.de> From: Oliver Fromme To: freebsd-fs@FreeBSD.ORG, kaluna@gmail.com, brooks@FreeBSD.ORG In-Reply-To: <20080707154805.GA57420@lor.one-eyed-alien.net> X-Newsgroups: list.freebsd-fs User-Agent: tin/1.8.3-20070201 ("Scotasay") (UNIX) (FreeBSD/6.2-STABLE-20070808 (i386)) MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-2.1.2 (lurza.secnetix.de [127.0.0.1]); Wed, 09 Jul 2008 16:24:00 +0200 (CEST) Cc: Subject: Re: Filesystem is not clean - run fsck X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: freebsd-fs@FreeBSD.ORG, kaluna@gmail.com, brooks@FreeBSD.ORG List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Jul 2008 14:24:02 -0000 Brooks Davis wrote: > On Sat, Jul 05, 2008 at 10:58:33AM +0200, Carlos Luna wrote: > > Hi I'd used freenas about 5 years without any problem. Now I can?t mount my > > raid volume and in his sourceforge forums seems they cant help me. Hope this > > list is the right list for my issue. > > > > When I try to fsck,I get: > > casa:/dev# fsck -t ufs -y /dev/pst0s1 > > ** /dev/pst0s1 > > ** Last Mounted on /mnt/raid > > ** Phase 1 - Check Blocks and Sizes > > -4439300862985009506 BAD I=86 > > 3443570138036206556 BAD I=86 > > -7476842757969057647 BAD I=86 > > -8078484667502176485 BAD I=86 > > 2249916482063805839 BAD I=86 > > -3291681609520367063 BAD I=86 > > 7780434385339928353 BAD I=86 > > -4372486048108189431 BAD I=86 > > 8774078035736727371 BAD I=86 > > -2035310265760485777 BAD I=86 > > 6848295312539782814 BAD I=86 > > EXCESSIVE BAD BLKS I=86 > > CONTINUE? yes > > > > ... > > .... > > > > UNKNOWN FILE TYPE I=7254140 > > CLEAR? yes > > > > UNKNOWN FILE TYPE I=7254141 > > CLEAR? yes > > > > UNKNOWN FILE TYPE I=7254142 > > CLEAR? yes > > > > UNKNOWN FILE TYPE I=7254143 > > CLEAR? yes > > > > fsck_ufs: cannot alloc 3037795832 bytes for inoinfo > > I have a lot of info there, 1 TB. I will appreciate any help. > > It looks like you have a somewhat large file system, apparently with a > lot of small files on it. The message indicates that you need to be > able to allocate over 3GB of address space to handle this. As such you > will need a 64-bit machine, ideally with 4GB or more RAM and probably > with a large swap partition. > > In theory it should be possible to write a constrained memory use version of > fsck, but to my knowledge no one has done so and I suspect it would be a > time consuming development effort. There's another possibility. I remember cases where the FS structures were damaged in a way that fsck picked up wrong size information, and then tried to allocate ridiculously large amounts of memory, even for a small file system. That could be the case here, too. Best regards Oliver -- Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M. Handelsregister: Registergericht Muenchen, HRA 74606, Geschäftsfuehrung: secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün- chen, HRB 125758, Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd "And believe me, as a C++ programmer, I don't hesitate to question the decisions of language designers. After a decent amount of C++ exposure, Python's flaws seem ridiculously small." -- Ville Vainio From owner-freebsd-fs@FreeBSD.ORG Wed Jul 9 19:13:05 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D66FE1065674 for ; Wed, 9 Jul 2008 19:13:05 +0000 (UTC) (envelope-from juri_mian@yahoo.com) Received: from n59.bullet.mail.sp1.yahoo.com (n59.bullet.mail.sp1.yahoo.com [98.136.44.43]) by mx1.freebsd.org (Postfix) with SMTP id B5E778FC17 for ; Wed, 9 Jul 2008 19:13:05 +0000 (UTC) (envelope-from juri_mian@yahoo.com) Received: from [216.252.122.219] by n59.bullet.mail.sp1.yahoo.com with NNFMP; 09 Jul 2008 19:13:05 -0000 Received: from [69.147.65.163] by t4.bullet.sp1.yahoo.com with NNFMP; 09 Jul 2008 19:13:05 -0000 Received: from [127.0.0.1] by omp408.mail.sp1.yahoo.com with NNFMP; 09 Jul 2008 19:13:05 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 348093.7952.bm@omp408.mail.sp1.yahoo.com Received: (qmail 22482 invoked by uid 60001); 9 Jul 2008 19:13:05 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Received:X-Mailer:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Message-ID; b=Pob6KcNkmwmJFaqhut9th8vxN1PG75PZDFukzLs/7PW1lrshBI5vtE+xkOmwAx0x4WTbhEAzhF8lMFuXga6gj8Qrqew08av43APzwOuFvJuqxoY8LKoxhwsOcaNGqvyFMWeqz+BLDOgFIOURI6OisBDDlG9J7LkmIFmZUeUWZwU=; Received: from [71.63.232.32] by web45601.mail.sp1.yahoo.com via HTTP; Wed, 09 Jul 2008 12:13:04 PDT X-Mailer: YahooMailWebService/0.7.199 Date: Wed, 9 Jul 2008 12:13:04 -0700 (PDT) From: Juri Mianovich To: Jeff Mohler In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Message-ID: <947384.22013.qm@web45601.mail.sp1.yahoo.com> X-Mailman-Approved-At: Wed, 09 Jul 2008 19:41:35 +0000 Cc: freebsd-fs@freebsd.org Subject: Re: 24 TB UFS2 reality check ? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: juri_mian@yahoo.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Jul 2008 19:13:05 -0000 Hello Jeff, --- On Tue, 7/8/08, Jeff Mohler wrote: > One drive has a what..maybe a 1 per 1.0 E15 bits transferred > uBER, and > you have 24x that of one drive, as each drive it it's > statistical crap > shoot. Each drive may NEVER hit uBER for you, but one may > do it > tomorrow. > > Plus, you have commodity firmware levels on those drives > and commodity > BER mechanisms, so you COULD argue you have another 2x > liability WRT > losing it all without HEFTY raid, at least 5+1. Thank you - I understand. You are worried because of the lack of redundancy. I didn't want to make my questions any more complicated than they were, but since we are on the topic, I will tell you that _in reality_ I will not make a 24 TB array, I will in fact use the raid-6 functionality (two parity drives) of my card and make a ~22 TB array. Does that address the concerns you were raising ? Does 22 data and 2 parity (raid 6) still make you very nervous, or does that completely change the scenario you were worried about ? Thanks. From owner-freebsd-fs@FreeBSD.ORG Thu Jul 10 01:02:19 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D796E1065685 for ; Thu, 10 Jul 2008 01:02:19 +0000 (UTC) (envelope-from speedtoys.racing@gmail.com) Received: from wa-out-1112.google.com (wa-out-1112.google.com [209.85.146.183]) by mx1.freebsd.org (Postfix) with ESMTP id 9D81D8FC14 for ; Thu, 10 Jul 2008 01:02:19 +0000 (UTC) (envelope-from speedtoys.racing@gmail.com) Received: by wa-out-1112.google.com with SMTP id j4so1773931wah.3 for ; Wed, 09 Jul 2008 18:02:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=Xq1gRwU6rCNh1V/uVdOmD91GscJdQihG2HDqo5s6Ev0=; b=QcsqAl9o89fGeKurOPUAqn6Osv6q69faTW/Ae7WnpN3lyaUSQsgOJ8Hm1HPh56x/l2 eIhUtjHZwpKA0YREFcwSgTVCyMPeKVZjMzNZlDdSCrUQe+sevqI28NNWJdivzEDN1kgg sRIJFIQUgG6lOHmboaqBts1Nvy0gs3JcwmJWs= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=nT4rhb/5mIct1SZgMT2bv+XDh6uNLOFcevgQOrLFTdiojuudJbxeQs5zoJCU94zUDz Ip4fHLpDzXhLuZOiCCHGRHHAg7OAqIm2s3/53ZrK1DFzZhxJmW92JXUN8dhWeyqkb8Au 3wk1zd6/qUHf4vs+XO6XJd8AWj55no4h3my2k= Received: by 10.115.32.8 with SMTP id k8mr10218934waj.89.1215651738994; Wed, 09 Jul 2008 18:02:18 -0700 (PDT) Received: by 10.114.102.10 with HTTP; Wed, 9 Jul 2008 18:02:18 -0700 (PDT) Message-ID: Date: Wed, 9 Jul 2008 18:02:18 -0700 From: "Jeff Mohler" To: "Alexandre Biancalana" In-Reply-To: <8e10486b0807091759m7cf4a04dsa4538594bc4a9304@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <947384.22013.qm@web45601.mail.sp1.yahoo.com> <8e10486b0807091759m7cf4a04dsa4538594bc4a9304@mail.gmail.com> Cc: freebsd-fs@freebsd.org, juri_mian@yahoo.com Subject: Re: 24 TB UFS2 reality check ? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Jul 2008 01:02:19 -0000 Lets see..a peak of maybe 25-30 random drive IOPS/sec at 15ms MINIMAL latency per IO (likely more like 35-40)..gonna be ugly. Complicated by normal load IOPS..you could expect it all to simply "dissapear" for a day while it reconstructs. On Wed, Jul 9, 2008 at 5:59 PM, Alexandre Biancalana wrote: > On 7/9/08, Juri Mianovich wrote: >> >> Hello Jeff, >> >> >> --- On Tue, 7/8/08, Jeff Mohler wrote: >> >> >> >> > One drive has a what..maybe a 1 per 1.0 E15 bits transferred >> > uBER, and >> > you have 24x that of one drive, as each drive it it's >> > statistical crap >> > shoot. Each drive may NEVER hit uBER for you, but one may >> > do it >> > tomorrow. >> > >> > Plus, you have commodity firmware levels on those drives >> > and commodity >> > BER mechanisms, so you COULD argue you have another 2x >> > liability WRT >> > losing it all without HEFTY raid, at least 5+1. >> >> >> >> Thank you - I understand. You are worried because of the lack of redundancy. >> >> I didn't want to make my questions any more complicated than they were, but since we are on the topic, I will tell you that _in reality_ I will not make a 24 TB array, I will in fact use the raid-6 functionality (two parity drives) of my card and make a ~22 TB array. >> >> Does that address the concerns you were raising ? Does 22 data and 2 parity (raid 6) still make you very nervous, or does that completely change the scenario you were worried about ? > > > I did be fewer nervous if you do 2 arrays of 11 disks... what`s the > time that it will take to do a rebuild of a failed drive in your > normal load ?? > From owner-freebsd-fs@FreeBSD.ORG Thu Jul 10 01:28:56 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1F7B21065672 for ; Thu, 10 Jul 2008 01:28:56 +0000 (UTC) (envelope-from biancalana@gmail.com) Received: from rv-out-0506.google.com (rv-out-0506.google.com [209.85.198.231]) by mx1.freebsd.org (Postfix) with ESMTP id DAEAB8FC19 for ; Thu, 10 Jul 2008 01:28:55 +0000 (UTC) (envelope-from biancalana@gmail.com) Received: by rv-out-0506.google.com with SMTP id b25so3982851rvf.43 for ; Wed, 09 Jul 2008 18:28:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=PooNuJZX0atfBXbT4iISUp3bJ+230xffmU7Bs0qnDUQ=; b=UpHjxsX0CZ+qfAOWiWZFZ7zh85Je/UKQiuAZPSHb/OMItOi1xGfqCq9UCXAiGcvR0z ENRxV7n8ReWfGSQ3XnJYyIvShVOPxueF+jykGVSVNuidsx8ACgb9kzPUVa+5AEQPtn31 Rbs/X+0djZwAW1NrCiRIDmzhAXyUFMaGIt1Bw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=Hm1SkYTj158r8ZnpwAzPwFOKJkEW9eTL+TQCVKCye4cCz/LDHt48rBv0sn7bFz1FS2 nu0l3bpKqGDekCqME9wNQVjhsjvdXVyb6BfOMGDQtXCslEPSapnw9a4ZaHgIOMpEC+Hv cAZMDCzJgWSf0s054OxEfn42enRnH4e3VLbCA= Received: by 10.140.127.13 with SMTP id z13mr4400035rvc.194.1215651594634; Wed, 09 Jul 2008 17:59:54 -0700 (PDT) Received: by 10.141.114.16 with HTTP; Wed, 9 Jul 2008 17:59:54 -0700 (PDT) Message-ID: <8e10486b0807091759m7cf4a04dsa4538594bc4a9304@mail.gmail.com> Date: Wed, 9 Jul 2008 21:59:54 -0300 From: "Alexandre Biancalana" To: juri_mian@yahoo.com In-Reply-To: <947384.22013.qm@web45601.mail.sp1.yahoo.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <947384.22013.qm@web45601.mail.sp1.yahoo.com> Cc: freebsd-fs@freebsd.org Subject: Re: 24 TB UFS2 reality check ? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Jul 2008 01:28:56 -0000 On 7/9/08, Juri Mianovich wrote: > > Hello Jeff, > > > --- On Tue, 7/8/08, Jeff Mohler wrote: > > > > > One drive has a what..maybe a 1 per 1.0 E15 bits transferred > > uBER, and > > you have 24x that of one drive, as each drive it it's > > statistical crap > > shoot. Each drive may NEVER hit uBER for you, but one may > > do it > > tomorrow. > > > > Plus, you have commodity firmware levels on those drives > > and commodity > > BER mechanisms, so you COULD argue you have another 2x > > liability WRT > > losing it all without HEFTY raid, at least 5+1. > > > > Thank you - I understand. You are worried because of the lack of redundancy. > > I didn't want to make my questions any more complicated than they were, but since we are on the topic, I will tell you that _in reality_ I will not make a 24 TB array, I will in fact use the raid-6 functionality (two parity drives) of my card and make a ~22 TB array. > > Does that address the concerns you were raising ? Does 22 data and 2 parity (raid 6) still make you very nervous, or does that completely change the scenario you were worried about ? I did be fewer nervous if you do 2 arrays of 11 disks... what`s the time that it will take to do a rebuild of a failed drive in your normal load ?? From owner-freebsd-fs@FreeBSD.ORG Thu Jul 10 15:55:19 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BBB77106566C for ; Thu, 10 Jul 2008 15:55:19 +0000 (UTC) (envelope-from juri_mian@yahoo.com) Received: from n68.bullet.mail.sp1.yahoo.com (n68.bullet.mail.sp1.yahoo.com [98.136.44.44]) by mx1.freebsd.org (Postfix) with SMTP id AB2E98FC1D for ; Thu, 10 Jul 2008 15:55:19 +0000 (UTC) (envelope-from juri_mian@yahoo.com) Received: from [216.252.122.217] by n68.bullet.mail.sp1.yahoo.com with NNFMP; 10 Jul 2008 15:55:18 -0000 Received: from [69.147.65.166] by t2.bullet.sp1.yahoo.com with NNFMP; 10 Jul 2008 15:55:18 -0000 Received: from [127.0.0.1] by omp501.mail.sp1.yahoo.com with NNFMP; 10 Jul 2008 15:55:18 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 867558.34047.bm@omp501.mail.sp1.yahoo.com Received: (qmail 96804 invoked by uid 60001); 10 Jul 2008 15:55:18 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Received:X-Mailer:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Message-ID; b=EC0rfusi1hdLwcTJugoFYxJozS/AkQYnTeKbG4XHXeNnqIHM0zUQ3IzOaNihLQ49gHZT/qwwPQIuBk37EJGzPGZ3kR0/w4wQJh1INl5KFXoCXq/gog6At8qOkmbeGeLNAcE4huiv/mopn0npow1HMZ+DqLfARb7q1omUlz6czts=; Received: from [71.63.232.32] by web45607.mail.sp1.yahoo.com via HTTP; Thu, 10 Jul 2008 08:55:17 PDT X-Mailer: YahooMailWebService/0.7.199 Date: Thu, 10 Jul 2008 08:55:17 -0700 (PDT) From: Juri Mianovich To: Alexandre Biancalana , Jeff Mohler In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Message-ID: <223496.96060.qm@web45607.mail.sp1.yahoo.com> X-Mailman-Approved-At: Thu, 10 Jul 2008 16:12:44 +0000 Cc: freebsd-fs@freebsd.org Subject: Re: 24 TB UFS2 reality check ? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: juri_mian@yahoo.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Jul 2008 15:55:19 -0000 Jeff, --- On Wed, 7/9/08, Jeff Mohler wrote: > Lets see..a peak of maybe 25-30 random drive IOPS/sec at > 15ms MINIMAL > latency per IO (likely more like 35-40)..gonna be ugly. > > Complicated by normal load IOPS..you could expect it all to > simply > "dissapear" for a day while it reconstructs. Once again, thank you very much - your comments are very helpful. So we've moved from "dangerous" (24 TB with no raid) to "inconvenient" (24 TB with raid 6). Two final questions: 1. What would _you_ do with 24 1 TB disks and a 24 port 3ware card ? Assume an i386, 4 GB machine, and that fsck is workable because of "newfs -i 131072" 2. What number should I ask my vendor (3ware) to do the rebuild calculations ? You are talking about IOPS/s - I think I should ask them how many IOPS/s the card does when rebuilding a 24 disk raid-6 array, and then combine that with the IOPS/s I see in my normal workload. How do you measure IOPS/s in FreeBSD on a running machine ? And, of course, any other comments appreciated. Thanks. From owner-freebsd-fs@FreeBSD.ORG Thu Jul 10 16:00:40 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7523E10656C3 for ; Thu, 10 Jul 2008 16:00:40 +0000 (UTC) (envelope-from juri_mian@yahoo.com) Received: from n77.bullet.mail.sp1.yahoo.com (n77.bullet.mail.sp1.yahoo.com [98.136.44.45]) by mx1.freebsd.org (Postfix) with SMTP id 4C5FF8FC0A for ; Thu, 10 Jul 2008 16:00:40 +0000 (UTC) (envelope-from juri_mian@yahoo.com) Received: from [216.252.122.216] by n77.bullet.mail.sp1.yahoo.com with NNFMP; 10 Jul 2008 16:00:40 -0000 Received: from [69.147.84.114] by t1.bullet.sp1.yahoo.com with NNFMP; 10 Jul 2008 16:00:40 -0000 Received: from [127.0.0.1] by omp203.mail.sp1.yahoo.com with NNFMP; 10 Jul 2008 16:00:40 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 47494.97793.bm@omp203.mail.sp1.yahoo.com Received: (qmail 22939 invoked by uid 60001); 10 Jul 2008 16:00:39 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Received:X-Mailer:Date:From:Reply-To:Subject:To:MIME-Version:Content-Type:Message-ID; b=yvtnXdJ4MEFmtN4Vq/9QiGbGlpEzRrb9e8ezK+MZJwBBwqGTxZ0gEqsfWoMqrjaKcGIxiS2KPBnaiZD8RGNrPnIBpANwG98N4hTrDo8at0lHA/rqb1MwlVEW5hZaxJNeBYE/8uQFVVLuULQwVR1zFnFlGNr8lGboqseubUZiyyA=; Received: from [71.63.232.32] by web45604.mail.sp1.yahoo.com via HTTP; Thu, 10 Jul 2008 09:00:39 PDT X-Mailer: YahooMailWebService/0.7.199 Date: Thu, 10 Jul 2008 09:00:39 -0700 (PDT) From: Juri Mianovich To: freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Message-ID: <806386.22750.qm@web45604.mail.sp1.yahoo.com> X-Mailman-Approved-At: Thu, 10 Jul 2008 16:12:55 +0000 Subject: the quota question ... one user with >2 TB owned files (but no quota set) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: juri_mian@yahoo.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Jul 2008 16:00:40 -0000 I am going to be running a large array. I will have quotas in the kernel and enabled BUT all users I set quotas on will be nowhere near the 2TB barrier I see people talking about recently. HOWEVER, at some point in the future, root or www (or both) users will _own more than_ 2 TB of files. They will not have a quota set on them, but they will in fact own >2 TB of files. Is this also a problem ? Or is the only problem actually _setting_ a quota larger than 2TB ? I assume the output in "repquota /my/fs" will be broken, and that is fine with me - I just don't want to corrupt or damage my filesystem (or existing quotas) the day that my www user goes over 2TB of owned files. Also, I am distrustful of merely testing this - just because things run fine for a day with quotas turned on and some user owning more than 2 TB of files doesn't mean it won't blow up at some future date in some interesting scenario - and that is why I am asking for opinions here rather than just creating >2 TB of files and turning on quotas. Does anyone out there already do this and can reassure me ? Thanks a lot. From owner-freebsd-fs@FreeBSD.ORG Thu Jul 10 17:25:25 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8BF23106567A for ; Thu, 10 Jul 2008 17:25:25 +0000 (UTC) (envelope-from gpalmer@freebsd.org) Received: from noop.in-addr.com (in-addr.broker.freenet6.net [IPv6:2001:5c0:8fff:fffe::214d]) by mx1.freebsd.org (Postfix) with ESMTP id 43AF18FC25 for ; Thu, 10 Jul 2008 17:25:25 +0000 (UTC) (envelope-from gpalmer@freebsd.org) Received: from gjp by noop.in-addr.com with local (Exim 4.54 (FreeBSD)) id 1KGztS-0005Hg-PN; Thu, 10 Jul 2008 13:25:22 -0400 Date: Thu, 10 Jul 2008 13:25:22 -0400 From: Gary Palmer To: Juri Mianovich Message-ID: <20080710172522.GA92945@in-addr.com> References: <223496.96060.qm@web45607.mail.sp1.yahoo.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <223496.96060.qm@web45607.mail.sp1.yahoo.com> Cc: freebsd-fs@freebsd.org Subject: Re: 24 TB UFS2 reality check ? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Jul 2008 17:25:25 -0000 On Thu, Jul 10, 2008 at 08:55:17AM -0700, Juri Mianovich wrote: > > Jeff, > > > --- On Wed, 7/9/08, Jeff Mohler wrote: > > > Lets see..a peak of maybe 25-30 random drive IOPS/sec at > > 15ms MINIMAL > > latency per IO (likely more like 35-40)..gonna be ugly. > > > > Complicated by normal load IOPS..you could expect it all to > > simply > > "dissapear" for a day while it reconstructs. > > > Once again, thank you very much - your comments are very helpful. > > So we've moved from "dangerous" (24 TB with no raid) to "inconvenient" (24 TB with raid 6). > > Two final questions: > > 1. What would _you_ do with 24 1 TB disks and a 24 port 3ware card ? > Assume an i386, 4 GB machine, and that fsck is workable because of > "newfs -i 131072" > > 2. What number should I ask my vendor (3ware) to do the rebuild > calculations ? You are talking about IOPS/s - I think I should ask > them how many IOPS/s the card does when rebuilding a 24 disk raid-6 > array, and then combine that with the IOPS/s I see in my normal workload. > At least on older 3Ware cards, and I suspect the one you're talking about will do it also, you can control the rebuild rate. There are 5 different options provided on my card (8506-4LP) for "background task rate" which either increases or decreases the IOPS used for the rebuild. > How do you measure IOPS/s in FreeBSD on a running machine ? iostat and/or systat are the two I use. Regards, Gary From owner-freebsd-fs@FreeBSD.ORG Thu Jul 10 17:38:56 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 18BB81065672 for ; Thu, 10 Jul 2008 17:38:56 +0000 (UTC) (envelope-from speedtoys.racing@gmail.com) Received: from yx-out-2324.google.com (yx-out-2324.google.com [74.125.44.28]) by mx1.freebsd.org (Postfix) with ESMTP id D31678FC29 for ; Thu, 10 Jul 2008 17:38:55 +0000 (UTC) (envelope-from speedtoys.racing@gmail.com) Received: by yx-out-2324.google.com with SMTP id 8so957072yxb.13 for ; Thu, 10 Jul 2008 10:38:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=dJvf1f+3kkEFlfhCmf8seyVPTZQ9IgvjfIxD0MkNbik=; b=iNyxvPGVXbZRgxnyDnkHXPkNTF0UOQObGH2BihU2Z/t6OEoca6qdtZ0LY+RM/zzjlb YbxKZQgQWr7YMy3j7y05AgOYiwLCq7jxm/pMvowQoelIPn+4TWL56uQb/AF9LOsfD+wd oW8wtD2/Nut6Rh4Hh1363sv+UNBUskJVtoZG0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=vPf0Ta7n9dQlW/3F+JiaGtO8c6E7eT/mu9q4j1/sPdG3WLophr/zcsNl1eOL68RvTD lYcelFHty06MZwloqWpgliEdXeBMthRkFoSE+yHB0Pm0HRNgN25BDhjtM0Gno5xhuESo G2WopRmqsRuEeayZDIi8u/yIduzKeFePv5pX8= Received: by 10.114.176.1 with SMTP id y1mr11556090wae.118.1215711534586; Thu, 10 Jul 2008 10:38:54 -0700 (PDT) Received: by 10.114.102.10 with HTTP; Thu, 10 Jul 2008 10:38:54 -0700 (PDT) Message-ID: Date: Thu, 10 Jul 2008 10:38:54 -0700 From: "Jeff Mohler" To: juri_mian@yahoo.com In-Reply-To: <223496.96060.qm@web45607.mail.sp1.yahoo.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <223496.96060.qm@web45607.mail.sp1.yahoo.com> Cc: freebsd-fs@freebsd.org Subject: Re: 24 TB UFS2 reality check ? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Jul 2008 17:38:56 -0000 > Two final questions: > > 1. What would _you_ do with 24 1 TB disks and a 24 port 3ware card ? Assume an i386, 4 GB machine, and that fsck is workable because of "newfs -i 131072" --- Im biased. If I had to have 24TB online, id get a netapp. 24TB of data has gotta be worth some real money, so I'd spend it. No..youre not wrong for NOT doing that, but, I wouldnt consider the 24TB nightmare myself, ive been there before. I never full healed from that mess. But if I had to steer you in a specific direction, plan for total failure. HDD's are built to do one thing. Fail. They occasionally hold data, but in reality, theyre just built to fail. Some are designed to fail sooner than others. You have 24 of them racing for failure. Sounds pessimistic, but, good backup & DR strategies are built around this. > 2. What number should I ask my vendor (3ware) to do the rebuild calculations ? You are talking about IOPS/s - I think I should ask them how many IOPS/s the card does when rebuilding a 24 disk raid-6 array, and then combine that with the IOPS/s I see in my normal workload. --- Rebuild calculations are based around how fast the drives are, cheap SATA is about 14ms track to track (longer full seek) and its highly random, and it has to compete with user/system workload. Thats just not possible to state, too many variables. IOPS are not a card issue, its a physical drive issue. 24TB of FCAL would rebuild faster than SATA, for example. The intelligence of the card itself could come into play, in case it is able to use command queuing to the drives/etc...and if the SATA drives fully support it as well. Depending what you want the system to do for users during the rebuild, prioritize the card appropriately. > How do you measure IOPS/s in FreeBSD on a running machine ? --- iostat -x is a pretty good way to measure that, for the most part. Im prepared to hear about different/better ways. From owner-freebsd-fs@FreeBSD.ORG Thu Jul 10 19:18:53 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F2DFB106567B for ; Thu, 10 Jul 2008 19:18:53 +0000 (UTC) (envelope-from dan@audilis.com) Received: from secure.athena.thinkhost.com (secure.athena.thinkhost.com [66.235.160.44]) by mx1.freebsd.org (Postfix) with ESMTP id D62288FC1B for ; Thu, 10 Jul 2008 19:18:53 +0000 (UTC) (envelope-from dan@audilis.com) Received: from [192.168.1.134] (c-76-16-122-236.hsd1.il.comcast.net [76.16.122.236]) (Authenticated sender: dan@sypher7.com) by c1-2A.thinkhost.com (Postfix) with ESMTPA id 520FE1112FFA for ; Thu, 10 Jul 2008 14:43:43 -0400 (EDT) Message-ID: <48765802.9060502@audilis.com> Date: Thu, 10 Jul 2008 13:42:10 -0500 From: "Daniel E. Lynn" Organization: Audilis Design Systems User-Agent: Thunderbird 2.0.0.14 (X11/20080505) MIME-Version: 1.0 To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Storing UFS snapshots externally? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Jul 2008 19:18:54 -0000 Greetings all, First, let me apologize if this has been asked before. If it has then my search skills must be lacking because I couldn't seem to find it in the lists anywhere. I'm wondering if it is feasible to store snapshots for UFS on a separate drive, and if so if there is an advisable way of doing it. The basic idea is that I'd like to be able to mitigate any write overhead of using a lot of snapshots by using a separate disk for them entirely. Here's the proposed setup: FreeBSD (/) is on ad0 Homedirs and userdata (/data) is on gm0 (ad2+ad3 mirrored) I've been successfully using snapshots for /data, and the overhead on this system doesn't seem too bad (yet) but if I have more than a dozen snapshots, I get the feeling it could get messy. It'd be great if I could store the snapshots for /data on / someplace. I just don't know if there are any "gotchyas" that might keep me from doing this. Obviously I could symlink the /data/.snap dir to someplace on /, but I'm a little weary it might not be that easy. Ideas? From owner-freebsd-fs@FreeBSD.ORG Thu Jul 10 19:23:11 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 24461106566B for ; Thu, 10 Jul 2008 19:23:11 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id AB9368FC24; Thu, 10 Jul 2008 19:23:09 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <4876619C.4050108@FreeBSD.org> Date: Thu, 10 Jul 2008 21:23:08 +0200 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.14 (Macintosh/20080421) MIME-Version: 1.0 To: "Daniel E. Lynn" References: <48765802.9060502@audilis.com> In-Reply-To: <48765802.9060502@audilis.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: Storing UFS snapshots externally? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Jul 2008 19:23:11 -0000 Daniel E. Lynn wrote: > Greetings all, > > First, let me apologize if this has been asked before. If it has then my > search skills must be lacking because I couldn't seem to find it in the > lists anywhere. > > I'm wondering if it is feasible to store snapshots for UFS on a separate > drive, and if so if there is an advisable way of doing it. The basic > idea is that I'd like to be able to mitigate any write overhead of using > a lot of snapshots by using a separate disk for them entirely. Here's > the proposed setup: No, the point of snapshots is that they are copy-on-write, so you don't copy the data until it changes. If you want to store them up externally, you have to copy the whole thing, and you should use a tool like dump, rsync, etc. Kris From owner-freebsd-fs@FreeBSD.ORG Thu Jul 10 20:05:03 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 448ED1065670 for ; Thu, 10 Jul 2008 20:05:03 +0000 (UTC) (envelope-from freebsd-fs@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id C82B78FC25 for ; Thu, 10 Jul 2008 20:05:02 +0000 (UTC) (envelope-from freebsd-fs@m.gmane.org) Received: from root by ciao.gmane.org with local (Exim 4.43) id 1KH1lG-0004Y1-4t for freebsd-fs@freebsd.org; Thu, 10 Jul 2008 19:25:02 +0000 Received: from 78-0-89-102.adsl.net.t-com.hr ([78.0.89.102]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 10 Jul 2008 19:25:02 +0000 Received: from ivoras by 78-0-89-102.adsl.net.t-com.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 10 Jul 2008 19:25:02 +0000 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-fs@freebsd.org From: Ivan Voras Date: Thu, 10 Jul 2008 21:23:58 +0200 Lines: 58 Message-ID: References: <48765802.9060502@audilis.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enigA775BEB77B91DB4D333D9DF6" X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: 78-0-89-102.adsl.net.t-com.hr User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) In-Reply-To: <48765802.9060502@audilis.com> X-Enigmail-Version: 0.95.6 Sender: news Subject: Re: Storing UFS snapshots externally? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Jul 2008 20:05:03 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigA775BEB77B91DB4D333D9DF6 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable Daniel E. Lynn wrote: > Greetings all, >=20 > First, let me apologize if this has been asked before. If it has then m= y=20 > search skills must be lacking because I couldn't seem to find it in the= =20 > lists anywhere. >=20 > I'm wondering if it is feasible to store snapshots for UFS on a separat= e=20 > drive, and if so if there is an advisable way of doing it. The basic=20 > idea is that I'd like to be able to mitigate any write overhead of usin= g=20 > a lot of snapshots by using a separate disk for them entirely. Here's=20 > the proposed setup: >=20 > FreeBSD (/) is on ad0 > Homedirs and userdata (/data) is on gm0 (ad2+ad3 mirrored) >=20 > I've been successfully using snapshots for /data, and the overhead on=20 > this system doesn't seem too bad (yet) but if I have more than a dozen = > snapshots, I get the feeling it could get messy. It'd be great if I=20 > could store the snapshots for /data on / someplace.=20 UFS snapshots don't copy the data into the "snapshot" file - they just=20 adjust internal references in the file system. The big file you get when = you create the snapshot isn't really a file in the traditional sense -=20 it consists of file system internal pointers to real data. Simplified,=20 when data gets changed on the "real" file system, *then* the old data=20 gets a separate copy in the snapshot. In short, there's no way other than manually copying (dd, tar) the data=20 from the snapshot to wherever. --------------enigA775BEB77B91DB4D333D9DF6 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFIdmHPldnAQVacBcgRAhNvAJ4rsx9g2HQkPxU1qy3O3sH0dWkcewCdERdv N4TwMdwkouvPmlfwrC2Y18k= =OGUV -----END PGP SIGNATURE----- --------------enigA775BEB77B91DB4D333D9DF6-- From owner-freebsd-fs@FreeBSD.ORG Fri Jul 11 08:40:22 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1960B106567E for ; Fri, 11 Jul 2008 08:40:22 +0000 (UTC) (envelope-from kaluna@gmail.com) Received: from rv-out-0506.google.com (rv-out-0506.google.com [209.85.198.231]) by mx1.freebsd.org (Postfix) with ESMTP id A49048FC1A for ; Fri, 11 Jul 2008 08:40:20 +0000 (UTC) (envelope-from kaluna@gmail.com) Received: by rv-out-0506.google.com with SMTP id b25so4590436rvf.43 for ; Fri, 11 Jul 2008 01:40:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:in-reply-to:mime-version:content-type:references; bh=E5/bibrl/rDOo2ArvlvMKYhif3pJSTvXzdQuZCk7WtI=; b=Wr9UBi5eElLmc2o9QgLMGF6HyqpshNWYd7g4LgkRpqfCrz89MdewNP5idqmrk3PTSA jPDohpOKGWOnwVa+nvtJG7fIJ7kTSiFuZ2dEyreoIjZHfGfURHGr9Mr+tBINTCRweU7L w2TnP+wyw1CQ+Hzo33elLyMIoSASh3ZjnLpms= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:in-reply-to:mime-version :content-type:references; b=fVsYPyyJBGZ8JbS/QrWtuWqnG6bIPp0QCH1szqbuZcTl9hWuvoitzHBqPFbTXTjvoR Sc0sBar9rlfyP60k2SSnwArXcbdCBA3MakI1w1UG6oJrmUM/lewFu1YWjCk+3Epc6jLm M6reLCHr7MedgaiWdq+bARAbbfTHeOWegLJMQ= Received: by 10.115.110.6 with SMTP id n6mr2833060wam.34.1215765620152; Fri, 11 Jul 2008 01:40:20 -0700 (PDT) Received: by 10.114.80.10 with HTTP; Fri, 11 Jul 2008 01:40:20 -0700 (PDT) Message-ID: <11c17ec30807110140xbeec510ke54eaf1829fc4894@mail.gmail.com> Date: Fri, 11 Jul 2008 10:40:20 +0200 From: "Carlos Luna" To: freebsd-fs@freebsd.org, kaluna@gmail.com, brooks@freebsd.org In-Reply-To: <200807091423.m69ENtJM075767@lurza.secnetix.de> MIME-Version: 1.0 References: <20080707154805.GA57420@lor.one-eyed-alien.net> <200807091423.m69ENtJM075767@lurza.secnetix.de> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Subject: Re: Filesystem is not clean - run fsck X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Jul 2008 08:40:22 -0000 Yes, that would be the case. There is a patch made by someone to the fsck t= o "solve" this, but who made the pacth said too "expect lot loose of date", son I'm just looking for a harmless solution. By the way I'm just triyng to mount it read-only. Regards Mike 2008/7/9 Oliver Fromme : > Brooks Davis wrote: > > On Sat, Jul 05, 2008 at 10:58:33AM +0200, Carlos Luna wrote: > > > Hi I'd used freenas about 5 years without any problem. Now I can?t > mount my > > > raid volume and in his sourceforge forums seems they cant help me. > Hope this > > > list is the right list for my issue. > > > > > > When I try to fsck,I get: > > > casa:/dev# fsck -t ufs -y /dev/pst0s1 > > > ** /dev/pst0s1 > > > ** Last Mounted on /mnt/raid > > > ** Phase 1 - Check Blocks and Sizes > > > -4439300862985009506 BAD I=3D86 > > > 3443570138036206556 BAD I=3D86 > > > -7476842757969057647 BAD I=3D86 > > > -8078484667502176485 BAD I=3D86 > > > 2249916482063805839 BAD I=3D86 > > > -3291681609520367063 BAD I=3D86 > > > 7780434385339928353 BAD I=3D86 > > > -4372486048108189431 BAD I=3D86 > > > 8774078035736727371 BAD I=3D86 > > > -2035310265760485777 BAD I=3D86 > > > 6848295312539782814 BAD I=3D86 > > > EXCESSIVE BAD BLKS I=3D86 > > > CONTINUE? yes > > > > > > ... > > > .... > > > > > > UNKNOWN FILE TYPE I=3D7254140 > > > CLEAR? yes > > > > > > UNKNOWN FILE TYPE I=3D7254141 > > > CLEAR? yes > > > > > > UNKNOWN FILE TYPE I=3D7254142 > > > CLEAR? yes > > > > > > UNKNOWN FILE TYPE I=3D7254143 > > > CLEAR? yes > > > > > > fsck_ufs: cannot alloc 3037795832 bytes for inoinfo > > > I have a lot of info there, 1 TB. I will appreciate any help. > > > > It looks like you have a somewhat large file system, apparently with a > > lot of small files on it. The message indicates that you need to be > > able to allocate over 3GB of address space to handle this. As such yo= u > > will need a 64-bit machine, ideally with 4GB or more RAM and probably > > with a large swap partition. > > > > In theory it should be possible to write a constrained memory use > version of > > fsck, but to my knowledge no one has done so and I suspect it would be= a > > time consuming development effort. > > There's another possibility. I remember cases where the FS > structures were damaged in a way that fsck picked up wrong > size information, and then tried to allocate ridiculously > large amounts of memory, even for a small file system. > > That could be the case here, too. > > Best regards > Oliver > > -- > Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M. > Handelsregister: Registergericht Muenchen, HRA 74606, Gesch=E4ftsfuehrun= g: > secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht M=FC= n- > chen, HRB 125758, Gesch=E4ftsf=FChrer: Maik Bachmann, Olaf Erb, Ralf Geb= hart > > FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd > > "And believe me, as a C++ programmer, I don't hesitate to question > the decisions of language designers. After a decent amount of C++ > exposure, Python's flaws seem ridiculously small." -- Ville Vainio > From owner-freebsd-fs@FreeBSD.ORG Fri Jul 11 10:25:47 2008 Return-Path: Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3D6101065670; Fri, 11 Jul 2008 10:25:47 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (unknown [IPv6:2a01:170:102f::2]) by mx1.freebsd.org (Postfix) with ESMTP id B9F438FC19; Fri, 11 Jul 2008 10:25:46 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (localhost [127.0.0.1]) by lurza.secnetix.de (8.14.1/8.14.1) with ESMTP id m6BAPiLB010157; Fri, 11 Jul 2008 12:25:44 +0200 (CEST) (envelope-from oliver.fromme@secnetix.de) Received: (from olli@localhost) by lurza.secnetix.de (8.14.1/8.14.1/Submit) id m6BAPiIY010156; Fri, 11 Jul 2008 12:25:44 +0200 (CEST) (envelope-from olli) Date: Fri, 11 Jul 2008 12:25:44 +0200 (CEST) Message-Id: <200807111025.m6BAPiIY010156@lurza.secnetix.de> From: Oliver Fromme To: freebsd-fs@FreeBSD.ORG, kaluna@gmail.com, brooks@FreeBSD.ORG In-Reply-To: <11c17ec30807110140xbeec510ke54eaf1829fc4894@mail.gmail.com> X-Newsgroups: list.freebsd-fs User-Agent: tin/1.8.3-20070201 ("Scotasay") (UNIX) (FreeBSD/6.2-STABLE-20070808 (i386)) MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-2.1.2 (lurza.secnetix.de [127.0.0.1]); Fri, 11 Jul 2008 12:25:45 +0200 (CEST) Cc: Subject: Re: Filesystem is not clean - run fsck X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: freebsd-fs@FreeBSD.ORG, kaluna@gmail.com, brooks@FreeBSD.ORG List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Jul 2008 10:25:47 -0000 Carlos Luna wrote: > Yes, that would be the case. There is a patch made by someone to the fsck to > "solve" this, but who made the pacth said too "expect lot loose of date", > son I'm just looking for a harmless solution. By the way I'm just triyng to > mount it read-only. You used fsck -y, so you should already expect to have lost data. Quoting from the manual page's description of the -y option: "this should be used with great caution as this is a free license to continue after essentially unlimited trouble has been encountered." Best regards Oliver -- Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M. Handelsregister: Registergericht Muenchen, HRA 74606, Geschäftsfuehrung: secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün- chen, HRB 125758, Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd "People still program in C. People keep writing shell scripts. *Most* people don't realize the shortcomings of the tools they are using because they a) don't reflect on their workflows and they are b) too lazy to check out alternatives to realize there is help." -- Simon 'corecode' Schubert From owner-freebsd-fs@FreeBSD.ORG Fri Jul 11 12:35:50 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EE94F106567A for ; Fri, 11 Jul 2008 12:35:50 +0000 (UTC) (envelope-from juri_mian@yahoo.com) Received: from n75.bullet.mail.sp1.yahoo.com (n75.bullet.mail.sp1.yahoo.com [98.136.44.51]) by mx1.freebsd.org (Postfix) with SMTP id CCDA18FC19 for ; Fri, 11 Jul 2008 12:35:50 +0000 (UTC) (envelope-from juri_mian@yahoo.com) Received: from [216.252.122.216] by n75.bullet.mail.sp1.yahoo.com with NNFMP; 11 Jul 2008 12:35:50 -0000 Received: from [69.147.65.164] by t1.bullet.sp1.yahoo.com with NNFMP; 11 Jul 2008 12:35:50 -0000 Received: from [127.0.0.1] by omp409.mail.sp1.yahoo.com with NNFMP; 11 Jul 2008 12:35:50 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 74749.75000.bm@omp409.mail.sp1.yahoo.com Received: (qmail 76260 invoked by uid 60001); 11 Jul 2008 12:35:50 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Received:X-Mailer:Date:From:Reply-To:Subject:To:MIME-Version:Content-Type:Message-ID; b=x53j+6NySkY8W1dOAXv6qAoaqEAV6bysWG2q7aLhF/D7P0zY8sALcgUyIo2S+u1HjlYPtFqd4DGto4KiylNox8paL44WiTuoiA8Ckd4MUsECsvZDlAOOhOywBevpAtnSr0+X61kY5RW0ik63O5f4yXS3yb1eproI759FXFRPcBg=; Received: from [71.63.232.32] by web45615.mail.sp1.yahoo.com via HTTP; Fri, 11 Jul 2008 05:35:49 PDT X-Mailer: YahooMailWebService/0.7.199 Date: Fri, 11 Jul 2008 05:35:49 -0700 (PDT) From: Juri Mianovich To: freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Message-ID: <964824.75951.qm@web45615.mail.sp1.yahoo.com> X-Mailman-Approved-At: Fri, 11 Jul 2008 13:06:12 +0000 Subject: RE: the quota question ... one user with >2 TB owned files (but no quota set) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: juri_mian@yahoo.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Jul 2008 12:35:51 -0000 > I am going to be running a large array. > > I will have quotas in the kernel and enabled BUT all users I set quotas > on will be nowhere near the 2TB barrier I see people talking about > recently. > > HOWEVER, at some point in the future, root or www (or both) users will > _own more than_ 2 TB of files. They will not have a quota set on them, > but they will in fact own >2 TB of files. > > Is this also a problem ? Or is the only problem actually _setting_ a > quota larger than 2TB ? > > I assume the output in "repquota /my/fs" will be broken, and that is > fine with me - I just don't want to corrupt or damage my filesystem (or > existing quotas) the day that my www user goes over 2TB of owned files. > > Also, I am distrustful of merely testing this - just because things run > fine for a day with quotas turned on and some user owning more than 2 TB > of files doesn't mean it won't blow up at some future date in some > interesting scenario - and that is why I am asking for opinions here > rather than just creating >2 TB of files and turning on quotas. > > Does anyone out there already do this and can reassure me ? I haven't heard anything - which is not surprising, since it doesn't sound like many people are using quotas these days. Does anyone have any general thoughts as to whether this will be dangerous or not ? I know (I assume) that repquota output for my root and www users will be broken, and that's fine - I just want to make sure that as soon as one user goes over 2TB of owned files the filesystem doesn't trash itself. Can the quota subsystem failing in some way cause data loss / filesystem inconsistencies ? From owner-freebsd-fs@FreeBSD.ORG Fri Jul 11 14:53:23 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 45FF9106564A for ; Fri, 11 Jul 2008 14:53:23 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [91.103.162.4]) by mx1.freebsd.org (Postfix) with ESMTP id 170B58FC23 for ; Fri, 11 Jul 2008 14:53:22 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from localhost (localhost.codelab.cz [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id A7F4A19E023; Fri, 11 Jul 2008 16:36:00 +0200 (CEST) Received: from [192.168.1.2] (r5bb235.net.upc.cz [86.49.61.235]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 6E01219E019; Fri, 11 Jul 2008 16:35:58 +0200 (CEST) Message-ID: <48776FE6.3060307@quip.cz> Date: Fri, 11 Jul 2008 16:36:22 +0200 From: Miroslav Lachman <000.fbsd@quip.cz> User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.12) Gecko/20050915 X-Accept-Language: cz, cs, en, en-us MIME-Version: 1.0 To: juri_mian@yahoo.com References: <964824.75951.qm@web45615.mail.sp1.yahoo.com> In-Reply-To: <964824.75951.qm@web45615.mail.sp1.yahoo.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: the quota question ... one user with >2 TB owned files (but no quota set) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Jul 2008 14:53:23 -0000 Juri Mianovich wrote: [...] > I haven't heard anything - which is not surprising, since it doesn't sound like many people are using quotas these days. > > Does anyone have any general thoughts as to whether this will be dangerous or not ? I know (I assume) that repquota output for my root and www users will be broken, and that's fine - I just want to make sure that as soon as one user goes over 2TB of owned files the filesystem doesn't trash itself. > > Can the quota subsystem failing in some way cause data loss / filesystem inconsistencies ? I think there are not so many users running similar configuration, it means not much experiences. But you can try your own test easily. Just install some test machine and try it with large sparse files (something like dd if=/dev/zero of=sparse-file bs=1 count=1 seek=1024k for 1M sparse file) Miroslav Lachman From owner-freebsd-fs@FreeBSD.ORG Fri Jul 11 17:52:14 2008 Return-Path: Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5A8261065671 for ; Fri, 11 Jul 2008 17:52:14 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (unknown [IPv6:2a01:170:102f::2]) by mx1.freebsd.org (Postfix) with ESMTP id CB7068FC0C for ; Fri, 11 Jul 2008 17:52:13 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (localhost [127.0.0.1]) by lurza.secnetix.de (8.14.1/8.14.1) with ESMTP id m6BHqBoG031071; Fri, 11 Jul 2008 19:52:11 +0200 (CEST) (envelope-from oliver.fromme@secnetix.de) Received: (from olli@localhost) by lurza.secnetix.de (8.14.1/8.14.1/Submit) id m6BHq8n2031070; Fri, 11 Jul 2008 19:52:08 +0200 (CEST) (envelope-from olli) Date: Fri, 11 Jul 2008 19:52:08 +0200 (CEST) Message-Id: <200807111752.m6BHq8n2031070@lurza.secnetix.de> From: Oliver Fromme To: freebsd-fs@FreeBSD.ORG, 000.fbsd@quip.cz, juri_mian@yahoo.com In-Reply-To: <48776FE6.3060307@quip.cz> X-Newsgroups: list.freebsd-fs User-Agent: tin/1.8.3-20070201 ("Scotasay") (UNIX) (FreeBSD/6.2-STABLE-20070808 (i386)) MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-2.1.2 (lurza.secnetix.de [127.0.0.1]); Fri, 11 Jul 2008 19:52:12 +0200 (CEST) Cc: Subject: Re: the quota question ... one user with >2 TB owned files (but no quota set) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: freebsd-fs@FreeBSD.ORG, 000.fbsd@quip.cz, juri_mian@yahoo.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Jul 2008 17:52:14 -0000 Miroslav Lachman wrote: > Juri Mianovich wrote: > [...] > > I haven't heard anything - which is not surprising, since it > > doesn't sound like many people are using quotas these days. > > > > Does anyone have any general thoughts as to whether this will be > > dangerous or not ? I know (I assume) that repquota output for my > > root and www users will be broken, and that's fine - I just want to > > make sure that as soon as one user goes over 2TB of owned files the > > filesystem doesn't trash itself. > > > > Can the quota subsystem failing in some way cause data loss / > > filesystem inconsistencies ? > > I think there are not so many users running similar configuration, it > means not much experiences. But you can try your own test easily. Just > install some test machine and try it with large sparse files That doesn't help, because only physically allocated space accounts towards quotas. An sparse file of 1 GB that consists entirely of zeros uses only 48 KB of physical disk space (with default UFS2 newfs parameters), so it uses only 48 KB from your quota, not 1 GB. > (something like dd if=/dev/zero of=sparse-file bs=1 count=1 seek=1024k > for 1M sparse file) There's an easier way to do that: truncate -s 1m sparse-file Best regards Oliver -- Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M. Handelsregister: Registergericht Muenchen, HRA 74606, Geschäftsfuehrung: secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün- chen, HRB 125758, Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd "Clear perl code is better than unclear awk code; but NOTHING comes close to unclear perl code" (taken from comp.lang.awk FAQ) From owner-freebsd-fs@FreeBSD.ORG Fri Jul 11 19:29:48 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B25781065671 for ; Fri, 11 Jul 2008 19:29:48 +0000 (UTC) (envelope-from salehoo@tradingtoolslist.com) Received: from mail.tradingtoolslist.com (mail.tradingtoolslist.com [64.187.120.161]) by mx1.freebsd.org (Postfix) with ESMTP id A06968FC12 for ; Fri, 11 Jul 2008 19:29:48 +0000 (UTC) (envelope-from salehoo@tradingtoolslist.com) Received: by mail.tradingtoolslist.com Postfix Date: Fri, 11 Jul 2008 15:05:26 -0400 From: "Wholesale Suppliers" To: freebsd-fs@freebsd.org Message-ID: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Find Fantastic Suppliers X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Jul 2008 19:29:48 -0000 SaleHoo gives you the most in-depth list of wholesale suppliers and distributors available today. They have gone through the internet and searched for all of the major suppliers, which are now at your fingertips. They are determined to help each and every single person as if their life depended on it, and we find this wonderful. If you are looking for goods at a price you can't give up, Click the link below! http://tradingtoolslist.com/c/oUHhGKu_uxQBSl3mmD4RQA.html?0 SaleHoo should be your first choice no questions asked. They even offer on-line chat for those of you who may be a little unsure of what it offers. To remove yourself from this list, click here http://tradingtoolslist.com/u/oUHhGKu_uxQBSl3mmD4RQA.html or write to us at: 1173 A. Second Ave, PNB 147New York NY 10065 From owner-freebsd-fs@FreeBSD.ORG Sat Jul 12 10:17:48 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 77DA41065674 for ; Sat, 12 Jul 2008 10:17:48 +0000 (UTC) (envelope-from kaluna@gmail.com) Received: from rv-out-0506.google.com (rv-out-0506.google.com [209.85.198.226]) by mx1.freebsd.org (Postfix) with ESMTP id 4603F8FC12 for ; Sat, 12 Jul 2008 10:17:48 +0000 (UTC) (envelope-from kaluna@gmail.com) Received: by rv-out-0506.google.com with SMTP id b25so5069235rvf.43 for ; Sat, 12 Jul 2008 03:17:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:in-reply-to:mime-version:content-type:references; bh=MxxOfJNPmVTJdDZZ7O1c7C9FXOIy+WtB7Ra3vDF8ycc=; b=DdqXcqnQwMlB7P/q7AwMwNA/xk/21JhZ+/zZ0q1XSRRgEtY2BaGXKCU7fZEVW6lh1T Zvj0yw83JCwDmZeWc+uamHxxI3UcLUD30q372lJhr9Pd2HbSY7HSoQZxZRp8lnTknhh3 euMmokMItHKEm0hAPC26mRAaoVMtX0TBhd/vU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:in-reply-to:mime-version :content-type:references; b=lHTTkPBxwFLQfIW6VuXcrMbObo3DWeqLA3NHXJVxjw9MQJFDrYfr7+KwDcLPcEoINT 1w2jhsEJ8soWdV1gV/QIZZamLtMqDk6y7LgKVFwyCUs6eD2kjVTI1JONNTuSiqA8qrnP uUj+DY7lXGhD7uhVN5Xb3qe7766oxwg5rGXHM= Received: by 10.114.89.1 with SMTP id m1mr15417159wab.126.1215857867436; Sat, 12 Jul 2008 03:17:47 -0700 (PDT) Received: by 10.114.80.10 with HTTP; Sat, 12 Jul 2008 03:17:47 -0700 (PDT) Message-ID: <11c17ec30807120317t1f344396rcd3cfd7ca6abaedd@mail.gmail.com> Date: Sat, 12 Jul 2008 12:17:47 +0200 From: "Carlos Luna" To: freebsd-fs@freebsd.org, kaluna@gmail.com, brooks@freebsd.org In-Reply-To: <200807111025.m6BAPiIY010156@lurza.secnetix.de> MIME-Version: 1.0 References: <11c17ec30807110140xbeec510ke54eaf1829fc4894@mail.gmail.com> <200807111025.m6BAPiIY010156@lurza.secnetix.de> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Subject: Re: Filesystem is not clean - run fsck X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 Jul 2008 10:17:48 -0000 Is there any other option? Regards Mike 2008/7/11 Oliver Fromme : > Carlos Luna wrote: > > Yes, that would be the case. There is a patch made by someone to the > fsck to > > "solve" this, but who made the pacth said too "expect lot loose of > date", > > son I'm just looking for a harmless solution. By the way I'm just triy= ng > to > > mount it read-only. > > You used fsck -y, so you should already expect to have > lost data. Quoting from the manual page's description > of the -y option: "this should be used with great caution > as this is a free license to continue after essentially > unlimited trouble has been encountered." > > Best regards > Oliver > > -- > Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M. > Handelsregister: Registergericht Muenchen, HRA 74606, Gesch=E4ftsfuehrun= g: > secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht M=FC= n- > chen, HRB 125758, Gesch=E4ftsf=FChrer: Maik Bachmann, Olaf Erb, Ralf Geb= hart > > FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd > > "People still program in C. People keep writing shell scripts. *Most* > people don't realize the shortcomings of the tools they are using because > they a) don't reflect on their workflows and they are b) too lazy to chec= k > out alternatives to realize there is help." -- Simon 'corecode' Schubert > From owner-freebsd-fs@FreeBSD.ORG Sat Jul 12 22:04:36 2008 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7A26F106564A; Sat, 12 Jul 2008 22:04:36 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 537B18FC1F; Sat, 12 Jul 2008 22:04:36 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (linimon@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id m6CM4a9X048114; Sat, 12 Jul 2008 22:04:36 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id m6CM4aEA048110; Sat, 12 Jul 2008 22:04:36 GMT (envelope-from linimon) Date: Sat, 12 Jul 2008 22:04:36 GMT Message-Id: <200807122204.m6CM4aEA048110@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-fs@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/125536: [ext2fs] ext 2 mounts cleanly but fails on commands like ls X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 Jul 2008 22:04:36 -0000 Old Synopsis: ext 2 mounts cleanly but fails on commands like ls New Synopsis: [ext2fs] ext 2 mounts cleanly but fails on commands like ls Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: linimon Responsible-Changed-When: Sat Jul 12 22:04:20 UTC 2008 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=125536