From owner-freebsd-fs@FreeBSD.ORG Sun Sep 4 00:00:26 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1E01F1065672 for ; Sun, 4 Sep 2011 00:00:26 +0000 (UTC) (envelope-from brodbd@uw.edu) Received: from mail-ew0-f54.google.com (mail-ew0-f54.google.com [209.85.215.54]) by mx1.freebsd.org (Postfix) with ESMTP id B3DEB8FC0C for ; Sun, 4 Sep 2011 00:00:06 +0000 (UTC) Received: by ewy1 with SMTP id 1so2266662ewy.13 for ; Sat, 03 Sep 2011 17:00:05 -0700 (PDT) MIME-Version: 1.0 Received: by 10.213.14.18 with SMTP id e18mr279125eba.47.1315094405542; Sat, 03 Sep 2011 17:00:05 -0700 (PDT) Received: by 10.213.22.210 with HTTP; Sat, 3 Sep 2011 17:00:05 -0700 (PDT) In-Reply-To: References: <14220705.747900.1315013770239.JavaMail.root@erie.cs.uoguelph.ca> Date: Sat, 3 Sep 2011 17:00:05 -0700 Message-ID: From: David Brodbeck To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: ZFSv28+NFSv4 poor file creation performance, "sync=disabled" has no effect X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Sep 2011 00:00:26 -0000 On Fri, Sep 2, 2011 at 6:36 PM, Rick Macklem wrote: > One post explained how disabling the ZIL can result in up to 5seconds worth > of changes being lost if/when the server crashes. (A lot can change on a > file > system in 5sec. The NFS protocol assumes all fs changes related to a file > creation are done before the server replies to the RPC. As such, disabling > the > ZIL does violate the protocol specs and means you are living dangerously.) > Yes, I'm aware of that. I chose to take that risk in my configuration because we frequently extract large tarballs, and performance with the default settings is not acceptable. Also, most other filesystems make no guarantees that data has been flushed to the platters, only that it's been sent to the disk hardware, so it really isn't a big increase in risk compared to our pre-ZFS setup. The sync=disabled parameter was, in my understanding, added as a per-filesystem way to achieve this risk/performance tradeoff, instead of zil_disable, which was system-wide. I posted the question because I'm not seeing any changes when I use sync=disabled. This is a bit of a show-stopper for us because performance with full syncing is not acceptable. I agree that it's not ideal, but we don't have the extra drive bays to add SSDs at this point; also it seems like people have had a lot of problems with SSDs and FreeBSD, so I'd prefer for the technology to mature a bit before I risk my entire pool by putting a ZIL on one. -- David Brodbeck System Administrator, Linguistics University of Washington From owner-freebsd-fs@FreeBSD.ORG Sun Sep 4 01:28:59 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7E65B106566C for ; Sun, 4 Sep 2011 01:28:59 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta15.westchester.pa.mail.comcast.net (qmta15.westchester.pa.mail.comcast.net [76.96.59.228]) by mx1.freebsd.org (Postfix) with ESMTP id 289BE8FC15 for ; Sun, 4 Sep 2011 01:28:58 +0000 (UTC) Received: from omta14.westchester.pa.mail.comcast.net ([76.96.62.60]) by qmta15.westchester.pa.mail.comcast.net with comcast id URTu1h0021HzFnQ5FRUzdi; Sun, 04 Sep 2011 01:28:59 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta14.westchester.pa.mail.comcast.net with comcast id URUx1h00N1t3BNj3aRUxTR; Sun, 04 Sep 2011 01:28:59 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id D9090102C1B; Sat, 3 Sep 2011 18:28:55 -0700 (PDT) Date: Sat, 3 Sep 2011 18:28:55 -0700 From: Jeremy Chadwick To: David Brodbeck Message-ID: <20110904012855.GA79580@icarus.home.lan> References: <14220705.747900.1315013770239.JavaMail.root@erie.cs.uoguelph.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@freebsd.org Subject: Re: ZFSv28+NFSv4 poor file creation performance, "sync=disabled" has no effect X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Sep 2011 01:28:59 -0000 On Sat, Sep 03, 2011 at 05:00:05PM -0700, David Brodbeck wrote: > On Fri, Sep 2, 2011 at 6:36 PM, Rick Macklem wrote: > > > One post explained how disabling the ZIL can result in up to 5seconds worth > > of changes being lost if/when the server crashes. (A lot can change on a > > file > > system in 5sec. The NFS protocol assumes all fs changes related to a file > > creation are done before the server replies to the RPC. As such, disabling > > the > > ZIL does violate the protocol specs and means you are living dangerously.) > > > > ... > > Also, most other filesystems make no guarantees that data has been > flushed to the platters, only that it's been sent to the disk > hardware, so it really isn't a big increase in risk compared to our > pre-ZFS setup. Can you explain where you got this impression from? Please provide actual references/validation/proof for non-network filesystems. Disk-level filesystems are more or less at the mercy of the applications using them; if a programmer calls open(2) without O_FSYNC or O_SYNC, or does not explicitly flush I/O to disk in their software, then is the filesystem really to blame? Furthermore, the world does know of some disks/devices/products where executing BIO_FLUSH (which on ATA disks should get translated into the ATA FLUSH or FLUSH EXT command) does not truly write the data to the platters. Again: the filesystem is innocent, blame the disk/device/manufacturer. > The sync=disabled parameter was, in my understanding, added as a > per-filesystem way to achieve this risk/performance tradeoff, instead of > zil_disable, which was system-wide. I posted the question because I'm not > seeing any changes when I use sync=disabled. pjd@, mm@, and others will need to comment on this. -fs is the best list for this, so I'm a little surprised no key members have chimed in here. What we (the community) need clarification regarding is whether or not OpenSolaris and/or Illumos still provides the zil_disable tunable on those OSes; if they do, FreeBSD should provide the same (which means removable of zil_disable on FreeBSD is effectively a regression). If said OSes do not provide it, then FreeBSD should not provide it. If said OSes moved to using the sync parameter where it works, yet it does not work on FreeBSD, then that's a bug and a PR should be filed + discussion induced. And yes, I'm aware that I have given the "try using the sync parameter instead" advice to others here on FreeBSD lists. > This is a bit of a show-stopper for us because performance with full syncing > is not acceptable. I agree that it's not ideal, but we don't have the extra > drive bays to add SSDs at this point; also it seems like people have had a > lot of problems with SSDs and FreeBSD, so I'd prefer for the technology to > mature a bit before I risk my entire pool by putting a ZIL on one. The risks/complexities and performance concerns with SSDs when used as part of a ZFS filesystem on FreeBSD greatly differ from that of using an SSD as a dedicated intent log (ZIL) device. What exactly do you need ZFS for given your requirements? Is it that you want a simple-to-use LVM-like filesystem that offers a vast in-memory cache but without the data integrity bits? -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Sun Sep 4 22:29:05 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C642D1065676 for ; Sun, 4 Sep 2011 22:29:05 +0000 (UTC) (envelope-from mm@FreeBSD.org) Received: from mail.vx.sk (mail.vx.sk [IPv6:2a01:4f8:100:1043::3]) by mx1.freebsd.org (Postfix) with ESMTP id 833AF8FC08 for ; Sun, 4 Sep 2011 22:29:05 +0000 (UTC) Received: from core.vx.sk (localhost [127.0.0.1]) by mail.vx.sk (Postfix) with ESMTP id 4ACD919A925; Mon, 5 Sep 2011 00:29:04 +0200 (CEST) X-Virus-Scanned: amavisd-new at mail.vx.sk Received: from mail.vx.sk ([127.0.0.1]) by core.vx.sk (mail.vx.sk [127.0.0.1]) (amavisd-new, port 10024) with LMTP id g71qNk9_OGkn; Mon, 5 Sep 2011 00:29:00 +0200 (CEST) Received: from [10.9.8.1] (188-167-78-15.dynamic.chello.sk [188.167.78.15]) by mail.vx.sk (Postfix) with ESMTPSA id 3D96B19A912; Mon, 5 Sep 2011 00:28:59 +0200 (CEST) Message-ID: <4E63FBAC.5070003@FreeBSD.org> Date: Mon, 05 Sep 2011 00:29:00 +0200 From: Martin Matuska User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:6.0.1) Gecko/20110830 Thunderbird/6.0.1 MIME-Version: 1.0 To: Jeremy Chadwick References: <14220705.747900.1315013770239.JavaMail.root@erie.cs.uoguelph.ca> <20110904012855.GA79580@icarus.home.lan> In-Reply-To: <20110904012855.GA79580@icarus.home.lan> X-Enigmail-Version: 1.3.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: ZFSv28+NFSv4 poor file creation performance, "sync=disabled" has no effect X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Sep 2011 22:29:05 -0000 On 4. 9. 2011 3:28, Jeremy Chadwick wrote: > pjd@, mm@, and others will need to comment on this. -fs is the best > list for this, so I'm a little surprised no key members have chimed in > here. > > What we (the community) need clarification regarding is whether or not > OpenSolaris and/or Illumos still provides the zil_disable tunable on > those OSes; if they do, FreeBSD should provide the same (which means > removable of zil_disable on FreeBSD is effectively a regression). If > said OSes do not provide it, then FreeBSD should not provide it. If > said OSes moved to using the sync parameter where it works, yet it does > not work on FreeBSD, then that's a bug and a PR should be filed + > discussion induced. The per-dataset synchronicity setting was proposed to OpenSolaris in PSARC/2010/108 and added in changeset 12294. It is now part of all ZFS implementations. PSARC: http://arc.opensolaris.org/caselog/PSARC/2010/108/20100401_neil.perrin Changed code: http://hg.openindiana.org/upstream/illumos/illumos-gate/rev/2a74b443e6b1 -- Martin Matuska FreeBSD committer http://blog.vx.sk From owner-freebsd-fs@FreeBSD.ORG Mon Sep 5 08:50:04 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 157CC1065670 for ; Mon, 5 Sep 2011 08:50:04 +0000 (UTC) (envelope-from pawel@dawidek.net) Received: from mail.dawidek.net (60.wheelsystems.com [83.12.187.60]) by mx1.freebsd.org (Postfix) with ESMTP id 7C3958FC15 for ; Mon, 5 Sep 2011 08:50:03 +0000 (UTC) Received: from localhost (58.wheelsystems.com [83.12.187.58]) by mail.dawidek.net (Postfix) with ESMTPSA id 19058D01; Mon, 5 Sep 2011 10:50:01 +0200 (CEST) Date: Mon, 5 Sep 2011 10:49:37 +0200 From: Pawel Jakub Dawidek To: Johan Hendriks Message-ID: <20110905084934.GC1662@garage.freebsd.pl> References: <4E60D992.3030802@gmail.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="jy6Sn24JjFx/iggw" Content-Disposition: inline In-Reply-To: <4E60D992.3030802@gmail.com> X-OS: FreeBSD 9.0-CURRENT amd64 User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@freebsd.org Subject: Re: ZFS on HAST and reboot. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Sep 2011 08:50:04 -0000 --jy6Sn24JjFx/iggw Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Sep 02, 2011 at 03:26:42PM +0200, Johan Hendriks wrote: > Hello all. >=20 > I just started using ZFS on top of HAST. >=20 > What i did was first glabel my disks like disk1 to disk3 > Then I created my hast devices in /etc/hast.conf >=20 > /etc/hast.conf looks like this. > i > resource disk1 { > on srv1 { > local /dev/label/disk1 > remote 192.168.5.41 > } > on srv2 { > local /dev/label/disk1 > remote 192.168.5.40 > } > } > resource disk2 { > on srv1 { > local /dev/label/disk2 > remote 192.168.5.41 > } > on srv2 { > local /dev/label/disk2 > remote 192.168.5.40 > } > } > resource disk3 { > on srv1 { > local /dev/label/disk3 > remote 192.168.5.41 > } > on srv2 { > local /dev/label/disk3 > remote 192.168.5.40 > } > } >=20 > This works. > I can set srv 1 to primary and srv 2 to secondary and visa versa. > hastctl role primary all and hastctl role secondary all. >=20 > Then i created the raidz on the master srv1 > zpool create storage raidz1 hast/disk1 hast/disk2 hast/disk3 >=20 > all looks good. > zpool status > pool: storage > state: ONLINE > scan: scrub repaired 0 in 0h0m with 0 errors on Wed Aug 31 20:49:19 2011 > config: >=20 > NAME STATE READ WRITE CKSUM > storage ONLINE 0 0 0 > raidz1-0 ONLINE 0 0 0 > hast/disk1 ONLINE 0 0 0 > hast/disk2 ONLINE 0 0 0 > hast/disk3 ONLINE 0 0 0 >=20 > errors: No known data errors >=20 > then i created the mountpoint and created zfs on it > # mkdir /usr/local/virtual > # zfs create storage/virtual > # zfs list > # zfs set mountpoint=3D/usr/local/virtual storage/virtual >=20 > # /etc/rc.d/zfs start and whooop there is my /usr/local/virtual zfs=20 > filesystem. > # mount > /dev/ada0p2 on / (ufs, local, journaled soft-updates) > devfs on /dev (devfs, local, multilabel) > storage on /storage (zfs, local, nfsv4acls) > storage/virtual on /usr/local/virtual (zfs, local, nfsv4acls) >=20 > if i do a zfs export -f storage on srv1 change the hast role to=20 > secondary and then set the hast role on srv2 to primary and do zfs=20 > import -f storage, i can see the files on srv2. >=20 > I am a happy camper :D >=20 > So it works like advertised. > Now i rebooted both machines. > all is working fine. >=20 > But if i reboot the server srv1 again, i can not import the pool=20 > anymore, it tells me the pool is already imported. > I do load the carp-hast-switch master file with ifstated. > This does set the hast role to primary. > But can not import the pool. > Now this can be true because i did not export it. > if i do a /etc/rc.d/zfs start, than it gets mounted and the pool is=20 > again available. >=20 > Is there a way i can do this automaticly. > In my understanding after a reboot zfs try's to start, but fails because= =20 > my hast providers are not yet ready. > Or am i doing something wrong and should i not do it this way. > Can i tell zfs to start after the hast providers are primary at reboot. You can see the message that pool is already imported, because when you reboot primary there is still info about the pool in /boot/zfs/zpool.cache. Pools that are mentioned in this file are automatically imported on boot (by the kernel), so importing such a pool will fail. You should still be able to mount file systems (zfs mount -a). What I'd recommend is not to use /etc/rc.d/zfs to mount file systems =66rom pools managed by HAST. Instead such pools should be imported by a script executed from HA software when it decides it should be primary. Also I'd recommend to avoid adding info about HAST pools to the /boot/zfs/zpool.cache file. You can do that by adding '-c none' option to 'zpool import'. This will tell ZFS not to cache info about the pool in zpool.cache. --=20 Pawel Jakub Dawidek http://www.wheelsystems.com FreeBSD committer http://www.FreeBSD.org Am I Evil? Yes, I Am! http://yomoli.com --jy6Sn24JjFx/iggw Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAk5kjR4ACgkQForvXbEpPzQcIQCfb9pe83DsyQYf4t+Tc7W5L7K7 eykAn29EgCqHBzM6FLTYg8by5/rRC+GE =8iWP -----END PGP SIGNATURE----- --jy6Sn24JjFx/iggw-- From owner-freebsd-fs@FreeBSD.ORG Mon Sep 5 09:16:46 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0D0A7106566B for ; Mon, 5 Sep 2011 09:16:46 +0000 (UTC) (envelope-from pawel@dawidek.net) Received: from mail.dawidek.net (60.wheelsystems.com [83.12.187.60]) by mx1.freebsd.org (Postfix) with ESMTP id B60238FC15 for ; Mon, 5 Sep 2011 09:16:45 +0000 (UTC) Received: from localhost (58.wheelsystems.com [83.12.187.58]) by mail.dawidek.net (Postfix) with ESMTPSA id 3FBE4D23; Mon, 5 Sep 2011 11:16:44 +0200 (CEST) Date: Mon, 5 Sep 2011 11:16:16 +0200 From: Pawel Jakub Dawidek To: David Brodbeck Message-ID: <20110905091616.GD1662@garage.freebsd.pl> References: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="iVCmgExH7+hIHJ1A" Content-Disposition: inline In-Reply-To: X-OS: FreeBSD 9.0-CURRENT amd64 User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@freebsd.org Subject: Re: ZFSv28+NFSv4 poor file creation performance, "sync=disabled" has no effect X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Sep 2011 09:16:46 -0000 --iVCmgExH7+hIHJ1A Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Sep 02, 2011 at 09:29:33AM -0700, David Brodbeck wrote: > I originally posted this on FreeBSD-questions, but it was suggested that I > bring it here. > I'm testing FreeBSD 9.0-BETA with an eye toward eventually using > FreeBSD 9.0 to replace some existing OpenSolaris 2008.11 > installations. I've found NFS file creation performance (as measured > by Bonnie++) is equally slow for both with default settings. However, > on OpenSolaris I disable the ZIL to improve file creation performance. > This tuning parameter was removed from FreeBSD 9.0; its replacement > is supposed to be the per-filesystem flag "sync", but setting this > flag seems to have no effect. >=20 > I did recompile the FreeBSD kernel without debugging features before > doing the tests, so I don't think this is a case of debugging code > slowing things down. >=20 > Here's the relevant data; these are all from bonnie++'s "sequential > create" benchmark. The NFS client was RedHat Enterprise Linux 5.6. >=20 > OpenSolaris 2008.11, default settings: 58/second > OpenSolaris 2008.11, with "zil_disable=3D1": 1258/second >=20 > FreeBSD 9.0-BETA, default settings: 107/second > FreeBSD 9.0-BETA, with "sync=3Ddisabled": 106/second >=20 > So it appears the "sync" ZFS parameter has no effect in FreeBSD. Has > anyone else seen this? Is there a way to improve NFS file creation > performance now that zil_disable has been removed? It would be good if you could try your test on FreeBSD directly. This way we could see if ZFS is to blame or NFS. --=20 Pawel Jakub Dawidek http://www.wheelsystems.com FreeBSD committer http://www.FreeBSD.org Am I Evil? Yes, I Am! http://yomoli.com --iVCmgExH7+hIHJ1A Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAk5kk2AACgkQForvXbEpPzSrgQCg7xJH3ZJM2KsuwlS4b0LGI6Vt rFEAoOEbRXKqHID2yLZjaFz6PEJJRPbj =I40w -----END PGP SIGNATURE----- --iVCmgExH7+hIHJ1A-- From owner-freebsd-fs@FreeBSD.ORG Mon Sep 5 11:07:08 2011 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9E3901065670 for ; Mon, 5 Sep 2011 11:07:08 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 499D08FC1B for ; Mon, 5 Sep 2011 11:07:08 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p85B781u065867 for ; Mon, 5 Sep 2011 11:07:08 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p85B77Px065865 for freebsd-fs@FreeBSD.org; Mon, 5 Sep 2011 11:07:07 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 5 Sep 2011 11:07:07 GMT Message-Id: <201109051107.p85B77Px065865@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-fs@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Sep 2011 11:07:08 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/160410 fs [smbfs] [hang] smbfs hangs when transferring large fil o kern/160283 fs [zfs] [patch] 'zfs list' does abort in make_dataset_ha o kern/159971 fs [ffs] [panic] panic with soft updates journaling durin o kern/159930 fs [ufs] [panic] kernel core o kern/159418 fs [tmpfs] [panic] tmpfs kernel panic: recursing on non r o kern/159402 fs [zfs][loader] symlinks cause I/O errors o kern/159357 fs [zfs] ZFS MAXNAMELEN macro has confusing name (off-by- o kern/159356 fs [zfs] [patch] ZFS NAME_ERR_DISKLIKE check is Solaris-s o kern/159351 fs [nfs] [patch] - divide by zero in mountnfs() o kern/159251 fs [zfs] [request]: add FLETCHER4 as DEDUP hash option o kern/159233 fs [ext2fs] [patch] fs/ext2fs: finish reallocblk implemen o kern/159232 fs [ext2fs] [patch] fs/ext2fs: merge ext2_readwrite into o kern/159077 fs [zfs] Can't cd .. with latest zfs version o kern/159048 fs [smbfs] smb mount corrupts large files o kern/159045 fs [zfs] [hang] ZFS scrub freezes system o kern/158839 fs [zfs] ZFS Bootloader Fails if there is a Dead Disk o kern/158802 fs [amd] amd(8) ICMP storm and unkillable process. o kern/158711 fs [ffs] [panic] panic in ffs_blkfree and ffs_valloc o kern/158231 fs [nullfs] panic on unmounting nullfs mounted over ufs o f kern/157929 fs [nfs] NFS slow read o kern/157722 fs [geli] unable to newfs a geli encrypted partition o kern/157399 fs [zfs] trouble with: mdconfig force delete && zfs strip o kern/157179 fs [zfs] zfs/dbuf.c: panic: solaris assert: arc_buf_remov o kern/156797 fs [zfs] [panic] Double panic with FreeBSD 9-CURRENT and o kern/156781 fs [zfs] zfs is losing the snapshot directory, p kern/156545 fs [ufs] mv could break UFS on SMP systems o kern/156193 fs [ufs] [hang] UFS snapshot hangs && deadlocks processes o kern/156168 fs [nfs] [panic] Kernel panic under concurrent access ove o kern/156039 fs [nullfs] [unionfs] nullfs + unionfs do not compose, re o kern/155615 fs [zfs] zfs v28 broken on sparc64 -current o kern/155587 fs [zfs] [panic] kernel panic with zfs o kern/155411 fs [regression] [8.2-release] [tmpfs]: mount: tmpfs : No o kern/155199 fs [ext2fs] ext3fs mounted as ext2fs gives I/O errors o bin/155104 fs [zfs][patch] use /dev prefix by default when importing o kern/154930 fs [zfs] cannot delete/unlink file from full volume -> EN o kern/154828 fs [msdosfs] Unable to create directories on external USB o kern/154491 fs [smbfs] smb_co_lock: recursive lock for object 1 o kern/154447 fs [zfs] [panic] Occasional panics - solaris assert somew p kern/154228 fs [md] md getting stuck in wdrain state o kern/153996 fs [zfs] zfs root mount error while kernel is not located o kern/153847 fs [nfs] [panic] Kernel panic from incorrect m_free in nf o kern/153753 fs [zfs] ZFS v15 - grammatical error when attempting to u o kern/153716 fs [zfs] zpool scrub time remaining is incorrect o kern/153695 fs [patch] [zfs] Booting from zpool created on 4k-sector o kern/153680 fs [xfs] 8.1 failing to mount XFS partitions o kern/153520 fs [zfs] Boot from GPT ZFS root on HP BL460c G1 unstable o kern/153418 fs [zfs] [panic] Kernel Panic occurred writing to zfs vol o kern/153351 fs [zfs] locking directories/files in ZFS o bin/153258 fs [patch][zfs] creating ZVOLs requires `refreservation' s kern/153173 fs [zfs] booting from a gzip-compressed dataset doesn't w o kern/153126 fs [zfs] vdev failure, zpool=peegel type=vdev.too_small p kern/152488 fs [tmpfs] [patch] mtime of file updated when only inode o kern/152022 fs [nfs] nfs service hangs with linux client [regression] o kern/151942 fs [zfs] panic during ls(1) zfs snapshot directory o kern/151905 fs [zfs] page fault under load in /sbin/zfs o kern/151845 fs [smbfs] [patch] smbfs should be upgraded to support Un o bin/151713 fs [patch] Bug in growfs(8) with respect to 32-bit overfl o kern/151648 fs [zfs] disk wait bug o kern/151629 fs [fs] [patch] Skip empty directory entries during name o kern/151330 fs [zfs] will unshare all zfs filesystem after execute a o kern/151326 fs [nfs] nfs exports fail if netgroups contain duplicate o kern/151251 fs [ufs] Can not create files on filesystem with heavy us o kern/151226 fs [zfs] can't delete zfs snapshot o kern/151111 fs [zfs] vnodes leakage during zfs unmount o kern/150503 fs [zfs] ZFS disks are UNAVAIL and corrupted after reboot o kern/150501 fs [zfs] ZFS vdev failure vdev.bad_label on amd64 o kern/150390 fs [zfs] zfs deadlock when arcmsr reports drive faulted o kern/150336 fs [nfs] mountd/nfsd became confused; refused to reload n o kern/150207 fs zpool(1): zpool import -d /dev tries to open weird dev o kern/149208 fs mksnap_ffs(8) hang/deadlock o kern/149173 fs [patch] [zfs] make OpenSolaris installa o kern/149015 fs [zfs] [patch] misc fixes for ZFS code to build on Glib o kern/149014 fs [zfs] [patch] declarations in ZFS libraries/utilities o kern/149013 fs [zfs] [patch] make ZFS makefiles use the libraries fro o kern/148504 fs [zfs] ZFS' zpool does not allow replacing drives to be o kern/148490 fs [zfs]: zpool attach - resilver bidirectionally, and re o kern/148368 fs [zfs] ZFS hanging forever on 8.1-PRERELEASE o bin/148296 fs [zfs] [loader] [patch] Very slow probe in /usr/src/sys o kern/148204 fs [nfs] UDP NFS causes overload o kern/148138 fs [zfs] zfs raidz pool commands freeze o kern/147903 fs [zfs] [panic] Kernel panics on faulty zfs device o kern/147881 fs [zfs] [patch] ZFS "sharenfs" doesn't allow different " o kern/147790 fs [zfs] zfs set acl(mode|inherit) fails on existing zfs o kern/147560 fs [zfs] [boot] Booting 8.1-PRERELEASE raidz system take o kern/147420 fs [ufs] [panic] ufs_dirbad, nullfs, jail panic (corrupt o kern/146941 fs [zfs] [panic] Kernel Double Fault - Happens constantly o kern/146786 fs [zfs] zpool import hangs with checksum errors o kern/146708 fs [ufs] [panic] Kernel panic in softdep_disk_write_compl o kern/146528 fs [zfs] Severe memory leak in ZFS on i386 o kern/146502 fs [nfs] FreeBSD 8 NFS Client Connection to Server s kern/145712 fs [zfs] cannot offline two drives in a raidz2 configurat o kern/145411 fs [xfs] [panic] Kernel panics shortly after mounting an o bin/145309 fs bsdlabel: Editing disk label invalidates the whole dev o kern/145272 fs [zfs] [panic] Panic during boot when accessing zfs on o kern/145246 fs [ufs] dirhash in 7.3 gratuitously frees hashes when it o kern/145238 fs [zfs] [panic] kernel panic on zpool clear tank o kern/145229 fs [zfs] Vast differences in ZFS ARC behavior between 8.0 o kern/145189 fs [nfs] nfsd performs abysmally under load o kern/144929 fs [ufs] [lor] vfs_bio.c + ufs_dirhash.c p kern/144447 fs [zfs] sharenfs fsunshare() & fsshare_main() non functi o kern/144416 fs [panic] Kernel panic on online filesystem optimization s kern/144415 fs [zfs] [panic] kernel panics on boot after zfs crash o kern/144234 fs [zfs] Cannot boot machine with recent gptzfsboot code o kern/143825 fs [nfs] [panic] Kernel panic on NFS client o bin/143572 fs [zfs] zpool(1): [patch] The verbose output from iostat o kern/143212 fs [nfs] NFSv4 client strange work ... o kern/143184 fs [zfs] [lor] zfs/bufwait LOR o kern/142878 fs [zfs] [vfs] lock order reversal o kern/142597 fs [ext2fs] ext2fs does not work on filesystems with real o kern/142489 fs [zfs] [lor] allproc/zfs LOR o kern/142466 fs Update 7.2 -> 8.0 on Raid 1 ends with screwed raid [re o kern/142306 fs [zfs] [panic] ZFS drive (from OSX Leopard) causes two o kern/142068 fs [ufs] BSD labels are got deleted spontaneously o kern/141897 fs [msdosfs] [panic] Kernel panic. msdofs: file name leng o kern/141463 fs [nfs] [panic] Frequent kernel panics after upgrade fro o kern/141305 fs [zfs] FreeBSD ZFS+sendfile severe performance issues ( o kern/141091 fs [patch] [nullfs] fix panics with DIAGNOSTIC enabled o kern/141086 fs [nfs] [panic] panic("nfs: bioread, not dir") on FreeBS o kern/141010 fs [zfs] "zfs scrub" fails when backed by files in UFS2 o kern/140888 fs [zfs] boot fail from zfs root while the pool resilveri o kern/140661 fs [zfs] [patch] /boot/loader fails to work on a GPT/ZFS- o kern/140640 fs [zfs] snapshot crash o kern/140068 fs [smbfs] [patch] smbfs does not allow semicolon in file o kern/139725 fs [zfs] zdb(1) dumps core on i386 when examining zpool c o kern/139715 fs [zfs] vfs.numvnodes leak on busy zfs p bin/139651 fs [nfs] mount(8): read-only remount of NFS volume does n o kern/139597 fs [patch] [tmpfs] tmpfs initializes va_gen but doesn't u o kern/139564 fs [zfs] [panic] 8.0-RC1 - Fatal trap 12 at end of shutdo o kern/139407 fs [smbfs] [panic] smb mount causes system crash if remot o kern/138662 fs [panic] ffs_blkfree: freeing free block o kern/138421 fs [ufs] [patch] remove UFS label limitations o kern/138202 fs mount_msdosfs(1) see only 2Gb o kern/136968 fs [ufs] [lor] ufs/bufwait/ufs (open) o kern/136945 fs [ufs] [lor] filedesc structure/ufs (poll) o kern/136944 fs [ffs] [lor] bufwait/snaplk (fsync) o kern/136873 fs [ntfs] Missing directories/files on NTFS volume o kern/136865 fs [nfs] [patch] NFS exports atomic and on-the-fly atomic p kern/136470 fs [nfs] Cannot mount / in read-only, over NFS o kern/135546 fs [zfs] zfs.ko module doesn't ignore zpool.cache filenam o kern/135469 fs [ufs] [panic] kernel crash on md operation in ufs_dirb o kern/135050 fs [zfs] ZFS clears/hides disk errors on reboot o kern/134491 fs [zfs] Hot spares are rather cold... o kern/133676 fs [smbfs] [panic] umount -f'ing a vnode-based memory dis o kern/133174 fs [msdosfs] [patch] msdosfs must support multibyte inter o kern/132960 fs [ufs] [panic] panic:ffs_blkfree: freeing free frag o kern/132397 fs reboot causes filesystem corruption (failure to sync b o kern/132331 fs [ufs] [lor] LOR ufs and syncer o kern/132237 fs [msdosfs] msdosfs has problems to read MSDOS Floppy o kern/132145 fs [panic] File System Hard Crashes o kern/131441 fs [unionfs] [nullfs] unionfs and/or nullfs not combineab o kern/131360 fs [nfs] poor scaling behavior of the NFS server under lo o kern/131342 fs [nfs] mounting/unmounting of disks causes NFS to fail o bin/131341 fs makefs: error "Bad file descriptor" on the mount poin o kern/130920 fs [msdosfs] cp(1) takes 100% CPU time while copying file o kern/130210 fs [nullfs] Error by check nullfs f kern/130133 fs [panic] [zfs] 'kmem_map too small' caused by make clea o kern/129760 fs [nfs] after 'umount -f' of a stale NFS share FreeBSD l o kern/129488 fs [smbfs] Kernel "bug" when using smbfs in smbfs_smb.c: o kern/129231 fs [ufs] [patch] New UFS mount (norandom) option - mostly o kern/129152 fs [panic] non-userfriendly panic when trying to mount(8) o kern/127787 fs [lor] [ufs] Three LORs: vfslock/devfs/vfslock, ufs/vfs f kern/127375 fs [zfs] If vm.kmem_size_max>"1073741823" then write spee o bin/127270 fs fsck_msdosfs(8) may crash if BytesPerSec is zero o kern/127029 fs [panic] mount(8): trying to mount a write protected zi f kern/126703 fs [panic] [zfs] _mtx_lock_sleep: recursed on non-recursi o kern/126287 fs [ufs] [panic] Kernel panics while mounting an UFS file o kern/125895 fs [ffs] [panic] kernel: panic: ffs_blkfree: freeing free s kern/125738 fs [zfs] [request] SHA256 acceleration in ZFS o kern/123939 fs [msdosfs] corrupts new files f sparc/123566 fs [zfs] zpool import issue: EOVERFLOW o kern/122380 fs [ffs] ffs_valloc:dup alloc (Soekris 4801/7.0/USB Flash o bin/122172 fs [fs]: amd(8) automount daemon dies on 6.3-STABLE i386, o bin/121898 fs [nullfs] pwd(1)/getcwd(2) fails with Permission denied o bin/121366 fs [zfs] [patch] Automatic disk scrubbing from periodic(8 o bin/121072 fs [smbfs] mount_smbfs(8) cannot normally convert the cha o kern/120483 fs [ntfs] [patch] NTFS filesystem locking changes o kern/120482 fs [ntfs] [patch] Sync style changes between NetBSD and F f kern/120210 fs [zfs] [panic] reboot after panic: solaris assert: arc_ o kern/118912 fs [2tb] disk sizing/geometry problem with large array o kern/118713 fs [minidump] [patch] Display media size required for a k o bin/118249 fs [ufs] mv(1): moving a directory changes its mtime o kern/118126 fs [nfs] [patch] Poor NFS server write performance o kern/118107 fs [ntfs] [panic] Kernel panic when accessing a file at N o kern/117954 fs [ufs] dirhash on very large directories blocks the mac o bin/117315 fs [smbfs] mount_smbfs(8) and related options can't mount o kern/117314 fs [ntfs] Long-filename only NTFS fs'es cause kernel pani o kern/117158 fs [zfs] zpool scrub causes panic if geli vdevs detach on o bin/116980 fs [msdosfs] [patch] mount_msdosfs(8) resets some flags f o conf/116931 fs lack of fsck_cd9660 prevents mounting iso images with o kern/116583 fs [ffs] [hang] System freezes for short time when using o bin/115361 fs [zfs] mount(8) gets into a state where it won't set/un o kern/114955 fs [cd9660] [patch] [request] support for mask,dirmask,ui o kern/114847 fs [ntfs] [patch] [request] dirmask support for NTFS ala o kern/114676 fs [ufs] snapshot creation panics: snapacct_ufs2: bad blo o bin/114468 fs [patch] [request] add -d option to umount(8) to detach o kern/113852 fs [smbfs] smbfs does not properly implement DFS referral o bin/113838 fs [patch] [request] mount(8): add support for relative p o bin/113049 fs [patch] [request] make quot(8) use getopt(3) and show o kern/112658 fs [smbfs] [patch] smbfs and caching problems (resolves b o kern/111843 fs [msdosfs] Long Names of files are incorrectly created o kern/111782 fs [ufs] dump(8) fails horribly for large filesystems s bin/111146 fs [2tb] fsck(8) fails on 6T filesystem o kern/109024 fs [msdosfs] [iconv] mount_msdosfs: msdosfs_iconv: Operat o kern/109010 fs [msdosfs] can't mv directory within fat32 file system o bin/107829 fs [2TB] fdisk(8): invalid boundary checking in fdisk / w o kern/106107 fs [ufs] left-over fsck_snapshot after unfinished backgro o kern/104406 fs [ufs] Processes get stuck in "ufs" state under persist o kern/104133 fs [ext2fs] EXT2FS module corrupts EXT2/3 filesystems o kern/103035 fs [ntfs] Directories in NTFS mounted disc images appear o kern/101324 fs [smbfs] smbfs sometimes not case sensitive when it's s o kern/99290 fs [ntfs] mount_ntfs ignorant of cluster sizes s bin/97498 fs [request] newfs(8) has no option to clear the first 12 o kern/97377 fs [ntfs] [patch] syntax cleanup for ntfs_ihash.c o kern/95222 fs [cd9660] File sections on ISO9660 level 3 CDs ignored o kern/94849 fs [ufs] rename on UFS filesystem is not atomic o bin/94810 fs fsck(8) incorrectly reports 'file system marked clean' o kern/94769 fs [ufs] Multiple file deletions on multi-snapshotted fil o kern/94733 fs [smbfs] smbfs may cause double unlock o kern/93942 fs [vfs] [patch] panic: ufs_dirbad: bad dir (patch from D o kern/92272 fs [ffs] [hang] Filling a filesystem while creating a sna o kern/91134 fs [smbfs] [patch] Preserve access and modification time a kern/90815 fs [smbfs] [patch] SMBFS with character conversions somet o kern/88657 fs [smbfs] windows client hang when browsing a samba shar o kern/88555 fs [panic] ffs_blkfree: freeing free frag on AMD 64 o kern/88266 fs [smbfs] smbfs does not implement UIO_NOCOPY and sendfi o bin/87966 fs [patch] newfs(8): introduce -A flag for newfs to enabl o kern/87859 fs [smbfs] System reboot while umount smbfs. o kern/86587 fs [msdosfs] rm -r /PATH fails with lots of small files o bin/85494 fs fsck_ffs: unchecked use of cg_inosused macro etc. o kern/80088 fs [smbfs] Incorrect file time setting on NTFS mounted vi o bin/74779 fs Background-fsck checks one filesystem twice and omits o kern/73484 fs [ntfs] Kernel panic when doing `ls` from the client si o bin/73019 fs [ufs] fsck_ufs(8) cannot alloc 607016868 bytes for ino o kern/71774 fs [ntfs] NTFS cannot "see" files on a WinXP filesystem o bin/70600 fs fsck(8) throws files away when it can't grow lost+foun o kern/68978 fs [panic] [ufs] crashes with failing hard disk, loose po o kern/65920 fs [nwfs] Mounted Netware filesystem behaves strange o kern/65901 fs [smbfs] [patch] smbfs fails fsx write/truncate-down/tr o kern/61503 fs [smbfs] mount_smbfs does not work as non-root o kern/55617 fs [smbfs] Accessing an nsmb-mounted drive via a smb expo o kern/51685 fs [hang] Unbounded inode allocation causes kernel to loc o kern/51583 fs [nullfs] [patch] allow to work with devices and socket o kern/36566 fs [smbfs] System reboot with dead smb mount and umount o kern/33464 fs [ufs] soft update inconsistencies after system crash o bin/27687 fs fsck(8) wrapper is not properly passing options to fsc o kern/18874 fs [2TB] 32bit NFS servers export wrong negative values t 246 problems total. From owner-freebsd-fs@FreeBSD.ORG Mon Sep 5 11:43:53 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EAA30106566C for ; Mon, 5 Sep 2011 11:43:53 +0000 (UTC) (envelope-from willem@nlnetlabs.nl) Received: from open.nlnetlabs.nl (open.nlnetlabs.nl [IPv6:2001:7b8:206:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id 5F9F38FC13 for ; Mon, 5 Sep 2011 11:43:53 +0000 (UTC) Received: from [IPv6:2001:7b8:206:1:216:76ff:fecd:6a66] (wilm.nlnetlabs.nl [IPv6:2001:7b8:206:1:216:76ff:fecd:6a66]) (authenticated bits=0) by open.nlnetlabs.nl (8.14.4/8.14.4) with ESMTP id p85BhpFS043852 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO) for ; Mon, 5 Sep 2011 13:43:52 +0200 (CEST) (envelope-from willem@nlnetlabs.nl) Message-ID: <4E64B5ED.9040508@nlnetlabs.nl> Date: Mon, 05 Sep 2011 13:43:41 +0200 From: Willem Toorop User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.20) Gecko/20110805 Thunderbird/3.1.12 MIME-Version: 1.0 To: freebsd-fs@freebsd.org X-Enigmail-Version: 1.1.2 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (open.nlnetlabs.nl [IPv6:2001:7b8:206:1::1]); Mon, 05 Sep 2011 13:43:52 +0200 (CEST) X-Spam-Status: No, score=-102.6 required=5.0 tests=AWL, BAYES_00, HELO_NO_DOMAIN, RP_MATCHES_RCVD,USER_IN_WHITELIST autolearn=ham version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on open.nlnetlabs.nl Subject: Cannot destroy zfs filesystem: I/O error X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Sep 2011 11:43:54 -0000 Hi, Due to a crash, one of my zfs filesystems broke (see output of zpool status -v below). According to the reference ( http://www.sun.com/msg/ZFS-8000-8A ) I just need the remove the broken file or directory. However tank/home/willem is a zfs filesystem itself. I cannot list, get properties, rename or destroy it. With everything I try, a "I/O error" is returned: # zfs list tank/home/willem cannot open 'tank/home/willem': I/O error # zfs get all tank/home/willem cannot open 'tank/home/willem': I/O error # zfs rename tank/home/willem tank/home/willem2 cannot open 'tank/home/willem': I/O error # zfs destroy tank/home/willem cannot open 'tank/home/willem': I/O error How do I get zfs to forget about tank/home/willem completely? The error is really annoying as it also breaks regular zfs practice: # zfs list -r tank/home cannot iterate filesystems: I/O error NAME USED AVAIL REFER MOUNTPOINT tank/home 46.2G 304G 2.91G /home This should have listed all home filesystems, but now only lists /home and the error: "cannot iterate filesystems: I/O error". -- Willem zpool status -v pool: tank state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scrub: scrub completed after 0h14m with 1 errors on Mon Sep 5 00:30:44 2011 config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 79 ad0s3d ONLINE 0 0 161 ad0s4d ONLINE 0 0 167 ad2s2d ONLINE 0 0 0 ad2s1h ONLINE 0 0 0 errors: Permanent errors have been detected in the following files: tank/home/willem:<0x0> From owner-freebsd-fs@FreeBSD.ORG Mon Sep 5 19:55:09 2011 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 01573106566B for ; Mon, 5 Sep 2011 19:55:08 +0000 (UTC) (envelope-from jeremie@le-hen.org) Received: from smtp5-g21.free.fr (unknown [IPv6:2a01:e0c:1:1599::14]) by mx1.freebsd.org (Postfix) with ESMTP id 665228FC0C for ; Mon, 5 Sep 2011 19:55:06 +0000 (UTC) Received: from endor.tataz.chchile.org (unknown [82.233.239.98]) by smtp5-g21.free.fr (Postfix) with ESMTP id 3586BD480CD; Mon, 5 Sep 2011 21:55:00 +0200 (CEST) Received: from felucia.tataz.chchile.org (felucia.tataz.chchile.org [192.168.1.9]) by endor.tataz.chchile.org (Postfix) with ESMTP id 2328533E05; Mon, 5 Sep 2011 19:54:59 +0000 (UTC) Received: by felucia.tataz.chchile.org (Postfix, from userid 1000) id 11DCCA131B; Mon, 5 Sep 2011 19:54:59 +0000 (UTC) Date: Mon, 5 Sep 2011 21:54:58 +0200 From: Jeremie Le Hen To: freebsd-fs@FreeBSD.org Message-ID: <20110905195458.GA7863@felucia.tataz.chchile.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) Cc: jeremie@le-hen.org Subject: Difficulties to use ZFS root: ROOT MOUNT ERROR X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Sep 2011 19:55:09 -0000 Hi list, I've followed the instructions documented here: http://wiki.freebsd.org/RootOnZFS/ZFSBootPartition The kernel starts correctly so this rules out any problem regarding boot0, zfsboot and ZFS loader. But when the kernel tries to mount the root filesystem, it fails with the following output: % Trying to mount root from zfs:zroot % ROOT MOUNT ERROR: % If you have invalid mount options, reboot, and first try the following from % the loader prompt: % % set vfs.root.mountfrom.options=rw % % and then remove the invalid mount options from /etc/fstab. % % Loader variables: % vfs.root.mountfrom=zfs:zroot % vfs.root.mountfrom.options=rw >From a netboot'd FreeBSD: # zfs import zroot % # zpool get bootfs zroot % NAME PROPERTY VALUE SOURCE % zroot bootfs zroot local % % # zfs list -o name,canmount,mountpoint % NAME CANMOUNT MOUNTPOINT % zroot on legacy % zroot/tmp on /tmp % zroot/usr on /usr % zroot/usr/home on /usr/home % zroot/usr/ports on /usr/ports % zroot/usr/ports/distfiles on /usr/ports/distfiles % zroot/usr/ports/packages on /usr/ports/packages % zroot/usr/src on /usr/src % zroot/usr/src8 on /usr/src8 % zroot/var on /var % zroot/var/crash on /var/crash % zroot/var/db on /var/db % zroot/var/db/pkg on /var/db/pkg % zroot/var/empty on /var/empty % zroot/var/log on /var/log % zroot/var/mail on /var/mail % zroot/var/run on /var/run % zroot/var/tmp on /var/tmp % # zfs export zroot /boot/zfs/zpool.cache exists in the zroot filesystem: % # zpool import -R /mnt zroot % # zfs set mountpoint=/ zroot % # ls -l /mnt/boot/zfs/zpool.cache % -rw-r--r-- 1 root wheel 924 Sep 5 07:31 /mnt/boot/zfs/zpool.cache % # grep zfs /mnt/boot/loader.conf /mnt/etc/rc.conf % /mnt/boot/loader.conf:zfs_load="YES" % /mnt/boot/loader.conf:vfs.root.mountfrom="zfs:zroot" % /mnt/etc/rc.conf:zfs_enable="YES" Any idea why this error occurs? Thanks -- Jeremie Le Hen Men are born free and equal. Later on, they're on their own. Jean Yanne From owner-freebsd-fs@FreeBSD.ORG Mon Sep 5 20:13:25 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B04F41065670 for ; Mon, 5 Sep 2011 20:13:25 +0000 (UTC) (envelope-from universite@ukr.net) Received: from otrada.od.ua (universite-1-pt.tunnel.tserv24.sto1.ipv6.he.net [IPv6:2001:470:27:140::2]) by mx1.freebsd.org (Postfix) with ESMTP id 1F7C48FC14 for ; Mon, 5 Sep 2011 20:13:24 +0000 (UTC) Received: from [IPv6:2001:470:28:140:91c2:948c:1352:ddcd] ([IPv6:2001:470:28:140:91c2:948c:1352:ddcd]) (authenticated bits=0) by otrada.od.ua (8.14.4/8.14.5) with ESMTP id p85KDLLj053164 for ; Mon, 5 Sep 2011 23:13:21 +0300 (EEST) (envelope-from universite@ukr.net) Message-ID: <4E652D58.1060304@ukr.net> Date: Mon, 05 Sep 2011 23:13:12 +0300 From: "Vladislav V. Prodan" User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:6.0.1) Gecko/20110830 Thunderbird/6.0.1 MIME-Version: 1.0 To: freebsd-fs@freebsd.org References: <20110905195458.GA7863@felucia.tataz.chchile.org> In-Reply-To: <20110905195458.GA7863@felucia.tataz.chchile.org> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-95.5 required=5.0 tests=FREEMAIL_FROM,FSL_RU_URL, RDNS_NONE, SPF_SOFTFAIL, T_TO_NO_BRKTS_FREEMAIL, USER_IN_WHITELIST autolearn=no version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mary-teresa.otrada.od.ua X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (otrada.od.ua [IPv6:2001:470:28:140::5]); Mon, 05 Sep 2011 23:13:23 +0300 (EEST) Subject: Re: Difficulties to use ZFS root: ROOT MOUNT ERROR X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Sep 2011 20:13:25 -0000 05.09.2011 22:54, Jeremie Le Hen wrote: > The kernel starts correctly so this rules out any problem regarding > boot0, zfsboot and ZFS loader. > > But when the kernel tries to mount the root filesystem, it fails with > the following output: > > % > > and then remove the > > invalid mount options from /etc/fstab. > > A > > ny idea why this err > > or occurs? cat /etc/fstab uname -r -- Vladislav V. Prodan VVP24-UANIC +380[67]4584408 +380[99]4060508 xmpp:vlad11@jabber.ru From owner-freebsd-fs@FreeBSD.ORG Mon Sep 5 21:04:02 2011 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CCBAE1065674 for ; Mon, 5 Sep 2011 21:04:02 +0000 (UTC) (envelope-from mm@FreeBSD.org) Received: from mail.vx.sk (mail.vx.sk [IPv6:2a01:4f8:100:1043::3]) by mx1.freebsd.org (Postfix) with ESMTP id 544428FC14 for ; Mon, 5 Sep 2011 21:04:02 +0000 (UTC) Received: from core.vx.sk (localhost [127.0.0.1]) by mail.vx.sk (Postfix) with ESMTP id 4AFF919B2E9; Mon, 5 Sep 2011 23:04:01 +0200 (CEST) X-Virus-Scanned: amavisd-new at mail.vx.sk Received: from mail.vx.sk ([127.0.0.1]) by core.vx.sk (mail.vx.sk [127.0.0.1]) (amavisd-new, port 10024) with LMTP id NSw0VF2LA6IR; Mon, 5 Sep 2011 23:03:59 +0200 (CEST) Received: from [10.9.8.1] (188-167-78-15.dynamic.chello.sk [188.167.78.15]) by mail.vx.sk (Postfix) with ESMTPSA id 18E0619B2DE; Mon, 5 Sep 2011 23:03:58 +0200 (CEST) Message-ID: <4E65393F.9070401@FreeBSD.org> Date: Mon, 05 Sep 2011 23:03:59 +0200 From: Martin Matuska User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:6.0.1) Gecko/20110830 Thunderbird/6.0.1 MIME-Version: 1.0 To: Jeremie Le Hen References: <20110905195458.GA7863@felucia.tataz.chchile.org> In-Reply-To: <20110905195458.GA7863@felucia.tataz.chchile.org> X-Enigmail-Version: 1.3.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org Subject: Re: Difficulties to use ZFS root: ROOT MOUNT ERROR X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Sep 2011 21:04:03 -0000 On 5. 9. 2011 21:54, Jeremie Le Hen wrote: > Hi list, > > I've followed the instructions documented here: > http://wiki.freebsd.org/RootOnZFS/ZFSBootPartition > > The kernel starts correctly so this rules out any problem regarding > boot0, zfsboot and ZFS loader. > > But when the kernel tries to mount the root filesystem, it fails with > the following output: > > % Trying to mount root from zfs:zroot > % ROOT MOUNT ERROR: > % If you have invalid mount options, reboot, and first try the following from > % the loader prompt: > % > % set vfs.root.mountfrom.options=rw > % > % and then remove the invalid mount options from /etc/fstab. > % > % Loader variables: > % vfs.root.mountfrom=zfs:zroot > % vfs.root.mountfrom.options=rw > > >From a netboot'd FreeBSD: > > # zfs import zroot > % # zpool get bootfs zroot > % NAME PROPERTY VALUE SOURCE > % zroot bootfs zroot local > % > % # zfs list -o name,canmount,mountpoint > % NAME CANMOUNT MOUNTPOINT > % zroot on legacy > % zroot/tmp on /tmp > % zroot/usr on /usr > % zroot/usr/home on /usr/home > % zroot/usr/ports on /usr/ports > % zroot/usr/ports/distfiles on /usr/ports/distfiles > % zroot/usr/ports/packages on /usr/ports/packages > % zroot/usr/src on /usr/src > % zroot/usr/src8 on /usr/src8 > % zroot/var on /var > % zroot/var/crash on /var/crash > % zroot/var/db on /var/db > % zroot/var/db/pkg on /var/db/pkg > % zroot/var/empty on /var/empty > % zroot/var/log on /var/log > % zroot/var/mail on /var/mail > % zroot/var/run on /var/run > % zroot/var/tmp on /var/tmp > % # zfs export zroot > > /boot/zfs/zpool.cache exists in the zroot filesystem: > > % # zpool import -R /mnt zroot > % # zfs set mountpoint=/ zroot > % # ls -l /mnt/boot/zfs/zpool.cache > % -rw-r--r-- 1 root wheel 924 Sep 5 07:31 /mnt/boot/zfs/zpool.cache > % # grep zfs /mnt/boot/loader.conf /mnt/etc/rc.conf > % /mnt/boot/loader.conf:zfs_load="YES" > % /mnt/boot/loader.conf:vfs.root.mountfrom="zfs:zroot" > % /mnt/etc/rc.conf:zfs_enable="YES" > > > Any idea why this error occurs? > > Thanks It might be a problem in the zpool.cache. If you read the zpool(8) manpage properly, you will find this: -R root Equivalent to "-o cachefile=none,altroot=root" If you mount a pool with an alternate root and want to update the cachefile, you have to explicitly state the cachefile. (e.g. zpool import -o altroot=/mnt -o cachefile=/tmp/zpool.cache zroot) Second, you should not have an exported pool for booting (from viewpoint of target system's zfs.cache). Third, you don't need a legacy mount for zroot. You can leave it to "/" but you don't have to. I personally prefer having everyting one level deeper (e.g. pool/root, pool/root/var, etc.). Therefore I suggest: zpool import -o altroot=/mnt -o cachefile=/tmp/zpool.cache zroot zfs set mountpoint=/ zroot cp /tmp/zpool.cache /mnt/boot/zfs/zpool.cache shutdown -r now (as you can see I have not exported the pool) P.S: mfsBSD is your friend here - http://mfsbsd.vx.sk -- Martin Matuska FreeBSD committer http://blog.vx.sk From owner-freebsd-fs@FreeBSD.ORG Tue Sep 6 14:28:52 2011 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5A6101065672; Tue, 6 Sep 2011 14:28:52 +0000 (UTC) (envelope-from jeremie@le-hen.org) Received: from smtp5-g21.free.fr (unknown [IPv6:2a01:e0c:1:1599::14]) by mx1.freebsd.org (Postfix) with ESMTP id 34C158FC16; Tue, 6 Sep 2011 14:28:46 +0000 (UTC) Received: from endor.tataz.chchile.org (unknown [82.233.239.98]) by smtp5-g21.free.fr (Postfix) with ESMTP id 1F4F0D48175; Tue, 6 Sep 2011 16:28:34 +0200 (CEST) Received: from felucia.tataz.chchile.org (felucia.tataz.chchile.org [192.168.1.9]) by endor.tataz.chchile.org (Postfix) with ESMTP id 09A1A33E05; Tue, 6 Sep 2011 14:28:34 +0000 (UTC) Received: by felucia.tataz.chchile.org (Postfix, from userid 1000) id EC667A121F; Tue, 6 Sep 2011 14:28:33 +0000 (UTC) Date: Tue, 6 Sep 2011 16:28:33 +0200 From: Jeremie Le Hen To: Martin Matuska Message-ID: <20110906142833.GG75185@felucia.tataz.chchile.org> References: <20110905195458.GA7863@felucia.tataz.chchile.org> <4E65393F.9070401@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4E65393F.9070401@FreeBSD.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@FreeBSD.org, Jeremie Le Hen Subject: Re: Difficulties to use ZFS root: ROOT MOUNT ERROR X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Sep 2011 14:28:52 -0000 On Mon, Sep 05, 2011 at 11:03:59PM +0200, Martin Matuska wrote: > Second, you should not have an exported pool for booting (from viewpoint > of target system's zfs.cache). I think that was the problem. I've rebooted without exporting my pool and the server successfully booted. Thanks a lot. Regards, -- Jeremie Le Hen Men are born free and equal. Later on, they're on their own. Jean Yanne From owner-freebsd-fs@FreeBSD.ORG Tue Sep 6 16:45:46 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1ADA6106566C for ; Tue, 6 Sep 2011 16:45:46 +0000 (UTC) (envelope-from brodbd@uw.edu) Received: from mail-ey0-f172.google.com (mail-ey0-f172.google.com [209.85.215.172]) by mx1.freebsd.org (Postfix) with ESMTP id 8A7B68FC17 for ; Tue, 6 Sep 2011 16:45:41 +0000 (UTC) Received: by eye4 with SMTP id 4so3905118eye.31 for ; Tue, 06 Sep 2011 09:45:41 -0700 (PDT) MIME-Version: 1.0 Received: by 10.213.20.219 with SMTP id g27mr1433966ebb.85.1315327540806; Tue, 06 Sep 2011 09:45:40 -0700 (PDT) Received: by 10.213.26.75 with HTTP; Tue, 6 Sep 2011 09:45:40 -0700 (PDT) In-Reply-To: <20110904012855.GA79580@icarus.home.lan> References: <14220705.747900.1315013770239.JavaMail.root@erie.cs.uoguelph.ca> <20110904012855.GA79580@icarus.home.lan> Date: Tue, 6 Sep 2011 09:45:40 -0700 Message-ID: From: David Brodbeck To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Re: ZFSv28+NFSv4 poor file creation performance, "sync=disabled" has no effect X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Sep 2011 16:45:46 -0000 On Sat, Sep 3, 2011 at 6:28 PM, Jeremy Chadwick wrote: > > Also, most other filesystems make no guarantees that data has been > > flushed to the platters, only that it's been sent to the disk > > hardware, so it really isn't a big increase in risk compared to our > > pre-ZFS setup. > > Can you explain where you got this impression from? Please provide > actual references/validation/proof for non-network filesystems. > My understanding is most filesystems only flush to the hardware disk cache, they don't force the disks to write to the platters like ZFS does, and that's why ZFS's NFS performance is slow compared to, say, UFS. It could be I'm mistaken. What we (the community) need clarification regarding is whether or not > OpenSolaris and/or Illumos still provides the zil_disable tunable on > those OSes; if they do, FreeBSD should provide the same (which means > removable of zil_disable on FreeBSD is effectively a regression). If > said OSes do not provide it, then FreeBSD should not provide it. If > said OSes moved to using the sync parameter where it works, yet it does > not work on FreeBSD, then that's a bug and a PR should be filed + > discussion induced. > My understanding is the upstream filesystems removed that feature and added the sync parameter. Testing against OpenIndiana to see whether sync makes a difference there is on my to-do list. I may get to it today; if so I'll let you know what I find. > The risks/complexities and performance concerns with SSDs when used as > part of a ZFS filesystem on FreeBSD greatly differ from that of using an > SSD as a dedicated intent log (ZIL) device. > Could you elaborate? I'm hearing a lot of stories of massive slowdowns after SSDs have been in use for a while, which doesn't sound promising if I'm going to add one to improve performance. Also my understanding is losing the ZIL means irretrievably losing the whole storage pool, so it creates another point of failure. > > What exactly do you need ZFS for given your requirements? Is it that > you want a simple-to-use LVM-like filesystem that offers a vast > in-memory cache but without the data integrity bits? > It's mostly the easy filesystem management I'm after. The problem with LVM-like solutions is you don't generally get a shared free space pool and expanding or shrinking a filesystem is a multi-step manual process. -- David Brodbeck System Administrator, Linguistics University of Washington From owner-freebsd-fs@FreeBSD.ORG Tue Sep 6 16:46:46 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F334E106566B for ; Tue, 6 Sep 2011 16:46:46 +0000 (UTC) (envelope-from brodbd@uw.edu) Received: from mail-ew0-f54.google.com (mail-ew0-f54.google.com [209.85.215.54]) by mx1.freebsd.org (Postfix) with ESMTP id 8D75C8FC15 for ; Tue, 6 Sep 2011 16:46:46 +0000 (UTC) Received: by ewy1 with SMTP id 1so3421774ewy.13 for ; Tue, 06 Sep 2011 09:46:45 -0700 (PDT) MIME-Version: 1.0 Received: by 10.213.14.18 with SMTP id e18mr1432593eba.47.1315327605118; Tue, 06 Sep 2011 09:46:45 -0700 (PDT) Received: by 10.213.26.75 with HTTP; Tue, 6 Sep 2011 09:46:45 -0700 (PDT) In-Reply-To: <20110905091616.GD1662@garage.freebsd.pl> References: <20110905091616.GD1662@garage.freebsd.pl> Date: Tue, 6 Sep 2011 09:46:45 -0700 Message-ID: From: David Brodbeck To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Re: ZFSv28+NFSv4 poor file creation performance, "sync=disabled" has no effect X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Sep 2011 16:46:47 -0000 On Mon, Sep 5, 2011 at 2:16 AM, Pawel Jakub Dawidek wrote: > It would be good if you could try your test on FreeBSD directly. This > way we could see if ZFS is to blame or NFS. > I'd be happy to try that if you know of a benchmarking utility that does synchronous writes. Local benchmarking unfortunately doesn't show the problem because bonnie++ doesn't sync after write. -- David Brodbeck System Administrator, Linguistics University of Washington From owner-freebsd-fs@FreeBSD.ORG Tue Sep 6 20:22:28 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 185A3106566C for ; Tue, 6 Sep 2011 20:22:28 +0000 (UTC) (envelope-from brodbd@uw.edu) Received: from mail-ew0-f54.google.com (mail-ew0-f54.google.com [209.85.215.54]) by mx1.freebsd.org (Postfix) with ESMTP id A4FD58FC08 for ; Tue, 6 Sep 2011 20:22:27 +0000 (UTC) Received: by ewy1 with SMTP id 1so3560921ewy.13 for ; Tue, 06 Sep 2011 13:22:26 -0700 (PDT) MIME-Version: 1.0 Received: by 10.213.13.136 with SMTP id c8mr1087323eba.123.1315340546267; Tue, 06 Sep 2011 13:22:26 -0700 (PDT) Received: by 10.213.26.75 with HTTP; Tue, 6 Sep 2011 13:22:26 -0700 (PDT) Date: Tue, 6 Sep 2011 13:22:26 -0700 Message-ID: From: David Brodbeck To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Slow NFSv4 performance, was: Re: ZFSv28+NFSv4 poor file creation performance, "sync=disabled" has no effect X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Sep 2011 20:22:28 -0000 On Mon, Sep 5, 2011 at 2:16 AM, Pawel Jakub Dawidek wrote: > It would be good if you could try your test on FreeBSD directly. This > way we could see if ZFS is to blame or NFS. > Okay, it appears it is in fact an NFSv4 problem, and I've been barking up the wrong tree. I shared out an NFSv4 mountpoint with a UFS ramdisk as the backing store, and I got the same slow results. I was fooled by the fact that the numbers were similar to what I got using OpenSolaris with the ZIL enabled. (This is what I get for making assumptions.) Interestingly enough, if I use NFSv3 instead of NFSv4, performance increases dramatically. It appears it's FreeBSD's NFSv4 server, or the way it's interacting with Linux's client, that's the culprit here. I'm not sure if there are knobs I should be tweaking to make it perform better; any suggestions on what to try next? -- David Brodbeck System Administrator, Linguistics University of Washington From owner-freebsd-fs@FreeBSD.ORG Wed Sep 7 04:48:04 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BB8551065670 for ; Wed, 7 Sep 2011 04:48:04 +0000 (UTC) (envelope-from peterjeremy@acm.org) Received: from mail14.syd.optusnet.com.au (mail14.syd.optusnet.com.au [211.29.132.195]) by mx1.freebsd.org (Postfix) with ESMTP id 4EB498FC08 for ; Wed, 7 Sep 2011 04:48:04 +0000 (UTC) Received: from server.vk2pj.dyndns.org (c220-239-116-103.belrs4.nsw.optusnet.com.au [220.239.116.103]) by mail14.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id p874m1dY004603 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 7 Sep 2011 14:48:02 +1000 X-Bogosity: Ham, spamicity=0.000000 Received: from server.vk2pj.dyndns.org (localhost.vk2pj.dyndns.org [127.0.0.1]) by server.vk2pj.dyndns.org (8.14.4/8.14.4) with ESMTP id p874m0Q3030235; Wed, 7 Sep 2011 14:48:00 +1000 (EST) (envelope-from peter@server.vk2pj.dyndns.org) Received: (from peter@localhost) by server.vk2pj.dyndns.org (8.14.4/8.14.4/Submit) id p874m0QC030234; Wed, 7 Sep 2011 14:48:00 +1000 (EST) (envelope-from peter) Date: Wed, 7 Sep 2011 14:48:00 +1000 From: Peter Jeremy To: freebsd-fs@freebsd.org Message-ID: <20110907044800.GA96277@server.vk2pj.dyndns.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="h31gzZEtNLTqOjlF" Content-Disposition: inline X-PGP-Key: http://members.optusnet.com.au/peterjeremy/pubkey.asc User-Agent: Mutt/1.5.21 (2010-09-15) Cc: avg@freebsd.org Subject: "can't load 'kernel'" on ZFS root X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Sep 2011 04:48:04 -0000 --h31gzZEtNLTqOjlF Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable I have a several systems with ZFS on mirrored root and have run into problems during recent 8-stable upgrades with loader reporting it was unable to load the kernel. On the first system, I did an installkernel, rebooted and all came up OK. I then did an installworld, rebooted and got (re-typed): Loading /boot/defaults/loader.conf Unable to load a kernel! | can't load 'kernel' and the loader prompt. After some research, I found bin/144214 which suggested my bootblocks were out of date. I booted from another system and installed new bootblocks on both disks, rebooted and got the same result. I then tried backing up and restoring all the files in /boot that were touched by installworld. This time when I tried to boot, I was greeted with "ZFS i/o error - all block copies unavailable" followed by the bootblock prompt. I tried loading zroot:/boot/zfsloader.old without success. At this point, I gave up and rolled back to a snapshot prior to the installworld (but after the installkernel) and everything worked. I have since verified that the bootblocks match the newly built pmbr and gptzfsboot and that the changes in r208892 are present in the sourcetree. On the second system, I was unable to boot after the installkernel and rolled back - I haven't checked how old the bootblocks are there. Firstly, does anyone have any ideas on how to move forward? I will rebuild both zroots at some point but that's not a fix. (I am reasonably confident I can reproduce the problem if someone has a patch to test). Secondly, I found that ZFS booting is extremely fragile - if your default kernel & modules won't load automatically, there's no way to boot the backup kernel because (as far as I can find) there's no way to manually load /boot/zfs/zpool.cache - without which the kernel can't mount the root FS. In both above cases, recovering the system required booting from recovery media. There needs to be a documented method for booting from a snapshot or clone. --=20 Peter Jeremy --h31gzZEtNLTqOjlF Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) iEYEARECAAYFAk5m94AACgkQ/opHv/APuIeABgCfY1TwefWE71QlhsEaZjjF3klL 9KkAn15FkjhfVQscdgCfvldV3Cu8rXHy =rBSz -----END PGP SIGNATURE----- --h31gzZEtNLTqOjlF-- From owner-freebsd-fs@FreeBSD.ORG Wed Sep 7 07:35:40 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E165F1065672; Wed, 7 Sep 2011 07:35:40 +0000 (UTC) (envelope-from artemb@gmail.com) Received: from mail-gw0-f49.google.com (mail-gw0-f49.google.com [74.125.83.49]) by mx1.freebsd.org (Postfix) with ESMTP id 8F9B48FC15; Wed, 7 Sep 2011 07:35:40 +0000 (UTC) Received: by gwb1 with SMTP id 1so4559969gwb.36 for ; Wed, 07 Sep 2011 00:35:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=2vKupEoQi9rAfLQyXyVm+FKarRDsHw4JnNBqLhuKlLo=; b=FYqD55k2Mj7QGhGJD7wI8tntbrtEyVHU8dqiy9UfepCadHAVETMPALsdCDfTjHuzaK jWelKA/y9W3s5etAsluy6H6shij2fVhowQoLORlyH89TiIeBJXuP1hhfDRkqPQOedyPJ 9Rduz+ingqLTAjCoSYPecTX7CueUZ8vIQ7NHs= MIME-Version: 1.0 Received: by 10.236.200.195 with SMTP id z43mr29563297yhn.127.1315380939911; Wed, 07 Sep 2011 00:35:39 -0700 (PDT) Sender: artemb@gmail.com Received: by 10.236.102.147 with HTTP; Wed, 7 Sep 2011 00:35:39 -0700 (PDT) In-Reply-To: <20110907044800.GA96277@server.vk2pj.dyndns.org> References: <20110907044800.GA96277@server.vk2pj.dyndns.org> Date: Wed, 7 Sep 2011 00:35:39 -0700 X-Google-Sender-Auth: 5cGIi123lDfucSJBGECtKd1FWsw Message-ID: From: Artem Belevich To: Peter Jeremy Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org, avg@freebsd.org Subject: Re: "can't load 'kernel'" on ZFS root X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Sep 2011 07:35:41 -0000 On Tue, Sep 6, 2011 at 9:48 PM, Peter Jeremy wrote: > Secondly, I found that ZFS booting is extremely fragile - if your > default kernel & modules won't load automatically, there's no way to > boot the backup kernel because (as far as I can find) there's no way > to manually load /boot/zfs/zpool.cache - without which the kernel > can't mount the root FS. =A0In both above cases, recovering the system > required booting from recovery media. =A0There needs to be a documented > method for booting from a snapshot or clone. The magic command is: load -t /boot/zfs/zpool.cache /boot/zfs/zpool.cache It makes me wonder, though -- if we're probing devices anyways, why is zpool.cache existence mandatory? According to the name it's a *cache*, presumably to speed up zpool detection on a normal boot. Perhaps we can fall back to probing all drives if zpool.cache is missing. Slower boot definitely beats no booting at all. --Artem From owner-freebsd-fs@FreeBSD.ORG Wed Sep 7 09:17:11 2011 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8F7D61065670 for ; Wed, 7 Sep 2011 09:17:11 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id E029E8FC1F for ; Wed, 7 Sep 2011 09:17:10 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id MAA11577; Wed, 07 Sep 2011 12:17:05 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1R1EFl-0004tJ-LL; Wed, 07 Sep 2011 12:17:05 +0300 Message-ID: <4E673690.1050100@FreeBSD.org> Date: Wed, 07 Sep 2011 12:17:04 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:6.0.2) Gecko/20110907 Thunderbird/6.0.2 MIME-Version: 1.0 To: Peter Jeremy References: <20110907044800.GA96277@server.vk2pj.dyndns.org> In-Reply-To: <20110907044800.GA96277@server.vk2pj.dyndns.org> X-Enigmail-Version: undefined Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org Subject: Re: "can't load 'kernel'" on ZFS root X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Sep 2011 09:17:11 -0000 on 07/09/2011 07:48 Peter Jeremy said the following: > Firstly, does anyone have any ideas on how to move forward? I will > rebuild both zroots at some point but that's not a fix. (I am > reasonably confident I can reproduce the problem if someone has > a patch to test). No patches, but you may want to try to further debug this problem using an approach described here: http://article.gmane.org/gmane.os.freebsd.current/135086 > Secondly, I found that ZFS booting is extremely fragile - if your > default kernel & modules won't load automatically, there's no way to > boot the backup kernel because (as far as I can find) there's no way > to manually load /boot/zfs/zpool.cache - without which the kernel > can't mount the root FS. In both above cases, recovering the system > required booting from recovery media. There needs to be a documented > method for booting from a snapshot or clone. AFAIK, FreeBSD doesn't support booting from ZFS snapshots (I wonder if any OS supports that). Additionally, FreeBSD currently doesn't provide an interactive way to choose from which fs to boot the kernel. I have a patch that provides that capability, but haven't found time yet to replace some hacks that I used with the proper code: http://people.freebsd.org/~avg/zfsboot.diff But the above is more of an advertising. Your main premise is incorrect. It is very well possible and trivial to boot a different kernel (on the same fs). See this PR for more details: http://www.freebsd.org/cgi/query-pr.cgi?pr=153804 -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Wed Sep 7 09:20:21 2011 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8FFFB106566B; Wed, 7 Sep 2011 09:20:21 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 4CB0A8FC16; Wed, 7 Sep 2011 09:20:19 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id MAA11623; Wed, 07 Sep 2011 12:20:18 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1R1EIs-0004tT-Dq; Wed, 07 Sep 2011 12:20:18 +0300 Message-ID: <4E673751.5080503@FreeBSD.org> Date: Wed, 07 Sep 2011 12:20:17 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:6.0.2) Gecko/20110907 Thunderbird/6.0.2 MIME-Version: 1.0 To: Artem Belevich , Pawel Jakub Dawidek , =?windows-1252?Q?Martin_Matu=9Aka?= References: <20110907044800.GA96277@server.vk2pj.dyndns.org> In-Reply-To: X-Enigmail-Version: undefined Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org Subject: Re: "can't load 'kernel'" on ZFS root X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Sep 2011 09:20:21 -0000 on 07/09/2011 10:35 Artem Belevich said the following: > It makes me wonder, though -- if we're probing devices anyways, why is > zpool.cache existence mandatory? According to the name it's a *cache*, > presumably to speed up zpool detection on a normal boot. Perhaps we > can fall back to probing all drives if zpool.cache is missing. Slower > boot definitely beats no booting at all. Very good point indeed. Pawel, Martin, do you know how the relevant code works? I suspect that you do :-) Maybe this could be improved trivially?... -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Wed Sep 7 09:42:09 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2A781106566C for ; Wed, 7 Sep 2011 09:42:09 +0000 (UTC) (envelope-from daniel@digsys.bg) Received: from smtp-sofia.digsys.bg (smtp-sofia.digsys.bg [193.68.3.230]) by mx1.freebsd.org (Postfix) with ESMTP id A7D958FC08 for ; Wed, 7 Sep 2011 09:42:08 +0000 (UTC) Received: from [192.92.129.101] ([192.92.129.101]) (authenticated bits=0) by smtp-sofia.digsys.bg (8.14.4/8.14.4) with ESMTP id p879frWh031522 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO) for ; Wed, 7 Sep 2011 12:42:01 +0300 (EEST) (envelope-from daniel@digsys.bg) Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Apple Message framework v1244.3) From: Daniel Kalchev In-Reply-To: <4E673751.5080503@FreeBSD.org> Date: Wed, 7 Sep 2011 12:41:53 +0300 Content-Transfer-Encoding: quoted-printable Message-Id: References: <20110907044800.GA96277@server.vk2pj.dyndns.org> <4E673751.5080503@FreeBSD.org> To: freebsd-fs@freebsd.org X-Mailer: Apple Mail (2.1244.3) Subject: Re: "can't load 'kernel'" on ZFS root X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Sep 2011 09:42:09 -0000 On Sep 7, 2011, at 12:20 , Andriy Gapon wrote: > on 07/09/2011 10:35 Artem Belevich said the following: >> It makes me wonder, though -- if we're probing devices anyways, why = is >> zpool.cache existence mandatory? According to the name it's a = *cache*, >> presumably to speed up zpool detection on a normal boot. Perhaps we >> can fall back to probing all drives if zpool.cache is missing. Slower >> boot definitely beats no booting at all. >=20 > Very good point indeed. >=20 > Pawel, Martin, do you know how the relevant code works? I suspect = that you do > :-) Maybe this could be improved trivially?... Imagine, you have an HAST pool, that is currently exported. Exporting = the zpool removes it from the spool.cache, so it is not = recognized/mounted at boot. If your node is a HAST secondary and not = supposed to touch that zpool, then you might accidentally import/use it, = if you ignore the zpool.cache contents. Also, you might not want to import a pool, that is connected to the = server, but not intended for importing. For example, in disaster = recovery. Or if it overlaps with mount points etc. Other than that, perhaps you could consider recognizing pools that = 'belong' to that host, but not present in the cache. Daniel= From owner-freebsd-fs@FreeBSD.ORG Wed Sep 7 09:46:25 2011 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id ACA03106564A; Wed, 7 Sep 2011 09:46:25 +0000 (UTC) (envelope-from pawel@dawidek.net) Received: from mail.dawidek.net (60.wheelsystems.com [83.12.187.60]) by mx1.freebsd.org (Postfix) with ESMTP id 094908FC0A; Wed, 7 Sep 2011 09:46:25 +0000 (UTC) Received: from localhost (58.wheelsystems.com [83.12.187.58]) by mail.dawidek.net (Postfix) with ESMTPSA id B0B195D4; Wed, 7 Sep 2011 11:46:22 +0200 (CEST) Date: Wed, 7 Sep 2011 11:45:56 +0200 From: Pawel Jakub Dawidek To: Andriy Gapon Message-ID: <20110907094554.GB1674@garage.freebsd.pl> References: <20110907044800.GA96277@server.vk2pj.dyndns.org> <4E673751.5080503@FreeBSD.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="qlTNgmc+xy1dBmNv" Content-Disposition: inline In-Reply-To: <4E673751.5080503@FreeBSD.org> X-OS: FreeBSD 9.0-CURRENT amd64 User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@FreeBSD.org, Artem Belevich , Martin =?utf-8?Q?Matu=C5=A1ka?= Subject: Re: "can't load 'kernel'" on ZFS root X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Sep 2011 09:46:25 -0000 --qlTNgmc+xy1dBmNv Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Sep 07, 2011 at 12:20:17PM +0300, Andriy Gapon wrote: > on 07/09/2011 10:35 Artem Belevich said the following: > > It makes me wonder, though -- if we're probing devices anyways, why is > > zpool.cache existence mandatory? According to the name it's a *cache*, > > presumably to speed up zpool detection on a normal boot. Perhaps we > > can fall back to probing all drives if zpool.cache is missing. Slower > > boot definitely beats no booting at all. >=20 > Very good point indeed. >=20 > Pawel, Martin, do you know how the relevant code works? I suspect that y= ou do > :-) Maybe this could be improved trivially?... The zpool.cache file contains pools that are automatically imported at system start-up. There might be pools visible in the system that are not suppose to be automatically imported (eg. a pool on iSCSI disks on secondary cluster node - importing such pool automatically will corrupt the data). This is also not entirely true that we probe all the devices. We probe them at boot, yes, because we don't even know where to look for zpool.cache file, but once kernel is loaded we use info from zpool.cache to open GEOM providers. We fall back to probing all GEOM providers if something has changed in disk configuration and we can't find all the vdevs (eg. disk name changed). --=20 Pawel Jakub Dawidek http://www.wheelsystems.com FreeBSD committer http://www.FreeBSD.org Am I Evil? Yes, I Am! http://yomoli.com --qlTNgmc+xy1dBmNv Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAk5nPVIACgkQForvXbEpPzS1+QCdGQkXLrr/c7citG0829SIEoMs TaIAoJD9z+QrEULBolLDUUf3TzwzzsUU =kdL9 -----END PGP SIGNATURE----- --qlTNgmc+xy1dBmNv-- From owner-freebsd-fs@FreeBSD.ORG Wed Sep 7 10:23:35 2011 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 39C65106564A; Wed, 7 Sep 2011 10:23:35 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id DEF378FC08; Wed, 7 Sep 2011 10:23:33 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id NAA13053; Wed, 07 Sep 2011 13:23:32 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1R1FI3-0004wl-R1; Wed, 07 Sep 2011 13:23:31 +0300 Message-ID: <4E674622.3040705@FreeBSD.org> Date: Wed, 07 Sep 2011 13:23:30 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:6.0.2) Gecko/20110907 Thunderbird/6.0.2 MIME-Version: 1.0 To: Pawel Jakub Dawidek References: <20110907044800.GA96277@server.vk2pj.dyndns.org> <4E673751.5080503@FreeBSD.org> <20110907094554.GB1674@garage.freebsd.pl> In-Reply-To: <20110907094554.GB1674@garage.freebsd.pl> X-Enigmail-Version: undefined Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org, Artem Belevich , =?windows-1252?Q?Martin_Matu=9Aka?= Subject: Re: "can't load 'kernel'" on ZFS root X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Sep 2011 10:23:35 -0000 on 07/09/2011 12:45 Pawel Jakub Dawidek said the following: > The zpool.cache file contains pools that are automatically imported at > system start-up. There might be pools visible in the system that are not > suppose to be automatically imported (eg. a pool on iSCSI disks on > secondary cluster node - importing such pool automatically will corrupt > the data). What about providing a special case for a pool from which we want to boot? Erm, not boot, but to mount a root filesystem. I suppose that we must know name/ID of that pool and if it is not present in zpool.cache, then we could try to find it. Or is this dangerous too? -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Wed Sep 7 16:35:13 2011 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7CCF3106566C; Wed, 7 Sep 2011 16:35:13 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 498B78FC08; Wed, 7 Sep 2011 16:35:12 +0000 (UTC) Received: from odyssey.starpoint.kiev.ua (alpha-e.starpoint.kiev.ua [212.40.38.101]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id TAA18844; Wed, 07 Sep 2011 19:35:10 +0300 (EEST) (envelope-from avg@FreeBSD.org) Message-ID: <4E679D3D.1000007@FreeBSD.org> Date: Wed, 07 Sep 2011 19:35:09 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:5.0) Gecko/20110705 Thunderbird/5.0 MIME-Version: 1.0 To: freebsd-current@FreeBSD.org, freebsd-fs@FreeBSD.org References: <20110901223646.14b8aae8@o2.pl> <4E60DBBD.1040703@FreeBSD.org> In-Reply-To: <4E60DBBD.1040703@FreeBSD.org> X-Enigmail-Version: 1.2pre Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: Sebastian Chmielewski Subject: Re: ZFS: i/o error - all block copies unavailable after upgrading to r225312 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Sep 2011 16:35:13 -0000 on 02/09/2011 16:35 Andriy Gapon said the following: > Then: > - obtain this patch http://people.freebsd.org/~avg/zfstest.head.diff > - cd sys/boot/zfs > - apply the patch to zfstest.c > - cc -I. -I../../cddl/boot/zfs zfstest.c -o zfstest > - run the resulting binary as root and provide your pool device(s) as > parameter(s); e.g.: > ./zfstest /dev/ada0p4 Thanks to a lot of excellent testing, debugging and analysis from Sebastian (which went behind the scenes) we now have this patch: http://people.freebsd.org/~avg/zfs-boot-gang.diff The patch introduces the following changes: - checksum is now verified for gang header blocks - checksum is now verified for reconstituted data of whole gang blocks (previously it is verified only for individual gang member leaf blocks) - reconstituted data of a whole gang block is now decompressed if the gang block is compressed The last change is _the_ change. If you use compression for a filesystem where your kernel resides and you get a problem with booting, then please test this patch and report back. Many thanks to Sebastian! Additional heap of thanks to Doug Rabson who came up with the idea and implementation of zfstest.c! This tool is of the immense help when debugging an issue like this one. -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Wed Sep 7 20:35:07 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 312C91065678; Wed, 7 Sep 2011 20:35:07 +0000 (UTC) (envelope-from joh.hendriks@gmail.com) Received: from mail-ew0-f54.google.com (mail-ew0-f54.google.com [209.85.215.54]) by mx1.freebsd.org (Postfix) with ESMTP id 6DAD68FC16; Wed, 7 Sep 2011 20:35:06 +0000 (UTC) Received: by ewy1 with SMTP id 1so27872ewy.13 for ; Wed, 07 Sep 2011 13:35:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=LlU37dm00L9mU15OUixl06hQ7ZOGF9DC4jwmbzm6ZDo=; b=m25+uL9nbFxFoW8KZ8mmgCWFH7IMeYKdRGyiuTQbq/EwlPRznNCtYdxgS87UTTRHjG Oo7LiDyAasc3UZd2QM0WGrCK0uq4FTTJlD6K6nbZ6TpGUJoE01cel1M0Xf2Pd98EUmvB h25opuFFz6mIAeySQ72+/6mLKzkjDvbUp2YiI= Received: by 10.14.18.4 with SMTP id k4mr2481272eek.30.1315427705430; Wed, 07 Sep 2011 13:35:05 -0700 (PDT) Received: from [192.168.1.13] (5ED0E470.cm-7-1d.dynamic.ziggo.nl [94.208.228.112]) by mx.google.com with ESMTPS id i6sm2782656eeb.11.2011.09.07.13.35.03 (version=SSLv3 cipher=OTHER); Wed, 07 Sep 2011 13:35:04 -0700 (PDT) Message-ID: <4E67D576.9030100@gmail.com> Date: Wed, 07 Sep 2011 22:35:02 +0200 From: Johan Hendriks User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:6.0.2) Gecko/20110902 Thunderbird/6.0.2 MIME-Version: 1.0 To: Pawel Jakub Dawidek References: <4E60D992.3030802@gmail.com> <20110905084934.GC1662@garage.freebsd.pl> In-Reply-To: <20110905084934.GC1662@garage.freebsd.pl> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: ZFS on HAST and reboot. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Sep 2011 20:35:07 -0000 On maandag 5 september 2011 10:49:37, Pawel Jakub Dawidek wrote: > On Fri, Sep 02, 2011 at 03:26:42PM +0200, Johan Hendriks wrote: >> Hello all. >> >> I just started using ZFS on top of HAST. >> >> What i did was first glabel my disks like disk1 to disk3 >> Then I created my hast devices in /etc/hast.conf >> >> /etc/hast.conf looks like this. >> i >> resource disk1 { >> on srv1 { >> local /dev/label/disk1 >> remote 192.168.5.41 >> } >> on srv2 { >> local /dev/label/disk1 >> remote 192.168.5.40 >> } >> } >> resource disk2 { >> on srv1 { >> local /dev/label/disk2 >> remote 192.168.5.41 >> } >> on srv2 { >> local /dev/label/disk2 >> remote 192.168.5.40 >> } >> } >> resource disk3 { >> on srv1 { >> local /dev/label/disk3 >> remote 192.168.5.41 >> } >> on srv2 { >> local /dev/label/disk3 >> remote 192.168.5.40 >> } >> } >> >> This works. >> I can set srv 1 to primary and srv 2 to secondary and visa versa. >> hastctl role primary all and hastctl role secondary all. >> >> Then i created the raidz on the master srv1 >> zpool create storage raidz1 hast/disk1 hast/disk2 hast/disk3 >> >> all looks good. >> zpool status >> pool: storage >> state: ONLINE >> scan: scrub repaired 0 in 0h0m with 0 errors on Wed Aug 31 20:49:19 2011 >> config: >> >> NAME STATE READ WRITE CKSUM >> storage ONLINE 0 0 0 >> raidz1-0 ONLINE 0 0 0 >> hast/disk1 ONLINE 0 0 0 >> hast/disk2 ONLINE 0 0 0 >> hast/disk3 ONLINE 0 0 0 >> >> errors: No known data errors >> >> then i created the mountpoint and created zfs on it >> # mkdir /usr/local/virtual >> # zfs create storage/virtual >> # zfs list >> # zfs set mountpoint=/usr/local/virtual storage/virtual >> >> # /etc/rc.d/zfs start and whooop there is my /usr/local/virtual zfs >> filesystem. >> # mount >> /dev/ada0p2 on / (ufs, local, journaled soft-updates) >> devfs on /dev (devfs, local, multilabel) >> storage on /storage (zfs, local, nfsv4acls) >> storage/virtual on /usr/local/virtual (zfs, local, nfsv4acls) >> >> if i do a zfs export -f storage on srv1 change the hast role to >> secondary and then set the hast role on srv2 to primary and do zfs >> import -f storage, i can see the files on srv2. >> >> I am a happy camper :D >> >> So it works like advertised. >> Now i rebooted both machines. >> all is working fine. >> >> But if i reboot the server srv1 again, i can not import the pool >> anymore, it tells me the pool is already imported. >> I do load the carp-hast-switch master file with ifstated. >> This does set the hast role to primary. >> But can not import the pool. >> Now this can be true because i did not export it. >> if i do a /etc/rc.d/zfs start, than it gets mounted and the pool is >> again available. >> >> Is there a way i can do this automaticly. >> In my understanding after a reboot zfs try's to start, but fails because >> my hast providers are not yet ready. >> Or am i doing something wrong and should i not do it this way. >> Can i tell zfs to start after the hast providers are primary at reboot. > > You can see the message that pool is already imported, because when you > reboot primary there is still info about the pool in > /boot/zfs/zpool.cache. Pools that are mentioned in this file are > automatically imported on boot (by the kernel), so importing such a pool > will fail. You should still be able to mount file systems (zfs mount -a). > > What I'd recommend is not to use /etc/rc.d/zfs to mount file systems > from pools managed by HAST. Instead such pools should be imported by a > script executed from HA software when it decides it should be primary. > > Also I'd recommend to avoid adding info about HAST pools to the > /boot/zfs/zpool.cache file. You can do that by adding '-c none' option > to 'zpool import'. This will tell ZFS not to cache info about the pool > in zpool.cache. > Thanks for your answer. One thing i can not seem to get done is the -c none option. # zpool import -c none storage failed to open cache file: No such file or directory It looks like it is looking for a cache file named none and not as advertised do not search or do not not cache the pool. Gr johan From owner-freebsd-fs@FreeBSD.ORG Wed Sep 7 22:19:36 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D3C30106564A for ; Wed, 7 Sep 2011 22:19:36 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.mail.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 8D9778FC15 for ; Wed, 7 Sep 2011 22:19:36 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqAEALzsZ06DaFvO/2dsb2JhbABDDoRHpBWBRgEBAQECAQEBASArIAsFFg4KAgINEgcCKQEJJgYIBwQBHASHVASlPJF3gSyELoERBJEgghKJfYYuM1Q X-IronPort-AV: E=Sophos;i="4.68,347,1312171200"; d="scan'208";a="133619405" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-annu-pri.mail.uoguelph.ca with ESMTP; 07 Sep 2011 18:19:35 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 8347BB3F0C; Wed, 7 Sep 2011 18:19:35 -0400 (EDT) Date: Wed, 7 Sep 2011 18:19:35 -0400 (EDT) From: Rick Macklem To: David Brodbeck Message-ID: <1356496286.948283.1315433975522.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.201] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - FF3.0 (Win)/6.0.10_GA_2692) Cc: freebsd-fs@freebsd.org Subject: Re: Slow NFSv4 performance, was: Re: ZFSv28+NFSv4 poor file creation performance, "sync=disabled" has no effect X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Sep 2011 22:19:37 -0000 David Brodbeck wrote: > On Mon, Sep 5, 2011 at 2:16 AM, Pawel Jakub Dawidek > wrote: > > > It would be good if you could try your test on FreeBSD directly. > > This > > way we could see if ZFS is to blame or NFS. > > > > Okay, it appears it is in fact an NFSv4 problem, and I've been barking > up > the wrong tree. I shared out an NFSv4 mountpoint with a UFS ramdisk as > the > backing store, and I got the same slow results. I was fooled by the > fact > that the numbers were similar to what I got using OpenSolaris with the > ZIL > enabled. (This is what I get for making assumptions.) > > Interestingly enough, if I use NFSv3 instead of NFSv4, performance > increases > dramatically. > Yea, I just tried a quick test using Fedora15 mounting the FreeBSD NFS server and then extracted a small tarball. Here's what the RPC counts were for NFSv3 and NFSv4 mounts (Open replaces Create for file creation in NFSv4): Client Getattr Setattr Read Write Create Open Fedora-NFSv3 12 249 35 75 60 0 Fedora-NFSv4 744 249 35 75 0 121 FreeBSDNFSv3 64 189 35 75 60 0 FreeBSDNFSv4 943 189 35 75 0 61 Two obvious things: 1 - Fedora15 does 2 Opens for each create, for NFSv4. 2 - NFSv4 does a massive # of Getattrs, however most of these will be in the compound RPCs with other Ops, so it doesn't mean that these are creating a larger load on the server. Put another way, the Getattrs done as a part of other RPCs don't get counted for NFSv3, but they are separate Ops in the NFSv4 compound and, therefore, do get counted. *** This was NOT a serious benchmark and I didn't bother to list all the RPC counts. Do not read this as a serious performance comparison, it was simply meant to get a hint as to why there was a difference between NFSv3 and NFSv4 for this case. So, my best guess is that doing twice as many Opens (vs NFSv3 Create RPCs) is why you see the difference. I'd also guess that disabling the ZIL on ZFS made the Opens perform faster, which helped alleviate the problem. (An NFSv4 Open operation will end up doing both VOP_GETATTR() and VOP_CREATE(). It will also do a VOP_SETATTR() for the exclusive open case.) > It appears it's FreeBSD's NFSv4 server, or the way it's interacting > with > Linux's client, that's the culprit here. I'm not sure if there are > knobs I > should be tweaking to make it perform better; any suggestions on what > to try > next? > Sorry, but I can't think of anything within the NFS server. It just does the operations that the client hands to it by performing the VOP_xxx() calls. I will take a look at a packet trace, but I doubt I'll learn anything useful from it. rick > -- > David Brodbeck > System Administrator, Linguistics > University of Washington > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Wed Sep 7 23:13:31 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DCBCF106564A for ; Wed, 7 Sep 2011 23:13:31 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 96D858FC08 for ; Wed, 7 Sep 2011 23:13:31 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqAEAOT5Z06DaFvO/2dsb2JhbABDDoRHpBWBRgEBAQECAQEBASArIAsFFg4KAgINEgcCKQEJJgYIBwQBHASHVASmApF6gSyELoERBJEgghKJfYYuM1Q X-IronPort-AV: E=Sophos;i="4.68,347,1312171200"; d="scan'208";a="136868082" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-jnhn-pri.mail.uoguelph.ca with ESMTP; 07 Sep 2011 19:13:30 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 90E64B3F32; Wed, 7 Sep 2011 19:13:30 -0400 (EDT) Date: Wed, 7 Sep 2011 19:13:30 -0400 (EDT) From: Rick Macklem To: David Brodbeck Message-ID: <12598691.949771.1315437210575.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.202] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - FF3.0 (Win)/6.0.10_GA_2692) Cc: freebsd-fs@freebsd.org Subject: Re: Slow NFSv4 performance, was: Re: ZFSv28+NFSv4 poor file creation performance, "sync=disabled" has no effect X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Sep 2011 23:13:31 -0000 David Brodbeck wrote: > On Mon, Sep 5, 2011 at 2:16 AM, Pawel Jakub Dawidek > wrote: > > > It would be good if you could try your test on FreeBSD directly. > > This > > way we could see if ZFS is to blame or NFS. > > > > Okay, it appears it is in fact an NFSv4 problem, and I've been barking > up > the wrong tree. I shared out an NFSv4 mountpoint with a UFS ramdisk as > the > backing store, and I got the same slow results. I was fooled by the > fact > that the numbers were similar to what I got using OpenSolaris with the > ZIL > enabled. (This is what I get for making assumptions.) > > Interestingly enough, if I use NFSv3 instead of NFSv4, performance > increases > dramatically. > Oops, it turned out that the 121 Opens for Fedora15 NFSv4 was because I had delegations enabled on the server (which is not the default). When I disabled that, I saw 61 Opens, just like the FreeBSD client and as I would have expected. So, the # of Opens is not the issue. Looking at the packet trace, I did notice that the Open compound is rather complex. It includes the following Ops: PUTFH SAVEFH OPEN GETFH GETATTR RESTOREFH GETATTR (most of these, except SAVEFH/RESTOREFH result in at least one VOP_xxx().) But, I'm afraid that doesn't really help you. Sorry, but no inspiration here, rick ps: Please ignore the last post. It was mostly irrelevant, except that enabling delegations will make it even worse in this case. > It appears it's FreeBSD's NFSv4 server, or the way it's interacting > with > Linux's client, that's the culprit here. I'm not sure if there are > knobs I > should be tweaking to make it perform better; any suggestions on what > to try > next? > > -- > David Brodbeck > System Administrator, Linguistics > University of Washington > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Wed Sep 7 23:17:56 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3DD791065675 for ; Wed, 7 Sep 2011 23:17:56 +0000 (UTC) (envelope-from brodbd@uw.edu) Received: from mail-ey0-f172.google.com (mail-ey0-f172.google.com [209.85.215.172]) by mx1.freebsd.org (Postfix) with ESMTP id BDBC68FC18 for ; Wed, 7 Sep 2011 23:17:55 +0000 (UTC) Received: by eye4 with SMTP id 4so112108eye.31 for ; Wed, 07 Sep 2011 16:17:54 -0700 (PDT) MIME-Version: 1.0 Received: by 10.213.14.18 with SMTP id e18mr1333eba.47.1315437474216; Wed, 07 Sep 2011 16:17:54 -0700 (PDT) Received: by 10.213.26.75 with HTTP; Wed, 7 Sep 2011 16:17:54 -0700 (PDT) In-Reply-To: <1356496286.948283.1315433975522.JavaMail.root@erie.cs.uoguelph.ca> References: <1356496286.948283.1315433975522.JavaMail.root@erie.cs.uoguelph.ca> Date: Wed, 7 Sep 2011 16:17:54 -0700 Message-ID: From: David Brodbeck To: Rick Macklem Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-fs@freebsd.org Subject: Re: Slow NFSv4 performance, was: Re: ZFSv28+NFSv4 poor file creation performance, "sync=disabled" has no effect X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Sep 2011 23:17:56 -0000 On Wed, Sep 7, 2011 at 3:19 PM, Rick Macklem wrote: > Yea, I just tried a quick test using Fedora15 mounting the FreeBSD NFS > server and then extracted a small tarball. Here's what the RPC counts > were for NFSv3 and NFSv4 mounts (Open replaces Create for file creation > in NFSv4): > > Client Getattr Setattr Read Write Create Open > Fedora-NFSv3 12 249 35 75 60 0 > Fedora-NFSv4 744 249 35 75 0 121 > FreeBSDNFSv3 64 189 35 75 60 0 > FreeBSDNFSv4 943 189 35 75 0 61 > > Two obvious things: > 1 - Fedora15 does 2 Opens for each create, for NFSv4. > 2 - NFSv4 does a massive # of Getattrs, however most of these will > be in the compound RPCs with other Ops, so it doesn't mean > that these are creating a larger load on the server. > Put another way, the Getattrs done as a part of other RPCs > don't get counted for NFSv3, but they are separate Ops in the > NFSv4 compound and, therefore, do get counted. > *** This was NOT a serious benchmark and I didn't bother to list all > the RPC counts. Do not read this as a serious performance comparison, > it was simply meant to get a hint as to why there was a difference > between NFSv3 and NFSv4 for this case. > I've also found this isn't just a Linux client problem. I tested a FreeBSD-9.0 client against a FreeBSD-9.0 server. Bonnie++ won't run on FreeBSD NFS mounts due to PR kern/26142, so I hacked up a quick script that times how long it takes to create 10,000 files. I averaged times over four runs; the backing filesystem was a UFS ramdisk. FreeBSD client -> FreeBSD server, NFSv4: 95.43s FreeBSD client -> FreeBSD server, NFSv3: 24.77s So basically NFSv4 against a FreeBSD server is nearly 4x slower at doing file creation than NFSv3, even when there's no disk latency involved. I did the same test with Linux and the results were similar, although the performance gap wasn't as wide: Linux client -> FreeBSD server, NFSv4: 71.25s Linux client -> FreeBSD server, NFSv3: 34.08s Sorry, but I can't think of anything within the NFS server. It just does > the operations that the client hands to it by performing the VOP_xxx() > calls. > I will take a look at a packet trace, but I doubt I'll learn anything > useful > from it. > Hm, okay. Unfortunately this has a big impact on us, because file creation speed happens to be one of the more user-visible aspects of server performance on our system. -- David Brodbeck System Administrator, Linguistics University of Washington From owner-freebsd-fs@FreeBSD.ORG Wed Sep 7 23:28:03 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7F2A81065673 for ; Wed, 7 Sep 2011 23:28:03 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 381B38FC17 for ; Wed, 7 Sep 2011 23:28:02 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ap4EAHf9Z06DaFvO/2dsb2JhbABDDoRHpBWBRgEBBSMEUhsOCgICDRIHAlkGE64KkXSBLIQugREEkzKQKzNU X-IronPort-AV: E=Sophos;i="4.68,348,1312171200"; d="scan'208";a="136869019" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-jnhn-pri.mail.uoguelph.ca with ESMTP; 07 Sep 2011 19:28:02 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 42D35B3F35; Wed, 7 Sep 2011 19:28:02 -0400 (EDT) Date: Wed, 7 Sep 2011 19:28:02 -0400 (EDT) From: Rick Macklem To: David Brodbeck Message-ID: <1890483230.950189.1315438082185.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.201] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - FF3.0 (Win)/6.0.10_GA_2692) Cc: freebsd-fs@freebsd.org Subject: Re: Slow NFSv4 performance, was: Re: ZFSv28+NFSv4 poor file creation performance, "sync=disabled" has no effect X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Sep 2011 23:28:03 -0000 David Brodbeck wrote: > On Wed, Sep 7, 2011 at 3:19 PM, Rick Macklem < rmacklem@uoguelph.ca > > wrote: > > > > Yea, I just tried a quick test using Fedora15 mounting the FreeBSD NFS > server and then extracted a small tarball. Here's what the RPC counts > were for NFSv3 and NFSv4 mounts (Open replaces Create for file > creation > in NFSv4): > > Client Getattr Setattr Read Write Create Open > Fedora-NFSv3 12 249 35 75 60 0 > Fedora-NFSv4 744 249 35 75 0 121 > FreeBSDNFSv3 64 189 35 75 60 0 > FreeBSDNFSv4 943 189 35 75 0 61 > > Two obvious things: > 1 - Fedora15 does 2 Opens for each create, for NFSv4. > 2 - NFSv4 does a massive # of Getattrs, however most of these will > be in the compound RPCs with other Ops, so it doesn't mean > that these are creating a larger load on the server. > Put another way, the Getattrs done as a part of other RPCs > don't get counted for NFSv3, but they are separate Ops in the > NFSv4 compound and, therefore, do get counted. > *** This was NOT a serious benchmark and I didn't bother to list all > the RPC counts. Do not read this as a serious performance comparison, > it was simply meant to get a hint as to why there was a difference > between NFSv3 and NFSv4 for this case. > > > > I've also found this isn't just a Linux client problem. > > > I tested a FreeBSD-9.0 client against a FreeBSD-9.0 server. Bonnie++ > won't run on FreeBSD NFS mounts due to PR kern/26142, so I hacked up a > quick script that times how long it takes to create 10,000 files. I > averaged times over four runs; the backing filesystem was a UFS > ramdisk. > > > FreeBSD client -> FreeBSD server, NFSv4: 95.43s > FreeBSD client -> FreeBSD server, NFSv3: 24.77s > > > So basically NFSv4 against a FreeBSD server is nearly 4x slower at > doing file creation than NFSv3, even when there's no disk latency > involved. > > > I did the same test with Linux and the results were similar, although > the performance gap wasn't as wide: > > > Linux client -> FreeBSD server, NFSv4: 71.25s > Linux client -> FreeBSD server, NFSv3: 34.08s > Yep, doesn't surprise me that the FreeBSD client generates the same result. Bottom line is that an NFSv4 Open (especially the exclusive open variant) is much more complex than an NFSv3 Create RPC. I showed how many Ops are in the Linux NFSv4 Open in the last post. Basically, unless you need the better byte range locking or ACLs, you can just use NFSv3. I don't think NFSv4 will ever replace NFSv3. It was designed for a somewhat different problem space. > > > Sorry, but I can't think of anything within the NFS server. It just > does > the operations that the client hands to it by performing the VOP_xxx() > calls. > I will take a look at a packet trace, but I doubt I'll learn anything > useful > from it. > > > > Hm, okay. Unfortunately this has a big impact on us, because file > creation speed happens to be one of the more user-visible aspects of > server performance on our system. > > -- > > David Brodbeck > System Administrator, Linguistics > University of Washington From owner-freebsd-fs@FreeBSD.ORG Wed Sep 7 23:35:15 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8E206106564A for ; Wed, 7 Sep 2011 23:35:15 +0000 (UTC) (envelope-from brodbd@uw.edu) Received: from mail-ew0-f54.google.com (mail-ew0-f54.google.com [209.85.215.54]) by mx1.freebsd.org (Postfix) with ESMTP id 23AC78FC16 for ; Wed, 7 Sep 2011 23:35:14 +0000 (UTC) Received: by ewy1 with SMTP id 1so88954ewy.13 for ; Wed, 07 Sep 2011 16:35:14 -0700 (PDT) MIME-Version: 1.0 Received: by 10.213.14.18 with SMTP id e18mr5488eba.47.1315438513775; Wed, 07 Sep 2011 16:35:13 -0700 (PDT) Received: by 10.213.26.75 with HTTP; Wed, 7 Sep 2011 16:35:13 -0700 (PDT) In-Reply-To: <1890483230.950189.1315438082185.JavaMail.root@erie.cs.uoguelph.ca> References: <1890483230.950189.1315438082185.JavaMail.root@erie.cs.uoguelph.ca> Date: Wed, 7 Sep 2011 16:35:13 -0700 Message-ID: From: David Brodbeck To: Rick Macklem Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-fs@freebsd.org Subject: Re: Slow NFSv4 performance, was: Re: ZFSv28+NFSv4 poor file creation performance, "sync=disabled" has no effect X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Sep 2011 23:35:15 -0000 On Wed, Sep 7, 2011 at 4:28 PM, Rick Macklem wrote: > Basically, unless you need the better byte range locking or ACLs, you can > just > use NFSv3. I don't think NFSv4 will ever replace NFSv3. It was designed > for a somewhat different problem space. > Unfortunately I do need the ACLs. We have a lot of shared projects and the group membership limit in NFS bites us pretty badly, so we use ACLs to work around it. There are other things in NFSv4 we need too, but ACLs are a biggie. I think I'm going to look at OpenIndiana next. It should give similar performance to what we're seeing with OpenSolaris, but unlike OpenSolaris it's still maintained. A major reason I was hoping to go with FreeBSD instead (other than familiarity) is better hardware support, though. Finding disk controllers that work with Solaris is a real headache. -- David Brodbeck System Administrator, Linguistics University of Washington From owner-freebsd-fs@FreeBSD.ORG Thu Sep 8 05:50:23 2011 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A02FE1065674; Thu, 8 Sep 2011 05:50:23 +0000 (UTC) (envelope-from peterjeremy@acm.org) Received: from mail14.syd.optusnet.com.au (mail14.syd.optusnet.com.au [211.29.132.195]) by mx1.freebsd.org (Postfix) with ESMTP id 300AF8FC15; Thu, 8 Sep 2011 05:50:22 +0000 (UTC) Received: from server.vk2pj.dyndns.org (c220-239-116-103.belrs4.nsw.optusnet.com.au [220.239.116.103]) by mail14.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id p885oJF0013751 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 8 Sep 2011 15:50:20 +1000 X-Bogosity: Ham, spamicity=0.000000 Received: from server.vk2pj.dyndns.org (localhost.vk2pj.dyndns.org [127.0.0.1]) by server.vk2pj.dyndns.org (8.14.5/8.14.4) with ESMTP id p885oIRO031819; Thu, 8 Sep 2011 15:50:18 +1000 (EST) (envelope-from peter@server.vk2pj.dyndns.org) Received: (from peter@localhost) by server.vk2pj.dyndns.org (8.14.5/8.14.4/Submit) id p885oGP8031818; Thu, 8 Sep 2011 15:50:16 +1000 (EST) (envelope-from peter) Date: Thu, 8 Sep 2011 15:50:16 +1000 From: Peter Jeremy To: Andriy Gapon Message-ID: <20110908055016.GB28874@server.vk2pj.dyndns.org> References: <20110901223646.14b8aae8@o2.pl> <4E60DBBD.1040703@FreeBSD.org> <4E679D3D.1000007@FreeBSD.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="9jxsPFA5p3P2qPhR" Content-Disposition: inline In-Reply-To: <4E679D3D.1000007@FreeBSD.org> X-PGP-Key: http://members.optusnet.com.au/peterjeremy/pubkey.asc User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@FreeBSD.org, freebsd-current@FreeBSD.org, Sebastian Chmielewski Subject: Re: ZFS: i/o error - all block copies unavailable after upgrading to r225312 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Sep 2011 05:50:23 -0000 --9jxsPFA5p3P2qPhR Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2011-Sep-07 19:35:09 +0300, Andriy Gapon wrote: >Thanks to a lot of excellent testing, debugging and analysis from Sebastian >(which went behind the scenes) we now have this patch: >http://people.freebsd.org/~avg/zfs-boot-gang.diff =2E.. >If you use compression for a filesystem where your kernel resides and >you get a problem with booting, then please test this patch and >report back. Since the problem isn't consistent (it can appear on some files but not others), I modified zfstest to take a pathname, rather than have the pathname hard-coded. Using this, I found an example file that works with the patch but not with the old ZFS code: pjdesk% ./zfstest.old /boot/kernel.old/iwn6000fw.ko.symbols /dev/ad0p3 |le= ss ZFS: i/o error - all block copies unavailable can't lookup pjdesk% ./zfstest.old /boot/kernel.old/iwn6000fw.ko.symbols /dev/ad0p3 |le= ss ZFS: i/o error - all block copies unavailable can't lookup pjdesk% ./zfstest.old /boot/kernel.old/iwn6000fw.ko.symbols /dev/ad0p3 |le= ss ZFS: i/o error - all block copies unavailable can't lookup pjdesk% ./zfstest /boot/kernel.old/iwn6000fw.ko.symbols /dev/ad0p3 |less = =20 ^?ELF^B^A^A ^@^@^@^@^@^@^@^@^A^@>^@^A^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@= ^@^@p =2E.. pjdesk% ./zfstest /boot/kernel.old/iwn6000fw.ko.symbols /dev/ad0p3 | cmp -= /boot/kernel.old/iwn6000fw.ko.symbols pjdesk% ./zfstest /boot/kernel.old/iwn6000fw.ko.symbols /dev/ad0p3 | cmp -= /boot/kernel.old/iwn6000fw.ko.symbols pjdesk% ./zfstest.old /boot/kernel.old/iwn6000fw.ko.symbols /dev/ad0p3 | c= mp - /boot/kernel.old/iwn6000fw.ko.symbols=20 stdin /boot/kernel.old/iwn6000fw.ko.symbols differ: char 1, line 1 pjdesk%=20 This was tested on an 8-STABLE system at r225392. I've built new bootblocks but not tested them yet - I will do that tomorrow. --=20 Peter Jeremy --9jxsPFA5p3P2qPhR Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) iEYEARECAAYFAk5oV5gACgkQ/opHv/APuIcQdACfUxBjX4w30gt+5BaxZRWHVMpe KmoAnAz8ki7P18u8cIGN4a4Gkbc2uC1H =Ov79 -----END PGP SIGNATURE----- --9jxsPFA5p3P2qPhR-- From owner-freebsd-fs@FreeBSD.ORG Thu Sep 8 09:27:17 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7AC26106564A for ; Thu, 8 Sep 2011 09:27:17 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from proxypop01.sare.net (proxypop01.sare.net [194.30.0.28]) by mx1.freebsd.org (Postfix) with ESMTP id 3D57B8FC18 for ; Thu, 8 Sep 2011 09:27:16 +0000 (UTC) Received: from [172.16.1.55] (izaro.sarenet.es [192.148.167.11]) by proxypop01.sare.net (Postfix) with ESMTPSA id 4F87263E950 for ; Thu, 8 Sep 2011 11:27:20 +0200 (CEST) From: Borja Marcos Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Date: Thu, 8 Sep 2011 11:27:14 +0200 Message-Id: To: freebsd-fs@freebsd.org Mime-Version: 1.0 (Apple Message framework v1084) X-Mailer: Apple Mail (2.1084) Subject: ZFS monitoring X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Sep 2011 09:27:17 -0000 Hello, Some time ago I wrote a FreeBSD data collector for Orca = (www.orcaware.com/orca). Seems it wasn't much of a success, but I use it = all the time :) I would like to add some ZFS aware monitoring. Any suggestions for key = statistics that would be useful to see in graphic form? Something like = cache hits/misses, etc. Borja. From owner-freebsd-fs@FreeBSD.ORG Thu Sep 8 09:50:16 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4EACA1065674 for ; Thu, 8 Sep 2011 09:50:16 +0000 (UTC) (envelope-from drb@karlov.mff.cuni.cz) Received: from mail.karlov.mff.cuni.cz (mail.karlov.mff.cuni.cz [195.113.27.114]) by mx1.freebsd.org (Postfix) with ESMTP id 0A3938FC0A for ; Thu, 8 Sep 2011 09:50:15 +0000 (UTC) Received: from nat-r.karlov.mff.cuni.cz ([195.113.27.73] helo=[10.32.82.27]) by mail.karlov.mff.cuni.cz with esmtpsa (TLSv1:CAMELLIA256-SHA:256) (Exim 4.68 (FreeBSD)) (envelope-from ) id 1R1azE-000K3t-Pi; Thu, 08 Sep 2011 11:33:32 +0200 Message-ID: <4E688BE4.8040602@karlov.mff.cuni.cz> Date: Thu, 08 Sep 2011 11:33:24 +0200 From: =?windows-1252?Q?Tom=E1=9A_Drbohlav?= User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20110624 Thunderbird/5.0 MIME-Version: 1.0 To: Borja Marcos , freebsd-fs@freebsd.org References: In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Cc: Subject: Re: ZFS monitoring X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Sep 2011 09:50:16 -0000 Hi, Dne 8.9.2011 11:27, Borja Marcos napsal(a): > > Hello, > > Some time ago I wrote a FreeBSD data collector for Orca (www.orcaware.com/orca). Seems it wasn't much of a success, but I use it all the time :) > > I would like to add some ZFS aware monitoring. Any suggestions for key statistics that would be useful to see in graphic form? Something like cache hits/misses, etc. we've got great experience with http://cuddletech.com/arc_summary/ (btw adapted to Nagios), actually, we use some version with added l2 cache stats. Drb > > > > > Borja. > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Thu Sep 8 10:33:45 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A9326106566B for ; Thu, 8 Sep 2011 10:33:45 +0000 (UTC) (envelope-from luchesar.iliev@gmail.com) Received: from mail-fx0-f54.google.com (mail-fx0-f54.google.com [209.85.161.54]) by mx1.freebsd.org (Postfix) with ESMTP id 2F86E8FC0C for ; Thu, 8 Sep 2011 10:33:44 +0000 (UTC) Received: by fxe4 with SMTP id 4so1886303fxe.13 for ; Thu, 08 Sep 2011 03:33:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=message-id:disposition-notification-to:date:from:user-agent :mime-version:to:cc:subject:references:in-reply-to :x-enigmail-version:openpgp:content-type:content-transfer-encoding; bh=l8RZ5ZcLRGJHfNXvycE9IZb2KZw5eyZ8zTPnQr0FH/0=; b=W2kTvgPGN28l0gAD3I13XDcP6zUUVb3J7TrJMCj8AfNJSYoUygMkk1eBBdvJQQj1Zv GwCnY7cOGuXMVBmG25RstxGFq8/VVwK1Yy07kabI/2V6wAu9SMargKHkWbcOy2vfThW7 DGhhPRp/fl2UNv16tPu6xYfUwYwF2jHb3daSc= Received: by 10.223.6.26 with SMTP id 26mr888838fax.40.1315476241789; Thu, 08 Sep 2011 03:04:01 -0700 (PDT) Received: from [79.124.93.41] ([79.124.93.41]) by mx.google.com with ESMTPS id m8sm1138992fae.17.2011.09.08.03.04.00 (version=SSLv3 cipher=OTHER); Thu, 08 Sep 2011 03:04:00 -0700 (PDT) Message-ID: <4E68930E.8000404@gmail.com> Date: Thu, 08 Sep 2011 13:03:58 +0300 From: "Luchesar V. ILIEV" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:6.0.2) Gecko/20110908 Thunderbird/6.0.2 MIME-Version: 1.0 To: =?UTF-8?B?VG9tw6HFoSBEcmJvaGxhdg==?= , Borja Marcos References: <4E688BE4.8040602@karlov.mff.cuni.cz> In-Reply-To: <4E688BE4.8040602@karlov.mff.cuni.cz> X-Enigmail-Version: undefined OpenPGP: id=9A1FEEFF; url=https://cert.acad.bg/pgp-keys/ Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: freebsd-fs@freebsd.org Subject: Re: ZFS monitoring X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Sep 2011 10:33:45 -0000 On 09/08/2011 12:33, Tomáš Drbohlav wrote: > > Dne 8.9.2011 11:27, Borja Marcos napsal(a): >> >> Some time ago I wrote a FreeBSD data collector for Orca >> (www.orcaware.com/orca). Seems it wasn't much of a success, but I use >> it all the time :) >> >> I would like to add some ZFS aware monitoring. Any suggestions for >> key statistics that would be useful to see in graphic form? Something >> like cache hits/misses, etc. > > we've got great experience with http://cuddletech.com/arc_summary/ (btw > adapted to Nagios), actually, we use some version with added l2 cache > stats. sysutils/zfs-stats is also quite handy in this regard http://www.freshports.org/sysutils/zfs-stats Cheers, Luchesar From owner-freebsd-fs@FreeBSD.ORG Thu Sep 8 13:54:08 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CCC3F1065674 for ; Thu, 8 Sep 2011 13:54:08 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from proxypop02.sare.net (proxypop02.sare.net [194.30.18.43]) by mx1.freebsd.org (Postfix) with ESMTP id 8F03B8FC16 for ; Thu, 8 Sep 2011 13:54:08 +0000 (UTC) Received: from [172.16.1.55] (izaro.sarenet.es [192.148.167.11]) by proxypop02.sare.net (Postfix) with ESMTPSA id 1D4D3109F110; Thu, 8 Sep 2011 15:53:27 +0200 (CEST) Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Borja Marcos In-Reply-To: <4E68930E.8000404@gmail.com> Date: Thu, 8 Sep 2011 15:54:06 +0200 Content-Transfer-Encoding: 7bit Message-Id: <09B87CF3-3703-4331-AD3F-4C450F12D1DD@sarenet.es> References: <4E688BE4.8040602@karlov.mff.cuni.cz> <4E68930E.8000404@gmail.com> To: Luchesar V. ILIEV X-Mailer: Apple Mail (2.1084) Cc: freebsd-fs@freebsd.org Subject: Re: ZFS monitoring X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Sep 2011 13:54:08 -0000 On Sep 8, 2011, at 12:03 PM, Luchesar V. ILIEV wrote: >> we've got great experience with http://cuddletech.com/arc_summary/ (btw >> adapted to Nagios), actually, we use some version with added l2 cache >> stats. > > sysutils/zfs-stats is also quite handy in this regard I mean something like this, including several ZFS statistics. (Slow, it's on a free ti.micro EC2 instance) http://devilator.frobula.com/ Borja. From owner-freebsd-fs@FreeBSD.ORG Thu Sep 8 14:48:19 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4926F106566B for ; Thu, 8 Sep 2011 14:48:19 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) by mx1.freebsd.org (Postfix) with ESMTP id 035768FC0C for ; Thu, 8 Sep 2011 14:48:18 +0000 (UTC) Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id 0BFF328424; Thu, 8 Sep 2011 16:29:43 +0200 (CEST) Received: from [192.168.1.2] (ip-86-49-61-235.net.upcbroadband.cz [86.49.61.235]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 47F8428423; Thu, 8 Sep 2011 16:29:42 +0200 (CEST) Message-ID: <4E68D155.5060507@quip.cz> Date: Thu, 08 Sep 2011 16:29:41 +0200 From: Miroslav Lachman <000.fbsd@quip.cz> User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.9.1.19) Gecko/20110420 Lightning/1.0b1 SeaMonkey/2.0.14 MIME-Version: 1.0 To: Borja Marcos References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: ZFS monitoring X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Sep 2011 14:48:19 -0000 Borja Marcos wrote: > > Hello, > > Some time ago I wrote a FreeBSD data collector for Orca (www.orcaware.com/orca). Seems it wasn't much of a success, but I use it all the time :) > > I would like to add some ZFS aware monitoring. Any suggestions for key statistics that would be useful to see in graphic form? Something like cache hits/misses, etc. You can do graphs for many things from sysctl kstat.zfs.misc and vfs.zfs I am personally graphing for example: kstat.zfs.misc.arcstats.size kstat.zfs.misc.arcstats.c vfs.zfs.arc_meta_limit vfs.zfs.arc_meta_used But there are other useful values. It depends on usage of the server. Miroslav Lachman From owner-freebsd-fs@FreeBSD.ORG Thu Sep 8 17:19:58 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D51EA1065798 for ; Thu, 8 Sep 2011 17:19:58 +0000 (UTC) (envelope-from luke@digital-crocus.com) Received: from mail.digital-crocus.com (node2.digital-crocus.com [91.209.244.128]) by mx1.freebsd.org (Postfix) with ESMTP id 822AB8FC1A for ; Thu, 8 Sep 2011 17:19:58 +0000 (UTC) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=dkselector; d=hybrid-logic.co.uk; h=Received:Received:Subject:From:Reply-To:To:Cc:In-Reply-To:References:Content-Type:Organization:Date:Message-ID:Mime-Version:X-Mailer:Content-Transfer-Encoding:X-Spam-Score:X-Digital-Crocus-Maillimit:X-Authenticated-Sender:X-Complaints:X-Admin:X-Abuse; b=Z2zw5Gms2/8YGQHrG4qqBvRdlpYHshkhEF8BTvgeOujpn/7M2kovWlJDWMRkZSC2Q88w1DyAnbEwvNEI1qzFAB0iXrBzQ3ZRSt8DXSfPHeLgEhuWI6ub8bYuKtkxAKE9; Received: from luke by mail.digital-crocus.com with local (Exim 4.69 (FreeBSD)) (envelope-from ) id 1R1iFm-000Go8-2s for freebsd-fs@freebsd.org; Thu, 08 Sep 2011 18:19:06 +0100 Received: from vlan111.pact.srf.ac.uk ([193.37.225.200] helo=[10.0.111.133]) by mail.digital-crocus.com with esmtpa (Exim 4.69 (FreeBSD)) (envelope-from ) id 1R1iFl-000Gnr-QM; Thu, 08 Sep 2011 18:19:06 +0100 From: Luke Marsden To: Martin Matuska In-Reply-To: <1314655349.7898.53.camel@pow> References: <1314646728.7898.44.camel@pow> <4E5BFC6F.5080507@FreeBSD.org> <1314655349.7898.53.camel@pow> Content-Type: text/plain; charset="UTF-8" Organization: Hybrid Web Cluster Date: Thu, 08 Sep 2011 18:19:48 +0100 Message-ID: <1315502388.11352.15.camel@pow> Mime-Version: 1.0 X-Mailer: Evolution 2.32.2 Content-Transfer-Encoding: 7bit X-Spam-Score: -1.0 X-Digital-Crocus-Maillimit: done X-Authenticated-Sender: luke X-Complaints: abuse@digital-crocus.com X-Admin: admin@digital-crocus.com X-Abuse: abuse@digital-crocus.com (Please include full headers in abuse reports) Cc: freebsd-fs@freebsd.org, tech@hybrid-logic.co.uk, Pawel Jakub Dawidek Subject: Re: ZFS hang in production on 8.2-RELEASE X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: luke@hybrid-logic.co.uk List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Sep 2011 17:19:58 -0000 On Mon, 2011-08-29 at 23:02 +0100, Luke Marsden wrote: > On Mon, 2011-08-29 at 22:54 +0200, Martin Matuska wrote: > > No, I think this is more likely fixed by pjd's bugfix in r224791 (MFC'ed > > to stable/8 as r225100). > > > > The corresponding patch is: > > http://people.freebsd.org/~pjd/patches/zfsdev_state_lock.patch > > > > Great, thanks! Will this patch apply to ZFS v15? We can't upgrade to > v28 yet. > We just got another hang in production, same procstat -kk on zfs umount -f as before: 52739 100186 zfs - mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 zfsvfs_teardown+0x269 zfs_umount+0x1c4 dounmount+0x32a unmount+0x38b syscallenter+0x1e5 syscall+0x4b Xfast_syscall+0xe2 Please advise whether the zfsdev_state_lock patch with apply to 8.2-RELEASE and whether it is wise to attempt to apply it? Thank you! -- Best Regards, Luke Marsden CTO, Hybrid Logic Ltd. Web: http://www.hybrid-cluster.com/ Hybrid Web Cluster - cloud web hosting Mobile: +447791750420 (UK) / +1-415-449-1165 (US) From owner-freebsd-fs@FreeBSD.ORG Thu Sep 8 19:54:33 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A0479106566B; Thu, 8 Sep 2011 19:54:33 +0000 (UTC) (envelope-from pawel@dawidek.net) Received: from mail.dawidek.net (60.wheelsystems.com [83.12.187.60]) by mx1.freebsd.org (Postfix) with ESMTP id 54CFC8FC14; Thu, 8 Sep 2011 19:54:33 +0000 (UTC) Received: from localhost (89-73-195-149.dynamic.chello.pl [89.73.195.149]) by mail.dawidek.net (Postfix) with ESMTPSA id E977EBDE; Thu, 8 Sep 2011 21:54:29 +0200 (CEST) Date: Thu, 8 Sep 2011 21:54:06 +0200 From: Pawel Jakub Dawidek To: luke@hybrid-logic.co.uk Message-ID: <20110908195405.GB1667@garage.freebsd.pl> References: <1314646728.7898.44.camel@pow> <4E5BFC6F.5080507@FreeBSD.org> <1314655349.7898.53.camel@pow> <1315502388.11352.15.camel@pow> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="nFreZHaLTZJo0R7j" Content-Disposition: inline In-Reply-To: <1315502388.11352.15.camel@pow> X-OS: FreeBSD 9.0-CURRENT amd64 User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@freebsd.org, tech@hybrid-logic.co.uk, Martin Matuska Subject: Re: ZFS hang in production on 8.2-RELEASE X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Sep 2011 19:54:33 -0000 --nFreZHaLTZJo0R7j Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Sep 08, 2011 at 06:19:48PM +0100, Luke Marsden wrote: > On Mon, 2011-08-29 at 23:02 +0100, Luke Marsden wrote: > > On Mon, 2011-08-29 at 22:54 +0200, Martin Matuska wrote: > > > No, I think this is more likely fixed by pjd's bugfix in r224791 (MFC= 'ed > > > to stable/8 as r225100). > > >=20 > > > The corresponding patch is: > > > http://people.freebsd.org/~pjd/patches/zfsdev_state_lock.patch > > >=20 > >=20 > > Great, thanks! Will this patch apply to ZFS v15? We can't upgrade to > > v28 yet. > >=20 >=20 > We just got another hang in production, same procstat -kk on zfs umount > -f as before: >=20 > 52739 100186 zfs - mi_switch+0x176 > sleepq_wait+0x42 _sleep+0x317 zfsvfs_teardown+0x269 zfs_umount+0x1c4 > dounmount+0x32a unmount+0x38b syscallenter+0x1e5 syscall+0x4b > Xfast_syscall+0xe2=20 >=20 > Please advise whether the zfsdev_state_lock patch with apply to > 8.2-RELEASE and whether it is wise to attempt to apply it? The zfsdev_state_lock patch won't help you, it fixes totally unrelated problem. Could you send backtraces of all processes? --=20 Pawel Jakub Dawidek http://www.wheelsystems.com FreeBSD committer http://www.FreeBSD.org Am I Evil? Yes, I Am! http://yomoli.com --nFreZHaLTZJo0R7j Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAk5pHV0ACgkQForvXbEpPzQySgCgl8uMWMXIv2L6VcSKj+IXXDLa u3oAn06fubxn8+j9nO5IqlR+cDkuJq0L =AFna -----END PGP SIGNATURE----- --nFreZHaLTZJo0R7j-- From owner-freebsd-fs@FreeBSD.ORG Thu Sep 8 19:59:11 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 507B71065674 for ; Thu, 8 Sep 2011 19:59:11 +0000 (UTC) (envelope-from luke@digital-crocus.com) Received: from mail.digital-crocus.com (node2.digital-crocus.com [91.209.244.128]) by mx1.freebsd.org (Postfix) with ESMTP id 007508FC16 for ; Thu, 8 Sep 2011 19:59:10 +0000 (UTC) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=dkselector; d=hybrid-logic.co.uk; h=Received:Received:Subject:From:To:Cc:In-Reply-To:References:Content-Type:Organization:Date:Message-ID:Mime-Version:X-Mailer:Content-Transfer-Encoding:X-Spam-Score:X-Digital-Crocus-Maillimit:X-Authenticated-Sender:X-Complaints:X-Admin:X-Abuse; b=T/U0dFTFinH4q410RUgqmcS0/wBgu+L0AsH183i2JNfQbreM3r1+r5UWWYAor22MG/O/xQAoeqNhmkTDd87sEFo6t5uICLFL2BSw8dTYwrsvq6p8bVe4gXEnnjfkUHfE; Received: from luke by mail.digital-crocus.com with local (Exim 4.69 (FreeBSD)) (envelope-from ) id 1R1kjq-000BFV-MC for freebsd-fs@freebsd.org; Thu, 08 Sep 2011 20:58:18 +0100 Received: from vlan111.pact.srf.ac.uk ([193.37.225.200] helo=[10.0.111.133]) by mail.digital-crocus.com with esmtpa (Exim 4.69 (FreeBSD)) (envelope-from ) id 1R1kjq-000BFD-AO; Thu, 08 Sep 2011 20:58:18 +0100 From: Luke Marsden To: Pawel Jakub Dawidek In-Reply-To: <20110908195405.GB1667@garage.freebsd.pl> References: <1314646728.7898.44.camel@pow> <4E5BFC6F.5080507@FreeBSD.org> <1314655349.7898.53.camel@pow> <1315502388.11352.15.camel@pow> <20110908195405.GB1667@garage.freebsd.pl> Content-Type: text/plain; charset="UTF-8" Organization: Hybrid Logic Date: Thu, 08 Sep 2011 20:59:03 +0100 Message-ID: <1315511943.11352.30.camel@pow> Mime-Version: 1.0 X-Mailer: Evolution 2.32.2 Content-Transfer-Encoding: 7bit X-Spam-Score: -1.0 X-Digital-Crocus-Maillimit: done X-Authenticated-Sender: luke X-Complaints: abuse@digital-crocus.com X-Admin: admin@digital-crocus.com X-Abuse: abuse@digital-crocus.com (Please include full headers in abuse reports) Cc: freebsd-fs@freebsd.org, tech@hybrid-logic.co.uk, Martin Matuska Subject: Re: ZFS hang in production on 8.2-RELEASE X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Sep 2011 19:59:11 -0000 On Thu, 2011-09-08 at 21:54 +0200, Pawel Jakub Dawidek wrote: > On Thu, Sep 08, 2011 at 06:19:48PM +0100, Luke Marsden wrote: > > On Mon, 2011-08-29 at 23:02 +0100, Luke Marsden wrote: > > > On Mon, 2011-08-29 at 22:54 +0200, Martin Matuska wrote: > > > > No, I think this is more likely fixed by pjd's bugfix in r224791 (MFC'ed > > > > to stable/8 as r225100). > > > > > > > > The corresponding patch is: > > > > http://people.freebsd.org/~pjd/patches/zfsdev_state_lock.patch > > > > > > > > > > Great, thanks! Will this patch apply to ZFS v15? We can't upgrade to > > > v28 yet. > > > > > > > We just got another hang in production, same procstat -kk on zfs umount > > -f as before: > > > > 52739 100186 zfs - mi_switch+0x176 > > sleepq_wait+0x42 _sleep+0x317 zfsvfs_teardown+0x269 zfs_umount+0x1c4 > > dounmount+0x32a unmount+0x38b syscallenter+0x1e5 syscall+0x4b > > Xfast_syscall+0xe2 > > > > Please advise whether the zfsdev_state_lock patch with apply to > > 8.2-RELEASE and whether it is wise to attempt to apply it? > > The zfsdev_state_lock patch won't help you, it fixes totally unrelated > problem. Could you send backtraces of all processes? > Thanks for getting back to me. As before: FreeBSD XXX 8.2-RELEASE FreeBSD 8.2-RELEASE #0 r219081M: Wed Mar 2 08:29:52 CET 2011 root@www4:/usr/obj/usr/src/sys/GENERIC amd64 There are 9 'zfs rename' processes and 1 'zfs umount -f' processes hung. Here is the procstat for the 'zfs umount -f': 13451 104337 zfs - mi_switch+0x176 sleepq_wait+0x42 _sleep+0x317 zfsvfs_teardown+0x269 zfs_umount+0x1c4 dounmount+0x32a unmount+0x38b syscallenter+0x1e5 syscall+0x4b Xfast_syscall+0xe2 And the 'zfs rename's all look the same: 20361 101049 zfs - mi_switch+0x176 sleepq_wait+0x42 __lockmgr_args+0x743 vop_stdlock+0x39 VOP_LOCK1_APV +0x46 _vn_lock+0x47 lookup+0x6e1 namei+0x53a kern_rmdirat+0xa4 syscallenter+0x1e5 syscall+0x4b Xfast_syscall+0xe2 An 'ls' on a directory which contains most of the system's ZFS mount-points (/hcfs) also hangs: 30073 101466 gnuls - mi_switch+0x176 sleepq_wait+0x42 __lockmgr_args+0x743 vop_stdlock+0x39 VOP_LOCK1_APV +0x46 _vn_lock+0x47 zfs_root+0x85 lookup+0x9b8 namei+0x53a vn_open_cred +0x3ac kern_openat+0x181 syscallenter+0x1e5 syscall+0x4b Xfast_syscall +0xe2 If I truss the 'ls' it hangs on the stat syscall: stat("/hcfs",{ mode=drwxr-xr-x ,inode=3,size=2012,blksize=16384 }) = 0 (0x0) There is also a 'find -s / ! ( -fstype zfs ) -prune -or -path /tmp -prune -or -path /usr/tmp -prune -or -path /var/tmp -prune -or -path /var/db/portsnap -prune -or -print' running which is also hung: 2650 101674 find - mi_switch+0x176 sleepq_wait+0x42 __lockmgr_args+0x743 vop_stdlock+0x39 VOP_LOCK1_APV +0x46 _vn_lock+0x47 zfs_root+0x85 lookup+0x9b8 namei+0x53a vn_open_cred +0x3ac kern_openat+0x181 syscallenter+0x1e5 syscall+0x4b Xfast_syscall +0xe2 However I/O to the presently mounted filesystems continues to work (even on parts of filesystems which are unlikely to be cached), and 'zfs list' showing all the filesystems (3,500 filesystems with ~100 snapshots per filesystem) also works. Any activity on the structure of the ZFS hierarchy *under the hcfs filesystem* crashes, such as a 'zfs create hpool/hcfs/test': 70868 101874 zfs - mi_switch+0x176 sleepq_wait+0x42 __lockmgr_args+0x743 vop_stdlock+0x39 VOP_LOCK1_APV +0x46 _vn_lock+0x47 lookup+0x6e1 namei+0x53a kern_mkdirat+0xce syscallenter+0x1e5 syscall+0x4b Xfast_syscall+0xe2 BUT "zfs create hpool/system/opt/hello" (a ZFS filesystem in the same pool, but not rooted on hpool/hcfs) does not hang, and succeeds normally. procstat -kk on the zfskern process gives: PID TID COMM TDNAME KSTACK 5 100045 zfskern arc_reclaim_thre mi_switch+0x176 sleepq_timedwait+0x42 _cv_timedwait+0x134 arc_reclaim_thread+0x2a9 fork_exit+0x118 fork_trampoline+0xe 5 100046 zfskern l2arc_feed_threa mi_switch+0x176 sleepq_timedwait+0x42 _cv_timedwait+0x134 l2arc_feed_thread+0x1ce fork_exit+0x118 fork_trampoline+0xe 5 100098 zfskern txg_thread_enter mi_switch+0x176 sleepq_wait+0x42 _cv_wait+0x129 txg_thread_wait+0x79 txg_quiesce_thread +0xb5 fork_exit+0x118 fork_trampoline+0xe 5 100099 zfskern txg_thread_enter mi_switch+0x176 sleepq_timedwait+0x42 _cv_timedwait+0x134 txg_thread_wait+0x3c txg_sync_thread+0x365 fork_exit+0x118 fork_trampoline+0xe -- Best Regards, Luke Marsden CTO, Hybrid Logic Ltd. Web: http://www.hybrid-cluster.com/ Hybrid Web Cluster - cloud web hosting Mobile: +447791750420 (UK) / +1-415-449-1165 (US) From owner-freebsd-fs@FreeBSD.ORG Thu Sep 8 21:45:26 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B5DCC106566C for ; Thu, 8 Sep 2011 21:45:26 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 74A3B8FC17 for ; Thu, 8 Sep 2011 21:45:26 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AkoHAGE2aU6DaFvO/2dsb2JhbABCFoQ/lF2PSIFwVjUCDRkCiGumKpIHgSyEMIERBJMziX2HNQ X-IronPort-AV: E=Sophos;i="4.68,352,1312171200"; d="scan'208";a="137008772" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-jnhn-pri.mail.uoguelph.ca with ESMTP; 08 Sep 2011 17:45:25 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id A101EB3F67; Thu, 8 Sep 2011 17:45:25 -0400 (EDT) Date: Thu, 8 Sep 2011 17:45:25 -0400 (EDT) From: Rick Macklem To: nfsv4@ietf.org Message-ID: <2113256952.1021105.1315518325647.JavaMail.root@erie.cs.uoguelph.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.202] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - FF3.0 (Win)/6.0.10_GA_2692) Cc: freebsd-fs@freebsd.org Subject: NFSv4.1 FreeBSD client for testing X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Sep 2011 21:45:26 -0000 First off, I hope no one minds the crosspost, but I thought some of the FreeBSD crowd might be interested... I've put a patch against FreeBSD9.0 (currently Beta2, but I don't think the NFS client code will change before the final release, so hopefully the patch will continue to apply) up in the hopes that server vendors (and anyone else, of course) will use it for testing. The main difference between what I tested during the June Bakeathon and this patch is support for the back channel. It does not yet have any support for pNFS. I'll need to figure out a way to set up a DS for testing before I can work on that. If you are interested, the patch is at: http://people.freebsd.org/~rmacklem/nfsv4.1-client Hopefully the Readme covers the basic setup. Good luck with it, if you try it, rick From owner-freebsd-fs@FreeBSD.ORG Fri Sep 9 02:08:04 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0AB251065695 for ; Fri, 9 Sep 2011 02:08:04 +0000 (UTC) (envelope-from Trond.Myklebust@netapp.com) Received: from mx2.netapp.com (mx2.netapp.com [216.240.18.37]) by mx1.freebsd.org (Postfix) with ESMTP id DD4088FC08 for ; Fri, 9 Sep 2011 02:08:03 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.68,353,1312182000"; d="scan'208";a="580137250" Received: from smtp1.corp.netapp.com ([10.57.156.124]) by mx2-out.netapp.com with ESMTP; 08 Sep 2011 18:39:29 -0700 Received: from svlrsexc1-prd.hq.netapp.com (svlrsexc1-prd.hq.netapp.com [10.57.115.30]) by smtp1.corp.netapp.com (8.13.1/8.13.1/NTAP-1.6) with ESMTP id p891dT6p015274; Thu, 8 Sep 2011 18:39:29 -0700 (PDT) Received: from SACMVEXC2-PRD.hq.netapp.com ([10.99.115.18]) by svlrsexc1-prd.hq.netapp.com with Microsoft SMTPSVC(6.0.3790.4675); Thu, 8 Sep 2011 18:39:29 -0700 Received: from 10.55.72.41 ([10.55.72.41]) by SACMVEXC2-PRD.hq.netapp.com ([10.99.115.16]) with Microsoft Exchange Server HTTP-DAV ; Fri, 9 Sep 2011 01:39:28 +0000 Received: from lade.trondhjem.org by SACMVEXC2-PRD.hq.netapp.com; 08 Sep 2011 21:39:25 -0400 From: Trond Myklebust To: Rick Macklem Date: Thu, 08 Sep 2011 21:39:25 -0400 In-Reply-To: <2113256952.1021105.1315518325647.JavaMail.root@erie.cs.uoguelph.ca> References: <2113256952.1021105.1315518325647.JavaMail.root@erie.cs.uoguelph.ca> Organization: NetApp Inc Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Mailer: Evolution 3.0.2 (3.0.2-3.fc15) Message-ID: <1315532365.3119.2.camel@lade.trondhjem.org> Mime-Version: 1.0 X-OriginalArrivalTime: 09 Sep 2011 01:39:29.0071 (UTC) FILETIME=[50BBF3F0:01CC6E91] Cc: freebsd-fs@freebsd.org, nfsv4@ietf.org Subject: Re: [nfsv4] NFSv4.1 FreeBSD client for testing X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Sep 2011 02:08:04 -0000 On Thu, 2011-09-08 at 17:45 -0400, Rick Macklem wrote:=20 > First off, I hope no one minds the crosspost, but I thought > some of the FreeBSD crowd might be interested... >=20 > I've put a patch against FreeBSD9.0 (currently Beta2, but I don't > think the NFS client code will change before the final release, so > hopefully the patch will continue to apply) up in the hopes that > server vendors (and anyone else, of course) will use it for testing. > The main difference between what I tested during the June Bakeathon > and this patch is support for the back channel. It does not yet have > any support for pNFS. I'll need to figure out a way to set up a DS for > testing before I can work on that. >=20 > If you are interested, the patch is at: > http://people.freebsd.org/~rmacklem/nfsv4.1-client >=20 > Hopefully the Readme covers the basic setup. >=20 > Good luck with it, if you try it, rick Cool... Are you going to be attending the Bakeathon next month? If not, can you supply a VM or something like that so that we can try to do some FreeBSD client testing against the various server implementations out there? Cheers Trond --=20 Trond Myklebust Linux NFS client maintainer NetApp Trond.Myklebust@netapp.com www.netapp.com From owner-freebsd-fs@FreeBSD.ORG Fri Sep 9 03:43:56 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0ED131065670 for ; Fri, 9 Sep 2011 03:43:56 +0000 (UTC) (envelope-from Tom.Haynes@netapp.com) Received: from mx2.netapp.com (mx2.netapp.com [216.240.18.37]) by mx1.freebsd.org (Postfix) with ESMTP id DF58B8FC13 for ; Fri, 9 Sep 2011 03:43:55 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.68,354,1312182000"; d="scan'208";a="580150956" Received: from smtp1.corp.netapp.com ([10.57.156.124]) by mx2-out.netapp.com with ESMTP; 08 Sep 2011 20:15:22 -0700 Received: from sacrsexc1-prd.hq.netapp.com (sacrsexc1-prd.hq.netapp.com [10.99.115.27]) by smtp1.corp.netapp.com (8.13.1/8.13.1/NTAP-1.6) with ESMTP id p893FKkV021528; Thu, 8 Sep 2011 20:15:21 -0700 (PDT) Received: from rtprsexc1-prd.hq.netapp.com ([10.100.161.114]) by sacrsexc1-prd.hq.netapp.com with Microsoft SMTPSVC(6.0.3790.4675); Thu, 8 Sep 2011 20:15:20 -0700 Received: from RTPMVEXC1-PRD.hq.netapp.com ([10.100.161.112]) by rtprsexc1-prd.hq.netapp.com with Microsoft SMTPSVC(6.0.3790.4675); Thu, 8 Sep 2011 23:15:18 -0400 Received: from vpn2ntap-39942.hq.netapp.com ([10.55.76.14]) by RTPMVEXC1-PRD.hq.netapp.com with Microsoft SMTPSVC(6.0.3790.3959); Thu, 8 Sep 2011 23:15:18 -0400 Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Thomas Haynes In-Reply-To: <1315532365.3119.2.camel@lade.trondhjem.org> Date: Thu, 8 Sep 2011 22:15:16 -0500 Content-Transfer-Encoding: 7bit Message-Id: References: <2113256952.1021105.1315518325647.JavaMail.root@erie.cs.uoguelph.ca> <1315532365.3119.2.camel@lade.trondhjem.org> To: Trond Myklebust X-Mailer: Apple Mail (2.1084) X-OriginalArrivalTime: 09 Sep 2011 03:15:18.0196 (UTC) FILETIME=[B37ACF40:01CC6E9E] Cc: freebsd-fs@freebsd.org, nfsv4@ietf.org Subject: Re: [nfsv4] NFSv4.1 FreeBSD client for testing X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Sep 2011 03:43:56 -0000 On Sep 8, 2011, at 8:39 PM, Trond Myklebust wrote: > On Thu, 2011-09-08 at 17:45 -0400, Rick Macklem wrote: >> First off, I hope no one minds the crosspost, but I thought >> some of the FreeBSD crowd might be interested... >> >> I've put a patch against FreeBSD9.0 (currently Beta2, but I don't >> think the NFS client code will change before the final release, so >> hopefully the patch will continue to apply) up in the hopes that >> server vendors (and anyone else, of course) will use it for testing. >> The main difference between what I tested during the June Bakeathon >> and this patch is support for the back channel. It does not yet have >> any support for pNFS. I'll need to figure out a way to set up a DS for >> testing before I can work on that. >> >> If you are interested, the patch is at: >> http://people.freebsd.org/~rmacklem/nfsv4.1-client >> >> Hopefully the Readme covers the basic setup. >> >> Good luck with it, if you try it, rick > > Cool... Are you going to be attending the Bakeathon next month? If not, > can you supply a VM or something like that so that we can try to do some > FreeBSD client testing against the various server implementations out > there? > I can run one, I got pretty good at it for a while for some testing. :-> > Cheers > Trond > -- > Trond Myklebust > Linux NFS client maintainer > > NetApp > Trond.Myklebust@netapp.com > www.netapp.com > > _______________________________________________ > nfsv4 mailing list > nfsv4@ietf.org > https://www.ietf.org/mailman/listinfo/nfsv4 From owner-freebsd-fs@FreeBSD.ORG Fri Sep 9 04:12:38 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 55E991065670 for ; Fri, 9 Sep 2011 04:12:38 +0000 (UTC) (envelope-from jhellenthal@gmail.com) Received: from mail-yx0-f182.google.com (mail-yx0-f182.google.com [209.85.213.182]) by mx1.freebsd.org (Postfix) with ESMTP id 0A20A8FC0A for ; Fri, 9 Sep 2011 04:12:37 +0000 (UTC) Received: by yxk36 with SMTP id 36so1517346yxk.13 for ; Thu, 08 Sep 2011 21:12:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to; bh=JaYLNYGBtGk/HH7Ejjd/mJN8M7IIUYMG3datb90ZRyM=; b=MePMKOE/d/8yNo5xIWjlcZI7qv5SGEDw60QXuarWjhH1EKO1/enNwSh8tsVscPPSMZ p97B2QWkqrLdOjBCEbabN0HyOs4+hYxnibiqatCb10Rzer3RXw4JlrWkul/sVcqGLjjd qhL/MVLVOUy+XstcbENPTzjlIVMFsH1yVsUnw= Received: by 10.236.37.230 with SMTP id y66mr9546806yha.10.1315540143601; Thu, 08 Sep 2011 20:49:03 -0700 (PDT) Received: from DataIX.net (adsl-99-190-81-85.dsl.klmzmi.sbcglobal.net [99.190.81.85]) by mx.google.com with ESMTPS id s77sm5486924yhe.22.2011.09.08.20.49.01 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 08 Sep 2011 20:49:02 -0700 (PDT) Sender: "J. Hellenthal" Received: from DataIX.net (localhost [127.0.0.1]) by DataIX.net (8.14.5/8.14.5) with ESMTP id p893mwxN070524 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 8 Sep 2011 23:48:58 -0400 (EDT) (envelope-from jhell@DataIX.net) Received: (from jhell@localhost) by DataIX.net (8.14.5/8.14.5/Submit) id p893mt3W070523; Thu, 8 Sep 2011 23:48:55 -0400 (EDT) (envelope-from jhell@DataIX.net) Date: Thu, 8 Sep 2011 23:48:55 -0400 From: Jason Hellenthal To: =?iso-8859-1?B?VG9t4T8=?= Drbohlav Message-ID: <20110909034855.GA70001@DataIX.net> References: <4E688BE4.8040602@karlov.mff.cuni.cz> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="2fHTh5uZTiUOsy+g" Content-Disposition: inline In-Reply-To: <4E688BE4.8040602@karlov.mff.cuni.cz> Cc: freebsd-fs@freebsd.org Subject: Re: ZFS monitoring X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Sep 2011 04:12:38 -0000 --2fHTh5uZTiUOsy+g Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Sep 08, 2011 at 11:33:24AM +0200, Tom=E1? Drbohlav wrote: > Hi, >=20 > Dne 8.9.2011 11:27, Borja Marcos napsal(a): > > > > Hello, > > > > Some time ago I wrote a FreeBSD data collector for Orca (www.orcaware.c= om/orca). Seems it wasn't much of a success, but I use it all the time :) > > > > I would like to add some ZFS aware monitoring. Any suggestions for key= statistics that would be useful to see in graphic form? Something like cac= he hits/misses, etc. >=20 > we've got great experience with http://cuddletech.com/arc_summary/ (btw= =20 > adapted to Nagios), actually, we use some version with added l2 cache sta= ts. >=20 > Drb >=20 You have considerable missed this then: https://jhell.googlecode.com/files/arc_summary.pl and the current repository version: http://bit.ly/arc_summary Which at this time are essentially the same. sysutils/zfs-stats is essentially a clone of an older version with slight modifications. --2fHTh5uZTiUOsy+g Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) Comment: http://bit.ly/0x89D8547E iQEcBAEBAgAGBQJOaYymAAoJEJBXh4mJ2FR+KtoH/2aV25GPyhF/tnTAgMj97ZH1 hwwLyyFVmDM3JFYqyiGVmJIJLGX6ckDn0VhG9aXRMTeOsNTS5L/BkULBRnxH/a05 9+mBfhWbghZfjwAs79n78cIpHgUPqWvL/tSpiexhyD9NTZf/KUr94cdShOlJG7u3 YwmqLpjQ99/WGD+VrY75tWPLz/CrjONaCo2LqYS9yxwj2+FiF4CUrtl15FmCY/x+ je9PJXQPaOa6B1D7277JoDwFMu//xh3N90xCkXhjUgLMjmSh43WK2zRoeap+Bf0z zy145onGsRxUBe6Rff8JjeTnliZ8BAcZrveCNeLsbU73Zn3BFQlugXcDjAegE1A= =2r+K -----END PGP SIGNATURE----- --2fHTh5uZTiUOsy+g-- From owner-freebsd-fs@FreeBSD.ORG Fri Sep 9 04:36:04 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D7E01106564A for ; Fri, 9 Sep 2011 04:36:04 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta07.westchester.pa.mail.comcast.net (qmta07.westchester.pa.mail.comcast.net [76.96.62.64]) by mx1.freebsd.org (Postfix) with ESMTP id 850908FC13 for ; Fri, 9 Sep 2011 04:36:04 +0000 (UTC) Received: from omta20.westchester.pa.mail.comcast.net ([76.96.62.71]) by qmta07.westchester.pa.mail.comcast.net with comcast id WUbv1h0021YDfWL57Uc4Jr; Fri, 09 Sep 2011 04:36:04 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta20.westchester.pa.mail.comcast.net with comcast id WUc21h01l1t3BNj3gUc38Z; Fri, 09 Sep 2011 04:36:04 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id A837F102C1B; Thu, 8 Sep 2011 21:36:01 -0700 (PDT) Date: Thu, 8 Sep 2011 21:36:01 -0700 From: Jeremy Chadwick To: Jason Hellenthal Message-ID: <20110909043601.GA49649@icarus.home.lan> References: <4E688BE4.8040602@karlov.mff.cuni.cz> <20110909034855.GA70001@DataIX.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20110909034855.GA70001@DataIX.net> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@freebsd.org Subject: Re: ZFS monitoring X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Sep 2011 04:36:04 -0000 On Thu, Sep 08, 2011 at 11:48:55PM -0400, Jason Hellenthal wrote: > > > On Thu, Sep 08, 2011 at 11:33:24AM +0200, Tom?? Drbohlav wrote: > > Hi, > > > > Dne 8.9.2011 11:27, Borja Marcos napsal(a): > > > > > > Hello, > > > > > > Some time ago I wrote a FreeBSD data collector for Orca (www.orcaware.com/orca). Seems it wasn't much of a success, but I use it all the time :) > > > > > > I would like to add some ZFS aware monitoring. Any suggestions for key statistics that would be useful to see in graphic form? Something like cache hits/misses, etc. > > > > we've got great experience with http://cuddletech.com/arc_summary/ (btw > > adapted to Nagios), actually, we use some version with added l2 cache stats. > > > > Drb > > > > You have considerable missed this then: > > > https://jhell.googlecode.com/files/arc_summary.pl > > and the current repository version: > > http://bit.ly/arc_summary > > Which at this time are essentially the same. > > sysutils/zfs-stats is essentially a clone of an older version with > slight modifications. I could have a field day fixing this perl script. Would you like me to contact you off-list discussing all the problems with it? There are many (at least 6 just from skimming). I would worry if this script was used by something like Nagios natively. Major eek. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Fri Sep 9 05:14:02 2011 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 80F23106566B; Fri, 9 Sep 2011 05:14:02 +0000 (UTC) (envelope-from peterjeremy@acm.org) Received: from mail27.syd.optusnet.com.au (mail27.syd.optusnet.com.au [211.29.133.168]) by mx1.freebsd.org (Postfix) with ESMTP id 0AE288FC12; Fri, 9 Sep 2011 05:14:01 +0000 (UTC) Received: from server.vk2pj.dyndns.org (c220-239-116-103.belrs4.nsw.optusnet.com.au [220.239.116.103]) by mail27.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id p895DxkM030018 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 9 Sep 2011 15:14:00 +1000 X-Bogosity: Ham, spamicity=0.000000 Received: from server.vk2pj.dyndns.org (localhost.vk2pj.dyndns.org [127.0.0.1]) by server.vk2pj.dyndns.org (8.14.5/8.14.4) with ESMTP id p895DvSC060002; Fri, 9 Sep 2011 15:13:57 +1000 (EST) (envelope-from peter@server.vk2pj.dyndns.org) Received: (from peter@localhost) by server.vk2pj.dyndns.org (8.14.5/8.14.4/Submit) id p895DvXX060001; Fri, 9 Sep 2011 15:13:57 +1000 (EST) (envelope-from peter) Date: Fri, 9 Sep 2011 15:13:57 +1000 From: Peter Jeremy To: Andriy Gapon Message-ID: <20110909051357.GA58043@server.vk2pj.dyndns.org> References: <20110901223646.14b8aae8@o2.pl> <4E60DBBD.1040703@FreeBSD.org> <4E679D3D.1000007@FreeBSD.org> <20110908055016.GB28874@server.vk2pj.dyndns.org> <4E685B8A.604@FreeBSD.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="PmA2V3Z32TCmWXqI" Content-Disposition: inline In-Reply-To: <4E685B8A.604@FreeBSD.org> X-PGP-Key: http://members.optusnet.com.au/peterjeremy/pubkey.asc User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@FreeBSD.org Subject: Re: ZFS: i/o error - all block copies unavailable after upgrading to r225312 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Sep 2011 05:14:02 -0000 --PmA2V3Z32TCmWXqI Content-Type: multipart/mixed; boundary="ZGiS0Q5IWpPtfppv" Content-Disposition: inline --ZGiS0Q5IWpPtfppv Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2011-Sep-08 09:07:06 +0300, Andriy Gapon wrote: >on 08/09/2011 08:50 Peter Jeremy said the following: >> Since the problem isn't consistent (it can appear on some files but not= =20 >> others), I modified zfstest to take a pathname, rather than have the=20 >> pathname hard-coded. > >I would appreciate your sharing back this enhancement. >Thanks for testing! See attached. The first argument is a pathname to open and the remaining arguments specify the pool as before. Note that I commented out the spa_all_status() so I could easily cmp(1) the output with the file read via the filesystem. On 2011-Sep-08 15:50:16 +1000, Peter Jeremy = wrote: >This was tested on an 8-STABLE system at r225392. I've built new >bootblocks but not tested them yet - I will do that tomorrow. Unfortunately, the resultant gptzfsboot causes a "BTX Halted" problem (see http://i.imgur.com/CuTut.jpg - apologies for the quality). Based on cs:eip and ss:esp, it looks like it just goes off into the weeds. I've tried building it on two different systems (including a full buildworld in case just running "make clean && make" in /usr/src/sys/boot was insufficient) and get the same binary. An unpatched gptzfsboot from r225392 works. I rather exceeded the time I should have invested in the problem and so have gone ahead with a zpool rebuild to get rid of the problematic gang-blocks. --=20 Peter Jeremy --ZGiS0Q5IWpPtfppv Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="zfstest.diff" Content-Transfer-Encoding: quoted-printable Index: zfstest.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D RCS file: /usr/ncvs/src/sys/boot/zfs/zfstest.c,v retrieving revision 1.1 diff -u -r1.1 zfstest.c --- zfstest.c 30 Jul 2010 13:54:15 -0000 1.1 +++ zfstest.c 7 Sep 2011 22:01:54 -0000 @@ -95,26 +95,35 @@ argc =3D 4; argv =3D av; } - for (i =3D 1; i < argc; i++) { - fd[i] =3D open(argv[i], O_RDONLY); + for (i =3D 0; i < argc-2; i++) { + fd[i] =3D open(argv[i+2], O_RDONLY); if (fd[i] < 0) continue; if (vdev_probe(vdev_read, &fd[i], NULL) !=3D 0) close(fd[i]); } - spa_all_status(); + //spa_all_status(); =20 spa =3D STAILQ_FIRST(&zfs_pools); - if (!spa || zfs_mount_pool(spa)) + if (spa =3D=3D NULL) { + printf("no pools\n"); exit(1); + } =20 - if (zfs_lookup(spa, "zfs.c", &dn)) + if (zfs_mount_pool(spa)) { + printf("can't mount pool\n"); exit(1); + } =20 + if (zfs_lookup(spa, argv[1], &dn)) { + printf("can't lookup\n"); + exit(1); + } off =3D 0; do { n =3D zfs_read(spa, &dn, buf, 512, off); write(1, buf, n); off +=3D n; } while (n =3D=3D 512); + return(0); } --ZGiS0Q5IWpPtfppv-- --PmA2V3Z32TCmWXqI Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) iEYEARECAAYFAk5poJUACgkQ/opHv/APuIdXWwCgkPIrzeeaSfOMTL/EN2WNM8C5 3ZUAmwZc/eWDt32VKdipUbopwNbf3eB8 =xYSy -----END PGP SIGNATURE----- --PmA2V3Z32TCmWXqI-- From owner-freebsd-fs@FreeBSD.ORG Fri Sep 9 06:59:56 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 038B5106564A for ; Fri, 9 Sep 2011 06:59:56 +0000 (UTC) (envelope-from drb@karlov.mff.cuni.cz) Received: from mail.karlov.mff.cuni.cz (mail.karlov.mff.cuni.cz [195.113.27.114]) by mx1.freebsd.org (Postfix) with ESMTP id B32568FC12 for ; Fri, 9 Sep 2011 06:59:55 +0000 (UTC) Received: from nat-r.karlov.mff.cuni.cz ([195.113.27.73] helo=[10.32.82.27]) by mail.karlov.mff.cuni.cz with esmtpsa (TLSv1:CAMELLIA256-SHA:256) (Exim 4.68 (FreeBSD)) (envelope-from ) id 1R1v43-000Bse-S1; Fri, 09 Sep 2011 08:59:51 +0200 Message-ID: <4E69B95A.9090906@karlov.mff.cuni.cz> Date: Fri, 09 Sep 2011 08:59:38 +0200 From: =?windows-1252?Q?Tom=E1=9A_Drbohlav?= User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20110624 Thunderbird/5.0 MIME-Version: 1.0 To: Jeremy Chadwick References: <4E688BE4.8040602@karlov.mff.cuni.cz> <20110909034855.GA70001@DataIX.net> <20110909043601.GA49649@icarus.home.lan> In-Reply-To: <20110909043601.GA49649@icarus.home.lan> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, Jason Hellenthal Subject: Re: ZFS monitoring X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Sep 2011 06:59:56 -0000 Hi! Dne 9.9.2011 6:36, Jeremy Chadwick napsal(a): > On Thu, Sep 08, 2011 at 11:48:55PM -0400, Jason Hellenthal wrote: >> >> >> On Thu, Sep 08, 2011 at 11:33:24AM +0200, Tom?? Drbohlav wrote: >>> Hi, >>> >>> Dne 8.9.2011 11:27, Borja Marcos napsal(a): >>>> >>>> Hello, >>>> >>>> Some time ago I wrote a FreeBSD data collector for Orca (www.orcaware.com/orca). Seems it wasn't much of a success, but I use it all the time :) >>>> >>>> I would like to add some ZFS aware monitoring. Any suggestions for key statistics that would be useful to see in graphic form? Something like cache hits/misses, etc. >>> >>> we've got great experience with http://cuddletech.com/arc_summary/ (btw >>> adapted to Nagios), actually, we use some version with added l2 cache stats. >>> >>> Drb >>> >> >> You have considerable missed this then: >> >> >> https://jhell.googlecode.com/files/arc_summary.pl >> >> and the current repository version: >> >> http://bit.ly/arc_summary >> >> Which at this time are essentially the same. >> >> sysutils/zfs-stats is essentially a clone of an older version with >> slight modifications. > > I could have a field day fixing this perl script. Would you like me to > contact you off-list discussing all the problems with it? There are > many (at least 6 just from skimming). It would be at least inspiring if you can shout out your opinion, do not mind in list or by e-mail, as you like. > I would worry if this script was used by something like Nagios natively. I am not sure what you mean 'natively', about 60% of checks made by Nagios for us is written by me, but I do consider it native. Just to be more precise, we push it to RRD through Nagios to see hwo is our ZFS doing. T:D > Major eek. > From owner-freebsd-fs@FreeBSD.ORG Fri Sep 9 09:07:48 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8E2DE106564A for ; Fri, 9 Sep 2011 09:07:48 +0000 (UTC) (envelope-from patpro@patpro.net) Received: from rack.patpro.net (rack.patpro.net [193.30.227.216]) by mx1.freebsd.org (Postfix) with ESMTP id 0E7608FC13 for ; Fri, 9 Sep 2011 09:07:48 +0000 (UTC) Received: from rack.patpro.net (localhost [127.0.0.1]) by rack.patpro.net (Postfix) with ESMTP id 0F9381CC020 for ; Fri, 9 Sep 2011 10:50:02 +0200 (CEST) X-Virus-Scanned: amavisd-new at patpro.net Received: from amavis-at-patpro.net ([127.0.0.1]) by rack.patpro.net (rack.patpro.net [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 5l8dBOljPNbo for ; Fri, 9 Sep 2011 10:50:00 +0200 (CEST) Received: from [127.0.0.1] (localhost [127.0.0.1]) by rack.patpro.net (Postfix) with ESMTP for ; Fri, 9 Sep 2011 10:50:00 +0200 (CEST) From: Patrick Proniewski Content-Type: multipart/signed; boundary=Apple-Mail-7-96066748; protocol="application/pkcs7-signature"; micalg=sha1 Date: Fri, 9 Sep 2011 10:49:58 +0200 Message-Id: To: FreeBSD Filesystems Mime-Version: 1.0 (Apple Message framework v1084) X-Mailer: Apple Mail (2.1084) X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: ZFS: zpool history size X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Sep 2011 09:07:48 -0000 --Apple-Mail-7-96066748 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Hello, I have a production server hosting about 250 web sites. I'm using ZFS, = so that I can create a dedicated FS whenever a user wants to create a = web site. I've started 6 mouths ago to dump `zpool history` into a file, daily, = and to commit the output to a remote subversion server, so that if I = need I could recreate my storage pool with all its properties. I've noticed that in fact the `zpool history` is limited. I've many = automated daily/weekly/monthly snapshots creations and that "noise" is = filling the history log and overwrites my precious history. Today, my history log is only 13 valuable lines long: # zpool history | grep -v @ History for 'tank': 2011-02-22.14:17:10 zpool create tank da1 2011-06-30.16:46:38 zfs create tank/Sites/m 2011-06-30.16:46:38 zfs set refquota=3D500M tank/Sites/m 2011-07-11.10:46:16 zfs create tank/Sites/sites/egales-uk 2011-07-11.10:46:17 zfs set refquota=3D500M tank/Sites/sites/egales-uk 2011-07-11.10:46:41 zfs create tank/Sites/sites/egales-es 2011-07-11.10:46:42 zfs set refquota=3D500M tank/Sites/sites/egales-es 2011-07-11.10:46:47 zfs create tank/Sites/sites/egales-ro 2011-07-11.10:46:47 zfs set refquota=3D500M tank/Sites/sites/egales-ro 2011-07-11.10:46:51 zfs create tank/Sites/sites/egales-se 2011-07-11.10:46:52 zfs set refquota=3D500M tank/Sites/sites/egales-se 2011-07-12.10:56:14 zfs set = mountpoint=3D/Sites/a_supprimer/perso-truchaud-2011071210 = tank/user/truchaud 2011-07-12.11:04:37 zfs destroy -r tank/user/truchaud Every interesting events between 2011-02-22 and 2011-06-30 is gone, = that's more than 700 lines of ZFS command. Thanks to svn, I lose = nothing. My questions are:=20 - what is the zpool history size limit?=20 - and is it possible to increase its value? regards, patpro --Apple-Mail-7-96066748-- From owner-freebsd-fs@FreeBSD.ORG Fri Sep 9 17:09:22 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AA08A106566B for ; Fri, 9 Sep 2011 17:09:22 +0000 (UTC) (envelope-from steve.f.thompson@gmail.com) Received: from mail-gy0-f182.google.com (mail-gy0-f182.google.com [209.85.160.182]) by mx1.freebsd.org (Postfix) with ESMTP id 1CC618FC27 for ; Fri, 9 Sep 2011 17:09:21 +0000 (UTC) Received: by gyf2 with SMTP id 2so2128468gyf.13 for ; Fri, 09 Sep 2011 10:09:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; bh=vHXLckB4gVuiVTxAuO6VBHP/bqyjk5ptnZ3XjdH6eK0=; b=HXZD19mwVt+k+P9yMSvm1fUp5KE84cSRsRVb6sUcG4yG+MoNd1nJcUMCbHFwmpIHWm i7vQV5VadMg8Zx9s7NFBUxXesAIPhBMMR/haIJphYUWmzNdfbYbRbDbcTN11lfrolgux 80BuFbKbwQOAsNbkyhfoQwIAZYl1cv8V4Wv5I= MIME-Version: 1.0 Received: by 10.236.144.193 with SMTP id n41mr13165604yhj.78.1315586287742; Fri, 09 Sep 2011 09:38:07 -0700 (PDT) Received: by 10.147.114.6 with HTTP; Fri, 9 Sep 2011 09:38:07 -0700 (PDT) Date: Fri, 9 Sep 2011 11:38:07 -0500 Message-ID: From: Steve Thompson To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: ZFS version 30 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Sep 2011 17:09:22 -0000 Does anyone know if there're any plans to get this into FreeBSD anytime soon? I really want transparent encryption! Thanks for any info. -Steve From owner-freebsd-fs@FreeBSD.ORG Fri Sep 9 17:35:06 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F2256106566C for ; Fri, 9 Sep 2011 17:35:06 +0000 (UTC) (envelope-from bfriesen@simple.dallas.tx.us) Received: from blade.simplesystems.org (blade.simplesystems.org [65.66.246.74]) by mx1.freebsd.org (Postfix) with ESMTP id 806BB8FC12 for ; Fri, 9 Sep 2011 17:35:06 +0000 (UTC) Received: from freddy.simplesystems.org (freddy.simplesystems.org [65.66.246.65]) by blade.simplesystems.org (8.14.4+Sun/8.14.4) with ESMTP id p89HZ4Wm013746; Fri, 9 Sep 2011 12:35:04 -0500 (CDT) Date: Fri, 9 Sep 2011 12:35:04 -0500 (CDT) From: Bob Friesenhahn X-X-Sender: bfriesen@freddy.simplesystems.org To: Steve Thompson In-Reply-To: Message-ID: References: User-Agent: Alpine 2.01 (GSO 1266 2009-07-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (blade.simplesystems.org [65.66.246.90]); Fri, 09 Sep 2011 12:35:04 -0500 (CDT) Cc: freebsd-fs@freebsd.org Subject: Re: ZFS version 30 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Sep 2011 17:35:07 -0000 On Fri, 9 Sep 2011, Steve Thompson wrote: > Does anyone know if there're any plans to get this into FreeBSD anytime > soon? I really want transparent encryption! Where is the source code for it? Has Oracle released code? Bob -- Bob Friesenhahn bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ From owner-freebsd-fs@FreeBSD.ORG Fri Sep 9 17:42:09 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E70D7106564A for ; Fri, 9 Sep 2011 17:42:09 +0000 (UTC) (envelope-from steve.f.thompson@gmail.com) Received: from mail-gx0-f179.google.com (mail-gx0-f179.google.com [209.85.161.179]) by mx1.freebsd.org (Postfix) with ESMTP id A39BD8FC12 for ; Fri, 9 Sep 2011 17:42:09 +0000 (UTC) Received: by gxk1 with SMTP id 1so1565619gxk.10 for ; Fri, 09 Sep 2011 10:42:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=RxhysKxE8CBFLhiPOCgFyFzf7W73vvs4MJ1WlVCiuSQ=; b=eNFVWYnaPzsYWryp5MVT0yh98zYILIlXXEiPuTjLKshAodXoo2md6ytw2SpTk+HR5o w68TPwbEzPlrZ+9VA67LVHzClzXyzj2QR2TsfuXM88sFb1NwXWeR2k+hJJR6WI8ChyCH 7LI79Xo35Uk7lIdiBvmPGco27XKstanYLhEPw= MIME-Version: 1.0 Received: by 10.236.173.131 with SMTP id v3mr13679203yhl.112.1315590129090; Fri, 09 Sep 2011 10:42:09 -0700 (PDT) Received: by 10.147.114.6 with HTTP; Fri, 9 Sep 2011 10:42:09 -0700 (PDT) In-Reply-To: References: Date: Fri, 9 Sep 2011 12:42:09 -0500 Message-ID: From: Steve Thompson To: Bob Friesenhahn Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-fs@freebsd.org Subject: Re: ZFS version 30 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Sep 2011 17:42:10 -0000 I don't personally know if Oracle has any intention of doing so, but I heard a rumor "some opensource groups are not sitting on its laurels while Oracle marches on". Anyone else heard anything about this? From owner-freebsd-fs@FreeBSD.ORG Fri Sep 9 17:56:32 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 82D74106564A for ; Fri, 9 Sep 2011 17:56:32 +0000 (UTC) (envelope-from bfriesen@simple.dallas.tx.us) Received: from blade.simplesystems.org (blade.simplesystems.org [65.66.246.74]) by mx1.freebsd.org (Postfix) with ESMTP id 460248FC18 for ; Fri, 9 Sep 2011 17:56:31 +0000 (UTC) Received: from freddy.simplesystems.org (freddy.simplesystems.org [65.66.246.65]) by blade.simplesystems.org (8.14.4+Sun/8.14.4) with ESMTP id p89HuVcA013825; Fri, 9 Sep 2011 12:56:31 -0500 (CDT) Date: Fri, 9 Sep 2011 12:56:31 -0500 (CDT) From: Bob Friesenhahn X-X-Sender: bfriesen@freddy.simplesystems.org To: Steve Thompson In-Reply-To: Message-ID: References: User-Agent: Alpine 2.01 (GSO 1266 2009-07-14) MIME-Version: 1.0 Content-Type: MULTIPART/MIXED; BOUNDARY="-559023410-52182039-1315590991=:4180" X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (blade.simplesystems.org [65.66.246.90]); Fri, 09 Sep 2011 12:56:31 -0500 (CDT) Cc: freebsd-fs@freebsd.org Subject: Re: ZFS version 30 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Sep 2011 17:56:32 -0000 This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. ---559023410-52182039-1315590991=:4180 Content-Type: TEXT/PLAIN; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8BIT On Fri, 9 Sep 2011, Steve Thompson wrote: > I don't personally know if Oracle has any intention of doing so, but I heard a rumor "some opensource groups are not > sitting on its laurels while Oracle marches on".  Anyone else heard anything about this? The zfs encryption code missed the window. Unless Oracle decides to make another code dump, any future activity is likely associated with the Illumos project (https://www.illumos.org/). I do know that key FreeBSD zfs maintainers collaborate with counterparts on the Illumos project to get zfs bugs fixed. I do see this forum entry: https://www.illumos.org/boards/1/topics/256 The main reason to expect/hope that Oracle will make more code dumps is that they have continued to maintain the many services associated with the OpenSolaris project. Bob -- Bob Friesenhahn bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ ---559023410-52182039-1315590991=:4180-- From owner-freebsd-fs@FreeBSD.ORG Fri Sep 9 22:20:36 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2254A106564A for ; Fri, 9 Sep 2011 22:20:36 +0000 (UTC) (envelope-from peterjeremy@acm.org) Received: from mail27.syd.optusnet.com.au (mail27.syd.optusnet.com.au [211.29.133.168]) by mx1.freebsd.org (Postfix) with ESMTP id A9F298FC0A for ; Fri, 9 Sep 2011 22:20:35 +0000 (UTC) Received: from server.vk2pj.dyndns.org (c220-239-116-103.belrs4.nsw.optusnet.com.au [220.239.116.103]) by mail27.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id p89MKW46006785 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 10 Sep 2011 08:20:33 +1000 X-Bogosity: Ham, spamicity=0.000000 Received: from server.vk2pj.dyndns.org (localhost.vk2pj.dyndns.org [127.0.0.1]) by server.vk2pj.dyndns.org (8.14.5/8.14.4) with ESMTP id p89MKVSo073208; Sat, 10 Sep 2011 08:20:31 +1000 (EST) (envelope-from peter@server.vk2pj.dyndns.org) Received: (from peter@localhost) by server.vk2pj.dyndns.org (8.14.5/8.14.4/Submit) id p89MKVDU073207; Sat, 10 Sep 2011 08:20:31 +1000 (EST) (envelope-from peter) Date: Sat, 10 Sep 2011 08:20:30 +1000 From: Peter Jeremy To: Steve Thompson Message-ID: <20110909222030.GB40207@server.vk2pj.dyndns.org> References: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="aM3YZ0Iwxop3KEKx" Content-Disposition: inline In-Reply-To: X-PGP-Key: http://members.optusnet.com.au/peterjeremy/pubkey.asc User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@freebsd.org Subject: Re: ZFS version 30 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Sep 2011 22:20:36 -0000 --aM3YZ0Iwxop3KEKx Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2011-Sep-09 11:38:07 -0500, Steve Thompson = wrote: >Does anyone know if there're any plans to get this into FreeBSD anytime >soon? I really want transparent encryption! ZFS v28 is the latest publicly available code and it's unlikely that features in later versions of ZFS on Solaris will be available in any free OS's unless Oracle releases the code. As others have said, the various FOSS groups using ZFS are co-ordinating their efforts but there is very little public information on their plans or progress. If you need encryption now, I suggest you investigate gdbe(4) and geli(4). You can run ZFS on top of either. --=20 Peter Jeremy --aM3YZ0Iwxop3KEKx Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) iEYEARECAAYFAk5qkS4ACgkQ/opHv/APuIeFRQCfXb4wUnYxkc9KYIZu4zmz7PC8 NLsAoKhandoktRAPxhsOkF58QSHFLr7A =XapT -----END PGP SIGNATURE----- --aM3YZ0Iwxop3KEKx-- From owner-freebsd-fs@FreeBSD.ORG Sat Sep 10 07:32:30 2011 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 41808106566B; Sat, 10 Sep 2011 07:32:30 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 2B9618FC15; Sat, 10 Sep 2011 07:32:28 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id KAA11926; Sat, 10 Sep 2011 10:32:25 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1R2I37-000Evl-EQ; Sat, 10 Sep 2011 10:32:25 +0300 Message-ID: <4E6B1285.70508@FreeBSD.org> Date: Sat, 10 Sep 2011 10:32:21 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:6.0.2) Gecko/20110907 Thunderbird/6.0.2 MIME-Version: 1.0 To: freebsd-current@FreeBSD.org, freebsd-fs@FreeBSD.org, Pawel Jakub Dawidek , Dimitry Andric References: <20110901223646.14b8aae8@o2.pl> <4E60DBBD.1040703@FreeBSD.org> <4E679D3D.1000007@FreeBSD.org> In-Reply-To: <4E679D3D.1000007@FreeBSD.org> X-Enigmail-Version: undefined Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Sebastian Chmielewski Subject: Re: ZFS: i/o error - all block copies unavailable after upgrading to r225312 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 Sep 2011 07:32:30 -0000 on 07/09/2011 19:35 Andriy Gapon said the following: > Thanks to a lot of excellent testing, debugging and analysis from Sebastian (which > went behind the scenes) we now have this patch: > http://people.freebsd.org/~avg/zfs-boot-gang.diff > > The patch introduces the following changes: > - checksum is now verified for gang header blocks > - checksum is now verified for reconstituted data of whole gang blocks > (previously it is verified only for individual gang member leaf blocks) > - reconstituted data of a whole gang block is now decompressed if the gang block > is compressed > > The last change is _the_ change. I am now investigating what looks like a miscompilation of the code by *gcc* after applying the patch. It seems that -mrtd option is to blame. I have found an older discussion about the -mrtd option causing trouble with clang: http://lists.freebsd.org/pipermail/freebsd-current/2011-August/026263.html There was a patch that made clang happy without disabling the flag, so I wonder if I made some subtle mistake in my patch. Or maybe it's better to disable mrtd altogether for the zfs boot blocks, just to stay on the safe side. Some technical details in the form of a diff with some superimposed comments: --- /home/avg/tmp/vdev_read_phys-mrtd.s 2011-09-10 01:50:54.500620864 +0300 +++ /home/avg/tmp/vdev_read_phys-no-mrtd.s 2011-09-10 01:49:59.157701373 +0300 @@ -29,16 +29,17 @@ ... <- in the code before this %edi gets assigned a pointer to a function movl 60(%ecx), %eax movl %eax, 24(%esp) movl %ecx, 20(%esp) + movl %edi, %ecx <- non-mrtd code saves the pointer popl %ebx popl %esi popl %edi <- %edi gets over-written with an unrelated value popl %ebp - jmp *%edi <- mrtd code calls some garbage code + jmp *%ecx <- non-mrtd code calls the correct code .L601: movl $5, %eax popl %ebx popl %esi popl %edi popl %ebp - ret $24 + ret The problem is in the patched vdev_read_phys function in zfsimpl.c. Unpatched version of the function doesn't seem to be affected. Any help/ideas will be greatly appreciated! -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Sat Sep 10 08:07:52 2011 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 305BF106564A; Sat, 10 Sep 2011 08:07:52 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id A77A28FC14; Sat, 10 Sep 2011 08:07:50 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id LAA12148; Sat, 10 Sep 2011 11:07:49 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1R2IbN-000EwX-1C; Sat, 10 Sep 2011 11:07:49 +0300 Message-ID: <4E6B1AD4.6080206@FreeBSD.org> Date: Sat, 10 Sep 2011 11:07:48 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:6.0.2) Gecko/20110907 Thunderbird/6.0.2 MIME-Version: 1.0 To: freebsd-current@FreeBSD.org, freebsd-fs@FreeBSD.org, Dimitry Andric References: <20110901223646.14b8aae8@o2.pl> <4E60DBBD.1040703@FreeBSD.org> <4E679D3D.1000007@FreeBSD.org> <4E6B1285.70508@FreeBSD.org> In-Reply-To: <4E6B1285.70508@FreeBSD.org> X-Enigmail-Version: undefined Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Pawel Jakub Dawidek Subject: Re: ZFS: i/o error - all block copies unavailable after upgrading to r225312 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 Sep 2011 08:07:52 -0000 on 10/09/2011 10:32 Andriy Gapon said the following: > I am now investigating what looks like a miscompilation of the code by *gcc* > after applying the patch. It seems that -mrtd option is to blame. > I have found an older discussion about the -mrtd option causing trouble with clang: > http://lists.freebsd.org/pipermail/freebsd-current/2011-August/026263.html > There was a patch that made clang happy without disabling the flag, so I wonder > if I made some subtle mistake in my patch. Or maybe it's better to disable mrtd > altogether for the zfs boot blocks, just to stay on the safe side. Actually, removing either -mrtd _or_ -fno-unit-at-a-time produces the correct code. Puzzled. > Some technical details in the form of a diff with some superimposed comments: > --- /home/avg/tmp/vdev_read_phys-mrtd.s 2011-09-10 01:50:54.500620864 +0300 > +++ /home/avg/tmp/vdev_read_phys-no-mrtd.s 2011-09-10 01:49:59.157701373 +0300 > @@ -29,16 +29,17 @@ > ... <- in the code before this %edi gets assigned a pointer to a function > movl 60(%ecx), %eax > movl %eax, 24(%esp) > movl %ecx, 20(%esp) > + movl %edi, %ecx <- non-mrtd code saves the pointer > popl %ebx > popl %esi > popl %edi <- %edi gets over-written with an unrelated value > popl %ebp > - jmp *%edi <- mrtd code calls some garbage code > + jmp *%ecx <- non-mrtd code calls the correct code > .L601: > movl $5, %eax > popl %ebx > popl %esi > popl %edi > popl %ebp > - ret $24 > + ret > > The problem is in the patched vdev_read_phys function in zfsimpl.c. > Unpatched version of the function doesn't seem to be affected. > > Any help/ideas will be greatly appreciated! > -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Sat Sep 10 09:23:09 2011 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C591D106566B for ; Sat, 10 Sep 2011 09:23:09 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 1CBDC8FC0A for ; Sat, 10 Sep 2011 09:23:08 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id MAA12646; Sat, 10 Sep 2011 12:23:04 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1R2JmC-000Ey8-Fp; Sat, 10 Sep 2011 12:23:04 +0300 Message-ID: <4E6B2C78.80102@FreeBSD.org> Date: Sat, 10 Sep 2011 12:23:04 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:6.0.2) Gecko/20110907 Thunderbird/6.0.2 MIME-Version: 1.0 To: Peter Jeremy References: <20110901223646.14b8aae8@o2.pl> <4E60DBBD.1040703@FreeBSD.org> <4E679D3D.1000007@FreeBSD.org> <20110908055016.GB28874@server.vk2pj.dyndns.org> <4E685B8A.604@FreeBSD.org> <20110909051357.GA58043@server.vk2pj.dyndns.org> In-Reply-To: <20110909051357.GA58043@server.vk2pj.dyndns.org> X-Enigmail-Version: undefined Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org Subject: Re: ZFS: i/o error - all block copies unavailable after upgrading to r225312 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 Sep 2011 09:23:09 -0000 on 09/09/2011 08:13 Peter Jeremy said the following: > On 2011-Sep-08 09:07:06 +0300, Andriy Gapon wrote: >> on 08/09/2011 08:50 Peter Jeremy said the following: >>> Since the problem isn't consistent (it can appear on some files but not >>> others), I modified zfstest to take a pathname, rather than have the >>> pathname hard-coded. >> >> I would appreciate your sharing back this enhancement. Thanks for >> testing! > > See attached. The first argument is a pathname to open and the remaining > arguments specify the pool as before. Note that I commented out the > spa_all_status() so I could easily cmp(1) the output with the file read via > the filesystem. Thank you. > On 2011-Sep-08 15:50:16 +1000, Peter Jeremy > wrote: >> This was tested on an 8-STABLE system at r225392. I've built new >> bootblocks but not tested them yet - I will do that tomorrow. > > Unfortunately, the resultant gptzfsboot causes a "BTX Halted" problem (see > http://i.imgur.com/CuTut.jpg - apologies for the quality). Based on cs:eip > and ss:esp, it looks like it just goes off into the weeds. Not an expert on BTX dumps, so can't tell anything from it. But see below. > I've tried building it on two different systems (including a full > buildworld in case just running "make clean && make" in /usr/src/sys/boot > was insufficient) and get the same binary. An unpatched gptzfsboot from > r225392 works. I rather exceeded the time I should have invested in the > problem and so have gone ahead with a zpool rebuild to get rid of the > problematic gang-blocks. I understand your situation. Good thing is that I've been able to reproduce what looks like the same issue. I should have used myself the same C flags for zfstest as those that are used for "real" (gpt)zfsboot compilation and I should have advised those same flags to the testers like you and Sebastian. Currently the problem looks like a miscompilation by GCC that was triggered by my change. Sigh. -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Sat Sep 10 09:39:26 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 48F24106566C for ; Sat, 10 Sep 2011 09:39:26 +0000 (UTC) (envelope-from olivier@gid0.org) Received: from mail-qw0-f45.google.com (mail-qw0-f45.google.com [209.85.216.45]) by mx1.freebsd.org (Postfix) with ESMTP id 0B2AE8FC08 for ; Sat, 10 Sep 2011 09:39:25 +0000 (UTC) Received: by qwj8 with SMTP id 8so2596097qwj.18 for ; Sat, 10 Sep 2011 02:39:25 -0700 (PDT) MIME-Version: 1.0 Received: by 10.229.20.14 with SMTP id d14mr2191728qcb.61.1315647564879; Sat, 10 Sep 2011 02:39:24 -0700 (PDT) Received: by 10.229.223.196 with HTTP; Sat, 10 Sep 2011 02:39:24 -0700 (PDT) In-Reply-To: <20110909222030.GB40207@server.vk2pj.dyndns.org> References: <20110909222030.GB40207@server.vk2pj.dyndns.org> Date: Sat, 10 Sep 2011 11:39:24 +0200 Message-ID: From: Olivier Smedts To: Peter Jeremy Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: Steve Thompson , freebsd-fs@freebsd.org Subject: Re: ZFS version 30 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 Sep 2011 09:39:26 -0000 2011/9/10 Peter Jeremy : > On 2011-Sep-09 11:38:07 -0500, Steve Thompson wrote: >>Does anyone know if there're any plans to get this into FreeBSD anytime >>soon? =A0I really want transparent encryption! > > ZFS v28 is the latest publicly available code and it's unlikely that > features in later versions of ZFS on Solaris will be available in any > free OS's unless Oracle releases the code. =A0As others have said, the > various FOSS groups using ZFS are co-ordinating their efforts but > there is very little public information on their plans or progress. After having acquired Sun, Oracle said (I think I read that) they won't change the license of the zfs sources and the developments will continue like they were, but they'll only disclose sources after a Solaris launch. Solaris Express is here, but still no Solaris 11. Any news on that ? > If you need encryption now, I suggest you investigate gdbe(4) and > geli(4). =A0You can run ZFS on top of either. > > -- > Peter Jeremy > --=20 Olivier Smedts=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=A0 _ =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 ASCII ribbon campaign ( ) e-mail: olivier@gid0.org=A0 =A0 =A0 =A0 - against HTML email & vCards=A0 X www: http://www.gid0.org=A0 =A0 - against proprietary attachments / \ =A0 "Il y a seulement 10 sortes de gens dans le monde : =A0 ceux qui comprennent le binaire, =A0 et ceux qui ne le comprennent pas." From owner-freebsd-fs@FreeBSD.ORG Sat Sep 10 09:46:54 2011 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 96480106566B; Sat, 10 Sep 2011 09:46:54 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 1B0248FC08; Sat, 10 Sep 2011 09:46:52 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id MAA12808; Sat, 10 Sep 2011 12:46:51 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1R2K9D-000Eyb-IP; Sat, 10 Sep 2011 12:46:51 +0300 Message-ID: <4E6B320A.4090606@FreeBSD.org> Date: Sat, 10 Sep 2011 12:46:50 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:6.0.2) Gecko/20110907 Thunderbird/6.0.2 MIME-Version: 1.0 To: freebsd-current@FreeBSD.org, freebsd-fs@FreeBSD.org, Dimitry Andric References: <20110901223646.14b8aae8@o2.pl> <4E60DBBD.1040703@FreeBSD.org> <4E679D3D.1000007@FreeBSD.org> <4E6B1285.70508@FreeBSD.org> <4E6B1AD4.6080206@FreeBSD.org> In-Reply-To: <4E6B1AD4.6080206@FreeBSD.org> X-Enigmail-Version: undefined Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Pawel Jakub Dawidek Subject: Re: ZFS: i/o error - all block copies unavailable after upgrading to r225312 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 Sep 2011 09:46:54 -0000 on 10/09/2011 11:07 Andriy Gapon said the following: > Actually, removing either -mrtd _or_ -fno-unit-at-a-time produces the correct > code. Puzzled. The problem is reproducible with base gcc and gcc42, it is not reproducible with gcc45, gcc46 and clang. -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Sat Sep 10 11:03:15 2011 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EF56F106566C; Sat, 10 Sep 2011 11:03:15 +0000 (UTC) (envelope-from peterjeremy@acm.org) Received: from mail27.syd.optusnet.com.au (mail27.syd.optusnet.com.au [211.29.133.168]) by mx1.freebsd.org (Postfix) with ESMTP id 3F1178FC0C; Sat, 10 Sep 2011 11:03:15 +0000 (UTC) Received: from server.vk2pj.dyndns.org (c220-239-116-103.belrs4.nsw.optusnet.com.au [220.239.116.103]) by mail27.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id p8AB3CIm017426 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 10 Sep 2011 21:03:13 +1000 X-Bogosity: Ham, spamicity=0.000000 Received: from server.vk2pj.dyndns.org (localhost.vk2pj.dyndns.org [127.0.0.1]) by server.vk2pj.dyndns.org (8.14.5/8.14.4) with ESMTP id p8AB3B54007414; Sat, 10 Sep 2011 21:03:11 +1000 (EST) (envelope-from peter@server.vk2pj.dyndns.org) Received: (from peter@localhost) by server.vk2pj.dyndns.org (8.14.5/8.14.4/Submit) id p8AB3BTc007413; Sat, 10 Sep 2011 21:03:11 +1000 (EST) (envelope-from peter) Date: Sat, 10 Sep 2011 21:03:10 +1000 From: Peter Jeremy To: Andriy Gapon Message-ID: <20110910110310.GA6263@server.vk2pj.dyndns.org> References: <20110901223646.14b8aae8@o2.pl> <4E60DBBD.1040703@FreeBSD.org> <4E679D3D.1000007@FreeBSD.org> <4E6B1285.70508@FreeBSD.org> <4E6B1AD4.6080206@FreeBSD.org> <4E6B320A.4090606@FreeBSD.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="zhXaljGHf11kAtnf" Content-Disposition: inline In-Reply-To: <4E6B320A.4090606@FreeBSD.org> X-PGP-Key: http://members.optusnet.com.au/peterjeremy/pubkey.asc User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@FreeBSD.org, Pawel Jakub Dawidek , freebsd-current@FreeBSD.org, Dimitry Andric Subject: Re: ZFS: i/o error - all block copies unavailable after upgrading to r225312 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 Sep 2011 11:03:16 -0000 --zhXaljGHf11kAtnf Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2011-Sep-10 12:46:50 +0300, Andriy Gapon wrote: >on 10/09/2011 11:07 Andriy Gapon said the following: >> Actually, removing either -mrtd _or_ -fno-unit-at-a-time produces the co= rrect >> code. Puzzled. > >The problem is reproducible with base gcc and gcc42, it is not reproducible >with gcc45, gcc46 and clang. I was just checking gcc44 & gcc46. gcc44 inlines the entire function and I couldn't quickly find the offending code to see if the bug was there or not. I agree you've triggered a gcc bug but I'm not sure of the correct approach to fix it. I've tried a few trivial code transforms within vdev_read_phys() but haven't stumbled on one that avoids the problem. Since -mrtd changes the calling convention, it's a more intrusive change. I'm not sure if there's any simple way to alter CFLAGS for a single file (since we only want to alter the zfsboot.c compilation. --=20 Peter Jeremy --zhXaljGHf11kAtnf Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) iEYEARECAAYFAk5rQ+4ACgkQ/opHv/APuIdXagCeMrMRpjGHkXnmxlmUG0v7GUGG YZgAoLzV4ypU07F261aURRMIARZa+2VF =imQJ -----END PGP SIGNATURE----- --zhXaljGHf11kAtnf-- From owner-freebsd-fs@FreeBSD.ORG Sat Sep 10 16:22:20 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F0F5210656A8 for ; Sat, 10 Sep 2011 16:22:20 +0000 (UTC) (envelope-from bfriesen@simple.dallas.tx.us) Received: from blade.simplesystems.org (blade.simplesystems.org [65.66.246.74]) by mx1.freebsd.org (Postfix) with ESMTP id B7F2F8FC0C for ; Sat, 10 Sep 2011 16:22:20 +0000 (UTC) Received: from freddy.simplesystems.org (freddy.simplesystems.org [65.66.246.65]) by blade.simplesystems.org (8.14.4+Sun/8.14.4) with ESMTP id p8AGMJon019395; Sat, 10 Sep 2011 11:22:19 -0500 (CDT) Date: Sat, 10 Sep 2011 11:22:19 -0500 (CDT) From: Bob Friesenhahn X-X-Sender: bfriesen@freddy.simplesystems.org To: Olivier Smedts In-Reply-To: Message-ID: References: <20110909222030.GB40207@server.vk2pj.dyndns.org> User-Agent: Alpine 2.01 (GSO 1266 2009-07-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (blade.simplesystems.org [65.66.246.90]); Sat, 10 Sep 2011 11:22:20 -0500 (CDT) Cc: freebsd-fs@freebsd.org Subject: Re: ZFS version 30 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 Sep 2011 16:22:21 -0000 On Sat, 10 Sep 2011, Olivier Smedts wrote: > > After having acquired Sun, Oracle said (I think I read that) they > won't change the license of the zfs sources and the developments will > continue like they were, but they'll only disclose sources after a > Solaris launch. Solaris Express is here, but still no Solaris 11. Any > news on that ? Even Solaris 10_u10 got retracted from the Oracle download site. No one knows anything (or will say anything) until there is an official statement from Oracle. It seems likely that Solaris 10_u10 and Solaris 11 will be released with some fanfare in conjunction with new Oracle hardware products (e.g. based on the SPARC T4 CPU). The Oracle "memo" was a leaked internal memo. Oracle has never clarified its intentions regarding OpenSolaris source code releases. We can only hope that Oracle does the right thing and at least releases zfs source code updates in order to avoid fragmentation of the filesystem format. Bob -- Bob Friesenhahn bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/