From owner-freebsd-fs@FreeBSD.ORG Sun Mar 24 05:40:42 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 39E72859 for ; Sun, 24 Mar 2013 05:40:42 +0000 (UTC) (envelope-from quartz@sneakertech.com) Received: from relay02.pair.com (relay02.pair.com [209.68.5.16]) by mx1.freebsd.org (Postfix) with SMTP id D496B74 for ; Sun, 24 Mar 2013 05:40:41 +0000 (UTC) Received: (qmail 10696 invoked by uid 0); 24 Mar 2013 05:40:40 -0000 Received: from 173.48.104.62 (HELO ?10.2.2.1?) (173.48.104.62) by relay02.pair.com with SMTP; 24 Mar 2013 05:40:40 -0000 X-pair-Authenticated: 173.48.104.62 Message-ID: <514E91D7.2020209@sneakertech.com> Date: Sun, 24 Mar 2013 01:40:39 -0400 From: Quartz User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:10.0.2) Gecko/20120216 Thunderbird/10.0.2 MIME-Version: 1.0 To: Jeremy Chadwick Subject: Re: ZFS: Failed pool causes system to hang References: <20130321044557.GA15977@icarus.home.lan> <514AA192.2090006@sneakertech.com> <20130321085304.GB16997@icarus.home.lan> <514B4D38.6090101@sneakertech.com> <514E166A.5070409@sneakertech.com> <20130323225237.GA85482@icarus.home.lan> In-Reply-To: <20130323225237.GA85482@icarus.home.lan> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 24 Mar 2013 05:40:42 -0000 http://sneakertech.com/-/dmesg.txt Honestly I have my doubts that anything in there will shed particular light on my zfs issue. From what I can tell this appears to be common to zfs cross platform (at least, solaris people have complained about this, and it looks like openindiana might be affected too). ______________________________________ it has a certain smooth-brained appeal From owner-freebsd-fs@FreeBSD.ORG Sun Mar 24 15:33:45 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 1CA5B8A2 for ; Sun, 24 Mar 2013 15:33:45 +0000 (UTC) (envelope-from jdc@koitsu.org) Received: from qmta01.emeryville.ca.mail.comcast.net (qmta01.emeryville.ca.mail.comcast.net [IPv6:2001:558:fe2d:43:76:96:30:16]) by mx1.freebsd.org (Postfix) with ESMTP id 02538FD2 for ; Sun, 24 Mar 2013 15:33:44 +0000 (UTC) Received: from omta02.emeryville.ca.mail.comcast.net ([76.96.30.19]) by qmta01.emeryville.ca.mail.comcast.net with comcast id FR5h1l0040QkzPwA1TZjjK; Sun, 24 Mar 2013 15:33:43 +0000 Received: from koitsu.strangled.net ([67.180.84.87]) by omta02.emeryville.ca.mail.comcast.net with comcast id FTZi1l00A1t3BNj8NTZils; Sun, 24 Mar 2013 15:33:42 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 1E48873A1C; Sun, 24 Mar 2013 08:33:42 -0700 (PDT) Date: Sun, 24 Mar 2013 08:33:42 -0700 From: Jeremy Chadwick To: Quartz Subject: Re: ZFS question Message-ID: <20130324153342.GA3687@icarus.home.lan> References: <20130321044557.GA15977@icarus.home.lan> <514AA192.2090006@sneakertech.com> <20130321085304.GB16997@icarus.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20130321085304.GB16997@icarus.home.lan> User-Agent: Mutt/1.5.21 (2010-09-15) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=comcast.net; s=q20121106; t=1364139223; bh=vj7+C3QoNIJYqjzTB7TuUKhPx22fXhaq4BxOYVPa+pc=; h=Received:Received:Received:Date:From:To:Subject:Message-ID: MIME-Version:Content-Type; b=chR15g7vYtgclHQ0tllAzZbEgdCvPrNtokc/wBrqXd66+85ukiIhaWuDSz2xQWnoZ fkiYlSsxKsOO9RROxZLCDhY6Z4+uhSEspK5NpMxiDfFNoxfeaBMkyRBSxujpiFVMiC xZ/KNJhaYp5jPc7Z10k3ZVJUJyymOr42yewW23bp8LqHmvFrNHxqQQgX7y1qtz8mP8 PshEci2saCrLWYaPIQIR2k9xdafrmjIfTM1NymUZROoanuJPqFRQE6goPGb+/X/rwv +BV010lpDZ1wiE6ol176KEJioWXhAEdFtBgkTFfAqaU9HekoJIw52rZ8cfgZqYfT6i sTqNeGD5rzfzQ== Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 24 Mar 2013 15:33:45 -0000 On Thu, Mar 21, 2013 at 01:53:04AM -0700, Jeremy Chadwick wrote: > There were fixes for very similar situations to this in stable/9 > recently -- I know because I was the person who reported such. mav@ and > ken@ worked out a series of kinks/bugs in CAM pertaining to pass(4) and > xpt(4) and some other things. You can read about that here: > > http://lists.freebsd.org/pipermail/freebsd-fs/2013-February/016515.html > http://lists.freebsd.org/pipermail/freebsd-fs/2013-February/016524.html > > For me to determine if those fixes address the above oddity while > testing, I would need to build stable/9 on this testbox. I can do that, > and will try to dedicate some time to it tomorrow. > > So in summary: there seem to be multiple issues shown above, but I can > confirm that failmode=continue **does** pass EIO to *running* processes > that are doing I/O. Subsequent I/O, however, is questionable at this > time. I've tested with stable/9 (r248571). The issues I noted in my previous post: http://lists.freebsd.org/pipermail/freebsd-fs/2013-March/016814.html ...still exist. Issue is 100% reproducible. However, commands like "zpool status" I also confirmed that **reads** from the now-busted pool do block/wait indefinitely -- this means you cannot Ctrl-C or Ctrl-Z or kill them in any way (this is completely normal for all *IXes (for some reason people think kill -9 will always cause a kernel to unlock/relinquish a thread, which is not the case)). For whatever reason "ls -l /pool" did return results, but I imagine these may be being returned from some caching mechanism within the kernel (either VFS or ZFS ARC, not sure). **Writes**, however, immediately return with EIO. This is with failmode=continue. -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Sun Mar 24 15:37:21 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 3961DA9B for ; Sun, 24 Mar 2013 15:37:21 +0000 (UTC) (envelope-from jdc@koitsu.org) Received: from qmta01.emeryville.ca.mail.comcast.net (qmta01.emeryville.ca.mail.comcast.net [IPv6:2001:558:fe2d:43:76:96:30:16]) by mx1.freebsd.org (Postfix) with ESMTP id 1B98C71 for ; Sun, 24 Mar 2013 15:37:21 +0000 (UTC) Received: from omta22.emeryville.ca.mail.comcast.net ([76.96.30.89]) by qmta01.emeryville.ca.mail.comcast.net with comcast id FScw1l0051vN32cA1TdMAz; Sun, 24 Mar 2013 15:37:21 +0000 Received: from koitsu.strangled.net ([67.180.84.87]) by omta22.emeryville.ca.mail.comcast.net with comcast id FTdL1l0061t3BNj8iTdLwu; Sun, 24 Mar 2013 15:37:20 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 363B273A1C; Sun, 24 Mar 2013 08:37:20 -0700 (PDT) Date: Sun, 24 Mar 2013 08:37:20 -0700 From: Jeremy Chadwick To: Quartz Subject: Re: ZFS: Failed pool causes system to hang Message-ID: <20130324153720.GA3983@icarus.home.lan> References: <20130321044557.GA15977@icarus.home.lan> <514AA192.2090006@sneakertech.com> <20130321085304.GB16997@icarus.home.lan> <514B4D38.6090101@sneakertech.com> <514E166A.5070409@sneakertech.com> <20130323225237.GA85482@icarus.home.lan> <514E91D7.2020209@sneakertech.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <514E91D7.2020209@sneakertech.com> User-Agent: Mutt/1.5.21 (2010-09-15) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=comcast.net; s=q20121106; t=1364139441; bh=HNVxiMkyDjRZfJfvvmN0udnhEPm9twbKHcQyTsGxzj8=; h=Received:Received:Received:Date:From:To:Subject:Message-ID: MIME-Version:Content-Type; b=TOPyUlsbBnqaAmZm+fugJSTePDB4/NgjoEq7v4ZPVipjG+bTycm6vPGwTF+j9X6+x 4VGwyNSBfNf0yVt459a9YrRzRFVQ/Y/ZvE/hn/CcxsWxI5PoCqcjz9nqm79jFeoBJ6 wc+CTLG8DkPsZT39OZ4oIguvUBsB7hwAV4oA029AbXCE9uvit2DpCOtjOaCNTIDUP/ 1KYj6BJRNQR81XVqZ5fG+TONbwwIqWswP80TJGQJv1X2utLnK4ge8RhPrOPxCJK1uE CruVMhFhxyCxURL+TNRwAiGdzQtwhelIMJ4mINXfVfqKDie7/vnGG7uO4ATwAyJUmC VpuSmx5TnusoA== Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 24 Mar 2013 15:37:21 -0000 On Sun, Mar 24, 2013 at 01:40:39AM -0400, Quartz wrote: > http://sneakertech.com/-/dmesg.txt > > Honestly I have my doubts that anything in there will shed > particular light on my zfs issue. From what I can tell this appears > to be common to zfs cross platform (at least, solaris people have > complained about this, and it looks like openindiana might be > affected too). In this case/at this point I wanted dmesg output so I could provide you working /boot/loader.conf lines for your configuration, with regards to "wiring down" device names. If you mess with controllers (e.g. disable one in the BIOS), this may break. Here you go: # "Wire down" device names (ada[0-5]) to each individual port # on the SATA/AHCI controller. This ensures that if we reboot # with a disk missing, the device names stay the same, and stay # attached to the same SATA/AHCI controller. # http://lists.freebsd.org/pipermail/freebsd-fs/2011-March/011036.html # # ada[01] = JMicron JMB363 # ada[234567] = Intel ICH10 # hint.scbus.0.at="ahcich0" hint.scbus.1.at="ahcich1" hint.scbus.2.at="ahcich2" hint.scbus.3.at="ahcich3" hint.scbus.4.at="ahcich4" hint.scbus.5.at="ahcich5" hint.scbus.6.at="ahcich6" hint.scbus.7.at="ahcich7" hint.ada.0.at="scbus0" hint.ada.1.at="scbus1" hint.ada.2.at="scbus2" hint.ada.3.at="scbus3" hint.ada.4.at="scbus4" hint.ada.5.at="scbus5" hint.ada.6.at="scbus6" hint.ada.7.at="scbus7" -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Sun Mar 24 15:38:05 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 2C60CB1F for ; Sun, 24 Mar 2013 15:38:05 +0000 (UTC) (envelope-from jdc@koitsu.org) Received: from qmta03.emeryville.ca.mail.comcast.net (qmta03.emeryville.ca.mail.comcast.net [IPv6:2001:558:fe2d:43:76:96:30:32]) by mx1.freebsd.org (Postfix) with ESMTP id 0C3EF88 for ; Sun, 24 Mar 2013 15:38:05 +0000 (UTC) Received: from omta11.emeryville.ca.mail.comcast.net ([76.96.30.36]) by qmta03.emeryville.ca.mail.comcast.net with comcast id FRQg1l0030mlR8UA3Te4a7; Sun, 24 Mar 2013 15:38:04 +0000 Received: from koitsu.strangled.net ([67.180.84.87]) by omta11.emeryville.ca.mail.comcast.net with comcast id FTe41l0081t3BNj8XTe4an; Sun, 24 Mar 2013 15:38:04 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 0C2CF73A1E; Sun, 24 Mar 2013 08:38:04 -0700 (PDT) Date: Sun, 24 Mar 2013 08:38:04 -0700 From: Jeremy Chadwick To: Quartz Subject: Re: ZFS: Failed pool causes system to hang Message-ID: <20130324153804.GA4052@icarus.home.lan> References: <20130321044557.GA15977@icarus.home.lan> <514AA192.2090006@sneakertech.com> <20130321085304.GB16997@icarus.home.lan> <514B4D38.6090101@sneakertech.com> <514E166A.5070409@sneakertech.com> <20130323225237.GA85482@icarus.home.lan> <514E91D7.2020209@sneakertech.com> <20130324153720.GA3983@icarus.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20130324153720.GA3983@icarus.home.lan> User-Agent: Mutt/1.5.21 (2010-09-15) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=comcast.net; s=q20121106; t=1364139484; bh=ZirwwODlp4yH/AqGfcHJIcVJAQSKkBZ+EM4kF5yfTRM=; h=Received:Received:Received:Date:From:To:Subject:Message-ID: MIME-Version:Content-Type; b=b2uxIN1ICtNl4wcBWQpegmPIBxRISiyKqIMShLyudtQesLvcUFyja6HMiKSZxRv6w QKDkA3QuSkLfkTPy2oXLKWjtasFcKVAq9ARmEP+yUmpEs3sUC30PxuBxpIUV6Dd9Eb 3/Fz0fWUeLwGvvIAoV9BAIOu2ZD2eiRC8WKnMrwDkMjvwZUWqhLXFAB7Sa5CNE06Cf GLLUKjFXWIKvmgaYTFMaAUadU123h58m9C1xalUtAO4RzZgsgu4wnqUbD4h8JhdSK5 sBTj7LtNLv+LlM7kv+0WRcu8Ka6XKlSlCYLozRWHkRjWMfGH8baND7Bhdmj86we+eR VPFlnqWlAM1KQ== Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 24 Mar 2013 15:38:05 -0000 On Sun, Mar 24, 2013 at 08:37:20AM -0700, Jeremy Chadwick wrote: > # "Wire down" device names (ada[0-5]) to each individual port This comment should have read "ada[0-7]", sorry. -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Sun Mar 24 15:54:49 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id F28B5446 for ; Sun, 24 Mar 2013 15:54:49 +0000 (UTC) (envelope-from jdc@koitsu.org) Received: from qmta06.emeryville.ca.mail.comcast.net (qmta06.emeryville.ca.mail.comcast.net [IPv6:2001:558:fe2d:43:76:96:30:56]) by mx1.freebsd.org (Postfix) with ESMTP id D7BE518C for ; Sun, 24 Mar 2013 15:54:49 +0000 (UTC) Received: from omta10.emeryville.ca.mail.comcast.net ([76.96.30.28]) by qmta06.emeryville.ca.mail.comcast.net with comcast id FStl1l0010cQ2SLA6Tupbu; Sun, 24 Mar 2013 15:54:49 +0000 Received: from koitsu.strangled.net ([67.180.84.87]) by omta10.emeryville.ca.mail.comcast.net with comcast id FTuo1l00V1t3BNj8WTupTA; Sun, 24 Mar 2013 15:54:49 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id CD7D773A1C; Sun, 24 Mar 2013 08:54:48 -0700 (PDT) Date: Sun, 24 Mar 2013 08:54:48 -0700 From: Jeremy Chadwick To: Quartz Subject: Re: ZFS question Message-ID: <20130324155448.GA4122@icarus.home.lan> References: <20130321044557.GA15977@icarus.home.lan> <514AA192.2090006@sneakertech.com> <20130321085304.GB16997@icarus.home.lan> <20130324153342.GA3687@icarus.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20130324153342.GA3687@icarus.home.lan> User-Agent: Mutt/1.5.21 (2010-09-15) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=comcast.net; s=q20121106; t=1364140489; bh=iNs4SGQ9HJweXQCfM6xL6Sh20a59ezADV4/ilKNhBU0=; h=Received:Received:Received:Date:From:To:Subject:Message-ID: MIME-Version:Content-Type; b=Dt4J+CAdfXfFNX7pID8mF2h5fXPtjbAinTY23giphVPK09NHwE7IaORbMTVj+Qah5 tl2cvzp3AX3N0kah/cbheLTdIWVi3V8rWrX354Y5d0jsMrgHTdCV35XEOyMFzOyiWw 59S/5Mxofv9CYVl4XzMAF6/w6ORtCzRzZmJUrvzySCNC0YWeiLHOuqQm7yll6DIvX2 W9qrGg/lyuS71gtel1QduOTKz4pPTjvQHLJP4hFrqw2hI/nKbazBfDKNj+XGIhiNg6 58Y6YTa3GiTSwWDKrZJNP+IftYTCGmSjSm2Kykpkx8JqLM/I+N1fm9plpHNIntxdVb edf93HDELtaAw== Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 24 Mar 2013 15:54:50 -0000 On Sun, Mar 24, 2013 at 08:33:42AM -0700, Jeremy Chadwick wrote: > However, commands like "zpool status" ...and seems a typo I made in vim caused the rest of my sentence to get deleted before I sent it out. This should have read: > However, commands like "zpool status" work just fine, but things like > "zpool destroy" and so on indefinitely block ("mount drain"), which to > me makes some degree of sense. To expand: for example, you've lost 3 disks of a 4-disk raidz2 pool, your data is buggered and you'll need to recover from backups. Yes, you will need to reboot for the ZFS layer to effectively "un-wedge" itself from whatever catatonic state its in. No argument: this is a bug somewhere, and my guess is that it relates to the confused state of the devices in CAM-land. But regardless, I think if you were to lose 3 of 4 disks on a raidz2 pool you'd have much more serious things to be worried about than "well crap I have to issue a reboot". And yes, I did test a reboot in the scenario I described -- the system did reboot without physically pressing the button. But then again, for remotely-managed systems, administrators should have the ability to remotely power-cycle or force resets (e.g. drop to DDB via serial console and force a reset). People who run servers remotely yet lack this capability are intentionally choosing to live dangerously and I do not condone such. These folks also make me wonder how they update world without remote console access, since to do it right you *must* drop to single-user for the installworld phase. I learned my lesson of "assuming" installworld would work from multi-user long ago when it broke one time and I ended up with a system with broken /libexec/ld-elf* binaries. Having to go to the datacenter 30 minutes away at 3 in the morning taught me to follow instructions. -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Sun Mar 24 18:26:19 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id B72D6770 for ; Sun, 24 Mar 2013 18:26:19 +0000 (UTC) (envelope-from quartz@sneakertech.com) Received: from relay01.pair.com (relay01.pair.com [209.68.5.15]) by mx1.freebsd.org (Postfix) with SMTP id 5E059DC4 for ; Sun, 24 Mar 2013 18:26:18 +0000 (UTC) Received: (qmail 84276 invoked by uid 0); 24 Mar 2013 18:26:10 -0000 Received: from 173.48.104.62 (HELO ?10.2.2.1?) (173.48.104.62) by relay01.pair.com with SMTP; 24 Mar 2013 18:26:10 -0000 X-pair-Authenticated: 173.48.104.62 Message-ID: <514F4542.7060100@sneakertech.com> Date: Sun, 24 Mar 2013 14:26:10 -0400 From: Quartz User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:10.0.2) Gecko/20120216 Thunderbird/10.0.2 MIME-Version: 1.0 To: Jeremy Chadwick Subject: Re: ZFS question References: <20130321044557.GA15977@icarus.home.lan> <514AA192.2090006@sneakertech.com> <20130321085304.GB16997@icarus.home.lan> <20130324153342.GA3687@icarus.home.lan> In-Reply-To: <20130324153342.GA3687@icarus.home.lan> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 24 Mar 2013 18:26:19 -0000 > I also confirmed that **reads** from the now-busted pool do block/wait > indefinitely Yeah, and that's the kicker. When my pool dies, I can't even check its status to figure out what the hell went wrong. >(for some reason people > think kill -9 will always cause a kernel to unlock/relinquish a thread, > which is not the case)). Oh believe me, from years of experience with linux/mac I wish it were true. >For whatever reason "ls -l /pool" did return > results, but I imagine these may be being returned from some caching > mechanism within the kernel (either VFS or ZFS ARC, not sure). I've noticed this too a couple times, but it only appears to happen for the top directory or so. Copy a folder with a lot of nested subfolders and then test with 'ls -R', that will kill it guaranteed. ______________________________________ it has a certain smooth-brained appeal From owner-freebsd-fs@FreeBSD.ORG Sun Mar 24 18:30:33 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 82C6B8ED for ; Sun, 24 Mar 2013 18:30:33 +0000 (UTC) (envelope-from quartz@sneakertech.com) Received: from relay00.pair.com (relay00.pair.com [209.68.5.9]) by mx1.freebsd.org (Postfix) with SMTP id 2781FDEB for ; Sun, 24 Mar 2013 18:30:32 +0000 (UTC) Received: (qmail 17751 invoked by uid 0); 24 Mar 2013 18:30:25 -0000 Received: from 173.48.104.62 (HELO ?10.2.2.1?) (173.48.104.62) by relay00.pair.com with SMTP; 24 Mar 2013 18:30:25 -0000 X-pair-Authenticated: 173.48.104.62 Message-ID: <514F4641.8000508@sneakertech.com> Date: Sun, 24 Mar 2013 14:30:25 -0400 From: Quartz User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:10.0.2) Gecko/20120216 Thunderbird/10.0.2 MIME-Version: 1.0 To: Jeremy Chadwick Subject: Re: ZFS: Failed pool causes system to hang References: <20130321044557.GA15977@icarus.home.lan> <514AA192.2090006@sneakertech.com> <20130321085304.GB16997@icarus.home.lan> <514B4D38.6090101@sneakertech.com> <514E166A.5070409@sneakertech.com> <20130323225237.GA85482@icarus.home.lan> <514E91D7.2020209@sneakertech.com> <20130324153720.GA3983@icarus.home.lan> In-Reply-To: <20130324153720.GA3983@icarus.home.lan> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 24 Mar 2013 18:30:33 -0000 > In this case/at this point I wanted dmesg output so I could provide you > working /boot/loader.conf lines for your configuration, with regards to > "wiring down" device names. OK, thanks. >If you mess with controllers (e.g. disable > one in the BIOS), this may break. Here you go: There's a high probability I'll be adding another sata card to the mix later (assuming I can get the hang thing resolved). > [snip] Huh, that syntax is a lot simpler than what I was expecting. I just copy that into /boot/loader.conf and that's all? ______________________________________ it has a certain smooth-brained appeal From owner-freebsd-fs@FreeBSD.ORG Sun Mar 24 18:54:17 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 73894ED9 for ; Sun, 24 Mar 2013 18:54:17 +0000 (UTC) (envelope-from quartz@sneakertech.com) Received: from relay01.pair.com (relay01.pair.com [209.68.5.15]) by mx1.freebsd.org (Postfix) with SMTP id 30918ED0 for ; Sun, 24 Mar 2013 18:54:16 +0000 (UTC) Received: (qmail 90458 invoked by uid 0); 24 Mar 2013 18:54:15 -0000 Received: from 173.48.104.62 (HELO ?10.2.2.1?) (173.48.104.62) by relay01.pair.com with SMTP; 24 Mar 2013 18:54:15 -0000 X-pair-Authenticated: 173.48.104.62 Message-ID: <514F4BD6.1060807@sneakertech.com> Date: Sun, 24 Mar 2013 14:54:14 -0400 From: Quartz User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:10.0.2) Gecko/20120216 Thunderbird/10.0.2 MIME-Version: 1.0 To: Jeremy Chadwick Subject: Re: ZFS: Failed pool causes system to hang References: <20130321044557.GA15977@icarus.home.lan> <514AA192.2090006@sneakertech.com> <20130321085304.GB16997@icarus.home.lan> <20130324153342.GA3687@icarus.home.lan> <20130324155448.GA4122@icarus.home.lan> In-Reply-To: <20130324155448.GA4122@icarus.home.lan> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 24 Mar 2013 18:54:17 -0000 >> However, commands like "zpool status" > > ...and seems a typo I made in vim caused the rest of my sentence to get > deleted before I sent it out. This should have read: > >> However, commands like "zpool status" work just fine, but things like >> "zpool destroy" and so on indefinitely block ("mount drain"), which to >> me makes some degree of sense. I'll have to double check this. I *know* I've run status and had it hang, but I'm not 100% certain if I've done it fast enough to guarantee that something else didn't hit the pool first. > Yes, you will need to reboot for the ZFS layer to effectively "un-wedge" > itself from whatever catatonic state its in. No argument: this is a bug > somewhere, and my guess is that it relates to the confused state of the > devices in CAM-land. But regardless, I think if you were to lose 3 of 4 > disks on a raidz2 pool you'd have much more serious things to be worried > about than "well crap I have to issue a reboot". My concern is proper investigation and damage control. The "it stopped working, guess I should reboot" is the windows way of administration. In the case of serious hardware failure, rebooting or otherwise continuing to provide power to the affected devices can be a very BAD thing. I'd like to have some idea of what the heck happened before I blindly powercycle something. > And yes, I did test a reboot in the scenario I described -- the system > did reboot without physically pressing the button. It *never* does for me. Ever. > People who run servers remotely yet lack this capability are > intentionally choosing [snip] Before you get up on a high horse and preach at me, consider a couple things: 1) Yes I can set that up, but this is a test box on my desk right now. 2) A hard reset is a hard reset is a hard reset. I'm not bitching that I have to physically walk over to the machine, I'm bitching that *THAT I HAVE TO RESET IT*. Being able to reset it remotely is NOT an acceptable solution or workaround, and has no bearing on my problem. ______________________________________ it has a certain smooth-brained appeal From owner-freebsd-fs@FreeBSD.ORG Sun Mar 24 19:19:55 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 483B661D for ; Sun, 24 Mar 2013 19:19:55 +0000 (UTC) (envelope-from chris@behanna.org) Received: from relay00.pair.com (relay00.pair.com [209.68.5.9]) by mx1.freebsd.org (Postfix) with SMTP id 06E4AFB6 for ; Sun, 24 Mar 2013 19:19:54 +0000 (UTC) Received: (qmail 28490 invoked by uid 0); 24 Mar 2013 19:19:54 -0000 Received: from 99.120.172.51 (HELO scythe.behanna.org) (99.120.172.51) by relay00.pair.com with SMTP; 24 Mar 2013 19:19:54 -0000 X-pair-Authenticated: 99.120.172.51 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.3 \(1503\)) Subject: Re: ZFS: Failed pool causes system to hang From: Chris BeHanna In-Reply-To: <514F4BD6.1060807@sneakertech.com> Date: Sun, 24 Mar 2013 14:19:53 -0500 Content-Transfer-Encoding: quoted-printable Message-Id: <3207E818-28B6-4F48-A100-4527F6D23936@behanna.org> References: <20130321044557.GA15977@icarus.home.lan> <514AA192.2090006@sneakertech.com> <20130321085304.GB16997@icarus.home.lan> <20130324153342.GA3687@icarus.home.lan> <20130324155448.GA4122@icarus.home.lan> <514F4BD6.1060807@sneakertech.com> To: FreeBSD FS X-Mailer: Apple Mail (2.1503) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 24 Mar 2013 19:19:55 -0000 On Mar 24, 2013, at 13:54 , Quartz wrote: >>> [...situations in which you *have* to reboot...] >=20 > [...all well and good, except for cases where someone has to = physically hit a button on a remote server...] >=20 >> People who run servers remotely yet lack this capability are >> intentionally choosing [snip] >=20 > Before you get up on a high horse and preach at me, consider a couple = things: >=20 > 1) Yes I can set that up, but this is a test box on my desk right now. >=20 > 2) A hard reset is a hard reset is a hard reset. I'm not bitching that = I have to physically walk over to the machine, I'm bitching that *THAT I = HAVE TO RESET IT*. Being able to reset it remotely is NOT an acceptable = solution or workaround, and has no bearing on my problem. Sure, there are bugs that inhibit the ability to reboot from = command-line. We shouldn't shrug them off, but we should also = acknowledge that the software is very complicated, and rooting out such = bugs takes time. Plus, this is a volunteer project. That said, there are some chassis that will let you drop to = lights-out management and force a reboot without having to physically = press a button, provided you have a way to reach the lights-out = management console. This often means, at least, a serial line = concentrator somewhere ("conserver" is simply awesome), or KVM-over-IP. Worst case, you have IP-addressable PDUs into which you can = telnet and hard-reset a particular outlet, to power-cycle a machine that = is truly, irrevocably stuck, without having to have a human walk over = and stick a paper clip into the reset hole. This should be a last = resort, though, as abrupt power-cycling shortens hardware life. --=20 Best, Chris BeHanna chris@behanna.org= From owner-freebsd-fs@FreeBSD.ORG Sun Mar 24 19:58:21 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 3156D1000 for ; Sun, 24 Mar 2013 19:58:21 +0000 (UTC) (envelope-from quartz@sneakertech.com) Received: from relay02.pair.com (relay02.pair.com [209.68.5.16]) by mx1.freebsd.org (Postfix) with SMTP id CA9A51C7 for ; Sun, 24 Mar 2013 19:58:20 +0000 (UTC) Received: (qmail 78655 invoked by uid 0); 24 Mar 2013 19:58:13 -0000 Received: from 173.48.104.62 (HELO ?10.2.2.1?) (173.48.104.62) by relay02.pair.com with SMTP; 24 Mar 2013 19:58:13 -0000 X-pair-Authenticated: 173.48.104.62 Message-ID: <514F5AD5.8000006@sneakertech.com> Date: Sun, 24 Mar 2013 15:58:13 -0400 From: Quartz User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:10.0.2) Gecko/20120216 Thunderbird/10.0.2 MIME-Version: 1.0 To: Chris BeHanna Subject: Re: ZFS: Failed pool causes system to hang References: <20130321044557.GA15977@icarus.home.lan> <514AA192.2090006@sneakertech.com> <20130321085304.GB16997@icarus.home.lan> <20130324153342.GA3687@icarus.home.lan> <20130324155448.GA4122@icarus.home.lan> <514F4BD6.1060807@sneakertech.com> <3207E818-28B6-4F48-A100-4527F6D23936@behanna.org> In-Reply-To: <3207E818-28B6-4F48-A100-4527F6D23936@behanna.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD FS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 24 Mar 2013 19:58:21 -0000 > Sure, there are bugs that inhibit the ability to reboot from > command-line. We shouldn't shrug them off, but we should also > acknowledge that the software is very complicated, and rooting out > such bugs takes time. Plus, this is a volunteer project. No, I'm not trying to say that I expect something as complicated as a whole OS to be bug free, and I do well recognize that it's no one's obligation to fix anything anyway.... I'm just saying that starting an argument about the remote-ness of the reboot is missing the point. > Worst case, you have IP-addressable PDUs That's what we usually go with- guaranteed to work with any hardware. That plus a vnc capable kvm. ______________________________________ it has a certain smooth-brained appeal From owner-freebsd-fs@FreeBSD.ORG Mon Mar 25 11:06:42 2013 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 83AEAFE6 for ; Mon, 25 Mar 2013 11:06:42 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id 66BCA92 for ; Mon, 25 Mar 2013 11:06:42 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.6/8.14.6) with ESMTP id r2PB6gbd007128 for ; Mon, 25 Mar 2013 11:06:42 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.6/8.14.6/Submit) id r2PB6g5S007125 for freebsd-fs@FreeBSD.org; Mon, 25 Mar 2013 11:06:42 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 25 Mar 2013 11:06:42 GMT Message-Id: <201303251106.r2PB6g5S007125@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-fs@FreeBSD.org Subject: Current problem reports assigned to freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Mar 2013 11:06:42 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/177240 fs [zfs] zpool import failed with state UNAVAIL but all d o kern/176978 fs [zfs] [panic] zfs send -D causes "panic: System call i o kern/176857 fs [softupdates] [panic] 9.1-RELEASE/amd64/GENERIC panic o bin/176253 fs zpool(8): zfs pool indentation is misleading/wrong o kern/176141 fs [zfs] sharesmb=on makes errors for sharenfs, and still o kern/175950 fs [zfs] Possible deadlock in zfs after long uptime o kern/175897 fs [zfs] operations on readonly zpool hang o kern/175179 fs [zfs] ZFS may attach wrong device on move o kern/175071 fs [ufs] [panic] softdep_deallocate_dependencies: unrecov o kern/174372 fs [zfs] Pagefault appears to be related to ZFS o kern/174315 fs [zfs] chflags uchg not supported o kern/174310 fs [zfs] root point mounting broken on CURRENT with multi o kern/174279 fs [ufs] UFS2-SU+J journal and filesystem corruption o kern/174060 fs [ext2fs] Ext2FS system crashes (buffer overflow?) o kern/173830 fs [zfs] Brain-dead simple change to ZFS error descriptio o kern/173718 fs [zfs] phantom directory in zraid2 pool f kern/173657 fs [nfs] strange UID map with nfsuserd o kern/173363 fs [zfs] [panic] Panic on 'zpool replace' on readonly poo o kern/173136 fs [unionfs] mounting above the NFS read-only share panic o kern/172348 fs [unionfs] umount -f of filesystem in use with readonly o kern/172334 fs [unionfs] unionfs permits recursive union mounts; caus o kern/171626 fs [tmpfs] tmpfs should be noisier when the requested siz o kern/171415 fs [zfs] zfs recv fails with "cannot receive incremental o kern/170945 fs [gpt] disk layout not portable between direct connect o bin/170778 fs [zfs] [panic] FreeBSD panics randomly o kern/170680 fs [nfs] Multiple NFS Client bug in the FreeBSD 7.4-RELEA o kern/170497 fs [xfs][panic] kernel will panic whenever I ls a mounted o kern/169945 fs [zfs] [panic] Kernel panic while importing zpool (afte o kern/169480 fs [zfs] ZFS stalls on heavy I/O o kern/169398 fs [zfs] Can't remove file with permanent error o kern/169339 fs panic while " : > /etc/123" o kern/169319 fs [zfs] zfs resilver can't complete o kern/168947 fs [nfs] [zfs] .zfs/snapshot directory is messed up when o kern/168942 fs [nfs] [hang] nfsd hangs after being restarted (not -HU o kern/168158 fs [zfs] incorrect parsing of sharenfs options in zfs (fs o kern/167979 fs [ufs] DIOCGDINFO ioctl does not work on 8.2 file syste o kern/167977 fs [smbfs] mount_smbfs results are differ when utf-8 or U o kern/167688 fs [fusefs] Incorrect signal handling with direct_io o kern/167685 fs [zfs] ZFS on USB drive prevents shutdown / reboot o kern/167612 fs [portalfs] The portal file system gets stuck inside po o kern/167272 fs [zfs] ZFS Disks reordering causes ZFS to pick the wron o kern/167260 fs [msdosfs] msdosfs disk was mounted the second time whe o kern/167109 fs [zfs] [panic] zfs diff kernel panic Fatal trap 9: gene o kern/167105 fs [nfs] mount_nfs can not handle source exports wiht mor o kern/167067 fs [zfs] [panic] ZFS panics the server o kern/167065 fs [zfs] boot fails when a spare is the boot disk o kern/167048 fs [nfs] [patch] RELEASE-9 crash when using ZFS+NULLFS+NF o kern/166912 fs [ufs] [panic] Panic after converting Softupdates to jo o kern/166851 fs [zfs] [hang] Copying directory from the mounted UFS di o kern/166477 fs [nfs] NFS data corruption. o kern/165950 fs [ffs] SU+J and fsck problem o kern/165521 fs [zfs] [hang] livelock on 1 Gig of RAM with zfs when 31 o kern/165392 fs Multiple mkdir/rmdir fails with errno 31 o kern/165087 fs [unionfs] lock violation in unionfs o kern/164472 fs [ufs] fsck -B panics on particular data inconsistency o kern/164370 fs [zfs] zfs destroy for snapshot fails on i386 and sparc o kern/164261 fs [nullfs] [patch] fix panic with NFS served from NULLFS o kern/164256 fs [zfs] device entry for volume is not created after zfs o kern/164184 fs [ufs] [panic] Kernel panic with ufs_makeinode o kern/163801 fs [md] [request] allow mfsBSD legacy installed in 'swap' o kern/163770 fs [zfs] [hang] LOR between zfs&syncer + vnlru leading to o kern/163501 fs [nfs] NFS exporting a dir and a subdir in that dir to o kern/162944 fs [coda] Coda file system module looks broken in 9.0 o kern/162860 fs [zfs] Cannot share ZFS filesystem to hosts with a hyph o kern/162751 fs [zfs] [panic] kernel panics during file operations o kern/162591 fs [nullfs] cross-filesystem nullfs does not work as expe o kern/162519 fs [zfs] "zpool import" relies on buggy realpath() behavi o kern/161968 fs [zfs] [hang] renaming snapshot with -r including a zvo o kern/161864 fs [ufs] removing journaling from UFS partition fails on o bin/161807 fs [patch] add option for explicitly specifying metadata o kern/161579 fs [smbfs] FreeBSD sometimes panics when an smb share is o kern/161533 fs [zfs] [panic] zfs receive panic: system ioctl returnin o kern/161438 fs [zfs] [panic] recursed on non-recursive spa_namespace_ o kern/161424 fs [nullfs] __getcwd() calls fail when used on nullfs mou o kern/161280 fs [zfs] Stack overflow in gptzfsboot o kern/161205 fs [nfs] [pfsync] [regression] [build] Bug report freebsd o kern/161169 fs [zfs] [panic] ZFS causes kernel panic in dbuf_dirty o kern/161112 fs [ufs] [lor] filesystem LOR in FreeBSD 9.0-BETA3 o kern/160893 fs [zfs] [panic] 9.0-BETA2 kernel panic o kern/160860 fs [ufs] Random UFS root filesystem corruption with SU+J o kern/160801 fs [zfs] zfsboot on 8.2-RELEASE fails to boot from root-o o kern/160790 fs [fusefs] [panic] VPUTX: negative ref count with FUSE o kern/160777 fs [zfs] [hang] RAID-Z3 causes fatal hang upon scrub/impo o kern/160706 fs [zfs] zfs bootloader fails when a non-root vdev exists o kern/160591 fs [zfs] Fail to boot on zfs root with degraded raidz2 [r o kern/160410 fs [smbfs] [hang] smbfs hangs when transferring large fil o kern/160283 fs [zfs] [patch] 'zfs list' does abort in make_dataset_ha o kern/159930 fs [ufs] [panic] kernel core o kern/159402 fs [zfs][loader] symlinks cause I/O errors o kern/159357 fs [zfs] ZFS MAXNAMELEN macro has confusing name (off-by- o kern/159356 fs [zfs] [patch] ZFS NAME_ERR_DISKLIKE check is Solaris-s o kern/159351 fs [nfs] [patch] - divide by zero in mountnfs() o kern/159251 fs [zfs] [request]: add FLETCHER4 as DEDUP hash option o kern/159077 fs [zfs] Can't cd .. with latest zfs version o kern/159048 fs [smbfs] smb mount corrupts large files o kern/159045 fs [zfs] [hang] ZFS scrub freezes system o kern/158839 fs [zfs] ZFS Bootloader Fails if there is a Dead Disk o kern/158802 fs amd(8) ICMP storm and unkillable process. o kern/158231 fs [nullfs] panic on unmounting nullfs mounted over ufs o f kern/157929 fs [nfs] NFS slow read o kern/157399 fs [zfs] trouble with: mdconfig force delete && zfs strip o kern/157179 fs [zfs] zfs/dbuf.c: panic: solaris assert: arc_buf_remov o kern/156797 fs [zfs] [panic] Double panic with FreeBSD 9-CURRENT and o kern/156781 fs [zfs] zfs is losing the snapshot directory, p kern/156545 fs [ufs] mv could break UFS on SMP systems o kern/156193 fs [ufs] [hang] UFS snapshot hangs && deadlocks processes o kern/156039 fs [nullfs] [unionfs] nullfs + unionfs do not compose, re o kern/155615 fs [zfs] zfs v28 broken on sparc64 -current o kern/155587 fs [zfs] [panic] kernel panic with zfs p kern/155411 fs [regression] [8.2-release] [tmpfs]: mount: tmpfs : No o kern/155199 fs [ext2fs] ext3fs mounted as ext2fs gives I/O errors o bin/155104 fs [zfs][patch] use /dev prefix by default when importing o kern/154930 fs [zfs] cannot delete/unlink file from full volume -> EN o kern/154828 fs [msdosfs] Unable to create directories on external USB o kern/154491 fs [smbfs] smb_co_lock: recursive lock for object 1 p kern/154228 fs [md] md getting stuck in wdrain state o kern/153996 fs [zfs] zfs root mount error while kernel is not located o kern/153753 fs [zfs] ZFS v15 - grammatical error when attempting to u o kern/153716 fs [zfs] zpool scrub time remaining is incorrect o kern/153695 fs [patch] [zfs] Booting from zpool created on 4k-sector o kern/153680 fs [xfs] 8.1 failing to mount XFS partitions o kern/153418 fs [zfs] [panic] Kernel Panic occurred writing to zfs vol o kern/153351 fs [zfs] locking directories/files in ZFS o bin/153258 fs [patch][zfs] creating ZVOLs requires `refreservation' s kern/153173 fs [zfs] booting from a gzip-compressed dataset doesn't w o bin/153142 fs [zfs] ls -l outputs `ls: ./.zfs: Operation not support o kern/153126 fs [zfs] vdev failure, zpool=peegel type=vdev.too_small o kern/152022 fs [nfs] nfs service hangs with linux client [regression] o kern/151942 fs [zfs] panic during ls(1) zfs snapshot directory o kern/151905 fs [zfs] page fault under load in /sbin/zfs o bin/151713 fs [patch] Bug in growfs(8) with respect to 32-bit overfl o kern/151648 fs [zfs] disk wait bug o kern/151629 fs [fs] [patch] Skip empty directory entries during name o kern/151330 fs [zfs] will unshare all zfs filesystem after execute a o kern/151326 fs [nfs] nfs exports fail if netgroups contain duplicate o kern/151251 fs [ufs] Can not create files on filesystem with heavy us o kern/151226 fs [zfs] can't delete zfs snapshot o kern/150503 fs [zfs] ZFS disks are UNAVAIL and corrupted after reboot o kern/150501 fs [zfs] ZFS vdev failure vdev.bad_label on amd64 o kern/150390 fs [zfs] zfs deadlock when arcmsr reports drive faulted o kern/150336 fs [nfs] mountd/nfsd became confused; refused to reload n o kern/149208 fs mksnap_ffs(8) hang/deadlock o kern/149173 fs [patch] [zfs] make OpenSolaris installa o kern/149015 fs [zfs] [patch] misc fixes for ZFS code to build on Glib o kern/149014 fs [zfs] [patch] declarations in ZFS libraries/utilities o kern/149013 fs [zfs] [patch] make ZFS makefiles use the libraries fro o kern/148504 fs [zfs] ZFS' zpool does not allow replacing drives to be o kern/148490 fs [zfs]: zpool attach - resilver bidirectionally, and re o kern/148368 fs [zfs] ZFS hanging forever on 8.1-PRERELEASE o kern/148138 fs [zfs] zfs raidz pool commands freeze o kern/147903 fs [zfs] [panic] Kernel panics on faulty zfs device o kern/147881 fs [zfs] [patch] ZFS "sharenfs" doesn't allow different " o kern/147420 fs [ufs] [panic] ufs_dirbad, nullfs, jail panic (corrupt o kern/146941 fs [zfs] [panic] Kernel Double Fault - Happens constantly o kern/146786 fs [zfs] zpool import hangs with checksum errors o kern/146708 fs [ufs] [panic] Kernel panic in softdep_disk_write_compl o kern/146528 fs [zfs] Severe memory leak in ZFS on i386 o kern/146502 fs [nfs] FreeBSD 8 NFS Client Connection to Server s kern/145712 fs [zfs] cannot offline two drives in a raidz2 configurat o kern/145411 fs [xfs] [panic] Kernel panics shortly after mounting an f bin/145309 fs bsdlabel: Editing disk label invalidates the whole dev o kern/145272 fs [zfs] [panic] Panic during boot when accessing zfs on o kern/145246 fs [ufs] dirhash in 7.3 gratuitously frees hashes when it o kern/145238 fs [zfs] [panic] kernel panic on zpool clear tank o kern/145229 fs [zfs] Vast differences in ZFS ARC behavior between 8.0 o kern/145189 fs [nfs] nfsd performs abysmally under load o kern/144929 fs [ufs] [lor] vfs_bio.c + ufs_dirhash.c p kern/144447 fs [zfs] sharenfs fsunshare() & fsshare_main() non functi o kern/144416 fs [panic] Kernel panic on online filesystem optimization s kern/144415 fs [zfs] [panic] kernel panics on boot after zfs crash o kern/144234 fs [zfs] Cannot boot machine with recent gptzfsboot code o kern/143825 fs [nfs] [panic] Kernel panic on NFS client o bin/143572 fs [zfs] zpool(1): [patch] The verbose output from iostat o kern/143212 fs [nfs] NFSv4 client strange work ... o kern/143184 fs [zfs] [lor] zfs/bufwait LOR o kern/142878 fs [zfs] [vfs] lock order reversal o kern/142597 fs [ext2fs] ext2fs does not work on filesystems with real o kern/142489 fs [zfs] [lor] allproc/zfs LOR o kern/142466 fs Update 7.2 -> 8.0 on Raid 1 ends with screwed raid [re o kern/142306 fs [zfs] [panic] ZFS drive (from OSX Leopard) causes two o kern/142068 fs [ufs] BSD labels are got deleted spontaneously o kern/141897 fs [msdosfs] [panic] Kernel panic. msdofs: file name leng o kern/141463 fs [nfs] [panic] Frequent kernel panics after upgrade fro o kern/141305 fs [zfs] FreeBSD ZFS+sendfile severe performance issues ( o kern/141091 fs [patch] [nullfs] fix panics with DIAGNOSTIC enabled o kern/141086 fs [nfs] [panic] panic("nfs: bioread, not dir") on FreeBS o kern/141010 fs [zfs] "zfs scrub" fails when backed by files in UFS2 o kern/140888 fs [zfs] boot fail from zfs root while the pool resilveri o kern/140661 fs [zfs] [patch] /boot/loader fails to work on a GPT/ZFS- o kern/140640 fs [zfs] snapshot crash o kern/140068 fs [smbfs] [patch] smbfs does not allow semicolon in file o kern/139725 fs [zfs] zdb(1) dumps core on i386 when examining zpool c o kern/139715 fs [zfs] vfs.numvnodes leak on busy zfs p bin/139651 fs [nfs] mount(8): read-only remount of NFS volume does n o kern/139407 fs [smbfs] [panic] smb mount causes system crash if remot o kern/138662 fs [panic] ffs_blkfree: freeing free block o kern/138421 fs [ufs] [patch] remove UFS label limitations o kern/138202 fs mount_msdosfs(1) see only 2Gb o kern/136968 fs [ufs] [lor] ufs/bufwait/ufs (open) o kern/136945 fs [ufs] [lor] filedesc structure/ufs (poll) o kern/136944 fs [ffs] [lor] bufwait/snaplk (fsync) o kern/136873 fs [ntfs] Missing directories/files on NTFS volume o kern/136865 fs [nfs] [patch] NFS exports atomic and on-the-fly atomic p kern/136470 fs [nfs] Cannot mount / in read-only, over NFS o kern/135546 fs [zfs] zfs.ko module doesn't ignore zpool.cache filenam o kern/135469 fs [ufs] [panic] kernel crash on md operation in ufs_dirb o kern/135050 fs [zfs] ZFS clears/hides disk errors on reboot o kern/134491 fs [zfs] Hot spares are rather cold... o kern/133676 fs [smbfs] [panic] umount -f'ing a vnode-based memory dis p kern/133174 fs [msdosfs] [patch] msdosfs must support multibyte inter o kern/132960 fs [ufs] [panic] panic:ffs_blkfree: freeing free frag o kern/132397 fs reboot causes filesystem corruption (failure to sync b o kern/132331 fs [ufs] [lor] LOR ufs and syncer o kern/132237 fs [msdosfs] msdosfs has problems to read MSDOS Floppy o kern/132145 fs [panic] File System Hard Crashes o kern/131441 fs [unionfs] [nullfs] unionfs and/or nullfs not combineab o kern/131360 fs [nfs] poor scaling behavior of the NFS server under lo o kern/131342 fs [nfs] mounting/unmounting of disks causes NFS to fail o bin/131341 fs makefs: error "Bad file descriptor" on the mount poin o kern/130920 fs [msdosfs] cp(1) takes 100% CPU time while copying file o kern/130210 fs [nullfs] Error by check nullfs o kern/129760 fs [nfs] after 'umount -f' of a stale NFS share FreeBSD l o kern/129488 fs [smbfs] Kernel "bug" when using smbfs in smbfs_smb.c: o kern/129231 fs [ufs] [patch] New UFS mount (norandom) option - mostly o kern/129152 fs [panic] non-userfriendly panic when trying to mount(8) o kern/127787 fs [lor] [ufs] Three LORs: vfslock/devfs/vfslock, ufs/vfs o bin/127270 fs fsck_msdosfs(8) may crash if BytesPerSec is zero o kern/127029 fs [panic] mount(8): trying to mount a write protected zi o kern/126287 fs [ufs] [panic] Kernel panics while mounting an UFS file o kern/125895 fs [ffs] [panic] kernel: panic: ffs_blkfree: freeing free s kern/125738 fs [zfs] [request] SHA256 acceleration in ZFS o kern/123939 fs [msdosfs] corrupts new files o kern/122380 fs [ffs] ffs_valloc:dup alloc (Soekris 4801/7.0/USB Flash o bin/122172 fs [fs]: amd(8) automount daemon dies on 6.3-STABLE i386, o bin/121898 fs [nullfs] pwd(1)/getcwd(2) fails with Permission denied o bin/121072 fs [smbfs] mount_smbfs(8) cannot normally convert the cha o kern/120483 fs [ntfs] [patch] NTFS filesystem locking changes o kern/120482 fs [ntfs] [patch] Sync style changes between NetBSD and F o kern/118912 fs [2tb] disk sizing/geometry problem with large array o kern/118713 fs [minidump] [patch] Display media size required for a k o kern/118318 fs [nfs] NFS server hangs under special circumstances o bin/118249 fs [ufs] mv(1): moving a directory changes its mtime o kern/118126 fs [nfs] [patch] Poor NFS server write performance o kern/118107 fs [ntfs] [panic] Kernel panic when accessing a file at N o kern/117954 fs [ufs] dirhash on very large directories blocks the mac o bin/117315 fs [smbfs] mount_smbfs(8) and related options can't mount o kern/117158 fs [zfs] zpool scrub causes panic if geli vdevs detach on o bin/116980 fs [msdosfs] [patch] mount_msdosfs(8) resets some flags f o conf/116931 fs lack of fsck_cd9660 prevents mounting iso images with o kern/116583 fs [ffs] [hang] System freezes for short time when using o bin/115361 fs [zfs] mount(8) gets into a state where it won't set/un o kern/114955 fs [cd9660] [patch] [request] support for mask,dirmask,ui o kern/114847 fs [ntfs] [patch] [request] dirmask support for NTFS ala o kern/114676 fs [ufs] snapshot creation panics: snapacct_ufs2: bad blo o bin/114468 fs [patch] [request] add -d option to umount(8) to detach o kern/113852 fs [smbfs] smbfs does not properly implement DFS referral o bin/113838 fs [patch] [request] mount(8): add support for relative p o bin/113049 fs [patch] [request] make quot(8) use getopt(3) and show o kern/112658 fs [smbfs] [patch] smbfs and caching problems (resolves b o kern/111843 fs [msdosfs] Long Names of files are incorrectly created o kern/111782 fs [ufs] dump(8) fails horribly for large filesystems s bin/111146 fs [2tb] fsck(8) fails on 6T filesystem o bin/107829 fs [2TB] fdisk(8): invalid boundary checking in fdisk / w o kern/106107 fs [ufs] left-over fsck_snapshot after unfinished backgro o kern/104406 fs [ufs] Processes get stuck in "ufs" state under persist o kern/104133 fs [ext2fs] EXT2FS module corrupts EXT2/3 filesystems o kern/103035 fs [ntfs] Directories in NTFS mounted disc images appear o kern/101324 fs [smbfs] smbfs sometimes not case sensitive when it's s o kern/99290 fs [ntfs] mount_ntfs ignorant of cluster sizes s bin/97498 fs [request] newfs(8) has no option to clear the first 12 o kern/97377 fs [ntfs] [patch] syntax cleanup for ntfs_ihash.c o kern/95222 fs [cd9660] File sections on ISO9660 level 3 CDs ignored o kern/94849 fs [ufs] rename on UFS filesystem is not atomic o bin/94810 fs fsck(8) incorrectly reports 'file system marked clean' o kern/94769 fs [ufs] Multiple file deletions on multi-snapshotted fil o kern/94733 fs [smbfs] smbfs may cause double unlock o kern/93942 fs [vfs] [patch] panic: ufs_dirbad: bad dir (patch from D o kern/92272 fs [ffs] [hang] Filling a filesystem while creating a sna o kern/91134 fs [smbfs] [patch] Preserve access and modification time a kern/90815 fs [smbfs] [patch] SMBFS with character conversions somet o kern/88657 fs [smbfs] windows client hang when browsing a samba shar o kern/88555 fs [panic] ffs_blkfree: freeing free frag on AMD 64 o bin/87966 fs [patch] newfs(8): introduce -A flag for newfs to enabl o kern/87859 fs [smbfs] System reboot while umount smbfs. o kern/86587 fs [msdosfs] rm -r /PATH fails with lots of small files o bin/85494 fs fsck_ffs: unchecked use of cg_inosused macro etc. o kern/80088 fs [smbfs] Incorrect file time setting on NTFS mounted vi o bin/74779 fs Background-fsck checks one filesystem twice and omits o kern/73484 fs [ntfs] Kernel panic when doing `ls` from the client si o bin/73019 fs [ufs] fsck_ufs(8) cannot alloc 607016868 bytes for ino o kern/71774 fs [ntfs] NTFS cannot "see" files on a WinXP filesystem o bin/70600 fs fsck(8) throws files away when it can't grow lost+foun o kern/68978 fs [panic] [ufs] crashes with failing hard disk, loose po o kern/65920 fs [nwfs] Mounted Netware filesystem behaves strange o kern/65901 fs [smbfs] [patch] smbfs fails fsx write/truncate-down/tr o kern/61503 fs [smbfs] mount_smbfs does not work as non-root o kern/55617 fs [smbfs] Accessing an nsmb-mounted drive via a smb expo o kern/51685 fs [hang] Unbounded inode allocation causes kernel to loc o kern/36566 fs [smbfs] System reboot with dead smb mount and umount o bin/27687 fs fsck(8) wrapper is not properly passing options to fsc o kern/18874 fs [2TB] 32bit NFS servers export wrong negative values t 301 problems total. From owner-freebsd-fs@FreeBSD.ORG Wed Mar 27 10:14:59 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 9A33C7B7 for ; Wed, 27 Mar 2013 10:14:59 +0000 (UTC) (envelope-from tevans.uk@googlemail.com) Received: from mail-la0-x22a.google.com (mail-la0-x22a.google.com [IPv6:2a00:1450:4010:c03::22a]) by mx1.freebsd.org (Postfix) with ESMTP id 2B68BB60 for ; Wed, 27 Mar 2013 10:14:58 +0000 (UTC) Received: by mail-la0-f42.google.com with SMTP id fe20so15363644lab.1 for ; Wed, 27 Mar 2013 03:14:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=U4qFRvKLj8SnMwpZ6DS5sXZrMKtR53ICFKmmkFYSkww=; b=rLrX9fMoVmAtcrNjsX6Md0ixUa0GXAFWcoRxQ8C4qnRBr9lHfDUVortKlyCPwyaEXg OBBvUg1mACAymr8fopbXh4JtybDyJAg0UXBQI2UU6oXwvndVsGYJU3BH1KHEItCL5Fmk wXbVrpDssTNWCE5bEtxFNnRG27IMplxc+pPsYx61Ub7UuxUgn6ExdnI7hk1iiZLd29Dq h+2/7NhkgaDmJzxQ3iCCphtSOKs04aL6Z/M0F+iyZ61O1WvNnrX9q4lBInWvKlvTYeBO Qz7ZOp+losI+BJcZuDfBYYxcKHVkLga0fqyLnECY69q+uGC0qE1bGRW9ZMLsbVOEmTp5 tyeA== MIME-Version: 1.0 X-Received: by 10.152.133.52 with SMTP id oz20mr10057682lab.30.1364379298105; Wed, 27 Mar 2013 03:14:58 -0700 (PDT) Received: by 10.112.26.135 with HTTP; Wed, 27 Mar 2013 03:14:57 -0700 (PDT) In-Reply-To: <20130322172657.95912@relay.ibs.dn.ua> References: <20130322172657.95912@relay.ibs.dn.ua> Date: Wed, 27 Mar 2013 10:14:57 +0000 Message-ID: Subject: Re: RAM amount recommendations for ZFS pools with ZIL and L2ARC on SSD From: Tom Evans To: Zeus Panchenko Content-Type: text/plain; charset=UTF-8 Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Mar 2013 10:14:59 -0000 On Fri, Mar 22, 2013 at 2:26 PM, Zeus Panchenko wrote: > hi all, > > while discovering the subj, I found recommendations at: > http://doc.freenas.org/index.php/Hardware_Recommendations#RAM > > - --------------------------------------------------------------------- > ... a general rule of thumb is 1 GB of RAM for every 1TB of storage ... > If you plan to use ZFS deduplication, a general rule of thumb is 5 GB > RAM per TB of storage to be deduplicated. > - --------------------------------------------------------------------- > > > so, are these recommendations correct for ZFS pools with ZIL and L2ARC > on SSD configurations? > > are there corellations between "RAM amount" and "with/without ZIL, L2ARC > on separate devices pool configuration" ? > Well, they wouldn't hurt. Personally, I've used much less than that - I have a 12 x 1.5 TB server with 8 GB RAM, which is only 0.5GB/TB. It's replacement will have significantly more, but only because RAM is cheap(er). Dedupe is a special beast, to get decent performance the whole dedupe table must fit into memory, Oracle have a sizing guide: http://www.oracle.com/technetwork/articles/servers-storage-admin/o11-113-size-zfs-dedup-1354231.html If you don't have a ZIL, a portion of your pool is used instead as ZIL. RAM is never used for ZIL. Remember that the ZIL is only for synchronous writes, which most writes are not. RAM is mainly used for the ARC cache, and RAM is a lot faster than an SSD, so the more RAM used for the ARC cache the better. Adding an SSD L2ARC in addition will increase the amount of things that can be kept cached in a (relatively) slow cache. If your problem is that you need that data in a faster cache, then you need more RAM. I think I am right in saying that only things that have been read can be stored in ARC, so how effective it is would depend upon your workload - how much read things do you need in the cache? Cheers Tom From owner-freebsd-fs@FreeBSD.ORG Wed Mar 27 10:38:12 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 1A032F95 for ; Wed, 27 Mar 2013 10:38:12 +0000 (UTC) (envelope-from velcroleaf@rocketmail.com) Received: from nm18-vm6.bullet.mail.gq1.yahoo.com (nm18-vm6.bullet.mail.gq1.yahoo.com [98.136.217.221]) by mx1.freebsd.org (Postfix) with SMTP id CBFC3DC7 for ; Wed, 27 Mar 2013 10:38:11 +0000 (UTC) Received: from [98.137.12.190] by nm18.bullet.mail.gq1.yahoo.com with NNFMP; 27 Mar 2013 10:35:46 -0000 Received: from [98.137.12.210] by tm11.bullet.mail.gq1.yahoo.com with NNFMP; 27 Mar 2013 10:35:46 -0000 Received: from [127.0.0.1] by omp1018.mail.gq1.yahoo.com with NNFMP; 27 Mar 2013 10:35:46 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 795023.87255.bm@omp1018.mail.gq1.yahoo.com Received: (qmail 75391 invoked by uid 60001); 27 Mar 2013 10:35:46 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rocketmail.com; s=s1024; t=1364380546; bh=1unV+TqNDz4Qg32CkZBVpvFwDuMZRqcq94RudtqP1s0=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Reply-To:Subject:To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=KzcbiY0uRBvUpif3164AqRtuVKyW8o1MFGy/Fuq94pM0fdKaFqh/ljB0NzORuMWx3wiHAsTiJDdDoeebO/clawuO+cEAkM0X7SMn7gGEMdSPsIuJDycj1YL3E7ZhBjDX97jMgnNaVvDlXCwn0+l9ZzJ0OCqMpazYzYozkYrNIlI= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=rocketmail.com; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Reply-To:Subject:To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=FaSiHbKuix25q9lKEhPN+kiUQsItkXisqbwQYZ3oD4sSXOMQaTpcNREhzTbkm4kR2uO7GIy9fK4vx2Jh6wgMMXmYrklv1gA5FpWZmwTkOBN+jBqc99d919uyVsHDP/Wxh7//15VHfxFJBLCUlR3eWev0qEUhvXeTFTsI39CsqtQ=; X-YMail-OSG: BBpf89AVM1k_0zj3.ADJl.oMpEWJpR75iBILBBryvXQ7Jhi ZmVL1_PBvpGJ4OcmltS_YbLZfzoI3ioRWjOgHy9Za5vyIpY77uNFU28fMdUh TULNgihThhE.u8VQ1FI8W3IUj7CxPh5mBAm8L1tIGee5jf5kXPRZVP2zdoB. H4brZ2VgnSucVMc7wCMcrQcdtH9aXqVuJbYkJmtkvqm_8mvTQ.pzBLwUJ7n_ Nwwd3YfoEelMarjjWilQ1hthUHoUDBsPmKZQuAsqFT67mfkkdlnAzJmQ4Zef wQnGgHw5JbrOEQnSSv7Q4kmDnqn5aHRWFbGyJEod6zOUQ67qmgtRPnNHpfdq WXN9S_FAr6hdOl0_csRV_u.Rep8LXmZ8wsyPQOGXJaTLb0FshzQteLbPyXle VD.4Z5ZQiL2.2fsz859xMeS7ZYbiVnXmddt0FtTctQRKArsLuabL5_UiI8J3 CeAXSVW4UQocQ.t5SV_2enovaMA-- Received: from [74.229.13.174] by web164504.mail.gq1.yahoo.com via HTTP; Wed, 27 Mar 2013 03:35:46 PDT X-Rocket-MIMEInfo: 002.001, U29tZXdoZXJlIGJldHdlZW4gOS4wLVJFTEVBU0UgYW5kIDkuMS1SRUxFQVNFLCBzb21ldGhpbmcgY2hhbmdlZCB0aGF0IAptYWtlcyBpdCBzbyBmaWxlc3lzdGVtcyBvbiB0d28gb2YgbXkgc2VydmVycyB3aXRoIHNpbWlsYXIgaGFyZHdhcmUgZG9uJ3QgY2xlYW5seSBzaHV0IGRvd24gZHVyaW5nIHNvZnQgcmVib290cy4KCkkgb25seSBzYXcgYSBjb3VwbGUgCmV4YW1wbGVzIG9mIHN1Y2Nlc3NmdWwgcmVib290aW5nIG9uIDkuMCB3aGlsZSB0cmFuc2l0aW9uaW5nIGZyb20gYW4gCmVhcmxpZXIgdmVyc2lvbiABMAEBAQE- X-Mailer: YahooMailWebService/0.8.139.530 Message-ID: <1364380546.45116.YahooMailNeo@web164504.mail.gq1.yahoo.com> Date: Wed, 27 Mar 2013 03:35:46 -0700 (PDT) From: Velcro Leaf Subject: filesystems don't properly dismount To: "freebsd-fs@freebsd.org" MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Velcro Leaf List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Mar 2013 10:38:12 -0000 Somewhere between 9.0-RELEASE and 9.1-RELEASE, something changed that =0Ama= kes it so filesystems on two of my servers with similar hardware don't clea= nly shut down during soft reboots.=0A=0AI only saw a couple =0Aexamples of = successful rebooting on 9.0 while transitioning from an =0Aearlier version = to 9.1, but filesystems consistently complain during =0Aboot-up that they w= ere not dismounted properly during every reboot in =0A9.1.=0A=0ABased on th= e release notes, it seems like it might have =0Asomething to do with the ne= w style of NFS.=A0 Is the old style still =0Alatent in 9.1?=A0 For example,= can you recompile the kernel with different=0A options to make the old sty= le of NFS the default?=A0 Am I barking up the =0Awrong tree?=A0 Any suggest= ions?=0A=0AThese are on mirrored RAID drives.=A0=0A If you need other hardw= are details, please explain specifically how to =0Acollect the information.= =A0 I'm no hardware expert.=A0 Thanks!=0A From owner-freebsd-fs@FreeBSD.ORG Wed Mar 27 12:06:03 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 5153C5F7 for ; Wed, 27 Mar 2013 12:06:03 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) by mx1.freebsd.org (Postfix) with ESMTP id 13B2D5E7 for ; Wed, 27 Mar 2013 12:06:02 +0000 (UTC) Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id D44A52842A; Wed, 27 Mar 2013 13:05:55 +0100 (CET) Received: from [192.168.1.2] (ip-89-177-49-222.net.upcbroadband.cz [89.177.49.222]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 946A728422; Wed, 27 Mar 2013 13:05:54 +0100 (CET) Message-ID: <5152E0A3.1090507@quip.cz> Date: Wed, 27 Mar 2013 13:05:55 +0100 From: Miroslav Lachman <000.fbsd@quip.cz> User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.9.1.19) Gecko/20110420 Lightning/1.0b1 SeaMonkey/2.0.14 MIME-Version: 1.0 To: Zeus Panchenko Subject: Re: RAM amount recommendations for ZFS pools with ZIL and L2ARC on SSD References: <20130322172657.95912@relay.ibs.dn.ua> In-Reply-To: <20130322172657.95912@relay.ibs.dn.ua> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Mar 2013 12:06:03 -0000 Zeus Panchenko wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > hi all, > > while discovering the subj, I found recommendations at: > http://doc.freenas.org/index.php/Hardware_Recommendations#RAM > > - --------------------------------------------------------------------- > ... a general rule of thumb is 1 GB of RAM for every 1TB of storage ... > If you plan to use ZFS deduplication, a general rule of thumb is 5 GB > RAM per TB of storage to be deduplicated. > - --------------------------------------------------------------------- > > > so, are these recommendations correct for ZFS pools with ZIL and L2ARC > on SSD configurations? > > are there corellations between "RAM amount" and "with/without ZIL, L2ARC > on separate devices pool configuration" ? The bigger L2ARC you have, the more RAM you need to store metadata about "what is in L2ARC". So if you have some system without L2ARC and you will add SSD as L2ARC, then you will need more RAM than before. It depends on records size and number of records stored in L2ARC. You can take a look at this thread http://www.mail-archive.com/zfs-discuss@opensolaris.org/msg34674.html http://www.mail-archive.com/zfs-discuss@opensolaris.org/msg34677.html Miroslav Lachman From owner-freebsd-fs@FreeBSD.ORG Wed Mar 27 14:29:29 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 41453A66 for ; Wed, 27 Mar 2013 14:29:29 +0000 (UTC) (envelope-from zeus@ibs.dn.ua) Received: from relay.ibs.dn.ua (relay.ibs.dn.ua [91.216.196.25]) by mx1.freebsd.org (Postfix) with ESMTP id B1DC02B5 for ; Wed, 27 Mar 2013 14:29:27 +0000 (UTC) Received: from ibs.dn.ua (relay.ibs.dn.ua [91.216.196.25]) by relay.ibs.dn.ua with ESMTP id r2RETEiM085826; Wed, 27 Mar 2013 16:29:15 +0200 (EET) Message-ID: <20130327162914.85824@relay.ibs.dn.ua> Date: Wed, 27 Mar 2013 16:29:14 +0300 From: Zeus Panchenko To: "Miroslav Lachman" <000.fbsd@quip.cz> Subject: Re: RAM amount recommendations for ZFS pools with ZIL and L2ARC on SSD In-reply-to: Your message of Wed, 27 Mar 2013 13:05:55 +0100 <5152E0A3.1090507@quip.cz> References: <20130322172657.95912@relay.ibs.dn.ua> <5152E0A3.1090507@quip.cz> Organization: I.B.S. LLC X-Mailer: MH-E 8.3.1; GNU Mailutils 2.99.97; GNU Emacs 24.0.93 X-Face: &sReWXo3Iwtqql1[My(t1Gkx; y?KF@KF`4X+'9Cs@PtK^y%}^.>Mtbpyz6U=,Op:KPOT.uG )Nvx`=er!l?WASh7KeaGhga"1[&yz$_7ir'cVp7o%CGbJ/V)j/=]vzvvcqcZkf; JDurQG6wTg+?/xA go`}1.Ze//K; Fk&/&OoHd'[b7iGt2UO>o(YskCT[_D)kh4!yY'<&:yt+zM=A`@`~9U+P[qS:f; #9z~ Or/Bo#N-'S'!'[3Wog'ADkyMqmGDvga?WW)qd=?)`Y&k=o}>!ST\ Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Zeus Panchenko List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Mar 2013 14:29:29 -0000 Miroslav Lachman <000.fbsd@quip.cz> wrote: > The bigger L2ARC you have, the more RAM you need to store metadata > ... > You can take a look at this thread > > http://www.mail-archive.com/zfs-discuss@opensolaris.org/msg34674.html > http://www.mail-archive.com/zfs-discuss@opensolaris.org/msg34677.html thank you very much, it is just what I wanted to know -- Zeus V. Panchenko jid:zeus@im.ibs.dn.ua IT Dpt., I.B.S. LLC GMT+2 (EET) From owner-freebsd-fs@FreeBSD.ORG Wed Mar 27 16:48:21 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 84F9A777 for ; Wed, 27 Mar 2013 16:48:21 +0000 (UTC) (envelope-from kaduk@mit.edu) Received: from dmz-mailsec-scanner-2.mit.edu (DMZ-MAILSEC-SCANNER-2.MIT.EDU [18.9.25.13]) by mx1.freebsd.org (Postfix) with ESMTP id 2E794D58 for ; Wed, 27 Mar 2013 16:48:20 +0000 (UTC) X-AuditID: 1209190d-b7fa66d0000008f6-4d-515321a2bc99 Received: from mailhub-auth-2.mit.edu ( [18.7.62.36]) by dmz-mailsec-scanner-2.mit.edu (Symantec Messaging Gateway) with SMTP id 71.F3.02294.2A123515; Wed, 27 Mar 2013 12:43:14 -0400 (EDT) Received: from outgoing.mit.edu (OUTGOING-AUTH-1.MIT.EDU [18.9.28.11]) by mailhub-auth-2.mit.edu (8.13.8/8.9.2) with ESMTP id r2RGhDw3023400; Wed, 27 Mar 2013 12:43:14 -0400 Received: from multics.mit.edu (SYSTEM-LOW-SIPB.MIT.EDU [18.187.2.37]) (authenticated bits=56) (User authenticated as kaduk@ATHENA.MIT.EDU) by outgoing.mit.edu (8.13.8/8.12.4) with ESMTP id r2RGhBX3019833 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Wed, 27 Mar 2013 12:43:13 -0400 Received: (from kaduk@localhost) by multics.mit.edu (8.12.9.20060308) id r2RGhAO6014533; Wed, 27 Mar 2013 12:43:10 -0400 (EDT) Date: Wed, 27 Mar 2013 12:43:10 -0400 (EDT) From: Benjamin Kaduk To: Velcro Leaf Subject: Re: filesystems don't properly dismount In-Reply-To: <1364380546.45116.YahooMailNeo@web164504.mail.gq1.yahoo.com> Message-ID: References: <1364380546.45116.YahooMailNeo@web164504.mail.gq1.yahoo.com> User-Agent: Alpine 1.10 (GSO 962 2008-03-14) MIME-Version: 1.0 Content-Type: MULTIPART/MIXED; BOUNDARY="-559023410-1970454349-1364402590=:9389" X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFmphleLIzCtJLcpLzFFi42IRYrdT0V2kGBxo8HaapMWxxz/ZLH5e+sPi wOQx49N8Fo+Hu9pYApiiuGxSUnMyy1KL9O0SuDLWXmtkKrjJXbHy2Tq2BsYznF2MnBwSAiYS Vx62skLYYhIX7q1n62Lk4hAS2Mco8ePPAWYIZyOjxJ2Ha9ghnENMEn+Xf2AEaRESaGCUaO3k A7FZBLQl3kxayQRiswmoSMx8s5ENxBYBip9ZehWsnlnAVGLXgk/MILawgIHEulldYPWcAp4S l89fZAexeQUcJNY+/wUU5wCa7yEx8a4QSFhUQEdi9f4pLBAlghInZz5hgRgZINH0+SnzBEbB WUhSs5CkIGxriQ1vpzJB2NoS92+2sS1gZFnFKJuSW6Wbm5iZU5yarFucnJiXl1qka6SXm1mi l5pSuokRFNickrw7GN8dVDrEKMDBqMTD2yAeHCjEmlhWXJl7iFGSg0lJlHeSPFCILyk/pTIj sTgjvqg0J7X4EKMEB7OSCK+FKFCONyWxsiq1KB8mJc3BoiTOeyXlpr+QQHpiSWp2ampBahFM VoaDQ0mC10oBqFGwKDU9tSItM6cEIc3EwQkynAdoeDxIDW9xQWJucWY6RP4Uo6KUOK8ASEIA JJFRmgfXC0s8rxjFgV4R5o0CqeIBJi247ldAg5mABk/96w8yuCQRISXVwJgZ3+5iyZ2wWHfO jrccCy/2x7ZLLnHMXPlqt7WJHtvflJyHWz/YmSzS/pi7Z9aRkz4/zJl65y8/W8i50dDGXH2W rTb3tlK1xYJ9F9V03kRXr1wpqql2VnyecpTV/sty8kuOzfc8o2wmEvcnZnfM4X+7ayRLVkRf MMtpaV0c5PcyeedPHT4ndiWW4oxEQy3mouJEAHw/9twXAwAA Cc: "freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Mar 2013 16:48:21 -0000 This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. ---559023410-1970454349-1364402590=:9389 Content-Type: TEXT/PLAIN; charset=iso-8859-1; format=flowed Content-Transfer-Encoding: QUOTED-PRINTABLE On Wed, 27 Mar 2013, Velcro Leaf wrote: > Somewhere between 9.0-RELEASE and 9.1-RELEASE, something changed that > makes it so filesystems on two of my servers with similar hardware don't = cleanly shut down during soft reboots. > > I only saw a couple > examples of successful rebooting on 9.0 while transitioning from an > earlier version to 9.1, but filesystems consistently complain during > boot-up that they were not dismounted properly during every reboot in > 9.1. > > Based on the release notes, it seems like it might have > something to do with the new style of NFS.=A0 Is the old style still > latent in 9.1?=A0 For example, can you recompile the kernel with differen= t > options to make the old style of NFS the default?=A0 Am I barking up the > wrong tree?=A0 Any suggestions? The default NFS implementation did not change between 9.0 and 9.1. Per the 9.0 release notes, you can change your kernel configuration to=20 control which implementation is used -- NFSCL/NFSD are the "new" versions,= =20 and NFSCLIENT/NFSSERVER are the old ones. Are you running GENERIC or a=20 custom configuration? -Ben Kaduk ---559023410-1970454349-1364402590=:9389-- From owner-freebsd-fs@FreeBSD.ORG Wed Mar 27 17:02:54 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 4A36DD37 for ; Wed, 27 Mar 2013 17:02:54 +0000 (UTC) (envelope-from velcroleaf@rocketmail.com) Received: from nm6-vm8.bullet.mail.gq1.yahoo.com (nm6-vm8.bullet.mail.gq1.yahoo.com [98.136.218.199]) by mx1.freebsd.org (Postfix) with SMTP id 11C60E31 for ; Wed, 27 Mar 2013 17:02:53 +0000 (UTC) Received: from [98.137.12.58] by nm6.bullet.mail.gq1.yahoo.com with NNFMP; 27 Mar 2013 17:00:42 -0000 Received: from [98.137.12.210] by tm3.bullet.mail.gq1.yahoo.com with NNFMP; 27 Mar 2013 17:00:42 -0000 Received: from [127.0.0.1] by omp1018.mail.gq1.yahoo.com with NNFMP; 27 Mar 2013 17:00:42 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 302472.88502.bm@omp1018.mail.gq1.yahoo.com Received: (qmail 47021 invoked by uid 60001); 27 Mar 2013 17:00:42 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rocketmail.com; s=s1024; t=1364403642; bh=ES8EjKXnUYSd+wRsgh/DN+jX94KprURtvNZkU/UFrU4=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=C8zGkXIZ7/kVWmCZZRHyU9yRPalxtW1JJs3JWeTaEQams8bwy7TeiTFNENA+3dI+Saizf0Dli+8XsqwmQ2daCuQg8C6UIFOMLyvNea3GHOAeJz9X7xm3aLXLyeFhGY/J1ZD04RdzyJakHV2jnbV0ULizywCVOpn+LiZ2jhagm0Y= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=rocketmail.com; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=PstK2prL8QzOdJvIIT6fzUBdhYYk6UVzH320F37/Jsaye6DCG/+8t9V9IF4BrerzsfjoQEl4ZRYXi9Fy+TwZb3/D7X/+IicvB7DNKC/c4jslVTHm9WXz4FWakeXQWTnMCzwWv4pXtA7gWGw9HRzWrA0wOi/sGl8t+0p6Q3smF0M=; X-YMail-OSG: 0jej8IEVM1l11fql1m2rn1b0y3DBzxopfbYtagsw0EEmNBX KYpETpATo_sJDdmz5TFSrlSt099KWwCWmHP1NtDmsJYfw1MjyW7wh0FhdfpO .6nnKFsP1DKD4O1A5C0O8mraUPd5spkB3Bk_oBZJHILqKFhZxBNH8V2MGSnS y9p8uK_ctDeVvQPfDAl.hHyiw2u33x80NjHZyMTYo4ZZ2Ztq70NPBKkhWjTl bj4vSiJV7uqLcZ1TdNPDQ.gUeuw70a_vdHYJDIO9J1CdIMsYNQBn6Ue5bX5S 32UqPQLlyGFQe_9V_d1PmVaDHHCCiCoIO20GMBGxgfEhC9oB3ssc.wQkYUWG RWwygPxvLSwp4vbXi4iguerFEIiZq5Ww0Fa6Htb0UNQ.N3XDtrRY_XXjXf9e 6kR6T7qRkQfEANwHfyWpvQ9GsXYCBzXwxeBwp0e9.gXlRwm94X_tRstMV1TT uBCQag.NWyE8szQeQTeCpJd4zGQ-- Received: from [74.229.13.174] by web164501.mail.gq1.yahoo.com via HTTP; Wed, 27 Mar 2013 10:00:42 PDT X-Rocket-MIMEInfo: 002.001, QmVuamFtaW4gS2FkdWsgd3JvdGU6Cgo.IFRoZSBkZWZhdWx0IE5GUyBpbXBsZW1lbnRhdGlvbiBkaWQgbm90IGNoYW5nZSBiZXR3ZWVuIDkuMCBhbmQgOS4xLgo.IFBlciB0aGUgOS4wIHJlbGVhc2Ugbm90ZXMsIHlvdSBjYW4gY2hhbmdlIHlvdXIga2VybmVsIGNvbmZpZ3VyYXRpb24gdG8gY29udHJvbCAKPiB3aGljaCBpbXBsZW1lbnRhdGlvbiBpcyB1c2VkIC0tIE5GU0NML05GU0QgYXJlIHRoZSAibmV3IiB2ZXJzaW9ucywgYW5kIAo.IE5GU0NMSUVOVC9ORlNTRVJWRVIgYXJlIHRoZSBvbGQgb25lcy7CoCABMAEBAQE- X-Mailer: YahooMailWebService/0.8.139.530 References: <1364380546.45116.YahooMailNeo@web164504.mail.gq1.yahoo.com> Message-ID: <1364403642.40164.YahooMailNeo@web164501.mail.gq1.yahoo.com> Date: Wed, 27 Mar 2013 10:00:42 -0700 (PDT) From: Velcro Leaf Subject: Re: filesystems don't properly dismount To: Benjamin Kaduk In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Cc: "freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Velcro Leaf List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Mar 2013 17:02:54 -0000 Benjamin Kaduk wrote:=0A=0A> The default NFS implementation did not change = between 9.0 and 9.1.=0A> Per the 9.0 release notes, you can change your ker= nel configuration to control =0A> which implementation is used -- NFSCL/NFS= D are the "new" versions, and =0A> NFSCLIENT/NFSSERVER are the old ones.=A0= Are you running GENERIC or a custom =0A> configuration?=0A=0AIt's a GENERI= C kernel, and I'm not actually using any networked filesystems.=A0 I saw th= at the NFS stuff happened prior to 9.0-RELEASE, but I'm looking for any sim= ple way to change the way filesystems are handled to see if it'll stop the = ugly dismounts.=0A=0AApparently the sparc64 ZFS loader has been changed, bu= t A) I honestly don't know what that means, and B) I'm not running a sparc6= 4 machine.=A0 Both malfunctioning servers are i386.=A0 I haven't updated th= e amd64 one yet, and am a little afraid to, even though 9.0 is about to go = EOL.=0A=0AI didn't see anything else in the changelog that would translate = into something simple I can do that might solve the problem.=0A=0AI'll make= a custom kernel with only the above changes and see if anything different = happens.=0A From owner-freebsd-fs@FreeBSD.ORG Wed Mar 27 18:41:42 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 6AB1F962 for ; Wed, 27 Mar 2013 18:41:42 +0000 (UTC) (envelope-from kaduk@mit.edu) Received: from dmz-mailsec-scanner-8.mit.edu (DMZ-MAILSEC-SCANNER-8.MIT.EDU [18.7.68.37]) by mx1.freebsd.org (Postfix) with ESMTP id 24D0A69C for ; Wed, 27 Mar 2013 18:41:41 +0000 (UTC) X-AuditID: 12074425-b7fec6d000007584-c2-51533c3381a1 Received: from mailhub-auth-2.mit.edu ( [18.7.62.36]) by dmz-mailsec-scanner-8.mit.edu (Symantec Messaging Gateway) with SMTP id 22.B6.30084.33C33515; Wed, 27 Mar 2013 14:36:35 -0400 (EDT) Received: from outgoing.mit.edu (OUTGOING-AUTH-1.MIT.EDU [18.9.28.11]) by mailhub-auth-2.mit.edu (8.13.8/8.9.2) with ESMTP id r2RIaY1G007610; Wed, 27 Mar 2013 14:36:35 -0400 Received: from multics.mit.edu (SYSTEM-LOW-SIPB.MIT.EDU [18.187.2.37]) (authenticated bits=56) (User authenticated as kaduk@ATHENA.MIT.EDU) by outgoing.mit.edu (8.13.8/8.12.4) with ESMTP id r2RIaWP8014004 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Wed, 27 Mar 2013 14:36:34 -0400 Received: (from kaduk@localhost) by multics.mit.edu (8.12.9.20060308) id r2RIaW91029204; Wed, 27 Mar 2013 14:36:32 -0400 (EDT) Date: Wed, 27 Mar 2013 14:36:32 -0400 (EDT) From: Benjamin Kaduk To: Velcro Leaf Subject: Re: filesystems don't properly dismount In-Reply-To: <1364403642.40164.YahooMailNeo@web164501.mail.gq1.yahoo.com> Message-ID: References: <1364380546.45116.YahooMailNeo@web164504.mail.gq1.yahoo.com> <1364403642.40164.YahooMailNeo@web164501.mail.gq1.yahoo.com> User-Agent: Alpine 1.10 (GSO 962 2008-03-14) MIME-Version: 1.0 Content-Type: MULTIPART/MIXED; BOUNDARY="-559023410-1969653652-1364409309=:9389" Content-ID: X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFlrFKsWRmVeSWpSXmKPExsUixG6nomtsExxo8PmclMWxxz/ZLH5e+sPi wOQx49N8Fo+Hu9pYApiiuGxSUnMyy1KL9O0SuDKmrg4umMtbMWviKZYGxpdcXYycHBICJhIT uveyQNhiEhfurWfrYuTiEBLYxyhx5sFDZghnI6PEpAlLoTKHmCTWP/vGAuE0MEp8mvwUrJ9F QFui9/FtRhCbTUBFYuabjWwgtghQ/MzSq2BxZgFTiV0LPjGD2MICBhLrZnUxgdicAp4SXVsu gtXwCjhITJx+CGrbLkaJvq6fYINEBXQkVu+fwgJRJChxcuYTFoihARKLdkyFWuAgceT/E5YJ jEKzkJTNQlI2C0kZhG0jcXfTW6i4tsT9m21sMDWnbneyLmBkW8Uom5JbpZubmJlTnJqsW5yc mJeXWqRroZebWaKXmlK6iREcJS6qOxgnHFI6xCjAwajEw9sgHhwoxJpYVlyZe4hRkoNJSZR3 lhVQiC8pP6UyI7E4I76oNCe1+BCjBAezkgivtTJQjjclsbIqtSgfJiXNwaIkznsj5aa/kEB6 YklqdmpqQWoRTFaGg0NJgpfTGqhRsCg1PbUiLTOnBCHNxMEJMpwHaPgGkMW8xQWJucWZ6RD5 U4yKUuK8AiDNAiCJjNI8uF5YEnvFKA70ijCvPEgVDzABwnW/AhrMBDR46l9/kMEliQgpqQZG odYc6Rqz19fFTvF9rxe3Lr/PuEXWacfLZ1M455xie2l0UN+npUWs8AvH536vo9M37debVs/R wyJ6+tyTb1M02vmeLPnrebLbvWFv8qV1Dz50/mIu3rG6/fOOreX/7GuXa6ScXmT07TZLXOMT pXvlCpunp0k5+AiGdxj9SFhU8vkHeyePytTXSizFGYmGWsxFxYkASHx+7D0DAAA= Cc: "freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Mar 2013 18:41:42 -0000 This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. ---559023410-1969653652-1364409309=:9389 Content-Type: TEXT/PLAIN; CHARSET=ISO-8859-15; FORMAT=flowed Content-Transfer-Encoding: QUOTED-PRINTABLE Content-ID: On Wed, 27 Mar 2013, Velcro Leaf wrote: > Benjamin Kaduk wrote: > >> The default NFS implementation did not change between 9.0 and 9.1. >> Per the 9.0 release notes, you can change your kernel configuration to c= ontrol=20 >> which implementation is used -- NFSCL/NFSD are the "new" versions, and= =20 >> NFSCLIENT/NFSSERVER are the old ones.=A0 Are you running GENERIC or a cu= stom=20 >> configuration? > > It's a GENERIC kernel, and I'm not actually using any networked=20 > filesystems.=A0 I saw that the NFS stuff happened prior to 9.0-RELEASE,= =20 > but I'm looking for any simple way to change the way filesystems are=20 > handled to see if it'll stop the ugly dismounts. > > I'll make a custom kernel with only the above changes and see if=20 > anything different happens. > If you don't have any networked filesystems mounted, then I would be very= =20 surprised if changing the version of NFS built into your kernel would have= =20 any effect here. At this point, I think the most useful information would be the kernel=20 messages from shutdown time, printed on the console. Do you have a serial= =20 console (makes this easy)? Otherwise, you may have to take a picture or=20 video of the machine as it shuts down. -Ben Kaduk ---559023410-1969653652-1364409309=:9389-- From owner-freebsd-fs@FreeBSD.ORG Thu Mar 28 00:36:44 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id B61DB963 for ; Thu, 28 Mar 2013 00:36:44 +0000 (UTC) (envelope-from velcroleaf@rocketmail.com) Received: from nm16-vm2.bullet.mail.gq1.yahoo.com (nm16-vm2.bullet.mail.gq1.yahoo.com [98.137.177.250]) by mx1.freebsd.org (Postfix) with SMTP id 7E1A9CC7 for ; Thu, 28 Mar 2013 00:36:44 +0000 (UTC) Received: from [98.137.12.62] by nm16.bullet.mail.gq1.yahoo.com with NNFMP; 28 Mar 2013 00:34:34 -0000 Received: from [98.137.12.224] by tm7.bullet.mail.gq1.yahoo.com with NNFMP; 28 Mar 2013 00:34:34 -0000 Received: from [127.0.0.1] by omp1032.mail.gq1.yahoo.com with NNFMP; 28 Mar 2013 00:34:34 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 375515.80036.bm@omp1032.mail.gq1.yahoo.com Received: (qmail 91180 invoked by uid 60001); 28 Mar 2013 00:34:34 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rocketmail.com; s=s1024; t=1364430874; bh=46PawcI2EZQ0BbZMoREgiMe1Zc06KmWcsCO3rG4c+DU=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=TNcG9laLZ7rB2uGtUb4NkSsPtyxAFrusEvXxtrHc3Kw9QBXIvvWt8Xg7pZPyJzjrVUQnqFJ8gX4oNfBxjCbLzzoVLzTSz8xlGIMvYKEnPP3Bwnoyh14Q8/1Wu1gE+EtpeHcA9wfR979+0TCxI3Z1hnh6Bfkdz+6Ri8UD8efPqtU= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=rocketmail.com; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=3W9aboHH6Ls8i9LioQiatZY6dqY3stCfOshlIV7Ih4/x8r8G0hlrYuN67ovkORMnes0M5vL4Bao/XhI9mUjyl3Y8RAHAjCBGZGxdpYfIoEAEP12pL7yg3d8JuTnVO2+j/qpOO24AGk0SrFiK5iAAMSHN4MbEUA7vuOg2MuBeO0s=; X-YMail-OSG: OHdFQqUVM1kMdQc.ONNwbYWCR4HHweVbiDG4gtlgIOcfHW3 QCN5qIYSD8vkCs0CTqxzHrvhLlVx5d1IVFTzVb.v9zK7aVha7_64TrMEO9f2 UdpjOq69gyDyl02_BDGHsJswgpfvadTA0KBxnBCs4291Rw1sWt7jK6js6WgK gYm3K9.y0NJxmUkZXlEepxDMrHlST7pej_mzWNLdVly1BPWJb4hciVq.f48y udaL7f0RFmC5IsXh9pDEklFGWSP0UrF6mBrZODy6CrreewcZce4gWkHz0e.H xdV_7UhZEWAvOb1QpjCMKg8tqBq2.W0OmA65AbwDEjScIVVchSD5hHc434oR v9AHy.vyReutdNCFBhtVsLSgPJiA9YmhqmO.v2ntvqDD1zI0wDvncqg_x2SN ahOuD9AXANwJTfJ8WJwzvtyu_.0T919qG_Qy4D5NhUkaVKnaZIX.BaRvW1Kh RVvqUvS1rGYl3F3rQAadqCETwBQ-- Received: from [192.70.171.82] by web164505.mail.gq1.yahoo.com via HTTP; Wed, 27 Mar 2013 17:34:34 PDT X-Rocket-MIMEInfo: 002.001, wqBCZW5qYW1pbiBLYWR1ayB3cm90ZToKCj4gQXQgdGhpcyBwb2ludCwgSSB0aGluayB0aGUgbW9zdCB1c2VmdWwgaW5mb3JtYXRpb24gd291bGQgYmUgdGhlIGtlcm5lbCBtZXNzYWdlcyAKPiBmcm9tIHNodXRkb3duIHRpbWUsIHByaW50ZWQgb24gdGhlIGNvbnNvbGUuwqAgRG8geW91IGhhdmUgYSBzZXJpYWwgY29uc29sZSAobWFrZXMgCj4gdGhpcyBlYXN5KT8gT3RoZXJ3aXNlLCB5b3UgbWF5IGhhdmUgdG8gdGFrZSBhIHBpY3R1cmUgb3IgdmlkZW8gb2YgdGhlIG1hY2hpbmUgYXMgCj4gaXQgc2h1dHMgZG8BMAEBAQE- X-Mailer: YahooMailWebService/0.8.139.530 References: <1364380546.45116.YahooMailNeo@web164504.mail.gq1.yahoo.com> <1364403642.40164.YahooMailNeo@web164501.mail.gq1.yahoo.com> Message-ID: <1364430874.88614.YahooMailNeo@web164505.mail.gq1.yahoo.com> Date: Wed, 27 Mar 2013 17:34:34 -0700 (PDT) From: Velcro Leaf Subject: Re: filesystems don't properly dismount To: Benjamin Kaduk In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Cc: "freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Velcro Leaf List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Mar 2013 00:36:44 -0000 =A0Benjamin Kaduk wrote:=0A=0A> At this point, I think the most useful info= rmation would be the kernel messages =0A> from shutdown time, printed on th= e console.=A0 Do you have a serial console (makes =0A> this easy)? Otherwis= e, you may have to take a picture or video of the machine as =0A> it shuts = down.=0A=0AThe video was a great idea.=A0 I'm glad I didn't put forth the e= ffort to dust off an old terminal, because now the two servers are quietly = and happily rebooting.=A0 This is good news, but also annoying.=0A=0AIt's p= ossible that every time I saw the system reboot sloppily, it was still in t= he process of being updated (that is, only part of the way through the free= bsd-update procedure).=0A=0AOnce I noticed the issue, I stopped rebooting u= nnecessarily, so it could be that I stopped rebooting right at the point at= which things worked themselves out.=0A=0AI've got some kernel tweaking to = do, so if the issue comes back, I'll be ready with the video.=A0 No more re= booting remotely until I either catch it in the act or give up the hunt.=0A= =0AThanks for your response, it helped me get a little farther down the roa= d.=0A From owner-freebsd-fs@FreeBSD.ORG Thu Mar 28 01:31:09 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id CF1CF2BE; Thu, 28 Mar 2013 01:31:09 +0000 (UTC) (envelope-from mdf356@gmail.com) Received: from mail-qe0-f42.google.com (mail-qe0-f42.google.com [209.85.128.42]) by mx1.freebsd.org (Postfix) with ESMTP id 85D82E7E; Thu, 28 Mar 2013 01:31:09 +0000 (UTC) Received: by mail-qe0-f42.google.com with SMTP id da11so4094046qeb.29 for ; Wed, 27 Mar 2013 18:31:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:date:message-id:subject:from:to :content-type; bh=vdSZtIZ5gTOVdSyF1yAT0BDvDa1QbESKB2quzyyRWo0=; b=iRvfR5L8lq+mPtSboXQ6tFwc4fUIK3WWR93fsT2tu9BmAGxeUXlo1P7HYW00sRBLWu H3bghK6Ckt7BwM4qiq6nm5GGgr14rEUnz9Fnxzbl7HCOGGyzt6DNqzBLOM5Ft7qJFFGy LpeaDu362inc1S5ChWfA6lW3RAiMCDUGglTrGc4iAq8uC4balLkmqYL4O2t5hYrqQu3v HSwIP4dpQf6kAmBaU3sC7oJs39apUwQSJzosbfEOvkJno2f1x3aGIvN4B1OrXy+AmM9j am42tCSXGIvnAdW9ZGr0eGBqrnjLLR5o3DKDgi+eS6G3GMG6M1JF/VTMTH1oM1RtNtJG EqWQ== MIME-Version: 1.0 X-Received: by 10.229.118.66 with SMTP id u2mr849759qcq.102.1364434263281; Wed, 27 Mar 2013 18:31:03 -0700 (PDT) Received: by 10.229.253.201 with HTTP; Wed, 27 Mar 2013 18:31:02 -0700 (PDT) Date: Wed, 27 Mar 2013 18:31:02 -0700 Message-ID: Subject: [RFC] use a shared lock for VOP_GETEXTATTR From: Matthew Fleming To: freebsd-current@freebsd.org, freebsd-fs@freebsd.org Content-Type: multipart/mixed; boundary=000e0cd5ccee951c6904d8f21bef X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Mar 2013 01:31:09 -0000 --000e0cd5ccee951c6904d8f21bef Content-Type: text/plain; charset=ISO-8859-1 VOP_GETEXTATTR is currently called with an exclusive lock, which seems like overkill for what is essentially a read operation. I had a look over the various in-tree filesystems and it didn't look like any of them will have a problem if a shared-mode lock is used for vop_getextattr. Does anyone know otherwise? Is someone using extended attributes regularly who can test this? Thanks, matthew --000e0cd5ccee951c6904d8f21bef Content-Type: application/octet-stream; name="0001-Use-a-shared-lock-for-VOP_GETEXTATTR-as-it-is-a-read.patch" Content-Disposition: attachment; filename="0001-Use-a-shared-lock-for-VOP_GETEXTATTR-as-it-is-a-read.patch" Content-Transfer-Encoding: base64 X-Attachment-Id: f_het8w2zm0 RnJvbSBhYTZkNmM3MTZiZDE5Y2UzMmUzYjkwODcyN2ZiZGE2NWQzNDViN2FiIE1vbiBTZXAgMTcg MDA6MDA6MDAgMjAwMQpGcm9tOiBNYXR0aGV3IEZsZW1pbmcgPG1kZkBGcmVlQlNELm9yZz4KRGF0 ZTogV2VkLCAyNyBNYXIgMjAxMyAxODoyMzoxNyAtMDcwMApTdWJqZWN0OiBbUEFUQ0hdIFVzZSBh IHNoYXJlZCBsb2NrIGZvciBWT1BfR0VURVhUQVRUUiwgYXMgaXQgaXMgYSByZWFkLWxpa2Ugb3Bl cmF0aW9uLgoKTUZDIGFmdGVyOgkxIHdlZWsKLS0tCiBzeXMva2Vybi92ZnNfZXh0YXR0ci5jIHwg ICAgMiArLQogc3lzL2tlcm4vdmZzX3Zub3BzLmMgICB8ICAgIDIgKy0KIDIgZmlsZXMgY2hhbmdl ZCwgMiBpbnNlcnRpb25zKCspLCAyIGRlbGV0aW9ucygtKQoKZGlmZiAtLWdpdCBhL3N5cy9rZXJu L3Zmc19leHRhdHRyLmMgYi9zeXMva2Vybi92ZnNfZXh0YXR0ci5jCmluZGV4IDg1ZmM4MzkuLjcw MGE3MGMgMTAwNjQ0Ci0tLSBhL3N5cy9rZXJuL3Zmc19leHRhdHRyLmMKKysrIGIvc3lzL2tlcm4v dmZzX2V4dGF0dHIuYwpAQCAtMzIxLDE3ICszMjEsMTcgQEAgZXh0YXR0cl9nZXRfdnAoc3RydWN0 IHZub2RlICp2cCwgaW50IGF0dHJuYW1lc3BhY2UsIGNvbnN0IGNoYXIgKmF0dHJuYW1lLAogICAg IHZvaWQgKmRhdGEsIHNpemVfdCBuYnl0ZXMsIHN0cnVjdCB0aHJlYWQgKnRkKQogewogCXN0cnVj dCB1aW8gYXVpbywgKmF1aW9wOwogCXN0cnVjdCBpb3ZlYyBhaW92OwogCXNzaXplX3QgY250Owog CXNpemVfdCBzaXplLCAqc2l6ZXA7CiAJaW50IGVycm9yOwogCi0Jdm5fbG9jayh2cCwgTEtfRVhD TFVTSVZFIHwgTEtfUkVUUlkpOworCXZuX2xvY2sodnAsIExLX1NIQVJFRCB8IExLX1JFVFJZKTsK IAogCS8qCiAJICogU2xpZ2h0bHkgdW51c3VhbCBzZW1hbnRpY3M6IGlmIHRoZSB1c2VyIHByb3Zp ZGVzIGEgTlVMTCBkYXRhCiAJICogcG9pbnRlciwgdGhleSBkb24ndCB3YW50IHRvIHJlY2VpdmUg dGhlIGRhdGEsIGp1c3QgdGhlIG1heGltdW0KIAkgKiByZWFkIGxlbmd0aC4KIAkgKi8KIAlhdWlv cCA9IE5VTEw7CiAJc2l6ZXAgPSBOVUxMOwpkaWZmIC0tZ2l0IGEvc3lzL2tlcm4vdmZzX3Zub3Bz LmMgYi9zeXMva2Vybi92ZnNfdm5vcHMuYwppbmRleCBjOGY4MzJmLi42ODQ5NzMwIDEwMDY0NAot LS0gYS9zeXMva2Vybi92ZnNfdm5vcHMuYworKysgYi9zeXMva2Vybi92ZnNfdm5vcHMuYwpAQCAt MTc1MywxNyArMTc1MywxNyBAQCB2bl9leHRhdHRyX2dldChzdHJ1Y3Qgdm5vZGUgKnZwLCBpbnQg aW9mbGcsIGludCBhdHRybmFtZXNwYWNlLAogCWF1aW8udWlvX2lvdmNudCA9IDE7CiAJYXVpby51 aW9fcncgPSBVSU9fUkVBRDsKIAlhdWlvLnVpb19zZWdmbGcgPSBVSU9fU1lTU1BBQ0U7CiAJYXVp by51aW9fdGQgPSB0ZDsKIAlhdWlvLnVpb19vZmZzZXQgPSAwOwogCWF1aW8udWlvX3Jlc2lkID0g KmJ1ZmxlbjsKIAogCWlmICgoaW9mbGcgJiBJT19OT0RFTE9DS0VEKSA9PSAwKQotCQl2bl9sb2Nr KHZwLCBMS19FWENMVVNJVkUgfCBMS19SRVRSWSk7CisJCXZuX2xvY2sodnAsIExLX1NIQVJFRCB8 IExLX1JFVFJZKTsKIAogCUFTU0VSVF9WT1BfTE9DS0VEKHZwLCAiSU9fTk9ERUxPQ0tFRCB3aXRo IG5vIHZwIGxvY2sgaGVsZCIpOwogCiAJLyogYXV0aG9yaXplIGF0dHJpYnV0ZSByZXRyaWV2YWwg YXMga2VybmVsICovCiAJZXJyb3IgPSBWT1BfR0VURVhUQVRUUih2cCwgYXR0cm5hbWVzcGFjZSwg YXR0cm5hbWUsICZhdWlvLCBOVUxMLCBOVUxMLAogCSAgICB0ZCk7CiAKIAlpZiAoKGlvZmxnICYg SU9fTk9ERUxPQ0tFRCkgPT0gMCkKLS0gCjEuNy4zLjIKCg== --000e0cd5ccee951c6904d8f21bef-- From owner-freebsd-fs@FreeBSD.ORG Thu Mar 28 01:45:26 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id E8835573; Thu, 28 Mar 2013 01:45:26 +0000 (UTC) (envelope-from mdf356@gmail.com) Received: from mail-qe0-f45.google.com (mail-qe0-f45.google.com [209.85.128.45]) by mx1.freebsd.org (Postfix) with ESMTP id 9CE7EEF8; Thu, 28 Mar 2013 01:45:26 +0000 (UTC) Received: by mail-qe0-f45.google.com with SMTP id b4so4903106qen.4 for ; Wed, 27 Mar 2013 18:45:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:content-type; bh=FS3mgNsZgh0E87awtM6qhaNO4G4sQSrgSsEaYl5XQpM=; b=x3oOiB0/sk3p0HvK/f9peduRw0JRH6nvxRIxknknFIFdolUfHm2/ZlHEEUWpdX2ufM o1tNmif3thx14P6hFFY/pT4aVkvMG9y1wZ1FBfxvd61g0ELD/4D3bY9JLaS+uyTO7Ibc Tct5RR+5ly9oudNzD1E29LL1c6Tte9/10zeTt14vMmE0vxKj8W9IhMyQfE7l07IuNTR6 8jrUbevS5Gbv1dPjDknB2ya6USsX6YZZz7DlWsakqTrx/3imis+ClRmPgGhwc4HZZ7yX nEbk7PKnxovWoj6ImuwwG4tSL76jkRpwbW8y+uX7atXp3QmX8ZD/LXUfWXn1zJq+PuiM I1Iw== MIME-Version: 1.0 X-Received: by 10.224.33.14 with SMTP id f14mr15790772qad.69.1364434672068; Wed, 27 Mar 2013 18:37:52 -0700 (PDT) Sender: mdf356@gmail.com Received: by 10.229.253.201 with HTTP; Wed, 27 Mar 2013 18:37:51 -0700 (PDT) In-Reply-To: References: Date: Wed, 27 Mar 2013 18:37:51 -0700 X-Google-Sender-Auth: 1Ya9g8m8aXcXnucHqY84C-jbqYY Message-ID: Subject: [RFC] use a shared lock for VOP_GETEXTATTR From: mdf@FreeBSD.org To: freebsd-current@freebsd.org, freebsd-fs@freebsd.org Content-Type: multipart/mixed; boundary=20cf3074b9a4f2ad3c04d8f23328 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Mar 2013 01:45:27 -0000 --20cf3074b9a4f2ad3c04d8f23328 Content-Type: text/plain; charset=ISO-8859-1 VOP_GETEXTATTR is currently called with an exclusive lock, which seems like overkill for what is essentially a read operation. I had a look over the various in-tree filesystems and it didn't look like any of them will have a problem if a shared-mode lock is used for vop_getextattr. Does anyone know otherwise? Is someone using extended attributes regularly who can test this? [sorry if this is a duplicate; I first sent from my non-FreeBSD mail] Thanks, matthew --20cf3074b9a4f2ad3c04d8f23328 Content-Type: application/octet-stream; name="0001-Use-a-shared-lock-for-VOP_GETEXTATTR-as-it-is-a-read.patch" Content-Disposition: attachment; filename="0001-Use-a-shared-lock-for-VOP_GETEXTATTR-as-it-is-a-read.patch" Content-Transfer-Encoding: base64 X-Attachment-Id: f_het94ldq0 RnJvbSBhYTZkNmM3MTZiZDE5Y2UzMmUzYjkwODcyN2ZiZGE2NWQzNDViN2FiIE1vbiBTZXAgMTcg MDA6MDA6MDAgMjAwMQpGcm9tOiBNYXR0aGV3IEZsZW1pbmcgPG1kZkBGcmVlQlNELm9yZz4KRGF0 ZTogV2VkLCAyNyBNYXIgMjAxMyAxODoyMzoxNyAtMDcwMApTdWJqZWN0OiBbUEFUQ0hdIFVzZSBh IHNoYXJlZCBsb2NrIGZvciBWT1BfR0VURVhUQVRUUiwgYXMgaXQgaXMgYSByZWFkLWxpa2Ugb3Bl cmF0aW9uLgoKTUZDIGFmdGVyOgkxIHdlZWsKLS0tCiBzeXMva2Vybi92ZnNfZXh0YXR0ci5jIHwg ICAgMiArLQogc3lzL2tlcm4vdmZzX3Zub3BzLmMgICB8ICAgIDIgKy0KIDIgZmlsZXMgY2hhbmdl ZCwgMiBpbnNlcnRpb25zKCspLCAyIGRlbGV0aW9ucygtKQoKZGlmZiAtLWdpdCBhL3N5cy9rZXJu L3Zmc19leHRhdHRyLmMgYi9zeXMva2Vybi92ZnNfZXh0YXR0ci5jCmluZGV4IDg1ZmM4MzkuLjcw MGE3MGMgMTAwNjQ0Ci0tLSBhL3N5cy9rZXJuL3Zmc19leHRhdHRyLmMKKysrIGIvc3lzL2tlcm4v dmZzX2V4dGF0dHIuYwpAQCAtMzIxLDE3ICszMjEsMTcgQEAgZXh0YXR0cl9nZXRfdnAoc3RydWN0 IHZub2RlICp2cCwgaW50IGF0dHJuYW1lc3BhY2UsIGNvbnN0IGNoYXIgKmF0dHJuYW1lLAogICAg IHZvaWQgKmRhdGEsIHNpemVfdCBuYnl0ZXMsIHN0cnVjdCB0aHJlYWQgKnRkKQogewogCXN0cnVj dCB1aW8gYXVpbywgKmF1aW9wOwogCXN0cnVjdCBpb3ZlYyBhaW92OwogCXNzaXplX3QgY250Owog CXNpemVfdCBzaXplLCAqc2l6ZXA7CiAJaW50IGVycm9yOwogCi0Jdm5fbG9jayh2cCwgTEtfRVhD TFVTSVZFIHwgTEtfUkVUUlkpOworCXZuX2xvY2sodnAsIExLX1NIQVJFRCB8IExLX1JFVFJZKTsK IAogCS8qCiAJICogU2xpZ2h0bHkgdW51c3VhbCBzZW1hbnRpY3M6IGlmIHRoZSB1c2VyIHByb3Zp ZGVzIGEgTlVMTCBkYXRhCiAJICogcG9pbnRlciwgdGhleSBkb24ndCB3YW50IHRvIHJlY2VpdmUg dGhlIGRhdGEsIGp1c3QgdGhlIG1heGltdW0KIAkgKiByZWFkIGxlbmd0aC4KIAkgKi8KIAlhdWlv cCA9IE5VTEw7CiAJc2l6ZXAgPSBOVUxMOwpkaWZmIC0tZ2l0IGEvc3lzL2tlcm4vdmZzX3Zub3Bz LmMgYi9zeXMva2Vybi92ZnNfdm5vcHMuYwppbmRleCBjOGY4MzJmLi42ODQ5NzMwIDEwMDY0NAot LS0gYS9zeXMva2Vybi92ZnNfdm5vcHMuYworKysgYi9zeXMva2Vybi92ZnNfdm5vcHMuYwpAQCAt MTc1MywxNyArMTc1MywxNyBAQCB2bl9leHRhdHRyX2dldChzdHJ1Y3Qgdm5vZGUgKnZwLCBpbnQg aW9mbGcsIGludCBhdHRybmFtZXNwYWNlLAogCWF1aW8udWlvX2lvdmNudCA9IDE7CiAJYXVpby51 aW9fcncgPSBVSU9fUkVBRDsKIAlhdWlvLnVpb19zZWdmbGcgPSBVSU9fU1lTU1BBQ0U7CiAJYXVp by51aW9fdGQgPSB0ZDsKIAlhdWlvLnVpb19vZmZzZXQgPSAwOwogCWF1aW8udWlvX3Jlc2lkID0g KmJ1ZmxlbjsKIAogCWlmICgoaW9mbGcgJiBJT19OT0RFTE9DS0VEKSA9PSAwKQotCQl2bl9sb2Nr KHZwLCBMS19FWENMVVNJVkUgfCBMS19SRVRSWSk7CisJCXZuX2xvY2sodnAsIExLX1NIQVJFRCB8 IExLX1JFVFJZKTsKIAogCUFTU0VSVF9WT1BfTE9DS0VEKHZwLCAiSU9fTk9ERUxPQ0tFRCB3aXRo IG5vIHZwIGxvY2sgaGVsZCIpOwogCiAJLyogYXV0aG9yaXplIGF0dHJpYnV0ZSByZXRyaWV2YWwg YXMga2VybmVsICovCiAJZXJyb3IgPSBWT1BfR0VURVhUQVRUUih2cCwgYXR0cm5hbWVzcGFjZSwg YXR0cm5hbWUsICZhdWlvLCBOVUxMLCBOVUxMLAogCSAgICB0ZCk7CiAKIAlpZiAoKGlvZmxnICYg SU9fTk9ERUxPQ0tFRCkgPT0gMCkKLS0gCjEuNy4zLjIKCg== --20cf3074b9a4f2ad3c04d8f23328-- From owner-freebsd-fs@FreeBSD.ORG Thu Mar 28 05:32:09 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 3A5911A1; Thu, 28 Mar 2013 05:32:09 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id A74138FE; Thu, 28 Mar 2013 05:32:08 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.6/8.14.6) with ESMTP id r2S5W5u4047614; Thu, 28 Mar 2013 07:32:05 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.8.0 kib.kiev.ua r2S5W5u4047614 Received: (from kostik@localhost) by tom.home (8.14.6/8.14.6/Submit) id r2S5W5T1047613; Thu, 28 Mar 2013 07:32:05 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 28 Mar 2013 07:32:05 +0200 From: Konstantin Belousov To: mdf@FreeBSD.org Subject: Re: [RFC] use a shared lock for VOP_GETEXTATTR Message-ID: <20130328053205.GF3794@kib.kiev.ua> References: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="MZTSf4zogrZAfwtT" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: freebsd-fs@freebsd.org, freebsd-current@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Mar 2013 05:32:09 -0000 --MZTSf4zogrZAfwtT Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Mar 27, 2013 at 06:37:51PM -0700, mdf@FreeBSD.org wrote: > VOP_GETEXTATTR is currently called with an exclusive lock, which seems > like overkill for what is essentially a read operation. I had a look > over the various in-tree filesystems and it didn't look like any of > them will have a problem if a shared-mode lock is used for > vop_getextattr. >=20 > Does anyone know otherwise? Is someone using extended attributes > regularly who can test this? I think this change should be fine. At least it seems to for UFS. What other filesystems did you audited ? --MZTSf4zogrZAfwtT Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iQIcBAEBAgAGBQJRU9XUAAoJEJDCuSvBvK1BKGgP+warWw/Mo9pcf9zIj+cBE3PZ 9sdXjWUcCKqIM1pOFlURH2odq9P1+5Va7wP2n5jYBYUdri0SoCfQ/C86jvUkvaNS xKrMtNXIq38eKCCWU34+rFNhF9bg+rS0KS8FXI5+nnsbDsWaB0PeCXBSrfudeTv+ HFfGFmGLQvJdzYfvZ6LWInWw8BhgyrLX0/A/xAnPwlZDLrw1RbtAL51XRH/PHliv KfsxlyMoiTqJLDR7jX0EPiIcg+WENyUgwQ/SJTZ4i3a52y8eXLlSzHIhrinHPmnE b3iCc3TTZCRJYx0cOJMjLzk/d1kIVTYgRZ+Sdap1UjHeFw+E8Ps3Ev2T3BCzYCzG eTZBdffzcDW6xxc7zYfNDeiUvKEq3uKsBY9AyQeZWn5vEOsFSYFXc4TV/Q/OT+VM /hcdbgqcjDh2rXVJo1XHkpb+6mMZN4HWOo/r8Q+jVk2q/5RCmeQCN8tmbmWI4jIr j/hYA4zvH6cuEJ3iDghDV4urgb11VZ/Ybyx3At26J4jlUqW0B9MvIbvyRTnGiQt5 A95rE6tR0sfiMCRYrXs+pWR82DboNtv5PaQ6Wo5M+/hcweU1xjuJ/ris6EJVXUuU vcHp2dGMabmPzzFNPuN6lLh0VtOx1M57wJ/3ZV6xdDdZAeKrI0/GoU+CvqT/ONPR pPPVG6W9gpNZ8OpRB1sl =4mql -----END PGP SIGNATURE----- --MZTSf4zogrZAfwtT-- From owner-freebsd-fs@FreeBSD.ORG Thu Mar 28 05:40:18 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 37B5379C; Thu, 28 Mar 2013 05:40:18 +0000 (UTC) (envelope-from mdf356@gmail.com) Received: from mail-qc0-x233.google.com (mail-qc0-x233.google.com [IPv6:2607:f8b0:400d:c01::233]) by mx1.freebsd.org (Postfix) with ESMTP id E260B977; Thu, 28 Mar 2013 05:40:17 +0000 (UTC) Received: by mail-qc0-f179.google.com with SMTP id b40so4068412qcq.10 for ; Wed, 27 Mar 2013 22:40:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=4ffQ2MMb83OZN/KkxP4eFjkYTLMRzM7AvYSi8QxhsyA=; b=sovq3PEQrj0Xy/kr+XQ3HvLWs6oOsKsjXDE8A6ycE7CO44BKgckGg+mKSeSQwYCiAl MnRMvdPO0pMB+v9+tvVcqk0n6KQdMLaMK40+jZWGWTyjS1aFrqaG7OO9g2fekb6QZLRg r+T614tgfScnn7+8G+o5hw1XrwSeTQU+Znvmnpjysr3VpfdL2NR7ElcPUb3Y4KP+mgcD lAHLYLEkEgMHNczMCCsYk1bsD5r4T6JjdAweCNxYnaSodXhn7yxtxPFw825tvaKq0ttC aZ0qBCRlS0HcrQbCC5JDyEcujMgyMRlrfhVytzvbJowZKuLyOHUSdJmp3YlJqQHbpV1b SYpA== MIME-Version: 1.0 X-Received: by 10.49.25.202 with SMTP id e10mr20122125qeg.49.1364449217129; Wed, 27 Mar 2013 22:40:17 -0700 (PDT) Sender: mdf356@gmail.com Received: by 10.229.253.201 with HTTP; Wed, 27 Mar 2013 22:40:16 -0700 (PDT) In-Reply-To: <20130328053205.GF3794@kib.kiev.ua> References: <20130328053205.GF3794@kib.kiev.ua> Date: Wed, 27 Mar 2013 22:40:16 -0700 X-Google-Sender-Auth: zJ7rBN1qp88H4U2Qe7csJJKC1E4 Message-ID: Subject: Re: [RFC] use a shared lock for VOP_GETEXTATTR From: mdf@FreeBSD.org To: Konstantin Belousov Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-fs@freebsd.org, freebsd-current@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Mar 2013 05:40:18 -0000 On Wed, Mar 27, 2013 at 10:32 PM, Konstantin Belousov wrote: > On Wed, Mar 27, 2013 at 06:37:51PM -0700, mdf@FreeBSD.org wrote: >> VOP_GETEXTATTR is currently called with an exclusive lock, which seems >> like overkill for what is essentially a read operation. I had a look >> over the various in-tree filesystems and it didn't look like any of >> them will have a problem if a shared-mode lock is used for >> vop_getextattr. >> >> Does anyone know otherwise? Is someone using extended attributes >> regularly who can test this? > > I think this change should be fine. At least it seems to for UFS. > > What other filesystems did you audited ? I looked over zfs, pseudofs, unionfs, ffs/ufs. None seemed to have any asserts on the lock type nor anything that looked like it would try to modify anything. zfs, I think it was, even used VOP_ISLOCKED to get the lock type in one path (but I think that was after a lookup(), so it may have been on a different vnode). Cheers, matthew From owner-freebsd-fs@FreeBSD.ORG Thu Mar 28 05:56:27 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id D3E8ECD9; Thu, 28 Mar 2013 05:56:27 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id 7461D9EC; Thu, 28 Mar 2013 05:56:27 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.6/8.14.6) with ESMTP id r2S5uN0K052050; Thu, 28 Mar 2013 07:56:23 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.8.0 kib.kiev.ua r2S5uN0K052050 Received: (from kostik@localhost) by tom.home (8.14.6/8.14.6/Submit) id r2S5uNj4052049; Thu, 28 Mar 2013 07:56:23 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 28 Mar 2013 07:56:23 +0200 From: Konstantin Belousov To: mdf@FreeBSD.org Subject: Re: [RFC] use a shared lock for VOP_GETEXTATTR Message-ID: <20130328055623.GG3794@kib.kiev.ua> References: <20130328053205.GF3794@kib.kiev.ua> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="XsW72sOHJGfuGe9h" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: freebsd-fs@freebsd.org, freebsd-current@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Mar 2013 05:56:27 -0000 --XsW72sOHJGfuGe9h Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Mar 27, 2013 at 10:40:16PM -0700, mdf@FreeBSD.org wrote: > On Wed, Mar 27, 2013 at 10:32 PM, Konstantin Belousov > wrote: > > On Wed, Mar 27, 2013 at 06:37:51PM -0700, mdf@FreeBSD.org wrote: > >> VOP_GETEXTATTR is currently called with an exclusive lock, which seems > >> like overkill for what is essentially a read operation. I had a look > >> over the various in-tree filesystems and it didn't look like any of > >> them will have a problem if a shared-mode lock is used for > >> vop_getextattr. > >> > >> Does anyone know otherwise? Is someone using extended attributes > >> regularly who can test this? > > > > I think this change should be fine. At least it seems to for UFS. > > > > What other filesystems did you audited ? >=20 > I looked over zfs, pseudofs, unionfs, ffs/ufs. None seemed to have > any asserts on the lock type nor anything that looked like it would > try to modify anything. zfs, I think it was, even used VOP_ISLOCKED > to get the lock type in one path (but I think that was after a > lookup(), so it may have been on a different vnode). VOPs usually do not assert the lock state, unless the explicit unlock/lock is done, or some other call is made which asserts the lock. If zfs is fine =66rom your inspection, I think that the patch can be committed without an issue. --XsW72sOHJGfuGe9h Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iQIcBAEBAgAGBQJRU9uHAAoJEJDCuSvBvK1B5+oP/1A+CLA6d4Uj4cBEZH71gsPd nI4QzQ8Mr1zOP2yLH9clbtT29NDXb/zwbFRzHFh4R8UD8mnQzr6Aebkt+obzctCz a3JT6WF4D+uCPL+E5cUTaVpaA/8guQO+MlP3ECN21KqqDI5mAfHEshSLcbHeJi2Q MAmWID6Wn0ONIk2027pjP2rayQE0ECpKEoEuahPNeHjTecstY2EKA4nITXb+dVLj mgC7zpHlLeH0PD6/PM47RBIc/AqTi6B2Mj3swYuIEUo2PJYMzNG+pdl9IYg+VNzZ KklxpSNRkBwUGfPGLiIKxR3Bvr404x3/C4rBWGyyIDYs75q5C4pUfUnKImJ8aerg fp4gfwSZ58xMV5yNUIPQRuKh7pQdf6ha5Kc4THEBKk4Cl4KL0SCRk1NS/vnn+IIR nDYog1g6h421Nvx9EIP/SoyYHlyni0J8Q9DDvu6eQEyUGwQdBkWqg8aKXF/7pRFr cjJTPHZ23zCbNQF+h4kIyrkqgcUYr1PQeCw77YN17rhfR7gfgAVoUxdCI6B+Un62 Pl/X9vte63S/LpbAu584ItlOrVFzFMXoYM+lcTmqra2InKtl0CtyzUg7lDrsuii8 j1SnN/EPuWJB8VNIWHp4n9O/Z5iHURbJLz1sT0POp+bocFiTHa8Nas8GGB8wcCYQ pqdGv5PhGVeLuQWQm+nT =2MpL -----END PGP SIGNATURE----- --XsW72sOHJGfuGe9h-- From owner-freebsd-fs@FreeBSD.ORG Thu Mar 28 18:27:58 2013 Return-Path: Delivered-To: freebsd-fs@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id A0387B37; Thu, 28 Mar 2013 18:27:58 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id 7A82CADB; Thu, 28 Mar 2013 18:27:58 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.6/8.14.6) with ESMTP id r2SIRwOB029952; Thu, 28 Mar 2013 18:27:58 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.6/8.14.6/Submit) id r2SIRwlm029951; Thu, 28 Mar 2013 18:27:58 GMT (envelope-from linimon) Date: Thu, 28 Mar 2013 18:27:58 GMT Message-Id: <201303281827.r2SIRwlm029951@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-amd64@FreeBSD.org, freebsd-fs@FreeBSD.org From: linimon@FreeBSD.org Subject: Re: kern/177445: [hast] HAST panic X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Mar 2013 18:27:58 -0000 Old Synopsis: HAST panic New Synopsis: [hast] HAST panic Responsible-Changed-From-To: freebsd-amd64->freebsd-fs Responsible-Changed-By: linimon Responsible-Changed-When: Thu Mar 28 18:27:43 UTC 2013 Responsible-Changed-Why: reclassify and assign. http://www.freebsd.org/cgi/query-pr.cgi?pr=177445 From owner-freebsd-fs@FreeBSD.ORG Fri Mar 29 02:28:52 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id E7923DE3 for ; Fri, 29 Mar 2013 02:28:52 +0000 (UTC) (envelope-from Kamil.Choudhury@anserinae.net) Received: from hrndva-omtalb.mail.rr.com (hrndva-omtalb.mail.rr.com [71.74.56.122]) by mx1.freebsd.org (Postfix) with ESMTP id B08E35F3 for ; Fri, 29 Mar 2013 02:28:52 +0000 (UTC) X-Authority-Analysis: v=2.0 cv=H5hZMpki c=1 sm=0 a=jrWvM0xCa/yzQsHFmCBh5A==:17 a=XtYZqdvDNwYA:10 a=3xLwOXIM4dgA:10 a=WWGGoYozHbgA:10 a=kj9zAlcOel0A:10 a=xqWC_Br6kY4A:10 a=NUNO_Q2GAAAA:8 a=5IDOE-LcbBkA:10 a=Q6Ojrw04AAAA:8 a=6SwBGEUBGz--ewoBKqQA:9 a=CjuIK1q_8ugA:10 a=qY5geOaZfuAA:10 a=jrWvM0xCa/yzQsHFmCBh5A==:117 X-Cloudmark-Score: 0 X-Authenticated-User: X-Originating-IP: 74.73.121.187 Received: from [74.73.121.187] ([74.73.121.187:62624] helo=janus.anserinae.net) by hrndva-oedge03.mail.rr.com (envelope-from ) (ecelerity 2.2.3.46 r()) with ESMTP id 99/BF-02677-12CF4515; Fri, 29 Mar 2013 02:27:45 +0000 Received: from JANUS.anserinae.net ([fe80::192c:4b89:9fe9:dc6d]) by janus.anserinae.net ([fe80::192c:4b89:9fe9:dc6d%11]) with mapi; Thu, 28 Mar 2013 22:27:44 -0400 From: Kamil Choudhury To: "freebsd-fs@freebsd.org" Subject: Building ZFS out of ISCSI LUNs? Thread-Topic: Building ZFS out of ISCSI LUNs? Thread-Index: Ac4sI+oaiWa6H9LnTPqWxjVOV0dXlA== Date: Fri, 29 Mar 2013 02:27:43 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Mar 2013 02:28:53 -0000 I stumbled across this thread on ServerFault:=20 http://serverfault.com/questions/419679/zfs-over-iscsi-high-availability-so= lution Summary: export LUNs from various (independent, share-nothing) storage nod= es, cobble them together into vdevs, and then create storage pools on them.= =20 Am I insane, or could this (with a layer to coordinate access to the LUNs)= be a pretty nifty way to create a distributed storage system? =20 From owner-freebsd-fs@FreeBSD.ORG Fri Mar 29 14:24:21 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 3CCAAB0A for ; Fri, 29 Mar 2013 14:24:21 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Received: from mail-ia0-x232.google.com (mail-ia0-x232.google.com [IPv6:2607:f8b0:4001:c02::232]) by mx1.freebsd.org (Postfix) with ESMTP id 12189883 for ; Fri, 29 Mar 2013 14:24:21 +0000 (UTC) Received: by mail-ia0-f178.google.com with SMTP id r13so430260iar.23 for ; Fri, 29 Mar 2013 07:24:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=veCJOBjp9ESYoLje7TNTn8GOKNiVRgzHxi/jWsSY67s=; b=k/8jS+WkyK5ZBI5jyuzj6YoxFbBRvU+MRnsrdSQboR+ezRBCKmBeJceo2o+2uGssO1 77uFmH4g8TCuCGdTkiOi5sJZNN9zkI8w3g+6RSrSH8zxMxiFrM0SlOMXNfgmw3HQAA0S xq1ShnhEHmOqsfC6RQejUfrLpCLBMXpFs2Udj2PY+NwlOpy3Zt2uKzAtLn6Qd5RcRqqU KrCid84OGoqpskRWQgLytOf3PnyuEft9JvXRuY5bvcsMe7km3zh3j4TskgwrVj0MS+KY GotVRMrNeP6Gm3fq1riNgXBLm8CLWAlZDanzUnqQMTQKWizYjwTZHvQS5PzaQzCXpZDv jfPw== MIME-Version: 1.0 X-Received: by 10.50.12.193 with SMTP id a1mr10008268igc.24.1364567060714; Fri, 29 Mar 2013 07:24:20 -0700 (PDT) Sender: jdavidlists@gmail.com Received: by 10.42.83.83 with HTTP; Fri, 29 Mar 2013 07:24:20 -0700 (PDT) In-Reply-To: References: Date: Fri, 29 Mar 2013 10:24:20 -0400 X-Google-Sender-Auth: pi3NeowRlSpCZaPJ95yEEtEtp_A Message-ID: Subject: Re: Building ZFS out of ISCSI LUNs? From: J David To: Kamil Choudhury Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Mar 2013 14:24:21 -0000 On Thu, Mar 28, 2013 at 10:27 PM, Kamil Choudhury < Kamil.Choudhury@anserinae.net> wrote: > Summary: export LUNs from various (independent, share-nothing) storage > nodes, cobble them together into vdevs, and then create storage pools on > them. > > Am I insane, or could this (with a layer to coordinate access to the > LUNs) be a pretty nifty way to create a distributed storage system? > It is possible, and I've done it as a test, but it's not a good idea. If a storage node (box of drives) dies, you lose every drive in it. To minimize the impact of losing a box of drives, you have to increase the number of boxes. Realistically the smallest number of drives per 1U server is 4. So if a node fails, you still lose 4 vdevs and your pool is probably shot until you fix it anyway. You can create four mirrors from one 4-drive box to another, but then if you lose a box it's one drive failure from there to permanent data loss. You can get more expensive boxes and put 10 drives per box and RAID 6 them together than mirror the result, but that's the exact opposite of putting ZFS close to the disks where it belongs, so performance dives some more. SAS shelves are cheap and SAS shelf failures are rarer than server chassis failures (fewer parts and good redundancy), so by swapping out a limited-complex node with good redundancy for a complex node with none, you've paid more for significantly less reliability and moderately awful performance. Also SAS is significantly faster at this than reading disks with SATA and exporting them with 1G ethernet via iSCSI. If you use 10G ethernet you will close the gap but of course you need multi path which means two 10G $witches. Finally, this is not in any way "distributed." The headend is still a single point of failure. Hooking an active and standby headend to the same pool of disks via Ethernet is conceptually easier than doing it via SAS, but the financial and performance costs aren't worth it. And even if you do that, you still have to put any ZIL/L2ARC SSD's in the shared storage too, otherwise the standby can't import the pool, and putting SSD on the wrong side of an ethernet switch when its raison d'=EAtre is low latency really blunts its effectiveness. So by the time you get two ZFS headends, two 10gig switches, and enough 1U "shared nothing" drive servers that you can build pools of usable size that don't get crushed by the loss of a single node, you have spent enough to buy a nice NetApp and a Mercedes to deliver it in. And you still have to solve the problem of how to fail over from the active headend to the standby. And the resulting performance will be embarrassing compared to a dual-head SAS setup at the same price. (Which admittedly still has the same how-to-failover problem.) ZFS is *not* a building block for a distributed storage system, or even a high-availability one. In fact, to the best of my knowledge, there are *zero* functioning production-quality distributed filesystems that can be implemented using only FreeBSD.