From owner-freebsd-fs@FreeBSD.ORG Sun Jun 23 09:24:38 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 7CB1F1DD for ; Sun, 23 Jun 2013 09:24:38 +0000 (UTC) (envelope-from gabor@zahemszky.hu) Received: from default-smtp.integrity.hu (default-smtp.integrity.hu [212.52.165.203]) by mx1.freebsd.org (Postfix) with ESMTP id 42BE21649 for ; Sun, 23 Jun 2013 09:24:37 +0000 (UTC) Received: by smtp.integrity.hu (Postfix, from userid 10000) id A3AFE134625D; Sun, 23 Jun 2013 11:15:45 +0200 (CEST) Received: from webmail.integrity.hu (mail-fe-1.integrity.hu [10.1.64.120]) (Authenticated sender: gabor@zahemszky.hu) by smtp.integrity.hu (Postfix) with ESMTPA id 114401346250 for ; Sun, 23 Jun 2013 11:15:44 +0200 (CEST) Received: from XawE/w7G0INgHr2gPKRKFrjZiKCVPzZ0mPvj26LZRe20lDRt4KguGw== (c9K7B4Wjx4af9IxzklBAz3Fa7a0Har0F) by webmail.integrity.hu with HTTP (HTTP/1.1 POST); Sun, 23 Jun 2013 11:15:44 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Date: Sun, 23 Jun 2013 11:15:44 +0200 From: gabor@zahemszky.hu To: Subject: ext2fs bug Message-ID: <9554b1ed02a342c347b17a2e71e38b73@zahemszky.hu> X-Sender: gabor@zahemszky.hu User-Agent: Roundcube Webmail/0.8.4 X-Virus-Scanned: clamav-milter 0.97.6 at mail-autosubmit X-Virus-Status: Clean X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Jun 2013 09:24:38 -0000 Hi! I've got 2 disks (a 1TB and a 2TB), these disk were written under Linux 2.4. I can mount them under FreeBSD, and I can reach about half of the files, but on some files, I only get some FS errors. In dmesg, I can see the following errors: g_vfs_done():ada2s1[READ(offset=-756558786560, length=4096)]error = 5 g_vfs_done():ada2s1[READ(offset=-899189821440, length=4096)]error = 5 g_vfs_done():ada2s1[READ(offset=-756558786560, length=4096)]error = 5 g_vfs_done():ada2s1[READ(offset=-899189821440, length=4096)]error = 5 g_vfs_done():ada2s1[READ(offset=-756558786560, length=4096)]error = 5 g_vfs_done():ada2s1[READ(offset=-899189821440, length=4096)]error = 5 g_vfs_done():ada2s1[READ(offset=-756558786560, length=4096)]error = 5 g_vfs_done():ada2s1[READ(offset=-899189821440, length=4096)]error = 5 g_vfs_done():ada2s1[READ(offset=-756558786560, length=4096)]error = 5 g_vfs_done():ada2s1[READ(offset=-899189821440, length=4096)]error = 5 The disk is not bad: I can read the same files under the original Linux, and on a newer Linux-machine, and I can read it under FreeBSD with the e2fsprogs package's debugfs tool (I can copy the files with the dump and rdump commmand from debugfs.) Itt looks, that there are some misinterpretation of some of the metainfo in the Ext2 fs-driver. (Or maybe some signed/unsigned bug in it.) Can anybody help? Can I help to anybody with some more information? Thanks, Gabor < Gabor at Zahemszky dot HU > PS: sometimes I can only see az empty directory. Sometimes, I can reach half of the files in a directry. But the kernel error message is the same. From owner-freebsd-fs@FreeBSD.ORG Sun Jun 23 09:35:50 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 8F08C4F8 for ; Sun, 23 Jun 2013 09:35:50 +0000 (UTC) (envelope-from jdc@koitsu.org) Received: from slow1-d.mail.gandi.net (slow1-d.mail.gandi.net [217.70.178.86]) by mx1.freebsd.org (Postfix) with ESMTP id 4EE5F1691 for ; Sun, 23 Jun 2013 09:35:50 +0000 (UTC) Received: from relay3-d.mail.gandi.net (relay3-d.mail.gandi.net [217.70.183.195]) by slow1-d.mail.gandi.net (Postfix) with ESMTP id E9EBA486523 for ; Sun, 23 Jun 2013 11:35:42 +0200 (CEST) Received: from mfilter18-d.gandi.net (mfilter18-d.gandi.net [217.70.178.146]) by relay3-d.mail.gandi.net (Postfix) with ESMTP id E8870A80C8; Sun, 23 Jun 2013 11:35:25 +0200 (CEST) X-Virus-Scanned: Debian amavisd-new at mfilter18-d.gandi.net Received: from relay3-d.mail.gandi.net ([217.70.183.195]) by mfilter18-d.gandi.net (mfilter18-d.gandi.net [10.0.15.180]) (amavisd-new, port 10024) with ESMTP id 1t-tA2zAtdjv; Sun, 23 Jun 2013 11:35:24 +0200 (CEST) X-Originating-IP: 76.102.14.35 Received: from jdc.koitsu.org (c-76-102-14-35.hsd1.ca.comcast.net [76.102.14.35]) (Authenticated sender: jdc@koitsu.org) by relay3-d.mail.gandi.net (Postfix) with ESMTPSA id BB6ACA80B8; Sun, 23 Jun 2013 11:35:23 +0200 (CEST) Received: by icarus.home.lan (Postfix, from userid 1000) id CF7A173A1C; Sun, 23 Jun 2013 02:35:21 -0700 (PDT) Date: Sun, 23 Jun 2013 02:35:21 -0700 From: Jeremy Chadwick To: gabor@zahemszky.hu Subject: Re: ext2fs bug Message-ID: <20130623093521.GA86307@icarus.home.lan> References: <9554b1ed02a342c347b17a2e71e38b73@zahemszky.hu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <9554b1ed02a342c347b17a2e71e38b73@zahemszky.hu> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Jun 2013 09:35:50 -0000 On Sun, Jun 23, 2013 at 11:15:44AM +0200, gabor@zahemszky.hu wrote: > Hi! > > I've got 2 disks (a 1TB and a 2TB), these disk were written under > Linux 2.4. > I can mount them under FreeBSD, and I can reach about half of the > files, but > on some files, I only get some FS errors. In dmesg, I can see the > following > errors: > > g_vfs_done():ada2s1[READ(offset=-756558786560, length=4096)]error = 5 > g_vfs_done():ada2s1[READ(offset=-899189821440, length=4096)]error = 5 > g_vfs_done():ada2s1[READ(offset=-756558786560, length=4096)]error = 5 > g_vfs_done():ada2s1[READ(offset=-899189821440, length=4096)]error = 5 > g_vfs_done():ada2s1[READ(offset=-756558786560, length=4096)]error = 5 > g_vfs_done():ada2s1[READ(offset=-899189821440, length=4096)]error = 5 > g_vfs_done():ada2s1[READ(offset=-756558786560, length=4096)]error = 5 > g_vfs_done():ada2s1[READ(offset=-899189821440, length=4096)]error = 5 > g_vfs_done():ada2s1[READ(offset=-756558786560, length=4096)]error = 5 > g_vfs_done():ada2s1[READ(offset=-899189821440, length=4096)]error = 5 > > The disk is not bad: I can read the same files under the original > Linux, > and on a newer Linux-machine, and I can read it under FreeBSD with > the e2fsprogs package's > debugfs tool (I can copy the files with the dump and rdump commmand > from debugfs.) > > Itt looks, that there are some misinterpretation of some of the > metainfo > in the Ext2 fs-driver. (Or maybe some signed/unsigned bug in it.) > > Can anybody help? Can I help to anybody with some more information? > > Thanks, > > Gabor < Gabor at Zahemszky dot HU > > > PS: sometimes I can only see az empty directory. Sometimes, I can > reach half of the files > in a directry. But the kernel error message is the same. What FreeBSD version are you using? In the past 3-4 weeks there have been *tons* of ext2fs fixes in stable/9, and I'm not exaggerating in the least. You will not find these fixes being backported to 9.1-RELEASE or earlier. I strongly recommend you try a recent stable/9 (i.e. past few days) with ext2fs. You can try a stable/9 ("RELENG_9") ISO snapshot here: https://pub.allbsd.org/FreeBSD-snapshots/ -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Sun Jun 23 09:43:15 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 8D44B738 for ; Sun, 23 Jun 2013 09:43:15 +0000 (UTC) (envelope-from gabor@zahemszky.hu) Received: from default-smtp.integrity.hu (default-smtp.integrity.hu [212.52.165.203]) by mx1.freebsd.org (Postfix) with ESMTP id 5226816BD for ; Sun, 23 Jun 2013 09:43:15 +0000 (UTC) Received: by smtp.integrity.hu (Postfix, from userid 10000) id A867F1346252; Sun, 23 Jun 2013 11:43:14 +0200 (CEST) Received: from webmail.integrity.hu (mail-fe-1.integrity.hu [10.1.64.120]) (Authenticated sender: gabor@zahemszky.hu) by smtp.integrity.hu (Postfix) with ESMTPA id 3CFBE1346252; Sun, 23 Jun 2013 11:43:14 +0200 (CEST) Received: from +YTNHfsHThRfvg4B2o9k1hVTDbwpMyoSdAkDl3bEmcTRRA50ZTS9rg== (Qx1vxEXaE5uWFT4iAghDYvKeIKRH0pIk) by webmail.integrity.hu with HTTP (HTTP/1.1 POST); Sun, 23 Jun 2013 11:43:13 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Date: Sun, 23 Jun 2013 11:43:13 +0200 From: gabor@zahemszky.hu To: Jeremy Chadwick Subject: Re: ext2fs bug In-Reply-To: <20130623093521.GA86307@icarus.home.lan> References: <9554b1ed02a342c347b17a2e71e38b73@zahemszky.hu> <20130623093521.GA86307@icarus.home.lan> Message-ID: X-Sender: gabor@zahemszky.hu User-Agent: Roundcube Webmail/0.8.4 X-Virus-Scanned: clamav-milter 0.97.6 at mail-autosubmit X-Virus-Status: Clean Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Jun 2013 09:43:15 -0000 Hi! > What FreeBSD version are you using? Sorry for the missing information: it's 9.1-p4 Amd64 > In the past 3-4 weeks there have been *tons* of ext2fs fixes in > stable/9, and I'm not exaggerating in the least. You will not find > these fixes being backported to 9.1-RELEASE or earlier. If they will be in 9.2 or in 10, it will be good. > I strongly recommend you try a recent stable/9 (i.e. past few days) > with ext2fs. You can try a stable/9 ("RELENG_9") ISO snapshot here: > > https://pub.allbsd.org/FreeBSD-snapshots/ Can I run these ISO-s from a flash drive as a Live system? (It's an HP microserver, and I don't have CD/DVD-player in it). Thanks, Gabor From owner-freebsd-fs@FreeBSD.ORG Sun Jun 23 09:52:22 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 84C61848 for ; Sun, 23 Jun 2013 09:52:22 +0000 (UTC) (envelope-from jdc@koitsu.org) Received: from relay3-d.mail.gandi.net (relay3-d.mail.gandi.net [217.70.183.195]) by mx1.freebsd.org (Postfix) with ESMTP id 4584216FC for ; Sun, 23 Jun 2013 09:52:22 +0000 (UTC) Received: from mfilter26-d.gandi.net (mfilter26-d.gandi.net [217.70.178.154]) by relay3-d.mail.gandi.net (Postfix) with ESMTP id E4789A80C2; Sun, 23 Jun 2013 11:52:10 +0200 (CEST) X-Virus-Scanned: Debian amavisd-new at mfilter26-d.gandi.net Received: from relay3-d.mail.gandi.net ([217.70.183.195]) by mfilter26-d.gandi.net (mfilter26-d.gandi.net [10.0.15.180]) (amavisd-new, port 10024) with ESMTP id IUod+RuFPB5a; Sun, 23 Jun 2013 11:52:09 +0200 (CEST) X-Originating-IP: 76.102.14.35 Received: from jdc.koitsu.org (c-76-102-14-35.hsd1.ca.comcast.net [76.102.14.35]) (Authenticated sender: jdc@koitsu.org) by relay3-d.mail.gandi.net (Postfix) with ESMTPSA id 0A92AA80B8; Sun, 23 Jun 2013 11:52:09 +0200 (CEST) Received: by icarus.home.lan (Postfix, from userid 1000) id 430AF73A1C; Sun, 23 Jun 2013 02:52:07 -0700 (PDT) Date: Sun, 23 Jun 2013 02:52:07 -0700 From: Jeremy Chadwick To: gabor@zahemszky.hu Subject: Re: ext2fs bug Message-ID: <20130623095207.GA86507@icarus.home.lan> References: <9554b1ed02a342c347b17a2e71e38b73@zahemszky.hu> <20130623093521.GA86307@icarus.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Jun 2013 09:52:22 -0000 On Sun, Jun 23, 2013 at 11:43:13AM +0200, gabor@zahemszky.hu wrote: > Hi! > > >What FreeBSD version are you using? > > Sorry for the missing information: it's 9.1-p4 Amd64 > > >In the past 3-4 weeks there have been *tons* of ext2fs fixes in > >stable/9, and I'm not exaggerating in the least. You will not find > >these fixes being backported to 9.1-RELEASE or earlier. > > If they will be in 9.2 or in 10, it will be good. > > >I strongly recommend you try a recent stable/9 (i.e. past few days) > >with ext2fs. You can try a stable/9 ("RELENG_9") ISO snapshot here: > > > >https://pub.allbsd.org/FreeBSD-snapshots/ > > Can I run these ISO-s from a flash drive as a Live system? (It's an > HP microserver, and I don't have CD/DVD-player in it). No you cannot use the ISO images -- ISO images are for CD/DVD drives; dd'ing them or "flashing" them to a USB memory stick/flash drive will not work generally speaking. (Note to readers: please do not bring ISOLINUX into this discussion -- I beg you, please do not, not just for the technical aspects but also because it's not necessary in this situation). However, you can download the relevant memstick.img files and dd those to a USB memory stick/flash drive and boot from (just like standard 9.1-RELEASE memstick images). Point: these are not ISOs, these are raw images. -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Sun Jun 23 10:29:23 2013 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 5CF56207 for ; Sun, 23 Jun 2013 10:29:23 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail109.syd.optusnet.com.au (mail109.syd.optusnet.com.au [211.29.132.80]) by mx1.freebsd.org (Postfix) with ESMTP id 0E64C17D8 for ; Sun, 23 Jun 2013 10:29:22 +0000 (UTC) Received: from c122-106-156-23.carlnfd1.nsw.optusnet.com.au (c122-106-156-23.carlnfd1.nsw.optusnet.com.au [122.106.156.23]) by mail109.syd.optusnet.com.au (Postfix) with ESMTPS id F3A14121A09; Sun, 23 Jun 2013 20:29:13 +1000 (EST) Date: Sun, 23 Jun 2013 20:29:07 +1000 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: gabor@zahemszky.hu Subject: Re: ext2fs bug In-Reply-To: <9554b1ed02a342c347b17a2e71e38b73@zahemszky.hu> Message-ID: <20130623200524.Q2646@besplex.bde.org> References: <9554b1ed02a342c347b17a2e71e38b73@zahemszky.hu> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.0 cv=K8x6hFqI c=1 sm=1 a=kj9zAlcOel0A:10 a=PO7r1zJSAAAA:8 a=JzwRw_2MAAAA:8 a=z8ET0h8izKQA:10 a=_Ewsh7-fr2BQT7zXL60A:9 a=CjuIK1q_8ugA:10 a=6DVhyThRLIsPgGPl:21 a=GB2nAgaG2CrPr4dT:21 a=ebeQFi2P/qHVC0Yw9JDJ4g==:117 Cc: freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Jun 2013 10:29:23 -0000 On Sun, 23 Jun 2013 gabor@zahemszky.hu wrote: > I've got 2 disks (a 1TB and a 2TB), these disk were written under Linux 2.4. > I can mount them under FreeBSD, and I can reach about half of the files, but > on some files, I only get some FS errors. In dmesg, I can see the following > errors: It's very broken at 1TB, since it is still using a quick fix from 2002 that preserved the old limit of signed 32-bit block numbers at the vfs level when that limit was increased to 64 bits. > g_vfs_done():ada2s1[READ(offset=-756558786560, length=4096)]error = 5 > g_vfs_done():ada2s1[READ(offset=-899189821440, length=4096)]error = 5 > ... The brokeness includes not refusing to mount file systems that are too large to work. I wouldn't risk mounting read-writes file systems that are too large to work. 1TB should work though. That's in real TB (2**40 bytes). ext2fs also has internal 32-bit block numbers. With a block size of 4K, this should give a limit of 16TB, but FreeBSD uses negative block numbers magically, so the limit will be 8TB after the main bugs are fixed. With a block size of 1K, the limit will be 2TB. Bruce From owner-freebsd-fs@FreeBSD.ORG Sun Jun 23 11:51:02 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 583F8CD3 for ; Sun, 23 Jun 2013 11:51:02 +0000 (UTC) (envelope-from gabor@zahemszky.hu) Received: from default-smtp.integrity.hu (default-smtp.integrity.hu [212.52.165.203]) by mx1.freebsd.org (Postfix) with ESMTP id 1BF8819A8 for ; Sun, 23 Jun 2013 11:51:01 +0000 (UTC) Received: by smtp.integrity.hu (Postfix, from userid 10000) id 462C81346264; Sun, 23 Jun 2013 13:51:00 +0200 (CEST) Received: from webmail.integrity.hu (mail-fe-1.integrity.hu [10.1.64.120]) (Authenticated sender: gabor@zahemszky.hu) by smtp.integrity.hu (Postfix) with ESMTPA id B76981346247 for ; Sun, 23 Jun 2013 13:50:59 +0200 (CEST) Received: from LNBXvuNp+vkdvddD8n55x1V1449g/V1hMUR/pbyqTcBkDUbFzQEBnQ== (VC8v1wL7fCz6VgQYkYG4uG7cC2SGiDEZ) by webmail.integrity.hu with HTTP (HTTP/1.1 POST); Sun, 23 Jun 2013 13:50:59 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Date: Sun, 23 Jun 2013 13:50:59 +0200 From: gabor@zahemszky.hu To: Subject: Re: ext2fs bug In-Reply-To: <20130623095207.GA86507@icarus.home.lan> References: <9554b1ed02a342c347b17a2e71e38b73@zahemszky.hu> <20130623093521.GA86307@icarus.home.lan> <20130623095207.GA86507@icarus.home.lan> Message-ID: <216bbb1389bb7e6e5ef11500d31204ad@zahemszky.hu> X-Sender: gabor@zahemszky.hu User-Agent: Roundcube Webmail/0.8.4 X-Virus-Scanned: clamav-milter 0.97.6 at mail-autosubmit X-Virus-Status: Clean X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Jun 2013 11:51:02 -0000 2013-06-23 11:52 időpontban Jeremy Chadwick ezt írta: > On Sun, Jun 23, 2013 at 11:43:13AM +0200, gabor@zahemszky.hu wrote: >> Hi! >> >> >What FreeBSD version are you using? >> >> Sorry for the missing information: it's 9.1-p4 Amd64 >> >> >In the past 3-4 weeks there have been *tons* of ext2fs fixes in >> >stable/9, and I'm not exaggerating in the least. You will not find >> >these fixes being backported to 9.1-RELEASE or earlier. >> >> If they will be in 9.2 or in 10, it will be good. >> >> >I strongly recommend you try a recent stable/9 (i.e. past few days) >> >with ext2fs. You can try a stable/9 ("RELENG_9") ISO snapshot >> here: >> > >> >https://pub.allbsd.org/FreeBSD-snapshots/ Well, I downloaded: https://pub.allbsd.org/FreeBSD-snapshots/amd64-amd64/9.1-RELENG_9-r252093-JPSNAP/iso/FreeBSD-9.1-RELENG_9-r252093-JPSNAP-amd64-amd64-memstick.img booted my machine from a flash drive with that IMG. Mounted the same ext2 prtitions and got the same result. The same files are missing, and the same errors on the console. :-( What next? Thanks, Gabor From owner-freebsd-fs@FreeBSD.ORG Mon Jun 24 11:06:45 2013 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 47E10FE for ; Mon, 24 Jun 2013 11:06:45 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id 28D411DBE for ; Mon, 24 Jun 2013 11:06:45 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r5OB6iiO000961 for ; Mon, 24 Jun 2013 11:06:44 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r5OB6ivi000959 for freebsd-fs@FreeBSD.org; Mon, 24 Jun 2013 11:06:44 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 24 Jun 2013 11:06:44 GMT Message-Id: <201306241106.r5OB6ivi000959@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-fs@FreeBSD.org Subject: Current problem reports assigned to freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Jun 2013 11:06:45 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o bin/178996 fs [zfs] [patch] error in message with zfs mount -> there o kern/178854 fs [ufs] FreeBSD kernel crash in UFS o kern/178713 fs [nfs] [patch] Correct WebNFS support in NFS server and o kern/178412 fs [smbfs] Coredump when smbfs mounted o kern/178388 fs [zfs] [patch] allow up to 8MB recordsize o kern/178349 fs [zfs] zfs scrub on deduped data could be much less see o kern/178329 fs [zfs] extended attributes leak o kern/178238 fs [nullfs] nullfs don't release i-nodes on unlink. f kern/178231 fs [nfs] 8.3 nfsv4 client reports "nfsv4 client/server pr o kern/178103 fs [kernel] [nfs] [patch] Correct support of index files o kern/177985 fs [zfs] disk usage problem when copying from one zfs dat o kern/177971 fs [nfs] FreeBSD 9.1 nfs client dirlist problem w/ nfsv3, o kern/177966 fs [zfs] resilver completes but subsequent scrub reports o kern/177658 fs [ufs] FreeBSD panics after get full filesystem with uf o kern/177536 fs [zfs] zfs livelock (deadlock) with high write-to-disk o kern/177445 fs [hast] HAST panic o kern/177240 fs [zfs] zpool import failed with state UNAVAIL but all d o kern/176978 fs [zfs] [panic] zfs send -D causes "panic: System call i o kern/176857 fs [softupdates] [panic] 9.1-RELEASE/amd64/GENERIC panic o bin/176253 fs zpool(8): zfs pool indentation is misleading/wrong o kern/176141 fs [zfs] sharesmb=on makes errors for sharenfs, and still o kern/175950 fs [zfs] Possible deadlock in zfs after long uptime o kern/175897 fs [zfs] operations on readonly zpool hang o kern/175179 fs [zfs] ZFS may attach wrong device on move o kern/175071 fs [ufs] [panic] softdep_deallocate_dependencies: unrecov o kern/174372 fs [zfs] Pagefault appears to be related to ZFS o kern/174315 fs [zfs] chflags uchg not supported o kern/174310 fs [zfs] root point mounting broken on CURRENT with multi o kern/174279 fs [ufs] UFS2-SU+J journal and filesystem corruption o kern/173830 fs [zfs] Brain-dead simple change to ZFS error descriptio o kern/173718 fs [zfs] phantom directory in zraid2 pool f kern/173657 fs [nfs] strange UID map with nfsuserd o kern/173363 fs [zfs] [panic] Panic on 'zpool replace' on readonly poo o kern/173136 fs [unionfs] mounting above the NFS read-only share panic o kern/172942 fs [smbfs] Unmounting a smb mount when the server became o kern/172348 fs [unionfs] umount -f of filesystem in use with readonly o kern/172334 fs [unionfs] unionfs permits recursive union mounts; caus o kern/171626 fs [tmpfs] tmpfs should be noisier when the requested siz o kern/171415 fs [zfs] zfs recv fails with "cannot receive incremental o kern/170945 fs [gpt] disk layout not portable between direct connect o bin/170778 fs [zfs] [panic] FreeBSD panics randomly o kern/170680 fs [nfs] Multiple NFS Client bug in the FreeBSD 7.4-RELEA o kern/170497 fs [xfs][panic] kernel will panic whenever I ls a mounted o kern/169945 fs [zfs] [panic] Kernel panic while importing zpool (afte o kern/169480 fs [zfs] ZFS stalls on heavy I/O o kern/169398 fs [zfs] Can't remove file with permanent error o kern/169339 fs panic while " : > /etc/123" o kern/169319 fs [zfs] zfs resilver can't complete o kern/168947 fs [nfs] [zfs] .zfs/snapshot directory is messed up when o kern/168942 fs [nfs] [hang] nfsd hangs after being restarted (not -HU o kern/168158 fs [zfs] incorrect parsing of sharenfs options in zfs (fs o kern/167979 fs [ufs] DIOCGDINFO ioctl does not work on 8.2 file syste o kern/167977 fs [smbfs] mount_smbfs results are differ when utf-8 or U o kern/167688 fs [fusefs] Incorrect signal handling with direct_io o kern/167685 fs [zfs] ZFS on USB drive prevents shutdown / reboot o kern/167612 fs [portalfs] The portal file system gets stuck inside po o kern/167272 fs [zfs] ZFS Disks reordering causes ZFS to pick the wron o kern/167260 fs [msdosfs] msdosfs disk was mounted the second time whe o kern/167109 fs [zfs] [panic] zfs diff kernel panic Fatal trap 9: gene o kern/167105 fs [nfs] mount_nfs can not handle source exports wiht mor o kern/167067 fs [zfs] [panic] ZFS panics the server o kern/167065 fs [zfs] boot fails when a spare is the boot disk o kern/167048 fs [nfs] [patch] RELEASE-9 crash when using ZFS+NULLFS+NF o kern/166912 fs [ufs] [panic] Panic after converting Softupdates to jo o kern/166851 fs [zfs] [hang] Copying directory from the mounted UFS di o kern/166477 fs [nfs] NFS data corruption. o kern/165950 fs [ffs] SU+J and fsck problem o kern/165521 fs [zfs] [hang] livelock on 1 Gig of RAM with zfs when 31 o kern/165392 fs Multiple mkdir/rmdir fails with errno 31 o kern/165087 fs [unionfs] lock violation in unionfs o kern/164472 fs [ufs] fsck -B panics on particular data inconsistency o kern/164370 fs [zfs] zfs destroy for snapshot fails on i386 and sparc o kern/164261 fs [nullfs] [patch] fix panic with NFS served from NULLFS o kern/164256 fs [zfs] device entry for volume is not created after zfs o kern/164184 fs [ufs] [panic] Kernel panic with ufs_makeinode o kern/163801 fs [md] [request] allow mfsBSD legacy installed in 'swap' o kern/163770 fs [zfs] [hang] LOR between zfs&syncer + vnlru leading to o kern/163501 fs [nfs] NFS exporting a dir and a subdir in that dir to o kern/162944 fs [coda] Coda file system module looks broken in 9.0 o kern/162860 fs [zfs] Cannot share ZFS filesystem to hosts with a hyph o kern/162751 fs [zfs] [panic] kernel panics during file operations o kern/162591 fs [nullfs] cross-filesystem nullfs does not work as expe o kern/162519 fs [zfs] "zpool import" relies on buggy realpath() behavi o kern/161968 fs [zfs] [hang] renaming snapshot with -r including a zvo o kern/161864 fs [ufs] removing journaling from UFS partition fails on o kern/161579 fs [smbfs] FreeBSD sometimes panics when an smb share is o kern/161533 fs [zfs] [panic] zfs receive panic: system ioctl returnin o kern/161438 fs [zfs] [panic] recursed on non-recursive spa_namespace_ o kern/161424 fs [nullfs] __getcwd() calls fail when used on nullfs mou o kern/161280 fs [zfs] Stack overflow in gptzfsboot o kern/161205 fs [nfs] [pfsync] [regression] [build] Bug report freebsd o kern/161169 fs [zfs] [panic] ZFS causes kernel panic in dbuf_dirty o kern/161112 fs [ufs] [lor] filesystem LOR in FreeBSD 9.0-BETA3 o kern/160893 fs [zfs] [panic] 9.0-BETA2 kernel panic f kern/160860 fs [ufs] Random UFS root filesystem corruption with SU+J o kern/160801 fs [zfs] zfsboot on 8.2-RELEASE fails to boot from root-o o kern/160790 fs [fusefs] [panic] VPUTX: negative ref count with FUSE o kern/160777 fs [zfs] [hang] RAID-Z3 causes fatal hang upon scrub/impo o kern/160706 fs [zfs] zfs bootloader fails when a non-root vdev exists o kern/160591 fs [zfs] Fail to boot on zfs root with degraded raidz2 [r o kern/160410 fs [smbfs] [hang] smbfs hangs when transferring large fil o kern/160283 fs [zfs] [patch] 'zfs list' does abort in make_dataset_ha o kern/159930 fs [ufs] [panic] kernel core o kern/159402 fs [zfs][loader] symlinks cause I/O errors o kern/159357 fs [zfs] ZFS MAXNAMELEN macro has confusing name (off-by- o kern/159356 fs [zfs] [patch] ZFS NAME_ERR_DISKLIKE check is Solaris-s o kern/159351 fs [nfs] [patch] - divide by zero in mountnfs() o kern/159251 fs [zfs] [request]: add FLETCHER4 as DEDUP hash option o kern/159077 fs [zfs] Can't cd .. with latest zfs version o kern/159048 fs [smbfs] smb mount corrupts large files o kern/159045 fs [zfs] [hang] ZFS scrub freezes system o kern/158839 fs [zfs] ZFS Bootloader Fails if there is a Dead Disk o kern/158802 fs amd(8) ICMP storm and unkillable process. o kern/158231 fs [nullfs] panic on unmounting nullfs mounted over ufs o f kern/157929 fs [nfs] NFS slow read o kern/157399 fs [zfs] trouble with: mdconfig force delete && zfs strip o kern/157179 fs [zfs] zfs/dbuf.c: panic: solaris assert: arc_buf_remov o kern/156797 fs [zfs] [panic] Double panic with FreeBSD 9-CURRENT and o kern/156781 fs [zfs] zfs is losing the snapshot directory, p kern/156545 fs [ufs] mv could break UFS on SMP systems o kern/156193 fs [ufs] [hang] UFS snapshot hangs && deadlocks processes o kern/156039 fs [nullfs] [unionfs] nullfs + unionfs do not compose, re o kern/155615 fs [zfs] zfs v28 broken on sparc64 -current o kern/155587 fs [zfs] [panic] kernel panic with zfs p kern/155411 fs [regression] [8.2-release] [tmpfs]: mount: tmpfs : No o kern/155199 fs [ext2fs] ext3fs mounted as ext2fs gives I/O errors o bin/155104 fs [zfs][patch] use /dev prefix by default when importing o kern/154930 fs [zfs] cannot delete/unlink file from full volume -> EN o kern/154828 fs [msdosfs] Unable to create directories on external USB o kern/154491 fs [smbfs] smb_co_lock: recursive lock for object 1 p kern/154228 fs [md] md getting stuck in wdrain state o kern/153996 fs [zfs] zfs root mount error while kernel is not located o kern/153753 fs [zfs] ZFS v15 - grammatical error when attempting to u o kern/153716 fs [zfs] zpool scrub time remaining is incorrect o kern/153695 fs [patch] [zfs] Booting from zpool created on 4k-sector o kern/153680 fs [xfs] 8.1 failing to mount XFS partitions o kern/153418 fs [zfs] [panic] Kernel Panic occurred writing to zfs vol o kern/153351 fs [zfs] locking directories/files in ZFS o bin/153258 fs [patch][zfs] creating ZVOLs requires `refreservation' s kern/153173 fs [zfs] booting from a gzip-compressed dataset doesn't w o bin/153142 fs [zfs] ls -l outputs `ls: ./.zfs: Operation not support o kern/153126 fs [zfs] vdev failure, zpool=peegel type=vdev.too_small o kern/152022 fs [nfs] nfs service hangs with linux client [regression] o kern/151942 fs [zfs] panic during ls(1) zfs snapshot directory o kern/151905 fs [zfs] page fault under load in /sbin/zfs o bin/151713 fs [patch] Bug in growfs(8) with respect to 32-bit overfl o kern/151648 fs [zfs] disk wait bug o kern/151629 fs [fs] [patch] Skip empty directory entries during name o kern/151330 fs [zfs] will unshare all zfs filesystem after execute a o kern/151326 fs [nfs] nfs exports fail if netgroups contain duplicate o kern/151251 fs [ufs] Can not create files on filesystem with heavy us o kern/151226 fs [zfs] can't delete zfs snapshot o kern/150503 fs [zfs] ZFS disks are UNAVAIL and corrupted after reboot o kern/150501 fs [zfs] ZFS vdev failure vdev.bad_label on amd64 o kern/150390 fs [zfs] zfs deadlock when arcmsr reports drive faulted o kern/150336 fs [nfs] mountd/nfsd became confused; refused to reload n o kern/149208 fs mksnap_ffs(8) hang/deadlock o kern/149173 fs [patch] [zfs] make OpenSolaris installa o kern/149015 fs [zfs] [patch] misc fixes for ZFS code to build on Glib o kern/149014 fs [zfs] [patch] declarations in ZFS libraries/utilities o kern/149013 fs [zfs] [patch] make ZFS makefiles use the libraries fro o kern/148504 fs [zfs] ZFS' zpool does not allow replacing drives to be o kern/148490 fs [zfs]: zpool attach - resilver bidirectionally, and re o kern/148368 fs [zfs] ZFS hanging forever on 8.1-PRERELEASE o kern/148138 fs [zfs] zfs raidz pool commands freeze o kern/147903 fs [zfs] [panic] Kernel panics on faulty zfs device o kern/147881 fs [zfs] [patch] ZFS "sharenfs" doesn't allow different " o kern/147420 fs [ufs] [panic] ufs_dirbad, nullfs, jail panic (corrupt o kern/146941 fs [zfs] [panic] Kernel Double Fault - Happens constantly o kern/146786 fs [zfs] zpool import hangs with checksum errors o kern/146708 fs [ufs] [panic] Kernel panic in softdep_disk_write_compl o kern/146528 fs [zfs] Severe memory leak in ZFS on i386 o kern/146502 fs [nfs] FreeBSD 8 NFS Client Connection to Server s kern/145712 fs [zfs] cannot offline two drives in a raidz2 configurat o kern/145411 fs [xfs] [panic] Kernel panics shortly after mounting an f bin/145309 fs bsdlabel: Editing disk label invalidates the whole dev o kern/145272 fs [zfs] [panic] Panic during boot when accessing zfs on o kern/145246 fs [ufs] dirhash in 7.3 gratuitously frees hashes when it o kern/145238 fs [zfs] [panic] kernel panic on zpool clear tank o kern/145229 fs [zfs] Vast differences in ZFS ARC behavior between 8.0 o kern/145189 fs [nfs] nfsd performs abysmally under load o kern/144929 fs [ufs] [lor] vfs_bio.c + ufs_dirhash.c p kern/144447 fs [zfs] sharenfs fsunshare() & fsshare_main() non functi o kern/144416 fs [panic] Kernel panic on online filesystem optimization s kern/144415 fs [zfs] [panic] kernel panics on boot after zfs crash o kern/144234 fs [zfs] Cannot boot machine with recent gptzfsboot code o kern/143825 fs [nfs] [panic] Kernel panic on NFS client o bin/143572 fs [zfs] zpool(1): [patch] The verbose output from iostat o kern/143212 fs [nfs] NFSv4 client strange work ... o kern/143184 fs [zfs] [lor] zfs/bufwait LOR o kern/142878 fs [zfs] [vfs] lock order reversal o kern/142597 fs [ext2fs] ext2fs does not work on filesystems with real o kern/142489 fs [zfs] [lor] allproc/zfs LOR o kern/142466 fs Update 7.2 -> 8.0 on Raid 1 ends with screwed raid [re o kern/142306 fs [zfs] [panic] ZFS drive (from OSX Leopard) causes two o kern/142068 fs [ufs] BSD labels are got deleted spontaneously o kern/141897 fs [msdosfs] [panic] Kernel panic. msdofs: file name leng o kern/141463 fs [nfs] [panic] Frequent kernel panics after upgrade fro o kern/141305 fs [zfs] FreeBSD ZFS+sendfile severe performance issues ( o kern/141091 fs [patch] [nullfs] fix panics with DIAGNOSTIC enabled o kern/141086 fs [nfs] [panic] panic("nfs: bioread, not dir") on FreeBS o kern/141010 fs [zfs] "zfs scrub" fails when backed by files in UFS2 o kern/140888 fs [zfs] boot fail from zfs root while the pool resilveri o kern/140661 fs [zfs] [patch] /boot/loader fails to work on a GPT/ZFS- o kern/140640 fs [zfs] snapshot crash o kern/140068 fs [smbfs] [patch] smbfs does not allow semicolon in file o kern/139725 fs [zfs] zdb(1) dumps core on i386 when examining zpool c o kern/139715 fs [zfs] vfs.numvnodes leak on busy zfs p bin/139651 fs [nfs] mount(8): read-only remount of NFS volume does n o kern/139407 fs [smbfs] [panic] smb mount causes system crash if remot o kern/138662 fs [panic] ffs_blkfree: freeing free block o kern/138421 fs [ufs] [patch] remove UFS label limitations o kern/138202 fs mount_msdosfs(1) see only 2Gb o kern/136968 fs [ufs] [lor] ufs/bufwait/ufs (open) o kern/136945 fs [ufs] [lor] filedesc structure/ufs (poll) o kern/136944 fs [ffs] [lor] bufwait/snaplk (fsync) o kern/136873 fs [ntfs] Missing directories/files on NTFS volume o kern/136865 fs [nfs] [patch] NFS exports atomic and on-the-fly atomic p kern/136470 fs [nfs] Cannot mount / in read-only, over NFS o kern/135546 fs [zfs] zfs.ko module doesn't ignore zpool.cache filenam o kern/135469 fs [ufs] [panic] kernel crash on md operation in ufs_dirb o kern/135050 fs [zfs] ZFS clears/hides disk errors on reboot o kern/134491 fs [zfs] Hot spares are rather cold... o kern/133676 fs [smbfs] [panic] umount -f'ing a vnode-based memory dis p kern/133174 fs [msdosfs] [patch] msdosfs must support multibyte inter o kern/132960 fs [ufs] [panic] panic:ffs_blkfree: freeing free frag o kern/132397 fs reboot causes filesystem corruption (failure to sync b o kern/132331 fs [ufs] [lor] LOR ufs and syncer o kern/132237 fs [msdosfs] msdosfs has problems to read MSDOS Floppy o kern/132145 fs [panic] File System Hard Crashes o kern/131441 fs [unionfs] [nullfs] unionfs and/or nullfs not combineab o kern/131360 fs [nfs] poor scaling behavior of the NFS server under lo o kern/131342 fs [nfs] mounting/unmounting of disks causes NFS to fail o bin/131341 fs makefs: error "Bad file descriptor" on the mount poin o kern/130920 fs [msdosfs] cp(1) takes 100% CPU time while copying file o kern/130210 fs [nullfs] Error by check nullfs o kern/129760 fs [nfs] after 'umount -f' of a stale NFS share FreeBSD l o kern/129488 fs [smbfs] Kernel "bug" when using smbfs in smbfs_smb.c: o kern/129231 fs [ufs] [patch] New UFS mount (norandom) option - mostly o kern/129152 fs [panic] non-userfriendly panic when trying to mount(8) o kern/127787 fs [lor] [ufs] Three LORs: vfslock/devfs/vfslock, ufs/vfs o bin/127270 fs fsck_msdosfs(8) may crash if BytesPerSec is zero o kern/127029 fs [panic] mount(8): trying to mount a write protected zi o kern/126287 fs [ufs] [panic] Kernel panics while mounting an UFS file o kern/125895 fs [ffs] [panic] kernel: panic: ffs_blkfree: freeing free s kern/125738 fs [zfs] [request] SHA256 acceleration in ZFS o kern/123939 fs [msdosfs] corrupts new files o kern/122380 fs [ffs] ffs_valloc:dup alloc (Soekris 4801/7.0/USB Flash o bin/122172 fs [fs]: amd(8) automount daemon dies on 6.3-STABLE i386, o bin/121898 fs [nullfs] pwd(1)/getcwd(2) fails with Permission denied o bin/121072 fs [smbfs] mount_smbfs(8) cannot normally convert the cha o kern/120483 fs [ntfs] [patch] NTFS filesystem locking changes o kern/120482 fs [ntfs] [patch] Sync style changes between NetBSD and F o kern/118912 fs [2tb] disk sizing/geometry problem with large array o kern/118713 fs [minidump] [patch] Display media size required for a k o kern/118318 fs [nfs] NFS server hangs under special circumstances o bin/118249 fs [ufs] mv(1): moving a directory changes its mtime o kern/118126 fs [nfs] [patch] Poor NFS server write performance o kern/118107 fs [ntfs] [panic] Kernel panic when accessing a file at N o kern/117954 fs [ufs] dirhash on very large directories blocks the mac o bin/117315 fs [smbfs] mount_smbfs(8) and related options can't mount o kern/117158 fs [zfs] zpool scrub causes panic if geli vdevs detach on o bin/116980 fs [msdosfs] [patch] mount_msdosfs(8) resets some flags f o conf/116931 fs lack of fsck_cd9660 prevents mounting iso images with o kern/116583 fs [ffs] [hang] System freezes for short time when using o bin/115361 fs [zfs] mount(8) gets into a state where it won't set/un o kern/114955 fs [cd9660] [patch] [request] support for mask,dirmask,ui o kern/114847 fs [ntfs] [patch] [request] dirmask support for NTFS ala o kern/114676 fs [ufs] snapshot creation panics: snapacct_ufs2: bad blo o bin/114468 fs [patch] [request] add -d option to umount(8) to detach o kern/113852 fs [smbfs] smbfs does not properly implement DFS referral o bin/113838 fs [patch] [request] mount(8): add support for relative p o bin/113049 fs [patch] [request] make quot(8) use getopt(3) and show o kern/112658 fs [smbfs] [patch] smbfs and caching problems (resolves b o kern/111843 fs [msdosfs] Long Names of files are incorrectly created o kern/111782 fs [ufs] dump(8) fails horribly for large filesystems s bin/111146 fs [2tb] fsck(8) fails on 6T filesystem o bin/107829 fs [2TB] fdisk(8): invalid boundary checking in fdisk / w o kern/106107 fs [ufs] left-over fsck_snapshot after unfinished backgro o kern/104406 fs [ufs] Processes get stuck in "ufs" state under persist o kern/104133 fs [ext2fs] EXT2FS module corrupts EXT2/3 filesystems o kern/103035 fs [ntfs] Directories in NTFS mounted disc images appear o kern/101324 fs [smbfs] smbfs sometimes not case sensitive when it's s o kern/99290 fs [ntfs] mount_ntfs ignorant of cluster sizes s bin/97498 fs [request] newfs(8) has no option to clear the first 12 o kern/97377 fs [ntfs] [patch] syntax cleanup for ntfs_ihash.c o kern/95222 fs [cd9660] File sections on ISO9660 level 3 CDs ignored o kern/94849 fs [ufs] rename on UFS filesystem is not atomic o bin/94810 fs fsck(8) incorrectly reports 'file system marked clean' o kern/94769 fs [ufs] Multiple file deletions on multi-snapshotted fil o kern/94733 fs [smbfs] smbfs may cause double unlock o kern/93942 fs [vfs] [patch] panic: ufs_dirbad: bad dir (patch from D o kern/92272 fs [ffs] [hang] Filling a filesystem while creating a sna o kern/91134 fs [smbfs] [patch] Preserve access and modification time a kern/90815 fs [smbfs] [patch] SMBFS with character conversions somet o kern/88657 fs [smbfs] windows client hang when browsing a samba shar o kern/88555 fs [panic] ffs_blkfree: freeing free frag on AMD 64 o bin/87966 fs [patch] newfs(8): introduce -A flag for newfs to enabl o kern/87859 fs [smbfs] System reboot while umount smbfs. o kern/86587 fs [msdosfs] rm -r /PATH fails with lots of small files o bin/85494 fs fsck_ffs: unchecked use of cg_inosused macro etc. o kern/80088 fs [smbfs] Incorrect file time setting on NTFS mounted vi o bin/74779 fs Background-fsck checks one filesystem twice and omits o kern/73484 fs [ntfs] Kernel panic when doing `ls` from the client si o bin/73019 fs [ufs] fsck_ufs(8) cannot alloc 607016868 bytes for ino o kern/71774 fs [ntfs] NTFS cannot "see" files on a WinXP filesystem o bin/70600 fs fsck(8) throws files away when it can't grow lost+foun o kern/68978 fs [panic] [ufs] crashes with failing hard disk, loose po o kern/65920 fs [nwfs] Mounted Netware filesystem behaves strange o kern/65901 fs [smbfs] [patch] smbfs fails fsx write/truncate-down/tr o kern/61503 fs [smbfs] mount_smbfs does not work as non-root o kern/55617 fs [smbfs] Accessing an nsmb-mounted drive via a smb expo o kern/51685 fs [hang] Unbounded inode allocation causes kernel to loc o kern/36566 fs [smbfs] System reboot with dead smb mount and umount o bin/27687 fs fsck(8) wrapper is not properly passing options to fsc o kern/18874 fs [2TB] 32bit NFS servers export wrong negative values t 316 problems total. From owner-freebsd-fs@FreeBSD.ORG Tue Jun 25 11:37:30 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 8057D26E for ; Tue, 25 Jun 2013 11:37:30 +0000 (UTC) (envelope-from thomas@gibfest.dk) Received: from mail.tyknet.dk (mail.tyknet.dk [176.9.9.186]) by mx1.freebsd.org (Postfix) with ESMTP id 404E613B8 for ; Tue, 25 Jun 2013 11:37:30 +0000 (UTC) Received: from [10.10.1.100] (217.71.4.82.static.router4.bolignet.dk [217.71.4.82]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mail.tyknet.dk (Postfix) with ESMTPSA id AC44D14F3F5 for ; Tue, 25 Jun 2013 13:27:52 +0200 (CEST) X-DKIM: OpenDKIM Filter v2.5.2 mail.tyknet.dk AC44D14F3F5 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=gibfest.dk; s=default; t=1372159673; bh=ksWTq+dCKZacoxK+NskGkvRSRP26vqJmFxy02kk5oOY=; h=Date:From:To:Subject; b=Me6eHiDgejpytl/wHcRcTQY8aIZ1MrzU85Sy2/tyA/T9sQwL4aiG6QGWRTt2aYCIC gs1k1Gy1kwJCEKuiGzGmS0A+8/QpZigpSJT8k6brgb7xtHu6574x8veNhiRrlvvs7Q hi5rju8uswuIFeqHLLdAxlEhl2jcwP5uPSdltEhg= Message-ID: <51C97EAF.3000901@gibfest.dk> Date: Tue, 25 Jun 2013 13:27:43 +0200 From: Thomas Steen Rasmussen User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20130509 Thunderbird/17.0.6 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: Reproducible ZFS jailed dataset panic after upgrading to latest 9-stable Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Jun 2013 11:37:30 -0000 Hello, To fix the mmap vulnerability I've upgraded one of my jail hosts from: "FreeBSD 9.1-STABLE #1: Sun Mar 17 08:48:35 UTC 2013" to: "FreeBSD 9.1-STABLE #3: Tue Jun 18 12:49:39 UTC 2013" One of the jails on this machine has a jailed zfs dataset: $ zfs get jailed gelipool/backups NAME PROPERTY VALUE SOURCE gelipool/backups jailed on local $ After the upgrade, when I start the jail, the machine panics. This is a remote zfs-only machine with swap on zfs, so far I have been unable to get a proper coredump. I have access to the console of the machine, and I have taken a couple of screenshots: http://imgur.com/2V0PBlf and http://imgur.com/OopP9Sp Any ideas what might have caused this ? It worked great before the upgrade to latest 9-STABLE. This is a production server, but I am willing to try any suggestions to get it working again. Thanks! Best regards, Thomas Steen Rasmussen ps. I have ordered a USB stick for the server which I will use to get a coredump, but I don't know when Hetzner will get around to it. From owner-freebsd-fs@FreeBSD.ORG Tue Jun 25 11:44:15 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 634F843C for ; Tue, 25 Jun 2013 11:44:15 +0000 (UTC) (envelope-from kp@sigsegv.be) Received: from mercury.codepro.be (mercury.codepro.be [95.142.164.132]) by mx1.freebsd.org (Postfix) with ESMTP id 2821515CA for ; Tue, 25 Jun 2013 11:44:14 +0000 (UTC) Received: from adrastea.jupiter.sigsegv.be (unknown [IPv6:2001:6f8:1498:1::3]) by mercury.codepro.be (Postfix) with ESMTP id C24DC165; Tue, 25 Jun 2013 13:44:07 +0200 (CEST) Received: from thebe.jupiter.sigsegv.be (thebe.jupiter.sigsegv.be [172.16.1.5]) by adrastea.jupiter.sigsegv.be (Postfix) with ESMTP id 704693BFB; Tue, 25 Jun 2013 13:44:07 +0200 (CEST) Received: by thebe.jupiter.sigsegv.be (Postfix, from userid 1000) id 242761C387; Tue, 25 Jun 2013 13:44:07 +0200 (CEST) Date: Tue, 25 Jun 2013 13:44:06 +0200 From: Kristof Provost To: Thomas Steen Rasmussen Subject: Re: Reproducible ZFS jailed dataset panic after upgrading to latest 9-stable Message-ID: <20130625114405.GB9254@thebe.jupiter.sigsegv.be> References: <51C97EAF.3000901@gibfest.dk> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <51C97EAF.3000901@gibfest.dk> X-PGP-Fingerprint: E114 D9EA 909E D469 8F57 17A5 7D15 91C6 9EFA F286 User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Jun 2013 11:44:15 -0000 On 2013-06-25 13:27:43 (+0200), Thomas Steen Rasmussen wrote: > Hello, > > To fix the mmap vulnerability I've upgraded one of my jail hosts from: > "FreeBSD 9.1-STABLE #1: Sun Mar 17 08:48:35 UTC 2013" > to: > "FreeBSD 9.1-STABLE #3: Tue Jun 18 12:49:39 UTC 2013" > > One of the jails on this machine has a jailed zfs dataset: > > $ zfs get jailed gelipool/backups > NAME PROPERTY VALUE SOURCE > gelipool/backups jailed on local > $ > > After the upgrade, when I start the jail, the machine panics. > > This is a remote zfs-only machine with swap on zfs, so far I have > been unable to get a proper coredump. I have access to the > console of the machine, and I have taken a couple of screenshots: > > http://imgur.com/2V0PBlf and http://imgur.com/OopP9Sp > > Any ideas what might have caused this ? It worked great before the > upgrade to latest 9-STABLE. This is a production server, but I am > willing to try any suggestions to get it working again. > I think you're hitting the same issue as me and Alexander Leidinger. Alexander said that the maintainer of the stress-test suite has managed to create a test case to trigger the issue, so hopefully a fix will be found soon. http://docs.freebsd.org/cgi/getmsg.cgi?fetch=250572+0+/usr/local/www/db/text/2013/freebsd-current/20130616.freebsd-current Regards, Kristof From owner-freebsd-fs@FreeBSD.ORG Tue Jun 25 11:50:58 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 2667772C for ; Tue, 25 Jun 2013 11:50:58 +0000 (UTC) (envelope-from prvs=1888902647=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id BFA741620 for ; Tue, 25 Jun 2013 11:50:57 +0000 (UTC) Received: from r2d2 ([82.69.141.170]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50004510795.msg for ; Tue, 25 Jun 2013 12:50:50 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Tue, 25 Jun 2013 12:50:50 +0100 (not processed: message from valid local sender) X-MDDKIM-Result: neutral (mail1.multiplay.co.uk) X-MDRemoteIP: 82.69.141.170 X-Return-Path: prvs=1888902647=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-fs@freebsd.org Message-ID: <0A183A77E7E740F69EDA19732CB06E39@multiplay.co.uk> From: "Steven Hartland" To: "Thomas Steen Rasmussen" , References: <51C97EAF.3000901@gibfest.dk> Subject: Re: Reproducible ZFS jailed dataset panic after upgrading to latest 9-stable Date: Tue, 25 Jun 2013 12:50:51 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Jun 2013 11:50:58 -0000 ----- Original Message ----- From: "Thomas Steen Rasmussen" To: Sent: Tuesday, June 25, 2013 12:27 PM Subject: Reproducible ZFS jailed dataset panic after upgrading to latest 9-stable > Hello, > > To fix the mmap vulnerability I've upgraded one of my jail hosts from: > "FreeBSD 9.1-STABLE #1: Sun Mar 17 08:48:35 UTC 2013" > to: > "FreeBSD 9.1-STABLE #3: Tue Jun 18 12:49:39 UTC 2013" > > One of the jails on this machine has a jailed zfs dataset: > > $ zfs get jailed gelipool/backups > NAME PROPERTY VALUE SOURCE > gelipool/backups jailed on local > $ > > After the upgrade, when I start the jail, the machine panics. > > This is a remote zfs-only machine with swap on zfs, so far I have > been unable to get a proper coredump. I have access to the > console of the machine, and I have taken a couple of screenshots: > > http://imgur.com/2V0PBlf and http://imgur.com/OopP9Sp > > Any ideas what might have caused this ? It worked great before the > upgrade to latest 9-STABLE. This is a production server, but I am > willing to try any suggestions to get it working again. > > Thanks! > > Best regards, > > Thomas Steen Rasmussen > > ps. I have ordered a USB stick for the server which I will use to get a > coredump, but I don't know when Hetzner will get around to it. What where you running before Thomas? Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-fs@FreeBSD.ORG Tue Jun 25 11:56:27 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id B0FDE902 for ; Tue, 25 Jun 2013 11:56:27 +0000 (UTC) (envelope-from kp@sigsegv.be) Received: from mercury.codepro.be (mercury.codepro.be [IPv6:2001:4b98:dc0:51:216:3eff:feb7:3147]) by mx1.freebsd.org (Postfix) with ESMTP id 74DC8167F for ; Tue, 25 Jun 2013 11:56:27 +0000 (UTC) Received: from adrastea.jupiter.sigsegv.be (unknown [IPv6:2001:6f8:1498:1::3]) by mercury.codepro.be (Postfix) with ESMTP id E1E2E165; Tue, 25 Jun 2013 13:56:25 +0200 (CEST) Received: from thebe.jupiter.sigsegv.be (thebe.jupiter.sigsegv.be [172.16.1.5]) by adrastea.jupiter.sigsegv.be (Postfix) with ESMTP id 860673C1B; Tue, 25 Jun 2013 13:56:25 +0200 (CEST) Received: by thebe.jupiter.sigsegv.be (Postfix, from userid 1000) id 1629B1C3CB; Tue, 25 Jun 2013 13:56:25 +0200 (CEST) Date: Tue, 25 Jun 2013 13:56:24 +0200 From: Kristof Provost To: Steven Hartland Subject: Re: Reproducible ZFS jailed dataset panic after upgrading to latest 9-stable Message-ID: <20130625115624.GC9254@thebe.jupiter.sigsegv.be> References: <51C97EAF.3000901@gibfest.dk> <0A183A77E7E740F69EDA19732CB06E39@multiplay.co.uk> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <0A183A77E7E740F69EDA19732CB06E39@multiplay.co.uk> X-PGP-Fingerprint: E114 D9EA 909E D469 8F57 17A5 7D15 91C6 9EFA F286 User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Jun 2013 11:56:27 -0000 On 2013-06-25 12:50:51 (+0100), Steven Hartland wrote: > From: "Thomas Steen Rasmussen" > > After the upgrade, when I start the jail, the machine panics. > > > > This is a remote zfs-only machine with swap on zfs, so far I have > > been unable to get a proper coredump. I have access to the > > console of the machine, and I have taken a couple of screenshots: > > What where you running before Thomas? > I don't know about Thomas, but the last version that works for me (that I know of) is stable/9 r249415 Regards, Kristof From owner-freebsd-fs@FreeBSD.ORG Tue Jun 25 19:22:49 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id C3B0965B for ; Tue, 25 Jun 2013 19:22:49 +0000 (UTC) (envelope-from mxb@alumni.chalmers.se) Received: from mail-lb0-x234.google.com (mail-lb0-x234.google.com [IPv6:2a00:1450:4010:c04::234]) by mx1.freebsd.org (Postfix) with ESMTP id 4476E1F35 for ; Tue, 25 Jun 2013 19:22:48 +0000 (UTC) Received: by mail-lb0-f180.google.com with SMTP id o10so1100195lbi.39 for ; Tue, 25 Jun 2013 12:22:47 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer :x-gm-message-state; bh=3gncpQNjN76vdmVXPvZDG9Ue/Z3yrOBTP24vR51N2vs=; b=ONpAqmjsO4NdjcDfKvWFveDXacruC4/J+9oZJeISYKpwelbazzflSdTjK6kAwd97rq dvmESr5VAURAnUzhUKa7CuxkPz377vnFikbLijg/uMFZXZF9dYjMtqUBrNJptvr/iqVn mbM9NEzOnm63BgRIl9aYhz8BgqdmFhRi+aFmSk8NULnaoD0IMq6XTYUXeVB9RAs1wIQO cwIjTjRNvOtfr3hPyrqC0eMPNahF6ZvYTPespQoAB7scPSuHNCCEH1hdHPIAJN34NS2B Ylqs/TQ0xowScsRflgui8Ba6G+Z008YnZkJJAZUjwlrOllqBx++UdgIPWk88GOWhFhLT E7Rw== X-Received: by 10.112.125.199 with SMTP id ms7mr573591lbb.29.1372188167277; Tue, 25 Jun 2013 12:22:47 -0700 (PDT) Received: from grey.home.unixconn.com (h-75-17.a183.priv.bahnhof.se. [46.59.75.17]) by mx.google.com with ESMTPSA id f8sm9292240lbr.10.2013.06.25.12.22.44 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 25 Jun 2013 12:22:45 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.5 \(1508\)) Subject: Re: zpool export/import on failover - The pool metadata is corrupted From: mxb In-Reply-To: <09717048-12BE-474B-9B20-F5E72D00152E@alumni.chalmers.se> Date: Tue, 25 Jun 2013 21:22:43 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <5A26ABDE-C7F2-41CC-A3D1-69310AB6BC36@alumni.chalmers.se> References: <016B635E-4EDC-4CDF-AC58-82AC39CBFF56@alumni.chalmers.se> <20130606223911.GA45807@icarus.home.lan> <20130606233417.GA46506@icarus.home.lan> <61E414CF-FCD3-42BB-9533-A40EA934DB99@alumni.chalmers.se> <09717048-12BE-474B-9B20-F5E72D00152E@alumni.chalmers.se> To: Jeremy Chadwick X-Mailer: Apple Mail (2.1508) X-Gm-Message-State: ALoCoQmZNUBWZzqpuwTRGyup5ZAPxjsulkP7SFMweGACJ54Qzp/Rl1RP14fmsxXeN7V/ekPtzuBj Cc: "freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Jun 2013 19:22:49 -0000 I think I'v found the root of this issue. Looks like "wiring down" disks the same way on both nodes (as suggested) = fixes this issue. //mxb On 20 jun 2013, at 12:30, mxb wrote: >=20 > Well, >=20 > I'm back to square one. >=20 > After some uptime and successful import/export from one node to = another, I eventually got 'metadata corruption'. > I had no problem with import/export while for ex. rebooting = master-node (nfs1), but not THIS time. > Metdata got corrupted while rebooting master-node?? >=20 > Any ideas?=20 >=20 > [root@nfs1 ~]# zpool import > pool: jbod > id: 7663925948774378610 > state: FAULTED > status: The pool metadata is corrupted. > action: The pool cannot be imported due to damaged devices or data. > see: http://illumos.org/msg/ZFS-8000-72 > config: >=20 > jbod FAULTED corrupted data > raidz3-0 ONLINE > da3 ONLINE > da4 ONLINE > da5 ONLINE > da6 ONLINE > da7 ONLINE > da8 ONLINE > da9 ONLINE > da10 ONLINE > da11 ONLINE > da12 ONLINE > cache > da13s2 > da14s2 > logs > mirror-1 ONLINE > da13s1 ONLINE > da14s1 ONLINE > [root@nfs1 ~]# zpool import jbod > cannot import 'jbod': I/O error > Destroy and re-create the pool from > a backup source. > [root@nfs1 ~]# >=20 > On 11 jun 2013, at 10:46, mxb wrote: >=20 >>=20 >> Thanks everyone whom replied. >> Removing local L2ARC cache disks (da1,da2) indeed showed to be a cure = to my problem. >>=20 >> Next is to test with add/remove after import/export as Jeremy = suggested. >>=20 >> //mxb >>=20 >> On 7 jun 2013, at 01:34, Jeremy Chadwick wrote: >>=20 >>> On Fri, Jun 07, 2013 at 12:51:14AM +0200, mxb wrote: >>>>=20 >>>> Sure, script is not perfects yet and does not handle many of stuff, = but moving highlight from zpool import/export to the script itself not = that >>>> clever,as this works most of the time. >>>>=20 >>>> Question is WHY ZFS corrupts metadata then it should not. = Sometimes. >>>> I'v seen stale of zpool then manually importing/exporting pool. >>>>=20 >>>>=20 >>>> On 7 jun 2013, at 00:39, Jeremy Chadwick wrote: >>>>=20 >>>>> On Fri, Jun 07, 2013 at 12:12:39AM +0200, mxb wrote: >>>>>>=20 >>>>>> Then MASTER goes down, CARP on the second node goes MASTER = (devd.conf, and script for lifting): >>>>>>=20 >>>>>> root@nfs2:/root # cat /etc/devd.conf >>>>>>=20 >>>>>>=20 >>>>>> notify 30 { >>>>>> match "system" "IFNET"; >>>>>> match "subsystem" "carp0"; >>>>>> match "type" "LINK_UP"; >>>>>> action "/etc/zfs_switch.sh active"; >>>>>> }; >>>>>>=20 >>>>>> notify 30 { >>>>>> match "system" "IFNET"; >>>>>> match "subsystem" "carp0"; >>>>>> match "type" "LINK_DOWN"; >>>>>> action "/etc/zfs_switch.sh backup"; >>>>>> }; >>>>>>=20 >>>>>> root@nfs2:/root # cat /etc/zfs_switch.sh >>>>>> #!/bin/sh >>>>>>=20 >>>>>> DATE=3D`date +%Y%m%d` >>>>>> HOSTNAME=3D`hostname` >>>>>>=20 >>>>>> ZFS_POOL=3D"jbod" >>>>>>=20 >>>>>>=20 >>>>>> case $1 in >>>>>> active) >>>>>> echo "Switching to ACTIVE and importing ZFS" | mail -s = ''$DATE': '$HOSTNAME' switching to ACTIVE' root >>>>>> sleep 10 >>>>>> /sbin/zpool import -f jbod >>>>>> /etc/rc.d/mountd restart >>>>>> /etc/rc.d/nfsd restart >>>>>> ;; >>>>>> backup) >>>>>> echo "Switching to BACKUP and exporting ZFS" | mail -s = ''$DATE': '$HOSTNAME' switching to BACKUP' root >>>>>> /sbin/zpool export jbod >>>>>> /etc/rc.d/mountd restart >>>>>> /etc/rc.d/nfsd restart >>>>>> ;; >>>>>> *) >>>>>> exit 0 >>>>>> ;; >>>>>> esac >>>>>>=20 >>>>>> This works, most of the time, but sometimes I'm forced to = re-create pool. Those machines suppose to go into prod. >>>>>> Loosing pool(and data inside it) stops me from deploy this setup. >>>>>=20 >>>>> This script looks highly error-prone. Hasty hasty... :-) >>>>>=20 >>>>> This script assumes that the "zpool" commands (import and export) = always >>>>> work/succeed; there is no exit code ($?) checking being used. >>>>>=20 >>>>> Since this is run from within devd(8): where does stdout/stderr go = to >>>>> when running a program/script under devd(8)? Does it effectively = go >>>>> to the bit bucket (/dev/null)? If so, you'd never know if the = import or >>>>> export actually succeeded or not (the export sounds more likely to = be >>>>> the problem point). >>>>>=20 >>>>> I imagine there would be some situations where the export would = fail >>>>> (some files on filesystems under pool "jbod" still in use), yet = CARP is >>>>> already blindly assuming everything will be fantastic. Surprise. >>>>>=20 >>>>> I also do not know if devd.conf(5) "action" commands spawn a = sub-shell >>>>> (/bin/sh) or not. If they don't, you won't be able to use things = like" >>>>> 'action "/etc/zfs_switch.sh active >> /var/log/failover.log";'. = You >>>>> would then need to implement the equivalent of logging within your >>>>> zfs_switch.sh script. >>>>>=20 >>>>> You may want to consider the -f flag to zpool import/export >>>>> (particularly export). However there are risks involved -- = userland >>>>> applications which have an fd/fh open on a file which is stored on = a >>>>> filesystem that has now completely disappeared can sometimes crash >>>>> (segfault) or behave very oddly (100% CPU usage, etc.) depending = on how >>>>> they're designed. >>>>>=20 >>>>> Basically what I'm trying to say is that devd(8) being used as a = form of >>>>> HA (high availability) and load balancing is not always possible. >>>>> Real/true HA (especially with SANs) is often done very differently = (now >>>>> you know why it's often proprietary. :-) ) >>>=20 >>> Add error checking to your script. That's my first and foremost >>> recommendation. It's not hard to do, really. :-) >>>=20 >>> After you do that and still experience the issue (e.g. you see no = actual >>> errors/issues during the export/import phases), I recommend removing >>> the "cache" devices which are "independent" on each system from the = pool >>> entirely. Quoting you (for readers, since I snipped it from my = previous >>> reply): >>>=20 >>>>>> Note, that ZIL(mirrored) resides on external enclosure. Only = L2ARC >>>>>> is both local and external - da1,da2, da13s2, da14s2 >>>=20 >>> I interpret this to mean the primary and backup nodes (physical = systems) >>> have actual disks which are not part of the "external enclosure". = If >>> that's the case -- those disks are always going to vary in their >>> contents and metadata. Those are never going to be 100% identical = all >>> the time (is this not obvious?). I'm surprised your stuff has = worked at >>> all using that model, honestly. >>>=20 >>> ZFS is going to bitch/cry if it cannot verify the integrity of = certain >>> things, all the way down to the L2ARC. That's my understanding of = it at >>> least, meaning there must always be "some" kind of metadata that has = to >>> be kept/maintained there. >>>=20 >>> Alternately you could try doing this: >>>=20 >>> zpool remove jbod cache daX daY ... >>> zpool export jbod >>>=20 >>> Then on the other system: >>>=20 >>> zpool import jbod >>> zpool add jbod cache daX daY ... >>>=20 >>> Where daX and daY are the disks which are independent to each system >>> (not on the "external enclosure"). >>>=20 >>> Finally, it would also be useful/worthwhile if you would provide=20 >>> "dmesg" from both systems and for you to explain the physical wiring >>> along with what device (e.g. daX) correlates with what exact thing = on >>> each system. (We right now have no knowledge of that, and your = terse >>> explanations imply we do -- we need to know more) >>>=20 >>> --=20 >>> | Jeremy Chadwick jdc@koitsu.org | >>> | UNIX Systems Administrator http://jdc.koitsu.org/ | >>> | Making life hard for others since 1977. PGP 4BD6C0CB | >>>=20 >>=20 >=20 From owner-freebsd-fs@FreeBSD.ORG Wed Jun 26 00:15:47 2013 Return-Path: Delivered-To: freebsd-fs@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 499C52BF; Wed, 26 Jun 2013 00:15:47 +0000 (UTC) (envelope-from gjb@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id 225E01DB0; Wed, 26 Jun 2013 00:15:47 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r5Q0FlqD063009; Wed, 26 Jun 2013 00:15:47 GMT (envelope-from gjb@freefall.freebsd.org) Received: (from gjb@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r5Q0Fklg063008; Wed, 26 Jun 2013 00:15:46 GMT (envelope-from gjb) Date: Wed, 26 Jun 2013 00:15:46 GMT Message-Id: <201306260015.r5Q0Fklg063008@freefall.freebsd.org> To: gjb@FreeBSD.org, freebsd-fs@FreeBSD.org, gjb@FreeBSD.org From: gjb@FreeBSD.org Subject: Re: bin/178996: [zfs] [patch] error in message with zfs mount -> there is no "mount -F zfs" in FreeBSD X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Jun 2013 00:15:47 -0000 Synopsis: [zfs] [patch] error in message with zfs mount -> there is no "mount -F zfs" in FreeBSD Responsible-Changed-From-To: freebsd-fs->gjb Responsible-Changed-By: gjb Responsible-Changed-When: Wed Jun 26 00:15:34 UTC 2013 Responsible-Changed-Why: Take. http://www.freebsd.org/cgi/query-pr.cgi?pr=178996 From owner-freebsd-fs@FreeBSD.ORG Wed Jun 26 04:45:38 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id CD3EBA67; Wed, 26 Jun 2013 04:45:38 +0000 (UTC) (envelope-from jdc@koitsu.org) Received: from relay3-d.mail.gandi.net (relay3-d.mail.gandi.net [217.70.183.195]) by mx1.freebsd.org (Postfix) with ESMTP id 8C8141D41; Wed, 26 Jun 2013 04:45:38 +0000 (UTC) Received: from mfilter22-d.gandi.net (mfilter22-d.gandi.net [217.70.178.150]) by relay3-d.mail.gandi.net (Postfix) with ESMTP id 06662A80B4; Wed, 26 Jun 2013 06:45:21 +0200 (CEST) X-Virus-Scanned: Debian amavisd-new at mfilter22-d.gandi.net Received: from relay3-d.mail.gandi.net ([217.70.183.195]) by mfilter22-d.gandi.net (mfilter22-d.gandi.net [10.0.15.180]) (amavisd-new, port 10024) with ESMTP id cXhP7RcYkWSy; Wed, 26 Jun 2013 06:45:19 +0200 (CEST) X-Originating-IP: 76.102.14.35 Received: from jdc.koitsu.org (c-76-102-14-35.hsd1.ca.comcast.net [76.102.14.35]) (Authenticated sender: jdc@koitsu.org) by relay3-d.mail.gandi.net (Postfix) with ESMTPSA id 69D02A80C2; Wed, 26 Jun 2013 06:45:17 +0200 (CEST) Received: by icarus.home.lan (Postfix, from userid 1000) id 60A0673A1C; Tue, 25 Jun 2013 21:45:15 -0700 (PDT) Date: Tue, 25 Jun 2013 21:45:15 -0700 From: Jeremy Chadwick To: gabor@zahemszky.hu Subject: Re: ext2fs bug Message-ID: <20130626044515.GA67521@icarus.home.lan> References: <9554b1ed02a342c347b17a2e71e38b73@zahemszky.hu> <20130623093521.GA86307@icarus.home.lan> <20130623095207.GA86507@icarus.home.lan> <216bbb1389bb7e6e5ef11500d31204ad@zahemszky.hu> MIME-Version: 1.0 Content-Type: text/plain; charset=unknown-8bit Content-Disposition: inline In-Reply-To: <216bbb1389bb7e6e5ef11500d31204ad@zahemszky.hu> User-Agent: Mutt/1.5.21 (2010-09-15) Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org, "Pedro F. Giffuni" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Jun 2013 04:45:38 -0000 On Sun, Jun 23, 2013 at 01:50:59PM +0200, gabor@zahemszky.hu wrote: > 2013-06-23 11:52 id=C5=91pontban Jeremy Chadwick ezt =C3=ADrta: > >On Sun, Jun 23, 2013 at 11:43:13AM +0200, gabor@zahemszky.hu wrote: > >>Hi! > >> > >>>What FreeBSD version are you using? > >> > >>Sorry for the missing information: it's 9.1-p4 Amd64 > >> > >>>In the past 3-4 weeks there have been *tons* of ext2fs fixes in > >>>stable/9, and I'm not exaggerating in the least. You will not find > >>>these fixes being backported to 9.1-RELEASE or earlier. > >> > >>If they will be in 9.2 or in 10, it will be good. > >> > >>>I strongly recommend you try a recent stable/9 (i.e. past few days) > >>>with ext2fs. You can try a stable/9 ("RELENG_9") ISO snapshot > >>here: > >>> > >>>https://pub.allbsd.org/FreeBSD-snapshots/ >=20 > Well, I downloaded: >=20 > https://pub.allbsd.org/FreeBSD-snapshots/amd64-amd64/9.1-RELENG_9-r2520= 93-JPSNAP/iso/FreeBSD-9.1-RELENG_9-r252093-JPSNAP-amd64-amd64-memstick.im= g >=20 > booted my machine from a flash drive with that IMG. Mounted the same > ext2 prtitions > and got the same result. The same files are missing, and the same > errors on the console. >=20 > :-( >=20 > What next? Possibly r252234 (just committed a few minutes ago to stable/9) may help you, but I'm not sure: http://svnweb.freebsd.org/base?view=3Drevision&revision=3D252234 I've CC'd the committer (pfg@) for his comments/insights. Pedro, the original problem reported is here: http://lists.freebsd.org/pipermail/freebsd-fs/2013-June/017497.html --=20 | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Wed Jun 26 10:29:11 2013 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 453B5786 for ; Wed, 26 Jun 2013 10:29:11 +0000 (UTC) (envelope-from skeletor@lissyara.su) Received: from mx.lissyara.su (mx.lissyara.su [91.227.18.11]) by mx1.freebsd.org (Postfix) with ESMTP id 063BE1C77 for ; Wed, 26 Jun 2013 10:29:10 +0000 (UTC) Received: from [195.234.69.50] (port=34089 helo=[10.5.5.55]) by mx.lissyara.su with esmtpsa (TLSv1:DHE-RSA-CAMELLIA256-SHA:256) (Exim 4.80.1 (FreeBSD)) (envelope-from ) id 1UrmVg-0007YT-BS for freebsd-fs@FreeBSD.org; Wed, 26 Jun 2013 13:59:32 +0400 Message-ID: <51CABB87.2020408@lissyara.su> Date: Wed, 26 Jun 2013 12:59:35 +0300 From: skeletor User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130510 Thunderbird/17.0.6 MIME-Version: 1.0 To: freebsd-fs@FreeBSD.org Subject: panic: Solaris(panic): zfs: allocating allocated segment Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Description: if spam count > 60 - this is spam X-Spam-Count: 0 X-Spam-Description: if spam count > 60 - this is spam X-Spam-Count: 0 X-Descriptions: powered by www.lissyara.su X-Bounce-ID: mx.lissyara.su X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: skeletor@lissyara.su List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Jun 2013 10:29:11 -0000 Hello all. I have the same trouble, that this http://lists.freebsd.org/pipermail/freebsd-current/2009-February/003039.html Does it already fised or no? OS FreeBSD 9.1 Release amd64 From owner-freebsd-fs@FreeBSD.ORG Wed Jun 26 13:08:53 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id CC8247CC for ; Wed, 26 Jun 2013 13:08:53 +0000 (UTC) (envelope-from skeletor@lissyara.su) Received: from mx.lissyara.su (mx.lissyara.su [91.227.18.11]) by mx1.freebsd.org (Postfix) with ESMTP id 8ABEE169D for ; Wed, 26 Jun 2013 13:08:53 +0000 (UTC) Received: from [195.234.69.50] (port=41525 helo=[10.5.5.55]) by mx.lissyara.su with esmtpsa (TLSv1:DHE-RSA-CAMELLIA256-SHA:256) (Exim 4.80.1 (FreeBSD)) (envelope-from ) id 1UrpSu-000PCc-MT for freebsd-fs@freebsd.org; Wed, 26 Jun 2013 17:08:52 +0400 Message-ID: <51CAE7E6.8000708@lissyara.su> Date: Wed, 26 Jun 2013 16:08:54 +0300 From: skeletor User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130510 Thunderbird/17.0.6 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: Re: panic: Solaris(panic): zfs: allocating allocated segment References: <51CABB87.2020408@lissyara.su> In-Reply-To: <51CABB87.2020408@lissyara.su> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spam-Description: if spam count > 60 - this is spam X-Spam-Count: 0 X-Descriptions: powered by www.lissyara.su X-Bounce-ID: mx.lissyara.su X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: skeletor@lissyara.su List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Jun 2013 13:08:53 -0000 26.06.2013 12:59, skeletor пишет: > Hello all. > > I have the same trouble, that this > http://lists.freebsd.org/pipermail/freebsd-current/2009-February/003039.html > > > Does it already fised or no? > > OS FreeBSD 9.1 Release amd64 From Solaris this pool successfully accessed, but in degraded state: # zpool status pool: dpool state: DEGRADED status: One or more devices are unavailable in response to persistent errors. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Determine if the device needs to be replaced, and clear the errors using 'zpool clear' or 'fmadm repaired', or replace the device with 'zpool replace'. Run 'zpool status -v' to see device specific details. scan: scrub repaired 64K in 0h3m with 0 errors on Sat Jun 22 17:10:17 2013 config: NAME STATE READ WRITE CKSUM dpool DEGRADED 0 0 0 raidz1-0 DEGRADED 0 0 0 c7t1d0p0 ONLINE 0 0 0 7844172536082216995 UNAVAIL 0 0 0 c7t3d0p0 ONLINE 0 0 0 errors: No known data errors From owner-freebsd-fs@FreeBSD.ORG Wed Jun 26 20:06:52 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 8A8B5F5F for ; Wed, 26 Jun 2013 20:06:52 +0000 (UTC) (envelope-from pfg@FreeBSD.org) Received: from nm11-vm7.bullet.mail.gq1.yahoo.com (nm11-vm7.bullet.mail.gq1.yahoo.com [98.136.218.174]) by mx1.freebsd.org (Postfix) with ESMTP id 45C3C19F7 for ; Wed, 26 Jun 2013 20:06:51 +0000 (UTC) Received: from [98.137.12.59] by nm11.bullet.mail.gq1.yahoo.com with NNFMP; 26 Jun 2013 20:04:59 -0000 Received: from [98.136.164.71] by tm4.bullet.mail.gq1.yahoo.com with NNFMP; 26 Jun 2013 20:04:59 -0000 Received: from [127.0.0.1] by smtp233.mail.gq1.yahoo.com with NNFMP; 26 Jun 2013 20:04:59 -0000 X-Yahoo-Newman-Id: 403832.53815.bm@smtp233.mail.gq1.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: 7QJctTwVM1naIV8qkebNcKfFXRoZYaQJg6wj1Eldtx37UxF pZ6Cty8Gb9HteL9LOp3DfwtTUm0aR_ieQ_WRjxUQ9Npbfabqi_M2J4XjiF9r h_3ogamTFierNICOpIZMcGiDiYh_FomS57XeGx1PzbtH3VGDWnJPR0PB6EtR nVOKNJJHBQGpJOrjFmxIJJrmJ1IQko85ZyBU3Lv_bFbZBosTsMEIRRJl7L2A JN5XRu5cCGRYM5LShewPvjgw6NtKbLscn3HZzd6u73P5Ne.Nj_DTsiEHY1Sy q29mmuNercHhdPZ9ESNaO4oCR5Jot66fCX57rZRt6Fwnj1PM7rBj_X1OOKib fUAiKi4qEdDvQk3f937dlCxcL_TJeREab2Ps3uCjwMb7W080OD5lCJe_3KyJ T8qyJ4w0HoUOBKQpkjSDGPKt2fTKgt5mOa8JSokRHDKCisMcKvmvZ0ApXqjr oqEXa_qhE58fohhopfrOgFGm8xMFWyzJPkP2ifDWtyq05.uhp4tU- X-Yahoo-SMTP: xcjD0guswBAZaPPIbxpWwLcp9Unf X-Rocket-Received: from [192.168.0.102] (pfg@190.157.126.109 with ) by smtp233.mail.gq1.yahoo.com with SMTP; 26 Jun 2013 20:04:59 +0000 UTC Message-ID: <51CB4968.3080503@FreeBSD.org> Date: Wed, 26 Jun 2013 15:04:56 -0500 From: Pedro Giffuni User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130611 Thunderbird/17.0.6 To: Jeremy Chadwick Subject: Re: ext2fs bug References: <9554b1ed02a342c347b17a2e71e38b73@zahemszky.hu> <20130623093521.GA86307@icarus.home.lan> <20130623095207.GA86507@icarus.home.lan> <216bbb1389bb7e6e5ef11500d31204ad@zahemszky.hu> <20130626044515.GA67521@icarus.home.lan> In-Reply-To: <20130626044515.GA67521@icarus.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset="unknown-8bit" X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-fs@freebsd.org, gabor@zahemszky.hu X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Jun 2013 20:06:52 -0000 From owner-freebsd-fs@FreeBSD.ORG Wed Jun 26 20:10:02 2013 Return-Path: Delivered-To: freebsd-fs@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id DB652FEA for ; Wed, 26 Jun 2013 20:10:02 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id CE11A1A10 for ; Wed, 26 Jun 2013 20:10:02 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r5QKA2kZ004856 for ; Wed, 26 Jun 2013 20:10:02 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r5QKA23E004855; Wed, 26 Jun 2013 20:10:02 GMT (envelope-from gnats) Date: Wed, 26 Jun 2013 20:10:02 GMT Message-Id: <201306262010.r5QKA23E004855@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org Cc: From: Allen Landsidel Subject: Re: kern/160790: [fusefs] [panic] VPUTX: negative ref count with FUSE X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Allen Landsidel List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Jun 2013 20:10:02 -0000 The following reply was made to PR kern/160790; it has been noted by GNATS. From: Allen Landsidel To: bug-followup@FreeBSD.org Cc: Subject: Re: kern/160790: [fusefs] [panic] VPUTX: negative ref count with FUSE Date: Wed, 26 Jun 2013 16:08:39 -0400 On RELENG_9 (9.1 stable), the crash is near instant now, and does not take hours to manifest. With a MooseFS FUSE filesystem mounted, simply deleting a single top level directory results in an instant kernel panic. Compiled with a debug kernel so I could catch it without having to be watching 24/7, I get a panic: Fatal trap 9: general protection fault while in kernel mode cpuid = 1; apic id = 01 ... current process = 904 (rm) [ thread pid 904 tid 100074 ] Stopped at M_FUSEFH: movb 0xffffffffff8182a5,%al Screenshots of the panic and backtrace here: http://i.imgur.com/LHwpdas.png http://i.imgur.com/5fpNd7E.png http://i.imgur.com/8O2yfV7.png From owner-freebsd-fs@FreeBSD.ORG Thu Jun 27 06:33:31 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id CF27E349 for ; Thu, 27 Jun 2013 06:33:31 +0000 (UTC) (envelope-from gabor@zahemszky.hu) Received: from default-smtp.integrity.hu (default-smtp.integrity.hu [212.52.165.203]) by mx1.freebsd.org (Postfix) with ESMTP id 93D101C78 for ; Thu, 27 Jun 2013 06:33:31 +0000 (UTC) Received: by smtp.integrity.hu (Postfix, from userid 10000) id 8A2A01346858; Thu, 27 Jun 2013 08:33:24 +0200 (CEST) Received: from webmail.integrity.hu (mail-fe-1.integrity.hu [10.1.64.120]) (Authenticated sender: gabor@zahemszky.hu) by smtp.integrity.hu (Postfix) with ESMTPA id A3D841346309 for ; Thu, 27 Jun 2013 08:33:23 +0200 (CEST) Received: from UR+FqybaKEoQBxyHpjx3aiucdEdhyAdFvnG5FCRbNjQeFijPUMKDug== (0bl/kfzshWCh0d/0ddRq9vk4B+xafVl6) by webmail.integrity.hu with HTTP (HTTP/1.1 POST); Thu, 27 Jun 2013 08:33:23 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Date: Thu, 27 Jun 2013 08:33:23 +0200 From: gabor@zahemszky.hu To: Subject: Re: ext2fs bug In-Reply-To: <51CB4968.3080503@FreeBSD.org> References: <9554b1ed02a342c347b17a2e71e38b73@zahemszky.hu> <20130623093521.GA86307@icarus.home.lan> <20130623095207.GA86507@icarus.home.lan> <216bbb1389bb7e6e5ef11500d31204ad@zahemszky.hu> <20130626044515.GA67521@icarus.home.lan> <51CB4968.3080503@FreeBSD.org> Message-ID: X-Sender: gabor@zahemszky.hu User-Agent: Roundcube Webmail/0.8.4 X-Virus-Scanned: clamav-milter 0.97.6 at mail-autosubmit X-Virus-Status: Clean X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jun 2013 06:33:31 -0000 2013-06-26 22:04 időpontban Pedro Giffuni ezt írta: > Pardon? Maybe some positive information about a patch eg. ? Thanks, Gabor From owner-freebsd-fs@FreeBSD.ORG Thu Jun 27 09:36:02 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 2C36CE4D for ; Thu, 27 Jun 2013 09:36:02 +0000 (UTC) (envelope-from mxb@alumni.chalmers.se) Received: from mail-lb0-f181.google.com (mail-lb0-f181.google.com [209.85.217.181]) by mx1.freebsd.org (Postfix) with ESMTP id A3B4616DB for ; Thu, 27 Jun 2013 09:36:01 +0000 (UTC) Received: by mail-lb0-f181.google.com with SMTP id w10so295820lbi.12 for ; Thu, 27 Jun 2013 02:36:00 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer :x-gm-message-state; bh=OtHsNTKiHZv9XX4uaDcyl1Ipi91MeiG+g4rqs12king=; b=QwHXmdiv5/qejaGAd6JcUVzYLYV66eX7g1d7ulsu6TGsooJoXQCZvk2civs1pUrn3R hjO6QPmU+YYyDHXAs8pLuFhO0yf74ePlqOCpkFjNj6E3tM0IWXvNJMc+JvAZdJ9FABtT SqtxyfoDA6BvjTBaEq00azq1JnJLeTawT35ILZHvyaKqcuA1DZ+GGIgev+5Ch/bj6r90 YNP+KsZRe1klFoKN+nr5dh9pq4qE2iN2Yu5toHLqV9mu+X1+frBbUJJafkq1tqGWmXVK etzOz6prH1tei1cc48f1J+ZzWTofRVqIo4w9TRmvPTa3edRKbzi9/CfZQEsOGOtw01Bk 8QzA== X-Received: by 10.112.55.104 with SMTP id r8mr3744599lbp.49.1372322065941; Thu, 27 Jun 2013 01:34:25 -0700 (PDT) Received: from grey.office.se.prisjakt.nu ([212.16.170.194]) by mx.google.com with ESMTPSA id b8sm734385lbr.12.2013.06.27.01.34.23 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 27 Jun 2013 01:34:24 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.5 \(1508\)) Subject: Re: zpool export/import on failover - The pool metadata is corrupted From: mxb In-Reply-To: <5A26ABDE-C7F2-41CC-A3D1-69310AB6BC36@alumni.chalmers.se> Date: Thu, 27 Jun 2013 10:34:22 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <47B6A89F-6444-485A-88DD-69A9A93D9B3F@alumni.chalmers.se> References: <016B635E-4EDC-4CDF-AC58-82AC39CBFF56@alumni.chalmers.se> <20130606223911.GA45807@icarus.home.lan> <20130606233417.GA46506@icarus.home.lan> <61E414CF-FCD3-42BB-9533-A40EA934DB99@alumni.chalmers.se> <09717048-12BE-474B-9B20-F5E72D00152E@alumni.chalmers.se> <5A26ABDE-C7F2-41CC-A3D1-69310AB6BC36@alumni.chalmers.se> To: Jeremy Chadwick X-Mailer: Apple Mail (2.1508) X-Gm-Message-State: ALoCoQkmM35BZ/S/+dcIYyM3p74hNWXZfeqAQE9PB7Q4iBzkk9tfxGgPh0Jmej+293YeGWIqVWAj Cc: "freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jun 2013 09:36:02 -0000 Notation for archives. I have so far not experienced any problems with both local (per head = unit) and external (on disk enclosure) caches while importing and exporting my pool. Disks I use on both nodes are identical - = manufacturer, size, model. da1,da2 - local da32,da33 - external Export/import is done WITHOUT removing/adding local disks.=20 root@nfs1:/root # zpool status pool: jbod state: ONLINE scan: scrub repaired 0 in 0h0m with 0 errors on Wed Jun 26 13:14:55 = 2013 config: NAME STATE READ WRITE CKSUM jbod ONLINE 0 0 0 raidz3-0 ONLINE 0 0 0 da10 ONLINE 0 0 0 da11 ONLINE 0 0 0 da12 ONLINE 0 0 0 da13 ONLINE 0 0 0 da14 ONLINE 0 0 0 da15 ONLINE 0 0 0 da16 ONLINE 0 0 0 da17 ONLINE 0 0 0 da18 ONLINE 0 0 0 da19 ONLINE 0 0 0 logs mirror-1 ONLINE 0 0 0 da32s1 ONLINE 0 0 0 da33s1 ONLINE 0 0 0 cache da32s2 ONLINE 0 0 0 da33s2 ONLINE 0 0 0 da1 ONLINE 0 0 0 da2 ONLINE 0 0 0 On 25 jun 2013, at 21:22, mxb wrote: >=20 > I think I'v found the root of this issue. > Looks like "wiring down" disks the same way on both nodes (as = suggested) fixes this issue. >=20 > //mxb >=20 > On 20 jun 2013, at 12:30, mxb wrote: >=20 >>=20 >> Well, >>=20 >> I'm back to square one. >>=20 >> After some uptime and successful import/export from one node to = another, I eventually got 'metadata corruption'. >> I had no problem with import/export while for ex. rebooting = master-node (nfs1), but not THIS time. >> Metdata got corrupted while rebooting master-node?? >>=20 >> Any ideas?=20 >>=20 >> [root@nfs1 ~]# zpool import >> pool: jbod >> id: 7663925948774378610 >> state: FAULTED >> status: The pool metadata is corrupted. >> action: The pool cannot be imported due to damaged devices or data. >> see: http://illumos.org/msg/ZFS-8000-72 >> config: >>=20 >> jbod FAULTED corrupted data >> raidz3-0 ONLINE >> da3 ONLINE >> da4 ONLINE >> da5 ONLINE >> da6 ONLINE >> da7 ONLINE >> da8 ONLINE >> da9 ONLINE >> da10 ONLINE >> da11 ONLINE >> da12 ONLINE >> cache >> da13s2 >> da14s2 >> logs >> mirror-1 ONLINE >> da13s1 ONLINE >> da14s1 ONLINE >> [root@nfs1 ~]# zpool import jbod >> cannot import 'jbod': I/O error >> Destroy and re-create the pool from >> a backup source. >> [root@nfs1 ~]# >>=20 >> On 11 jun 2013, at 10:46, mxb wrote: >>=20 >>>=20 >>> Thanks everyone whom replied. >>> Removing local L2ARC cache disks (da1,da2) indeed showed to be a = cure to my problem. >>>=20 >>> Next is to test with add/remove after import/export as Jeremy = suggested. >>>=20 >>> //mxb >>>=20 >>> On 7 jun 2013, at 01:34, Jeremy Chadwick wrote: >>>=20 >>>> On Fri, Jun 07, 2013 at 12:51:14AM +0200, mxb wrote: >>>>>=20 >>>>> Sure, script is not perfects yet and does not handle many of = stuff, but moving highlight from zpool import/export to the script = itself not that >>>>> clever,as this works most of the time. >>>>>=20 >>>>> Question is WHY ZFS corrupts metadata then it should not. = Sometimes. >>>>> I'v seen stale of zpool then manually importing/exporting pool. >>>>>=20 >>>>>=20 >>>>> On 7 jun 2013, at 00:39, Jeremy Chadwick wrote: >>>>>=20 >>>>>> On Fri, Jun 07, 2013 at 12:12:39AM +0200, mxb wrote: >>>>>>>=20 >>>>>>> Then MASTER goes down, CARP on the second node goes MASTER = (devd.conf, and script for lifting): >>>>>>>=20 >>>>>>> root@nfs2:/root # cat /etc/devd.conf >>>>>>>=20 >>>>>>>=20 >>>>>>> notify 30 { >>>>>>> match "system" "IFNET"; >>>>>>> match "subsystem" "carp0"; >>>>>>> match "type" "LINK_UP"; >>>>>>> action "/etc/zfs_switch.sh active"; >>>>>>> }; >>>>>>>=20 >>>>>>> notify 30 { >>>>>>> match "system" "IFNET"; >>>>>>> match "subsystem" "carp0"; >>>>>>> match "type" "LINK_DOWN"; >>>>>>> action "/etc/zfs_switch.sh backup"; >>>>>>> }; >>>>>>>=20 >>>>>>> root@nfs2:/root # cat /etc/zfs_switch.sh >>>>>>> #!/bin/sh >>>>>>>=20 >>>>>>> DATE=3D`date +%Y%m%d` >>>>>>> HOSTNAME=3D`hostname` >>>>>>>=20 >>>>>>> ZFS_POOL=3D"jbod" >>>>>>>=20 >>>>>>>=20 >>>>>>> case $1 in >>>>>>> active) >>>>>>> echo "Switching to ACTIVE and importing ZFS" | = mail -s ''$DATE': '$HOSTNAME' switching to ACTIVE' root >>>>>>> sleep 10 >>>>>>> /sbin/zpool import -f jbod >>>>>>> /etc/rc.d/mountd restart >>>>>>> /etc/rc.d/nfsd restart >>>>>>> ;; >>>>>>> backup) >>>>>>> echo "Switching to BACKUP and exporting ZFS" | = mail -s ''$DATE': '$HOSTNAME' switching to BACKUP' root >>>>>>> /sbin/zpool export jbod >>>>>>> /etc/rc.d/mountd restart >>>>>>> /etc/rc.d/nfsd restart >>>>>>> ;; >>>>>>> *) >>>>>>> exit 0 >>>>>>> ;; >>>>>>> esac >>>>>>>=20 >>>>>>> This works, most of the time, but sometimes I'm forced to = re-create pool. Those machines suppose to go into prod. >>>>>>> Loosing pool(and data inside it) stops me from deploy this = setup. >>>>>>=20 >>>>>> This script looks highly error-prone. Hasty hasty... :-) >>>>>>=20 >>>>>> This script assumes that the "zpool" commands (import and export) = always >>>>>> work/succeed; there is no exit code ($?) checking being used. >>>>>>=20 >>>>>> Since this is run from within devd(8): where does stdout/stderr = go to >>>>>> when running a program/script under devd(8)? Does it effectively = go >>>>>> to the bit bucket (/dev/null)? If so, you'd never know if the = import or >>>>>> export actually succeeded or not (the export sounds more likely = to be >>>>>> the problem point). >>>>>>=20 >>>>>> I imagine there would be some situations where the export would = fail >>>>>> (some files on filesystems under pool "jbod" still in use), yet = CARP is >>>>>> already blindly assuming everything will be fantastic. Surprise. >>>>>>=20 >>>>>> I also do not know if devd.conf(5) "action" commands spawn a = sub-shell >>>>>> (/bin/sh) or not. If they don't, you won't be able to use things = like" >>>>>> 'action "/etc/zfs_switch.sh active >> /var/log/failover.log";'. = You >>>>>> would then need to implement the equivalent of logging within = your >>>>>> zfs_switch.sh script. >>>>>>=20 >>>>>> You may want to consider the -f flag to zpool import/export >>>>>> (particularly export). However there are risks involved -- = userland >>>>>> applications which have an fd/fh open on a file which is stored = on a >>>>>> filesystem that has now completely disappeared can sometimes = crash >>>>>> (segfault) or behave very oddly (100% CPU usage, etc.) depending = on how >>>>>> they're designed. >>>>>>=20 >>>>>> Basically what I'm trying to say is that devd(8) being used as a = form of >>>>>> HA (high availability) and load balancing is not always possible. >>>>>> Real/true HA (especially with SANs) is often done very = differently (now >>>>>> you know why it's often proprietary. :-) ) >>>>=20 >>>> Add error checking to your script. That's my first and foremost >>>> recommendation. It's not hard to do, really. :-) >>>>=20 >>>> After you do that and still experience the issue (e.g. you see no = actual >>>> errors/issues during the export/import phases), I recommend = removing >>>> the "cache" devices which are "independent" on each system from the = pool >>>> entirely. Quoting you (for readers, since I snipped it from my = previous >>>> reply): >>>>=20 >>>>>>> Note, that ZIL(mirrored) resides on external enclosure. Only = L2ARC >>>>>>> is both local and external - da1,da2, da13s2, da14s2 >>>>=20 >>>> I interpret this to mean the primary and backup nodes (physical = systems) >>>> have actual disks which are not part of the "external enclosure". = If >>>> that's the case -- those disks are always going to vary in their >>>> contents and metadata. Those are never going to be 100% identical = all >>>> the time (is this not obvious?). I'm surprised your stuff has = worked at >>>> all using that model, honestly. >>>>=20 >>>> ZFS is going to bitch/cry if it cannot verify the integrity of = certain >>>> things, all the way down to the L2ARC. That's my understanding of = it at >>>> least, meaning there must always be "some" kind of metadata that = has to >>>> be kept/maintained there. >>>>=20 >>>> Alternately you could try doing this: >>>>=20 >>>> zpool remove jbod cache daX daY ... >>>> zpool export jbod >>>>=20 >>>> Then on the other system: >>>>=20 >>>> zpool import jbod >>>> zpool add jbod cache daX daY ... >>>>=20 >>>> Where daX and daY are the disks which are independent to each = system >>>> (not on the "external enclosure"). >>>>=20 >>>> Finally, it would also be useful/worthwhile if you would provide=20 >>>> "dmesg" from both systems and for you to explain the physical = wiring >>>> along with what device (e.g. daX) correlates with what exact thing = on >>>> each system. (We right now have no knowledge of that, and your = terse >>>> explanations imply we do -- we need to know more) >>>>=20 >>>> --=20 >>>> | Jeremy Chadwick jdc@koitsu.org = | >>>> | UNIX Systems Administrator http://jdc.koitsu.org/ = | >>>> | Making life hard for others since 1977. PGP 4BD6C0CB = | >>>>=20 >>>=20 >>=20 >=20 From owner-freebsd-fs@FreeBSD.ORG Thu Jun 27 09:43:19 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 21080DE for ; Thu, 27 Jun 2013 09:43:19 +0000 (UTC) (envelope-from araujobsdport@gmail.com) Received: from mail-wi0-x22d.google.com (mail-wi0-x22d.google.com [IPv6:2a00:1450:400c:c05::22d]) by mx1.freebsd.org (Postfix) with ESMTP id 762FA1755 for ; Thu, 27 Jun 2013 09:43:18 +0000 (UTC) Received: by mail-wi0-f173.google.com with SMTP id hq4so2889961wib.12 for ; Thu, 27 Jun 2013 02:43:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:reply-to:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=+C45zZmHxK5oibCoJbysQT40YhKvBR14B891WS+jvUM=; b=rEAcL1tgKurlAoo5z9pblwTjN0MGnm7jYIj1q6mDru8J0qCUtCWOn1+m79n/FVYXqY 6K8XzmOrsy0xx3GS6P5j2/kdCp5P6jGpV2C4pqt6VIE5Km6UGOu6T6mi3LFe/Rlpig2m 3LYiVR8kVzeiNk82lLd5n1w56d0RjJbSFZUk7jRC0PbkpnE13siqKPbtJVXcnRAyXcbi JBtfqJ6HrUfgMMtpLcWuPzUPewwuTW6aYM7AUEYeGw/dqOPJWQ5PkyWWUvOk1wTIDeZ1 c5uPVkt5koBcnccQGsZ9sui3CiUvBGk8fz3wjjPcgdOcjnGMdLeHL/y9HjmNC9/fCEmr 6YZw== MIME-Version: 1.0 X-Received: by 10.180.20.228 with SMTP id q4mr5465696wie.1.1372326197648; Thu, 27 Jun 2013 02:43:17 -0700 (PDT) Received: by 10.180.73.180 with HTTP; Thu, 27 Jun 2013 02:43:17 -0700 (PDT) In-Reply-To: <47B6A89F-6444-485A-88DD-69A9A93D9B3F@alumni.chalmers.se> References: <016B635E-4EDC-4CDF-AC58-82AC39CBFF56@alumni.chalmers.se> <20130606223911.GA45807@icarus.home.lan> <20130606233417.GA46506@icarus.home.lan> <61E414CF-FCD3-42BB-9533-A40EA934DB99@alumni.chalmers.se> <09717048-12BE-474B-9B20-F5E72D00152E@alumni.chalmers.se> <5A26ABDE-C7F2-41CC-A3D1-69310AB6BC36@alumni.chalmers.se> <47B6A89F-6444-485A-88DD-69A9A93D9B3F@alumni.chalmers.se> Date: Thu, 27 Jun 2013 17:43:17 +0800 Message-ID: Subject: Re: zpool export/import on failover - The pool metadata is corrupted From: Marcelo Araujo To: mxb Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: araujo@FreeBSD.org List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jun 2013 09:43:19 -0000 For this failover solution, did you create a heartbeat or something such like that? How do you avoid split-brain? Best Regards. 2013/6/27 mxb > > Notation for archives. > > I have so far not experienced any problems with both local (per head unit) > and external (on disk enclosure) caches while importing > and exporting my pool. Disks I use on both nodes are identical - > manufacturer, size, model. > > da1,da2 - local > da32,da33 - external > > Export/import is done WITHOUT removing/adding local disks. > > root@nfs1:/root # zpool status > pool: jbod > state: ONLINE > scan: scrub repaired 0 in 0h0m with 0 errors on Wed Jun 26 13:14:55 2013 > config: > > NAME STATE READ WRITE CKSUM > jbod ONLINE 0 0 0 > raidz3-0 ONLINE 0 0 0 > da10 ONLINE 0 0 0 > da11 ONLINE 0 0 0 > da12 ONLINE 0 0 0 > da13 ONLINE 0 0 0 > da14 ONLINE 0 0 0 > da15 ONLINE 0 0 0 > da16 ONLINE 0 0 0 > da17 ONLINE 0 0 0 > da18 ONLINE 0 0 0 > da19 ONLINE 0 0 0 > logs > mirror-1 ONLINE 0 0 0 > da32s1 ONLINE 0 0 0 > da33s1 ONLINE 0 0 0 > cache > da32s2 ONLINE 0 0 0 > da33s2 ONLINE 0 0 0 > da1 ONLINE 0 0 0 > da2 ONLINE 0 0 0 > > On 25 jun 2013, at 21:22, mxb wrote: > > > > > I think I'v found the root of this issue. > > Looks like "wiring down" disks the same way on both nodes (as suggested) > fixes this issue. > > > > //mxb > > > > On 20 jun 2013, at 12:30, mxb wrote: > > > >> > >> Well, > >> > >> I'm back to square one. > >> > >> After some uptime and successful import/export from one node to > another, I eventually got 'metadata corruption'. > >> I had no problem with import/export while for ex. rebooting master-node > (nfs1), but not THIS time. > >> Metdata got corrupted while rebooting master-node?? > >> > >> Any ideas? > >> > >> [root@nfs1 ~]# zpool import > >> pool: jbod > >> id: 7663925948774378610 > >> state: FAULTED > >> status: The pool metadata is corrupted. > >> action: The pool cannot be imported due to damaged devices or data. > >> see: http://illumos.org/msg/ZFS-8000-72 > >> config: > >> > >> jbod FAULTED corrupted data > >> raidz3-0 ONLINE > >> da3 ONLINE > >> da4 ONLINE > >> da5 ONLINE > >> da6 ONLINE > >> da7 ONLINE > >> da8 ONLINE > >> da9 ONLINE > >> da10 ONLINE > >> da11 ONLINE > >> da12 ONLINE > >> cache > >> da13s2 > >> da14s2 > >> logs > >> mirror-1 ONLINE > >> da13s1 ONLINE > >> da14s1 ONLINE > >> [root@nfs1 ~]# zpool import jbod > >> cannot import 'jbod': I/O error > >> Destroy and re-create the pool from > >> a backup source. > >> [root@nfs1 ~]# > >> > >> On 11 jun 2013, at 10:46, mxb wrote: > >> > >>> > >>> Thanks everyone whom replied. > >>> Removing local L2ARC cache disks (da1,da2) indeed showed to be a cure > to my problem. > >>> > >>> Next is to test with add/remove after import/export as Jeremy > suggested. > >>> > >>> //mxb > >>> > >>> On 7 jun 2013, at 01:34, Jeremy Chadwick wrote: > >>> > >>>> On Fri, Jun 07, 2013 at 12:51:14AM +0200, mxb wrote: > >>>>> > >>>>> Sure, script is not perfects yet and does not handle many of stuff, > but moving highlight from zpool import/export to the script itself not that > >>>>> clever,as this works most of the time. > >>>>> > >>>>> Question is WHY ZFS corrupts metadata then it should not. Sometimes. > >>>>> I'v seen stale of zpool then manually importing/exporting pool. > >>>>> > >>>>> > >>>>> On 7 jun 2013, at 00:39, Jeremy Chadwick wrote: > >>>>> > >>>>>> On Fri, Jun 07, 2013 at 12:12:39AM +0200, mxb wrote: > >>>>>>> > >>>>>>> Then MASTER goes down, CARP on the second node goes MASTER > (devd.conf, and script for lifting): > >>>>>>> > >>>>>>> root@nfs2:/root # cat /etc/devd.conf > >>>>>>> > >>>>>>> > >>>>>>> notify 30 { > >>>>>>> match "system" "IFNET"; > >>>>>>> match "subsystem" "carp0"; > >>>>>>> match "type" "LINK_UP"; > >>>>>>> action "/etc/zfs_switch.sh active"; > >>>>>>> }; > >>>>>>> > >>>>>>> notify 30 { > >>>>>>> match "system" "IFNET"; > >>>>>>> match "subsystem" "carp0"; > >>>>>>> match "type" "LINK_DOWN"; > >>>>>>> action "/etc/zfs_switch.sh backup"; > >>>>>>> }; > >>>>>>> > >>>>>>> root@nfs2:/root # cat /etc/zfs_switch.sh > >>>>>>> #!/bin/sh > >>>>>>> > >>>>>>> DATE=`date +%Y%m%d` > >>>>>>> HOSTNAME=`hostname` > >>>>>>> > >>>>>>> ZFS_POOL="jbod" > >>>>>>> > >>>>>>> > >>>>>>> case $1 in > >>>>>>> active) > >>>>>>> echo "Switching to ACTIVE and importing ZFS" | > mail -s ''$DATE': '$HOSTNAME' switching to ACTIVE' root > >>>>>>> sleep 10 > >>>>>>> /sbin/zpool import -f jbod > >>>>>>> /etc/rc.d/mountd restart > >>>>>>> /etc/rc.d/nfsd restart > >>>>>>> ;; > >>>>>>> backup) > >>>>>>> echo "Switching to BACKUP and exporting ZFS" | > mail -s ''$DATE': '$HOSTNAME' switching to BACKUP' root > >>>>>>> /sbin/zpool export jbod > >>>>>>> /etc/rc.d/mountd restart > >>>>>>> /etc/rc.d/nfsd restart > >>>>>>> ;; > >>>>>>> *) > >>>>>>> exit 0 > >>>>>>> ;; > >>>>>>> esac > >>>>>>> > >>>>>>> This works, most of the time, but sometimes I'm forced to > re-create pool. Those machines suppose to go into prod. > >>>>>>> Loosing pool(and data inside it) stops me from deploy this setup. > >>>>>> > >>>>>> This script looks highly error-prone. Hasty hasty... :-) > >>>>>> > >>>>>> This script assumes that the "zpool" commands (import and export) > always > >>>>>> work/succeed; there is no exit code ($?) checking being used. > >>>>>> > >>>>>> Since this is run from within devd(8): where does stdout/stderr go > to > >>>>>> when running a program/script under devd(8)? Does it effectively go > >>>>>> to the bit bucket (/dev/null)? If so, you'd never know if the > import or > >>>>>> export actually succeeded or not (the export sounds more likely to > be > >>>>>> the problem point). > >>>>>> > >>>>>> I imagine there would be some situations where the export would fail > >>>>>> (some files on filesystems under pool "jbod" still in use), yet > CARP is > >>>>>> already blindly assuming everything will be fantastic. Surprise. > >>>>>> > >>>>>> I also do not know if devd.conf(5) "action" commands spawn a > sub-shell > >>>>>> (/bin/sh) or not. If they don't, you won't be able to use things > like" > >>>>>> 'action "/etc/zfs_switch.sh active >> /var/log/failover.log";'. You > >>>>>> would then need to implement the equivalent of logging within your > >>>>>> zfs_switch.sh script. > >>>>>> > >>>>>> You may want to consider the -f flag to zpool import/export > >>>>>> (particularly export). However there are risks involved -- userland > >>>>>> applications which have an fd/fh open on a file which is stored on a > >>>>>> filesystem that has now completely disappeared can sometimes crash > >>>>>> (segfault) or behave very oddly (100% CPU usage, etc.) depending on > how > >>>>>> they're designed. > >>>>>> > >>>>>> Basically what I'm trying to say is that devd(8) being used as a > form of > >>>>>> HA (high availability) and load balancing is not always possible. > >>>>>> Real/true HA (especially with SANs) is often done very differently > (now > >>>>>> you know why it's often proprietary. :-) ) > >>>> > >>>> Add error checking to your script. That's my first and foremost > >>>> recommendation. It's not hard to do, really. :-) > >>>> > >>>> After you do that and still experience the issue (e.g. you see no > actual > >>>> errors/issues during the export/import phases), I recommend removing > >>>> the "cache" devices which are "independent" on each system from the > pool > >>>> entirely. Quoting you (for readers, since I snipped it from my > previous > >>>> reply): > >>>> > >>>>>>> Note, that ZIL(mirrored) resides on external enclosure. Only L2ARC > >>>>>>> is both local and external - da1,da2, da13s2, da14s2 > >>>> > >>>> I interpret this to mean the primary and backup nodes (physical > systems) > >>>> have actual disks which are not part of the "external enclosure". If > >>>> that's the case -- those disks are always going to vary in their > >>>> contents and metadata. Those are never going to be 100% identical all > >>>> the time (is this not obvious?). I'm surprised your stuff has worked > at > >>>> all using that model, honestly. > >>>> > >>>> ZFS is going to bitch/cry if it cannot verify the integrity of certain > >>>> things, all the way down to the L2ARC. That's my understanding of it > at > >>>> least, meaning there must always be "some" kind of metadata that has > to > >>>> be kept/maintained there. > >>>> > >>>> Alternately you could try doing this: > >>>> > >>>> zpool remove jbod cache daX daY ... > >>>> zpool export jbod > >>>> > >>>> Then on the other system: > >>>> > >>>> zpool import jbod > >>>> zpool add jbod cache daX daY ... > >>>> > >>>> Where daX and daY are the disks which are independent to each system > >>>> (not on the "external enclosure"). > >>>> > >>>> Finally, it would also be useful/worthwhile if you would provide > >>>> "dmesg" from both systems and for you to explain the physical wiring > >>>> along with what device (e.g. daX) correlates with what exact thing on > >>>> each system. (We right now have no knowledge of that, and your terse > >>>> explanations imply we do -- we need to know more) > >>>> > >>>> -- > >>>> | Jeremy Chadwick jdc@koitsu.org | > >>>> | UNIX Systems Administrator http://jdc.koitsu.org/ | > >>>> | Making life hard for others since 1977. PGP 4BD6C0CB | > >>>> > >>> > >> > > > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > -- Marcelo Araujo araujo@FreeBSD.org From owner-freebsd-fs@FreeBSD.ORG Thu Jun 27 11:26:17 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 15D1C6B8 for ; Thu, 27 Jun 2013 11:26:17 +0000 (UTC) (envelope-from mxb@alumni.chalmers.se) Received: from mail-lb0-f171.google.com (mail-lb0-f171.google.com [209.85.217.171]) by mx1.freebsd.org (Postfix) with ESMTP id 77B181C9F for ; Thu, 27 Jun 2013 11:26:15 +0000 (UTC) Received: by mail-lb0-f171.google.com with SMTP id 13so349740lba.30 for ; Thu, 27 Jun 2013 04:26:14 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :message-id:references:to:x-mailer:x-gm-message-state; bh=TGtl12ZxEPibbeUC6DbtdyMu8xGZrrrgSfGwMvx3idY=; b=azfpyhiiPWRJax5RAKucXfiUBoj5WIRW9OiYZQI+6E5TIvPjHZl/x5WlTYxCATz4WB g2+OBv7MWkS3y3A3N5HGwVOs8ABsmv1OYaay0N0tYwT9m+haxAt6fQ2fvWdPnWB1aDkH lRxYEfvr9zI16vXncrJY1cw16xJfq294mcSjyR8Uwf4HAL7mlpW6ec22kRBZi6ISHYE2 Bg4hiXflOeSCPsLfwZ2fGFxyDO3jsRhVvwSuS4BKeDj5Zy7A4t+IiVwq/KFIrA+iz8AQ pwaFaPRTpYaynr8F9DWGY3GH27TYigw7xJSejHFyD6S0CPv8/Kc2Ybw7BsWsXumXB6hA yk2w== X-Received: by 10.152.8.37 with SMTP id o5mr3789742laa.87.1372328556192; Thu, 27 Jun 2013 03:22:36 -0700 (PDT) Received: from grey.office.se.prisjakt.nu ([212.16.170.194]) by mx.google.com with ESMTPSA id ea14sm890106lbb.11.2013.06.27.03.22.33 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 27 Jun 2013 03:22:35 -0700 (PDT) Mime-Version: 1.0 (Mac OS X Mail 6.5 \(1508\)) Subject: Re: zpool export/import on failover - The pool metadata is corrupted From: mxb In-Reply-To: Date: Thu, 27 Jun 2013 12:22:32 +0200 Message-Id: References: <016B635E-4EDC-4CDF-AC58-82AC39CBFF56@alumni.chalmers.se> <20130606223911.GA45807@icarus.home.lan> <20130606233417.GA46506@icarus.home.lan> <61E414CF-FCD3-42BB-9533-A40EA934DB99@alumni.chalmers.se> <09717048-12BE-474B-9B20-F5E72D00152E@alumni.chalmers.se> <5A26ABDE-C7F2-41CC-A3D1-69310AB6BC36@alumni.chalmers.se> <47B6A89F-6444-485A-88DD-69A9A93D9B3F@alumni.chalmers.se> To: araujo@FreeBSD.org X-Mailer: Apple Mail (2.1508) X-Gm-Message-State: ALoCoQlfKsMVajYk2kh9ZaDpIjL6xzghViP+tq7zQx72c5iJ7Ht3x67tCsUUCzjcj+1pFJ0L57g6 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jun 2013 11:26:17 -0000 This solution is built on top of CARP. One of nodes is (as of advskew) a preferred master. Triggered chain is CARP -> devd -> failover_script.sh (zfs = import/export) On 27 jun 2013, at 11:43, Marcelo Araujo = wrote: > For this failover solution, did you create a heartbeat or something = such like that? How do you avoid split-brain? >=20 > Best Regards. >=20 >=20 > 2013/6/27 mxb >=20 > Notation for archives. >=20 > I have so far not experienced any problems with both local (per head = unit) and external (on disk enclosure) caches while importing > and exporting my pool. Disks I use on both nodes are identical - = manufacturer, size, model. >=20 > da1,da2 - local > da32,da33 - external >=20 > Export/import is done WITHOUT removing/adding local disks. >=20 > root@nfs1:/root # zpool status > pool: jbod > state: ONLINE > scan: scrub repaired 0 in 0h0m with 0 errors on Wed Jun 26 13:14:55 = 2013 > config: >=20 > NAME STATE READ WRITE CKSUM > jbod ONLINE 0 0 0 > raidz3-0 ONLINE 0 0 0 > da10 ONLINE 0 0 0 > da11 ONLINE 0 0 0 > da12 ONLINE 0 0 0 > da13 ONLINE 0 0 0 > da14 ONLINE 0 0 0 > da15 ONLINE 0 0 0 > da16 ONLINE 0 0 0 > da17 ONLINE 0 0 0 > da18 ONLINE 0 0 0 > da19 ONLINE 0 0 0 > logs > mirror-1 ONLINE 0 0 0 > da32s1 ONLINE 0 0 0 > da33s1 ONLINE 0 0 0 > cache > da32s2 ONLINE 0 0 0 > da33s2 ONLINE 0 0 0 > da1 ONLINE 0 0 0 > da2 ONLINE 0 0 0 >=20 > On 25 jun 2013, at 21:22, mxb wrote: >=20 > > > > I think I'v found the root of this issue. > > Looks like "wiring down" disks the same way on both nodes (as = suggested) fixes this issue. > > > > //mxb > > > > On 20 jun 2013, at 12:30, mxb wrote: > > > >> > >> Well, > >> > >> I'm back to square one. > >> > >> After some uptime and successful import/export from one node to = another, I eventually got 'metadata corruption'. > >> I had no problem with import/export while for ex. rebooting = master-node (nfs1), but not THIS time. > >> Metdata got corrupted while rebooting master-node?? > >> > >> Any ideas? > >> > >> [root@nfs1 ~]# zpool import > >> pool: jbod > >> id: 7663925948774378610 > >> state: FAULTED > >> status: The pool metadata is corrupted. > >> action: The pool cannot be imported due to damaged devices or data. > >> see: http://illumos.org/msg/ZFS-8000-72 > >> config: > >> > >> jbod FAULTED corrupted data > >> raidz3-0 ONLINE > >> da3 ONLINE > >> da4 ONLINE > >> da5 ONLINE > >> da6 ONLINE > >> da7 ONLINE > >> da8 ONLINE > >> da9 ONLINE > >> da10 ONLINE > >> da11 ONLINE > >> da12 ONLINE > >> cache > >> da13s2 > >> da14s2 > >> logs > >> mirror-1 ONLINE > >> da13s1 ONLINE > >> da14s1 ONLINE > >> [root@nfs1 ~]# zpool import jbod > >> cannot import 'jbod': I/O error > >> Destroy and re-create the pool from > >> a backup source. > >> [root@nfs1 ~]# > >> > >> On 11 jun 2013, at 10:46, mxb wrote: > >> > >>> > >>> Thanks everyone whom replied. > >>> Removing local L2ARC cache disks (da1,da2) indeed showed to be a = cure to my problem. > >>> > >>> Next is to test with add/remove after import/export as Jeremy = suggested. > >>> > >>> //mxb > >>> > >>> On 7 jun 2013, at 01:34, Jeremy Chadwick wrote: > >>> > >>>> On Fri, Jun 07, 2013 at 12:51:14AM +0200, mxb wrote: > >>>>> > >>>>> Sure, script is not perfects yet and does not handle many of = stuff, but moving highlight from zpool import/export to the script = itself not that > >>>>> clever,as this works most of the time. > >>>>> > >>>>> Question is WHY ZFS corrupts metadata then it should not. = Sometimes. > >>>>> I'v seen stale of zpool then manually importing/exporting pool. > >>>>> > >>>>> > >>>>> On 7 jun 2013, at 00:39, Jeremy Chadwick wrote: > >>>>> > >>>>>> On Fri, Jun 07, 2013 at 12:12:39AM +0200, mxb wrote: > >>>>>>> > >>>>>>> Then MASTER goes down, CARP on the second node goes MASTER = (devd.conf, and script for lifting): > >>>>>>> > >>>>>>> root@nfs2:/root # cat /etc/devd.conf > >>>>>>> > >>>>>>> > >>>>>>> notify 30 { > >>>>>>> match "system" "IFNET"; > >>>>>>> match "subsystem" "carp0"; > >>>>>>> match "type" "LINK_UP"; > >>>>>>> action "/etc/zfs_switch.sh active"; > >>>>>>> }; > >>>>>>> > >>>>>>> notify 30 { > >>>>>>> match "system" "IFNET"; > >>>>>>> match "subsystem" "carp0"; > >>>>>>> match "type" "LINK_DOWN"; > >>>>>>> action "/etc/zfs_switch.sh backup"; > >>>>>>> }; > >>>>>>> > >>>>>>> root@nfs2:/root # cat /etc/zfs_switch.sh > >>>>>>> #!/bin/sh > >>>>>>> > >>>>>>> DATE=3D`date +%Y%m%d` > >>>>>>> HOSTNAME=3D`hostname` > >>>>>>> > >>>>>>> ZFS_POOL=3D"jbod" > >>>>>>> > >>>>>>> > >>>>>>> case $1 in > >>>>>>> active) > >>>>>>> echo "Switching to ACTIVE and importing ZFS" | = mail -s ''$DATE': '$HOSTNAME' switching to ACTIVE' root > >>>>>>> sleep 10 > >>>>>>> /sbin/zpool import -f jbod > >>>>>>> /etc/rc.d/mountd restart > >>>>>>> /etc/rc.d/nfsd restart > >>>>>>> ;; > >>>>>>> backup) > >>>>>>> echo "Switching to BACKUP and exporting ZFS" | = mail -s ''$DATE': '$HOSTNAME' switching to BACKUP' root > >>>>>>> /sbin/zpool export jbod > >>>>>>> /etc/rc.d/mountd restart > >>>>>>> /etc/rc.d/nfsd restart > >>>>>>> ;; > >>>>>>> *) > >>>>>>> exit 0 > >>>>>>> ;; > >>>>>>> esac > >>>>>>> > >>>>>>> This works, most of the time, but sometimes I'm forced to = re-create pool. Those machines suppose to go into prod. > >>>>>>> Loosing pool(and data inside it) stops me from deploy this = setup. > >>>>>> > >>>>>> This script looks highly error-prone. Hasty hasty... :-) > >>>>>> > >>>>>> This script assumes that the "zpool" commands (import and = export) always > >>>>>> work/succeed; there is no exit code ($?) checking being used. > >>>>>> > >>>>>> Since this is run from within devd(8): where does stdout/stderr = go to > >>>>>> when running a program/script under devd(8)? Does it = effectively go > >>>>>> to the bit bucket (/dev/null)? If so, you'd never know if the = import or > >>>>>> export actually succeeded or not (the export sounds more likely = to be > >>>>>> the problem point). > >>>>>> > >>>>>> I imagine there would be some situations where the export would = fail > >>>>>> (some files on filesystems under pool "jbod" still in use), yet = CARP is > >>>>>> already blindly assuming everything will be fantastic. = Surprise. > >>>>>> > >>>>>> I also do not know if devd.conf(5) "action" commands spawn a = sub-shell > >>>>>> (/bin/sh) or not. If they don't, you won't be able to use = things like" > >>>>>> 'action "/etc/zfs_switch.sh active >> /var/log/failover.log";'. = You > >>>>>> would then need to implement the equivalent of logging within = your > >>>>>> zfs_switch.sh script. > >>>>>> > >>>>>> You may want to consider the -f flag to zpool import/export > >>>>>> (particularly export). However there are risks involved -- = userland > >>>>>> applications which have an fd/fh open on a file which is stored = on a > >>>>>> filesystem that has now completely disappeared can sometimes = crash > >>>>>> (segfault) or behave very oddly (100% CPU usage, etc.) = depending on how > >>>>>> they're designed. > >>>>>> > >>>>>> Basically what I'm trying to say is that devd(8) being used as = a form of > >>>>>> HA (high availability) and load balancing is not always = possible. > >>>>>> Real/true HA (especially with SANs) is often done very = differently (now > >>>>>> you know why it's often proprietary. :-) ) > >>>> > >>>> Add error checking to your script. That's my first and foremost > >>>> recommendation. It's not hard to do, really. :-) > >>>> > >>>> After you do that and still experience the issue (e.g. you see no = actual > >>>> errors/issues during the export/import phases), I recommend = removing > >>>> the "cache" devices which are "independent" on each system from = the pool > >>>> entirely. Quoting you (for readers, since I snipped it from my = previous > >>>> reply): > >>>> > >>>>>>> Note, that ZIL(mirrored) resides on external enclosure. Only = L2ARC > >>>>>>> is both local and external - da1,da2, da13s2, da14s2 > >>>> > >>>> I interpret this to mean the primary and backup nodes (physical = systems) > >>>> have actual disks which are not part of the "external enclosure". = If > >>>> that's the case -- those disks are always going to vary in their > >>>> contents and metadata. Those are never going to be 100% = identical all > >>>> the time (is this not obvious?). I'm surprised your stuff has = worked at > >>>> all using that model, honestly. > >>>> > >>>> ZFS is going to bitch/cry if it cannot verify the integrity of = certain > >>>> things, all the way down to the L2ARC. That's my understanding = of it at > >>>> least, meaning there must always be "some" kind of metadata that = has to > >>>> be kept/maintained there. > >>>> > >>>> Alternately you could try doing this: > >>>> > >>>> zpool remove jbod cache daX daY ... > >>>> zpool export jbod > >>>> > >>>> Then on the other system: > >>>> > >>>> zpool import jbod > >>>> zpool add jbod cache daX daY ... > >>>> > >>>> Where daX and daY are the disks which are independent to each = system > >>>> (not on the "external enclosure"). > >>>> > >>>> Finally, it would also be useful/worthwhile if you would provide > >>>> "dmesg" from both systems and for you to explain the physical = wiring > >>>> along with what device (e.g. daX) correlates with what exact = thing on > >>>> each system. (We right now have no knowledge of that, and your = terse > >>>> explanations imply we do -- we need to know more) > >>>> > >>>> -- > >>>> | Jeremy Chadwick = jdc@koitsu.org | > >>>> | UNIX Systems Administrator = http://jdc.koitsu.org/ | > >>>> | Making life hard for others since 1977. PGP = 4BD6C0CB | > >>>> > >>> > >> > > >=20 > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >=20 >=20 >=20 > --=20 > Marcelo Araujo > araujo@FreeBSD.org From owner-freebsd-fs@FreeBSD.ORG Thu Jun 27 13:13:54 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 2FDC4414 for ; Thu, 27 Jun 2013 13:13:54 +0000 (UTC) (envelope-from zoltan.arnold.nagy@gmail.com) Received: from mail-oa0-x232.google.com (mail-oa0-x232.google.com [IPv6:2607:f8b0:4003:c02::232]) by mx1.freebsd.org (Postfix) with ESMTP id 01C1B131E for ; Thu, 27 Jun 2013 13:13:53 +0000 (UTC) Received: by mail-oa0-f50.google.com with SMTP id k7so804103oag.37 for ; Thu, 27 Jun 2013 06:13:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=Fb3LaeU/FQO+GKmkCSa6RAcNmwVJN5LQ/iv8vfyRJx8=; b=goNEFYAAW1a1xwWyCMZPtl/hfK3ot6J6vT2e6DebZZsaN23sXKbzjgr/UvQs8LbrD7 bJ/Y6+4ZC0dGGDkAf6QK+TCOAZWYuP+/HRht3qUsypdPoy3Dj68xXEtJfuQFKQD6ZuRx gwAHjTPuowY2QCT5M2xU1xJw+VSRzGM5Krae2X+kbnWDwYhWz+M89Igu7PbnEMKGfQym ANnyfAY+uT3yxQxjLVinT1CKbKzp7mEmAgtxYAlj4xzDbzhCo8McQ1LbP7tlS09fRArE rjzzVH6KogNi/BE6GdY/ZVJDmzf8Ox+C04rA+fZIs0LAdskDcPtFZZzrqn8nS8SCBEww rgcQ== MIME-Version: 1.0 X-Received: by 10.182.81.233 with SMTP id d9mr4151821oby.43.1372338833455; Thu, 27 Jun 2013 06:13:53 -0700 (PDT) Received: by 10.76.126.195 with HTTP; Thu, 27 Jun 2013 06:13:53 -0700 (PDT) Date: Thu, 27 Jun 2013 15:13:53 +0200 Message-ID: Subject: ZFS-backed NFS export with vSphere From: Zoltan Arnold NAGY To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jun 2013 13:13:54 -0000 Hi list, I'd love to have a ZFS-backed NFS export as my VM datastore, but as much as I'd like to tune it, the performance doesn't even get close to Solaris 11's. I currently have the system set up as this: pool: tank state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 da0 ONLINE 0 0 0 da1 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 da2 ONLINE 0 0 0 da3 ONLINE 0 0 0 logs ada0p4 ONLINE 0 0 0 spares da4 AVAIL ada0 is a samsung 840pro SSD, which I'm using for system+ZIL. daX is 1TB, 7200rpm seagate disks. (from this test's perspective, if I use a separate ZIL device or just a partition, doesn't matter - I get roughly the same numbers). The first thing I noticed is that the FSINFO reply from FreeBSD is advertising untunable values (I did not find them documented either in the manpage, or as a sysctl). rtmax, rtpref, wtmax, wtpref: 64k (fbsd), 1M (solaris) dtpref: 64k (fbsd), 8k (solaris) After manually patching the nfs code (changing NFS_MAXBSIZE to 1M instead of MAXBSIZE) to adversize the same read/write values (didn't touch dtpref), my performance went up from 17MB/s to 76MB/s. Is there a reason NFS_MAXBSIZE is not tunable and/or is it so slow? Here's my iozone output (which is run on an ext4 partition created on a linux VM which has a disk backed by the NFS exported from the FreeBSD box): Record Size 4096 KB File size set to 2097152 KB Command line used: iozone -b results.xls -r 4m -s 2g -t 6 -i 0 -i 1 -i 2 Output is in Kbytes/sec Time Resolution = 0.000001 seconds. Processor cache size set to 1024 Kbytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. Throughput test with 6 processes Each process writes a 2097152 Kbyte file in 4096 Kbyte records Children see throughput for 6 initial writers = 76820.31 KB/sec Parent sees throughput for 6 initial writers = 74899.44 KB/sec Min throughput per process = 12298.62 KB/sec Max throughput per process = 12972.72 KB/sec Avg throughput per process = 12803.38 KB/sec Min xfer = 1990656.00 KB Children see throughput for 6 rewriters = 76030.99 KB/sec Parent sees throughput for 6 rewriters = 75062.91 KB/sec Min throughput per process = 12620.45 KB/sec Max throughput per process = 12762.80 KB/sec Avg throughput per process = 12671.83 KB/sec Min xfer = 2076672.00 KB Children see throughput for 6 readers = 114221.39 KB/sec Parent sees throughput for 6 readers = 113942.71 KB/sec Min throughput per process = 18920.14 KB/sec Max throughput per process = 19183.80 KB/sec Avg throughput per process = 19036.90 KB/sec Min xfer = 2068480.00 KB Children see throughput for 6 re-readers = 117018.50 KB/sec Parent sees throughput for 6 re-readers = 116917.01 KB/sec Min throughput per process = 19436.28 KB/sec Max throughput per process = 19590.40 KB/sec Avg throughput per process = 19503.08 KB/sec Min xfer = 2080768.00 KB Children see throughput for 6 random readers = 110072.68 KB/sec Parent sees throughput for 6 random readers = 109698.99 KB/sec Min throughput per process = 18260.33 KB/sec Max throughput per process = 18442.55 KB/sec Avg throughput per process = 18345.45 KB/sec Min xfer = 2076672.00 KB Children see throughput for 6 random writers = 76389.71 KB/sec Parent sees throughput for 6 random writers = 74816.45 KB/sec Min throughput per process = 12592.09 KB/sec Max throughput per process = 12843.75 KB/sec Avg throughput per process = 12731.62 KB/sec Min xfer = 2056192.00 KB The other interesting this is that you can notice the system doesn't cache the data file to ram (the box has 32G), so even for re-reads I get miserable numbers. With solaris, the re-reads happen at nearly wire spead. Any ideas what else I could tune? While 76MB/s is much better than the original 17MB I was seeing, it's still far from Solaris's ~220MB/s... Thanks a lot, Zoltan From owner-freebsd-fs@FreeBSD.ORG Thu Jun 27 13:29:23 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 19C807EB for ; Thu, 27 Jun 2013 13:29:23 +0000 (UTC) (envelope-from feld@feld.me) Received: from out2-smtp.messagingengine.com (out2-smtp.messagingengine.com [66.111.4.26]) by mx1.freebsd.org (Postfix) with ESMTP id E2D131478 for ; Thu, 27 Jun 2013 13:29:22 +0000 (UTC) Received: from compute4.internal (compute4.nyi.mail.srv.osa [10.202.2.44]) by gateway1.nyi.mail.srv.osa (Postfix) with ESMTP id 406CA20C22 for ; Thu, 27 Jun 2013 09:29:22 -0400 (EDT) Received: from web3.nyi.mail.srv.osa ([10.202.2.213]) by compute4.internal (MEProxy); Thu, 27 Jun 2013 09:29:22 -0400 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=feld.me; h= message-id:from:to:mime-version:content-transfer-encoding :content-type:in-reply-to:references:subject:date; s=mesmtp; bh= lGsvy2/I/7K3TISQ37t32PqB2wI=; b=OZNiLA9e4EIzJpkiY3WT12I2eVfvavnx eFhDGzdoyo99UAYOOeoXBMsxYAFx8dlwTIdMbDNOBxRskxCfA+a3iqSa/ACSg+sW wO0afF81BuVKIOTbt0i8TjoMVdtUds7mWhx6o4MXSXKVCjlHp3oEJrCspe1smzqi JA3V7qZ+Has= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d= messagingengine.com; h=message-id:from:to:mime-version :content-transfer-encoding:content-type:in-reply-to:references :subject:date; s=smtpout; bh=lGsvy2/I/7K3TISQ37t32PqB2wI=; b=qfm 2hftRouA7Mr0FficO/XBBXfxpGh8xWuVki8B95/Ft2Q/VN4KjWQvQrvIgPln6yUI V1u7kg9uXVtRIgwYlCg6x7Vu8NCC7ZonadXRnasTS48gLzzakJGyg1YpCthw0NVd tHgrzaTC7iuBWaU0f1Nt+/3od7OIYxTPo+r/jAaI= Received: by web3.nyi.mail.srv.osa (Postfix, from userid 99) id 1B725B00003; Thu, 27 Jun 2013 09:29:22 -0400 (EDT) Message-Id: <1372339762.24419.140661249149329.65B90A12@webmail.messagingengine.com> X-Sasl-Enc: SFtwWgeGBu2UNfBlU8cJmYIGD/bQAKIHVThBpeM6OgKs 1372339762 From: Mark Felder To: freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain X-Mailer: MessagingEngine.com Webmail Interface - ajax-5ae8e04c In-Reply-To: References: Subject: Re: ZFS-backed NFS export with vSphere Date: Thu, 27 Jun 2013 08:29:22 -0500 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jun 2013 13:29:23 -0000 On Thu, Jun 27, 2013, at 8:13, Zoltan Arnold NAGY wrote: > Any ideas what else I could tune? While 76MB/s is much better than the > original 17MB I was seeing, it's still far from Solaris's ~220MB/s... > I've never seen NFS exports on ZFS get anywhere close to what Solaris can do. Then again, Solaris spent a lot of time making sure their ZFS implementation played well with their NFS implementation. There is ongoing work, but I'm not sure what improvements are available in the FreeBSD camp. I can, however, report that iSCSI connections to ZFS zvols vis the istgt daemon work quite well. From owner-freebsd-fs@FreeBSD.ORG Thu Jun 27 15:07:51 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id D4D89AF2 for ; Thu, 27 Jun 2013 15:07:51 +0000 (UTC) (envelope-from biancalana@gmail.com) Received: from mail-vb0-x233.google.com (mail-vb0-x233.google.com [IPv6:2607:f8b0:400c:c02::233]) by mx1.freebsd.org (Postfix) with ESMTP id 9BBB31C66 for ; Thu, 27 Jun 2013 15:07:51 +0000 (UTC) Received: by mail-vb0-f51.google.com with SMTP id x17so740727vbf.38 for ; Thu, 27 Jun 2013 08:07:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=53ttVKODMrGZ0L4KjM0snmloNb3+Lhik3WqPa7cQWZU=; b=MaM3CgebXZTAMZMfJIERSsOB5knB2Af5UG+rLbwRN5iqyna1TaFJJ28iyAjU6lYuS8 XPV3UvB8hXxYPsmUCu4CD/dqnRr6m6z4oZP9xhOnPhgtuwMCFA/e4jsndY+tKp8qiX/f svMf5vVEnbf+524eF4m43MidAMfu1QU0ldPRNgr+U8L8Eh93LiuFXlUP4691JLliAfjt r7Q8AdP0WQI8/q3UoxFnA58lYibNnPCFH18TKsfgl4wfJR+Wn1Lqi4aYDHfDiFz65mQF IFrFBexnvuJSxuegPgKu+lOgZoYv5yRwygDiFcN9fGQ3PujY1nPzq7UWoIpr2DL52+Ta 1pjQ== MIME-Version: 1.0 X-Received: by 10.220.124.1 with SMTP id s1mr3604154vcr.93.1372345671013; Thu, 27 Jun 2013 08:07:51 -0700 (PDT) Received: by 10.58.77.34 with HTTP; Thu, 27 Jun 2013 08:07:50 -0700 (PDT) Date: Thu, 27 Jun 2013 12:07:50 -0300 Message-ID: Subject: ZFS mount volume freeze From: Alexandre Biancalana To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jun 2013 15:07:51 -0000 Hi list, I'm using FreeBSD-8.3-RELEASE-p7 amd64, actually this is a FreeNAS 8.3.1-RELEASE but everything described here were also tested booting machine with a usb memstick using FreeBSD-10.0-CURRENT-amd64-20130623-r252101-memstick image with the same results. I've a home file server where I store my personal files, backups and virtual machine images served through nfs to a XCP. Yesterday my VMs stopped and all commands (ls, find) issued to the VMs volume (through NFS or local on fileserver) started to freeze hanging forever. Then I reboot the machine and it starts to hang after zpool import and before volumes get mounted. This machine has 8GB RAM and AMD Bulldozer six-core CPU. I have two zpool: $ zpool status pool: datastore0 state: ONLINE scan: resilvered 834G in 41h27m with 0 errors on Fri Jun 14 15:59:54 2013 config: NAME STATE READ WRITE CKSUM datastore0 ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gptid/d761e927-d2e2-11e2-bddc-902b34f47b09 ONLINE 0 0 0 gptid/daa3440d-d3c8-11e2-8405-902b34f47b09 ONLINE 0 0 0 errors: No known data errors pool: datastore1 state: ONLINE scan: scrub repaired 1.63G in 1h47m with 0 errors on Thu Jun 27 01:06:59 2013 config: NAME STATE READ WRITE CKSUM datastore1 ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 ada2 ONLINE 0 0 0 ada4 ONLINE 0 0 0 logs gptid/74f920fd-d301-11e2-8405-902b34f47b09 ONLINE 0 0 0 errors: No known data errors $ zfs list NAME USED AVAIL REFER MOUNTPOINT datastore0 1.21T 33.7G 2.12M /mnt/datastore0 datastore0/ale 534G 33.7G 9.10G /mnt/datastore0/ale datastore0/ale/backup 525G 33.7G 143G /mnt/datastore0/ale/backup datastore0/forensics 42.1G 33.7G 42.1G /mnt/datastore0/forensics datastore0/jail_plugins 1.54G 593M 1.42G /mnt/datastore0/jail_plugins datastore0/jail_plugins_archive 278M 746M 278M /mnt/datastore0/jail_plugins_archive datastore0/mirror 520G 33.7G 387G /mnt/datastore0/mirror datastore1 291G 171G 2.29M /datastore1 datastore1/iso 51.1G 28.9G 51.1G /tmp/iso datastore1/vms 238G 144G 55.6G /datastore1/vms datastore0 => mirror 2 1TB SATA datastore1 => mirror 2 500G SATA So I boot in single mode and can successfully import both pools, datastore0 is working perfectly I can mount all their volumes and access all data without any problem. The problem is with a specify volume where I store the VM files is located in datastore1/vms, when I try to mount this volume, no matter if I try todo zfs mount -a or zfs mount datastore1/vms the command hangs forever. I've another volume in that zpool (datastore1/iso) that I can mount and use without any problem. I already tried to do a scrub, that corrected some data but even after that I can mount that volume. Now I'm with the machine online with all volume mounted but datastore1/vms and waiting a zdb datastore1 to finish, (I can provide the cmd output too). How this machine is my home file server I'm free to do almost everything to troubleshoot that. Any ideas on how troubleshoot that further ? Regards, Alexandre From owner-freebsd-fs@FreeBSD.ORG Thu Jun 27 15:31:16 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 77408242 for ; Thu, 27 Jun 2013 15:31:16 +0000 (UTC) (envelope-from biancalana@gmail.com) Received: from mail-ve0-x22d.google.com (mail-ve0-x22d.google.com [IPv6:2607:f8b0:400c:c01::22d]) by mx1.freebsd.org (Postfix) with ESMTP id 3D1291DEC for ; Thu, 27 Jun 2013 15:31:16 +0000 (UTC) Received: by mail-ve0-f173.google.com with SMTP id jw11so810206veb.18 for ; Thu, 27 Jun 2013 08:31:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=yWclSMhilPWAg6xw8jjOrbmxBZzVJriz+Vgx/2KKfKY=; b=UxbESUhuqtxZ4H7edrvjwxfBhvUYWRaHiMqTeHSnH1JJxiDKx/fz3RgPZzNY5IpoNc 6IhEh+z8HvNN3FXzINrbHPmpgVUnWzRrleeyleaiQ9jBqChe/Dxrambb6q1+lrvzP2UL JvIfNtYX8RSoY6utABASp4CtHiAIGqBn25EfZceNozKunX81W1NhyyVR6Rt7rD10po+p 96UrDmfVyj4eJTddJm+MaWmPucj8C/XASbOcgsKAiswLvGB88QBviXVLVWmmWD6y9ZB6 6c4DPW09cyPrO4WPVe+XyPlFuzyTqRaWiWYoM3sSCHUlvbfLxaS235h/pjcXrYUlOnHD MW6g== MIME-Version: 1.0 X-Received: by 10.220.71.74 with SMTP id g10mr3716061vcj.83.1372347075671; Thu, 27 Jun 2013 08:31:15 -0700 (PDT) Received: by 10.58.77.34 with HTTP; Thu, 27 Jun 2013 08:31:15 -0700 (PDT) In-Reply-To: <20130627152304.GA21262@icarus.home.lan> References: <20130627152304.GA21262@icarus.home.lan> Date: Thu, 27 Jun 2013 12:31:15 -0300 Message-ID: Subject: Re: ZFS mount volume freeze From: Alexandre Biancalana To: Jeremy Chadwick , freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jun 2013 15:31:16 -0000 On Thu, Jun 27, 2013 at 12:23 PM, Jeremy Chadwick wrote: > > > > How this machine is my home file server I'm free to do almost everything > to > > troubleshoot that. > > > > Any ideas on how troubleshoot that further ? > > Are there any messages on console ("dmesg") indicating something > anomalous going on? > Hi Jeremy, No messages in dmesg or console... From owner-freebsd-fs@FreeBSD.ORG Thu Jun 27 15:36:41 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 9E230469 for ; Thu, 27 Jun 2013 15:36:41 +0000 (UTC) (envelope-from jdc@koitsu.org) Received: from relay3-d.mail.gandi.net (relay3-d.mail.gandi.net [217.70.183.195]) by mx1.freebsd.org (Postfix) with ESMTP id 5D91A1E64 for ; Thu, 27 Jun 2013 15:36:41 +0000 (UTC) Received: from mfilter13-d.gandi.net (mfilter13-d.gandi.net [217.70.178.141]) by relay3-d.mail.gandi.net (Postfix) with ESMTP id 4A39EA8115; Thu, 27 Jun 2013 17:36:24 +0200 (CEST) X-Virus-Scanned: Debian amavisd-new at mfilter13-d.gandi.net Received: from relay3-d.mail.gandi.net ([217.70.183.195]) by mfilter13-d.gandi.net (mfilter13-d.gandi.net [10.0.15.180]) (amavisd-new, port 10024) with ESMTP id 3mQOTYgXI3p3; Thu, 27 Jun 2013 17:36:22 +0200 (CEST) X-Originating-IP: 76.102.14.35 Received: from jdc.koitsu.org (c-76-102-14-35.hsd1.ca.comcast.net [76.102.14.35]) (Authenticated sender: jdc@koitsu.org) by relay3-d.mail.gandi.net (Postfix) with ESMTPSA id 67B07A80D3; Thu, 27 Jun 2013 17:36:22 +0200 (CEST) Received: by icarus.home.lan (Postfix, from userid 1000) id 76ED773A1C; Thu, 27 Jun 2013 08:36:20 -0700 (PDT) Date: Thu, 27 Jun 2013 08:36:20 -0700 From: Jeremy Chadwick To: Alexandre Biancalana Subject: Re: ZFS mount volume freeze Message-ID: <20130627153620.GA21491@icarus.home.lan> References: <20130627152304.GA21262@icarus.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jun 2013 15:36:41 -0000 On Thu, Jun 27, 2013 at 12:31:15PM -0300, Alexandre Biancalana wrote: > On Thu, Jun 27, 2013 at 12:23 PM, Jeremy Chadwick wrote: > > > > > > > How this machine is my home file server I'm free to do almost everything > > to > > > troubleshoot that. > > > > > > Any ideas on how troubleshoot that further ? > > > > Are there any messages on console ("dmesg") indicating something > > anomalous going on? > > > > Hi Jeremy, > > No messages in dmesg or console... Are you using deduplication or compression on any of the filesystems or the pool? -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Thu Jun 27 15:59:41 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 34E10C01 for ; Thu, 27 Jun 2013 15:59:41 +0000 (UTC) (envelope-from thomas.e.zander@googlemail.com) Received: from mail-ve0-x22a.google.com (mail-ve0-x22a.google.com [IPv6:2607:f8b0:400c:c01::22a]) by mx1.freebsd.org (Postfix) with ESMTP id F37361F7F for ; Thu, 27 Jun 2013 15:59:40 +0000 (UTC) Received: by mail-ve0-f170.google.com with SMTP id 14so849561vea.15 for ; Thu, 27 Jun 2013 08:59:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=uHCYmamKJ6tIDiRaN72WMTKCwJSCRaAR+flSPkCpCI8=; b=LwTRKSM+vPWKdgJNVIcXuOTIJfUQJMvJ6fLTE1WjhQWK3WgN4fGiwDvCx6j+eowBTS Tk+/FyFdAE6ehPIEvqO552Pd60MV4iZIRPSOXlLpzBM5oQ/kUPbdjmOFPkXLr+UWWgmD 9qVjrOZlVlkqYocyFGEVsLl4cnzVIQVAcjmF3/SdAct98PWHTNJoPXCVp2Q2dBSbDDWq 34JVh24s9S9TKlzU+L/Wy8jwP9UVQk6C/8zGr4NCrUd1YD3JmikvtqpSbMVri3bx3poM 4Yn2ozPGR1EKoVqN6JMLlXsjUKG+mXlHCzsOWP2AC5h4CQK9j6WdM+GygB16cIhaEsRT Su5w== MIME-Version: 1.0 X-Received: by 10.220.101.69 with SMTP id b5mr3771819vco.55.1372348780343; Thu, 27 Jun 2013 08:59:40 -0700 (PDT) Received: by 10.220.219.81 with HTTP; Thu, 27 Jun 2013 08:59:40 -0700 (PDT) Date: Thu, 27 Jun 2013 17:59:40 +0200 Message-ID: Subject: Persistent L2ARC From: Thomas Zander To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jun 2013 15:59:41 -0000 Hi, I was curious whether we have this on the radar/roadmap: https://www.illumos.org/issues/3525 It's declared feature-complete, just needs testing. Best wishes Riggs From owner-freebsd-fs@FreeBSD.ORG Thu Jun 27 17:26:30 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 76C31251 for ; Thu, 27 Jun 2013 17:26:30 +0000 (UTC) (envelope-from biancalana@gmail.com) Received: from mail-vb0-x22a.google.com (mail-vb0-x22a.google.com [IPv6:2607:f8b0:400c:c02::22a]) by mx1.freebsd.org (Postfix) with ESMTP id 3B2C9138C for ; Thu, 27 Jun 2013 17:26:30 +0000 (UTC) Received: by mail-vb0-f42.google.com with SMTP id i3so908171vbh.1 for ; Thu, 27 Jun 2013 10:26:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=/ZC1QiZ/O2w+S1l+tjWMND/jBlj0FdB+zXmhCFYsl00=; b=ymdcJliZ4WStgCmrfaDzeEJvX9gSQlTGtYNawEtis9beQftEaFWycvu11WC0sri8wm Z82MuN0OC7qtqH2wNw2zSgRsveP5Tfbk39j3i5n0ZASzOT68FxLHJk6nJP+2kEx5aoEf eEqvpOBDE56D6cljjO2mDI2tYrH6Ijd/FjJwQpOuEI3RTsJSbPS3FZZ3sSFilMDywv84 AC6y4O2/ky5EcsRs/jfH37fsSnRcUx0feYjxCGt7rRa5jI1IgXCoXjeKAU8FRHJcq2Of r4nXew3XE8ui99xAepDTR8Iuhm+ETup3dj5tEcleIAY1l0ukcTQtWDUOxxyLhp7sCv1r LPGQ== MIME-Version: 1.0 X-Received: by 10.58.207.195 with SMTP id ly3mr3857900vec.77.1372353989757; Thu, 27 Jun 2013 10:26:29 -0700 (PDT) Received: by 10.58.77.34 with HTTP; Thu, 27 Jun 2013 10:26:29 -0700 (PDT) In-Reply-To: <20130627153620.GA21491@icarus.home.lan> References: <20130627152304.GA21262@icarus.home.lan> <20130627153620.GA21491@icarus.home.lan> Date: Thu, 27 Jun 2013 14:26:29 -0300 Message-ID: Subject: Re: ZFS mount volume freeze From: Alexandre Biancalana To: Jeremy Chadwick Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jun 2013 17:26:30 -0000 On Thu, Jun 27, 2013 at 12:36 PM, Jeremy Chadwick wrote: > On Thu, Jun 27, 2013 at 12:31:15PM -0300, Alexandre Biancalana wrote: > > On Thu, Jun 27, 2013 at 12:23 PM, Jeremy Chadwick > wrote: > > > > > > > > > > How this machine is my home file server I'm free to do almost > everything > > > to > > > > troubleshoot that. > > > > > > > > Any ideas on how troubleshoot that further ? > > > > > > Are there any messages on console ("dmesg") indicating something > > > anomalous going on? > > > > > > > Hi Jeremy, > > > > No messages in dmesg or console... > > Are you using deduplication or compression on any of the filesystems or > the pool? > Sorry, I forgot to give that info... # zfs get all datastore1/vms NAME PROPERTY VALUE SOURCE datastore1/vms type filesystem - datastore1/vms creation Sun Jun 9 11:43 2013 - datastore1/vms used 238G - datastore1/vms available 144G - datastore1/vms referenced 55.6G - datastore1/vms compressratio 1.94x - datastore1/vms mounted no - datastore1/vms quota none local datastore1/vms reservation none local datastore1/vms recordsize 128K default datastore1/vms mountpoint /datastore1/vms inherited from datastore1 datastore1/vms sharenfs off default datastore1/vms checksum on default datastore1/vms compression on inherited from datastore1 datastore1/vms atime off local datastore1/vms devices on default datastore1/vms exec on default datastore1/vms setuid on default datastore1/vms readonly off default datastore1/vms jailed off default datastore1/vms snapdir hidden default datastore1/vms aclmode passthrough inherited from datastore1 datastore1/vms aclinherit passthrough inherited from datastore1 datastore1/vms canmount on default datastore1/vms xattr on default datastore1/vms copies 1 default datastore1/vms version 5 - datastore1/vms utf8only off - datastore1/vms normalization none - datastore1/vms casesensitivity sensitive - datastore1/vms vscan off default datastore1/vms nbmand off default datastore1/vms sharesmb off default datastore1/vms refquota 200G local datastore1/vms refreservation 200G local datastore1/vms primarycache all default datastore1/vms secondarycache all default datastore1/vms usedbysnapshots 60.4M - datastore1/vms usedbydataset 55.6G - datastore1/vms usedbychildren 0 - datastore1/vms usedbyrefreservation 183G - datastore1/vms logbias latency default datastore1/vms dedup sha256,verify inherited from datastore1 datastore1/vms mlslabel - datastore1/vms sync standard default datastore1/vms refcompressratio 1.94x - datastore1/vms written 17.3G - This pool was created using 4k blocks and also has a 8G SSD used as zil From owner-freebsd-fs@FreeBSD.ORG Thu Jun 27 21:58:20 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 39782E69 for ; Thu, 27 Jun 2013 21:58:20 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id C958B1180 for ; Thu, 27 Jun 2013 21:58:19 +0000 (UTC) X-Cloudmark-SP-Filtered: true X-Cloudmark-SP-Result: v=1.1 cv=u+Bwc9JL7tMNtl/i9xObSTPSFclN5AOtXcIZY5dPsHA= c=1 sm=2 a=ctSXsGKhotwA:10 a=FKkrIqjQGGEA:10 a=-c-2Dywjgf4A:10 a=IkcTkHD0fZMA:10 a=6I5d2MoRAAAA:8 a=2Sv3OlsCs7apXaJz1EwA:9 a=QEXdDO2ut3YA:10 a=SV7veod9ZcQA:10 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqMEANS0zFGDaFve/2dsb2JhbABbgzpJgwe8BYEYdIIjAQEBAQIBAQEBICsgCwUWGAICDRkCKQEJJgYIBwQBHASHZwYMqgCRN4EmjH1+NAeCT4EWA5E1g0uDbpAcgy0gMoEDNw X-IronPort-AV: E=Sophos;i="4.87,954,1363147200"; d="scan'208";a="37401102" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 27 Jun 2013 17:58:13 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 23C80B405A; Thu, 27 Jun 2013 17:58:13 -0400 (EDT) Date: Thu, 27 Jun 2013 17:58:13 -0400 (EDT) From: Rick Macklem To: Zoltan Arnold NAGY Message-ID: <1508973822.292566.1372370293123.JavaMail.root@uoguelph.ca> In-Reply-To: Subject: Re: ZFS-backed NFS export with vSphere MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.202] X-Mailer: Zimbra 7.2.1_GA_2790 (ZimbraWebClient - FF3.0 (Win)/7.2.1_GA_2790) Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jun 2013 21:58:20 -0000 Zoltan Nagy wrote: > Hi list, > > I'd love to have a ZFS-backed NFS export as my VM datastore, but as > much as > I'd like to tune > it, the performance doesn't even get close to Solaris 11's. > > I currently have the system set up as this: > > pool: tank > state: ONLINE > scan: none requested > config: > > NAME STATE READ WRITE CKSUM > tank ONLINE 0 0 0 > mirror-0 ONLINE 0 0 0 > da0 ONLINE 0 0 0 > da1 ONLINE 0 0 0 > mirror-1 ONLINE 0 0 0 > da2 ONLINE 0 0 0 > da3 ONLINE 0 0 0 > logs > ada0p4 ONLINE 0 0 0 > spares > da4 AVAIL > > ada0 is a samsung 840pro SSD, which I'm using for system+ZIL. > daX is 1TB, 7200rpm seagate disks. > (from this test's perspective, if I use a separate ZIL device or just > a > partition, doesn't matter - I get roughly the same numbers). > > The first thing I noticed is that the FSINFO reply from FreeBSD is > advertising untunable values (I did not find them documented either > in the > manpage, or as a sysctl). > > rtmax, rtpref, wtmax, wtpref: 64k (fbsd), 1M (solaris) > dtpref: 64k (fbsd), 8k (solaris) > > After manually patching the nfs code (changing NFS_MAXBSIZE to 1M > instead > of MAXBSIZE) to adversize the same read/write values (didn't touch > dtpref), > my performance went up from 17MB/s to 76MB/s. > > Is there a reason NFS_MAXBSIZE is not tunable and/or is it so slow? > For exporting other file system types (UFS, ...) the buffer cache is used and MAXBSIZE is the largest block you can use for the buffer cache. Some increase of MAXBSIZE would be nice. (I've tried 128Kb without observing difficulties and from what I've been told 128Kb is the ZFS block size.) > Here's my iozone output (which is run on an ext4 partition created on > a > linux VM which has a disk backed by the NFS exported from the FreeBSD > box): > > Record Size 4096 KB > File size set to 2097152 KB > Command line used: iozone -b results.xls -r 4m -s 2g -t 6 -i 0 -i > 1 -i 2 > Output is in Kbytes/sec > Time Resolution = 0.000001 seconds. > Processor cache size set to 1024 Kbytes. > Processor cache line size set to 32 bytes. > File stride size set to 17 * record size. > Throughput test with 6 processes > Each process writes a 2097152 Kbyte file in 4096 Kbyte records > > Children see throughput for 6 initial writers = 76820.31 > KB/sec > Parent sees throughput for 6 initial writers = 74899.44 > KB/sec > Min throughput per process = 12298.62 KB/sec > Max throughput per process = 12972.72 KB/sec > Avg throughput per process = 12803.38 KB/sec > Min xfer = 1990656.00 KB > > Children see throughput for 6 rewriters = 76030.99 KB/sec > Parent sees throughput for 6 rewriters = 75062.91 KB/sec > Min throughput per process = 12620.45 KB/sec > Max throughput per process = 12762.80 KB/sec > Avg throughput per process = 12671.83 KB/sec > Min xfer = 2076672.00 KB > > Children see throughput for 6 readers = 114221.39 > KB/sec > Parent sees throughput for 6 readers = 113942.71 KB/sec > Min throughput per process = 18920.14 KB/sec > Max throughput per process = 19183.80 KB/sec > Avg throughput per process = 19036.90 KB/sec > Min xfer = 2068480.00 KB > > Children see throughput for 6 re-readers = 117018.50 KB/sec > Parent sees throughput for 6 re-readers = 116917.01 KB/sec > Min throughput per process = 19436.28 KB/sec > Max throughput per process = 19590.40 KB/sec > Avg throughput per process = 19503.08 KB/sec > Min xfer = 2080768.00 KB > > Children see throughput for 6 random readers = 110072.68 > KB/sec > Parent sees throughput for 6 random readers = 109698.99 > KB/sec > Min throughput per process = 18260.33 KB/sec > Max throughput per process = 18442.55 KB/sec > Avg throughput per process = 18345.45 KB/sec > Min xfer = 2076672.00 KB > > Children see throughput for 6 random writers = 76389.71 > KB/sec > Parent sees throughput for 6 random writers = 74816.45 > KB/sec > Min throughput per process = 12592.09 KB/sec > Max throughput per process = 12843.75 KB/sec > Avg throughput per process = 12731.62 KB/sec > Min xfer = 2056192.00 KB > > The other interesting this is that you can notice the system doesn't > cache > the data file to ram (the box has 32G), so even for re-reads I get > miserable numbers. With solaris, the re-reads happen at nearly wire > spead. > > Any ideas what else I could tune? While 76MB/s is much better than > the > original 17MB I was seeing, it's still far from Solaris's ~220MB/s... > > Thanks a lot, > Zoltan > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@FreeBSD.ORG Thu Jun 27 22:16:49 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id A60271C2 for ; Thu, 27 Jun 2013 22:16:49 +0000 (UTC) (envelope-from zoltan.arnold.nagy@gmail.com) Received: from mail-ob0-x234.google.com (mail-ob0-x234.google.com [IPv6:2607:f8b0:4003:c01::234]) by mx1.freebsd.org (Postfix) with ESMTP id 728B11244 for ; Thu, 27 Jun 2013 22:16:49 +0000 (UTC) Received: by mail-ob0-f180.google.com with SMTP id eh20so1305083obb.11 for ; Thu, 27 Jun 2013 15:16:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=+ke7xQYX0bzW46AFZgAomeKDGW4C0sar/Vbi0D/QnQI=; b=INiaX5BGwHqPcNruGoax0PADtZK8zEuZGh5D37z2qqwp5JWN/45Dt15WDxXXp1Tw09 PwM7cRFs4qWM/53TexJRUQ3eoJCcz0lnWxvG0GK8HIDKcX5eLsA0Qaj+rkvUEmww3ZfK No8sdz5qWTV1icwZ55GLe3Ry5ZBndwMt6jNbhvQidMGHW9rVY3vu3Umny7XLE15VXbur JwOQ+P6qCHFCOse1hJAa0xa2EnUkqdaow8SkYq+a28+O75SdEt8QhyvRcWoClZLDcAks 6QeC/lrxGchl+np2AxTGXvrBpx1zNKV9ifgw7YF6SO5X19KPmTUizrDlDveoJR8LFO2v hKXw== MIME-Version: 1.0 X-Received: by 10.60.62.238 with SMTP id b14mr3858517oes.90.1372371409001; Thu, 27 Jun 2013 15:16:49 -0700 (PDT) Received: by 10.76.126.195 with HTTP; Thu, 27 Jun 2013 15:16:48 -0700 (PDT) In-Reply-To: <1508973822.292566.1372370293123.JavaMail.root@uoguelph.ca> References: <1508973822.292566.1372370293123.JavaMail.root@uoguelph.ca> Date: Fri, 28 Jun 2013 00:16:48 +0200 Message-ID: Subject: Re: ZFS-backed NFS export with vSphere From: Zoltan Arnold NAGY To: Rick Macklem Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jun 2013 22:16:49 -0000 Right. As I said, increasing it to 1M increased my throughput from 17MB/s to 76MB/s. However, the SSD can do much more random writes; any idea why I don't see the ZIL go over this value? (vSphere always uses sync writes). Thanks, Zoltan On Thu, Jun 27, 2013 at 11:58 PM, Rick Macklem wrote: > Zoltan Nagy wrote: > > Hi list, > > > > I'd love to have a ZFS-backed NFS export as my VM datastore, but as > > much as > > I'd like to tune > > it, the performance doesn't even get close to Solaris 11's. > > > > I currently have the system set up as this: > > > > pool: tank > > state: ONLINE > > scan: none requested > > config: > > > > NAME STATE READ WRITE CKSUM > > tank ONLINE 0 0 0 > > mirror-0 ONLINE 0 0 0 > > da0 ONLINE 0 0 0 > > da1 ONLINE 0 0 0 > > mirror-1 ONLINE 0 0 0 > > da2 ONLINE 0 0 0 > > da3 ONLINE 0 0 0 > > logs > > ada0p4 ONLINE 0 0 0 > > spares > > da4 AVAIL > > > > ada0 is a samsung 840pro SSD, which I'm using for system+ZIL. > > daX is 1TB, 7200rpm seagate disks. > > (from this test's perspective, if I use a separate ZIL device or just > > a > > partition, doesn't matter - I get roughly the same numbers). > > > > The first thing I noticed is that the FSINFO reply from FreeBSD is > > advertising untunable values (I did not find them documented either > > in the > > manpage, or as a sysctl). > > > > rtmax, rtpref, wtmax, wtpref: 64k (fbsd), 1M (solaris) > > dtpref: 64k (fbsd), 8k (solaris) > > > > After manually patching the nfs code (changing NFS_MAXBSIZE to 1M > > instead > > of MAXBSIZE) to adversize the same read/write values (didn't touch > > dtpref), > > my performance went up from 17MB/s to 76MB/s. > > > > Is there a reason NFS_MAXBSIZE is not tunable and/or is it so slow? > > > For exporting other file system types (UFS, ...) the buffer cache is > used and MAXBSIZE is the largest block you can use for the buffer cache. > Some increase of MAXBSIZE would be nice. (I've tried 128Kb without > observing > difficulties and from what I've been told 128Kb is the ZFS block size.) > > > Here's my iozone output (which is run on an ext4 partition created on > > a > > linux VM which has a disk backed by the NFS exported from the FreeBSD > > box): > > > > Record Size 4096 KB > > File size set to 2097152 KB > > Command line used: iozone -b results.xls -r 4m -s 2g -t 6 -i 0 -i > > 1 -i 2 > > Output is in Kbytes/sec > > Time Resolution = 0.000001 seconds. > > Processor cache size set to 1024 Kbytes. > > Processor cache line size set to 32 bytes. > > File stride size set to 17 * record size. > > Throughput test with 6 processes > > Each process writes a 2097152 Kbyte file in 4096 Kbyte records > > > > Children see throughput for 6 initial writers = 76820.31 > > KB/sec > > Parent sees throughput for 6 initial writers = 74899.44 > > KB/sec > > Min throughput per process = 12298.62 KB/sec > > Max throughput per process = 12972.72 KB/sec > > Avg throughput per process = 12803.38 KB/sec > > Min xfer = 1990656.00 KB > > > > Children see throughput for 6 rewriters = 76030.99 KB/sec > > Parent sees throughput for 6 rewriters = 75062.91 KB/sec > > Min throughput per process = 12620.45 KB/sec > > Max throughput per process = 12762.80 KB/sec > > Avg throughput per process = 12671.83 KB/sec > > Min xfer = 2076672.00 KB > > > > Children see throughput for 6 readers = 114221.39 > > KB/sec > > Parent sees throughput for 6 readers = 113942.71 KB/sec > > Min throughput per process = 18920.14 KB/sec > > Max throughput per process = 19183.80 KB/sec > > Avg throughput per process = 19036.90 KB/sec > > Min xfer = 2068480.00 KB > > > > Children see throughput for 6 re-readers = 117018.50 KB/sec > > Parent sees throughput for 6 re-readers = 116917.01 KB/sec > > Min throughput per process = 19436.28 KB/sec > > Max throughput per process = 19590.40 KB/sec > > Avg throughput per process = 19503.08 KB/sec > > Min xfer = 2080768.00 KB > > > > Children see throughput for 6 random readers = 110072.68 > > KB/sec > > Parent sees throughput for 6 random readers = 109698.99 > > KB/sec > > Min throughput per process = 18260.33 KB/sec > > Max throughput per process = 18442.55 KB/sec > > Avg throughput per process = 18345.45 KB/sec > > Min xfer = 2076672.00 KB > > > > Children see throughput for 6 random writers = 76389.71 > > KB/sec > > Parent sees throughput for 6 random writers = 74816.45 > > KB/sec > > Min throughput per process = 12592.09 KB/sec > > Max throughput per process = 12843.75 KB/sec > > Avg throughput per process = 12731.62 KB/sec > > Min xfer = 2056192.00 KB > > > > The other interesting this is that you can notice the system doesn't > > cache > > the data file to ram (the box has 32G), so even for re-reads I get > > miserable numbers. With solaris, the re-reads happen at nearly wire > > spead. > > > > Any ideas what else I could tune? While 76MB/s is much better than > > the > > original 17MB I was seeing, it's still far from Solaris's ~220MB/s... > > > > Thanks a lot, > > Zoltan > > _______________________________________________ > > freebsd-fs@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > > From owner-freebsd-fs@FreeBSD.ORG Thu Jun 27 22:41:31 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 1E84BA56 for ; Thu, 27 Jun 2013 22:41:31 +0000 (UTC) (envelope-from matthew.ahrens@delphix.com) Received: from mail-lb0-x22d.google.com (mail-lb0-x22d.google.com [IPv6:2a00:1450:4010:c04::22d]) by mx1.freebsd.org (Postfix) with ESMTP id 98ABD1341 for ; Thu, 27 Jun 2013 22:41:30 +0000 (UTC) Received: by mail-lb0-f173.google.com with SMTP id v1so690728lbd.32 for ; Thu, 27 Jun 2013 15:41:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=delphix.com; s=google; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=ABoPbsUHz2U+cYvmEMFehmPCtb1KlJvUblRayNtIkFM=; b=QN/rOLFftEJiAZTt78I7Pk2fD/vS8GgWYpVpw9Ggu/85ENzFCKufHYW3Bid0a2IuFz 66x+4cZKOgJ+V8TSge5fLSb5kywBru5xBjWop2d3Woh5haiYefl/a2LpLmxH5OYJxLj0 iIa96v7AyNjMCFJh7IdbsI3DBIXoJHf29q8DM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:x-gm-message-state; bh=ABoPbsUHz2U+cYvmEMFehmPCtb1KlJvUblRayNtIkFM=; b=o2rIhO8R/PHiRdMUe54qkmUcT+q+3rOwYKxlMSkJYVC2BERc7KfCou7sfOShI35ckY tqnu5VxVcl+jtaBT2nU42EcTQN+191OkSJ3DbkpZWVhgAxxRK1IL4/MbCR8SZ5CdJMia lDPedJpJKpoGf653o/Gn3cul7QntsOL1LATKLXxhtRmhNQRX8nUdeu30w8wBpmOp4+XT weSRpsQBb6Q21fksWxKaY9VY2+Wj/+CerDy1RJPw6wG8QOCYIs3ZZViiOX2am+H6B21t mIgZmZ0YUMouX4DV0Y4+7qFcGv4DII5CYPxwrA+q1tOnfkYrDiBAUJMwL+8664MGPSqg EVGA== MIME-Version: 1.0 X-Received: by 10.112.16.105 with SMTP id f9mr5144703lbd.69.1372372889493; Thu, 27 Jun 2013 15:41:29 -0700 (PDT) Received: by 10.114.25.33 with HTTP; Thu, 27 Jun 2013 15:41:29 -0700 (PDT) In-Reply-To: References: Date: Thu, 27 Jun 2013 15:41:29 -0700 Message-ID: Subject: Re: Persistent L2ARC From: Matthew Ahrens To: Thomas Zander X-Gm-Message-State: ALoCoQmrMrxr0SvHc9NJrkhq3Xxq3gAtFQI2hl2WT2ZFmAGxp7+9YI5XhrBKNfmP8cUIG2KVNBEJ Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-fs , George Wilson X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jun 2013 22:41:31 -0000 Yes, George and I need to finish reviewing it for illumos. If you're able to test it out and report back, that would be great! --matt On Thu, Jun 27, 2013 at 8:59 AM, Thomas Zander < thomas.e.zander@googlemail.com> wrote: > Hi, > > I was curious whether we have this on the radar/roadmap: > https://www.illumos.org/issues/3525 > > It's declared feature-complete, just needs testing. > > Best wishes > Riggs > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@FreeBSD.ORG Fri Jun 28 07:41:48 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 10D99B2E for ; Fri, 28 Jun 2013 07:41:48 +0000 (UTC) (envelope-from thomas.e.zander@googlemail.com) Received: from mail-vb0-x236.google.com (mail-vb0-x236.google.com [IPv6:2607:f8b0:400c:c02::236]) by mx1.freebsd.org (Postfix) with ESMTP id CAA4519CA for ; Fri, 28 Jun 2013 07:41:47 +0000 (UTC) Received: by mail-vb0-f54.google.com with SMTP id q12so1443267vbe.41 for ; Fri, 28 Jun 2013 00:41:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=qPxnqNjpaMmCruvhnGXmzPQotd/cvVGB/PU5f89V93Q=; b=qi3GfZTvslYAiKHV027NBFgWrHSbaueEK8n+d3cunAatKKR5BlS04boKZoqGbuwKzk uEjXHNjQrHVWMEqoRgaFaQNj+AgmFbIw+5UPkvACW/uASt8FrKFGyGCDVgW/unZVpzRR qoE4lXk/OAuGpqVcSc/tyh0QPQqoI8HJR4Survhv5ZmRjGftpSkw53WdV4qehu3Ogp4U MXKYkm3zzD3vYJThvrTldHXmxT3Yl2ISxFWTyC2Pv0XWzYkjrYbHTKkS5D7j/WEREVlK tB85YY3pvnfUTH6RzmLAd7WB/bcYWWuHgUGSYCAu1j9SZnltB7gIQKeE7Bf3jdDrYskE nxqg== MIME-Version: 1.0 X-Received: by 10.59.9.69 with SMTP id dq5mr4908961ved.87.1372405307322; Fri, 28 Jun 2013 00:41:47 -0700 (PDT) Received: by 10.220.219.81 with HTTP; Fri, 28 Jun 2013 00:41:47 -0700 (PDT) In-Reply-To: References: Date: Fri, 28 Jun 2013 09:41:47 +0200 Message-ID: Subject: Re: Persistent L2ARC From: Thomas Zander To: Matthew Ahrens Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-fs , George Wilson X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jun 2013 07:41:48 -0000 On 28 June 2013 00:41, Matthew Ahrens wrote: > Yes, George and I need to finish reviewing it for illumos. If you're able > to test it out and report back, that would be great! You mean on illumos or on a patch for FreeBSD head? Right now I don't have a test box but I might have one in a couple of weeks. If so, I'd be happy to test and report. Best regards Riggs From owner-freebsd-fs@FreeBSD.ORG Fri Jun 28 16:52:53 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 7A69BCE2 for ; Fri, 28 Jun 2013 16:52:53 +0000 (UTC) (envelope-from matthew.ahrens@delphix.com) Received: from mail-la0-x22b.google.com (mail-la0-x22b.google.com [IPv6:2a00:1450:4010:c03::22b]) by mx1.freebsd.org (Postfix) with ESMTP id F2DAB18AC for ; Fri, 28 Jun 2013 16:52:51 +0000 (UTC) Received: by mail-la0-f43.google.com with SMTP id gw10so2366374lab.2 for ; Fri, 28 Jun 2013 09:52:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=delphix.com; s=google; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=Hx4xZ6tTq+C5/OYwGX7PSaP1vbn4wuM+OldEb4lJQYs=; b=RD+sb/nyc6IEgRrXEkZXnyAnEHPmqBtQKDhOsMUL3QAdxTgHmQdhGRxLlLx1amBJ0+ LRPlfr6bcpU3GxSyOh/yhCWEaV12T7Erm3Ogb6kae0m0TPrY/TCDlikxTv+JvVGEg7gg t+kEpvpjk4mWzU6dTo9QIBUmp82jJr9bsoS+A= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:x-gm-message-state; bh=Hx4xZ6tTq+C5/OYwGX7PSaP1vbn4wuM+OldEb4lJQYs=; b=B/aPeIoblOZekvjdBvBzcrR113T1FOkplTLiR1ZP26yF5kWdJULCQZIEZEZpavgbws Gl9d5sUoAiuJo+Gc/Pfuh3S+9dAsGcE4jkP7qMtYkZnuPsq2+No2Fqn7cr6QytJcE3p5 duGd7pO0hD01POcB3p8sKfr2r+k82ppjOCB5mWDWI9avW72eLwT9ywQP7/pa7sfPIxUm flrjeDJkVCdRPXqr6SxNGeiwlxQrLJMoFTlftlJtQqGYAd3VUPXwc4IJ8LO6bSGdeGc1 T2LdZbp3i1gRZKS6Tn0CraXM07OgnQVClXArevn6QkFQ0nipEjs8qwamjltFdfYyN6p0 kFLQ== MIME-Version: 1.0 X-Received: by 10.152.28.129 with SMTP id b1mr6907913lah.51.1372438371033; Fri, 28 Jun 2013 09:52:51 -0700 (PDT) Received: by 10.114.25.33 with HTTP; Fri, 28 Jun 2013 09:52:50 -0700 (PDT) In-Reply-To: References: Date: Fri, 28 Jun 2013 09:52:50 -0700 Message-ID: Subject: Re: Persistent L2ARC From: Matthew Ahrens To: Thomas Zander X-Gm-Message-State: ALoCoQlbm52+6B6kPJr7SkAYdr6KXWYmCg1lY6ebpdmRJuriGdB33W7f5pQX5puB2uY17X/oDEr1 Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-fs , =?UTF-8?Q?Sa=C5=A1o_Kiselkov?= , George Wilson X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jun 2013 16:52:53 -0000 On Fri, Jun 28, 2013 at 12:41 AM, Thomas Zander < thomas.e.zander@googlemail.com> wrote: > On 28 June 2013 00:41, Matthew Ahrens wrote: > > Yes, George and I need to finish reviewing it for illumos. If you're > able > > to test it out and report back, that would be great! > > You mean on illumos or on a patch for FreeBSD head? > Right now I don't have a test box but I might have one in a couple of > weeks. If so, I'd be happy to test and report. > We are reviewing the changes against illumos, but testing on either illumos or a patched FreeBSD would be appreciated. --matt From owner-freebsd-fs@FreeBSD.ORG Fri Jun 28 17:19:36 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id D4C3974B for ; Fri, 28 Jun 2013 17:19:36 +0000 (UTC) (envelope-from skiselkov.ml@gmail.com) Received: from mail-we0-x234.google.com (mail-we0-x234.google.com [IPv6:2a00:1450:400c:c03::234]) by mx1.freebsd.org (Postfix) with ESMTP id 64CCF19EA for ; Fri, 28 Jun 2013 17:19:36 +0000 (UTC) Received: by mail-we0-f180.google.com with SMTP id w56so1476462wes.11 for ; Fri, 28 Jun 2013 10:19:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=FbuXOYmSasMOtAniXmRQbMsKHAQLz6s59JvLLDCftTk=; b=Yab/7nAoAOD2LnbiDJDOpWfybncMVg3QIDAUZcnCdxuwVz25ANfbTgCbxzfDC5c1Ib R+jGzhxT3RY/QpvOAsbRWcqKtZA3Re/GDZ2OwSdhvmOL4hLXEpvVdZNMaIzoO2AgPH/9 zYuBtB0veHtuDxDfTR+DoZrEO8lPVaAfFiR7Pw/Knnw0Jmes0dxKguObmRMcDybP2+K0 eVyJ+SP1pm/hn9cPccKThFmsRCWXU/nn59AowHlUSu6GfcwmDv078OigztbT7n4Aa5gn KzwFylq0fjoh+XOgEYNoUDNkM370P+36mILeJ1n13DgVpxEpU7c2g/mB+bmoAAszaOuj eHjQ== X-Received: by 10.194.237.38 with SMTP id uz6mr10467906wjc.73.1372439975587; Fri, 28 Jun 2013 10:19:35 -0700 (PDT) Received: from [192.168.0.2] (cpc9-cmbg15-2-0-cust426.5-4.cable.virginmedia.com. [86.30.201.171]) by mx.google.com with ESMTPSA id p1sm25230773wix.9.2013.06.28.10.19.33 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 28 Jun 2013 10:19:34 -0700 (PDT) Message-ID: <51CDC5A3.6030400@gmail.com> Date: Fri, 28 Jun 2013 18:19:31 +0100 From: Saso Kiselkov User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:17.0) Gecko/20130509 Thunderbird/17.0.6 MIME-Version: 1.0 To: Matthew Ahrens Subject: Re: Persistent L2ARC References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs , George Wilson X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jun 2013 17:19:36 -0000 On 28/06/2013 17:52, Matthew Ahrens wrote: > > On Fri, Jun 28, 2013 at 12:41 AM, Thomas Zander > > > wrote: > > On 28 June 2013 00:41, Matthew Ahrens > wrote: > > Yes, George and I need to finish reviewing it for illumos. If > you're able > > to test it out and report back, that would be great! > > You mean on illumos or on a patch for FreeBSD head? > Right now I don't have a test box but I might have one in a couple of > weeks. If so, I'd be happy to test and report. > > > We are reviewing the changes against illumos, but testing on either > illumos or a patched FreeBSD would be appreciated. Hi guys, As luck would have it I'm right now in the middle of refactoring some of the persistency code to get it ready for upstreaming. I'm doing a full nightly and will be testing my changes today. If all goes well I'll publish a new webrev for review this weekend or on Monday. Matt & George: I've built in your suggestion to store a hint about persistent L2ARC in the vdev configuration nvlist. Now we only attempt a rebuild if the vdev is marked as "persistency-capable". I've also gotten rid of the rotor L2 uberblock region - now it's just a simple 4k block at the start of the device that we write. We'll leave the flash wear leveling to the piece of hardware that knows best: the flash controller. Cheers, -- Saso From owner-freebsd-fs@FreeBSD.ORG Fri Jun 28 21:10:54 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 3A9F0834; Fri, 28 Jun 2013 21:10:54 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from mail-wi0-x22d.google.com (mail-wi0-x22d.google.com [IPv6:2a00:1450:400c:c05::22d]) by mx1.freebsd.org (Postfix) with ESMTP id 2B37816F4; Fri, 28 Jun 2013 21:10:53 +0000 (UTC) Received: by mail-wi0-f173.google.com with SMTP id hq4so1250294wib.12 for ; Fri, 28 Jun 2013 14:10:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:reply-to:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=yFWEg61/nFsMuA+QAW8QOXml5VhWHsYJH6O4/xjNiHQ=; b=RElnHC4vhHm/8eGjlGO/axftaQbmAa4gz6LV2JHa4s1m4y5LZkJCsHvYxUJwh1atjd KFfU2aqW/uga+56rLjCOQ2M70JpaRRyPmzCLUtKyVTDU3LwBYvGY764nUx43hU949FrL 8BScmW4U4w8j0VRHkzboEPZuaG5LRVxka7Ll2/Eur0QNiMo/g15fEO6dUh3B19CxUL6F 4lu5PueYV3b2fWCqe7smt7/AQwZcvSvXypE4QE+g9cZC0sdku3Sa46p0R8gjN9ZELy/d t/ajLjEJQouLD6HkKc7RQLpcyc3dgBsMYq7eWJw1IqWHb38HN9yZgDF1NpJwOV0snCO2 SASg== MIME-Version: 1.0 X-Received: by 10.194.174.4 with SMTP id bo4mr11081091wjc.40.1372453852259; Fri, 28 Jun 2013 14:10:52 -0700 (PDT) Sender: asmrookie@gmail.com Received: by 10.194.154.135 with HTTP; Fri, 28 Jun 2013 14:10:52 -0700 (PDT) In-Reply-To: <201306282100.r5SL08kx093999@svn.freebsd.org> References: <201306282100.r5SL08kx093999@svn.freebsd.org> Date: Fri, 28 Jun 2013 23:10:52 +0200 X-Google-Sender-Auth: cS8kwkfU-DGxxD1uSrVY1daamSI Message-ID: Subject: Re: svn commit: r252356 - in head: contrib/smbfs/mount_smbfs etc/defaults etc/mtree include lib lib/libprocstat rescue/rescue sbin/mount share/examples share/examples/etc share/mk sys/conf sys/kern sys... From: Attilio Rao To: Davide Italiano Content-Type: text/plain; charset=UTF-8 Cc: svn-src-head@freebsd.org, FreeBSD FS , svn-src-all@freebsd.org, src-committers@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: attilio@FreeBSD.org List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jun 2013 21:10:54 -0000 On Fri, Jun 28, 2013 at 11:00 PM, Davide Italiano wrote: > Author: davide > Date: Fri Jun 28 21:00:08 2013 > New Revision: 252356 > URL: http://svnweb.freebsd.org/changeset/base/252356 > > Log: > - Trim an unused and bogus Makefile for mount_smbfs. > - Reconnect with some minor modifications, in particular now selsocket() > internals are adapted to use sbintime units after recent'ish calloutng > switch. I've updated the page at: https://wiki.freebsd.org/NONMPSAFE_DEORBIT_VFS which reflects completition of the project and the re-insert of smbfs. During VFS Non-MPSAFE deorbit we lost the following in-kernel support: CodaFS, HPFS, NTFS, NWFS, PortalFS, XFS even if for some of those (most notably NTFS) there is an userland module for FUSE available. Thanks, Attilio -- Peace can only be achieved by understanding - A. Einstein From owner-freebsd-fs@FreeBSD.ORG Sat Jun 29 00:19:35 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 5A270742 for ; Sat, 29 Jun 2013 00:19:35 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id E37F51D67 for ; Sat, 29 Jun 2013 00:19:34 +0000 (UTC) X-Cloudmark-SP-Filtered: true X-Cloudmark-SP-Result: v=1.1 cv=u+Bwc9JL7tMNtl/i9xObSTPSFclN5AOtXcIZY5dPsHA= c=1 sm=2 a=4x594vOIrDwA:10 a=FKkrIqjQGGEA:10 a=-c-2Dywjgf4A:10 a=IkcTkHD0fZMA:10 a=6I5d2MoRAAAA:8 a=0z9gZXBKqnB2xuKoH10A:9 a=QEXdDO2ut3YA:10 a=SV7veod9ZcQA:10 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqMEAEAmzlGDaFve/2dsb2JhbABbgztJgwe8BIEgdIIjAQEBAwEBAQEgKyALBRYYAgINGQIpAQkmBggHBAEcBIdoBgypNJEPgSaMbhB+NAeCUYEWA5E1g0uDbpAcgy0gMoEDNw X-IronPort-AV: E=Sophos;i="4.87,962,1363147200"; d="scan'208";a="37769364" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 28 Jun 2013 20:19:27 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 44EB4B3F4F; Fri, 28 Jun 2013 20:19:27 -0400 (EDT) Date: Fri, 28 Jun 2013 20:19:27 -0400 (EDT) From: Rick Macklem To: Zoltan Arnold NAGY Message-ID: <2073363197.51063.1372465167242.JavaMail.root@uoguelph.ca> In-Reply-To: Subject: Re: ZFS-backed NFS export with vSphere MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.203] X-Mailer: Zimbra 7.2.1_GA_2790 (ZimbraWebClient - FF3.0 (Win)/7.2.1_GA_2790) Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Jun 2013 00:19:35 -0000 Zoltan Nagy wrote: > > > > > Right. As I said, increasing it to 1M increased my throughput from > 17MB/s to 76MB/s. > I vaguely recall running into problems with UFS when I made MAXBSIZE > 128Kbytes, but it has been a while since I tried it. Even doubling it would be a significant change in the default, with the potential for secondary effects I have no idea on. (That's why I haven't even doubled it in head as of yet. I may get that daring someday, after having an email list discussion about implications of it.) You didn't mention what client you are using? (The FreeBSD client won't do rsize/wsize greater than MAXBSIZE either, so it will be capped at 64Kbytes.) You can also try things like changing rsize,wsize and readahead in the client mount. (Most clients have some variant of these mount options.) Also, ken@ recently added NFSv3 file handle affinity support, which apparently helps reading from ZFS, since without it some ZFS algorithm that recognized sequential access fails and that slows things down. (ie. You could try a recent 10/current or stable/9 on the server and see what effect that has.) > However, the SSD can do much more random writes; any idea why I don't > see the ZIL go over > this value? > (vSphere always uses sync writes). > That's a question for someone else. I have never used ZFS, rick > Thanks, > Zoltan > > > > > On Thu, Jun 27, 2013 at 11:58 PM, Rick Macklem < rmacklem@uoguelph.ca > > wrote: > > > > > Zoltan Nagy wrote: > > Hi list, > > > > I'd love to have a ZFS-backed NFS export as my VM datastore, but as > > much as > > I'd like to tune > > it, the performance doesn't even get close to Solaris 11's. > > > > I currently have the system set up as this: > > > > pool: tank > > state: ONLINE > > scan: none requested > > config: > > > > NAME STATE READ WRITE CKSUM > > tank ONLINE 0 0 0 > > mirror-0 ONLINE 0 0 0 > > da0 ONLINE 0 0 0 > > da1 ONLINE 0 0 0 > > mirror-1 ONLINE 0 0 0 > > da2 ONLINE 0 0 0 > > da3 ONLINE 0 0 0 > > logs > > ada0p4 ONLINE 0 0 0 > > spares > > da4 AVAIL > > > > ada0 is a samsung 840pro SSD, which I'm using for system+ZIL. > > daX is 1TB, 7200rpm seagate disks. > > (from this test's perspective, if I use a separate ZIL device or > > just > > a > > partition, doesn't matter - I get roughly the same numbers). > > > > The first thing I noticed is that the FSINFO reply from FreeBSD is > > advertising untunable values (I did not find them documented either > > in the > > manpage, or as a sysctl). > > > > rtmax, rtpref, wtmax, wtpref: 64k (fbsd), 1M (solaris) > > dtpref: 64k (fbsd), 8k (solaris) > > > > After manually patching the nfs code (changing NFS_MAXBSIZE to 1M > > instead > > of MAXBSIZE) to adversize the same read/write values (didn't touch > > dtpref), > > my performance went up from 17MB/s to 76MB/s. > > > > Is there a reason NFS_MAXBSIZE is not tunable and/or is it so slow? > > > For exporting other file system types (UFS, ...) the buffer cache is > used and MAXBSIZE is the largest block you can use for the buffer > cache. > Some increase of MAXBSIZE would be nice. (I've tried 128Kb without > observing > difficulties and from what I've been told 128Kb is the ZFS block > size.) > > > > > Here's my iozone output (which is run on an ext4 partition created > > on > > a > > linux VM which has a disk backed by the NFS exported from the > > FreeBSD > > box): > > > > Record Size 4096 KB > > File size set to 2097152 KB > > Command line used: iozone -b results.xls -r 4m -s 2g -t 6 -i 0 -i > > 1 -i 2 > > Output is in Kbytes/sec > > Time Resolution = 0.000001 seconds. > > Processor cache size set to 1024 Kbytes. > > Processor cache line size set to 32 bytes. > > File stride size set to 17 * record size. > > Throughput test with 6 processes > > Each process writes a 2097152 Kbyte file in 4096 Kbyte records > > > > Children see throughput for 6 initial writers = 76820.31 > > KB/sec > > Parent sees throughput for 6 initial writers = 74899.44 > > KB/sec > > Min throughput per process = 12298.62 KB/sec > > Max throughput per process = 12972.72 KB/sec > > Avg throughput per process = 12803.38 KB/sec > > Min xfer = 1990656.00 KB > > > > Children see throughput for 6 rewriters = 76030.99 KB/sec > > Parent sees throughput for 6 rewriters = 75062.91 KB/sec > > Min throughput per process = 12620.45 KB/sec > > Max throughput per process = 12762.80 KB/sec > > Avg throughput per process = 12671.83 KB/sec > > Min xfer = 2076672.00 KB > > > > Children see throughput for 6 readers = 114221.39 > > KB/sec > > Parent sees throughput for 6 readers = 113942.71 KB/sec > > Min throughput per process = 18920.14 KB/sec > > Max throughput per process = 19183.80 KB/sec > > Avg throughput per process = 19036.90 KB/sec > > Min xfer = 2068480.00 KB > > > > Children see throughput for 6 re-readers = 117018.50 KB/sec > > Parent sees throughput for 6 re-readers = 116917.01 KB/sec > > Min throughput per process = 19436.28 KB/sec > > Max throughput per process = 19590.40 KB/sec > > Avg throughput per process = 19503.08 KB/sec > > Min xfer = 2080768.00 KB > > > > Children see throughput for 6 random readers = 110072.68 > > KB/sec > > Parent sees throughput for 6 random readers = 109698.99 > > KB/sec > > Min throughput per process = 18260.33 KB/sec > > Max throughput per process = 18442.55 KB/sec > > Avg throughput per process = 18345.45 KB/sec > > Min xfer = 2076672.00 KB > > > > Children see throughput for 6 random writers = 76389.71 > > KB/sec > > Parent sees throughput for 6 random writers = 74816.45 > > KB/sec > > Min throughput per process = 12592.09 KB/sec > > Max throughput per process = 12843.75 KB/sec > > Avg throughput per process = 12731.62 KB/sec > > Min xfer = 2056192.00 KB > > > > The other interesting this is that you can notice the system > > doesn't > > cache > > the data file to ram (the box has 32G), so even for re-reads I get > > miserable numbers. With solaris, the re-reads happen at nearly wire > > spead. > > > > Any ideas what else I could tune? While 76MB/s is much better than > > the > > original 17MB I was seeing, it's still far from Solaris's > > ~220MB/s... > > > > Thanks a lot, > > Zoltan > > > > _______________________________________________ > > freebsd-fs@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > > To unsubscribe, send any mail to " > > freebsd-fs-unsubscribe@freebsd.org " > > > > From owner-freebsd-fs@FreeBSD.ORG Sat Jun 29 05:43:27 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 28C2AC5F for ; Sat, 29 Jun 2013 05:43:27 +0000 (UTC) (envelope-from gabor@zahemszky.hu) Received: from default-smtp.integrity.hu (default-smtp.integrity.hu [212.52.165.203]) by mx1.freebsd.org (Postfix) with ESMTP id E19281984 for ; Sat, 29 Jun 2013 05:43:25 +0000 (UTC) Received: by smtp.integrity.hu (Postfix, from userid 10000) id 3B53413468BB; Sat, 29 Jun 2013 07:43:18 +0200 (CEST) Received: from webmail.integrity.hu (mail-fe-1.integrity.hu [10.1.64.120]) (Authenticated sender: gabor@zahemszky.hu) by smtp.integrity.hu (Postfix) with ESMTPA id BA5BD13468B3 for ; Sat, 29 Jun 2013 07:43:17 +0200 (CEST) Received: from 6L/a03BugnnTrrUqtoKnF8OVspdzwzW/pjYKFLVjSj+56q+9pKzitg== (7oq7C8Ff4Cq/8Sx6qLRGQoC0ZuRNNyaK) by webmail.integrity.hu with HTTP (HTTP/1.1 POST); Sat, 29 Jun 2013 07:43:16 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Date: Sat, 29 Jun 2013 07:43:16 +0200 From: gabor@zahemszky.hu To: Subject: Re: ext2fs bug In-Reply-To: References: <9554b1ed02a342c347b17a2e71e38b73@zahemszky.hu> <20130623093521.GA86307@icarus.home.lan> <20130623095207.GA86507@icarus.home.lan> <216bbb1389bb7e6e5ef11500d31204ad@zahemszky.hu> <20130626044515.GA67521@icarus.home.lan> <51CB4968.3080503@FreeBSD.org> Message-ID: <55383573d731798d9b683e31e401ce2a@zahemszky.hu> X-Sender: gabor@zahemszky.hu User-Agent: Roundcube Webmail/0.8.4 X-Virus-Scanned: clamav-milter 0.97.6 at mail-autosubmit X-Virus-Status: Clean X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Jun 2013 05:43:27 -0000 Hi list, hi Pedro! The next part of my ext2fs saga. Yesterday, I downloaded FreeBSD-10.0-HEAD-r252302-JPSNAP-amd64-amd64-memstick.img FreeBSD-9.1-RELENG_9-r252246-JPSNAP-amd64-amd64-memstick.img these daily snapshot images from allbsd.org. Rebboted the machine, and got the same problem with both of them: - some missing directories - some missing files - reaching some of the available files generate errors and when I try to list / reach these files, got the next errors on console: g_vfs_done():ada2s1[READ(offset=-756558786560, length=4096)]error = 5 g_vfs_done():ada2s1[READ(offset=-899189821440, length=4096)]error = 5 g_vfs_done():ada2s1[READ(offset=-756558786560, length=4096)]error = 5 So nothing changed. Gabor < Gabor at Zahemszky dot HU > From owner-freebsd-fs@FreeBSD.ORG Sat Jun 29 11:00:02 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 4737097A for ; Sat, 29 Jun 2013 11:00:02 +0000 (UTC) (envelope-from longwitz@incore.de) Received: from dss.incore.de (dss.incore.de [195.145.1.138]) by mx1.freebsd.org (Postfix) with ESMTP id CB2131169 for ; Sat, 29 Jun 2013 11:00:01 +0000 (UTC) Received: from inetmail.dmz (inetmail.dmz [10.3.0.3]) by dss.incore.de (Postfix) with ESMTP id 716915C31E for ; Sat, 29 Jun 2013 12:54:49 +0200 (CEST) X-Virus-Scanned: amavisd-new at incore.de Received: from dss.incore.de ([10.3.0.3]) by inetmail.dmz (inetmail.dmz [10.3.0.3]) (amavisd-new, port 10024) with LMTP id 1bmLVREKHaNN for ; Sat, 29 Jun 2013 12:54:48 +0200 (CEST) Received: from mail.incore (fwintern.dmz [10.0.0.253]) by dss.incore.de (Postfix) with ESMTP id 1E8495C149 for ; Sat, 29 Jun 2013 12:54:48 +0200 (CEST) Received: from bsdmhs.longwitz (unknown [192.168.99.6]) by mail.incore (Postfix) with ESMTP id 7772E50881 for ; Sat, 29 Jun 2013 12:54:47 +0200 (CEST) Message-ID: <51CEBCF7.5040505@incore.de> Date: Sat, 29 Jun 2013 12:54:47 +0200 From: Andreas Longwitz User-Agent: Thunderbird 2.0.0.19 (X11/20090113) MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: Reproducable ZFS crash when starting a jail in 8-stable Content-Type: text/plain; charset=ISO-8859-15 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Jun 2013 11:00:02 -0000 The problem occurs after an update of 8-stable from r248120 to r252111. My server has system disks da0, da1 with gmirror/gjournal for rootfs, usr, var and home partitions and glabeled data disks da2, da3 with zfs for prod and backup. Applications run in two jails on the zfs disks with nullfs mounts: At boot the servers crashs when /etc/rc.d/jail tries to start the jails, on the console I see (I use ddb.conf to handle crash by ddb): Fatal trap 12: page fault while in kernel mode cpuid = 1; apic id = 01 fault virtual address = 0x0 fault code = supervisor read instruction, page not present instruction pointer = 0x20:0x0 stack pointer = 0x28:0xffffff8245853930 frame pointer = 0x28:0xffffff82458539e0 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 4411 (initial thread) [thread pid 4411 tid 100460 ] Stopped at 0: *** error reading from address 0 *** db:0:kdb.enter.default> watchdog No argument provided, disabling watchdog db:0:kdb.enter.default> call doadump Dumping 472 out of 8179 MB:..4%..11%..21%..31%..41%..51%..61%..72%..82%..92% Dump complete >From kgdb output I see pid 4411 is the zfs/initial thread: (kgdb) where #0 doadump () at /usr/src/sys/kern/kern_shutdown.c:266 #1 0xffffffff801f877c in db_fncall (dummy1=, dummy2=, dummy3=, dummy4=) at /usr/src/sys/ddb/db_command.c:548 #2 0xffffffff801f8a2d in db_command (last_cmdp=0xffffffff8086b5c0, cmd_table=, dopager=0) at /usr/src/sys/ddb/db_command.c:445 #3 0xffffffff801fd0e3 in db_script_exec (scriptname=0xffffffff80657b9e "kdb.enter.default", warnifnotfound=0) at /usr/src/sys/ddb/db_script.c:302 #4 0xffffffff801fd1de in db_script_kdbenter (eventname=) at /usr/src/sys/ddb/db_script.c:325 #5 0xffffffff801fadc4 in db_trap (type=, code=) at /usr/src/sys/ddb/db_main.c:230 #6 0xffffffff80432981 in kdb_trap (type=12, code=0, tf=0xffffff8245853880) at /usr/src/sys/kern/subr_kdb.c:654 #7 0xffffffff805dbbed in trap_fatal (frame=0xffffff8245853880, eva=) at /usr/src/sys/amd64/amd64/trap.c:844 #8 0xffffffff805dbf6e in trap_pfault (frame=0xffffff8245853880, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:765 #9 0xffffffff805dc32b in trap (frame=0xffffff8245853880) at /usr/src/sys/amd64/amd64/trap.c:457 #10 0xffffffff805c2534 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:228 #11 0x0000000000000000 in ?? () (kgdb) info thread * 412 Thread 100460 (PID=4411: zfs/initial thread) doadump () at /usr/src/sys/kern/kern_shutdown.c:266 411 Thread 100461 (PID=4404: sh) sched_switch (td=0xffffff009e26d000, newtd=, flags=) at /usr/src/sys/kern/sched_ule.c:1932 .... 221 Thread 100272 (PID=7: zfskern/txg_thread_enter) sched_switch (td=0xffffff0002c95000, newtd=, flags=) at /usr/src/sys/kern/sched_ule.c:1932 220 Thread 100271 (PID=7: zfskern/txg_thread_enter) sched_switch (td=0xffffff0002c95470, newtd=, flags=) at /usr/src/sys/kern/sched_ule.c:1932 219 Thread 100069 (PID=7: zfskern/l2arc_feed_thread) sched_switch (td=0xffffff0002957000, newtd=, flags=) at /usr/src/sys/kern/sched_ule.c:1932 218 Thread 100068 (PID=7: zfskern/arc_reclaim_thread) sched_switch (td=0xffffff0002957470, newtd=, flags=) at /usr/src/sys/kern/sched_ule.c:1932 ... 156 Thread 100270 (PID=0: kernel/zfs_vn_rele_taskq) sched_switch (td=0xffffff0002c958e0, newtd=, flags=) at /usr/src/sys/kern/sched_ule.c:1932 155 Thread 100269 (PID=0: kernel/zio_ioctl_intr) sched_switch (td=0xffffff0002d92470, newtd=, flags=) at /usr/src/sys/kern/sched_ule.c:1932 154 Thread 100268 (PID=0: kernel/zio_ioctl_issue) sched_switch (td=0xffffff0002d9e000, newtd=, flags=) at /usr/src/sys/kern/sched_ule.c:1932 153 Thread 100267 (PID=0: kernel/zio_claim_intr) sched_switch (td=0xffffff0002d9e8e0, newtd=, flags=) at /usr/src/sys/kern/sched_ule.c:1932 152 Thread 100266 (PID=0: kernel/zio_claim_issue) sched_switch (td=0xffffff0002d9a8e0, newtd=, flags=) at /usr/src/sys/kern/sched_ule.c:1932 .... >From the kerneldump I can give backtraces of all threads or any other information. Some more informations: === root@serv07 (pts/1) -> gmirror status Name Status Components mirror/gmsv07 COMPLETE da0 (ACTIVE) da1 (ACTIVE) === root@serv07 (pts/1) -> glabel status Name Status Components label/9241A7D4 N/A da2 label/C2477N17 N/A da3 === root@serv07 (pts/2) -> zpool status pool: mpool state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM mpool ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 label/9241A7D4 ONLINE 0 0 0 label/C2477N17 ONLINE 0 0 0 errors: No known data errors === root@serv07 (pts/2) -> zpool list NAME USED AVAIL REFER MOUNTPOINT mpool 108G 806G 31K /mpool mpool/backup 545M 806G 485M /backup mpool/jail_deb_backup 33K 806G 33K /backup/jail/deb mpool/jail_deb_prod 32,6G 806G 32,6G /prod/jail/deb mpool/jail_pvz_backup 32K 806G 32K /backup/jail/pvz mpool/jail_pvz_prod 54,7G 806G 54,7G /prod/jail/pvz mpool/prod 20,0G 806G 20,0G /prod cat /etc/fstab.deb (fstab.pvz analogue): # Device Mountpoint FStype Options Dump Pass# /usr/jail/deb /jail/deb/usr nullfs rw 0 0 /var/jail/deb /jail/deb/var nullfs rw 0 0 /home/jail/deb /jail/deb/home nullfs rw 0 0 /tmp/jail/deb /jail/deb/tmp nullfs rw 0 0 /prod/jail/deb /jail/deb/prod nullfs rw 0 0 /backup/jail/deb /jail/deb/backup nullfs rw 0 0 On server without zfs the problem does not exist, jails on r252111 are running fine. Andreas Longwitz From owner-freebsd-fs@FreeBSD.ORG Sat Jun 29 19:53:14 2013 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 7FF3EB42 for ; Sat, 29 Jun 2013 19:53:14 +0000 (UTC) (envelope-from pfg@FreeBSD.org) Received: from nm2-vm2.bullet.mail.gq1.yahoo.com (nm2-vm2.bullet.mail.gq1.yahoo.com [98.136.218.129]) by mx1.freebsd.org (Postfix) with ESMTP id 4BE391232 for ; Sat, 29 Jun 2013 19:53:13 +0000 (UTC) Received: from [98.137.12.191] by nm2.bullet.mail.gq1.yahoo.com with NNFMP; 29 Jun 2013 19:51:16 -0000 Received: from [208.71.42.207] by tm12.bullet.mail.gq1.yahoo.com with NNFMP; 29 Jun 2013 19:51:16 -0000 Received: from [127.0.0.1] by smtp218.mail.gq1.yahoo.com with NNFMP; 29 Jun 2013 19:51:16 -0000 X-Yahoo-Newman-Id: 344041.75380.bm@smtp218.mail.gq1.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: rtc8KdcVM1kFgOwFQXRGt9poOrQSNzh9saQHzbfP_3bCWRx ctcdpJKVfBvDCLX1JorVo8HrYe8PXEdmiAp6HLfwdheX7W_V3sfxphvzMgAh zKCucqQtcsHWmx9t3U.Pg7b6bfuws9dVj1GMImh5c6S3Vg7fZC.vNEGNL0_U R4rJ3YmAvSACs9BRUJIWNJi5Qgmo4WTr4JZ1iWyzMmoWtqakCiJJpdmo9P28 FVVKmcpB.Jqn0rNS5uaNz0fceBSFRb83MmXL6Zoqas_vLtUe0oahFMwwJDbv GLFfY6vVzzF1pqLuYn2EzfsPv.2wYEPN2KQx7IJisraR1f38x0g522FphIsk 5bz6iiIQf43UIqRwim_l2P2U4Vtftm6kuMj_HSFbvBG3Ct62AU90X_VFtszS .WqYDuhD_QtgySx4YFkk1WPlzHnYRZhV2JfKUuNjhsIJ1lqRdagtqDlXu32a y2FqzSSXM1GTqSIah7RASC3OBYQvXTc8oWPdV4SL7qtuVlSx21kMFS.5WxrB LqIQge84L90Ho73G0APC4TSfofypxbqQOLfHebq8zrc8Bq6Bjs3wyoC4CR59 75AptAlfvkNku X-Yahoo-SMTP: xcjD0guswBAZaPPIbxpWwLcp9Unf X-Rocket-Received: from [192.168.0.102] (pfg@190.157.126.109 with ) by smtp218.mail.gq1.yahoo.com with SMTP; 29 Jun 2013 19:51:14 +0000 UTC Message-ID: <51CF3AAF.6040503@FreeBSD.org> Date: Sat, 29 Jun 2013 14:51:11 -0500 From: Pedro Giffuni User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130611 Thunderbird/17.0.6 MIME-Version: 1.0 To: =?ISO-8859-1?Q?Zahemszky_G=E1bor?= Subject: Re: ext2fs bug References: <1372048524.86572.YahooMailMobile@web162106.mail.bf1.yahoo.com> <20130624072032.775c1eed@Picasso.Zahemszky.HU> In-Reply-To: <20130624072032.775c1eed@Picasso.Zahemszky.HU> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Jun 2013 19:53:14 -0000 (Hope the message makes it through this time) Hello Gabor; So far I have been only doing cleanups to the filesystem so that it looks like and behaves more like UFS. Unfortunately there are some old bugs in the ext2fs support. I think if I just bump the block numbers to 64 bits the main problems you are seeing may be gone but it would be like an attempt to hide the garbage under the carpet. There is some interesting work coming to ext2fs hopefully we will get some time later on to fix the issues properly. My current disks are much smaller than yours so I can't reproduce the issue. Sorry that I can't help but, did I mention ZFS is much better? ;). Pedro. From owner-freebsd-fs@FreeBSD.ORG Sat Jun 29 22:07:49 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 8AA7990D for ; Sat, 29 Jun 2013 22:07:49 +0000 (UTC) (envelope-from mm@FreeBSD.org) Received: from mail.vx.sk (mail.vx.sk [176.9.45.25]) by mx1.freebsd.org (Postfix) with ESMTP id 2F11917F2 for ; Sat, 29 Jun 2013 22:07:48 +0000 (UTC) Received: from core.vx.sk (localhost [127.0.0.2]) by mail.vx.sk (Postfix) with ESMTP id 0073935C7E; Sun, 30 Jun 2013 00:07:48 +0200 (CEST) X-Virus-Scanned: amavisd-new at mail.vx.sk Received: from mail.vx.sk by core.vx.sk (amavisd-new, unix socket) with LMTP id RtVOoljWdKnz; Sun, 30 Jun 2013 00:07:47 +0200 (CEST) Received: from [10.9.8.144] (chello085216226145.chello.sk [85.216.226.145]) by mail.vx.sk (Postfix) with ESMTPSA id 3C82635C77; Sun, 30 Jun 2013 00:07:47 +0200 (CEST) Message-ID: <51CF5AB0.8040406@FreeBSD.org> Date: Sun, 30 Jun 2013 00:07:44 +0200 From: Martin Matuska User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130623 Thunderbird/17.0.7 MIME-Version: 1.0 To: Andreas Longwitz Subject: Re: Reproducable ZFS crash when starting a jail in 8-stable References: <51CEBCF7.5040505@incore.de> In-Reply-To: <51CEBCF7.5040505@incore.de> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Jun 2013 22:07:49 -0000 Fixed in r252380 (head), MFC scheduled for July 2, 2013 On 2013-06-29 12:54, Andreas Longwitz wrote: > The problem occurs after an update of 8-stable from r248120 to r252111. > > My server has system disks da0, da1 with gmirror/gjournal for rootfs, > usr, var and home partitions and glabeled data disks da2, da3 with zfs > for prod and backup. Applications run in two jails on the zfs disks with > nullfs mounts: > > At boot the servers crashs when /etc/rc.d/jail tries to start the jails, > on the console I see (I use ddb.conf to handle crash by ddb): > > Fatal trap 12: page fault while in kernel mode > cpuid = 1; apic id = 01 > fault virtual address = 0x0 > fault code = supervisor read instruction, page not present > instruction pointer = 0x20:0x0 > stack pointer = 0x28:0xffffff8245853930 > frame pointer = 0x28:0xffffff82458539e0 > code segment = base 0x0, limit 0xfffff, type 0x1b > = DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags = interrupt enabled, resume, IOPL = 0 > current process = 4411 (initial thread) > [thread pid 4411 tid 100460 ] > Stopped at 0: *** error reading from address 0 *** > db:0:kdb.enter.default> watchdog > No argument provided, disabling watchdog > db:0:kdb.enter.default> call doadump > Dumping 472 out of 8179 MB:..4%..11%..21%..31%..41%..51%..61%..72%..82%..92% > Dump complete > > From kgdb output I see pid 4411 is the zfs/initial thread: > > (kgdb) where > #0 doadump () at /usr/src/sys/kern/kern_shutdown.c:266 > #1 0xffffffff801f877c in db_fncall (dummy1=, > dummy2=, > dummy3=, dummy4=) at > /usr/src/sys/ddb/db_command.c:548 > #2 0xffffffff801f8a2d in db_command (last_cmdp=0xffffffff8086b5c0, > cmd_table=, dopager=0) > at /usr/src/sys/ddb/db_command.c:445 > #3 0xffffffff801fd0e3 in db_script_exec (scriptname=0xffffffff80657b9e > "kdb.enter.default", warnifnotfound=0) > at /usr/src/sys/ddb/db_script.c:302 > #4 0xffffffff801fd1de in db_script_kdbenter (eventname= out>) at /usr/src/sys/ddb/db_script.c:325 > #5 0xffffffff801fadc4 in db_trap (type=, > code=) > at /usr/src/sys/ddb/db_main.c:230 > #6 0xffffffff80432981 in kdb_trap (type=12, code=0, > tf=0xffffff8245853880) at /usr/src/sys/kern/subr_kdb.c:654 > #7 0xffffffff805dbbed in trap_fatal (frame=0xffffff8245853880, > eva=) > at /usr/src/sys/amd64/amd64/trap.c:844 > #8 0xffffffff805dbf6e in trap_pfault (frame=0xffffff8245853880, > usermode=0) at /usr/src/sys/amd64/amd64/trap.c:765 > #9 0xffffffff805dc32b in trap (frame=0xffffff8245853880) at > /usr/src/sys/amd64/amd64/trap.c:457 > #10 0xffffffff805c2534 in calltrap () at > /usr/src/sys/amd64/amd64/exception.S:228 > #11 0x0000000000000000 in ?? () > > (kgdb) info thread > * 412 Thread 100460 (PID=4411: zfs/initial thread) doadump () at > /usr/src/sys/kern/kern_shutdown.c:266 > 411 Thread 100461 (PID=4404: sh) sched_switch (td=0xffffff009e26d000, > newtd=, > flags=) at /usr/src/sys/kern/sched_ule.c:1932 > .... > 221 Thread 100272 (PID=7: zfskern/txg_thread_enter) sched_switch > (td=0xffffff0002c95000, newtd=, > flags=) at /usr/src/sys/kern/sched_ule.c:1932 > 220 Thread 100271 (PID=7: zfskern/txg_thread_enter) sched_switch > (td=0xffffff0002c95470, newtd=, > flags=) at /usr/src/sys/kern/sched_ule.c:1932 > 219 Thread 100069 (PID=7: zfskern/l2arc_feed_thread) sched_switch > (td=0xffffff0002957000, newtd=, > flags=) at /usr/src/sys/kern/sched_ule.c:1932 > 218 Thread 100068 (PID=7: zfskern/arc_reclaim_thread) sched_switch > (td=0xffffff0002957470, > newtd=, flags=) at > /usr/src/sys/kern/sched_ule.c:1932 > ... > 156 Thread 100270 (PID=0: kernel/zfs_vn_rele_taskq) sched_switch > (td=0xffffff0002c958e0, newtd=, > flags=) at /usr/src/sys/kern/sched_ule.c:1932 > 155 Thread 100269 (PID=0: kernel/zio_ioctl_intr) sched_switch > (td=0xffffff0002d92470, newtd=, > flags=) at /usr/src/sys/kern/sched_ule.c:1932 > 154 Thread 100268 (PID=0: kernel/zio_ioctl_issue) sched_switch > (td=0xffffff0002d9e000, newtd=, > flags=) at /usr/src/sys/kern/sched_ule.c:1932 > 153 Thread 100267 (PID=0: kernel/zio_claim_intr) sched_switch > (td=0xffffff0002d9e8e0, newtd=, > flags=) at /usr/src/sys/kern/sched_ule.c:1932 > 152 Thread 100266 (PID=0: kernel/zio_claim_issue) sched_switch > (td=0xffffff0002d9a8e0, newtd=, > flags=) at /usr/src/sys/kern/sched_ule.c:1932 > .... > > From the kerneldump I can give backtraces of all threads or any other > information. > > Some more informations: > > === root@serv07 (pts/1) -> gmirror status > Name Status Components > mirror/gmsv07 COMPLETE da0 (ACTIVE) > da1 (ACTIVE) > > === root@serv07 (pts/1) -> glabel status > Name Status Components > label/9241A7D4 N/A da2 > label/C2477N17 N/A da3 > > === root@serv07 (pts/2) -> zpool status > pool: mpool > state: ONLINE > scan: none requested > config: > > NAME STATE READ WRITE CKSUM > mpool ONLINE 0 0 0 > mirror-0 ONLINE 0 0 0 > label/9241A7D4 ONLINE 0 0 0 > label/C2477N17 ONLINE 0 0 0 > > errors: No known data errors > > === root@serv07 (pts/2) -> zpool list > NAME USED AVAIL REFER MOUNTPOINT > mpool 108G 806G 31K /mpool > mpool/backup 545M 806G 485M /backup > mpool/jail_deb_backup 33K 806G 33K /backup/jail/deb > mpool/jail_deb_prod 32,6G 806G 32,6G /prod/jail/deb > mpool/jail_pvz_backup 32K 806G 32K /backup/jail/pvz > mpool/jail_pvz_prod 54,7G 806G 54,7G /prod/jail/pvz > mpool/prod 20,0G 806G 20,0G /prod > > cat /etc/fstab.deb (fstab.pvz analogue): > # Device Mountpoint FStype Options Dump Pass# > /usr/jail/deb /jail/deb/usr nullfs rw 0 0 > /var/jail/deb /jail/deb/var nullfs rw 0 0 > /home/jail/deb /jail/deb/home nullfs rw 0 0 > /tmp/jail/deb /jail/deb/tmp nullfs rw 0 0 > /prod/jail/deb /jail/deb/prod nullfs rw 0 0 > /backup/jail/deb /jail/deb/backup nullfs rw 0 0 > > On server without zfs the problem does not exist, jails on r252111 > are running fine. > > Andreas Longwitz > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Sat Jun 29 23:22:33 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id A42E8399; Sat, 29 Jun 2013 23:22:33 +0000 (UTC) (envelope-from longwitz@incore.de) Received: from dss.incore.de (dss.incore.de [195.145.1.138]) by mx1.freebsd.org (Postfix) with ESMTP id 69895197E; Sat, 29 Jun 2013 23:22:33 +0000 (UTC) Received: from inetmail.dmz (inetmail.dmz [10.3.0.3]) by dss.incore.de (Postfix) with ESMTP id 3735F5C163; Sun, 30 Jun 2013 01:22:32 +0200 (CEST) X-Virus-Scanned: amavisd-new at incore.de Received: from dss.incore.de ([10.3.0.3]) by inetmail.dmz (inetmail.dmz [10.3.0.3]) (amavisd-new, port 10024) with LMTP id HcU9kpANewCb; Sun, 30 Jun 2013 01:22:31 +0200 (CEST) Received: from mail.incore (fwintern.dmz [10.0.0.253]) by dss.incore.de (Postfix) with ESMTP id 59F585C027; Sun, 30 Jun 2013 01:22:31 +0200 (CEST) Received: from bsdmhs.longwitz (unknown [192.168.99.6]) by mail.incore (Postfix) with ESMTP id 0F66350886; Sun, 30 Jun 2013 01:22:30 +0200 (CEST) Message-ID: <51CF6C36.5080208@incore.de> Date: Sun, 30 Jun 2013 01:22:30 +0200 From: Andreas Longwitz User-Agent: Thunderbird 2.0.0.19 (X11/20090113) MIME-Version: 1.0 To: Martin Matuska Subject: Re: Reproducable ZFS crash when starting a jail in 8-stable References: <51CEBCF7.5040505@incore.de> <51CF5AB0.8040406@FreeBSD.org> In-Reply-To: <51CF5AB0.8040406@FreeBSD.org> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Jun 2013 23:22:33 -0000 Martin Matuska wrote: > Fixed in r252380 (head), MFC scheduled for July 2, 2013 Thanks for quick response. With the patch r252380 (from head) my 8-stable system using zfs and jails runs ok now. Andreas Longwitz