From owner-freebsd-fs@FreeBSD.ORG Sun Feb 1 07:24:40 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AFD3C10657A4 for ; Sun, 1 Feb 2009 07:24:40 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from mail35.syd.optusnet.com.au (mail35.syd.optusnet.com.au [211.29.133.51]) by mx1.freebsd.org (Postfix) with ESMTP id 3FAC48FC22 for ; Sun, 1 Feb 2009 07:24:40 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from server.vk2pj.dyndns.org (c122-106-216-167.belrs3.nsw.optusnet.com.au [122.106.216.167]) by mail35.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id n117Oass022221 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sun, 1 Feb 2009 18:24:38 +1100 X-Bogosity: Ham, spamicity=0.000000 Received: from server.vk2pj.dyndns.org (localhost.vk2pj.dyndns.org [127.0.0.1]) by server.vk2pj.dyndns.org (8.14.3/8.14.3) with ESMTP id n117OaKE093694; Sun, 1 Feb 2009 18:24:36 +1100 (EST) (envelope-from peter@server.vk2pj.dyndns.org) Received: (from peter@localhost) by server.vk2pj.dyndns.org (8.14.3/8.14.3/Submit) id n117OXqK093693; Sun, 1 Feb 2009 18:24:33 +1100 (EST) (envelope-from peter) Date: Sun, 1 Feb 2009 18:24:32 +1100 From: Peter Jeremy To: Doug Rabson Message-ID: <20090201072432.GA25276@server.vk2pj.dyndns.org> References: <9461581F-F354-486D-961D-3FD5B1EF007C@rabson.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="h31gzZEtNLTqOjlF" Content-Disposition: inline In-Reply-To: <9461581F-F354-486D-961D-3FD5B1EF007C@rabson.org> X-PGP-Key: http://members.optusnet.com.au/peterjeremy/pubkey.asc User-Agent: Mutt/1.5.18 (2008-05-17) Cc: freebsd-fs@freebsd.org Subject: Re: Booting from ZFS raidz X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 01 Feb 2009 07:24:41 -0000 --h31gzZEtNLTqOjlF Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2008-Dec-17 18:25:51 +0000, Doug Rabson wrote: >I've been working on adding raidz and raidz2 support to the boot code =20 >and I have a patch which could use some testing if anyone here is =20 >interested. This http://people.freebsd.org/~dfr/=20 >raidzboot-17122008.diff adds support for raidz and raidz2. The easiest =20 >way to prepare a bootable pool is to put a GPT boot partition on each =20 >disk that will make up the raidz pool and install gptzfsboot on the =20 >boot partition of every drive. This sounds great so I thought I'd try it. Unfortunately, it didn't work on my degraded pool [ZFS managed to kill a disk and I thought I'd experiment]. When I tried to boot, I got: ZFS: i/o error - all block copies unavailable ZFS: can't read MOS ZFS: unexpected object set type lld FreeBSD/i386 boot Default: tank:/boot/loader boot: The boot loader is up-to-date and was built with 'LOADER_ZFS_SUPPORT'. Any ideas? --=20 Peter Jeremy --h31gzZEtNLTqOjlF Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.10 (FreeBSD) iEYEARECAAYFAkmFTjAACgkQ/opHv/APuIfVNgCgwPyiB/cG5TMRKtw1GUSPRUhu lE4AoLy/TiVSXxfdIab+DFKpmeKsMW/s =wsDR -----END PGP SIGNATURE----- --h31gzZEtNLTqOjlF-- From owner-freebsd-fs@FreeBSD.ORG Mon Feb 2 11:06:51 2009 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0460810656BD for ; Mon, 2 Feb 2009 11:06:51 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id DB8EC8FC2A for ; Mon, 2 Feb 2009 11:06:50 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id n12B6oep094417 for ; Mon, 2 Feb 2009 11:06:50 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id n12B6oQP094413 for freebsd-fs@FreeBSD.org; Mon, 2 Feb 2009 11:06:50 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 2 Feb 2009 11:06:50 GMT Message-Id: <200902021106.n12B6oQP094413@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-fs@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Feb 2009 11:06:52 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/131086 fs [ext2fs] mkfs.ext2 creates rotten partition o kern/131084 fs [xfs] xfs destroys itself after copying data o kern/131081 fs [zfs] User cannot delete a file when a ZFS dataset is o kern/131009 fs [ext2fs] [hang] System freezes when attempting to copy o kern/130979 fs [smbfs] [panic] boot/kernel/smbfs.ko o kern/130920 fs [msdosfs] cp(1) takes 100% CPU time while copying file o kern/130229 fs [iconv] usermount fails on fs that need iconv o kern/130210 fs [nullfs] Error by check nullfs o bin/130105 fs [zfs] zfs send -R dumps core o kern/129760 fs [nfs] after 'umount -f' of a stale NFS share FreeBSD l o kern/129231 fs [ufs] [patch] New UFS mount (norandom) option - mostly o kern/129174 fs [nfs] [zfs] [panic] NFS v3 Panic when under high load o kern/129152 fs [panic] non-userfriendly panic when trying to mount(8) o kern/129084 fs [udf] [panic] udf panic: getblk: size(67584) > MAXBSIZ f kern/128829 fs smbd(8) causes periodic panic on 7-RELEASE o kern/128633 fs [zfs] [lor] lock order reversal in zfs o kern/128514 fs [zfs] [mpt] problems with ZFS and LSILogic SAS/SATA Ad f kern/128173 fs [ext2fs] ls gives "Input/output error" on mounted ext3 o kern/127420 fs [gjournal] [panic] Journal overflow on gmirrored gjour o kern/127213 fs [tmpfs] sendfile on tmpfs data corruption o kern/127029 fs [panic] mount(8): trying to mount a write protected zi o kern/126287 fs [ufs] [panic] Kernel panics while mounting an UFS file f kern/125536 fs [ext2fs] ext 2 mounts cleanly but fails on commands li o kern/125149 fs [nfs] [panic] changing into .zfs dir from nfs client c f kern/124621 fs [ext3] [patch] Cannot mount ext2fs partition o kern/122888 fs [zfs] zfs hang w/ prefetch on, zil off while running t o bin/122172 fs [fs]: amd(8) automount daemon dies on 6.3-STABLE i386, o bin/121072 fs [smbfs] mount_smbfs(8) cannot normally convert the cha o bin/118249 fs mv(1): moving a directory changes its mtime o kern/116170 fs [panic] Kernel panic when mounting /tmp o kern/114955 fs [cd9660] [patch] [request] support for mask,dirmask,ui o kern/114847 fs [ntfs] [patch] [request] dirmask support for NTFS ala o kern/114676 fs [ufs] snapshot creation panics: snapacct_ufs2: bad blo o bin/114468 fs [patch] [request] add -d option to umount(8) to detach o bin/113838 fs [patch] [request] mount(8): add support for relative p o bin/113049 fs [patch] [request] make quot(8) use getopt(3) and show o kern/112658 fs [smbfs] [patch] smbfs and caching problems (resolves b o kern/93942 fs [vfs] [patch] panic: ufs_dirbad: bad dir (patch from D 38 problems total. From owner-freebsd-fs@FreeBSD.ORG Mon Feb 2 12:29:29 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 625AF1065670 for ; Mon, 2 Feb 2009 12:29:29 +0000 (UTC) (envelope-from antik@bsd.ee) Received: from sorbesgroup.com (mail.sorbesgroup.com [217.159.241.118]) by mx1.freebsd.org (Postfix) with ESMTP id 1E83A8FC1F for ; Mon, 2 Feb 2009 12:29:29 +0000 (UTC) (envelope-from antik@bsd.ee) Received: from localhost (localhost.localdomain [127.0.0.1]) by sorbesgroup.com (Postfix) with ESMTP id 999D13C528F0 for ; Mon, 2 Feb 2009 13:58:25 +0200 (EET) Received: from sorbesgroup.com ([127.0.0.1]) by localhost (sorbesgroup.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 00773-09 for ; Mon, 2 Feb 2009 13:58:24 +0200 (EET) Received: from [192.168.0.80] (andrei [192.168.0.80]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by sorbesgroup.com (Postfix) with ESMTP id 489373C528EB for ; Mon, 2 Feb 2009 13:58:24 +0200 (EET) Message-ID: <4986E2F2.8070903@bsd.ee> Date: Mon, 02 Feb 2009 14:11:30 +0200 From: Andrei Kolu User-Agent: Thunderbird 2.0.0.19 (Windows/20081209) MIME-Version: 1.0 To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: amavisd-new at localhost Subject: zfs compression and nfs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Feb 2009 12:29:29 -0000 Hi, I encouontered strange problem with zfs compressed volume that is shared out over nfs. volume is created with command: # zpool create example /dev/da1 # zfs set compression=gzip data/configuration Now my "data" is shared with NFS and all servers have access to "configuration" volume. All NFS clients can write to volume and show written files over network. What is missing is files from server side- it does not show any file on compressed volume that is written by clients over NFS. If I copy same files/directories to nfs root eg. "data" then I can access files from server. Where are my files? # zpool status pool: data state: ONLINE scrub: scrub in progress, 19.58% done, 1h9m to go config: NAME STATE READ WRITE CKSUM data ONLINE 0 0 0 da1 ONLINE 0 0 0 errors: No known data errors From owner-freebsd-fs@FreeBSD.ORG Mon Feb 2 13:40:57 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 89F3F106566C for ; Mon, 2 Feb 2009 13:40:57 +0000 (UTC) (envelope-from antik@bsd.ee) Received: from sorbesgroup.com (mail.sorbesgroup.com [217.159.241.118]) by mx1.freebsd.org (Postfix) with ESMTP id 14EF88FC2F for ; Mon, 2 Feb 2009 13:40:56 +0000 (UTC) (envelope-from antik@bsd.ee) Received: from localhost (localhost.localdomain [127.0.0.1]) by sorbesgroup.com (Postfix) with ESMTP id 6E4323C528F1 for ; Mon, 2 Feb 2009 15:27:46 +0200 (EET) Received: from sorbesgroup.com ([127.0.0.1]) by localhost (sorbesgroup.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 03694-07 for ; Mon, 2 Feb 2009 15:27:45 +0200 (EET) Received: from [192.168.0.80] (andrei [192.168.0.80]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by sorbesgroup.com (Postfix) with ESMTP id 4BA2E3C528FF for ; Mon, 2 Feb 2009 15:27:45 +0200 (EET) Message-ID: <4986F7E3.6020404@bsd.ee> Date: Mon, 02 Feb 2009 15:40:51 +0200 From: Andrei Kolu User-Agent: Thunderbird 2.0.0.19 (Windows/20081209) MIME-Version: 1.0 CC: freebsd-fs@freebsd.org References: <4986E2F2.8070903@bsd.ee> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: amavisd-new at localhost Subject: Re: zfs compression and nfs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Feb 2009 13:40:57 -0000 Markus Gebert wrote: > Hi Andrei > > Am 02.02.2009 um 13:11 schrieb Andrei Kolu: > >> I encouontered strange problem with zfs compressed volume that is >> shared out over nfs. >> >> volume is created with command: >> >> # zpool create example /dev/da1 >> >> # zfs set compression=gzip data/configuration > > Since 'zfs set' is usually used on a file system (i.e. not a > directory), I assume 'data/configuration' is a zfs filesystem separate > from 'data/'. > Yes, it is created with command (I forgot to add it in my first post): # zfs create data/configuration # mount data on /data (zfs, NFS exported, local) data/configuration on /data/configuration (zfs, local) data/iscsi on /data/iscsi (zfs, local) > >> Now my "data" is shared with NFS and all servers have access to >> "configuration" volume. All NFS clients can write to volume and show >> written files over network. What is missing is files from server >> side- it does not show any file on compressed volume that is written >> by clients over NFS. If I copy same files/directories to nfs root eg. >> "data" then I can access files from server. Where are my files? > > > I don't think this is related to compression. > > If 'data/' and 'data/configuration' really happen to be different > filesystems and you're mounting only 'data/' on the client, the > behaviour you're seeing is expected. What's happening is that you're > client is able to to see the configuration _directory_ inside the > mounted 'data/' filesystem. But since the nfsclient won't be able to > cross filesystem boundaries on the server (nfs restriction), changing > to that directory and writing a file on the client will actually > result in the file being written to the 'data/' filesystem on the > server (inside it's 'configuration' _directory_). You are not seeing > these files on the server, because there 'data/configuration' is > actually you're compressed zfs filesysten that never got a write. You > should be able to make the lost files visible on the server by > umounting 'data/configuration': > > # zfs umount data/configuration > > Of course this does not solve your problem. I guess you need to export > 'data/configuration' too and mount it on the client. > But I can see "configuration" directory from NFS client!? If I understand correctly then NFS can't use "filesystem on filesystem" for example my case with "data/configuration"? Can I compress "data" then? All other subfilesystems will be compressed also? How can I see what compression ratio I got on compressed filesystem? So many questions... From owner-freebsd-fs@FreeBSD.ORG Mon Feb 2 13:45:54 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5DEBA10656D8 for ; Mon, 2 Feb 2009 13:45:54 +0000 (UTC) (envelope-from markus.gebert@hostpoint.ch) Received: from mail.adm.hostpoint.ch (mail.adm.hostpoint.ch [217.26.48.124]) by mx1.freebsd.org (Postfix) with ESMTP id 202148FC08 for ; Mon, 2 Feb 2009 13:45:54 +0000 (UTC) (envelope-from markus.gebert@hostpoint.ch) Received: from adsl-202-186-fixip.tiscali.ch ([212.254.202.186]:55316 helo=dynip72.office.hostpoint.internal) by mail.adm.hostpoint.ch with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.69 (FreeBSD)) (envelope-from ) id 1LTyue-000D82-Sh; Mon, 02 Feb 2009 14:32:33 +0100 Message-Id: From: Markus Gebert To: Andrei Kolu In-Reply-To: <4986E2F2.8070903@bsd.ee> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v930.3) Date: Mon, 2 Feb 2009 14:32:23 +0100 References: <4986E2F2.8070903@bsd.ee> X-Mailer: Apple Mail (2.930.3) Cc: freebsd-fs@freebsd.org Subject: Re: zfs compression and nfs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Feb 2009 13:45:55 -0000 Hi Andrei Am 02.02.2009 um 13:11 schrieb Andrei Kolu: > I encouontered strange problem with zfs compressed volume that is > shared out over nfs. > > volume is created with command: > > # zpool create example /dev/da1 > > # zfs set compression=gzip data/configuration Since 'zfs set' is usually used on a file system (i.e. not a directory), I assume 'data/configuration' is a zfs filesystem separate from 'data/'. > Now my "data" is shared with NFS and all servers have access to > "configuration" volume. All NFS clients can write to volume and show > written files over network. What is missing is files from server > side- it does not show any file on compressed volume that is written > by clients over NFS. If I copy same files/directories to nfs root > eg. "data" then I can access files from server. Where are my files? I don't think this is related to compression. If 'data/' and 'data/configuration' really happen to be different filesystems and you're mounting only 'data/' on the client, the behaviour you're seeing is expected. What's happening is that you're client is able to to see the configuration _directory_ inside the mounted 'data/' filesystem. But since the nfsclient won't be able to cross filesystem boundaries on the server (nfs restriction), changing to that directory and writing a file on the client will actually result in the file being written to the 'data/' filesystem on the server (inside it's 'configuration' _directory_). You are not seeing these files on the server, because there 'data/configuration' is actually you're compressed zfs filesysten that never got a write. You should be able to make the lost files visible on the server by umounting 'data/configuration': # zfs umount data/configuration Of course this does not solve your problem. I guess you need to export 'data/configuration' too and mount it on the client. Markus From owner-freebsd-fs@FreeBSD.ORG Mon Feb 2 14:31:09 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D641710656F2 for ; Mon, 2 Feb 2009 14:31:09 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [91.103.162.4]) by mx1.freebsd.org (Postfix) with ESMTP id 92B318FC2F for ; Mon, 2 Feb 2009 14:31:09 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from localhost (localhost.codelab.cz [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id 852AC19E023; Mon, 2 Feb 2009 15:11:53 +0100 (CET) Received: from [192.168.1.2] (r5bb235.net.upc.cz [86.49.61.235]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 0AD4819E027; Mon, 2 Feb 2009 15:11:48 +0100 (CET) Message-ID: <4986FF23.40502@quip.cz> Date: Mon, 02 Feb 2009 15:11:47 +0100 From: Miroslav Lachman <000.fbsd@quip.cz> User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.12) Gecko/20050915 X-Accept-Language: cz, cs, en, en-us MIME-Version: 1.0 To: Andrei Kolu References: <4986E2F2.8070903@bsd.ee> <4986F7E3.6020404@bsd.ee> In-Reply-To: <4986F7E3.6020404@bsd.ee> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: zfs compression and nfs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Feb 2009 14:31:11 -0000 Andrei Kolu wrote: > Markus Gebert wrote: [...] >> Of course this does not solve your problem. I guess you need to export >> 'data/configuration' too and mount it on the client. >> > But I can see "configuration" directory from NFS client!? If I > understand correctly then NFS can't use "filesystem on filesystem" for > example my case with "data/configuration"? Can I compress "data" then? > All other subfilesystems will be compressed also? How can I see what > compression ratio I got on compressed filesystem? 'zfs get -r compressratio' will show you ratio for all ZFS filesystems. Miroslav Lachman From owner-freebsd-fs@FreeBSD.ORG Mon Feb 2 14:33:44 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 132B1106566B for ; Mon, 2 Feb 2009 14:33:44 +0000 (UTC) (envelope-from markus.gebert@hostpoint.ch) Received: from mail.adm.hostpoint.ch (mail.adm.hostpoint.ch [217.26.48.124]) by mx1.freebsd.org (Postfix) with ESMTP id C7D888FC24 for ; Mon, 2 Feb 2009 14:33:43 +0000 (UTC) (envelope-from markus.gebert@hostpoint.ch) Received: from adsl-202-186-fixip.tiscali.ch ([212.254.202.186]:56264 helo=dynip72.office.hostpoint.internal) by mail.adm.hostpoint.ch with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.69 (FreeBSD)) (envelope-from ) id 1LTzrq-000E80-9M; Mon, 02 Feb 2009 15:33:42 +0100 Message-Id: <71E28FE1-0A16-499B-B240-A7D9BAC2D8FF@hostpoint.ch> From: Markus Gebert To: Andrei Kolu In-Reply-To: <4986F7E3.6020404@bsd.ee> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v930.3) Date: Mon, 2 Feb 2009 15:33:41 +0100 References: <4986E2F2.8070903@bsd.ee> <4986F7E3.6020404@bsd.ee> X-Mailer: Apple Mail (2.930.3) Cc: freebsd-fs@freebsd.org Subject: Re: zfs compression and nfs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Feb 2009 14:33:44 -0000 Andrei Kolu wrote: >> >> >>> Now my "data" is shared with NFS and all servers have access to >>> "configuration" volume. All NFS clients can write to volume and >>> show written files over network. What is missing is files from >>> server side- it does not show any file on compressed volume that >>> is written by clients over NFS. If I copy same files/directories >>> to nfs root eg. "data" then I can access files from server. Where >>> are my files? >> >> >> I don't think this is related to compression. >> >> If 'data/' and 'data/configuration' really happen to be different >> filesystems and you're mounting only 'data/' on the client, the >> behaviour you're seeing is expected. What's happening is that >> you're client is able to to see the configuration _directory_ >> inside the mounted 'data/' filesystem. But since the nfsclient >> won't be able to cross filesystem boundaries on the server (nfs >> restriction), changing to that directory and writing a file on the >> client will actually result in the file being written to the >> 'data/' filesystem on the server (inside it's 'configuration' >> _directory_). You are not seeing these files on the server, because >> there 'data/configuration' is actually you're compressed zfs >> filesysten that never got a write. You should be able to make the >> lost files visible on the server by umounting 'data/configuration': >> >> # zfs umount data/configuration >> >> Of course this does not solve your problem. I guess you need to >> export 'data/configuration' too and mount it on the client. >> > But I can see "configuration" directory from NFS client!? Yes, you can, because that directory is part of the 'data/' filesystem and used (by zfs on the server) as a mount point for the 'data/ configuration' filesystem. > If I understand correctly then NFS can't use "filesystem on > filesystem" for example my case with "data/configuration"? Well, at least nfsv3 and lower don't have this ability for sure. I once heard that nfsv4 might do it, but I at least for me, that didn't work on FreeBSD (tested with 7.0 which has only quite basic nfsv4 support AFAIK). But you can mount all your zfs filesystems on the client, i.e.: # mkdir /mnt/data # mount_nfs -3 server:/data /mnt/data # mount_nfs -3 server:/data/configuration /mnt/data/configuration > Can I compress "data" then? You could, since data is just another zfs filesystem. But if you mount like stated above, you should already have achieved your goal. > All other subfilesystems will be compressed also? 'compression' is a zfs property. If you set a property on the top- level zfs of a pool, then usually it will be inherited by all filesystems within the pool. But you can override properties for subfilesystems. > How can I see what compression ratio I got on compressed filesystem? # zfs get compressratio data/configuration btw: # man zfs Markus From owner-freebsd-fs@FreeBSD.ORG Mon Feb 2 15:06:28 2009 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A61EF1065673 for ; Mon, 2 Feb 2009 15:06:28 +0000 (UTC) (envelope-from vova@sw.ru) Received: from relay.sw.ru (mailhub.sw.ru [195.214.232.25]) by mx1.freebsd.org (Postfix) with ESMTP id A69E08FC18 for ; Mon, 2 Feb 2009 15:06:27 +0000 (UTC) (envelope-from vova@sw.ru) Received: from vbook.fbsd.ru ([10.30.1.111]) (authenticated bits=0) by relay.sw.ru (8.13.4/8.13.4) with ESMTP id n12EikYh010447 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 2 Feb 2009 17:44:47 +0300 (MSK) Received: from vova by vbook.fbsd.ru with local (Exim 4.69 (FreeBSD)) (envelope-from ) id 1LU02Y-00046s-64; Mon, 02 Feb 2009 17:44:46 +0300 From: Vladimir Grebenschikov To: amistry@am-productions.biz Content-Type: text/plain Content-Transfer-Encoding: 7bit Organization: SWsoft Date: Mon, 02 Feb 2009 17:44:45 +0300 Message-Id: <1233585885.56108.1.camel@localhost> Mime-Version: 1.0 X-Mailer: Evolution 2.24.3 FreeBSD GNOME Team Port Sender: Vladimir Grebenschikov Cc: fs@freebsd.org Subject: failed to build sysutils/fusefs-kmod on recent 8-CURRENT X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: vova@fbsd.ru List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Feb 2009 15:06:29 -0000 ===> Found saved configuration for fusefs-kmod-0.3.0_5 ===> Extracting for fusefs-kmod-0.3.9.p1.20080208_5 => MD5 Checksum OK for fuse4bsd/498acaef33b0.tar.gz. => SHA256 Checksum OK for fuse4bsd/498acaef33b0.tar.gz. ===> Patching for fusefs-kmod-0.3.9.p1.20080208_5 ===> Applying FreeBSD patches for fusefs-kmod-0.3.9.p1.20080208_5 ===> fusefs-kmod-0.3.9.p1.20080208_5 depends on package: fusefs-libs>2.4.1 - found ===> fusefs-kmod-0.3.9.p1.20080208_5 depends on executable: deplate - found ===> Configuring for fusefs-kmod-0.3.9.p1.20080208_5 ===> Building for fusefs-kmod-0.3.9.p1.20080208_5 ===> fuse_module (all) Warning: Object directory not changed from original /usr/ports/sysutils/fusefs-kmod/work/fuse4bsd-498acaef33b0/fuse_module @ -> /usr/src/sys machine -> /usr/src/sys/i386/include awk -f @/tools/vnode_if.awk @/kern/vnode_if.src -p awk -f @/tools/vnode_if.awk @/kern/vnode_if.src -q awk -f @/tools/vnode_if.awk @/kern/vnode_if.src -h cc -O2 -pipe -fno-strict-aliasing -Werror -D_KERNEL -DKLD_MODULE -nostdinc -I../include -I. -I@ -I@/contrib/altq -finline-limit=8000 --param inline-unit-growth=100 --param large-function-growth=1000 -fno-common -mno-align-long-strings -mpreferred-stack-boundary=2 -mno-mmx -mno-3dnow -mno-sse -mno-sse2 -mno-sse3 -ffreestanding -fstack-protector -std=iso9899:1999 -fstack-protector -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -fformat-extensions -c fuse_main.c cc -O2 -pipe -fno-strict-aliasing -Werror -D_KERNEL -DKLD_MODULE -nostdinc -I../include -I. -I@ -I@/contrib/altq -finline-limit=8000 --param inline-unit-growth=100 --param large-function-growth=1000 -fno-common -mno-align-long-strings -mpreferred-stack-boundary=2 -mno-mmx -mno-3dnow -mno-sse -mno-sse2 -mno-sse3 -ffreestanding -fstack-protector -std=iso9899:1999 -fstack-protector -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -fformat-extensions -c fuse_msg.c cc -O2 -pipe -fno-strict-aliasing -Werror -D_KERNEL -DKLD_MODULE -nostdinc -I../include -I. -I@ -I@/contrib/altq -finline-limit=8000 --param inline-unit-growth=100 --param large-function-growth=1000 -fno-common -mno-align-long-strings -mpreferred-stack-boundary=2 -mno-mmx -mno-3dnow -mno-sse -mno-sse2 -mno-sse3 -ffreestanding -fstack-protector -std=iso9899:1999 -fstack-protector -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -fformat-extensions -c fuse_dev.c cc1: warnings being treated as errors fuse_dev.c: In function 'fusedev_clone': fuse_dev.c:556: warning: implicit declaration of function 'unit2minor' fuse_dev.c:556: warning: nested extern declaration of 'unit2minor' -- Vladimir B. Grebenschikov vova@fbsd.ru From owner-freebsd-fs@FreeBSD.ORG Mon Feb 2 15:39:37 2009 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D62E01065674 for ; Mon, 2 Feb 2009 15:39:37 +0000 (UTC) (envelope-from aryeh.friedman@gmail.com) Received: from mta1.srv.hcvlny.cv.net (mta1.srv.hcvlny.cv.net [167.206.4.196]) by mx1.freebsd.org (Postfix) with ESMTP id AAF948FC1A for ; Mon, 2 Feb 2009 15:39:37 +0000 (UTC) (envelope-from aryeh.friedman@gmail.com) Received: from flosoft.no-ip.biz (ool-435559b8.dyn.optonline.net [67.85.89.184]) by mta1.srv.hcvlny.cv.net (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTP id <0KEG008M123RXNI0@mta1.srv.hcvlny.cv.net> for fs@freebsd.org; Mon, 02 Feb 2009 10:09:30 -0500 (EST) Received: from flosoft.no-ip.biz (localhost [IPv6:::1]) by flosoft.no-ip.biz (8.14.3/8.14.3) with ESMTP id n12F9L73008798; Mon, 02 Feb 2009 10:09:24 -0500 Date: Mon, 02 Feb 2009 10:09:21 -0500 From: "Aryeh M. Friedman" In-reply-to: <1233585885.56108.1.camel@localhost> To: vova@fbsd.ru Message-id: <49870CA1.9010804@gmail.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT References: <1233585885.56108.1.camel@localhost> User-Agent: Thunderbird 2.0.0.19 (X11/20090201) Cc: amistry@am-productions.biz, fs@freebsd.org Subject: Re: failed to build sysutils/fusefs-kmod on recent 8-CURRENT X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Feb 2009 15:39:38 -0000 Vladimir Grebenschikov wrote: > ===> Found saved configuration for fusefs-kmod-0.3.0_5 > ===> Extracting for fusefs-kmod-0.3.9.p1.20080208_5 > => MD5 Checksum OK for fuse4bsd/498acaef33b0.tar.gz. > => SHA256 Checksum OK for fuse4bsd/498acaef33b0.tar.gz. > ===> Patching for fusefs-kmod-0.3.9.p1.20080208_5 > ===> Applying FreeBSD patches for fusefs-kmod-0.3.9.p1.20080208_5 > ===> fusefs-kmod-0.3.9.p1.20080208_5 depends on package: fusefs-libs>2.4.1 - found > ===> fusefs-kmod-0.3.9.p1.20080208_5 depends on executable: deplate - found > ===> Configuring for fusefs-kmod-0.3.9.p1.20080208_5 > ===> Building for fusefs-kmod-0.3.9.p1.20080208_5 > ===> fuse_module (all) > Warning: Object directory not changed from original /usr/ports/sysutils/fusefs-kmod/work/fuse4bsd-498acaef33b0/fuse_module > @ -> /usr/src/sys > machine -> /usr/src/sys/i386/include > awk -f @/tools/vnode_if.awk @/kern/vnode_if.src -p > awk -f @/tools/vnode_if.awk @/kern/vnode_if.src -q > awk -f @/tools/vnode_if.awk @/kern/vnode_if.src -h > cc -O2 -pipe -fno-strict-aliasing -Werror -D_KERNEL -DKLD_MODULE -nostdinc -I../include -I. -I@ -I@/contrib/altq -finline-limit=8000 --param inline-unit-growth=100 --param large-function-growth=1000 -fno-common -mno-align-long-strings -mpreferred-stack-boundary=2 -mno-mmx -mno-3dnow -mno-sse -mno-sse2 -mno-sse3 -ffreestanding -fstack-protector -std=iso9899:1999 -fstack-protector -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -fformat-extensions -c fuse_main.c > cc -O2 -pipe -fno-strict-aliasing -Werror -D_KERNEL -DKLD_MODULE -nostdinc -I../include -I. -I@ -I@/contrib/altq -finline-limit=8000 --param inline-unit-growth=100 --param large-function-growth=1000 -fno-common -mno-align-long-strings -mpreferred-stack-boundary=2 -mno-mmx -mno-3dnow -mno-sse -mno-sse2 -mno-sse3 -ffreestanding -fstack-protector -std=iso9899:1999 -fstack-protector -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -fformat-extensions -c fuse_msg.c > cc -O2 -pipe -fno-strict-aliasing -Werror -D_KERNEL -DKLD_MODULE -nostdinc -I../include -I. -I@ -I@/contrib/altq -finline-limit=8000 --param inline-unit-growth=100 --param large-function-growth=1000 -fno-common -mno-align-long-strings -mpreferred-stack-boundary=2 -mno-mmx -mno-3dnow -mno-sse -mno-sse2 -mno-sse3 -ffreestanding -fstack-protector -std=iso9899:1999 -fstack-protector -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -fformat-extensions -c fuse_dev.c > cc1: warnings being treated as errors > fuse_dev.c: In function 'fusedev_clone': > fuse_dev.c:556: warning: implicit declaration of function 'unit2minor' > fuse_dev.c:556: warning: nested extern declaration of 'unit2minor' > > Even though it is a total hack I had no issue by just remove the unit2minor call and using the raw param From owner-freebsd-fs@FreeBSD.ORG Mon Feb 2 15:41:14 2009 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 71B0410656D6 for ; Mon, 2 Feb 2009 15:41:14 +0000 (UTC) (envelope-from aryeh.friedman@gmail.com) Received: from mta1.srv.hcvlny.cv.net (mta1.srv.hcvlny.cv.net [167.206.4.196]) by mx1.freebsd.org (Postfix) with ESMTP id 475048FC1E for ; Mon, 2 Feb 2009 15:41:14 +0000 (UTC) (envelope-from aryeh.friedman@gmail.com) Received: from flosoft.no-ip.biz (ool-435559b8.dyn.optonline.net [67.85.89.184]) by mta1.srv.hcvlny.cv.net (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTP id <0KEG000VC26O29N0@mta1.srv.hcvlny.cv.net> for fs@freebsd.org; Mon, 02 Feb 2009 10:11:14 -0500 (EST) Received: from flosoft.no-ip.biz (localhost [IPv6:::1]) by flosoft.no-ip.biz (8.14.3/8.14.3) with ESMTP id n12FBAsC021382; Mon, 02 Feb 2009 10:11:11 -0500 Date: Mon, 02 Feb 2009 10:11:10 -0500 From: "Aryeh M. Friedman" In-reply-to: <49870CA1.9010804@gmail.com> To: vova@fbsd.ru Message-id: <49870D0E.3010004@gmail.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT References: <1233585885.56108.1.camel@localhost> <49870CA1.9010804@gmail.com> User-Agent: Thunderbird 2.0.0.19 (X11/20090201) Cc: amistry@am-productions.biz, fs@freebsd.org Subject: Re: failed to build sysutils/fusefs-kmod on recent 8-CURRENT X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Feb 2009 15:41:15 -0000 Aryeh M. Friedman wrote: > Vladimir Grebenschikov wrote: >> ===> Found saved configuration for fusefs-kmod-0.3.0_5 >> ===> Extracting for fusefs-kmod-0.3.9.p1.20080208_5 >> => MD5 Checksum OK for fuse4bsd/498acaef33b0.tar.gz. >> => SHA256 Checksum OK for fuse4bsd/498acaef33b0.tar.gz. >> ===> Patching for fusefs-kmod-0.3.9.p1.20080208_5 >> ===> Applying FreeBSD patches for fusefs-kmod-0.3.9.p1.20080208_5 >> ===> fusefs-kmod-0.3.9.p1.20080208_5 depends on package: >> fusefs-libs>2.4.1 - found >> ===> fusefs-kmod-0.3.9.p1.20080208_5 depends on executable: deplate >> - found >> ===> Configuring for fusefs-kmod-0.3.9.p1.20080208_5 >> ===> Building for fusefs-kmod-0.3.9.p1.20080208_5 >> ===> fuse_module (all) >> Warning: Object directory not changed from original >> /usr/ports/sysutils/fusefs-kmod/work/fuse4bsd-498acaef33b0/fuse_module >> @ -> /usr/src/sys >> machine -> /usr/src/sys/i386/include >> awk -f @/tools/vnode_if.awk @/kern/vnode_if.src -p >> awk -f @/tools/vnode_if.awk @/kern/vnode_if.src -q >> awk -f @/tools/vnode_if.awk @/kern/vnode_if.src -h >> cc -O2 -pipe -fno-strict-aliasing -Werror -D_KERNEL -DKLD_MODULE >> -nostdinc -I../include -I. -I@ -I@/contrib/altq -finline-limit=8000 >> --param inline-unit-growth=100 --param large-function-growth=1000 >> -fno-common -mno-align-long-strings -mpreferred-stack-boundary=2 >> -mno-mmx -mno-3dnow -mno-sse -mno-sse2 -mno-sse3 -ffreestanding >> -fstack-protector -std=iso9899:1999 -fstack-protector -Wall >> -Wredundant-decls -Wnested-externs -Wstrict-prototypes >> -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef >> -Wno-pointer-sign -fformat-extensions -c fuse_main.c >> cc -O2 -pipe -fno-strict-aliasing -Werror -D_KERNEL -DKLD_MODULE >> -nostdinc -I../include -I. -I@ -I@/contrib/altq -finline-limit=8000 >> --param inline-unit-growth=100 --param large-function-growth=1000 >> -fno-common -mno-align-long-strings -mpreferred-stack-boundary=2 >> -mno-mmx -mno-3dnow -mno-sse -mno-sse2 -mno-sse3 -ffreestanding >> -fstack-protector -std=iso9899:1999 -fstack-protector -Wall >> -Wredundant-decls -Wnested-externs -Wstrict-prototypes >> -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef >> -Wno-pointer-sign -fformat-extensions -c fuse_msg.c >> cc -O2 -pipe -fno-strict-aliasing -Werror -D_KERNEL -DKLD_MODULE >> -nostdinc -I../include -I. -I@ -I@/contrib/altq -finline-limit=8000 >> --param inline-unit-growth=100 --param large-function-growth=1000 >> -fno-common -mno-align-long-strings -mpreferred-stack-boundary=2 >> -mno-mmx -mno-3dnow -mno-sse -mno-sse2 -mno-sse3 -ffreestanding >> -fstack-protector -std=iso9899:1999 -fstack-protector -Wall >> -Wredundant-decls -Wnested-externs -Wstrict-prototypes >> -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef >> -Wno-pointer-sign -fformat-extensions -c fuse_dev.c >> cc1: warnings being treated as errors >> fuse_dev.c: In function 'fusedev_clone': >> fuse_dev.c:556: warning: implicit declaration of function 'unit2minor' >> fuse_dev.c:556: warning: nested extern declaration of 'unit2minor' >> >> > Even though it is a total hack I had no issue by just remove the > unit2minor call and using the raw param > Forgot yo mention (even though off topic) the same hack *DOES NOT* work for x11/nvidia-driver if that is an issue for you From owner-freebsd-fs@FreeBSD.ORG Mon Feb 2 20:46:41 2009 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E10331065705; Mon, 2 Feb 2009 20:46:41 +0000 (UTC) (envelope-from gavin@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id B61E28FC1E; Mon, 2 Feb 2009 20:46:41 +0000 (UTC) (envelope-from gavin@FreeBSD.org) Received: from freefall.freebsd.org (gavin@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id n12Kkf0s038053; Mon, 2 Feb 2009 20:46:41 GMT (envelope-from gavin@freefall.freebsd.org) Received: (from gavin@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id n12KkfiG038049; Mon, 2 Feb 2009 20:46:41 GMT (envelope-from gavin) Date: Mon, 2 Feb 2009 20:46:41 GMT Message-Id: <200902022046.n12KkfiG038049@freefall.freebsd.org> To: gavin@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-fs@FreeBSD.org From: gavin@FreeBSD.org Cc: Subject: Re: kern/68978: [panic] [ufs] crashes with failing hard disk, loose pointers in kernel? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Feb 2009 20:46:42 -0000 Old Synopsis: [firewire] firewire crashes with failing hard disk, loose pointers in kernel? New Synopsis: [panic] [ufs] crashes with failing hard disk, loose pointers in kernel? Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: gavin Responsible-Changed-When: Mon Feb 2 20:41:44 UTC 2009 Responsible-Changed-Why: Over to maintainer(s). To be honest, I'm not sure this will be a priority - the UFS code is known to expect that the underlying media is 100% working. I'll keep this PR open though, as some progress has been made recently on fixing up these assumptions. http://www.freebsd.org/cgi/query-pr.cgi?pr=68978 From owner-freebsd-fs@FreeBSD.ORG Tue Feb 3 01:33:11 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4C2131065674 for ; Tue, 3 Feb 2009 01:33:11 +0000 (UTC) (envelope-from jmrueda@diatel.upm.es) Received: from aurora.diatel.upm.es (aurora.diatel.upm.es [138.100.49.70]) by mx1.freebsd.org (Postfix) with ESMTP id E6F928FC17 for ; Tue, 3 Feb 2009 01:33:10 +0000 (UTC) (envelope-from jmrueda@diatel.upm.es) Received: from [172.29.9.10] ([172.29.9.10]) by aurora.diatel.upm.es (8.14.3/8.14.3) with ESMTP id n131MoiB002975; Tue, 3 Feb 2009 02:22:50 +0100 (CET) (envelope-from jmrueda@diatel.upm.es) Message-ID: <49879C62.6070509@diatel.upm.es> Date: Tue, 03 Feb 2009 02:22:42 +0100 From: =?ISO-8859-1?Q?Javier_Mart=EDn_Rueda?= User-Agent: Thunderbird 2.0.0.19 (Windows/20081209) MIME-Version: 1.0 To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Raidz2 pool with single disk failure is faulted X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Feb 2009 01:33:11 -0000 On a FreeBSD 7.1-PRERELEASE amd64 system I had a raidz2 pool made up of 8 disks. Due to some things I tried in the past, the pool was currently like this: z1 ONLINE raidz2 ONLINE mirror/gm0 ONLINE mirror/gm1 ONLINE da2 ONLINE da3 ONLINE da4 ONLINE da5 ONLINE da6 ONLINE da7 ONLINE da2 to da7 where originally mirror/gm2 to mirror/gm7, but I replaced them little by little, eliminating the corresponding gmirrors at the same time. I don't think this is relevant for what I'm goint to explain, but I mention it just in case... One day, after a system reboot, one of the disks (da4) was dead and FreeBSD renamed all of the other disks that used to be after it (da5 became da4, da6 became da5, and da7 became da6). The pool was unavailable (da4 to da6 marked as corrupt and da7 as unavailable) because I suppose ZFS couldn't match the contents in the last 3 disks to their new names. I was able to fix this by inserting a blank new disk, rebooting, now the disk names were correct again, and the pool showed up as degraded because da4 was unavailable, but usable. I resilvered the pool and everything was back to normal. Yesterday, another disk died after a system reboot and the pool was unavailable again because of the automatic renaming of the SCSI disks. However, this time I didn't substitute it by a blank disk, but for another identical disk which I had been using in the past in a different ZFS pool on a different computer, but with the same name (z1) and same characteristics (raidz2, 8 disks). The disk hadn't been erased and its pool hadn't been destroyed, so it still had whatever ZFS stored in it. After rebooting, it seems ZFS got confused or something when it found out about two different active pools with the same name, etc. and it faulted the pool. I stopped ZFS, wiped the beginning and end of the disk with zeroes, but the problem persisted. Finally, I tried to export and import the pool, as I read somewhere that may help, but zpool import complains about an I/O error (which I imagine is ficticious, because all of the disks are find, I can read from them with dd no problem). The current situation is this: # zpool import pool: z1 id: 8828203687312199578 state: FAULTED status: One or more devices contains corrupted data. action: The pool cannot be imported due to damaged devices or data. The pool may be active on on another system, but can be imported using the '-f' flag. see: http://www.sun.com/msg/ZFS-8000-5E config: z1 FAULTED corrupted data raidz2 ONLINE mirror/gm0 ONLINE mirror/gm1 ONLINE da2 ONLINE da3 ONLINE da4 UNAVAIL corrupted data da5 ONLINE da6 ONLINE da7 ONLINE # zpool import -f z1 cannot import 'z1': I/O error By the way, before exporting the pool, the CKSUM column in "zpool status" showed 6 errors. However, zpool status -v didn't give any additional information. How come the pool is faulted if it is raidz2 and 7 out of 8 disks are reported as fine? Any idea how to recover the pool? The data has to be in there, as I haven't done any other destructive operation, as far as I can think of, and I imagine it should be some stupid little detail. I have dumped all of the labels in the 8 disks with zdb -l, and I don't see anything peculiar. They are fine in the 7 online disks, and it doesn't exist in the da4 disk. Is there some kind of diagnostic tools similar to dumpfs, but for zfs? I can provide additional information if needed. From owner-freebsd-fs@FreeBSD.ORG Tue Feb 3 04:03:04 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EC32D1065673 for ; Tue, 3 Feb 2009 04:03:04 +0000 (UTC) (envelope-from morganw@chemikals.org) Received: from warped.bluecherry.net (unknown [IPv6:2001:440:eeee:fffb::2]) by mx1.freebsd.org (Postfix) with ESMTP id 2EC038FC08 for ; Tue, 3 Feb 2009 04:03:04 +0000 (UTC) (envelope-from morganw@chemikals.org) Received: from volatile.chemikals.org (morganw-1-pt.tunnel.tserv8.dal1.ipv6.he.net [IPv6:2001:470:1f0e:47e::2]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by warped.bluecherry.net (Postfix) with ESMTPSA id 8764FA1FD20A; Mon, 2 Feb 2009 22:03:00 -0600 (CST) Received: from localhost (morganw@localhost [127.0.0.1]) by volatile.chemikals.org (8.14.3/8.14.3) with ESMTP id n1342rCb013620; Mon, 2 Feb 2009 22:02:54 -0600 (CST) (envelope-from morganw@chemikals.org) Date: Mon, 2 Feb 2009 22:02:53 -0600 (CST) From: Wes Morgan To: =?ISO-8859-15?Q?Javier_Mart=EDn_Rueda?= In-Reply-To: <49879C62.6070509@diatel.upm.es> Message-ID: References: <49879C62.6070509@diatel.upm.es> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: MULTIPART/MIXED; BOUNDARY="3265970123-156712375-1233633774=:10729" Cc: freebsd-fs@freebsd.org Subject: Re: Raidz2 pool with single disk failure is faulted X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Feb 2009 04:03:05 -0000 This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. --3265970123-156712375-1233633774=:10729 Content-Type: TEXT/PLAIN; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8BIT On Tue, 3 Feb 2009, Javier Martín Rueda wrote: > On a FreeBSD 7.1-PRERELEASE amd64 system I had a raidz2 pool made up of 8 > disks. Due to some things I tried in the past, the pool was currently like > this: > > z1 ONLINE > raidz2 ONLINE > mirror/gm0 ONLINE > mirror/gm1 ONLINE > da2 ONLINE > da3 ONLINE > da4 ONLINE > da5 ONLINE > da6 ONLINE > da7 ONLINE > > da2 to da7 where originally mirror/gm2 to mirror/gm7, but I replaced them > little by little, eliminating the corresponding gmirrors at the same time. I > don't think this is relevant for what I'm goint to explain, but I mention it > just in case... > > One day, after a system reboot, one of the disks (da4) was dead and FreeBSD > renamed all of the other disks that used to be after it (da5 became da4, da6 > became da5, and da7 became da6). The pool was unavailable (da4 to da6 marked > as corrupt and da7 as unavailable) because I suppose ZFS couldn't match the > contents in the last 3 disks to their new names. I was able to fix this by > inserting a blank new disk, rebooting, now the disk names were correct again, > and the pool showed up as degraded because da4 was unavailable, but usable. I > resilvered the pool and everything was back to normal. > > Yesterday, another disk died after a system reboot and the pool was > unavailable again because of the automatic renaming of the SCSI disks. > However, this time I didn't substitute it by a blank disk, but for another > identical disk which I had been using in the past in a different ZFS pool on > a different computer, but with the same name (z1) and same characteristics > (raidz2, 8 disks). The disk hadn't been erased and its pool hadn't been > destroyed, so it still had whatever ZFS stored in it. > > After rebooting, it seems ZFS got confused or something when it found out > about two different active pools with the same name, etc. and it faulted the > pool. I stopped ZFS, wiped the beginning and end of the disk with zeroes, but > the problem persisted. Finally, I tried to export and import the pool, as I > read somewhere that may help, but zpool import complains about an I/O error > (which I imagine is ficticious, because all of the disks are find, I can read > from them with dd no problem). > > The current situation is this: > > # zpool import > pool: z1 > id: 8828203687312199578 > state: FAULTED > status: One or more devices contains corrupted data. > action: The pool cannot be imported due to damaged devices or data. > The pool may be active on on another system, but can be imported using > the '-f' flag. > see: http://www.sun.com/msg/ZFS-8000-5E > config: > > z1 FAULTED corrupted data > raidz2 ONLINE > mirror/gm0 ONLINE > mirror/gm1 ONLINE > da2 ONLINE > da3 ONLINE > da4 UNAVAIL corrupted data > da5 ONLINE > da6 ONLINE > da7 ONLINE > # zpool import -f z1 > cannot import 'z1': I/O error > > By the way, before exporting the pool, the CKSUM column in "zpool status" > showed 6 errors. However, zpool status -v didn't give any additional > information. > > How come the pool is faulted if it is raidz2 and 7 out of 8 disks are > reported as fine? Any idea how to recover the pool? The data has to be in > there, as I haven't done any other destructive operation, as far as I can > think of, and I imagine it should be some stupid little detail. > > I have dumped all of the labels in the 8 disks with zdb -l, and I don't see > anything peculiar. They are fine in the 7 online disks, and it doesn't exist > in the da4 disk. > > Is there some kind of diagnostic tools similar to dumpfs, but for zfs? > > I can provide additional information if needed. I would try removing /boot/zfs/zpool.cache and re-importing, and if that doesn't work detach da4 device (camcontrol stop da4 or so) and see if it will import. Also make sure you wiped at least 512k from the front of the drive. --3265970123-156712375-1233633774=:10729-- From owner-freebsd-fs@FreeBSD.ORG Tue Feb 3 07:09:06 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 80AB11065674 for ; Tue, 3 Feb 2009 07:09:06 +0000 (UTC) (envelope-from jmrueda@diatel.upm.es) Received: from aurora.diatel.upm.es (aurora.diatel.upm.es [138.100.49.70]) by mx1.freebsd.org (Postfix) with ESMTP id EC86C8FC16 for ; Tue, 3 Feb 2009 07:09:05 +0000 (UTC) (envelope-from jmrueda@diatel.upm.es) Received: from [172.29.9.10] ([172.29.9.10]) by aurora.diatel.upm.es (8.14.3/8.14.3) with ESMTP id n1378vug004981; Tue, 3 Feb 2009 08:08:58 +0100 (CET) (envelope-from jmrueda@diatel.upm.es) Message-ID: <4987ED81.6080008@diatel.upm.es> Date: Tue, 03 Feb 2009 08:08:49 +0100 From: =?ISO-8859-1?Q?Javier_Mart=EDn_Rueda?= User-Agent: Thunderbird 2.0.0.19 (Windows/20081209) MIME-Version: 1.0 To: Wes Morgan References: <49879C62.6070509@diatel.upm.es> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit Cc: freebsd-fs@freebsd.org Subject: Re: Raidz2 pool with single disk failure is faulted X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Feb 2009 07:09:06 -0000 Wes Morgan escribió: > On Tue, 3 Feb 2009, Javier Martín Rueda wrote: > >> On a FreeBSD 7.1-PRERELEASE amd64 system I had a raidz2 pool made up >> of 8 disks. Due to some things I tried in the past, the pool was >> currently like this: >> >> z1 ONLINE >> raidz2 ONLINE >> mirror/gm0 ONLINE >> mirror/gm1 ONLINE >> da2 ONLINE >> da3 ONLINE >> da4 ONLINE >> da5 ONLINE >> da6 ONLINE >> da7 ONLINE >> >> da2 to da7 where originally mirror/gm2 to mirror/gm7, but I replaced >> them little by little, eliminating the corresponding gmirrors at the >> same time. I don't think this is relevant for what I'm goint to >> explain, but I mention it just in case... >> >> One day, after a system reboot, one of the disks (da4) was dead and >> FreeBSD renamed all of the other disks that used to be after it (da5 >> became da4, da6 became da5, and da7 became da6). The pool was >> unavailable (da4 to da6 marked as corrupt and da7 as unavailable) >> because I suppose ZFS couldn't match the contents in the last 3 disks >> to their new names. I was able to fix this by inserting a blank new >> disk, rebooting, now the disk names were correct again, and the pool >> showed up as degraded because da4 was unavailable, but usable. I >> resilvered the pool and everything was back to normal. >> >> Yesterday, another disk died after a system reboot and the pool was >> unavailable again because of the automatic renaming of the SCSI >> disks. However, this time I didn't substitute it by a blank disk, but >> for another identical disk which I had been using in the past in a >> different ZFS pool on a different computer, but with the same name >> (z1) and same characteristics (raidz2, 8 disks). The disk hadn't been >> erased and its pool hadn't been destroyed, so it still had whatever >> ZFS stored in it. >> >> After rebooting, it seems ZFS got confused or something when it found >> out about two different active pools with the same name, etc. and it >> faulted the pool. I stopped ZFS, wiped the beginning and end of the >> disk with zeroes, but the problem persisted. Finally, I tried to >> export and import the pool, as I read somewhere that may help, but >> zpool import complains about an I/O error (which I imagine is >> ficticious, because all of the disks are find, I can read from them >> with dd no problem). >> >> The current situation is this: >> >> # zpool import >> pool: z1 >> id: 8828203687312199578 >> state: FAULTED >> status: One or more devices contains corrupted data. >> action: The pool cannot be imported due to damaged devices or data. >> The pool may be active on on another system, but can be >> imported using >> the '-f' flag. >> see: http://www.sun.com/msg/ZFS-8000-5E >> config: >> >> z1 FAULTED corrupted data >> raidz2 ONLINE >> mirror/gm0 ONLINE >> mirror/gm1 ONLINE >> da2 ONLINE >> da3 ONLINE >> da4 UNAVAIL corrupted data >> da5 ONLINE >> da6 ONLINE >> da7 ONLINE >> # zpool import -f z1 >> cannot import 'z1': I/O error >> >> By the way, before exporting the pool, the CKSUM column in "zpool >> status" showed 6 errors. However, zpool status -v didn't give any >> additional information. >> >> How come the pool is faulted if it is raidz2 and 7 out of 8 disks are >> reported as fine? Any idea how to recover the pool? The data has to >> be in there, as I haven't done any other destructive operation, as >> far as I can think of, and I imagine it should be some stupid little >> detail. >> >> I have dumped all of the labels in the 8 disks with zdb -l, and I >> don't see anything peculiar. They are fine in the 7 online disks, and >> it doesn't exist in the da4 disk. >> >> Is there some kind of diagnostic tools similar to dumpfs, but for zfs? >> >> I can provide additional information if needed. > > I would try removing /boot/zfs/zpool.cache and re-importing, and if > that doesn't work detach da4 device (camcontrol stop da4 or so) and > see if it will import. > > Also make sure you wiped at least 512k from the front of the drive. I tried all that, but nothing worked. I've tried to trace what's going on in the kernel when I try to import the pool. The problem seems to be in dsl_pool_open(). In the first part of the function, there is this call: err = zap_lookup(dp->dp_meta_objset, DMU_POOL_DIRECTORY_OBJECT, DMU_POOL_ROOT_DATASET, sizeof (uint64_t), 1, &dp->dp_root_dir_obj); if (err) goto out; zap_lookup() was returning EIO, but I don't think it is a real I/O problem, but a checksumming problem, because I also got these messages: zio 0xffffff000bcb5810 vdev raidz offset 6eb6d6d9000 stage 15 error 86 retry #1 for read to raidz offset 6eb6d6d9000 zio 0xffffff000bcb5810 vdev raidz offset 6eb6d6d9000 stage 15 error 86 zio 0xffffff000bcb5810 vdev raidz offset 6eb6d6d9000 stage 16 error 86 zio 0xffffff000bcb5810 vdev raidz offset 6eb6d6d9000 stage 17 error 86 zio 0xffffff000bcb5ac0 vdev offset 0 stage 14 error 86 zio 0xffffff005a5dbac0 vdev raidz offset c03bf45800 stage 15 error 86 retry #1 for read to raidz offset c03bf45800 zio 0xffffff005a5dbac0 vdev raidz offset c03bf45800 stage 15 error 86 zio 0xffffff005a5dbac0 vdev raidz offset c03bf45800 stage 16 error 86 zio 0xffffff005a5dbac0 vdev raidz offset c03bf45800 stage 17 error 86 zio 0xffffff000bcb5ac0 vdev offset 0 stage 14 error 86 zio 0xffffff005a534ac0 vdev raidz offset 5902760f800 stage 15 error 86 retry #1 for read to raidz offset 5902760f800 zio 0xffffff005a534ac0 vdev raidz offset 5902760f800 stage 15 error 86 zio 0xffffff005a534ac0 vdev raidz offset 5902760f800 stage 16 error 86 zio 0xffffff005a534ac0 vdev raidz offset 5902760f800 stage 17 error 86 zio 0xffffff000bcb5ac0 vdev offset 0 stage 14 error 86 zio 0xffffff000bcb5ac0 vdev offset 0 stage 15 error 86 retry #1 for read to offset 0 zio 0xffffff0003ebbac0 vdev raidz offset 6eb6d6d9000 stage 15 error 86 retry #1 for read to raidz offset 6eb6d6d9000 zio 0xffffff0003ebbac0 vdev raidz offset 6eb6d6d9000 stage 15 error 86 zio 0xffffff0003ebbac0 vdev raidz offset 6eb6d6d9000 stage 16 error 86 zio 0xffffff0003ebbac0 vdev raidz offset 6eb6d6d9000 stage 17 error 86 zio 0xffffff000bcb5ac0 vdev offset 0 stage 14 error 86 zio 0xffffff0003eba2b0 vdev raidz offset c03bf45800 stage 15 error 86 retry #1 for read to raidz offset c03bf45800 zio 0xffffff0003eba2b0 vdev raidz offset c03bf45800 stage 15 error 86 zio 0xffffff0003eba2b0 vdev raidz offset c03bf45800 stage 16 error 86 zio 0xffffff0003eba2b0 vdev raidz offset c03bf45800 stage 17 error 86 zio 0xffffff000bcb5ac0 vdev offset 0 stage 14 error 86 zio 0xffffff000bc93ac0 vdev raidz offset 5902760f800 stage 15 error 86 retry #1 for read to raidz offset 5902760f800 zio 0xffffff000bc93ac0 vdev raidz offset 5902760f800 stage 15 error 86 zio 0xffffff000bc93ac0 vdev raidz offset 5902760f800 stage 16 error 86 zio 0xffffff000bc93ac0 vdev raidz offset 5902760f800 stage 17 error 86 zio 0xffffff000bcb5ac0 vdev offset 0 stage 14 error 86 zio 0xffffff000bcb5ac0 vdev offset 0 stage 15 error 86 zio 0xffffff000bcb5ac0 vdev offset 0 stage 16 error 86 zio 0xffffff000bcb5ac0 vdev offset 0 stage 17 error 86 zio 0xffffff0003f44000 vdev offset 0 stage 17 error 5 Error 86 seems to be ECKSUM, so I decided to disable checksumming and see what happened. To disable checksumming I edited /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio_checksum.c and just put a return 0 at the beginning of zio_checksum_error(). I tried to import again, and still didn't work. Only this time, zap_lookup() was returning ENOENT, which I imagine it means that ZFS cannot locate the root of the pool or something like that. Any ideas? From owner-freebsd-fs@FreeBSD.ORG Tue Feb 3 12:40:51 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3CC24106566C for ; Tue, 3 Feb 2009 12:40:51 +0000 (UTC) (envelope-from jmrueda@diatel.upm.es) Received: from aurora.diatel.upm.es (aurora.diatel.upm.es [138.100.49.70]) by mx1.freebsd.org (Postfix) with ESMTP id BD3FE8FC1C for ; Tue, 3 Feb 2009 12:40:50 +0000 (UTC) (envelope-from jmrueda@diatel.upm.es) Received: from [172.29.9.10] ([172.29.9.10]) by aurora.diatel.upm.es (8.14.3/8.14.3) with ESMTP id n13CeiHJ005877; Tue, 3 Feb 2009 13:40:45 +0100 (CET) (envelope-from jmrueda@diatel.upm.es) Message-ID: <49883B45.3040606@diatel.upm.es> Date: Tue, 03 Feb 2009 13:40:37 +0100 From: =?ISO-8859-1?Q?Javier_Mart=EDn_Rueda?= User-Agent: Thunderbird 2.0.0.19 (Windows/20081209) MIME-Version: 1.0 To: Wes Morgan , freebsd-fs@freebsd.org References: <49879C62.6070509@diatel.upm.es> <4987ED81.6080008@diatel.upm.es> In-Reply-To: <4987ED81.6080008@diatel.upm.es> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Subject: Re: Raidz2 pool with single disk failure is faulted X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Feb 2009 12:40:51 -0000 I solved the problem. This is how I did it, in case one day it prevents somebody from jumping in front of a train :-) First of all, I got some insight from various sites, mailing list archives, documents, etc. Among them, maybe these two were more helpful: http://mail.opensolaris.org/pipermail/zfs-discuss/2008-October/051643.html http://opensolaris.org/os/community/zfs/docs/ondiskformat0822.pdf I suspected that maybe my uberblock was somehow corrupted, and thought it would be worthwhile to rollback to an earlier uberblock. However, my pool was raidz2 and the examples I had seen about how to do this were with simple pools, so I tried a different approach, which in the end proved very successful: First, I added a couple of printf to vdev_uberblock_load_done(), which is in /sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_label.c: --- vdev_label.c.orig 2009-02-03 13:14:35.000000000 +0100 +++ vdev_label.c 2009-02-03 13:14:52.000000000 +0100 @@ -659,10 +659,12 @@ if (zio->io_error == 0 && uberblock_verify(ub) == 0) { mutex_enter(&spa->spa_uberblock_lock); + printf("JMR: vdev_uberblock_load_done ub_txg=%qd ub_timestamp=%qd\n", ub->ub_txg, ub->ub_timestamp); if (vdev_uberblock_compare(ub, ubbest) > 0) *ubbest = *ub; mutex_exit(&spa->spa_uberblock_lock); } + printf("JMR: vdev_uberblock_load_done ubbest ub_txg=%qd ub_timestamp=%qd\n", ubbest->ub_txg, ubbest->ub_timestamp); zio_buf_free(zio->io_data, zio->io_size); } After compiling and loading the zfs.ko module, I executed "zpool import" and these messages came up: ... JMR: vdev_uberblock_load_done ub_txg=4254783 ub_timestamp=1233545538 JMR: vdev_uberblock_load_done ubbest ub_txg=4254783 ub_timestamp=1233545538 JMR: vdev_uberblock_load_done ub_txg=4254782 ub_timestamp=1233545533 JMR: vdev_uberblock_load_done ubbest ub_txg=4254783 ub_timestamp=1233545538 JMR: vdev_uberblock_load_done ub_txg=4254781 ub_timestamp=1233545528 JMR: vdev_uberblock_load_done ubbest ub_txg=4254783 ub_timestamp=1233545538 JMR: vdev_uberblock_load_done ub_txg=4254780 ub_timestamp=1233545523 JMR: vdev_uberblock_load_done ubbest ub_txg=4254783 ub_timestamp=1233545538 JMR: vdev_uberblock_load_done ub_txg=4254779 ub_timestamp=1233545518 JMR: vdev_uberblock_load_done ubbest ub_txg=4254783 ub_timestamp=1233545538 JMR: vdev_uberblock_load_done ub_txg=4254778 ub_timestamp=1233545513 ... JMR: vdev_uberblock_load_done ubbest ub_txg=4254783 ub_timestamp=1233545538 So, the uberblock with transaction group 4254783 was the most recent. I convinced ZFS to use an earlier one with this patch (note the second expression I added to the if statement): --- vdev_label.c.orig 2009-02-03 13:14:35.000000000 +0100 +++ vdev_label.c 2009-02-03 13:25:43.000000000 +0100 @@ -659,10 +659,12 @@ if (zio->io_error == 0 && uberblock_verify(ub) == 0) { mutex_enter(&spa->spa_uberblock_lock); - if (vdev_uberblock_compare(ub, ubbest) > 0) + printf("JMR: vdev_uberblock_load_done ub_txg=%qd ub_timestamp=%qd\n", ub->ub_txg, ub->ub_timestamp); + if (vdev_uberblock_compare(ub, ubbest) > 0 && ub->ub_txg < 4254783) *ubbest = *ub; mutex_exit(&spa->spa_uberblock_lock); } + printf("JMR: vdev_uberblock_load_done ubbest ub_txg=%qd ub_timestamp=%qd\n", ubbest->ub_txg, ubbest->ub_timestamp); zio_buf_free(zio->io_data, zio->io_size); } After compiling and loading the zfs.ko module, I executed "zpool import" and the pool was still faulted. So, I decremented the limit txg to "< 4254782" and this time the zpool came up as ONLINE. After crossing my fingers I executed "zpool import z1", and it worked ok. No data loss, everything back to normal. The only curious thing I've noticed is this: # zpool status pool: z1 state: ONLINE status: One or more devices could not be used because the label is missing or invalid. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Replace the device using 'zpool replace'. see: http://www.sun.com/msg/ZFS-8000-4J scrub: resilver completed with 0 errors on Tue Feb 3 09:26:40 2009 config: NAME STATE READ WRITE CKSUM z1 ONLINE 0 0 0 raidz2 ONLINE 0 0 0 mirror/gm0 ONLINE 0 0 0 mirror/gm1 ONLINE 0 0 0 da2 ONLINE 0 0 0 da3 ONLINE 0 0 0 8076139616933977534 UNAVAIL 0 0 0 was /dev/da4 da5 ONLINE 0 0 0 da6 ONLINE 0 0 0 da7 ONLINE 0 0 0 errors: No known data errors As you can see, the raidz2 vdev is marked as ONLINE, when I think it should be DEGRADED. Nevertheless, the pool is readable and writeable, and so far I haven't detected any problem. To be safe, I am extracting all the data and I will recreate the pool again from scratch, just in case. Pending questions: 1) Why did the "supposed corruption" happened in the first place? I advise people not to mix disks from different zpools with the same name in the same computer. That's what I did, and maybe it's what caused my problems. 2) Rolling back to an earlier uberblock seems to solve some faulted zpool problems. I think it would be interesting to have a program that let you do it in a user-friendly way (after warning you about the dangers, etc.). From owner-freebsd-fs@FreeBSD.ORG Tue Feb 3 13:36:39 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 142271065675 for ; Tue, 3 Feb 2009 13:36:39 +0000 (UTC) (envelope-from morganw@chemikals.org) Received: from cdptpa-omtalb.mail.rr.com (cdptpa-omtalb.mail.rr.com [75.180.132.121]) by mx1.freebsd.org (Postfix) with ESMTP id C6A908FC20 for ; Tue, 3 Feb 2009 13:36:38 +0000 (UTC) (envelope-from morganw@chemikals.org) Received: from shop.chemikals.org ([75.182.5.141]) by cdptpa-omta06.mail.rr.com with ESMTP id <20090203130753.DRRB22988.cdptpa-omta06.mail.rr.com@shop.chemikals.org>; Tue, 3 Feb 2009 13:07:53 +0000 Received: from localhost (morganw@localhost [127.0.0.1]) by shop.chemikals.org (8.14.3/8.14.3) with ESMTP id n13D7qEG076456; Tue, 3 Feb 2009 08:07:53 -0500 (EST) (envelope-from morganw@chemikals.org) Date: Tue, 3 Feb 2009 08:07:47 -0500 (EST) From: Wesley Morgan To: =?ISO-8859-15?Q?Javier_Mart=EDn_Rueda?= In-Reply-To: <49883B45.3040606@diatel.upm.es> Message-ID: References: <49879C62.6070509@diatel.upm.es> <4987ED81.6080008@diatel.upm.es> <49883B45.3040606@diatel.upm.es> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: MULTIPART/MIXED; BOUNDARY="2866877063-78565638-1233666473=:61897" Cc: freebsd-fs@freebsd.org Subject: Re: Raidz2 pool with single disk failure is faulted X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Feb 2009 13:36:39 -0000 This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. --2866877063-78565638-1233666473=:61897 Content-Type: TEXT/PLAIN; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8BIT On Tue, 3 Feb 2009, Javier Martín Rueda wrote: > I solved the problem. This is how I did it, in case one day it prevents > somebody from jumping in front of a train :-) > > First of all, I got some insight from various sites, mailing list archives, > documents, etc. Among them, maybe these two were more helpful: > > http://mail.opensolaris.org/pipermail/zfs-discuss/2008-October/051643.html > http://opensolaris.org/os/community/zfs/docs/ondiskformat0822.pdf > > I suspected that maybe my uberblock was somehow corrupted, and thought it > would be worthwhile to rollback to an earlier uberblock. However, my pool was > raidz2 and the examples I had seen about how to do this were with simple > pools, so I tried a different approach, which in the end proved very > successful: > > First, I added a couple of printf to vdev_uberblock_load_done(), which is in > /sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_label.c: > > --- vdev_label.c.orig 2009-02-03 13:14:35.000000000 +0100 > +++ vdev_label.c 2009-02-03 13:14:52.000000000 +0100 > @@ -659,10 +659,12 @@ > > if (zio->io_error == 0 && uberblock_verify(ub) == 0) { > mutex_enter(&spa->spa_uberblock_lock); > + printf("JMR: vdev_uberblock_load_done ub_txg=%qd > ub_timestamp=%qd\n", ub->ub_txg, ub->ub_timestamp); > if (vdev_uberblock_compare(ub, ubbest) > 0) > *ubbest = *ub; > mutex_exit(&spa->spa_uberblock_lock); > } > + printf("JMR: vdev_uberblock_load_done ubbest ub_txg=%qd > ub_timestamp=%qd\n", ubbest->ub_txg, ubbest->ub_timestamp); > > zio_buf_free(zio->io_data, zio->io_size); > } > > After compiling and loading the zfs.ko module, I executed "zpool import" and > these messages came up: > > ... > JMR: vdev_uberblock_load_done ub_txg=4254783 ub_timestamp=1233545538 > JMR: vdev_uberblock_load_done ubbest ub_txg=4254783 ub_timestamp=1233545538 > JMR: vdev_uberblock_load_done ub_txg=4254782 ub_timestamp=1233545533 > JMR: vdev_uberblock_load_done ubbest ub_txg=4254783 ub_timestamp=1233545538 > JMR: vdev_uberblock_load_done ub_txg=4254781 ub_timestamp=1233545528 > JMR: vdev_uberblock_load_done ubbest ub_txg=4254783 ub_timestamp=1233545538 > JMR: vdev_uberblock_load_done ub_txg=4254780 ub_timestamp=1233545523 > JMR: vdev_uberblock_load_done ubbest ub_txg=4254783 ub_timestamp=1233545538 > JMR: vdev_uberblock_load_done ub_txg=4254779 ub_timestamp=1233545518 > JMR: vdev_uberblock_load_done ubbest ub_txg=4254783 ub_timestamp=1233545538 > JMR: vdev_uberblock_load_done ub_txg=4254778 ub_timestamp=1233545513 > ... > JMR: vdev_uberblock_load_done ubbest ub_txg=4254783 ub_timestamp=1233545538 > > So, the uberblock with transaction group 4254783 was the most recent. I > convinced ZFS to use an earlier one with this patch (note the second > expression I added to the if statement): > > --- vdev_label.c.orig 2009-02-03 13:14:35.000000000 +0100 > +++ vdev_label.c 2009-02-03 13:25:43.000000000 +0100 > @@ -659,10 +659,12 @@ > > if (zio->io_error == 0 && uberblock_verify(ub) == 0) { > mutex_enter(&spa->spa_uberblock_lock); > - if (vdev_uberblock_compare(ub, ubbest) > 0) > + printf("JMR: vdev_uberblock_load_done ub_txg=%qd > ub_timestamp=%qd\n", ub->ub_txg, ub->ub_timestamp); > + if (vdev_uberblock_compare(ub, ubbest) > 0 && ub->ub_txg < > 4254783) > *ubbest = *ub; > mutex_exit(&spa->spa_uberblock_lock); > } > + printf("JMR: vdev_uberblock_load_done ubbest ub_txg=%qd > ub_timestamp=%qd\n", ubbest->ub_txg, ubbest->ub_timestamp); > > zio_buf_free(zio->io_data, zio->io_size); > } > > After compiling and loading the zfs.ko module, I executed "zpool import" and > the pool was still faulted. So, I decremented the limit txg to "< 4254782" > and this time the zpool came up as ONLINE. After crossing my fingers I > executed "zpool import z1", and it worked ok. No data loss, everything back > to normal. The only curious thing I've noticed is this: > > # zpool status > pool: z1 > state: ONLINE > status: One or more devices could not be used because the label is missing or > invalid. Sufficient replicas exist for the pool to continue > functioning in a degraded state. > action: Replace the device using 'zpool replace'. > see: http://www.sun.com/msg/ZFS-8000-4J > scrub: resilver completed with 0 errors on Tue Feb 3 09:26:40 2009 > config: > > NAME STATE READ WRITE CKSUM > z1 ONLINE 0 0 0 > raidz2 ONLINE 0 0 0 > mirror/gm0 ONLINE 0 0 0 > mirror/gm1 ONLINE 0 0 0 > da2 ONLINE 0 0 0 > da3 ONLINE 0 0 0 > 8076139616933977534 UNAVAIL 0 0 0 was /dev/da4 > da5 ONLINE 0 0 0 > da6 ONLINE 0 0 0 > da7 ONLINE 0 0 0 > > errors: No known data errors > > As you can see, the raidz2 vdev is marked as ONLINE, when I think it should > be DEGRADED. Nevertheless, the pool is readable and writeable, and so far I > haven't detected any problem. To be safe, I am extracting all the data and I > will recreate the pool again from scratch, just in case. > > > Pending questions: > > 1) Why did the "supposed corruption" happened in the first place? I advise > people not to mix disks from different zpools with the same name in the same > computer. That's what I did, and maybe it's what caused my problems. > > 2) Rolling back to an earlier uberblock seems to solve some faulted zpool > problems. I think it would be interesting to have a program that let you do > it in a user-friendly way (after warning you about the dangers, etc.). > It would be interesting to see if the txid from all of your labels was the same. I would highly advise scrubbing your array. I believe the reason that your "da4" is showing up with only a uuid is because zfs is now recognizing that the da4 it sees is not the correct one. Still very curious how you ended up in that situation. I wonder if you had corruption that was unknown before you removed da4. --2866877063-78565638-1233666473=:61897-- From owner-freebsd-fs@FreeBSD.ORG Tue Feb 3 13:41:27 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 948D9106567C for ; Tue, 3 Feb 2009 13:41:27 +0000 (UTC) (envelope-from jmrueda@diatel.upm.es) Received: from aurora.diatel.upm.es (aurora.diatel.upm.es [138.100.49.70]) by mx1.freebsd.org (Postfix) with ESMTP id 0F3B68FC18 for ; Tue, 3 Feb 2009 13:41:26 +0000 (UTC) (envelope-from jmrueda@diatel.upm.es) Received: from [172.29.9.10] ([172.29.9.10]) by aurora.diatel.upm.es (8.14.3/8.14.3) with ESMTP id n13DfLFC006032; Tue, 3 Feb 2009 14:41:21 +0100 (CET) (envelope-from jmrueda@diatel.upm.es) Message-ID: <49884979.7080103@diatel.upm.es> Date: Tue, 03 Feb 2009 14:41:13 +0100 From: =?ISO-8859-1?Q?Javier_Mart=EDn_Rueda?= User-Agent: Thunderbird 2.0.0.19 (Windows/20081209) MIME-Version: 1.0 To: Wesley Morgan References: <49879C62.6070509@diatel.upm.es> <4987ED81.6080008@diatel.upm.es> <49883B45.3040606@diatel.upm.es> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit Cc: freebsd-fs@freebsd.org Subject: Re: Raidz2 pool with single disk failure is faulted X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Feb 2009 13:41:28 -0000 Wesley Morgan escribió: > On Tue, 3 Feb 2009, Javier Martín Rueda wrote: > >> I solved the problem. This is how I did it, in case one day it >> prevents somebody from jumping in front of a train :-) >> >> First of all, I got some insight from various sites, mailing list >> archives, documents, etc. Among them, maybe these two were more helpful: >> >> http://mail.opensolaris.org/pipermail/zfs-discuss/2008-October/051643.html >> >> http://opensolaris.org/os/community/zfs/docs/ondiskformat0822.pdf >> >> I suspected that maybe my uberblock was somehow corrupted, and >> thought it would be worthwhile to rollback to an earlier uberblock. >> However, my pool was raidz2 and the examples I had seen about how to >> do this were with simple pools, so I tried a different approach, >> which in the end proved very successful: >> >> First, I added a couple of printf to vdev_uberblock_load_done(), >> which is in >> /sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_label.c: >> >> --- vdev_label.c.orig 2009-02-03 13:14:35.000000000 +0100 >> +++ vdev_label.c 2009-02-03 13:14:52.000000000 +0100 >> @@ -659,10 +659,12 @@ >> >> if (zio->io_error == 0 && uberblock_verify(ub) == 0) { >> mutex_enter(&spa->spa_uberblock_lock); >> + printf("JMR: vdev_uberblock_load_done ub_txg=%qd >> ub_timestamp=%qd\n", ub->ub_txg, ub->ub_timestamp); >> if (vdev_uberblock_compare(ub, ubbest) > 0) >> *ubbest = *ub; >> mutex_exit(&spa->spa_uberblock_lock); >> } >> + printf("JMR: vdev_uberblock_load_done ubbest ub_txg=%qd >> ub_timestamp=%qd\n", ubbest->ub_txg, ubbest->ub_timestamp); >> >> zio_buf_free(zio->io_data, zio->io_size); >> } >> >> After compiling and loading the zfs.ko module, I executed "zpool >> import" and these messages came up: >> >> ... >> JMR: vdev_uberblock_load_done ub_txg=4254783 ub_timestamp=1233545538 >> JMR: vdev_uberblock_load_done ubbest ub_txg=4254783 >> ub_timestamp=1233545538 >> JMR: vdev_uberblock_load_done ub_txg=4254782 ub_timestamp=1233545533 >> JMR: vdev_uberblock_load_done ubbest ub_txg=4254783 >> ub_timestamp=1233545538 >> JMR: vdev_uberblock_load_done ub_txg=4254781 ub_timestamp=1233545528 >> JMR: vdev_uberblock_load_done ubbest ub_txg=4254783 >> ub_timestamp=1233545538 >> JMR: vdev_uberblock_load_done ub_txg=4254780 ub_timestamp=1233545523 >> JMR: vdev_uberblock_load_done ubbest ub_txg=4254783 >> ub_timestamp=1233545538 >> JMR: vdev_uberblock_load_done ub_txg=4254779 ub_timestamp=1233545518 >> JMR: vdev_uberblock_load_done ubbest ub_txg=4254783 >> ub_timestamp=1233545538 >> JMR: vdev_uberblock_load_done ub_txg=4254778 ub_timestamp=1233545513 >> ... >> JMR: vdev_uberblock_load_done ubbest ub_txg=4254783 >> ub_timestamp=1233545538 >> >> So, the uberblock with transaction group 4254783 was the most recent. >> I convinced ZFS to use an earlier one with this patch (note the >> second expression I added to the if statement): >> >> --- vdev_label.c.orig 2009-02-03 13:14:35.000000000 +0100 >> +++ vdev_label.c 2009-02-03 13:25:43.000000000 +0100 >> @@ -659,10 +659,12 @@ >> >> if (zio->io_error == 0 && uberblock_verify(ub) == 0) { >> mutex_enter(&spa->spa_uberblock_lock); >> - if (vdev_uberblock_compare(ub, ubbest) > 0) >> + printf("JMR: vdev_uberblock_load_done ub_txg=%qd >> ub_timestamp=%qd\n", ub->ub_txg, ub->ub_timestamp); >> + if (vdev_uberblock_compare(ub, ubbest) > 0 && >> ub->ub_txg < 4254783) >> *ubbest = *ub; >> mutex_exit(&spa->spa_uberblock_lock); >> } >> + printf("JMR: vdev_uberblock_load_done ubbest ub_txg=%qd >> ub_timestamp=%qd\n", ubbest->ub_txg, ubbest->ub_timestamp); >> >> zio_buf_free(zio->io_data, zio->io_size); >> } >> >> After compiling and loading the zfs.ko module, I executed "zpool >> import" and the pool was still faulted. So, I decremented the limit >> txg to "< 4254782" and this time the zpool came up as ONLINE. After >> crossing my fingers I executed "zpool import z1", and it worked ok. >> No data loss, everything back to normal. The only curious thing I've >> noticed is this: >> >> # zpool status >> pool: z1 >> state: ONLINE >> status: One or more devices could not be used because the label is >> missing or >> invalid. Sufficient replicas exist for the pool to continue >> functioning in a degraded state. >> action: Replace the device using 'zpool replace'. >> see: http://www.sun.com/msg/ZFS-8000-4J >> scrub: resilver completed with 0 errors on Tue Feb 3 09:26:40 2009 >> config: >> >> NAME STATE READ WRITE CKSUM >> z1 ONLINE 0 0 0 >> raidz2 ONLINE 0 0 0 >> mirror/gm0 ONLINE 0 0 0 >> mirror/gm1 ONLINE 0 0 0 >> da2 ONLINE 0 0 0 >> da3 ONLINE 0 0 0 >> 8076139616933977534 UNAVAIL 0 0 0 was /dev/da4 >> da5 ONLINE 0 0 0 >> da6 ONLINE 0 0 0 >> da7 ONLINE 0 0 0 >> >> errors: No known data errors >> >> As you can see, the raidz2 vdev is marked as ONLINE, when I think it >> should be DEGRADED. Nevertheless, the pool is readable and writeable, >> and so far I haven't detected any problem. To be safe, I am >> extracting all the data and I will recreate the pool again from >> scratch, just in case. >> >> >> Pending questions: >> >> 1) Why did the "supposed corruption" happened in the first place? I >> advise people not to mix disks from different zpools with the same >> name in the same computer. That's what I did, and maybe it's what >> caused my problems. >> >> 2) Rolling back to an earlier uberblock seems to solve some faulted >> zpool problems. I think it would be interesting to have a program >> that let you do it in a user-friendly way (after warning you about >> the dangers, etc.). >> > > > It would be interesting to see if the txid from all of your labels was > the same. I would highly advise scrubbing your array. I did a zdb -l in all the healthy disks, and all the labels (4 copies x 7 devices) were identical, except for the "guid" field at the beginning. That's the vdev's guid, so I think it's normal it's different for each disk. The txg field was identical in all of them. > > I believe the reason that your "da4" is showing up with only a uuid is > because zfs is now recognizing that the da4 it sees is not the correct > one. Still very curious how you ended up in that situation. I wonder > if you had corruption that was unknown before you removed da4. Definitely the current da4 has nothing to do with the zpool. First it belonged to a different zpool and later I zeroed the beginning and end. The GUID that is listed in "zpool status" is the same one that appears in the zpool labels for the old da4. I don't recall seeing any corruption before, and I scrubbed the pool from time to time. By the way, thinking again about this, the timestamp on the most recent uberblock was 6:32 CET, which also coincides with the time that the server froze, while the change of disks took place about 2-3 hours later. So, maybe the change of disks had nothing to do with all this after all. The disks are connected to a RAID controller, although they are exported in pass-through mode. From owner-freebsd-fs@FreeBSD.ORG Tue Feb 3 15:03:58 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 814FA106564A for ; Tue, 3 Feb 2009 15:03:58 +0000 (UTC) (envelope-from dan.cojocar@gmail.com) Received: from an-out-0708.google.com (an-out-0708.google.com [209.85.132.244]) by mx1.freebsd.org (Postfix) with ESMTP id 41ECB8FC19 for ; Tue, 3 Feb 2009 15:03:57 +0000 (UTC) (envelope-from dan.cojocar@gmail.com) Received: by an-out-0708.google.com with SMTP id b38so841793ana.13 for ; Tue, 03 Feb 2009 07:03:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type:content-transfer-encoding; bh=K4kIMsvQjEDm4KXsKU1ifckFb/2YNQ64BnCsfkDNGsQ=; b=E+4VttUXEiYQCJAntmIX23rAFOWFCZMQp8TngG+yEe1xtBs+U8rntFIcFkRwkyHmR3 nGchQiZo62V2II9iohkidzGM/JZe1se32/29tVnxreSB1Cv7EHQcWsxhnQU5r/GTBaP1 crcE0GIci8IXS52B4LCsNQzxD8Jy2GfggghBo= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type :content-transfer-encoding; b=xshraO718UGwGi6f4H/KV/kq62tupTgsy7HPOezSM7NM7dDi/jJe6XLv8pg6ZG9eJw WDEoYt9Sc8Tq8tBImX9q4LuErRyXiArqB7BAwFt+w9zBYhyPUQwaBmvFW+JO58qlVmE8 DMfR6UNxACTgrMLi6rAZJWH/pAkyOyilT950Y= MIME-Version: 1.0 Received: by 10.100.211.3 with SMTP id j3mr3674965ang.42.1233671608456; Tue, 03 Feb 2009 06:33:28 -0800 (PST) Date: Tue, 3 Feb 2009 16:33:28 +0200 Message-ID: From: Dan Cojocar To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: zfs replace disk has failed X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Feb 2009 15:03:58 -0000 Hello all, In a mirror(ad1,ad2) configuration one of my disk(ad1) had failed, after replacing the failed disk with a new one using: zpool replace tank ad1 I have noticed that the replace is taking too long and that the system is not responding, after restart the new disk was not recognized any more in bios :(, I have tested also in another box and the disk was not recognized there too. I have installed a new one on the same location (ad1 I think). Then the zpool status has reported something like this (this is from memory because I have made many changes back then, I don't remember exactly if the online disk was ad1 or ad2): zpool status pool: tank state: DEGRADED scrub: none requested config: NAME STATE READ WRITE CKSUM tank DEGRADED 0 0 0 mirror DEGRADED 0 0 0 replacing UNAVAIL 0 387 0 insufficient replicas 10193841952954445329 REMOVED 0 0 0 was /dev/ad1/old 9318348042598806923 FAULTED 0 0 0 was /dev/ad1 ad2 ONLINE 0 0 0 At this stage I was thinking that if I will attach the new disk (ad1) to the mirror I will get sufficient replicas to detach 9318348042598806923 (this one was the disk that has failed the second time), so I did an attach, after the resilvering process has completed with success, I had: zpool status pool: tank state: DEGRADED scrub: none requested config: NAME STATE READ WRITE CKSUM tank DEGRADED 0 0 0 mirror DEGRADED 0 0 0 replacing UNAVAIL 0 387 0 insufficient replicas 10193841952954445329 REMOVED 0 0 0 was /dev/ad1/old 9318348042598806923 FAULTED 0 0 0 was /dev/ad1 ad2 ONLINE 0 0 0 ad1 ONLINE 0 0 0 And I'm not able to detach 9318348042598806923 :(, and another bad news is that if I try to access something under /tank the operation is hanging, eg: if I do a ls /tank is freezing and if I do in another console: zpool status which was working before ls, now it's freezing too. What should I do next? Thanks, Dan From owner-freebsd-fs@FreeBSD.ORG Tue Feb 3 15:40:06 2009 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A7604106568A for ; Tue, 3 Feb 2009 15:40:06 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 7BEA28FC20 for ; Tue, 3 Feb 2009 15:40:06 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id n13Fe6Ru049195 for ; Tue, 3 Feb 2009 15:40:06 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id n13Fe64U049194; Tue, 3 Feb 2009 15:40:06 GMT (envelope-from gnats) Date: Tue, 3 Feb 2009 15:40:06 GMT Message-Id: <200902031540.n13Fe64U049194@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org From: Jaakko Heinonen Cc: Subject: Re: kern/131009: System freezes when attempting to copy from one mounted (USB-disk-resident) ext2 filesystem to another X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Jaakko Heinonen List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Feb 2009 15:40:07 -0000 The following reply was made to PR kern/131009; it has been noted by GNATS. From: Jaakko Heinonen To: Donald Allen Cc: bug-followup@FreeBSD.org Subject: Re: kern/131009: System freezes when attempting to copy from one mounted (USB-disk-resident) ext2 filesystem to another Date: Tue, 3 Feb 2009 17:33:47 +0200 On 2009-01-26, Donald Allen wrote: > Thank you for the patch. Building a kernel is next on my agenda, and I > will install the fix when I do. Can you confirm that the patch fixed the problem for you? -- Jaakko From owner-freebsd-fs@FreeBSD.ORG Tue Feb 3 16:13:08 2009 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3AEE41065692; Tue, 3 Feb 2009 16:13:08 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 0BF8B8FC0A; Tue, 3 Feb 2009 16:13:08 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (linimon@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id n13GD7Bd076867; Tue, 3 Feb 2009 16:13:07 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id n13GD7Ll076863; Tue, 3 Feb 2009 16:13:07 GMT (envelope-from linimon) Date: Tue, 3 Feb 2009 16:13:07 GMT Message-Id: <200902031613.n13GD7Ll076863@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-fs@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/131342: [nfs] mounting/unmounting of disks causes NFS to fail X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Feb 2009 16:13:12 -0000 Synopsis: [nfs] mounting/unmounting of disks causes NFS to fail Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: linimon Responsible-Changed-When: Tue Feb 3 16:12:58 UTC 2009 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=131342 From owner-freebsd-fs@FreeBSD.ORG Tue Feb 3 16:40:36 2009 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0111C106566C; Tue, 3 Feb 2009 16:40:36 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id C99CC8FC1B; Tue, 3 Feb 2009 16:40:35 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (linimon@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id n13GeZ8x098426; Tue, 3 Feb 2009 16:40:35 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id n13GeZvO098410; Tue, 3 Feb 2009 16:40:35 GMT (envelope-from linimon) Date: Tue, 3 Feb 2009 16:40:35 GMT Message-Id: <200902031640.n13GeZvO098410@freefall.freebsd.org> To: donaldcallen@gmail.com, linimon@FreeBSD.org, freebsd-fs@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/131009: [ext2fs] [hang] System freezes when attempting to copy from one mounted (USB-disk-resident) ext2 filesystem to another X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Feb 2009 16:40:36 -0000 Synopsis: [ext2fs] [hang] System freezes when attempting to copy from one mounted (USB-disk-resident) ext2 filesystem to another State-Changed-From-To: open->closed State-Changed-By: linimon State-Changed-When: Tue Feb 3 16:39:25 UTC 2009 State-Changed-Why: Closed at submitter's request. http://www.freebsd.org/cgi/query-pr.cgi?pr=131009 From owner-freebsd-fs@FreeBSD.ORG Tue Feb 3 17:30:03 2009 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9282F10656D8 for ; Tue, 3 Feb 2009 17:30:03 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 7FBC18FC2E for ; Tue, 3 Feb 2009 17:30:03 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id n13HU3p0030502 for ; Tue, 3 Feb 2009 17:30:03 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id n13HU3sj030496; Tue, 3 Feb 2009 17:30:03 GMT (envelope-from gnats) Date: Tue, 3 Feb 2009 17:30:03 GMT Message-Id: <200902031730.n13HU3sj030496@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org From: Russell Cattelan Cc: Subject: Re: kern/131084: [xfs] xfs destroys itself after copying data X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Russell Cattelan List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Feb 2009 17:30:04 -0000 The following reply was made to PR kern/131084; it has been noted by GNATS. From: Russell Cattelan To: bug-followup@FreeBSD.org, estellnb@gmail.com Cc: Subject: Re: kern/131084: [xfs] xfs destroys itself after copying data Date: Tue, 03 Feb 2009 10:53:25 -0600 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Write support is not anywhere close to finished. I have a much more modern version of XFS working with FreeBSD current but it also is very early stages of being able to write the and xfs filesystem. - -Russell Cattelan -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJiHaFNRmM+OaGhBgRAkYHAJ9lnFSN64WSKTdOnn35y2+7DlPl5wCfb//H CCCrq6fRlHtlhTt7+3pX/UM= =hRXC -----END PGP SIGNATURE----- From owner-freebsd-fs@FreeBSD.ORG Wed Feb 4 13:33:06 2009 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4BFEA1065670; Wed, 4 Feb 2009 13:33:06 +0000 (UTC) (envelope-from gavin@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 21F2E8FC14; Wed, 4 Feb 2009 13:33:06 +0000 (UTC) (envelope-from gavin@FreeBSD.org) Received: from freefall.freebsd.org (gavin@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id n14DX6MR091594; Wed, 4 Feb 2009 13:33:06 GMT (envelope-from gavin@freefall.freebsd.org) Received: (from gavin@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id n14DX5xt091590; Wed, 4 Feb 2009 13:33:06 GMT (envelope-from gavin) Date: Wed, 4 Feb 2009 13:33:06 GMT Message-Id: <200902041333.n14DX5xt091590@freefall.freebsd.org> To: gavin@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-fs@FreeBSD.org From: gavin@FreeBSD.org Cc: Subject: Re: kern/131356: [tmpfs][patch] unlink(2) on tmpfs removs wrong files with hard-links X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Feb 2009 13:33:06 -0000 Synopsis: [tmpfs][patch] unlink(2) on tmpfs removs wrong files with hard-links Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: gavin Responsible-Changed-When: Wed Feb 4 13:13:39 UTC 2009 Responsible-Changed-Why: Over to maintainer(s) http://www.freebsd.org/cgi/query-pr.cgi?pr=131356 From owner-freebsd-fs@FreeBSD.ORG Wed Feb 4 16:13:38 2009 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 91C6E106564A; Wed, 4 Feb 2009 16:13:38 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 689538FC12; Wed, 4 Feb 2009 16:13:38 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (linimon@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id n14GDc0v012130; Wed, 4 Feb 2009 16:13:38 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id n14GDckg012126; Wed, 4 Feb 2009 16:13:38 GMT (envelope-from linimon) Date: Wed, 4 Feb 2009 16:13:38 GMT Message-Id: <200902041613.n14GDckg012126@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-fs@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/131360: [nfs] poor scaling behavior of the NFS server under load X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Feb 2009 16:13:39 -0000 Synopsis: [nfs] poor scaling behavior of the NFS server under load Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: linimon Responsible-Changed-When: Wed Feb 4 16:13:29 UTC 2009 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=131360 From owner-freebsd-fs@FreeBSD.ORG Wed Feb 4 18:40:22 2009 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 04C0510656BD; Wed, 4 Feb 2009 18:40:22 +0000 (UTC) (envelope-from remko@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id CC1028FC13; Wed, 4 Feb 2009 18:40:21 +0000 (UTC) (envelope-from remko@FreeBSD.org) Received: from freefall.freebsd.org (remko@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id n14IeL2f022746; Wed, 4 Feb 2009 18:40:21 GMT (envelope-from remko@freefall.freebsd.org) Received: (from remko@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id n14IeLqI022736; Wed, 4 Feb 2009 18:40:21 GMT (envelope-from remko) Date: Wed, 4 Feb 2009 18:40:21 GMT Message-Id: <200902041840.n14IeLqI022736@freefall.freebsd.org> To: remko@FreeBSD.org, freebsd-i386@FreeBSD.org, freebsd-fs@FreeBSD.org From: remko@FreeBSD.org Cc: Subject: Re: bin/131341: makefs: error "Bad file descriptor" on the mount point of md-presentation makefs image X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Feb 2009 18:40:23 -0000 Old Synopsis: error "Bad file descriptor" on the mount point of md-presentation makefs image New Synopsis: makefs: error "Bad file descriptor" on the mount point of md-presentation makefs image Responsible-Changed-From-To: freebsd-i386->freebsd-fs Responsible-Changed-By: remko Responsible-Changed-When: Wed Feb 4 18:40:08 UTC 2009 Responsible-Changed-Why: reassign to -fs http://www.freebsd.org/cgi/query-pr.cgi?pr=131341 From owner-freebsd-fs@FreeBSD.ORG Wed Feb 4 20:52:36 2009 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AC4F71065675; Wed, 4 Feb 2009 20:52:36 +0000 (UTC) (envelope-from remko@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 81A6D8FC1A; Wed, 4 Feb 2009 20:52:36 +0000 (UTC) (envelope-from remko@FreeBSD.org) Received: from freefall.freebsd.org (remko@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id n14KqaQC024686; Wed, 4 Feb 2009 20:52:36 GMT (envelope-from remko@freefall.freebsd.org) Received: (from remko@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id n14KqaNM024682; Wed, 4 Feb 2009 20:52:36 GMT (envelope-from remko) Date: Wed, 4 Feb 2009 20:52:36 GMT Message-Id: <200902042052.n14KqaNM024682@freefall.freebsd.org> To: remko@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-fs@FreeBSD.org From: remko@FreeBSD.org Cc: Subject: Re: kern/131353: gjournal kernel lock X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Feb 2009 20:52:37 -0000 Synopsis: gjournal kernel lock Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: remko Responsible-Changed-When: Wed Feb 4 20:52:26 UTC 2009 Responsible-Changed-Why: reassign to -fs http://www.freebsd.org/cgi/query-pr.cgi?pr=131353 From owner-freebsd-fs@FreeBSD.ORG Thu Feb 5 20:52:11 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D21B2106566B for ; Thu, 5 Feb 2009 20:52:11 +0000 (UTC) (envelope-from admin@kedvenc.hu) Received: from site.hu (site.hu [212.92.23.185]) by mx1.freebsd.org (Postfix) with ESMTP id 8B49E8FC0A for ; Thu, 5 Feb 2009 20:52:11 +0000 (UTC) (envelope-from admin@kedvenc.hu) Received: from www-data by site.hu with local (Exim 4.63 #1 (Debian)) id 1LVAmd-0002Ax-5m for ; Thu, 05 Feb 2009 21:25:11 +0100 Received: from 92-249-229-208.pool.digikabel.hu (92-249-229-208.pool.digikabel.hu [92.249.229.208]) by www.site.hu (Horde MIME library) with HTTP; Thu, 05 Feb 2009 21:25:11 +0100 Message-ID: <20090205212511.jrmipmazeo4o0c8w@www.site.hu> X-Priority: 1 (Highest) Date: Thu, 05 Feb 2009 21:25:11 +0100 From: Joe7 To: freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-2; DelSp="Yes"; format="flowed" Content-Disposition: inline Content-Transfer-Encoding: 7bit User-Agent: Internet Messaging Program (IMP) H3 (4.1.4) Subject: nfs delay causing broken files X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Feb 2009 20:52:12 -0000 Hi there, I'd like to know how to tune NFS (or related kernel) settings so the NFS client writes would have NO delay? I'm getting application level errors as NFS client writes data, then application would read the data from NFS master just 1 second later and it's not there yet, so outdated data is read. Thanks for any ideas in avance, Joe From owner-freebsd-fs@FreeBSD.ORG Fri Feb 6 16:55:43 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7BFA21065672 for ; Fri, 6 Feb 2009 16:55:43 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from aeryn.cs.uoguelph.ca (aeryn.cs.uoguelph.ca [131.104.20.160]) by mx1.freebsd.org (Postfix) with ESMTP id 3C4D98FC0A for ; Fri, 6 Feb 2009 16:55:43 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from muncher.cs.uoguelph.ca (muncher.cs.uoguelph.ca [131.104.91.102]) by aeryn.cs.uoguelph.ca (8.13.1/8.13.1) with ESMTP id n16GYS5n022004; Fri, 6 Feb 2009 11:34:29 -0500 Received: from localhost (rmacklem@localhost) by muncher.cs.uoguelph.ca (8.11.7p3+Sun/8.11.6) with ESMTP id n16GcFf08129; Fri, 6 Feb 2009 11:38:15 -0500 (EST) X-Authentication-Warning: muncher.cs.uoguelph.ca: rmacklem owned process doing -bs Date: Fri, 6 Feb 2009 11:38:14 -0500 (EST) From: Rick Macklem X-X-Sender: rmacklem@muncher.cs.uoguelph.ca To: Joe7 In-Reply-To: <20090205212511.jrmipmazeo4o0c8w@www.site.hu> Message-ID: References: <20090205212511.jrmipmazeo4o0c8w@www.site.hu> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Scanned-By: MIMEDefang 2.63 on 131.104.20.161 Cc: freebsd-fs@freebsd.org Subject: Re: nfs delay causing broken files X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Feb 2009 16:55:44 -0000 On Thu, 5 Feb 2009, Joe7 wrote: > > Hi there, > > I'd like to know how to tune NFS (or related kernel) settings so the NFS > client writes would have NO delay? > I'm getting application level errors as NFS client writes data, then > application would read the data from NFS master just 1 second later and it's > not there yet, so outdated data is read. > You will have to enable synchronous writing (which will be a big performance hit, but...): - either mount with the "sync" option OR - have the apps open the files with O_SYNC (or O_FSYNC) If the recent writes need to be visible on other clients (and not just the NFS server), you will also have to bypass the client side caching for readers. This can be done by setting nfs_directio_enable to non-zero using sysctl and opening the files with O_DIRECT. (I think setting acregmax=0 as a mount option should achieve the same result, if you can't add O_DIRECT to the apps.) Again, a big performance hit, but... As an historical aside, way back when (before NFSv3) I concocked something callled not quite nfs, which included a cache coherency protocol, but it didn't catch on. Cache coherency seems to have never been a priority for the NFS crowd. Even in NFSv4, there isn't a cache coherency protocol, nor any interest in adding one. (In fairness, coherency between multiple clients can be achieved in NFSv4 for data by the clients putting byte range locks on all the byte ranges.) rick From owner-freebsd-fs@FreeBSD.ORG Fri Feb 6 17:22:33 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 07052106566B for ; Fri, 6 Feb 2009 17:22:33 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from mailhub.cs.uoguelph.ca (mailhub.cs.uoguelph.ca [131.104.94.205]) by mx1.freebsd.org (Postfix) with ESMTP id A14508FC0C for ; Fri, 6 Feb 2009 17:22:32 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from muncher.cs.uoguelph.ca (muncher.cs.uoguelph.ca [131.104.91.102]) by mailhub.cs.uoguelph.ca (8.13.1/8.13.1) with ESMTP id n16HMThn025119; Fri, 6 Feb 2009 12:22:29 -0500 Received: from localhost (rmacklem@localhost) by muncher.cs.uoguelph.ca (8.11.7p3+Sun/8.11.6) with ESMTP id n16HQGo14774; Fri, 6 Feb 2009 12:26:16 -0500 (EST) X-Authentication-Warning: muncher.cs.uoguelph.ca: rmacklem owned process doing -bs Date: Fri, 6 Feb 2009 12:26:16 -0500 (EST) From: Rick Macklem X-X-Sender: rmacklem@muncher.cs.uoguelph.ca To: Joe7 In-Reply-To: Message-ID: References: <20090205212511.jrmipmazeo4o0c8w@www.site.hu> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Scanned-By: MIMEDefang 2.63 on 131.104.94.205 Cc: freebsd-fs@freebsd.org Subject: Re: nfs delay causing broken files X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Feb 2009 17:22:33 -0000 On Fri, 6 Feb 2009, Rick Macklem wrote: [stuff snipped] > > If the recent writes need to be visible on other clients (and not just the > NFS server), you will also have to bypass the client side caching for > readers. This can be done by setting nfs_directio_enable to non-zero > using sysctl and opening the files with O_DIRECT. (I think setting > acregmax=0 as a mount option should achieve the same result, if you can't > add O_DIRECT to the apps.) Again, a big performance hit, but... > I know it's weird to reply to my own post, but I realized I should clarify that setting acregmax=0 will only achieve this approximately, based on the clock resolution used for the file's modify time. Setting acregmax=0 should disable client side attribute caching, such that the client always does a Getattr against the server. Then, if the mtime attribute for the file has changed since the client cached data, it will be purged. As such, this only works when the mtime has changed and that will be based upon clock resolution (and if the mtime got saved on the server's disk, for the case where the server has crashed/rebooted). Have a good weekend, rick From owner-freebsd-fs@FreeBSD.ORG Fri Feb 6 17:27:21 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5B9AD1065670 for ; Fri, 6 Feb 2009 17:27:21 +0000 (UTC) (envelope-from admin@kedvenc.hu) Received: from site.hu (site.hu [212.92.23.185]) by mx1.freebsd.org (Postfix) with ESMTP id 159E78FC1E for ; Fri, 6 Feb 2009 17:27:20 +0000 (UTC) (envelope-from admin@kedvenc.hu) Received: from www-data by site.hu with local (Exim 4.63 #1 (Debian)) id 1LVUU2-0003Kn-Fq; Fri, 06 Feb 2009 18:27:18 +0100 Received: from 92-249-229-208.pool.digikabel.hu (92-249-229-208.pool.digikabel.hu [92.249.229.208]) by www.site.hu (Horde MIME library) with HTTP; Fri, 06 Feb 2009 18:27:18 +0100 Message-ID: <20090206182718.dhsvxzt7r4sswos4@www.site.hu> X-Priority: 3 (Normal) Date: Fri, 06 Feb 2009 18:27:18 +0100 From: Joe7 To: Rick Macklem References: <20090205212511.jrmipmazeo4o0c8w@www.site.hu> In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-2; DelSp="Yes"; format="flowed" Content-Disposition: inline Content-Transfer-Encoding: quoted-printable User-Agent: Internet Messaging Program (IMP) H3 (4.1.4) Cc: freebsd-fs@freebsd.org Subject: Re: nfs delay causing broken files X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Feb 2009 17:27:21 -0000 Hi, Thank you for reply, but: I'm afraid sync mount is not an option: mount_nfs: -o sync: option not supported Tried acregmax=3D0 already, and that made no difference unfortunately. Application level O_DIRECT is not an option for us, so I'm wondering =20 what can be done at mount level? thanks, Joe Id=E9zet (Rick Macklem ): > > > On Fri, 6 Feb 2009, Rick Macklem wrote: > > [stuff snipped] >> >> If the recent writes need to be visible on other clients (and not just th= e >> NFS server), you will also have to bypass the client side caching for >> readers. This can be done by setting nfs_directio_enable to non-zero >> using sysctl and opening the files with O_DIRECT. (I think setting >> acregmax=3D0 as a mount option should achieve the same result, if you can= 't >> add O_DIRECT to the apps.) Again, a big performance hit, but... >> > I know it's weird to reply to my own post, but I realized I should clarify > that setting acregmax=3D0 will only achieve this approximately, based on > the clock resolution used for the file's modify time. Setting acregmax=3D0 > should disable client side attribute caching, such that the client always > does a Getattr against the server. Then, if the mtime attribute for the > file has changed since the client cached data, it will be purged. As such, > this only works when the mtime has changed and that will be based upon > clock resolution (and if the mtime got saved on the server's disk, for the > case where the server has crashed/rebooted). > > Have a good weekend, rick From owner-freebsd-fs@FreeBSD.ORG Fri Feb 6 18:36:43 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 40093106566C for ; Fri, 6 Feb 2009 18:36:43 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from acadia.cs.uoguelph.ca (acadia.cs.uoguelph.ca [131.104.94.221]) by mx1.freebsd.org (Postfix) with ESMTP id D866F8FC12 for ; Fri, 6 Feb 2009 18:36:42 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from muncher.cs.uoguelph.ca (muncher.cs.uoguelph.ca [131.104.91.102]) by acadia.cs.uoguelph.ca (8.13.1/8.13.1) with ESMTP id n16IafWJ018888; Fri, 6 Feb 2009 13:36:41 -0500 Received: from localhost (rmacklem@localhost) by muncher.cs.uoguelph.ca (8.11.7p3+Sun/8.11.6) with ESMTP id n16IeSW24436; Fri, 6 Feb 2009 13:40:28 -0500 (EST) X-Authentication-Warning: muncher.cs.uoguelph.ca: rmacklem owned process doing -bs Date: Fri, 6 Feb 2009 13:40:28 -0500 (EST) From: Rick Macklem X-X-Sender: rmacklem@muncher.cs.uoguelph.ca To: Joe7 In-Reply-To: <20090206182718.dhsvxzt7r4sswos4@www.site.hu> Message-ID: References: <20090205212511.jrmipmazeo4o0c8w@www.site.hu> <20090206182718.dhsvxzt7r4sswos4@www.site.hu> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Scanned-By: MIMEDefang 2.63 on 131.104.94.221 Cc: freebsd-fs@freebsd.org Subject: Re: nfs delay causing broken files X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Feb 2009 18:36:43 -0000 On Fri, 6 Feb 2009, Joe7 wrote: > Hi, > > Thank you for reply, but: > I'm afraid sync mount is not an option: > mount_nfs: -o sync: option not supported > > Tried acregmax=0 already, and that made no difference unfortunately. > Application level O_DIRECT is not an option for us, so I'm wondering what can > be done at mount level? > Oops, I saw that MNT_SYNCHRONOUS would set IO_SYNC, but didn't check to see if the mount would actually work. Sorry about that. Hmm, I think that, without O_DIRECT. you are SOL (shit outa luck:-). rick From owner-freebsd-fs@FreeBSD.ORG Fri Feb 6 19:03:47 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 34D531065716 for ; Fri, 6 Feb 2009 19:03:47 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from gigi.cs.uoguelph.ca (gigi.cs.uoguelph.ca [131.104.94.210]) by mx1.freebsd.org (Postfix) with ESMTP id D14478FC2D for ; Fri, 6 Feb 2009 19:03:46 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from muncher.cs.uoguelph.ca (muncher.cs.uoguelph.ca [131.104.91.102]) by gigi.cs.uoguelph.ca (8.13.1/8.13.1) with ESMTP id n16J3j63000985; Fri, 6 Feb 2009 14:03:45 -0500 Received: from localhost (rmacklem@localhost) by muncher.cs.uoguelph.ca (8.11.7p3+Sun/8.11.6) with ESMTP id n16J7Wi28342; Fri, 6 Feb 2009 14:07:32 -0500 (EST) X-Authentication-Warning: muncher.cs.uoguelph.ca: rmacklem owned process doing -bs Date: Fri, 6 Feb 2009 14:07:32 -0500 (EST) From: Rick Macklem X-X-Sender: rmacklem@muncher.cs.uoguelph.ca To: admin@kedvenc.hu Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Scanned-By: MIMEDefang 2.63 on 131.104.94.210 Cc: freebsd-fs@freebsd.org Subject: Re: nfs delay causing broken files X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Feb 2009 19:03:47 -0000 Well, maybe not SOL if you are willing to hack the kernel sources. If you change the following line in sys/nfsclient/nfs_bio.c as follows: if (nfs_directio_enable && (ioflag & IO_DIRECT) && vp->v_type == VREG) return nfs_directio_write(vp, uio, cred, ioflag); to if (nfs_directio_enable && vp->v_type == VREG) return nfs_directio_write(vp, uio, cred, ioflag); then all writes would be pushed to the server if nfs_directio_enable is set non-zero by sysctl. (In other words, O_DIRECT would be set for all opens on the NFS mounts.) Ugly, but if it fixes your problem...rick ps: I'm thinking that a mount option that does this might be useful? From owner-freebsd-fs@FreeBSD.ORG Fri Feb 6 19:37:47 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 605AB10657F5 for ; Fri, 6 Feb 2009 19:37:47 +0000 (UTC) (envelope-from admin@kedvenc.hu) Received: from site.hu (site.hu [212.92.23.185]) by mx1.freebsd.org (Postfix) with ESMTP id 1BC368FC22 for ; Fri, 6 Feb 2009 19:37:47 +0000 (UTC) (envelope-from admin@kedvenc.hu) Received: from www-data by site.hu with local (Exim 4.63 #1 (Debian)) id 1LVWWH-00015t-Nt; Fri, 06 Feb 2009 20:37:45 +0100 Received: from 92-249-229-208.pool.digikabel.hu (92-249-229-208.pool.digikabel.hu [92.249.229.208]) by www.site.hu (Horde MIME library) with HTTP; Fri, 06 Feb 2009 20:37:45 +0100 Message-ID: <20090206203745.57a035vq2so44ok0@www.site.hu> X-Priority: 3 (Normal) Date: Fri, 06 Feb 2009 20:37:45 +0100 From: Joe7 To: Rick Macklem References: In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-2; DelSp="Yes"; format="flowed" Content-Disposition: inline Content-Transfer-Encoding: quoted-printable User-Agent: Internet Messaging Program (IMP) H3 (4.1.4) Cc: freebsd-fs@freebsd.org Subject: Re: nfs delay causing broken files X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Feb 2009 19:37:48 -0000 Okay, So although it's likely that kernel hack would do it, am I right that a wrapper script with open(...O_DIRECT) would do the job? Basicly i'm creating an file with imagemagick and wanna place that on =20 the nfs server. So I assume if I create the file locally and copy over using a little =20 script that uses open(.. O_DIRECT), it would just work? Application is PHP+imagemagick binary thus pretty high level compared =20 to this stuff, but please let me know if you agree! Thanks Joe Id=E9zet (Rick Macklem ): > Well, maybe not SOL if you are willing to hack the kernel sources. If you > change the following line in sys/nfsclient/nfs_bio.c as follows: > =09if (nfs_directio_enable && (ioflag & IO_DIRECT) && vp->v_type =3D=3D VR= EG) > =09=09return nfs_directio_write(vp, uio, cred, ioflag); > to > =09if (nfs_directio_enable && vp->v_type =3D=3D VREG) > =09=09return nfs_directio_write(vp, uio, cred, ioflag); > > then all writes would be pushed to the server if nfs_directio_enable is > set non-zero by sysctl. (In other words, O_DIRECT would be set for all > opens on the NFS mounts.) > > Ugly, but if it fixes your problem...rick > ps: I'm thinking that a mount option that does this might be useful? From owner-freebsd-fs@FreeBSD.ORG Fri Feb 6 21:32:58 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B9C3A10656E0 for ; Fri, 6 Feb 2009 21:32:58 +0000 (UTC) (envelope-from admin@kedvenc.hu) Received: from site.hu (site.hu [212.92.23.185]) by mx1.freebsd.org (Postfix) with ESMTP id 77B068FC1E for ; Fri, 6 Feb 2009 21:32:57 +0000 (UTC) (envelope-from admin@kedvenc.hu) Received: from www-data by site.hu with local (Exim 4.63 #1 (Debian)) id 1LVYJj-0006po-Vh for ; Fri, 06 Feb 2009 22:32:56 +0100 Received: from 92-249-229-208.pool.digikabel.hu (92-249-229-208.pool.digikabel.hu [92.249.229.208]) by www.site.hu (Horde MIME library) with HTTP; Fri, 06 Feb 2009 22:32:55 +0100 Message-ID: <20090206223255.nx3wziczkwso0w4o@www.site.hu> X-Priority: 3 (Normal) Date: Fri, 06 Feb 2009 22:32:55 +0100 From: Joe7 To: freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-2; DelSp="Yes"; format="flowed" Content-Disposition: inline Content-Transfer-Encoding: quoted-printable User-Agent: Internet Messaging Program (IMP) H3 (4.1.4) Subject: Re: nfs delay causing broken files X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Feb 2009 21:32:59 -0000 So I tried, wrote a tenliner c script that copies the files ouF =3D open(argv[2], O_WRONLY | O_CREAT | O_DIRECT | O_SYNC) opening the target with that, but tested and the same problem occurs am I missing something obvious? Thanks for all your help, Joe Id=E9zet (Rick Macklem ): > > > On Fri, 6 Feb 2009, Joe7 wrote: > >> Okay, >> >> So although it's likely that kernel hack would do it, >> am I right that a wrapper script with open(...O_DIRECT) would do the job? >> >> Basicly i'm creating an file with imagemagick and wanna place that =20 >> on the nfs server. >> So I assume if I create the file locally and copy over using a =20 >> little script that uses open(.. O_DIRECT), it would just work? >> Application is PHP+imagemagick binary thus pretty high level =20 >> compared to this stuff, but please let me know if you agree! >> > Sounds like it might work. Good luck with it, rick From owner-freebsd-fs@FreeBSD.ORG Sat Feb 7 15:40:06 2009 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2D5C71065702 for ; Sat, 7 Feb 2009 15:40:06 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 1C1E28FC18 for ; Sat, 7 Feb 2009 15:40:06 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id n17Fe55w062751 for ; Sat, 7 Feb 2009 15:40:05 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id n17Fe5nN062750; Sat, 7 Feb 2009 15:40:05 GMT (envelope-from gnats) Date: Sat, 7 Feb 2009 15:40:05 GMT Message-Id: <200902071540.n17Fe5nN062750@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org From: Martin Birgmeier Cc: Subject: Re: kern/131360: [nfs] poor scaling behavior of the NFS server under load X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Martin Birgmeier List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 07 Feb 2009 15:40:06 -0000 The following reply was made to PR kern/131360; it has been noted by GNATS. From: Martin Birgmeier To: bug-followup@FreeBSD.org Cc: Subject: Re: kern/131360: [nfs] poor scaling behavior of the NFS server under load Date: Sat, 7 Feb 2009 16:31:21 +0100 (CET) I am now very sure that this is an interaction with pppoa, and it is also worse than I originally thought: it will even lead to failed NFS transactions for the client. Here is what I have: Machine A ('server', a mini home server) does the following: - connecting to the Internet using usermode ppp over pppoa over an Alcatel ADSL modem - NFS serving FreeBSD sources Machine B does the following: - Mounting the FreeBSD sources from A (using amd), under directory /vol/SRC/FreeBSD/HEAD/src - Compiling the FreeBSD sources: make -j4 buildworld, such that the corresponding obj is local (via amd again) Especially in the first phase of the buildworld (clean, depend, obj), there is a lot of simultaneous NFS traffic from B to A. As soon as a download is started at A (going via pppoa, of course), the load on A rises to very high values (> 20 not uncommon). This may lead to B aborting the compile, it just did that with "directory not found". Both machines are running 7.1.0. No such problem happended when both were running 6.3.0. From owner-freebsd-fs@FreeBSD.ORG Sat Feb 7 16:40:06 2009 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EA88E106566C for ; Sat, 7 Feb 2009 16:40:06 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id BB2FA8FC08 for ; Sat, 7 Feb 2009 16:40:06 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id n17Ge6pI007704 for ; Sat, 7 Feb 2009 16:40:06 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id n17Ge6oQ007703; Sat, 7 Feb 2009 16:40:06 GMT (envelope-from gnats) Date: Sat, 7 Feb 2009 16:40:06 GMT Message-Id: <200902071640.n17Ge6oQ007703@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org From: Martin Birgmeier Cc: Subject: Re: kern/131360: [nfs] poor scaling behavior of the NFS server under load X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Martin Birgmeier List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 07 Feb 2009 16:40:07 -0000 The following reply was made to PR kern/131360; it has been noted by GNATS. From: Martin Birgmeier To: bug-followup@FreeBSD.org Cc: Subject: Re: kern/131360: [nfs] poor scaling behavior of the NFS server under load Date: Sat, 7 Feb 2009 17:32:15 +0100 (CET) o.k. pppoa does not have (much) to do with it... even when it is not running, the excessive load happens. From owner-freebsd-fs@FreeBSD.ORG Sat Feb 7 17:00:17 2009 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3FEA2106564A for ; Sat, 7 Feb 2009 17:00:17 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 2E2998FC0A for ; Sat, 7 Feb 2009 17:00:17 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id n17H0Hce021580 for ; Sat, 7 Feb 2009 17:00:17 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id n17H0HJT021579; Sat, 7 Feb 2009 17:00:17 GMT (envelope-from gnats) Date: Sat, 7 Feb 2009 17:00:17 GMT Message-Id: <200902071700.n17H0HJT021579@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org From: Martin Birgmeier Cc: Subject: Re: kern/131360: [nfs] poor scaling behavior of the NFS server under load X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Martin Birgmeier List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 07 Feb 2009 17:00:17 -0000 The following reply was made to PR kern/131360; it has been noted by GNATS. From: Martin Birgmeier To: bug-followup@FreeBSD.org Cc: Subject: Re: kern/131360: [nfs] poor scaling behavior of the NFS server under load Date: Sat, 7 Feb 2009 17:59:05 +0100 (CET) o.k. more info... I have this on the server (machine A in my previous post); it is showing quite a high number for "Server Ret-Failed": # nfsstat Client Info: Rpc Counts: Getattr Setattr Lookup Readlink Read Write Create Remove 2304000 0 221 165 0 0 0 0 Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access 0 0 0 0 0 22 0 0 Mknod Fsstat Fsinfo PathConf Commit 0 58 0 0 0 Rpc Info: TimedOut Invalid X Replies Retries Requests 0 0 17 17 2304466 Cache Info: Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW Hits Misses 0 2304000 885499 221 0 0 0 0 BioRLHits Misses BioD Hits Misses DirE Hits Misses 532202 165 89 11 50 0 Server Info: Getattr Setattr Lookup Readlink Read Write Create Remove 8 19 3123468 82 34382 3156 19 11 Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access 0 0 11 0 0 9522 0 868616 Mknod Fsstat Fsinfo PathConf Commit 0 31396 23 0 147 Server Ret-Failed 3046139 Server Faults 0 Server Cache Stats: Inprog Idem Non-idem Misses 0 51 0 412 Server Write Gathering: WriteOps WriteRPC Opsaved 3156 3156 0 and this on the client (machine B): # nfsstat Client Info: Rpc Counts: Getattr Setattr Lookup Readlink Read Write Create Remove 48679649 0 3035823 58 26400 0 0 0 Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access 0 0 0 0 0 7255 0 767940 Mknod Fsstat Fsinfo PathConf Commit 0 27347 1 0 0 Rpc Info: TimedOut Invalid X Replies Retries Requests 0 0 0 1 52543821 Cache Info: Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW Hits Misses 46870061 49414909 64135869 3035823 1144711 26389 0 0 BioRLHits Misses BioD Hits Misses DirE Hits Misses 8748412 58 43987 7218 25358 0 Server Info: Getattr Setattr Lookup Readlink Read Write Create Remove 0 0 0 0 0 0 0 0 Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access 0 0 0 0 0 0 0 0 Mknod Fsstat Fsinfo PathConf Commit 0 0 0 0 0 Server Ret-Failed 0 Server Faults 0 Server Cache Stats: Inprog Idem Non-idem Misses 0 0 0 0 Server Write Gathering: WriteOps WriteRPC Opsaved 0 0 0 From owner-freebsd-fs@FreeBSD.ORG Sat Feb 7 20:46:52 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E79AC1065674 for ; Sat, 7 Feb 2009 20:46:52 +0000 (UTC) (envelope-from peter@vk2pj.dyndns.org) Received: from fallbackmx08.syd.optusnet.com.au (fallbackmx08.syd.optusnet.com.au [211.29.132.10]) by mx1.freebsd.org (Postfix) with ESMTP id 6604C8FC16 for ; Sat, 7 Feb 2009 20:46:52 +0000 (UTC) (envelope-from peter@vk2pj.dyndns.org) Received: from mail16.syd.optusnet.com.au (mail16.syd.optusnet.com.au [211.29.132.197]) by fallbackmx08.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id n17K9OV0018833 for ; Sun, 8 Feb 2009 07:09:24 +1100 Received: from test71.vk2pj.dyndns.org (c122-106-216-167.belrs3.nsw.optusnet.com.au [122.106.216.167]) by mail16.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id n17K9L56031121 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Sun, 8 Feb 2009 07:09:22 +1100 X-Bogosity: Ham, spamicity=0.000000 Received: from test71.vk2pj.dyndns.org (localhost.vk2pj.dyndns.org [127.0.0.1]) by test71.vk2pj.dyndns.org (8.14.3/8.14.3) with ESMTP id n17K9KNT009914 for ; Sun, 8 Feb 2009 07:09:20 +1100 (EST) (envelope-from peter@test71.vk2pj.dyndns.org) Received: (from peter@localhost) by test71.vk2pj.dyndns.org (8.14.3/8.14.3/Submit) id n17K9JXN009913 for freebsd-fs@freebsd.org; Sun, 8 Feb 2009 07:09:19 +1100 (EST) (envelope-from peter) Date: Sun, 8 Feb 2009 07:09:19 +1100 From: Peter Jeremy To: freebsd-fs@freebsd.org Message-ID: <20090207200918.GA58657@test71.vk2pj.dyndns.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="GvXjxJ+pjyke8COw" Content-Disposition: inline X-PGP-Key: http://members.optusnet.com.au/peterjeremy/pubkey.asc User-Agent: Mutt/1.5.18 (2008-05-17) Subject: Unable to pwd in ZFS snapshot X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 07 Feb 2009 20:46:53 -0000 --GvXjxJ+pjyke8COw Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable I'm running -current from late last year (just after the ZFS v13 import) and have found that I can't determine the current working directory inside a snapshot: # df -k /usr/ports/.zfs/snapshot/pre_7.4 Filesystem 1024-blocks Used Avail Capacity Mounted on tank/ports@pre_7.4 877019136 31293824 845725312 4% /usr/ports/.zfs= /snap shot/pre_7.4 # cd /usr/ports/.zfs/snapshot/pre_7.4 # pwd pwd: .: No such file or directory # ls . =2Ecvsignore accessibility dns math security CHANGES arabic editors mbone shells COPYRIGHT archivers emulators misc sysutils CVS astro finance multimedia textproc GIDs audio french net ukrainian INDEX-7 benchmarks ftp net-im vietnamese INDEX-8 biology games net-mgmt work KNOBS build.cvsupdate german net-p2p www LEGAL build.diffs graphics news x11 MOVED cad hebrew packages x11-clocks Makefile chinese hungarian palm x11-drivers Mk comms irc polish x11-fm README converters japanese ports-mgmt x11-fonts Templates databases java portuguese x11-servers Tools deskutils korean print x11-themes UIDs devel lang russian x11-toolkits UPDATING distfiles mail science x11-wm #=20 This breaks (eg) make. I got around it by cloning the snapshot but this behaviour strikes me as counter-intuitive (and the error message leaves something to be desired). --=20 Peter Jeremy --GvXjxJ+pjyke8COw Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.10 (FreeBSD) iEYEARECAAYFAkmN6m4ACgkQ/opHv/APuIfoUACdEKTd9KlftfT/4WGMfJt4QD+6 TFgAniYt34ZbL2gBfHFSP1okBCS68nTk =SGv5 -----END PGP SIGNATURE----- --GvXjxJ+pjyke8COw-- From owner-freebsd-fs@FreeBSD.ORG Sat Feb 7 21:25:37 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 972A0106564A for ; Sat, 7 Feb 2009 21:25:37 +0000 (UTC) (envelope-from freebsd-listen@fabiankeil.de) Received: from smtprelay11.ispgateway.de (smtprelay11.ispgateway.de [80.67.29.28]) by mx1.freebsd.org (Postfix) with ESMTP id 52E118FC24 for ; Sat, 7 Feb 2009 21:25:37 +0000 (UTC) (envelope-from freebsd-listen@fabiankeil.de) Received: from [88.153.16.241] (helo=localhost) by smtprelay11.ispgateway.de with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.68) (envelope-from ) id 1LVuTa-0008Cb-0I for freebsd-fs@freebsd.org; Sat, 07 Feb 2009 22:12:34 +0100 Date: Sat, 7 Feb 2009 22:12:02 +0100 From: Fabian Keil To: freebsd-fs@freebsd.org Message-ID: <20090207221202.1a19456a@fabiankeil.de> In-Reply-To: <20090207200918.GA58657@test71.vk2pj.dyndns.org> References: <20090207200918.GA58657@test71.vk2pj.dyndns.org> X-Mailer: Claws Mail 3.7.0 (GTK+ 2.14.7; i386-portbld-freebsd8.0) X-PGP-KEY-URL: http://www.fabiankeil.de/gpg-keys/freebsd-listen-2008-08-18.asc Mime-Version: 1.0 Content-Type: multipart/signed; boundary="Sig_/8YQS2.hNh_aQuSsnBoUpYf2"; protocol="application/pgp-signature"; micalg=PGP-SHA1 X-Df-Sender: 775067 Subject: Re: Unable to pwd in ZFS snapshot X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 07 Feb 2009 21:25:38 -0000 --Sig_/8YQS2.hNh_aQuSsnBoUpYf2 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Peter Jeremy wrote: > I'm running -current from late last year (just after the ZFS v13 > import) and have found that I can't determine the current working > directory inside a snapshot: > # df -k /usr/ports/.zfs/snapshot/pre_7.4 > Filesystem 1024-blocks Used Avail Capacity Mounted on > tank/ports@pre_7.4 877019136 31293824 845725312 4% /usr/ports/.z= fs/snap > shot/pre_7.4 > # cd /usr/ports/.zfs/snapshot/pre_7.4 > # pwd > pwd: .: No such file or directory I can reproduce this on: FreeBSD TP51.local 8.0-CURRENT FreeBSD 8.0-CURRENT #30: Sat Feb 7 19:37:07= CET 2009 fk@TP51.local:/usr/obj/usr/src/sys/THINKPAD i386 It seems to work with bash's builtin though: fk@TP51 /tank/privoxy/.zfs/snapshot/2009-02-07 $df -k . Filesystem 1024-blocks Used Avail Capacity Mounted on tank/privoxy@2009-02-07 3704832 45824 3659008 1% /tank/privoxy/.= zfs/snapshot/2009-02-07 fk@TP51 /tank/privoxy/.zfs/snapshot/2009-02-07 $/bin/pwd pwd: .: No such file or directory fk@TP51 /tank/privoxy/.zfs/snapshot/2009-02-07 $pwd /tank/privoxy/.zfs/snapshot/2009-02-07 Fabian --Sig_/8YQS2.hNh_aQuSsnBoUpYf2 Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (FreeBSD) iEYEARECAAYFAkmN+SIACgkQBYqIVf93VJ3YBwCgq3KGFPVblZCaVK1vjbqUzvg8 kT0AnjKLGM3U8bKKh+JYJ230xU80Tyze =QU/B -----END PGP SIGNATURE----- --Sig_/8YQS2.hNh_aQuSsnBoUpYf2-- From owner-freebsd-fs@FreeBSD.ORG Sat Feb 7 22:04:32 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C4B05106566B for ; Sat, 7 Feb 2009 22:04:32 +0000 (UTC) (envelope-from morganw@chemikals.org) Received: from cdptpa-omtalb.mail.rr.com (cdptpa-omtalb.mail.rr.com [75.180.132.123]) by mx1.freebsd.org (Postfix) with ESMTP id 872A08FC1A for ; Sat, 7 Feb 2009 22:04:32 +0000 (UTC) (envelope-from morganw@chemikals.org) Received: from shop.chemikals.org ([75.182.5.141]) by cdptpa-omta04.mail.rr.com with ESMTP id <20090207220431.IFUK23506.cdptpa-omta04.mail.rr.com@shop.chemikals.org>; Sat, 7 Feb 2009 22:04:31 +0000 Received: from localhost (morganw@localhost [127.0.0.1]) by shop.chemikals.org (8.14.3/8.14.3) with ESMTP id n17M4UXY004808; Sat, 7 Feb 2009 17:04:30 -0500 (EST) (envelope-from morganw@chemikals.org) Date: Sat, 7 Feb 2009 17:04:30 -0500 (EST) From: Wesley Morgan To: Dan Cojocar In-Reply-To: Message-ID: References: User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-fs@freebsd.org Subject: Re: zfs replace disk has failed X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 07 Feb 2009 22:04:33 -0000 On Tue, 3 Feb 2009, Dan Cojocar wrote: > Hello all, > In a mirror(ad1,ad2) configuration one of my disk(ad1) had failed, > after replacing the failed disk with a new one using: > zpool replace tank ad1 > I have noticed that the replace is taking too long and that the system > is not responding, after restart the new disk was not recognized any > more in bios :(, I have tested also in another box and the disk was > not recognized there too. > I have installed a new one on the same location (ad1 I think). Then > the zpool status has reported something like this (this is from memory > because I have made many changes back then, I don't remember exactly > if the online disk was ad1 or ad2): > > zpool status > pool: tank > state: DEGRADED > scrub: none requested > config: > > NAME STATE READ WRITE CKSUM > tank DEGRADED 0 0 0 > mirror DEGRADED 0 0 0 > replacing UNAVAIL 0 387 0 > insufficient replicas > 10193841952954445329 REMOVED 0 0 0 was /dev/ad1/old > 9318348042598806923 FAULTED 0 0 0 was /dev/ad1 > ad2 ONLINE 0 0 0 > At this stage I was thinking that if I will attach the new disk (ad1) > to the mirror I will get sufficient replicas to detach > 9318348042598806923 (this one was the disk that has failed the second > time), so I did an attach, after the resilvering process has completed > with success, I had: > zpool status > pool: tank > state: DEGRADED > scrub: none requested > config: > > NAME STATE READ WRITE CKSUM > tank DEGRADED 0 0 0 > mirror DEGRADED 0 0 0 > replacing UNAVAIL 0 387 0 > insufficient replicas > 10193841952954445329 REMOVED 0 0 0 was /dev/ad1/old > 9318348042598806923 FAULTED 0 0 0 was /dev/ad1 > ad2 ONLINE 0 0 0 > ad1 ONLINE 0 0 0 > And I'm not able to detach 9318348042598806923 :(, and another bad > news is that if I try to access something under /tank the operation is > hanging, eg: if I do a ls /tank is freezing and if I do in another > console: zpool status which was working before ls, now it's freezing > too. > What should I do next? > Thanks, > Dan ZFS seems to fall over on itself if a disk replacement is interrupted and the replacement drive goes missing. By attaching the disk, you now have a 3-way mirror. The two possibilties for you would be to roll the array back to a previous txg, which I'm not at all sure would work, or to create a fake device the same size as the array devices and put a label on it that emulates the missing device, and you can then cancel the replacement. Once the replacement is cancelled, you should be able to remove the nonexistent device. Note, that the labels are all checksummed with sha256 so it's not a simple hex edit (unless you can calculate checksums by hand also!). If you send me the first 512k of either ad1 or ad2 (off-list of course), I can alter the labels to be the missing guids, and you can use md devices and sparse files to fool zpool. -- This .signature sanitized for your protection