From owner-freebsd-fs@FreeBSD.ORG Sat Feb 18 15:48:24 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1A4D8106564A for ; Sat, 18 Feb 2012 15:48:24 +0000 (UTC) (envelope-from shuey@fmepnet.org) Received: from mail-vx0-f182.google.com (mail-vx0-f182.google.com [209.85.220.182]) by mx1.freebsd.org (Postfix) with ESMTP id C40A88FC08 for ; Sat, 18 Feb 2012 15:48:23 +0000 (UTC) Received: by vcmm1 with SMTP id m1so4170808vcm.13 for ; Sat, 18 Feb 2012 07:48:23 -0800 (PST) Received-SPF: pass (google.com: domain of shuey@fmepnet.org designates 10.52.88.235 as permitted sender) client-ip=10.52.88.235; Authentication-Results: mr.google.com; spf=pass (google.com: domain of shuey@fmepnet.org designates 10.52.88.235 as permitted sender) smtp.mail=shuey@fmepnet.org Received: from mr.google.com ([10.52.88.235]) by 10.52.88.235 with SMTP id bj11mr6473722vdb.119.1329580103207 (num_hops = 1); Sat, 18 Feb 2012 07:48:23 -0800 (PST) MIME-Version: 1.0 Received: by 10.52.88.235 with SMTP id bj11mr5186686vdb.119.1329578716693; Sat, 18 Feb 2012 07:25:16 -0800 (PST) Received: by 10.220.64.141 with HTTP; Sat, 18 Feb 2012 07:25:16 -0800 (PST) X-Originating-IP: [98.223.59.225] Date: Sat, 18 Feb 2012 10:25:16 -0500 Message-ID: From: Michael Shuey To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Gm-Message-State: ALoCoQkSGKb5aL3H15tHyD4qEkxs5m24D6ednPuWbHwNfjsFg0EZdwOrb6UNV0kBNX4VSM8MpkgO Subject: ZFS size reduced, 100% full, on fbsd9 upgrade X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 18 Feb 2012 15:48:24 -0000 I'm upgrading a server from 8.2p6 to 9.0-RELEASE, and I've tried both make in the source tree and freebsd-update and I get the same strange result. As soon as I boot to the fbsd9 kernel, even booting into single-user mode, the pool's size is greatly reduced. All filesystems show 100% full (0 bytes free space), nothing can be written to the pool (probably a side-effect of being 100% full), and dmesg shows several of "Solaris: WARNING: metaslab_free_dva(): bad DVA 0:5978620460544" warnings (with different numbers). Switching kernels back to the 8.2p6 kernel restores things to normal, but I'd really like to finish my fbsd9 upgrade. The system is a 64-bit Intel box with 4 GB of memory, and 8 disks in a raidz2 pool called "pool". It's booted to the 8.2p6 kernel now, and scrubbing the pool, but last time I did this (roughly a week ago) it was fine. / is a gmirror, but /usr, /tmp, and /var all come from the pool. Normally, the pool has 1.2 TB of free space, and is version 15 (zfs version 4). Some disks are WD drives, with 4k native sectors, but some time ago I rebuilt the pool to use a native 4k sector size (ashift=12). Over time, I've been slowly replacing disks (1 at a time) to increase the free space in the pool. Also, the system experienced severe failure recently; the power supply blew, and took out the memory (and presumably motherboard). I replaced these last week with known-good board/memory/processor/PS, and it's been running fine since. Any suggestions? Is it possible I've got some nasty pool corruption going on - and if so, how do I go about fixing it? Any advice would be appreciated. This is a backup server, so I could rebuild its contents from the primary, but I'd rather fix it if possible (since I want to do a fbsd9 upgrade on the primary next).