From owner-freebsd-stable@FreeBSD.ORG Sun Feb 5 23:23:26 2012 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4E30C106566B; Sun, 5 Feb 2012 23:23:26 +0000 (UTC) (envelope-from morgan.s.reed@gmail.com) Received: from mail-pw0-f54.google.com (mail-pw0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id 1D0CC8FC15; Sun, 5 Feb 2012 23:23:25 +0000 (UTC) Received: by pbdv10 with SMTP id v10so5698220pbd.13 for ; Sun, 05 Feb 2012 15:23:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type:content-transfer-encoding; bh=X1nX0Pyu6nRVvzl5VkC+gvSxKSR9lgNxlTEBgVyywHY=; b=Dukq5R4vUC8Uhvb6dNMfBJna3befs3YYF7OUmPSrsnmOHNHT2bCT7RgHgJB/eOW98p JusEdGGGCmEbrTJiGBCrjRIYuOwXMxmBQMTNZttfNRuYAM8knSPes9kQvRyx2amda6+7 S4jSn5p7tV8ZH5o0vEqWeYL19xS0i0B9ejmQk= Received: by 10.68.135.137 with SMTP id ps9mr41821503pbb.40.1328484205452; Sun, 05 Feb 2012 15:23:25 -0800 (PST) MIME-Version: 1.0 Received: by 10.68.65.226 with HTTP; Sun, 5 Feb 2012 15:23:05 -0800 (PST) In-Reply-To: <4F2E65EC.60107@FreeBSD.org> References: <4F2E65EC.60107@FreeBSD.org> From: Morgan Reed Date: Mon, 6 Feb 2012 10:23:05 +1100 Message-ID: To: Andriy Gapon Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-stable@freebsd.org Subject: Re: ZFS panics on pool moved from OpenSolaris X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Feb 2012 23:23:26 -0000 Hi Andriy, Thanks for that, the patch has significantly improved matters, I'm now able to run a find across part of the drive without issue, however I'm still seeing the panics on some directories, stack trace below; panic: avl_find() succeeded inside avl_add() cpuid =3D 0 KDB: stack backtrace: #0 0xc0a4b157 at kdb_backtrace+0x47 #1 0xc0a186b7 at panic+0x117 #2 0xc5a2d7b2 at avl_add+0x52 #3 0xc5ac44e6 at zfs_fuid_table_load+0x1f6 #4 0xc5ac479e at zfs_fuid_init+0x14e #5 0xc5ac4893 at zfs_fuid_find_by_idx+0xc3 #6 0xc5ac48ed at zfs_fuid_map_id+0x2d #7 0xc5ac492f at zfs_groupmember+0x2f #8 0xc5adbdcb at zfs_zaccess_aces_check+0x1db #9 0xc5adc257 at zfs_zaccess+0xb7 #10 0xc5afa7d4 at zfs_freebsd_getattr+0x1f4 #11 0xc0d69322 at VOP_GETATTR_APV+0x42 #12 0xc0ab81c9 at vn_stat+0x79 #13 0xc0aaefdd at kern_statat_vnhook+0xfd #14 0xc0aaf1cc at kern_statat+0x3c #15 0xc0aaf156 at kern_lstat+0x36 #16 0xc0aaf1ff at sys_lstat+0x2f #17 0xc0d49315 at syscall+0x355 Same trace as previously on 9.0. I'm following your advice to Karli and throwing some printfs into zfs_fuid_table_load, I'll advise if I find anything enlightening. Thanks, Morgan On Sun, Feb 5, 2012 at 22:20, Andriy Gapon wrote: > > Please see this thread: > http://lists.freebsd.org/pipermail/freebsd-fs/2011-December/013215.html > It looks like the same issue. > The patch has been committed in head, not sure if it's MFCed. > > on 05/02/2012 06:57 Morgan Reed said the following: >> Hi all, >> >> =A0 =A0 =A0I'm experiencing an issue in migrating my NAS from OpenSolari= s >> over to FreeBSD, I've tried both releng_8_2 and releng_9 I have >> similar issues in both cases. >> >> The pool is a RAID-Z pool comprising 4 1TB drives, it was originally >> created on OpenSolaris (not sure what version, 2010.09 maybe, it was >> one of the last ones prior to the Oracle acquisition), pool was a V14 >> pool, initially I built a FreeBSD-8.2 system to migrate the pool to, >> migrated it over OK, upgraded it from V14 to V15, but later testing >> revealed something wasn't happy, when listing certain directories (and >> even doing an ls -la at the root of the pool) resulted in a kernel >> panic (Mostly GENERIC kernel, rebuilt with KVA_PAGES 512 but other >> than that stock); >> >> panic: avl_find() =A0succeeded inside avl_add() >> cpuid =3D 0 >> KDB: stack backtrace: >> #0 0x808e0d07 at kdb_backtrace+0x47 >> #1 0x808b1dc7 at panic+0x117 >> #2 0x862e6602 at avl_add+0x52 >> #3 0x8635c136 at zfs_fuid_table_load+0x1f6 >> #4 0x8635c3ee at zfs_fuid_init+0x14e >> #5 0x8635c4d7 at zfs_fuid_find_by_idx+0xb7 >> #6 0x8635c52d at zfs_fuid_map_id+0x2d >> #7 0x8635d56f at zfs_groupmember+0x2f >> #8 0x8636df0b at zfs_zaccess_aces_check+0x1db >> #9 0x8636377 at zfs_zaccess+0x57 >> #10 0x8636d6fb at zfs_zaccess_rwx+0x3b >> #11 0x86385f61 at zfs_freebsd_access+0xf1 >> #12 0x80c02ea2 at VOP_ACCESS_APV+0x42 >> #13 0x809457cf at change_dir+0x5f >> #14 0x809467b1 at kern_chdir+0x81 >> #15 0x80946a22 at chdir+0x22 >> #16 0x808eca39 at syscallenter+0x329 >> #17 0x80be4e14 at syscall+0x34 >> >> Looks like something in the permissions structure was causing grief, >> tried running a scrub across the pool, didn't resolve the issue. >> >> After spending some time fighting with it I decided that it wasn't >> worth the effort, and I upgraded to FreeBSD-9.0 to see if that would >> assist (I normally avoid x.0 releases), once again pool imported fine, >> however I was still seeing similar panics, ran a scrub across the >> pool, still not happy, also upgraded the pool to v28 tried again, when >> that failed I scrubbed again but still no joy. >> >> As a matter of interest I booted an OpenIndiana live CD and tried >> copying the directories contents to another location, I am now able to >> list the directories. However there are still issues. >> >> The issue seems to have shifted slightly, stack trace from a recent >> panic is below (GENERIC kernel on 9.0-RELEASE); >> >> panic: avl_find() =A0succeeded inside avl_add() >> cpuid =3D 0 >> KDB: stack backtrace: >> #0 0xc0a4b157 at kdb_backtrace+0x47 >> #1 0xc0a186b7 at panic+0x117 >> #2 0xc5a2d7b2 at avl_add+0x52 >> #3 0xc5ac44e6 at zfs_fuid_table_load+0x1f6 >> #4 0xc5ac479e at zfs_fuid_init+0x14e >> #5 0xc5ac4893 at zfs_fuid_find_by_idx+0xc3 >> #6 0xc5ac48ed at zfs_fuid_map_id+0x2d >> #7 0xc5ac492f at zfs_groupmember+0x2f >> #8 0xc5adbdcb at zfs_zaccess_aces_check+0x1db >> #9 0xc5adc257 at zfs_zaccess+0xb7 >> #10 0xc5afa7d4 at zfs_freebsd_getattr+0x1f4 >> #11 0xc0d69322 at VOP_GETATTR_APV+0x42 >> #12 0xc0ab81c9 at vn_stat+0x79 >> #13 0xc0aaefdd at kern_statat_vnhook+0xfd >> #14 0xc0aaf1cc at kern_statat+0x3c >> #15 0xc0aaf156 at kern_lstat+0x36 >> #16 0xc0aaf1ff at sys_lstat+0x2f >> #17 0xc0d49315 at syscall+0x355 >> >> This time it appears to be related to some extended attribute(s), I >> can do an ls on one of the directories in question but an ls -la >> causes a panic, so it would seem that it's some attribute which is >> only shown in the long form of the ls output that is causing the >> issue. >> >> I've done some digging around via the magic of google and this seems >> to be a fairly common issue, but I've not found a solution for it >> (barring copying the data off, recreating the pool and restoring the >> data, I'd like to avoid this if at all possible. >> >> If I could determine what the problematic attribute was and a means to >> strip it (be that from FreeBSD or from an OpenIndiana liveCD) I think >> that will get me back up and running. >> >> If anybody can provide some suggestions as to what I may be able to do >> to resolve this issue in situ I would be very grateful. > > > > -- > Andriy Gapon --=20 "They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety." -- Benjamin Franklin, 1759