From owner-freebsd-fs@FreeBSD.ORG Thu Jul 30 12:51:49 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7CFCF106564A; Thu, 30 Jul 2009 12:51:49 +0000 (UTC) (envelope-from serenity@exscape.org) Received: from ch-smtp01.sth.basefarm.net (ch-smtp01.sth.basefarm.net [80.76.149.212]) by mx1.freebsd.org (Postfix) with ESMTP id F3DD78FC1B; Thu, 30 Jul 2009 12:51:48 +0000 (UTC) (envelope-from serenity@exscape.org) Received: from c83-253-252-234.bredband.comhem.se ([83.253.252.234]:55335 helo=mx.exscape.org) by ch-smtp01.sth.basefarm.net with esmtp (Exim 4.68) (envelope-from ) id 1MWV6Y-0002xN-5V; Thu, 30 Jul 2009 14:51:32 +0200 Received: from [192.168.1.5] (macbookpro [192.168.1.5]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by mx.exscape.org (Postfix) with ESMTPSA id E98681734E3; Thu, 30 Jul 2009 14:51:30 +0200 (CEST) Message-Id: <71A038EC-02B1-4606-96C2-5E84BE80F005@exscape.org> From: Thomas Backman To: Andriy Gapon In-Reply-To: <4A718E03.6030909@freebsd.org> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v935.3) Date: Thu, 30 Jul 2009 14:51:28 +0200 References: <20090727072503.GA52309@jpru.ffm.jpru.de> <4A6E06E6.9030300@mail.zedat.fu-berlin.de> <4A6EC9E2.5070200@icyb.net.ua> <20090729084723.GD1586@garage.freebsd.pl> <4A7030B6.8010205@icyb.net.ua> <97D5950F-4E4D-4446-AC22-92679135868D@exscape.org> <4A7048A9.4020507@icyb.net.ua> <52AA86CB-6C06-4370-BA73-CE19175467D0@exscape.org> <4A705299.8060504@icyb.net.ua> <4A7054E1.5060402@icyb.net.ua> <5918824D-A67C-43E6-8685-7B72A52B9CAE@exscape.org> <4A705E50.8070307@icyb.net.ua> <4A70728C.7020004@freebsd.org> <6D47A34B-0753-4CED-BF3D-C505B37748FC@exscape.org> <4A708455.5070304@freebsd.org> <86983A55-E5C4-4C04-A4C7-0AE9A9EE37A3@exscape.org> <4A718E03.6030909@freebsd.org> X-Mailer: Apple Mail (2.935.3) X-Originating-IP: 83.253.252.234 X-Scan-Result: No virus found in message 1MWV6Y-0002xN-5V. X-Scan-Signature: ch-smtp01.sth.basefarm.net 1MWV6Y-0002xN-5V b5e743d7e5f9cfd0cf05591b9c5c7e44 Cc: freebsd-fs@freebsd.org, FreeBSD current , Pawel Jakub Dawidek Subject: Re: zfs: Fatal trap 12: page fault while in kernel mode X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Jul 2009 12:51:49 -0000 On Jul 30, 2009, at 14:11, Andriy Gapon wrote: > on 29/07/2009 21:04 Thomas Backman said the following: >> Thanks for your work :) >> However, bad news: it didn't help. It *might* have gotten us further, >> though, because the DDB backtrace now looks like this: >> >> _sx_xlock_hard() >> _sx_xlock() >> zfs_znode_free() >> zfs_freebsd_inactive() >> VOP_INACTIVE_APV() >> vinactive() >> vput() >> dounmount() >> unmount() >> syscall() >> XFast_syscall() >> > > Oh my bad. I missed the fact that recycle would do zfs_znode_free, > so it seems > like zfs_znode_free was called twice on the same znode. > Could you please try replacing > zfs_znode_free(zp); > with > vrecycle(vp, curthread); > in the same block (instead of adding the latter before the former). > Sorry, if this looks like shooting in the dark - because this is > what it is. I am > not familiar with the code and it's hard to follow all possibilities > without good > understanding. New panic. :( Damnit! I think I'm using svn + http://people.freebsd.org/~pjd/patches/zfs_vnops.c.2.patch + your change, now... Unread portion of the kernel message buffer: GEOM_GATE: Device ggate1482 destroyed. panic: solaris assert: zp != ((void *)0), file: /usr/src/sys/modules/ zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c, line: 4359 cpuid = 0 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a panic() at panic+0x182 zfs_freebsd_reclaim() at zfs_freebsd_reclaim+0x244 VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0x4a vgonel() at vgonel+0x12e vrecycle() at vrecycle+0x7d zfs_freebsd_inactive() at zfs_freebsd_inactive+0x1a VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0x4a vinactive() at vinactive+0x6a vput() at vput+0x1c6 dounmount() at dounmount+0x4af unmount() at unmount+0x3c8 syscall() at syscall+0x28f Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (22, FreeBSD ELF64, unmount), rip = 0x80104e9ec, rsp = 0x7fffffffaa98, rbp = 0x801223300 --- KDB: enter: panic 0xffffff00452971d8: tag zfs, type VDIR usecount 0, writecount 0, refcount 1 mountedhere 0 flags (VI_DOOMED|VI_DOINGINACT) lock type zfs: EXCL by thread 0xffffff0019ff6000 (pid 1425) panic: from debugger ... #11 0xffffffff8033a9cb in panic (fmt=Variable "fmt" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:558 #12 0xffffffff80b11124 in zfs_freebsd_reclaim () from /boot/kernel/ zfs.ko #13 0xffffffff805c5c2a in VOP_RECLAIM_APV (vop=0x0, a=0xffffff803eaf8930) at vnode_if.c:1926 #14 0xffffffff803c839e in vgonel (vp=0xffffff00452971d8) at vnode_if.h: 830 #15 0xffffffff803ca7ad in vrecycle (vp=0xffffff00452971d8, td=Variable "td" is not available. ) at /usr/src/sys/kern/vfs_subr.c:2504 #16 0xffffffff80b10aaa in zfs_freebsd_inactive () from /boot/kernel/ zfs.ko #17 0xffffffff805c5b5a in VOP_INACTIVE_APV (vop=0xffffffff80b882a0, a=0xffffff803eaf89f0) at vnode_if.c:1863 #18 0xffffffff803c6aaa in vinactive (vp=0xffffff00452971d8, td=0xffffff0019ff6000) at vnode_if.h:807 #19 0xffffffff803cbf26 in vput (vp=0xffffff00452971d8) at /usr/src/sys/kern/vfs_subr.c:2257 #20 0xffffffff803c57ef in dounmount (mp=0xffffff0001d058d0, flags=0, td=Variable "td" is not available. ) at /usr/src/sys/kern/vfs_mount.c:1333 #21 0xffffffff803c5df8 in unmount (td=0xffffff0019ff6000, uap=0xffffff803eaf8bf0) at /usr/src/sys/kern/vfs_mount.c:1174 #22 0xffffffff805980bf in syscall (frame=0xffffff803eaf8c80) at /usr/src/sys/amd64/amd64/trap.c:984 #23 0xffffffff8057e2c1 in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:373 #24 0x000000080104e9ec in ?? () FWIW: Line 4359 (panic line): zfs_freebsd_reclaim(ap) ... { vnode_t *vp = ap->a_vp; znode_t *zp = VTOZ(vp); ASSERT(ap != NULL); // added by me ASSERT(vp != NULL); // added by me >>> ASSERT(zp != NULL); // line 4359 --------------- zfs_inactive(vnode_t *vp, cred_t *cr, caller_context_t *ct) { znode_t *zp = VTOZ(vp); zfsvfs_t *zfsvfs = zp->z_zfsvfs; int error; rw_enter(&zfsvfs->z_teardown_inactive_lock, RW_READER); if (zp->z_dbuf == NULL) { /* * The fs has been unmounted, or we did a * suspend/resume and this file no longer exists. */ VI_LOCK(vp); vp->v_count = 0; /* count arrives as 1 */ vp->v_data = NULL; VI_UNLOCK(vp); rw_exit(&zfsvfs->z_teardown_inactive_lock); ZTOV(zp) = NULL; vrecycle(vp, curthread); // zfs_znode_free(zp); return; } Regards, Thomas PS. ... and thanks again for working to solve this. :)