From owner-freebsd-current@FreeBSD.ORG Fri Aug 21 11:00:38 2009 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7B099106568B; Fri, 21 Aug 2009 11:00:38 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (chello087206049004.chello.pl [87.206.49.4]) by mx1.freebsd.org (Postfix) with ESMTP id 535B58FC52; Fri, 21 Aug 2009 11:00:36 +0000 (UTC) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 2144145C9B; Fri, 21 Aug 2009 13:00:34 +0200 (CEST) Received: from localhost (pjd-w.wheel.pl [10.0.1.1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id 6AA6645684; Fri, 21 Aug 2009 13:00:28 +0200 (CEST) Date: Fri, 21 Aug 2009 13:00:31 +0200 From: Pawel Jakub Dawidek To: Thomas Backman Message-ID: <20090821110031.GB1962@garage.freebsd.pl> References: <7F161876-8DA7-4617-98B6-7CD54C691BC6@exscape.org> <306284EA-C89C-433C-9D33-E6CF44305800@exscape.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="Y7xTucakfITjPcLV" Content-Disposition: inline In-Reply-To: <306284EA-C89C-433C-9D33-E6CF44305800@exscape.org> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 8.0-CURRENT i386 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-5.9 required=4.5 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.0.4 Cc: freebsd-fs@freebsd.org, FreeBSD current Subject: Re: Yet another ZFS recv panic; old but rarely seen X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Aug 2009 11:00:38 -0000 --Y7xTucakfITjPcLV Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Aug 21, 2009 at 11:47:35AM +0200, Thomas Backman wrote: > On Aug 21, 2009, at 08:51, Thomas Backman wrote: >=20 > >Ugh. Bad news again: another zfs send/recv panic during an =20 > >incremental backup. > > > >Unread portion of the kernel message buffer: > >panic: dirtying dbuf obj=3Db213 lvl=3D1 blkid=3D2 but not tx_held > > > >cpuid =3D 0 > >KDB: stack backtrace: > >db_trace_self_wrapper() at db_trace_self_wrapper+0x2a > >panic() at panic+0x182 > >dmu_tx_dirty_buf() at dmu_tx_dirty_buf+0x28f > >dbuf_dirty() at dbuf_dirty+0x69 > >dnode_free_range() at dnode_free_range+0x80d > >dnode_reallocate() at dnode_reallocate+0x131 > >dmu_object_reclaim() at dmu_object_reclaim+0x99 > >dmu_recv_stream() at dmu_recv_stream+0x1446 > >zfs_ioc_recv() at zfs_ioc_recv+0x25a > >zfsdev_ioctl() at zfsdev_ioctl+0x8a > >devfs_ioctl_f() at devfs_ioctl_f+0x77 > >kern_ioctl() at kern_ioctl+0xf6ioctl() at ioctl+0xfd > >syscall() at syscall+0x28f > >Xfast_syscall() at Xfast_syscall+0xe1 > >--- syscall (54, FreeBSD ELF64, ioctl), rip =3D 0x800fe5f7c, rsp =3D =20 > >0x7fffffff8fb8, rbp =3D 0x7fffffff9cf0 --- > >KDB: enter: panic > >panic: from debugger > >cpuid =3D 0 > >Uptime: 4h52m26s > > > >Looks *eerily* similar to this panic fron OpenSolaris:=20 > >http://mail.opensolaris.org/pipermail/zfs-code/2008-September/000694.html > > > >GDB backtrace isn't of that much more use, I guess: > >#11 0xffffffff8036d02b in panic (fmt=3DVariable "fmt" is not available. > >) > > at /usr/src/sys/kern/kern_shutdown.c:562 > >#12 0xffffffff80b4765f in dmu_tx_dirty_buf () from /boot/kernel/zfs.ko > >#13 0xffffffff80b3a519 in dbuf_dirty () from /boot/kernel/zfs.ko > >#14 0xffffffff80b4b68d in dnode_free_range () from /boot/kernel/zfs.ko > >#15 0xffffffff80b4c461 in dnode_reallocate () from /boot/kernel/zfs.ko > >#16 0xffffffff80b42569 in dmu_object_reclaim () from /boot/kernel/=20 > >zfs.ko > >#17 0xffffffff80b421b6 in dmu_recv_stream () from /boot/kernel/zfs.ko > >#18 0xffffffff80ba430a in zfs_ioc_recv () from /boot/kernel/zfs.ko > >#19 0xffffff002ac13d68 in ?? () > >#20 0xffffff002aa6c320 in ?? () > >#21 0xffffff002ae15000 in ?? () > >#22 0xffffff0002891400 in ?? () > >#23 0xffffff00028f2800 in ?? () > >#24 0xffffff00744a1ab8 in ?? () > >... > >#34 0xffffff803e7fc860 in ?? () > >#35 0xffffffff805b699f in uma_zalloc_arg (zone=3D0xffffff00183c6600, > > udata=3D0xffffff00744a1000, flags=3D-128) at /usr/src/sys/vm/=20 > >uma_core.c:1990 > >Previous frame inner to this frame (corrupt stack?) > >(kgdb) > > > >Apparently, I've gotten this once before, at r195910 (+ patches, not =20 > >such which ones at that time), on July 30th. Same DDB backtrace, =20 > >same broken GDB backtrace. > > > >Regards, > >Thomas >=20 > I found some more info mere minutes after posting this (figures; =20 > that's why I prefer media where you can edit your posts!), but had =20 > other things to do. So, here's some more: >=20 > OpenSolaris bug ID: 6754448 (=20 > http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=3D6754448 ) > Fixed in build 108:=20 > http://dlc.sun.com/osol/on/downloads/b108/on-changelog-b108.html > Changelogs are to be found on that page (just search for "6754448", =20 > with a history/diff link on each source file's page. Unfortunately =20 > (unless FreeBSD suffers from both, that is), they apparently fixed two = =20 > bugs in the same batch, making it harder - at least for *me* - to see =20 > what changes relate to *this* panic. > Still, I'm guessing this will help, unless the code is too much out of = =20 > sync with OpenSolaris. > I'm also guessing Pawel already knows waaaaaaay more about their =20 > system than I do (... which is about nothing), so I'll probably shut =20 > up now... ;) Right, the bug is already fixed in OpenSolaris. If you can reproduce the problem, you might try this patch: http://people.freebsd.org/~pjd/patches/dirtying_dbuf.patch --=20 Pawel Jakub Dawidek http://www.wheel.pl pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --Y7xTucakfITjPcLV Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQFKjn5PForvXbEpPzQRAiUbAJ9+4N2km8QzGfVtwzfZIjtmXBdJBwCcDJ8D +kSfuJlTH3Ban7+VTwJAAsM= =RHhz -----END PGP SIGNATURE----- --Y7xTucakfITjPcLV--