From owner-freebsd-current@FreeBSD.ORG Sat Jun 20 08:23:03 2009 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5F3041065672 for ; Sat, 20 Jun 2009 08:23:03 +0000 (UTC) (envelope-from mat.macy@gmail.com) Received: from an-out-0708.google.com (an-out-0708.google.com [209.85.132.249]) by mx1.freebsd.org (Postfix) with ESMTP id 146548FC0C for ; Sat, 20 Jun 2009 08:23:02 +0000 (UTC) (envelope-from mat.macy@gmail.com) Received: by an-out-0708.google.com with SMTP id c3so978994ana.13 for ; Sat, 20 Jun 2009 01:23:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:received:in-reply-to :references:date:x-google-sender-auth:message-id:subject:from:to:cc :content-type:content-transfer-encoding; bh=V3EHx5EgJcY4my/J9xbMQzpsOl4sBRfwqWU5KoxKKCw=; b=f+W/bIF+heGytqcr+g1vovucXhjgzI12DgLWDeg2+1/LB3W1aB9OcX5v/GODAyl8qC KbiQ0WATwgmMdZNPBMjakmmbnzZOV40Zk3w8A5HQxGjD8QVfNPDnoCf9qGDy7Z3BmFd3 v2XoYPUAyLSeCgHga7BJLKIWqD+OnF2ycyIq8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; b=JHEGKVI8+MOnHWJz+iJeEuXsaa2wBJFhSdAsYDb6fUZtZJRhq0tXrlcYGtGYv5QDhM kUydQB1w+N82OIuXqtLCQ62vjkWZzIH8KWs0DLeHeNt+L2KIYwlte6yo5eMeR/hxFwJc NbNY+PwkyLDnZ3awjbi5BT0Rx9OW+GqDIGPVU= MIME-Version: 1.0 Sender: mat.macy@gmail.com Received: by 10.100.108.8 with SMTP id g8mr5071406anc.66.1245486182198; Sat, 20 Jun 2009 01:23:02 -0700 (PDT) In-Reply-To: <72163521-40BF-4764-8B74-5446A88DFBF8@exscape.org> References: <72163521-40BF-4764-8B74-5446A88DFBF8@exscape.org> Date: Sat, 20 Jun 2009 01:23:02 -0700 X-Google-Sender-Auth: 34eaf1e1e0b091e9 Message-ID: <3c1674c90906200123l75b4eb29y57d5b48f89c19a5b@mail.gmail.com> From: Kip Macy To: Thomas Backman Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: FreeBSD current Subject: Re: "New" ZFS crash on FS (pool?) unmount/export X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Jun 2009 08:23:03 -0000 A force unmount causes all vnodes dirty data to be flushed and then the vnodes to be released. A regular unmount will not complete if there are open files referencing it (hence the use of force unmount on shutdown). Pawel had previously disabled forced unmount because of known "issues". I chose to enable it because it guarantees that dirty data is flushed to disk before shutdown. It looks like we may be reclaiming a vnode that has already been freed. I haven't seen this issue myself, so if you can identify more specifics on how to reproduce it would greatly increase the likelihood of my being able to fix it in the near future. Thanks, Kip On Sat, Jun 20, 2009 at 12:11 AM, Thomas Backman wrot= e: > I just ran into this tonight. Not sure exactly what triggered it - the bo= x > stopped responding to pings at 02:07AM and it has a cron backup job using > zfs send/recv at 02:00, so I'm guessing it's related, even though the bac= kup > probably should have finished before then... Hmm. Anyway. > > r194478. > > kernel trap 12 with interrupts disabled > > Fatal trap 12: page fault while in kernel mode > cpuid =3D 0; apic id =3D 00 > fault virtual address =A0 =3D 0x288 > fault code =A0 =A0 =A0 =A0 =A0 =A0 =A0=3D supervisor read data, page not = present > instruction pointer =A0 =A0 =3D 0x20:0xffffffff805a4989 > stack pointer =A0 =A0 =A0 =A0 =A0 =3D 0x28:0xffffff803e8b57e0 > frame pointer =A0 =A0 =A0 =A0 =A0 =3D 0x28:0xffffff803e8b5840 > code segment =A0 =A0 =A0 =A0 =A0 =A0=3D base 0x0, limit 0xfffff, type 0x1= b > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=3D DPL 0, pres 1, long 1,= def32 0, gran 1 > processor eflags =A0 =A0 =A0 =A0=3D resume, IOPL =3D 0 > current process =A0 =A0 =A0 =A0 =3D 57514 (zpool) > panic: from debugger > cpuid =3D 0 > Uptime: 10h22m13s > Physical memory: 2027 MB > > (kgdb) bt > #0 =A0doadump () at pcpu.h:223 > #1 =A00xffffffff8059c409 in boot (howto=3D260) at > /usr/src/sys/kern/kern_shutdown.c:419 > #2 =A00xffffffff8059c85c in panic (fmt=3DVariable "fmt" is not available. > ) at /usr/src/sys/kern/kern_shutdown.c:575 > #3 =A00xffffffff801f1377 in db_panic (addr=3DVariable "addr" is not avail= able. > ) at /usr/src/sys/ddb/db_command.c:478 > #4 =A00xffffffff801f1781 in db_command (last_cmdp=3D0xffffffff80c38620, > cmd_table=3DVariable "cmd_table" is not available. > ) at /usr/src/sys/ddb/db_command.c:445 > #5 =A00xffffffff801f19d0 in db_command_loop () at > /usr/src/sys/ddb/db_command.c:498 > #6 =A00xffffffff801f3969 in db_trap (type=3DVariable "type" is not availa= ble. > ) at /usr/src/sys/ddb/db_main.c:229 > #7 =A00xffffffff805ce465 in kdb_trap (type=3D12, code=3D0, tf=3D0xffffff8= 03e8b5730) > at /usr/src/sys/kern/subr_kdb.c:534 > #8 =A00xffffffff8088715d in trap_fatal (frame=3D0xffffff803e8b5730, eva= =3DVariable > "eva" is not available. > ) at /usr/src/sys/amd64/amd64/trap.c:847 > #9 =A00xffffffff80887fb2 in trap (frame=3D0xffffff803e8b5730) at > /usr/src/sys/amd64/amd64/trap.c:345 > #10 0xffffffff8086e007 in calltrap () at > /usr/src/sys/amd64/amd64/exception.S:223 > #11 0xffffffff805a4989 in _sx_xlock_hard (sx=3D0xffffff0043557d50, > tid=3D18446742975830720512, opts=3DVariable "opts" is not available. > ) > =A0 =A0at /usr/src/sys/kern/kern_sx.c:575 > #12 0xffffffff805a52fe in _sx_xlock (sx=3DVariable "sx" is not available. > ) at sx.h:155 > #13 0xffffffff80fe2995 in zfs_freebsd_reclaim () from /boot/kernel/zfs.ko > #14 0xffffffff808cefca in VOP_RECLAIM_APV (vop=3D0xffffff0043557d38, > a=3D0xffffff0043557d50) at vnode_if.c:1926 > #15 0xffffffff80626f6e in vgonel (vp=3D0xffffff00437a7938) at vnode_if.h:= 830 > #16 0xffffffff8062b528 in vflush (mp=3D0xffffff0060f2a000, rootrefs=3D0, > flags=3D0, td=3D0xffffff0061528000) > =A0 =A0at /usr/src/sys/kern/vfs_subr.c:2450 > #17 0xffffffff80fdd3a8 in zfs_umount () from /boot/kernel/zfs.ko > #18 0xffffffff8062420a in dounmount (mp=3D0xffffff0060f2a000, > flags=3D1626513408, td=3DVariable "td" is not available. > ) > =A0 =A0at /usr/src/sys/kern/vfs_mount.c:1287 > #19 0xffffffff80624975 in unmount (td=3D0xffffff0061528000, > uap=3D0xffffff803e8b5c00) > =A0 =A0at /usr/src/sys/kern/vfs_mount.c:1172 > #20 0xffffffff8088783f in syscall (frame=3D0xffffff803e8b5c90) at > /usr/src/sys/amd64/amd64/trap.c:984 > #21 0xffffffff8086e290 in Xfast_syscall () at > /usr/src/sys/amd64/amd64/exception.S:364 > #22 0x000000080104e49c in ?? () > Previous frame inner to this frame (corrupt stack?) > > BTW, I got a (one) "force unmount is experimental" on the console. On > regular shutdown I usually get one per filesystem, it seems (at least 10) > and this pool should contain exactly as many filesystems as the root pool > since it's a copy of it. On running the backup script manually post-crash= , > though, I didn't get any. > > Also worth noting is that I was running DTrace all night to test its > stability. I'm pretty sure the script was > dtrace -n 'syscall::open:entry { @a[copyinstr(arg0)] =3D count(); }' > > 0 swap was used and 277700 pages (~1084 MB or 50%) RAM was free, accordin= g > to the core.txt. > > Regards, > Thomas > _______________________________________________ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org= " > --=20 When bad men combine, the good must associate; else they will fall one by one, an unpitied sacrifice in a contemptible struggle. Edmund Burke