From owner-freebsd-bugs@FreeBSD.ORG Tue Dec 26 12:30:14 2006 Return-Path: X-Original-To: freebsd-bugs@hub.freebsd.org Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 4247E16A412 for ; Tue, 26 Dec 2006 12:30:14 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [69.147.83.40]) by mx1.freebsd.org (Postfix) with ESMTP id 1FB1C13C48C for ; Tue, 26 Dec 2006 12:30:14 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.13.4/8.13.4) with ESMTP id kBQCUDP2092253 for ; Tue, 26 Dec 2006 12:30:13 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.13.4/8.13.4/Submit) id kBQCUDiV092252; Tue, 26 Dec 2006 12:30:13 GMT (envelope-from gnats) Resent-Date: Tue, 26 Dec 2006 12:30:13 GMT Resent-Message-Id: <200612261230.kBQCUDiV092252@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Geoffrey Giesemann Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0CC1916A407 for ; Tue, 26 Dec 2006 12:21:30 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (www.freebsd.org [69.147.83.33]) by mx1.freebsd.org (Postfix) with ESMTP id F0B8A13C474 for ; Tue, 26 Dec 2006 12:21:29 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (localhost [127.0.0.1]) by www.freebsd.org (8.13.1/8.13.1) with ESMTP id kBQCLTvA033909 for ; Tue, 26 Dec 2006 12:21:29 GMT (envelope-from nobody@www.freebsd.org) Received: (from nobody@localhost) by www.freebsd.org (8.13.1/8.13.1/Submit) id kBQCLT91033908; Tue, 26 Dec 2006 12:21:29 GMT (envelope-from nobody) Message-Id: <200612261221.kBQCLT91033908@www.freebsd.org> Date: Tue, 26 Dec 2006 12:21:29 GMT From: Geoffrey Giesemann To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.0 Cc: Subject: kern/107206: Background fsck causes kernel panic X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Dec 2006 12:30:14 -0000 >Number: 107206 >Category: kern >Synopsis: Background fsck causes kernel panic >Confidential: no >Severity: critical >Priority: low >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Tue Dec 26 12:30:13 GMT 2006 >Closed-Date: >Last-Modified: >Originator: Geoffrey Giesemann >Release: RELENG_6 >Organization: >Environment: FreeBSD iddqd.local 6.2-PRERELEASE FreeBSD 6.2-PRERELEASE #0: Sun Dec 17 02:33:32 EST 2006 geoffwa@iddqd.local:/usr/obj/usr/src/sys/IDDQD i386 >Description: I'm running a dual P3 machine with an SMP kernel, with all the hard drives hanging off a single Areca ARC-1120 (arcmsr). I migrated to a new larger RAID5 volume on the machine recently. Trying to do this with snapshots caused the odd (difficult-to-reproduce) kernel panic (I used dd instead). To speed up the migration I enabled write-caching to the new volumes. Several days later, writing files to the array via Samba causes a kernel panic. Following this, the system reboots and proceeds to crash in the background fsck. This occurred twice before I booted to a 6.1 install cd, ran fsck in the foreground (which worked fine), and got everything running again. I had savecore turned on after the snapshot problems, so I now have three large coredumps. # kgdb /usr/obj/usr/src/sys/IDDQD/kernel.debug /var/crash/vmcore.0 [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd". Unread portion of the kernel message buffer: panic: softdep_setup_freeblocks: inode busy cpuid = 0 Uptime: 4h58m59s Dumping 1023 MB (2 chunks) chunk 0: 1MB (159 pages) ... ok chunk 1: 1023MB (261872 pages) 1007 991 975 959 943 927 911 895 879 863 847 831 815 799 783 767 751 735 719 703 687 671 655 639 623 607 591 575 559 543 527 511 495 479 463 447 431 415 399 383 367 351 335 319 303 287 271 255 239 223 207 191 175 159 143 127 111 95 79 63 47 31 15 #0 doadump () at pcpu.h:165 165 __asm __volatile("movl %%fs:0,%0" : "=r" (td)); (kgdb) where #0 doadump () at pcpu.h:165 #1 0xc061e541 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409 #2 0xc061e934 in panic (fmt=0xc0905fc5 "softdep_setup_freeblocks: inode busy") at /usr/src/sys/kern/kern_shutdown.c:565 #3 0xc07bb7f9 in softdep_setup_freeblocks (ip=0xc6002c60, length=Unhandled dwarf expression opcode 0x93 ) at /usr/src/sys/ufs/ffs/ffs_softdep.c:2271 #4 0xc07ae85d in ffs_truncate (vp=0xc5fff000, length=0, flags=3072, cred=0x0, td=0xc531b180) at /usr/src/sys/ufs/ffs/ffs_inode.c:278 #5 0xc07d0abe in ufs_inactive (ap=0x0) at /usr/src/sys/ufs/ufs/ufs_inode.c:126 #6 0xc08b364e in VOP_INACTIVE_APV (vop=0x0, a=0x0) at vnode_if.c:1535 #7 0xc068faf2 in vinactive (vp=0xc5fff000, td=0x0) at vnode_if.h:795 #8 0xc068f81c in vput (vp=0xc5fff000) at /usr/src/sys/kern/vfs_subr.c:2158 #9 0xc0697539 in kern_unlink (td=0xc531b180, path=0xbfbfe4b0
, pathseg=UIO_USERSPACE) at /usr/src/sys/kern/vfs_syscalls.c:1722 #10 0xc0697322 in unlink (td=0x0, uap=0x0) at /usr/src/sys/kern/vfs_syscalls.c:1658 #11 0xc089d6f0 in syscall (frame= {tf_fs = -1078001605, tf_es = 135528507, tf_ds = -1078001605, tf_edi = 1, tf_esi = 135534336, tf_ebp = -1077942072, tf_isp = -412762780, tf_ebx = 135461640, tf_edx = 0, tf_ecx = 5, tf_eax = 10, tf_trapno = 0, tf_err = 2, tf_eip = 674363207, tf_cs = 51, tf_eflags = 642, tf_esp = -1077944196, tf_ss = 59}) at /usr/src/sys/i386/i386/trap.c:983 #12 0xc088501f in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:200 #13 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) # kgdb /usr/obj/usr/src/sys/IDDQD/kernel.debug /var/crash/vmcore.1 [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd". Unread portion of the kernel message buffer: g_vfs_done():ufs/var[WRITE(offset=1355972608, length=16384)]error = 5 panic: initiate_write_inodeblock_ufs2: already started cpuid = 1 Uptime: 4m18s Dumping 1023 MB (2 chunks) chunk 0: 1MB (159 pages) ... ok chunk 1: 1023MB (261872 pages) 1007 991 975 959 943 927 911 895 879 863 847 831 815 799 783 767 751 735 719 703 687 671 655 639 623 607 591 575 559 543 527 511 495 479 463 447 431 415 399 383 367 351 335 319 303 287 271 255 239 223 207 191 175 159 143 127 111 95 79 63 47 31 15 #0 doadump () at pcpu.h:165 165 __asm __volatile("movl %%fs:0,%0" : "=r" (td)); (kgdb) where #0 doadump () at pcpu.h:165 #1 0xc061e541 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409 #2 0xc061e934 in panic (fmt=0xc090659d "initiate_write_inodeblock_ufs2: already started") at /usr/src/sys/kern/kern_shutdown.c:565 #3 0xc07bf8ff in initiate_write_inodeblock_ufs2 (inodedep=0xc550ab80, bp=0x0) at /usr/src/sys/ufs/ffs/ffs_softdep.c:4022 #4 0xc07beed0 in softdep_disk_io_initiation (bp=0xd8d1d488) at /usr/src/sys/ufs/ffs/ffs_softdep.c:3757 #5 0xc07c9e9b in ffs_geom_strategy (bo=0xc4f2f3f0, bp=0xd8d1d488) at buf.h:433 #6 0xc0677cc9 in bufwrite (bp=0xd8d1d488) at buf.h:426 #7 0xc07c9cf8 in ffs_bufwrite (bp=0xd8d1d488) at /usr/src/sys/ufs/ffs/ffs_vfsops.c:1740 #8 0xc07ae167 in ffs_update (vp=0xc51bc440, waitfor=1) at buf.h:410 #9 0xc07ca480 in ffs_syncvnode (vp=0xc51bc440, waitfor=1) at /usr/src/sys/ufs/ffs/ffs_vnops.c:330 #10 0xc07ae760 in ffs_truncate (vp=0xc51bc440, length=8192, flags=2048, cred=0xc4afe780, td=0xc4d2d780) at /usr/src/sys/ufs/ffs/ffs_inode.c:268 #11 0xc07d5b4d in ufs_setattr (ap=0x0) at /usr/src/sys/ufs/ufs/ufs_vnops.c:565 #12 0xc08b2c2e in VOP_SETATTR_APV (vop=0x0, a=0xe5346c14) at vnode_if.c:588 #13 0xc069a135 in ftruncate (td=0xc4d2d780, uap=0xe5346d04) at vnode_if.h:314 #14 0xc089d6f0 in syscall (frame= {tf_fs = 59, tf_es = 59, tf_ds = 59, tf_edi = 7496, tf_esi = 135566592, tf_ebp = -1077944920, tf_isp = -449548956, tf_ebx = 674493184, tf_edx = 0, tf_ecx = 246, tf_eax = 198, tf_trapno = 12, tf_err = 2, tf_eip = 674431511, tf_cs = 51, tf_eflags = 582, tf_esp = -1077944964, tf_ss = 59}) at /usr/src/sys/i386/i386/trap.c:983 #15 0xc088501f in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:200 #16 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) # kgdb /usr/obj/usr/src/sys/IDDQD/kernel.debug /var/crash/vmcore.2 [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd". Unread portion of the kernel message buffer: panic: handle_written_filepage: attached cpuid = 0 Uptime: 5m42s Dumping 1023 MB (2 chunks) chunk 0: 1MB (159 pages) ... ok chunk 1: 1023MB (261872 pages) 1007 991 975 959 943 927 911 895 879 863 847 831 815 799 783 767 751 735 719 703 687 671 655 639 623 607 591 575 559 543 527 511 495 479 463 447 431 415 399 383 367 351 335 319 303 287 271 255 239 223 207 191 175 159 143 127 111 95 79 63 47 31 15 #0 doadump () at pcpu.h:165 165 __asm __volatile("movl %%fs:0,%0" : "=r" (td)); (kgdb) where #0 doadump () at pcpu.h:165 #1 0xc061e541 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409 #2 0xc061e934 in panic (fmt=0xc09069a0 "handle_written_filepage: attached") at /usr/src/sys/kern/kern_shutdown.c:565 #3 0xc07c0ff5 in handle_written_filepage (pagedep=0xc52bb900, bp=0xd8be3378) at /usr/src/sys/ufs/ffs/ffs_softdep.c:4772 #4 0xc07c0246 in softdep_disk_write_complete (bp=0xd8be3378) at /usr/src/sys/ufs/ffs/ffs_softdep.c:4244 #5 0xc067cec2 in bufdone (bp=0xd8be3378) at buf.h:440 #6 0xc05d57ad in g_vfs_done (bip=0x0) at /usr/src/sys/geom/geom_vfs.c:86 #7 0xc067cae9 in biodone (bp=0xc52b6738) at /usr/src/sys/kern/vfs_bio.c:2911 #8 0xc05d2cc3 in g_io_schedule_up (tp=0xc4b02000) at /usr/src/sys/geom/geom_io.c:490 #9 0xc05d2fe8 in g_up_procbody () at /usr/src/sys/geom/geom_kern.c:95 #10 0xc05fff90 in fork_exit (callout=0xc05d2f30 , arg=0x0, frame=0x0) at /usr/src/sys/kern/kern_fork.c:821 #11 0xc088502c in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:208 I'm willing to dick around with the box a bit to reproduce the problem, but I'd like to avoid losing any data in the process. >How-To-Repeat: >Fix: >Release-Note: >Audit-Trail: >Unformatted: