From owner-freebsd-current@FreeBSD.ORG Tue Aug 12 12:04:11 2003 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E640937B401 for ; Tue, 12 Aug 2003 12:04:11 -0700 (PDT) Received: from vimes.aminor.no (vimes.aminor.no [213.187.177.2]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5A48543F3F for ; Tue, 12 Aug 2003 12:04:09 -0700 (PDT) (envelope-from eivind@aminor.no) Received: from [192.168.0.2] (rincewind.eivind [192.168.0.2]) by vimes.aminor.no (Postfix) with ESMTP id 72D3978E16 for ; Tue, 12 Aug 2003 21:04:04 +0200 (CEST) Date: Tue, 12 Aug 2003 21:05:42 +0200 From: Eivind Olsen To: freebsd-current@freebsd.org Message-ID: <10206765.1060722342@[192.168.0.2]> X-Mailer: Mulberry/3.0.3 (Win32) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline Subject: Crash in g_dev_strategy / CURRENT as of yesterday. X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Aug 2003 19:04:12 -0000 Hello. Some of you might have seen my previous mailings regarding crashes in g_dev_strategy under FreeBSD 5.1 (RELENG_5_1). I have now upgraded to CURRENT (cvsupped, compiled and installed yesterday) and I still see similar crashes (but not identical crashes, I can't see any mention of Vinum here). My kernel is basically GENERIC with a few small changes: eivind@vimes:/root/kernels > diff VIMES /usr/src/sys/i386/conf/GENERIC 25c25 < ident VIMES --- > ident GENERIC 63,66c63,66 < #options INVARIANTS #Enable calls of extra sanity checking < #options INVARIANT_SUPPORT #Extra sanity checks of internal structures, required by INVARIANTS < #options WITNESS #Enable checks to detect deadlocks and cycles < #options WITNESS_SKIPSPIN #Don't run witness on spinlocks for speed --- > options INVARIANTS #Enable calls of extra sanity checking > options INVARIANT_SUPPORT #Extra sanity checks of internal structures, required by INVARIANTS > options WITNESS #Enable checks to detect deadlocks and cycles > options WITNESS_SKIPSPIN #Don't run witness on spinlocks for speed 266,271d265 < < # This option is a subset of the IPFILTER option. < options IPFILTER #ipfilter support < options IPFILTER_LOG #ipfilter logging < options IPFILTER_DEFAULT_BLOCK #block all packets by default < eivind@vimes:/root/kernels > Here is what I saw in DDB (manually typed in here). I'm no kernel hacker so I'm not sure if I should have done anything else to dig up more information: Fatal trap 12: page fault while in kernel mode fault virtual address = 0x14 fault code = supervisor write, page not present instruction pointer = 0x8:0xc0308af9 stack pointer = 0x10:0xcaf239a0 frame pointer = 0x10:0xcaf239d4 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 5 (pagedaemon) kernel: type 12 trap, code=0 Stopped at g_dev_strategy+0x29: movl %eax,0x14(%ebx) db> show reg cs 0x8 ds 0xc0350010 sysctl_sysctl_name+0x80 es 0xcaf20010 fs 0xc0d30018 ss 0x10 eax 0xf96000 ecx 0 edx 0 ebx 0 esp 0xcaf239a0 ebp 0xcaf239d4 esi 0xc5abc9b0 edi 0xc1fa9cc0 eip 0xc0308af9 g_dev_strategy+0x29 efl 0x10206 dr0 0 dr1 0 dr2 0 dr3 0 dr4 0xffff0ff0 dr5 0x400 dr6 0xffff0ff0 dr7 0x400 g_dev_strategy+0x29: movl %eax,0x14(%ebx) db> trace g_dev_strategy(c5abc9b0,0,c0338523,caf23a24,0) at g_dev_strategy+0x29 spec_xstrategy(c2133920,c5abc9b0,caf23a1c,c0305b18,caf23a44) at spec_xstrategy+0x23e spec_specstrategy(caf23a44,caf23a68,c04a13a4,caf23a44,0) at spec_specstrategy+0x1b spec_vnoperate(caf23a44,0,4,caf23a68,c04a0d16) at spec_vnoperate+0x18 swapdev_strategy(c5abc9b0,200,0,40f9a,0) at swapdev_strategy+0x204 swap_pager_putpages(c2c23128,caf23b7c,4,0,caf23ae0) at swap_pager_putpages+0x452 vm_pageout_flush(caf23b7c,4,0,1,c2615690) at vm_pageout_flush+0x18b vm_pageout_clean(c093d228,48,0,caf23c28,0) at vm_pageout_clean+0x2ed vm_pageout_scan(0,c0626aa0,44,c055b4fa,1f4) at vm_pageout_scan+0x7bd vm_pageout(0,caf23d48,0,0,0) at vm_pageout+0x368 fork_exit(c04b6f90,0,caf23d48) at fork_exit+0xb1 fork_trampoline() at fork_trampoline+0x8 -- trap 0x1, eip = 0, esp = 0xcaf23d7c, ebp = 0 --- db> panic panic: from debugger Debugger("panic") Fatal trap 3: breakpoint instruction fault while in kernel mode instruction pointer = 0x8:0xc04e1da4 stack pointer = 0x10:0xcaf23718 frame pointer = 0x10:0xcaf23724 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = IOPL = 0 currnet process = 5 (pagedaemon) Stopped at g_dev_strategy+0x29: movl %eax,0x14(%ebx) db> panic ..and then it's saving a crashdump.. At the time of the crash, the machine had an uptime of approx. 1 day and 4 hours. With RELENG_5_1 the crashes also happened something like once a day. eivind@vimes:~/tmp/debug/CURRENT-2003-08-11 > gdb -k kernel.debug vmcore.1 GNU gdb 5.2.1 (FreeBSD) Copyright 2002 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-undermydesk-freebsd"... panic: from debugger panic messages: --- Syntax error: Unterminated quoted string --- Reading symbols from /usr/obj/usr/src/sys/VIMES/modules/usr/src/sys/modules/vinum/vinum.ko.debug ...done. Loaded symbols for /usr/obj/usr/src/sys/VIMES/modules/usr/src/sys/modules/vinum/vinum.ko.debug Reading symbols from /usr/obj/usr/src/sys/VIMES/modules/usr/src/sys/modules/ipfw/ipfw.ko.debug.. .done. Loaded symbols for /usr/obj/usr/src/sys/VIMES/modules/usr/src/sys/modules/ipfw/ipfw.ko.debug Reading symbols from /boot/kernel/dragon_saver.ko...done. Loaded symbols for /boot/kernel/dragon_saver.ko #0 doadump () at /usr/src/sys/kern/kern_shutdown.c:240 240 dumping++; (kgdb) bt #0 doadump () at /usr/src/sys/kern/kern_shutdown.c:240 #1 0xc03461c0 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:372 #2 0xc03465a8 in panic () at /usr/src/sys/kern/kern_shutdown.c:550 #3 0xc01753b2 in db_panic () at /usr/src/sys/ddb/db_command.c:450 #4 0xc0175312 in db_command (last_cmdp=0xc05eeae0, cmd_table=0x0, aux_cmd_tablep=0xc0573374, aux_cmd_tablep_end=0xc057338c) at /usr/src/sys/ddb/db_command.c:346 #5 0xc0175455 in db_command_loop () at /usr/src/sys/ddb/db_command.c:472 #6 0xc0178465 in db_trap (type=12, code=0) at /usr/src/sys/ddb/db_trap.c:73 #7 0xc04e1aec in kdb_trap (type=12, code=0, regs=0xcaf23960) at /usr/src/sys/i386/i386/db_interface.c:172 #8 0xc04f4466 in trap_fatal (frame=0xcaf23960, eva=0) at /usr/src/sys/i386/i386/trap.c:816 #9 0xc04f4132 in trap_pfault (frame=0xcaf23960, usermode=0, eva=20) at /usr/src/sys/i386/i386/trap.c:735 #10 0xc04f3c65 in trap (frame= {tf_fs = -1059913704, tf_es = -890109936, tf_ds = -1070268400, tf_edi = -1040540480, tf_esi = -978597456, tf_ebp = -890095148, tf_isp = -890095220, tf_ebx = 0, tf_edx = 0, tf_ecx = 0, tf_eax = 16343040, tf_trapno = 12, tf_err = 2, tf_eip = -1070560519, tf_cs = 8, tf_eflags = 66054, tf_esp = -978597456, tf_ss = -1067143852}) at /usr/src/sys/i386/i386/trap.c:420 #11 0xc04e3498 in calltrap () at {standard input}:102 #12 0xc030697e in spec_xstrategy (vp=0xc1fa9cc0, bp=0x0) at /usr/src/sys/fs/specfs/spec_vnops.c:512 #13 0xc03069ab in spec_specstrategy (ap=0x0) at /usr/src/sys/fs/specfs/spec_vnops.c:529 #14 0xc0305b18 in spec_vnoperate (ap=0x0) at /usr/src/sys/fs/specfs/spec_vnops.c:122 #15 0xc04a13a4 in swapdev_strategy (a_bp=0xc5abc9b0) at vnode_if.h:1141 #16 0xc04a0282 in swap_pager_putpages (object=0x0, m=0xcaf23b7c, count=4, sync=0, rtvals=0xcaf23ae0) at /usr/src/sys/vm/swap_pager.c:1326 #17 0xc04b524b in vm_pageout_flush (mc=0xcaf23b7c, count=4, flags=0, is_object_locked=1) at /usr/src/sys/vm/vm_pager.h:145 #18 0xc04b506d in vm_pageout_clean (m=0xc0ad6200) at /usr/src/sys/vm/vm_pageout.c:351 #19 0xc04b64ed in vm_pageout_scan (pass=0) at /usr/src/sys/vm/vm_pageout.c:1015 #20 0xc04b72f8 in vm_pageout () at /usr/src/sys/vm/vm_pageout.c:1509 #21 0xc032ef61 in fork_exit (callout=0xc04b6f90 , arg=0x0, frame=0x0) at /usr/src/sys/kern/kern_fork.c:790 (kgdb) Is this of any help at all to anyone? Suggestions as to what I should try out etc.? -- Regards / Hilsen Eivind Olsen