From owner-freebsd-bugs@FreeBSD.ORG Fri May 16 20:20:01 2008 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 46FD1106566C for ; Fri, 16 May 2008 20:20:01 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 237E18FC27 for ; Fri, 16 May 2008 20:20:01 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id m4GKK0Q8086669 for ; Fri, 16 May 2008 20:20:00 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id m4GKK09C086668; Fri, 16 May 2008 20:20:00 GMT (envelope-from gnats) Resent-Date: Fri, 16 May 2008 20:20:00 GMT Resent-Message-Id: <200805162020.m4GKK09C086668@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, C Fan Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A022E106564A for ; Fri, 16 May 2008 20:17:42 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21]) by mx1.freebsd.org (Postfix) with ESMTP id 8E4688FC19 for ; Fri, 16 May 2008 20:17:42 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (localhost [127.0.0.1]) by www.freebsd.org (8.14.2/8.14.2) with ESMTP id m4GKGTKE040472 for ; Fri, 16 May 2008 20:16:29 GMT (envelope-from nobody@www.freebsd.org) Received: (from nobody@localhost) by www.freebsd.org (8.14.2/8.14.1/Submit) id m4GKGTdI040471; Fri, 16 May 2008 20:16:29 GMT (envelope-from nobody) Message-Id: <200805162016.m4GKGTdI040471@www.freebsd.org> Date: Fri, 16 May 2008 20:16:29 GMT From: C Fan To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 Cc: Subject: kern/123741: [netgraph] [panic] kernel panic due to netgraph mpd X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 May 2008 20:20:01 -0000 >Number: 123741 >Category: kern >Synopsis: [netgraph] [panic] kernel panic due to netgraph mpd >Confidential: no >Severity: critical >Priority: high >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Fri May 16 20:20:00 UTC 2008 >Closed-Date: >Last-Modified: >Originator: C Fan >Release: FreeBSD 7.0 stable >Organization: >Environment: FreeBSD xxx 7.0-STABLE FreeBSD 7.0-STABLE #6: Sat May 10 05:27:18 PDT 2008 root@xxx:/usr/obj/usr/src/sys/XXX i386 >Description: I'm having a system, console, and keyboard locked up problems on both of my production server and home server after upgrading to Freebsd 7.0. There is a mpd VPN setup between these 2 servers. A PR has been filed regarding to the problem. http://www.freebsd.org/cgi/query-pr.cgi?pr=123729 After the upgrade, my home server always got locked up after few hours. I compiled my home server with DDB and KDB options. The problem seemed to be gone. However, after few days, I got a kernel panic and core dump. Fatal trap 12: page fault while i kernel mode cpuid = 0; apic id = 00 fault virtua address = 0x8 fault code = supervisor write, page not present instruction pointer = 0x20:0xc2f62a9b stack pointer = 0x28:0xd615ebc4 frame pointer = 0x28:0xd615ebe8 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 99236 (ngctl) [thread pid 99236 tid 100214 ] Stopped at ng_address_path+0x4d: movl %eax,0x8(%ebx) db> show msgbuf .. lot of these messages <7>TCP: [x.x.x.x]:443 to [y.y.y.y]:56747 tcpflags 0x19;tcp_do_segment: FIN_WAIT_1: Received 23 bytes of data after socket was closed, sending RST and removing tcpcb db> bt Tracing pid 99236 tid 100214 td 0xc383aaa0 ng_address_path(0,0,c332cbe2,0,c293a000,...) at ng_address_path+0x4d ngd_connect(c334c000,c332cbe0,c383aaa0,25,d615ec60,...) at ngd_connect+0x65 soconnect(c334c000,c332cbe0,c383aaa0,c065c456,bfbfed2c,...) at soconnect+0x52 kern_connect(c383aaa0,4,c332cbe0,c332cbe0,0,...) at kern_connect+0x56 syscall(d615ed38) at syscall+0x32e Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (98, FreeBSD ELF32, connect), eip = 0x281a2783, esp = 0xbfbfecdc, ebp=0xbfbfecf8 kgdb -c vmcore.0 kernel.debug [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd". There is no member named pathname. Reading symbols from /boot/kernel/acpi.ko...Reading symbols from /boot/kernel/acpi.ko.symbols...done. done. Loaded symbols for /boot/kernel/acpi.ko Reading symbols from /boot/kernel/linux.ko...Reading symbols from /boot/kernel/linux.ko.symbols...done. done. Loaded symbols for /boot/kernel/linux.ko Reading symbols from /boot/kernel/ng_socket.ko...Reading symbols from /boot/kernel/ng_socket.ko.symbols...done. done. Loaded symbols for /boot/kernel/ng_socket.ko Reading symbols from /boot/kernel/netgraph.ko...Reading symbols from /boot/kernel/netgraph.ko.symbols...done. done. Loaded symbols for /boot/kernel/netgraph.ko Reading symbols from /boot/kernel/ng_iface.ko...Reading symbols from /boot/kernel/ng_iface.ko.symbols...done. done. Loaded symbols for /boot/kernel/ng_iface.ko Reading symbols from /boot/kernel/ng_ppp.ko...Reading symbols from /boot/kernel/ng_ppp.ko.symbols...done. done. Loaded symbols for /boot/kernel/ng_ppp.ko Reading symbols from /boot/kernel/ng_bpf.ko...Reading symbols from /boot/kernel/ng_bpf.ko.symbols...done. done. Loaded symbols for /boot/kernel/ng_bpf.ko Reading symbols from /boot/kernel/ng_vjc.ko...Reading symbols from /boot/kernel/ng_vjc.ko.symbols...done. done. Loaded symbols for /boot/kernel/ng_vjc.ko Reading symbols from /boot/kernel/ng_pptpgre.ko...Reading symbols from /boot/kernel/ng_pptpgre.ko.symbols...done. done. Loaded symbols for /boot/kernel/ng_pptpgre.ko Reading symbols from /boot/kernel/ng_ksocket.ko...Reading symbols from /boot/kernel/ng_ksocket.ko.symbols...done. done. Loaded symbols for /boot/kernel/ng_ksocket.ko Reading symbols from /boot/kernel/fire_saver.ko...Reading symbols from /boot/kernel/fire_saver.ko.symbols...done. done. Loaded symbols for /boot/kernel/fire_saver.ko Reading symbols from /boot/kernel/ng_mppc.ko...Reading symbols from /boot/kernel/ng_mppc.ko.symbols...done. done. Loaded symbols for /boot/kernel/ng_mppc.ko Reading symbols from /boot/kernel/rc4.ko...Reading symbols from /boot/kernel/rc4.ko.symbols...done. done. Loaded symbols for /boot/kernel/rc4.ko Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0x8 fault code = supervisor write, page not present instruction pointer = 0x20:0xc2f62a9b stack pointer = 0x28:0xd615ebc4 frame pointer = 0x28:0xd615ebe8 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 99236 (ngctl) panic: from debugger cpuid = 0 Uptime: 4d22h59m14s Physical memory: 501 MB Dumping 88 MB: 73 57 41 25 9 #0 doadump () at pcpu.h:195 195 pcpu.h: No such file or directory. in pcpu.h (kgdb) bt #0 doadump () at pcpu.h:195 #1 0xc06057d6 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418 #2 0xc0605ac9 in panic (fmt=Variable "fmt" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:572 #3 0xc04b2be9 in db_panic (addr=Could not find the frame base for "db_panic". ) at /usr/src/sys/ddb/db_command.c:446 #4 0xc04b3307 in db_command (last_cmdp=0xc0929614, cmd_table=0x0, dopager=1) at /usr/src/sys/ddb/db_command.c:413 #5 0xc04b340a in db_command_loop () at /usr/src/sys/ddb/db_command.c:466 #6 0xc04b4d7e in db_trap (type=12, code=0) at /usr/src/sys/ddb/db_main.c:228 #7 0xc062e15c in kdb_trap (type=12, code=0, tf=0xd615eb84) at /usr/src/sys/kern/subr_kdb.c:524 #8 0xc0860055 in trap_fatal (frame=0xd615eb84, eva=8) at /usr/src/sys/i386/i386/trap.c:890 #9 0xc08602e9 in trap_pfault (frame=0xd615eb84, usermode=0, eva=8) at /usr/src/sys/i386/i386/trap.c:812 #10 0xc0860c76 in trap (frame=0xd615eb84) at /usr/src/sys/i386/i386/trap.c:490 #11 0xc084869b in calltrap () at /usr/src/sys/i386/i386/exception.s:139 #12 0xc2f62a9b in ng_address_path (here=0x0, item=0x0, address=0xc332cbe2 "ngctl99236:", retaddr=0) at /usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c:3599 ---Type to continue, or q to quit--- #13 0xc2f5cb25 in ngd_connect (so=0xc334c000, nam=0xc332cbe0, td=0xc383aaa0) at /usr/src/sys/modules/netgraph/socket/../../../netgraph/ng_socket.c:737 #14 0xc06568f7 in soconnect (so=0xc334c000, nam=0xc332cbe0, td=0xc383aaa0) at /usr/src/sys/kern/uipc_socket.c:765 #15 0xc065ce62 in kern_connect (td=0xc383aaa0, fd=4, sa=0xc332cbe0) at /usr/src/sys/kern/uipc_syscalls.c:558 #16 0xc065cfe7 in connect (td=0xc383aaa0, uap=0xd615ecfc) at /usr/src/sys/kern/uipc_syscalls.c:526 #17 0xc0860635 in syscall (frame=0xd615ed38) at /usr/src/sys/i386/i386/trap.c:1035 #18 0xc0848700 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:196 #19 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) f 12 #12 0xc2f62a9b in ng_address_path (here=0x0, item=0x0, address=0xc332cbe2 "ngctl99236:", retaddr=0) at /usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c:3599 3599 NGI_SET_NODE(item, dest); (kgdb) list 3594 error = ng_path2noderef(here, address, &dest, &hook); 3595 if (error) { 3596 NG_FREE_ITEM(item); 3597 return (error); 3598 } 3599 NGI_SET_NODE(item, dest); 3600 if ( hook) { 3601 NG_HOOK_REF(hook); /* don't let it go while on the queue */ 3602 NGI_SET_HOOK(item, hook); 3603 } Copyright (c) 1992-2008 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 7.0-STABLE #6: Sat May 10 05:27:18 PDT 2008 root@xxx:/usr/obj/usr/src/sys/XXX Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Pentium(R) 4 CPU 2.40GHz (2394.01-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0xf25 Stepping = 5 Features=0xbfebfbff Features2=0x4400 Logical CPUs per core: 2 real memory = 534970368 (510 MB) avail memory = 513785856 (489 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 [chifung@marx /var/log]$ dmesg | less Copyright (c) 1992-2008 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 7.0-STABLE #6: Sat May 10 05:27:18 PDT 2008 root@xxx:/usr/obj/usr/src/sys/XXX Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Pentium(R) 4 CPU 2.40GHz (2394.01-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0xf25 Stepping = 5 Features=0xbfebfbff Features2=0x4400 Logical CPUs per core: 2 real memory = 534970368 (510 MB) avail memory = 513785856 (489 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) acpi0: reservation of 0, a0000 (3) failed acpi0: reservation of 100000, 1ff00000 (3) failed Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 cpu0: on acpi0 p4tcc0: on cpu0 cpu1: on acpi0 p4tcc1: on cpu1 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 agp0: on hostb0 pcib1: at device 1.0 on pci0 pci1: on pcib1 vgapci0: mem 0xf5000000-0xf5ffffff,0xf8000000-0xfbffffff irq 16 at device 0.0 on pci1 pcib2: at device 3.0 on pci0 pci2: on pcib2 em0: port 0xac00-0xac1f mem 0xf7000000-0xf701ffff irq 18 at device 1.0 on pci2 em0: [FILTER] em0: Ethernet address: 00:07:e9:3b:bc:37 uhci0: port 0xcc00-0xcc1f irq 16 at device 29.0 on pci0 uhci0: [GIANT-LOCKED] uhci0: [ITHREAD] usb0: on uhci0 usb0: USB revision 1.0 uhub0: on usb0 uhub0: 2 ports with 2 removable, self powered uhci1: port 0xd000-0xd01f irq 19 at device 29.1 on pci0 uhci1: [GIANT-LOCKED] uhci1: [ITHREAD] usb1: on uhci1 usb1: USB revision 1.0 uhub1: on usb1 uhub1: 2 ports with 2 removable, self powered uhci2: port 0xd400-0xd41f irq 18 at device 29.2 on pci0 uhci2: [GIANT-LOCKED] uhci2: [ITHREAD] usb2: on uhci2 usb2: USB revision 1.0 uhub2: on usb2 uhub2: 2 ports with 2 removable, self powered uhci3: port 0xd800-0xd81f irq 16 at device 29.3 on pci0 uhci3: [GIANT-LOCKED] uhci3: [ITHREAD] usb3: on uhci3 usb3: USB revision 1.0 uhub3: on usb3 uhub3: 2 ports with 2 removable, self powered ehci0: mem 0xf7200000-0xf72003ff irq 23 at device 29.7 on pci0 ehci0: [GIANT-LOCKED] ehci0: [ITHREAD] usb4: EHCI version 1.0 usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3 usb4: on ehci0 usb4: USB revision 2.0 uhub4: on usb4 uhub4: 8 ports with 8 removable, self powered pcib3: at device 30.0 on pci0 pci3: on pcib3 rl0: port 0xbc00-0xbc7f mem 0xf7100000-0xf710007f irq 22 at device 1.0 on pci3 miibus0: on rl0 rlphy0: PHY 0 on miibus0 rlphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto rl0: Ethernet address: 00:a0:4b:04:91:b4 rl0: [ITHREAD] isab0: at device 31.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xffa0-0xffaf at device 31.1 on pci0 ata0: on atapci0 ata0: [ITHREAD] ata1: on atapci0 ata1: [ITHREAD] atapci1: port 0xec00-0xec07,0xe800-0xe803,0xe400-0xe407,0xe000-0xe003,0xdc00-0xdc0f irq 18 at device 31.2 on pci0 atapci1: [ITHREAD] ata2: on atapci1 ata2: [ITHREAD] ata3: on atapci1 ata3: [ITHREAD] pci0: at device 31.3 (no driver attached) acpi_button0: on acpi0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] psm0: irq 12 on atkbdc0 psm0: [GIANT-LOCKED] psm0: [ITHREAD] psm0: model Generic PS/2 mouse, device ID 0 fdc0: port 0x3f0-0x3f1,0x3f2-0x3f3,0x3f4-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: [FILTER] fd0: <1440-KB 3.5" drive> on fdc0 drive 0 sio0: configured irq 4 not in bitmap of probed irqs 0 sio0: port may not be enabled sio0: configured irq 4 not in bitmap of probed irqs 0 sio0: port may not be enabled sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 sio0: type 16550A sio0: [FILTER] cryptosoft0: on motherboard pmtimer0 on isa0 ppc0: at port 0x378-0x37f irq 7 on isa0 ppc0: Generic chipset (EPP/NIBBLE) in COMPATIBLE mode ppbus0: on ppc0 ppbus0: [ITHREAD] plip0: on ppbus0 lpt0: on ppbus0 lpt0: Interrupt-driven port ppi0: on ppbus0 ppc0: [GIANT-LOCKED] ppc0: [ITHREAD] sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> sio1: configured irq 3 not in bitmap of probed irqs 0 sio1: port may not be enabled vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 Timecounters tick every 1.000 msec IPsec: Initialized Security Association Processing. ad0: 57241MB at ata0-master UDMA100 SMP: AP CPU #1 Launched! Trying to mount root from ufs:/dev/ad0s1a WARNING: /var/backups was not properly dismounted WARNING: attempt to net_add_domain(netgraph) after domainfinalize() WARNING: /crypt was not properly dismounted >How-To-Repeat: Setup VPN using mpd between 2 servers. >Fix: >Release-Note: >Audit-Trail: >Unformatted: