From owner-freebsd-bugs@FreeBSD.ORG Tue May 6 14:00:07 2008 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 488ED1065676 for ; Tue, 6 May 2008 14:00:07 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 27EAA8FC27 for ; Tue, 6 May 2008 14:00:07 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id m46E07i2088647 for ; Tue, 6 May 2008 14:00:07 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id m46E06W9088646; Tue, 6 May 2008 14:00:06 GMT (envelope-from gnats) Resent-Date: Tue, 6 May 2008 14:00:06 GMT Resent-Message-Id: <200805061400.m46E06W9088646@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, matthew.seaman@thebunker.net Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6678B106566B for ; Tue, 6 May 2008 13:55:05 +0000 (UTC) (envelope-from root@obol.hosted-at.thebunker.net) Received: from obol.hosted-at.thebunker.net (obol.hosted-at.thebunker.net [213.129.86.74]) by mx1.freebsd.org (Postfix) with ESMTP id 41F7C8FC2B for ; Tue, 6 May 2008 13:55:03 +0000 (UTC) (envelope-from root@obol.hosted-at.thebunker.net) Received: from obol.hosted-at.thebunker.net (localhost [127.0.0.1]) by obol.hosted-at.thebunker.net (8.14.2/8.14.2) with ESMTP id m46DM4YA001185 for ; Tue, 6 May 2008 14:22:04 +0100 (BST) (envelope-from root@obol.hosted-at.thebunker.net) Received: (from root@localhost) by obol.hosted-at.thebunker.net (8.14.2/8.14.2/Submit) id m46DM4hT001184; Tue, 6 May 2008 14:22:04 +0100 (BST) (envelope-from root) Message-Id: <200805061322.m46DM4hT001184@obol.hosted-at.thebunker.net> Date: Tue, 6 May 2008 14:22:04 +0100 (BST) From: matthew.seaman@thebunker.net To: FreeBSD-gnats-submit@FreeBSD.org X-Send-Pr-Version: 3.113 Cc: Subject: kern/123463: repeatable crash related to ipsec-tools X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: matthew.seaman@thebunker.net List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 May 2008 14:00:07 -0000 >Number: 123463 >Category: kern >Synopsis: repeatable crash related to ipsec-tools >Confidential: no >Severity: critical >Priority: high >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Tue May 06 14:00:06 UTC 2008 >Closed-Date: >Last-Modified: >Originator: Matthew Seaman >Release: FreeBSD 7.0-RELEASE-p1 amd64 >Organization: The Bunker >Environment: System: FreeBSD obol.hosted-at.thebunker.net 7.0-RELEASE-p1 FreeBSD 7.0-RELEASE-p1 #3: Sun May 4 10:46:11 BST 2008 root@obol.hosted-at.thebunker.net:/usr/obj/usr/src/sys/OBOL amd64 >Description: I have a new HP DL140G3 server runing RELENG_7_0 which has been stable up to now. However the combination of configuring it as an IPSec tunnel end-point and then turning on some Nagios monitoring via the tunnel causes the machine to crash within a few minutes. kgdb backtrace attached from the latest crash attached. I'm using racoon from security/ipsec-tools for IKE -- I had tried previously using security/isakmpd but in that case I found the process would run fine for maybe 20 minutes, then get into a loop where it chewed up lots of RAM very fast, until the kernel killed it. >How-To-Repeat: >Fix: --- kgdb.out begins here --- Script started on Tue May 6 14:01:27 2008 [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd". Unread portion of the kernel message buffer: Fatal trap 9: general protection fault while in kernel mode cpuid = 3; apic id = 03 instruction pointer = 0x8:0xffffffff80706048 stack pointer = 0x10:0xffffffffae5cbf30 frame pointer = 0x10:0xffffff0001e1c300 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 779 (snmpd) trap number = 9 panic: general protection fault cpuid = 3 GEOM_MIRROR: Device gm0: rebuilding provider da0 stopped. Uptime: 7m23s Physical memory: 2034 MB Dumping 314 MB: 299 283 267 251 235 219 203 187 171 155 139 123 107 91 75 59 43 27 11 #0 doadump () at pcpu.h:194 194 __asm __volatile("movq %%gs:0,%0" : "=r" (td)); (kgdb) backtrace #0 doadump () at pcpu.h:194 #1 0x0000000000000004 in ?? () #2 0xffffffff8045b9cf in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409 #3 0xffffffff8045bdf8 in panic (fmt=0x104
) at /usr/src/sys/kern/kern_shutdown.c:563 #4 0xffffffff8071f8ca in trap_fatal (frame=0xffffff00014f8350, eva=18446742974219888848) at /usr/src/sys/amd64/amd64/trap.c:724 #5 0xffffffff80720388 in trap (frame=0xffffffffae5cbe80) at /usr/src/sys/amd64/amd64/trap.c:526 #6 0xffffffff8070738e in calltrap () at /usr/src/sys/amd64/amd64/exception.S:169 #7 0xffffffff80706048 in bus_dmamap_load_mbuf_sg (dmat=0xffffff00012a7a00, map=0x0, m0=Variable "m0" is not available. ) at /usr/src/sys/amd64/amd64/busdma_machdep.c:816 #8 0xffffffff80270a85 in bge_start_locked (ifp=0xffffff0001277000) at /usr/src/sys/dev/bge/if_bge.c:3390 #9 0xffffffff802714a7 in bge_start (ifp=0xffffff0001277000) at /usr/src/sys/dev/bge/if_bge.c:3572 #10 0xffffffff804e977e in ether_output_frame (ifp=0xffffff0001277000, m=0xffffff0001977100) at /usr/src/sys/net/if_ethersubr.c:405 #11 0xffffffff804e9cdf in ether_output (ifp=0xffffff0001277000, m=0xffffff0001977100, dst=Variable "dst" is not available. ) at /usr/src/sys/net/if_ethersubr.c:374 #12 0xffffffff805333e9 in ip_output (m=0xffffff0001977100, opt=Variable "opt" is not available. ) at /usr/src/sys/netinet/ip_output.c:583 #13 0xffffffff805bf747 in ipsec_process_done (m=0xffffff000177bc00, isr=0xffffff000174f800) at /usr/src/sys/netipsec/ipsec_output.c:177 #14 0xffffffff805cd8f8 in esp_output_cb (crp=0xffffff0001e24cb8) at /usr/src/sys/netipsec/xform_esp.c:965 #15 0xffffffff80606109 in crypto_done (crp=0xffffff0001e24cb8) at /usr/src/sys/opencrypto/crypto.c:1148 #16 0xffffffff8060934c in swcr_process (dev=Variable "dev" is not available. ) at /usr/src/sys/opencrypto/cryptosoft.c:975 #17 0xffffffff80606e89 in crypto_invoke (cap=Variable "cap" is not available. ) at cryptodev_if.h:53 #18 0xffffffff80607974 in crypto_dispatch (crp=0xffffff0001e24cb8) at /usr/src/sys/opencrypto/crypto.c:798 #19 0xffffffff805cdf91 in esp_output (m=0xffffff000161c360, isr=0xffffff000174f800, mp=Variable "mp" is not available. ) at /usr/src/sys/netipsec/xform_esp.c:875 #20 0xffffffff805bf95b in ipsec4_process_packet (m=0xffffff000177bc00, isr=0xffffff000174f800, flags=Variable "flags" is not available. ) at /usr/src/sys/netipsec/ipsec_output.c:486 #21 0xffffffff805312e7 in ip_ipsec_output (m=0xffffffffae5cc8b8, inp=0xffffff000168c360, flags=0xffffffffae5cc8ac, error=0xffffffffae5cc8f8, ro=Variable "ro" is not available. ) at /usr/src/sys/netinet/ip_ipsec.c:331 #22 0xffffffff80532814 in ip_output (m=0xffffff000177bc00, opt=Variable "opt" is not available. ) at /usr/src/sys/netinet/ip_output.c:418 #23 0xffffffff80594ab3 in udp_send (so=Variable "so" is not available. ) at /usr/src/sys/netinet/udp_usrreq.c:972 #24 0xffffffff804abb60 in sosend_dgram (so=0xffffff0001aadae0, addr=0xffffff000161c090, uio=Variable "uio" is not available. ) at /usr/src/sys/kern/uipc_socket.c:1053 #25 0xffffffff804af176 in kern_sendit (td=0xffffff00014f8350, s=11, mp=0xffffffffae5ccb10, flags=0, control=0x0, segflg=Variable "segflg" is not available. ) at /usr/src/sys/kern/uipc_syscalls.c:789 #26 0xffffffff804b1c6a in sendit (td=0xffffff00014f8350, s=11, mp=0xffffffffae5ccb10, flags=0) at /usr/src/sys/kern/uipc_syscalls.c:730 #27 0xffffffff804b1d4a in sendto (td=Variable "td" is not available. ) at /usr/src/sys/kern/uipc_syscalls.c:841 #28 0xffffffff8071fedc in syscall (frame=0xffffffffae5ccc70) at /usr/src/sys/amd64/amd64/trap.c:852 #29 0xffffffff8070759b in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:290 #30 0x00000008018d607c in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) up #1 0x0000000000000004 in ?? () (kgdb) up #2 0xffffffff8045b9cf in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409 409 doadump(); (kgdb) up #3 0xffffffff8045bdf8 in panic (fmt=0x104
) at /usr/src/sys/kern/kern_shutdown.c:563 563 boot(bootopt); (kgdb) up #4 0xffffffff8071f8ca in trap_fatal (frame=0xffffff00014f8350, eva=18446742974219888848) at /usr/src/sys/amd64/amd64/trap.c:724 724 panic("%s", trap_msg[type]); (kgdb) up #5 0xffffffff80720388 in trap (frame=0xffffffffae5cbe80) at /usr/src/sys/amd64/amd64/trap.c:526 526 trap_fatal(frame, 0); (kgdb) up #6 0xffffffff8070738e in calltrap () at /usr/src/sys/amd64/amd64/exception.S:169 169 call trap Current language: auto; currently asm (kgdb) up #7 0xffffffff80706048 in bus_dmamap_load_mbuf_sg (dmat=0xffffff00012a7a00, map=0x0, m0=Variable "m0" is not available. ) at /usr/src/sys/amd64/amd64/busdma_machdep.c:816 816 if (m->m_len > 0) { Current language: auto; currently c (kgdb) list 811 int first = 1; 812 bus_addr_t lastaddr = 0; 813 struct mbuf *m; 814 815 for (m = m0; m != NULL && error == 0; m = m->m_next) { 816 if (m->m_len > 0) { 817 error = _bus_dmamap_load_buffer(dmat, map, 818 m->m_data, m->m_len, 819 NULL, flags, &lastaddr, 820 segs, nsegs, first); (kgdb) quit Script done on Tue May 6 14:05:50 2008 --- kgdb.out ends here --- --- dmesg.boot begins here --- Copyright (c) 1992-2008 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 7.0-RELEASE-p1 #3: Sun May 4 10:46:11 BST 2008 root@obol.hosted-at.thebunker.net:/usr/obj/usr/src/sys/OBOL Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(R) CPU E5335 @ 2.00GHz (1995.01-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x6fb Stepping = 11 Features=0xbfebfbff Features2=0x4e33d AMD Features=0x20100800 AMD Features2=0x1 Cores per package: 4 usable memory = 2133483520 (2034 MB) avail memory = 2058792960 (1963 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-47 on motherboard kbd1 at kbdmux0 ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413) hptrr: HPT RocketRAID controller driver v1.1 (May 4 2008 10:46:04) acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0 cpu0: on acpi0 p4tcc0: on cpu0 cpu1: on acpi0 p4tcc1: on cpu1 cpu2: on acpi0 p4tcc2: on cpu2 cpu3: on acpi0 p4tcc3: on cpu3 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: at device 2.0 on pci0 pci1: on pcib1 pcib2: irq 16 at device 0.0 on pci1 pci2: on pcib2 pcib3: irq 16 at device 0.0 on pci2 pci3: on pcib3 pcib4: at device 0.3 on pci1 pci7: on pcib4 mpt0: port 0x2000-0x20ff mem 0xdc210000-0xdc213fff,0xdc200000-0xdc20ffff irq 24 at device 1.0 on pci7 mpt0: [ITHREAD] mpt0: MPI Version=1.5.14.0 mpt0: mpt_cam_event: 0x16 mpt0: mpt_cam_event: 0x16 mpt0: mpt_cam_event: 0x16 mpt0: mpt_cam_event: 0x12 mpt0: mpt_cam_event: 0x12 mpt0: mpt_cam_event: 0x16 mpt0: mpt_cam_event: 0x16 mpt0: mpt_cam_event: 0x16 pcib5: at device 3.0 on pci0 pci8: on pcib5 pcib6: at device 4.0 on pci0 pci12: on pcib6 pcib7: at device 5.0 on pci0 pci13: on pcib7 pcib8: at device 6.0 on pci0 pci14: on pcib8 pcib9: at device 7.0 on pci0 pci15: on pcib9 pcib10: at device 28.0 on pci0 pci22: on pcib10 bge0: mem 0xdc300000-0xdc30ffff irq 16 at device 0.0 on pci22 miibus0: on bge0 brgphy0: PHY 1 on miibus0 brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto bge0: Ethernet address: 00:1e:0b:5a:b2:e4 bge0: [ITHREAD] pcib11: at device 28.1 on pci0 pci23: on pcib11 bge1: mem 0xdc400000-0xdc40ffff irq 17 at device 0.0 on pci23 miibus1: on bge1 brgphy1: PHY 1 on miibus1 brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto bge1: Ethernet address: 00:1e:0b:5a:b2:e5 bge1: [ITHREAD] uhci0: port 0x1800-0x181f irq 23 at device 29.0 on pci0 uhci0: [GIANT-LOCKED] uhci0: [ITHREAD] usb0: on uhci0 usb0: USB revision 1.0 uhub0: on usb0 uhub0: 2 ports with 2 removable, self powered uhci1: port 0x1820-0x183f irq 23 at device 29.1 on pci0 uhci1: [GIANT-LOCKED] uhci1: [ITHREAD] usb1: on uhci1 usb1: USB revision 1.0 uhub1: on usb1 uhub1: 2 ports with 2 removable, self powered uhci2: port 0x1840-0x185f irq 23 at device 29.2 on pci0 uhci2: [GIANT-LOCKED] uhci2: [ITHREAD] usb2: on uhci2 usb2: USB revision 1.0 uhub2: on usb2 uhub2: 2 ports with 2 removable, self powered ehci0: mem 0xdc000000-0xdc0003ff irq 23 at device 29.7 on pci0 ehci0: [GIANT-LOCKED] ehci0: [ITHREAD] usb3: EHCI version 1.0 usb3: companion controllers, 2 ports each: usb0 usb1 usb2 usb3: on ehci0 usb3: USB revision 2.0 uhub3: on usb3 uhub3: 6 ports with 6 removable, self powered pcib12: at device 30.0 on pci0 pci24: on pcib12 vgapci0: mem 0xde000000-0xdeffffff,0xdc500000-0xdc503fff,0xdc800000-0xdcffffff irq 17 at device 2.0 on pci24 isab0: at device 31.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1860-0x186f at device 31.1 on pci0 ata0: on atapci0 ata0: [ITHREAD] ata1: on atapci0 ata1: [ITHREAD] atapci1: port 0x1890-0x1897,0x1884-0x1887,0x1888-0x188f,0x1880-0x1883,0x1870-0x187f mem 0xdc000400-0xdc0007ff irq 19 at device 31.2 on pci0 atapci1: [ITHREAD] ata2: on atapci1 ata2: [ITHREAD] ata3: on atapci1 ata3: [ITHREAD] pci0: at device 31.3 (no driver attached) acpi_button0: on acpi0 sio0: configured irq 4 not in bitmap of probed irqs 0 sio0: port may not be enabled sio0: configured irq 4 not in bitmap of probed irqs 0 sio0: port may not be enabled sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 sio0: type 16550A sio0: [FILTER] cryptosoft0: on motherboard orm0: at iomem 0xc0000-0xc7fff,0xc8000-0xc8fff,0xdc000-0xdffff on isa0 atkbdc0: at port 0x60,0x64 on isa0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] ppc0: cannot reserve I/O port range sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> sio1: configured irq 3 not in bitmap of probed irqs 0 sio1: port may not be enabled vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 ukbd0: on uhub2 kbd2 at ukbd0 ums0: on uhub2 ums0: 8 buttons and Z dir. Timecounters tick every 1.000 msec Fast IPsec: Initialized Security Association Processing. hptrr: no controller detected. acd0: CDRW at ata0-master UDMA33 da0 at mpt0 bus 0 target 1 lun 0 da0: Fixed Direct Access SCSI-5 device da0: 300.000MB/s transfers da0: Command Queueing Enabled da0: 152627MB (312581808 512 byte sectors: 255H 63S/T 19457C) da1 at mpt0 bus 0 target 2 lun 0 da1: Fixed Direct Access SCSI-5 device da1: 300.000MB/s transfers da1: Command Queueing Enabled da1: 152627MB (312581808 512 byte sectors: 255H 63S/T 19457C) SMP: AP CPU #1 Launched! SMP: AP CPU #3 Launched! SMP: AP CPU #2 Launched! GEOM_MIRROR: Device mirror/gm0 launched (1/2). GEOM_MIRROR: Device gm0: rebuilding provider da0. Trying to mount root from ufs:/dev/mirror/gm0s1a WARNING: / was not properly dismounted --- dmesg.boot ends here --- --- OBOL begins here --- # # Kernel config for FreeBSD 7.0+ server # include GENERIC ident OBOL nooptions SCHED_4BSD options SCHED_ULE device crypto device cryptodev options IPSEC options IPSEC_DEBUG options ALTQ # # That's All Folks! # --- OBOL ends here --- >Release-Note: >Audit-Trail: >Unformatted: