From owner-freebsd-bugs@FreeBSD.ORG Mon Sep 17 14:50:12 2012 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3BEC5106564A for ; Mon, 17 Sep 2012 14:50:12 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 135C28FC12 for ; Mon, 17 Sep 2012 14:50:12 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q8HEoBKV045235 for ; Mon, 17 Sep 2012 14:50:11 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q8HEoB1S045234; Mon, 17 Sep 2012 14:50:11 GMT (envelope-from gnats) Resent-Date: Mon, 17 Sep 2012 14:50:11 GMT Resent-Message-Id: <201209171450.q8HEoB1S045234@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Eugene Grosbein Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 44C06106564A for ; Mon, 17 Sep 2012 14:43:42 +0000 (UTC) (envelope-from eugen@k-45-pc-1.sd.rdtc.ru) Received: from k-45-pc-1.sd.rdtc.ru (k-45-pc-1.sd.rdtc.ru [62.231.160.109]) by mx1.freebsd.org (Postfix) with ESMTP id 3A2F28FC14 for ; Mon, 17 Sep 2012 14:43:40 +0000 (UTC) Received: from k-45-pc-1.sd.rdtc.ru (localhost [127.0.0.1]) by k-45-pc-1.sd.rdtc.ru (8.14.5/8.14.5) with ESMTP id q8HEhDdl041303; Mon, 17 Sep 2012 21:43:13 +0700 (NOVT) (envelope-from eugen@k-45-pc-1.sd.rdtc.ru) Received: (from root@localhost) by k-45-pc-1.sd.rdtc.ru (8.14.5/8.14.5/Submit) id q8HEh7fR041243; Mon, 17 Sep 2012 21:43:07 +0700 (NOVT) (envelope-from eugen) Message-Id: <201209171443.q8HEh7fR041243@k-45-pc-1.sd.rdtc.ru> Date: Mon, 17 Sep 2012 21:43:07 +0700 (NOVT) From: Eugene Grosbein To: FreeBSD-gnats-submit@FreeBSD.org X-Send-Pr-Version: 3.113 Cc: Subject: kern/171711: [dummynet] [panic] Kernel panic in dummynet X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Sep 2012 14:50:12 -0000 >Number: 171711 >Category: kern >Synopsis: [dummynet] [panic] Kernel panic in dummynet >Confidential: no >Severity: serious >Priority: medium >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Mon Sep 17 14:50:11 UTC 2012 >Closed-Date: >Last-Modified: >Originator: Eugene Grosbein >Release: FreeBSD 8.3-STABLE amd64 >Organization: RDTC JSC >Environment: System: FreeBSD k-45-pc-1.sd.rdtc.ru 8.3-STABLE FreeBSD 8.3-STABLE #37: Wed Jun 13 12:25:17 NOVT 2012 root@k-45-pc-1.sd.rdtc.ru:/usr/local/obj/home/src/sys/PPPOE amd64 >Description: I run pretty busy mpd-5.6 based PPPoE access server (about 1700 simultaneous connections at most loaded hours). It uses dummynet extensively: each connecting user obtains its own dynamic dummynet pipes. Some time ago this server has crashed. Crashdump points to some dummynet-related code: GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Unread portion of the kernel message buffer: 0 (dummynet) trap number = 12 panic: page fault cpuid = 0 KDB: stack backtrace: db_trace_self_wrapper() at 0xffffffff801adf1a = db_trace_self_wrapper+0x2a kdb_backtrace() at 0xffffffff80330827 = kdb_backtrace+0x37 panic() at 0xffffffff802fd48e = panic+0x1ce trap_fatal() at 0xffffffff804f3f80 = trap_fatal+0x290 trap_pfault() at 0xffffffff804f430e = trap_pfault+0x23e trap() at 0xffffffff804f47de = trap+0x3ce calltrap() at 0xffffffff804dac04 = calltrap+0x8 --- trap 0xc, rip = 0x1, rsp = 0xffffff8122a9ea20, rbp = 0xffffff8122a9ea40 --- uart_z8530_class() at 0x1 uma_zfree_arg() at 0xffffffff804b53da = uma_zfree_arg+0x3a m_freem() at 0xffffffff8035c4a7 = m_freem+0x37 dummynet_send() at 0xffffffff8040017d = dummynet_send+0x2d dummynet_task() at 0xffffffff80400496 = dummynet_task+0x1c6 taskqueue_run_locked() at 0xffffffff8033cdf5 = taskqueue_run_locked+0x85 taskqueue_thread_loop() at 0xffffffff8033cf8e = taskqueue_thread_loop+0x4e fork_exit() at 0xffffffff802d13cf = fork_exit+0x11f fork_trampoline() at 0xffffffff804db14e = fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8122a9ecf0, rbp = 0 --- Uptime: 14d8h5m59s Dumping 754 out of 4079 MB:..3%..11%..22%..32%..41%..51%..62%..73%..81%..92% Reading symbols from /boot/kernel/ipmi.ko...done. Loaded symbols for /boot/kernel/ipmi.ko #0 doadump () at /home/src/sys/kern/kern_shutdown.c:268 268 if (textdump_pending) (kgdb) bt #0 doadump () at /home/src/sys/kern/kern_shutdown.c:268 #1 0xffffffff802fcf8a in boot (howto=260) at /home/src/sys/kern/kern_shutdown.c:448 #2 0xffffffff802fd467 in panic (fmt=0x1
) at /home/src/sys/kern/kern_shutdown.c:639 #3 0xffffffff804f3f80 in trap_fatal (frame=0xc, eva=Variable "eva" is not available. ) at /home/src/sys/amd64/amd64/trap.c:848 #4 0xffffffff804f430e in trap_pfault (frame=0xffffff8122a9e970, usermode=0) at /home/src/sys/amd64/amd64/trap.c:764 #5 0xffffffff804f47de in trap (frame=0xffffff8122a9e970) at /home/src/sys/amd64/amd64/trap.c:457 #6 0xffffffff804dac04 in calltrap () at /home/src/sys/amd64/amd64/exception.S:228 #7 0x0000000000000001 in ?? () #8 0xffffffff802eb967 in mb_dtor_pack (mem=Variable "mem" is not available. ) at /home/src/sys/kern/kern_mbuf.c:453 #9 0xffffffff804b53da in uma_zfree_arg (zone=0xffffff00df773780, item=0xffffff0054691b00, udata=0x0) at /home/src/sys/vm/uma_core.c:2543 #10 0xffffffff8035c4a7 in m_freem (mb=0x0) at mbuf.h:562 #11 0xffffffff8040017d in dummynet_send (m=0xffffff0054691b00) at /home/src/sys/netinet/ipfw/ip_dn_io.c:705 #12 0xffffffff80400496 in dummynet_task (context=Variable "context" is not available. ) at /home/src/sys/netinet/ipfw/ip_dn_io.c:615 #13 0xffffffff8033cdf5 in taskqueue_run_locked (queue=0xffffff0003a09380) at /home/src/sys/kern/subr_taskqueue.c:250 #14 0xffffffff8033cf8e in taskqueue_thread_loop (arg=Variable "arg" is not available. ) at /home/src/sys/kern/subr_taskqueue.c:387 #15 0xffffffff802d13cf in fork_exit (callout=0xffffffff8033cf40 , arg=0xffffffff80769e80, frame=0xffffff8122a9ec40) at /home/src/sys/kern/kern_fork.c:876 #16 0xffffffff804db14e in fork_trampoline () at /home/src/sys/amd64/amd64/exception.S:602 #17 0x0000000000000000 in ?? () #18 0x0000000000000000 in ?? () #19 0x0000000000000000 in ?? () #20 0x0000000000000000 in ?? () #21 0x0000000000000000 in ?? () #22 0x0000000000000000 in ?? () #23 0x0000000000000000 in ?? () #24 0x0000000000000000 in ?? () #25 0x0000000000000000 in ?? () #26 0x0000000000000000 in ?? () #27 0x0000000000000000 in ?? () #28 0x0000000000000000 in ?? () #29 0x0000000000000000 in ?? () #30 0x0000000000000000 in ?? () #31 0x0000000000000000 in ?? () #32 0x0000000000000000 in ?? () #33 0x0000000000000000 in ?? () #34 0x0000000000000000 in ?? () #35 0x0000000000000000 in ?? () #36 0x0000000000000000 in ?? () #37 0x0000000000000000 in ?? () #38 0x0000000000000000 in ?? () #39 0x0000000000000000 in ?? () #40 0x0000000000000000 in ?? () #41 0xffffffff80763f00 in sleepq_chains () #42 0xffffff0003a12d18 in ?? () #43 0x0000000000000000 in ?? () #44 0xffffff0003a128e0 in ?? () ---Type to continue, or q to quit--- #45 0xffffff8122a9eaf0 in ?? () #46 0xffffff8122a9ea98 in ?? () #47 0xffffff0001c0e470 in ?? () #48 0xffffffff80323972 in sched_switch (td=0xffffffff8033cf40, newtd=0xffffffff80769e80, flags=Variable "flags" is not available. ) at /home/src/sys/kern/sched_ule.c:1886 Previous frame inner to this frame (corrupt stack?) Here comes its kernel configuration file: cpu HAMMER ident PPPOE # To statically compile in device wiring instead of /boot/device.hints #hints "GENERIC.hints" # Default places to look for devices. # Use the following to compile in values accessible to the kernel # through getenv() (or kenv(1) in userland). The format of the file # is 'variable=value', see kenv(1) # # env "GENERIC.env" makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols options SCHED_ULE # ULE scheduler options PREEMPTION # Enable kernel thread preemption options INET # InterNETworking options INET6 # IPv6 communications protocols #options SCTP # Stream Control Transmission Protocol options FFS # Berkeley Fast Filesystem options SOFTUPDATES # Enable FFS soft updates support options UFS_ACL # Support for access control lists options UFS_DIRHASH # Improve performance on big directories options UFS_GJOURNAL # Enable gjournal-based UFS journaling options MD_ROOT # MD is a potential root device #options NFSCLIENT # Network Filesystem Client #options NFSSERVER # Network Filesystem Server #options NFSLOCKD # Network Lock Manager options NFS_ROOT # NFS usable as /, requires NFSCLIENT #options MSDOSFS # MSDOS Filesystem #options CD9660 # ISO 9660 Filesystem #options PROCFS # Process filesystem (requires PSEUDOFS) options PSEUDOFS # Pseudo-filesystem framework options GEOM_PART_GPT # GUID Partition Tables. options GEOM_LABEL # Provides labelization #options GEOM_JOURNAL options COMPAT_43TTY # BSD 4.3 TTY compat (sgtty) options COMPAT_FREEBSD32 # Compatible with i386 binaries #options COMPAT_FREEBSD4 # Compatible with FreeBSD4 #options COMPAT_FREEBSD5 # Compatible with FreeBSD5 #options COMPAT_FREEBSD6 # Compatible with FreeBSD6 #options COMPAT_FREEBSD7 # Compatible with FreeBSD7 #options SCSI_DELAY=5000 # Delay (in ms) before probing SCSI options KTRACE # ktrace(1) support options STACK # stack(9) support options SYSVSHM # SYSV-style shared memory options SYSVMSG # SYSV-style message queues options SYSVSEM # SYSV-style semaphores options P1003_1B_SEMAPHORES # POSIX-style semaphores options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions options PRINTF_BUFR_SIZE=512 # Prevent printf output being interspersed. options KBD_INSTALL_CDEV # install a CDEV entry in /dev options HWPMC_HOOKS # Necessary kernel hooks for hwpmc(4) options AUDIT # Security event auditing options MAC # TrustedBSD MAC Framework #options FLOWTABLE # per-cpu routing cache #options KDTRACE_FRAME # Ensure frames are compiled in #options KDTRACE_HOOKS # Kernel DTrace hooks options INCLUDE_CONFIG_FILE # Include this file in kernel # Make an SMP-capable kernel by default options SMP # Symmetric MultiProcessor Kernel # CPU frequency control device cpufreq # Bus support. device acpi device pci # Floppy drives #device fdc # ATA and ATAPI devices device ata device atadisk # ATA disk drives device atapicd # ATAPI CDROM drives # SCSI peripherals device scbus # SCSI bus (required for SCSI) device da # Direct Access (disks) device cd # CD device pass # Passthrough device (direct SCSI access) # atkbdc0 controls both the keyboard and the PS/2 mouse device atkbdc # AT keyboard controller device atkbd # AT keyboard device psm # PS/2 mouse device kbdmux # keyboard multiplexer device vga # VGA video card driver # syscons is the default console driver, resembling an SCO console device sc # Serial (COM) ports device uart # Generic UART driver # PCI Ethernet NICs. device em # Intel PRO/1000 Gigabit Ethernet Family device igb # Pseudo devices. device loop # Network loopback device random # Entropy device device ether # Ethernet support device vlan # 802.1Q VLAN support device pty # BSD-style compatibility pseudo ttys device md # Memory "disks" device gif # IPv6 and IPv4 tunneling device faith # IPv6-to-IPv4 relaying (translation) device firmware # firmware assist module device snp device bpf # Berkeley packet filter # USB support #options USB_DEBUG # enable debug msgs #options USB_VERBOSE device uhci # UHCI PCI->USB interface device ehci # EHCI PCI->USB interface (USB 2.0) device usb # USB Bus (required) device ukbd # Keyboard device umass # Disks/Mass storage - Requires scbus and da device ums # Mouse device ucom # USB support for Prolific PL-2303 serial adapters device uplcom # USB support for Silicon Laboratories CP2101/CP2102 based USB serial adapters device uslcom #options IPSEC #device crypto options NETGRAPH options NETGRAPH_ETHER options NETGRAPH_IFACE options NETGRAPH_MPPC_ENCRYPTION options NETGRAPH_PPP options NETGRAPH_PPPOE options NETGRAPH_SOCKET options NETGRAPH_TCPMSS options NETGRAPH_TEE options NETGRAPH_VJC options IPFIREWALL options IPFIREWALL_FORWARD options DUMMYNET options VFS_AIO device smbus device smb device ichsmb device iicbus device iicbb device ic device iic device iicsmb device coretemp device ichwd device nvram device lagg options KDB options KDB_TRACE options KDB_UNATTENDED options DDB options DDB_NUMSYM #options NETGRAPH_DEBUG #options INVARIANT_SUPPORT #options INVARIANTS #options DEBUG_MEMGUARD #options BREAK_TO_DEBUGGER options ALT_BREAK_TO_DEBUGGER device bridge >How-To-Repeat: Run busy router with lots of dummynet dynamic pipes, lots of traffic inside pipes and high rate of pipe creation/expiration. >Fix: Unknown for me. >Release-Note: >Audit-Trail: >Unformatted: