From owner-freebsd-stable@FreeBSD.ORG Mon Nov 24 22:58:59 2008 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 76F0C1065670 for ; Mon, 24 Nov 2008 22:58:59 +0000 (UTC) (envelope-from citrin@citrin.ru) Received: from mail-chaos.rambler.ru (mail-chaos.rambler.ru [81.19.68.130]) by mx1.freebsd.org (Postfix) with ESMTP id B21138FC1B for ; Mon, 24 Nov 2008 22:58:58 +0000 (UTC) (envelope-from citrin@citrin.ru) Received: from cmb.rambler.ramblermedia.com (unknown [91.78.189.73]) (Authenticated sender: citrin@citrin.ru) by mail-chaos.rambler.ru (Postfix) with ESMTPSA id 269961705C for ; Tue, 25 Nov 2008 01:48:40 +0300 (MSK) Message-ID: <492B2F46.9000709@citrin.ru> Date: Tue, 25 Nov 2008 01:48:38 +0300 From: Anton Yuzhaninov User-Agent: Thunderbird 2.0.0.18 (Macintosh/20081105) MIME-Version: 1.0 To: freebsd-stable@FreeBSD.org Content-Type: text/plain; charset=KOI8-R; format=flowed Content-Transfer-Encoding: 7bit Cc: Subject: RELENG_7 panic under load: vm_page_unwire: invalid wire count: 0 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Nov 2008 22:58:59 -0000 Box with fresh RELENG_7 panic under heavy network load (more than 50k connections). This panics seems to be senfile(2) related, because when sendfile disabled in nginx, I can't reproduce the problem. Backtrace in all cases like this: # kgdb kernel /spool/crash/vmcore.1 GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Unread portion of the kernel message buffer: panic: vm_page_unwire: invalid wire count: 0 cpuid = 0 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a panic() at panic+0x182 vm_page_unwire() at vm_page_unwire+0x84 sf_buf_mext() at sf_buf_mext+0x3c mb_free_ext() at mb_free_ext+0x99 sbdrop_internal() at sbdrop_internal+0x1e8 tcp_do_segment() at tcp_do_segment+0x1512 tcp_input() at tcp_input+0x7f7 ip_input() at ip_input+0xa8 ether_demux() at ether_demux+0x1b4 ether_input() at ether_input+0x1bb bge_intr() at bge_intr+0x3ca ithread_loop() at ithread_loop+0x180 fork_exit() at fork_exit+0x11f fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffea28fd30, rbp = 0 --- Uptime: 36m47s Physical memory: 4087 MB Dumping 708 MB: 693 677 661 645 629 613 597 581 565 549 533 517 501 485 469 453 437 421 405 389 373 357 341 325 309 293 277 261 245 229 213 197 181 165 149 133 117 101 85 69 53 37 21 5 #0 doadump () at pcpu.h:195 195 __asm __volatile("movq %%gs:0,%0" : "=r" (td)); (kgdb) bt #0 doadump () at pcpu.h:195 #1 0xffffffff8031adf8 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418 #2 0xffffffff8031b25c in panic (fmt=Variable "fmt" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:574 #3 0xffffffff8044a084 in vm_page_unwire (m=Variable "m" is not available. ) at /usr/src/sys/vm/vm_page.c:1410 #4 0xffffffff80379a4c in sf_buf_mext (addr=Variable "addr" is not available. ) at /usr/src/sys/kern/uipc_syscalls.c:1720 #5 0xffffffff8036e9c9 in mb_free_ext (m=0xffffff0081f93d00) at /usr/src/sys/kern/uipc_mbuf.c:257 #6 0xffffffff80372c38 in sbdrop_internal (sb=0xffffff00b4161458, len=2896) at mbuf.h:515 #7 0xffffffff803d6532 in tcp_do_segment (m=0xffffff0075c23b00, th=0xffffff0075c53024, so=0xffffff00b41612d0, tp=0xffffff00b4154b60, drop_hdrlen=52, tlen=0) at /usr/src/sys/netinet/tcp_input.c:2042 #8 0xffffffff803d7bc7 in tcp_input (m=0xffffff0075c23b00, off0=20) at /usr/src/sys/netinet/tcp_input.c:846 #9 0xffffffff803cf108 in ip_input (m=0xffffff0075c23b00) at /usr/src/sys/netinet/ip_input.c:665 #10 0xffffffff803b8004 in ether_demux (ifp=0xffffff0001255800, m=0xffffff0075c23b00) at /usr/src/sys/net/if_ethersubr.c:834 #11 0xffffffff803b825b in ether_input (ifp=0xffffff0001255800, m=0xffffff0075c23b00) at /usr/src/sys/net/if_ethersubr.c:692 #12 0xffffffff801bcf5a in bge_intr (xsc=Variable "xsc" is not available. ) at /usr/src/sys/dev/bge/if_bge.c:3160 #13 0xffffffff802fb5f0 in ithread_loop (arg=0xffffff0003711840) at /usr/src/sys/kern/kern_intr.c:1088 #14 0xffffffff802f7f7f in fork_exit (callout=0xffffffff802fb470 , arg=0xffffff0003711840, frame=0xffffffffea28fc80) at /usr/src/sys/kern/kern_fork.c:804 #15 0xffffffff8045b88e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:455 #16 0x0000000000000000 in ?? () #17 0x0000000000000000 in ?? () #18 0x0000000000000001 in ?? () in /boot/loader.conf I have: vm.kmem_size=1536M # 2 Mb KVA/kmem net.inet.tcp.tcbhashsize=131072 # 64M KVA kern.maxbcache=64M # 4M KVA kern.ipc.maxpipekva=4M # net.inet.tcp.syncache.hashsize=1024 net.inet.tcp.syncache.bucketlimit=100 in /etc/sysctl.conf # 576 Mb KVA/kmem kern.ipc.nmbclusters=262144 kern.ipc.nmbjumbop=65536 kern.ipc.maxsockets=307200 kern.ipc.somaxconn=4096 kern.maxfiles=307200 kern.maxfilesperproc=102400 $ sysctl vm.kvm_free vm.kvm_free: 327151616 netstat -m output, several seconds before panic: 380270/63895/444165 mbufs in use (current/cache/total) 14141/29273/43414/262144 mbuf clusters in use (current/cache/total/max) 14141/29251 mbuf+clusters out of packet secondary zone in use (current/cache) 0/9/9/65536 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) 123349K/74555K/197905K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/0/0 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 1 requests for I/O initiated by sendfile 0 calls to protocol drain routines -- Anton Yuzhaninov