From owner-freebsd-stable@FreeBSD.ORG Thu Aug 17 08:21:05 2006 Return-Path: X-Original-To: freebsd-stable@freebsd.org Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 572FD16A4E0; Thu, 17 Aug 2006 08:21:05 +0000 (UTC) (envelope-from pvh@wfeet.za.net) Received: from ctb-mesg8.saix.net (ctb-mesg8.saix.net [196.25.240.78]) by mx1.FreeBSD.org (Postfix) with ESMTP id B95BB43D60; Thu, 17 Aug 2006 08:21:03 +0000 (GMT) (envelope-from pvh@wfeet.za.net) Received: from leftside.wfeet.za.net (dsl-146-188-25.telkomadsl.co.za [165.146.188.25]) by ctb-mesg8.saix.net (Postfix) with ESMTP id 8FC0D3619; Thu, 17 Aug 2006 10:20:57 +0200 (SAST) Received: from [192.168.0.1] by leftside.wfeet.za.net with esmtp (Exim 4.60 (FreeBSD)) (envelope-from ) id 1GDd7Y-0000TZ-Aa; Thu, 17 Aug 2006 10:20:58 +0200 Message-ID: <44E426E5.2000703@wfeet.za.net> Date: Thu, 17 Aug 2006 10:20:53 +0200 From: Peter van Heusden User-Agent: Thunderbird 1.5.0.5 (Windows/20060719) MIME-Version: 1.0 To: John Baldwin , freebsd-stable@freebsd.org References: <44DED670.9050601@wfeet.za.net> <200608140944.01232.jhb@freebsd.org> <44E0BE16.7000803@wfeet.za.net> <200608141430.38015.jhb@freebsd.org> In-Reply-To: <200608141430.38015.jhb@freebsd.org> X-Enigmail-Version: 0.94.0.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-SA-Exim-Connect-IP: 192.168.0.1 X-SA-Exim-Mail-From: pvh@wfeet.za.net X-SA-Exim-Scanned: No (on leftside.wfeet.za.net); SAEximRunCond expanded to false Cc: Subject: Re: Unexplained kernel panic on 5-STABLE (now in 6-STABLE) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Aug 2006 08:21:05 -0000 Thanks for the advice John. I upgraded to 6-STABLE and just got a kernel panic again. Before I list the dump, I'd like to mention two messages I see in my syslog. Firstly, often I get something like this: kernel: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 151698, size: 28672 (though the last message like this was at 9:18 this morning and the kernel paniced at about 10:04) and secondly, during boot, I get messages like this: kernel: acpi: bad read from port 0xcfc (32) kernel: acpi: bad write to port 0xcf8 (32), val 0x80002084 Anyway, on with the kgdb output: leftside# kgdb kernel.debug /var/crash/vmcore.27 [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd". Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode fault virtual address = 0x1b fault code = supervisor write, page not present instruction pointer = 0x20:0xc08a98ca stack pointer = 0x28:0xcbdd4cc4 frame pointer = 0x28:0xcbdd4ce0 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 34 (pagedaemon) trap number = 12 panic: page fault Uptime: 19h6m20s Dumping 251 MB (2 chunks) chunk 0: 1MB (160 pages) ... ok chunk 1: 251MB (64252 pages) 236 220 204 188 172 156 140 124 108 92 76 60 44 28 12 #0 doadump () at pcpu.h:165 165 __asm __volatile("movl %%fs:0,%0" : "=r" (td)); (kgdb) list *0xc08a98ca 0xc08a98ca is in pmap_ts_referenced (atomic.h:149). 144 static __inline int 145 atomic_cmpset_int(volatile u_int *dst, u_int exp, u_int src) 146 { 147 int res = exp; 148 149 __asm __volatile ( 150 " " __XSTRING(MPLOCKED) " " 151 " cmpxchgl %2,%1 ; " 152 " setz %%al ; " 153 " movzbl %%al,%0 ; " (kgdb) backtrace #0 doadump () at pcpu.h:165 #1 0xc069cee6 in boot (howto=260) at /usr/freebsd6/src/sys/kern/kern_shutdown.c:409 #2 0xc069d17c in panic (fmt=0xc0903b6f "%s") at /usr/freebsd6/src/sys/kern/kern_shutdown.c:565 #3 0xc08ac6d4 in trap_fatal (frame=0xcbdd4c84, eva=27) at /usr/freebsd6/src/sys/i386/i386/trap.c:836 #4 0xc08ac43b in trap_pfault (frame=0xcbdd4c84, usermode=0, eva=27) at /usr/freebsd6/src/sys/i386/i386/trap.c:744 #5 0xc08ac079 in trap (frame= {tf_fs = -874708984, tf_es = -1064697816, tf_ds = -1063452632, tf_edi = -1054911176, tf_esi = 4, tf_ebp = -874689312, tf_isp = -874689360, tf_ebx = -1050447872, tf_edx = -1, tf_ecx = -1038927232, tf_eax = 4, tf_trapno = 12, tf_err = 2, tf_eip = -1064658742, tf_cs = 32, tf_eflags = 66050, tf_esp = -874689320, tf_ss = 0}) at /usr/freebsd6/src/sys/i386/i386/trap.c:434 #6 0xc089a7fa in calltrap () at /usr/freebsd6/src/sys/i386/i386/exception.s:139 #7 0xc08a98ca in pmap_ts_referenced (m=0xc11f5538) at atomic.h:149 #8 0xc0815e59 in vm_pageout_page_stats () at /usr/freebsd6/src/sys/vm/vm_pageout.c:1401 #9 0xc0816192 in vm_pageout () at /usr/freebsd6/src/sys/vm/vm_pageout.c:1546 #10 0xc0687434 in fork_exit (callout=0xc0815ef8 , arg=0x0, frame=0xcbdd4d38) at /usr/freebsd6/src/sys/kern/kern_fork.c:805 #11 0xc089a85c in fork_trampoline () at /usr/freebsd6/src/sys/i386/i386/exception.s:208 Thanks for all the help, Peter John Baldwin wrote: > On Monday 14 August 2006 14:16, Peter van Heusden wrote: > >> Thanks. That gives the following output: >> >> #9 0xc0801295 in trap_pfault (frame=0xd1231b68, usermode=0x0, eva=0x3) >> at /usr/src/sys/i386/i386/trap.c:714 >> #10 0xc0800fa5 in trap (frame= >> {tf_fs = 0xc1e20018, tf_es = 0x10, tf_ds = 0x10, tf_edi = 0x0, >> tf_esi = 0xc1045420, tf_ebp = 0xd1231bb8, tf_isp = 0xd1231b94, tf_ebx = >> 0xc1045458, tf_edx = 0xffffffff, tf_ecx = 0xc28cf000, tf_eax = >> 0xc1045434, tf_trapno = 0xc, tf_err = 0x2, tf_eip = 0xc07b6bbe, tf_cs = >> 0x8, tf_eflags = 0x10286, tf_esp = 0xc102dd08, tf_ss = 0xc1edf570}) >> at /usr/src/sys/i386/i386/trap.c:427 >> #11 0xc07f0eea in calltrap () at /usr/src/sys/i386/i386/exception.s:140 >> ... >> #25 0xc07b6bbe in uma_zalloc_arg (zone=0xc1045420, udata=0x0, flags=0x1) >> at /usr/src/sys/vm/uma_core.c:1895 >> #26 0xc07fc97f in get_pv_entry () at uma.h:276 >> > > So it got a nested page fault inside the VM, basically. > > >> Does this help? It seems that fork() was called, and then something went >> wrong from there. One common feature of these panics seems to be that >> they happen when my server (an aging P3 700Mhz with 256 MB of RAM that >> is put to use for all sorts of network services) is under quite heavy load. >> > > To be honest, I'd update it to 6.1 or even 6-stable as this is likely already > fixed in 6.x. > >