From owner-freebsd-current@FreeBSD.ORG Tue Aug 5 04:12:54 2003 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E4EB837B401 for ; Tue, 5 Aug 2003 04:12:54 -0700 (PDT) Received: from mailbox.univie.ac.at (mailbox.univie.ac.at [131.130.1.27]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9B39D43F3F for ; Tue, 5 Aug 2003 04:12:53 -0700 (PDT) (envelope-from l.ertl@univie.ac.at) Received: from pcle.cc.univie.ac.at (pcle.cc.univie.ac.at [131.130.2.107]) by mailbox.univie.ac.at (8.12.2/8.12.2) with ESMTP id h75BChNL051458 for ; Tue, 5 Aug 2003 13:12:46 +0200 Date: Tue, 5 Aug 2003 13:12:39 +0200 (CEST) From: Lukas Ertl To: freebsd-current@freebsd.org Message-ID: <20030805130808.W574@korben.in.tern> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=ISO-8859-1 Content-Transfer-Encoding: QUOTED-PRINTABLE X-DCC-ZID-Univie-Metrics: mailbox 4244; Body=0 Fuz1=0 Fuz2=0 Subject: panic every few hours, pmap related? X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Aug 2003 11:12:55 -0000 Hi, since this weekend my highly loaded newsserver panics every few hours with the following traceback. Any ideas? 5.1-CURRENT FreeBSD 5.1-CURRENT #6: Mon Aug 4 21:54:06 CEST 2003 Stopped at pmap_remove_all+0x38: xchgl %edx,0(%eax) db> where pmap_remove_all(c0f73de0,40,0,f,c0d5e998) at pmap_remove_all+0x38 vfs_busy_pages(d28d1d48,1,db8a2000,e0ba7b18,c03599d9) at vfs_busy_pages+0x1= 78 bwrite(d28d1d48,e0ba7bc8,c0257f2e,d28d1d48,d28d1e78) at bwrite+0x380 bawrite(d28d1d48,d28d1e78,18,c613a390,c6437b68) at bawrite+0x1c cluster_wbuild(c6437b68,4000,1c2,0,6) at cluster_wbuild+0x90e vfs_bio_awrite(d29fdc08,0,0,c613a390,e0ba7c78) at vfs_bio_awrite+0x25d ffs_fsync(e0ba7cc4,20002,c613a390,c03a38c0,0) at ffs_fsync+0x382 sched_sync(0,e0ba7d48,0,0,0) at sched_sync+0x204 fork_exit(c02620b0,0,e0ba7d48) at fork_exit+0xb1 fork_trampoline() at fork_trampoline+0x1a = --- trap 0x1, eip =3D 0, esp =3D 0xe0ba7d7c, ebp =3D 0 --- Script started on Mon Aug 4 23:57:55 2003 [root@newscore crash]# gdb -k kernel.debug vmcore.0 GNU gdb 5.2.1 (FreeBSD) Copyright 2002 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you ar= e welcome to change it and/or distribute copies of it under certain condition= s. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-undermydesk-freebsd"... panic: from debugger panic messages: --- Fatal trap 12: page fault while in kernel mode cpuid =3D 2; lapic.id =3D 06000000 fault virtual address=09=3D 0xbfceea70 fault code=09=09=3D supervisor write, page not present instruction pointer=09=3D 0x8:0xc035d588 stack pointer=09 =3D 0x10:0xe0ba7a98 frame pointer=09 =3D 0x10:0xe0ba7ab0 code segment=09=09=3D base 0x0, limit 0xfffff, type 0x1b =09=09=09=3D DPL 0, pres 1, def32 1, gran 1 processor eflags=09=3D interrupt enabled, resume, IOPL =3D 0 current process=09=09=3D 41 (syncer) panic: from debugger cpuid =3D 2; lapic.id =3D 06000000 Fatal trap 3: breakpoint instruction fault while in kernel mode cpuid =3D 2; lapic.id =3D 06000000 instruction pointer=09=3D 0x8:0xc0347b65 stack pointer=09 =3D 0x10:0xe0ba7800 frame pointer=09 =3D 0x10:0xe0ba780c code segment=09=09=3D base 0x0, limit 0xfffff, type 0x1b =09=09=09=3D DPL 0, pres 1, def32 1, gran 1 processor eflags=09=3D IOPL =3D 0 current process=09=09=3D 41 (syncer) panic: from debugger cpuid =3D 2; lapic.id =3D 06000000 boot() called on cpu#2 Uptime: 1h36m33s Dumping 1023 MB 16 32 48 64 80 96 112 128 144 160 176 192 208 224 240 256 272 288 304 320 = 336 352 368 384 400 416 432 448 464 480 496 512 528 544 560 576 592 608 624= 640 656 672 688 704 720 736 752 768 784 800 816 832 848 864 880 896 912 92= 8 944 960 976 992 1008 --- Reading symbols from /usr/obj/usr/src/sys/NEWSCORE/modules/usr/src/sys/modu= les/acpi/acpi.ko.debug...done. Loaded symbols for /usr/obj/usr/src/sys/NEWSCORE/modules/usr/src/sys/module= s/acpi/acpi.ko.debug #0 doadump () at /usr/src/sys/kern/kern_shutdown.c:240 240=09=09dumping++; (kgdb) wher=08 =08=08 =08=08 =08=08 =08bt full #0 doadump () at /usr/src/sys/kern/kern_shutdown.c:240 No locals. #1 0xc0203c61 in boot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:3= 72 No locals. #2 0xc02040b8 in panic () at /usr/src/sys/kern/kern_shutdown.c:550 =09td =3D (struct thread *) 0xc613a390 =09bootopt =3D 260 =09newpanic =3D 0 =09ap =3D 0xe0ba7850 "\by=BA=E0\222\222\024=C0\210=D55=C0" =09buf =3D "from debugger", '\0' #3 0xc0149332 in db_panic () at /usr/src/sys/ddb/db_command.c:450 No locals. #4 0xc0149292 in db_command (last_cmdp=3D0xc03e4a60, cmd_table=3D0xc03bb90= 0, aux_cmd_tablep=3D0xc03b5fb8, aux_cmd_tablep_end=3D0xc03b5fbc) at /usr/src/sys/ddb/db_command.c:346 =09cmd =3D (struct command *) 0xc03799dc =09t =3D 0 =09modif =3D "\0S>=C0=A8\204B=C0\230x=BA=E0\r\0\0\0 pA=C0\r\0\0\0\001\0\0\0= =B8x=BA=E0\226=D63=C0 V@=C0\aK\0 =A0pA=C0@=CC?=C0=C0S>=C0x\0\0\0=C0S>=C0=A8= \204B=C0=DCx=BA=E0=D1=B1\024=C0\f=B38=C0\200=AF\024=C0\0\0\0\0\020\0\0\0=E8= x=BA=E0=F8x=BA=E0]=A8\024=C0\f=B38=C0=A8\204B=C0\by=BA=E0\020\0\0" =09addr =3D -1070213752 =09count =3D 1 =09have_addr =3D 0 =09result =3D 0 #5 0xc01493d5 in db_command_loop () at /usr/src/sys/ddb/db_command.c:472 No locals. #6 0xc014c3f5 in db_trap (type=3D12, code=3D0) at /usr/src/sys/ddb/db_trap= =2Ec:73 =09bkpt =3D 0 #7 0xc034785c in kdb_trap (type=3D12, code=3D0, regs=3D0xe0ba7a58) at /usr/src/sys/i386/i386/db_interface.c:172 =09ef =3D 582 =09ddb_mode =3D 1 #8 0xc0361c16 in trap_fatal (frame=3D0xe0ba7a58, eva=3D0) at /usr/src/sys/i386/i386/trap.c:816 =09code =3D 16 ---Type to continue, or q to quit--- =09type =3D 12 =09ss =3D 16 =09esp =3D 0 =09softseg =3D {ssd_base =3D 0, ssd_limit =3D 1048575, ssd_type =3D 27, ssd_dpl =3D 0, ssd_p =3D 1, ssd_xx =3D 0, ssd_xx1 =3D 0, ssd_def32 =3D 1,= ssd_gran =3D 1} #9 0xc03618c2 in trap_pfault (frame=3D0xe0ba7a58, usermode=3D0, eva=3D3218= 008688) at /usr/src/sys/i386/i386/trap.c:735 =09va =3D 3218006016 =09vm =3D (struct vmspace *) 0x0 =09map =3D 0x1 =09rv =3D 1 =09ftype =3D 2 '\002' =09td =3D (struct thread *) 0xc613a390 =09p =3D (struct proc *) 0xc6139d3c #10 0xc036141d in trap (frame=3D {tf_fs =3D -963313640, tf_es =3D 409075728, tf_ds =3D -474808304, tf_= edi =3D -1057538592, tf_esi =3D 16, tf_ebp =3D -524649808, tf_isp =3D -5246= 49852, tf_ebx =3D -1044790456, tf_edx =3D 0, tf_ecx =3D -474795080, tf_eax = =3D -1076958608, tf_trapno =3D 12, tf_err =3D 2, tf_eip =3D -1070213752, tf= _cs =3D 8, tf_eflags =3D 66118, tf_esp =3D -963262032, tf_ss =3D 1000980480= }) at /usr/src/sys/i386/i386/trap.c:420 =09td =3D (struct thread *) 0xc613a390 =09p =3D (struct proc *) 0xc6139d3c =09sticks =3D 3323175824 =09i =3D 0 =09ucode =3D 0 =09type =3D 12 =09code =3D 2 =09eva =3D 3218008688 #11 0xc0349298 in calltrap () at {standard input}:103 No locals. #12 0xc0253a08 in vfs_busy_pages (bp=3D0xc0f73de0, clear_modify=3D1) at /usr/src/sys/kern/vfs_bio.c:3370 =09m =3D 0xc1b9c348 =09obj =3D 0x0 =09foff =3D 7438336 ---Type to continue, or q to quit--- =09i =3D 16 =09bogus =3D 0 #13 0xc024df20 in bwrite (bp=3D0xd28d1d48) at /usr/src/sys/kern/vfs_bio.c:8= 59 =09oldflags =3D 1677721604 =09newbp =3D (struct buf *) 0xd298d638 #14 0xc024eb0c in bawrite (bp=3D0x0) at /usr/src/sys/kern/vfs_bio.c:1148 No locals. #15 0xc0257f2e in cluster_wbuild (vp=3D0xc6437b68, size=3D16384, start_lbn= =3D456, len=3D6) at /usr/src/sys/kern/vfs_cluster.c:985 =09bp =3D (struct buf *) 0xd28d1d48 =09tbp =3D (struct buf *) 0xd298d638 =09i =3D 6 =09j =3D 4 =09totalwritten =3D 98304 =09dbsize =3D 32 #16 0xc02504dd in vfs_bio_awrite (bp=3D0xd29fdc08) at /usr/src/sys/kern/vfs_bio.c:1691 =09i =3D 6 =09j =3D 0 =09lblkno =3D 450 =09vp =3D (struct vnode *) 0xc6437b68 =09ncl =3D 0 =09nwritten =3D 0 =09size =3D 16384 =09maxcl =3D 8 #17 0xc02f6872 in ffs_fsync (ap=3D0xe0ba7cc4) at /usr/src/sys/ufs/ffs/ffs_vnops.c:268 =09vp =3D (struct vnode *) 0xc6437b68 =09ip =3D (struct inode *) 0xd29fdc08 =09bp =3D (struct buf *) 0xd29fdc08 =09nbp =3D (struct buf *) 0xd2b9cf00 =09error =3D 0 =09wait =3D 0 =09passes =3D 4 =09skipmeta =3D 0 ---Type to continue, or q to quit--- =09lbn =3D 456 #18 0xc02622b4 in sched_sync () at vnode_if.h:627 =09slp =3D (struct synclist *) 0xc61994ec =09vp =3D (struct vnode *) 0xc6437b68 =09mp =3D (struct mount *) 0xc636b200 =09starttime =3D 1060033156 =09td =3D (struct thread *) 0xc613a390 #19 0xc01ec621 in fork_exit (callout=3D0xc02620b0 , arg=3D0x0, frame=3D0x0) at /usr/src/sys/kern/kern_fork.c:794 =09td =3D (struct thread *) 0x0 =09p =3D (struct proc *) 0xc6139d3c (kgdb) quit [root@newscore crash]# exit Script done on Mon Aug 4 23:58:35 2003 --=20 Lukas Ertl eMail: l.ertl@univie.ac.at UNIX Systemadministrator Tel.: (+43 1) 4277-14073 Vienna University Computer Center Fax.: (+43 1) 4277-9140 University of Vienna http://mailbox.univie.ac.at/~le/