From owner-freebsd-questions@FreeBSD.ORG Sun Mar 16 01:00:09 2014 Return-Path: Delivered-To: questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BDAD0B69 for ; Sun, 16 Mar 2014 01:00:09 +0000 (UTC) Received: from zoom.lafn.org (zoom.lafn.org [108.92.93.123]) by mx1.freebsd.org (Postfix) with ESMTP id 9ECE096C for ; Sun, 16 Mar 2014 01:00:09 +0000 (UTC) Received: from [10.0.1.3] (static-71-177-216-148.lsanca.fios.verizon.net [71.177.216.148]) (authenticated bits=0) by zoom.lafn.org (8.14.7/8.14.2) with ESMTP id s2G0jNa7085923 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO) for ; Sat, 15 Mar 2014 17:45:23 -0700 (PDT) (envelope-from bc979@lafn.org) From: Doug Hardie Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Subject: Frequent Page Faults Message-Id: Date: Sat, 15 Mar 2014 17:45:23 -0700 To: "questions@freebsd.org FreeBSD" Mime-Version: 1.0 (Mac OS X Mail 6.6 \(1510\)) X-Mailer: Apple Mail (2.1510) X-Virus-Scanned: clamav-milter 0.98 at zoom.lafn.org X-Virus-Status: Clean X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 16 Mar 2014 01:00:09 -0000 I have a system running: FreeBSD zoon.lafn.org 7.2-RELEASE-p5 FreeBSD 7.2-RELEASE-p5 #3: Thu Aug = 19 20:09:11 PDT 2010 This morning it started crashing frequently. The system had no issues = prior to today. Sometimes it auto reboots, others it just hangs. The = console messages are always very similar. I do have core dumps for 3 of = them. At first it appeared that the problem was being caused by an = attack on port 110. While the attack was in process, the system would = stay up for only a few minutes. After discovering and blocking the = attack, the system remained up a couple hours but crashed while I was = watching it. I am beginning to suspect a HW issue that was worsened by = the load of the attack, but not caused directly by the attack. I have been in the process of building a new set of disks for this = system using 9.2, but thats not complete yet. In addition it will take = a couple days to get the disks on site. Its a remote facility. I found = several references to issues with this problem and 7.2 that have = apparently been fixed. However, I have been running 7.2 since it first = came out on this system without any similar issues. Actually, I don't = recall any issues at all with 7.2. The source for this system no longer = exists. My development system has been upgraded to 9.2. Its a modified = kernel with some of the older processors commented out and includes = QUOTA and ALTQ. I don't recall any other changes. I can hurry up the setup of the 9.2 system, but it would be at least = until Wed before it could be installed at tried. If that would correct = the problem that would be great. However, I have a concern that there = is also a HW issue here and am not sure how to identify such. My review = of the dumps shows that acpi is always involved. Don't know for sure = what that implies though. I don't believe the os has degraded and it = has not been touched since 2010. That pretty much leaves me with a HW = issue. Any ideas where the problem is will be appreciated. Here are a couple of the dumps. The first one was while the attack was = in progress. The second after it was terminated. -------------------------------------------------------------- zoon# kgdb /boot/kernel/kernel vmcore.0 GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you = are welcome to change it and/or distribute copies of it under certain = conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for = details. This GDB was configured as "i386-marcel-freebsd"... Unread portion of the kernel message buffer: 3) trap number =3D 12 panic: page fault cpuid =3D 3 Uptime: 3m48s Physical memory: 1993 MB Dumping 187 MB: 172 156 140 124 Fatal trap 12: page fault while in kernel mode cpuid =3D 1; apic id =3D 01 fault virtual address =3D 0x4 fault code =3D supervisor write, page not present instruction pointer =3D 0x20:0xc0c72c00 stack pointer =3D 0x28:0xe566faf4 frame pointer =3D 0x28:0xe566fb14 code segment =3D base 0x0, limit 0xfffff, type 0x1b =3D DPL 0, pres 1, def32 1, gran 1 processor eflags =3D interrupt enabled, resume, IOPL =3D 0 current process =3D 4 (g_down) trap number =3D 12 108 92 76 60 44 28 12 Reading symbols from /boot/kernel/fdescfs.ko...Reading symbols from = /boot/kernel/fdescfs.ko.symbols...done. done. Loaded symbols for /boot/kernel/fdescfs.ko Reading symbols from /boot/kernel/pflog.ko...Reading symbols from = /boot/kernel/pflog.ko.symbols...done. done. Loaded symbols for /boot/kernel/pflog.ko Reading symbols from /boot/kernel/pf.ko...Reading symbols from = /boot/kernel/pf.ko.symbols...done. done. Loaded symbols for /boot/kernel/pf.ko Reading symbols from /boot/kernel/acpi.ko...Reading symbols from = /boot/kernel/acpi.ko.symbols...done. done. Loaded symbols for /boot/kernel/acpi.ko #0 doadump () at pcpu.h:196 196 pcpu.h: No such file or directory. in pcpu.h (kgdb) where #0 doadump () at pcpu.h:196 #1 0xc07a5e27 in boot (howto=3D260) at = /usr2/src/sys/kern/kern_shutdown.c:418 #2 0xc07a60f9 in panic (fmt=3DVariable "fmt" is not available. ) at /usr2/src/sys/kern/kern_shutdown.c:574 #3 0xc0aa792c in trap_fatal (frame=3D0xc4ff8c48, eva=3D1361334589) at /usr2/src/sys/i386/i386/trap.c:939 #4 0xc0aa7b90 in trap_pfault (frame=3D0xc4ff8c48, usermode=3D0, = eva=3D1361334589) at /usr2/src/sys/i386/i386/trap.c:852 #5 0xc0aa8512 in trap (frame=3D0xc4ff8c48) at = /usr2/src/sys/i386/i386/trap.c:530 #6 0xc0a8d62b in calltrap () at /usr2/src/sys/i386/i386/exception.s:159 #7 0xc0e21715 in acpi_cpu_c1 () at = /usr2/src/sys/modules/acpi/acpi/../../../i386/acpica/acpi_machdep.c:550 #8 0xc0e1a594 in acpi_cpu_idle () at = /usr2/src/sys/modules/acpi/acpi/../../../dev/acpica/acpi_cpu.c:943 #9 0xc0a97f78 in cpu_idle () at /usr2/src/sys/i386/i386/machdep.c:1183 #10 0xc07c7904 in sched_idletd (dummy=3D0x0) at /usr2/src/sys/kern/sched_ule.c:2681 #11 0xc07808d9 in fork_exit (callout=3D0xc07c7640 , = arg=3D0x0,=20 frame=3D0xc4ff8d38) at /usr2/src/sys/kern/kern_fork.c:810 #12 0xc0a8d6a0 in fork_trampoline () at = /usr2/src/sys/i386/i386/exception.s:264 -------------------------------------------------------------- zoon# kgdb /boot/kernel/kernel vmcore.2 GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you = are welcome to change it and/or distribute copies of it under certain = conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for = details. This GDB was configured as "i386-marcel-freebsd"... Unread portion of the kernel message buffer: Fatal double fault: eip =3D 0xc0e21715 esp =3D 0xc4ff8d80 ebp =3D 0xc4ff8c88 cpuid =3D 3; apic id =3D 03 panic: double fault cpuid =3D 3 Uptime: 2h19m49s Physical memory: 1993 MB Dumping 187 MB: 172 156 140 124 108 92 76 60 44 28 12 Reading symbols from /boot/kernel/fdescfs.ko...Reading symbols from = /boot/kernel/fdescfs.ko.symbols...done. done. Loaded symbols for /boot/kernel/fdescfs.ko Reading symbols from /boot/kernel/pflog.ko...Reading symbols from = /boot/kernel/pflog.ko.symbols...done. done. Loaded symbols for /boot/kernel/pflog.ko Reading symbols from /boot/kernel/pf.ko...Reading symbols from = /boot/kernel/pf.ko.symbols...done. done. Loaded symbols for /boot/kernel/pf.ko Reading symbols from /boot/kernel/acpi.ko...Reading symbols from = /boot/kernel/acpi.ko.symbols...done. done. Loaded symbols for /boot/kernel/acpi.ko #0 doadump () at pcpu.h:196 196 pcpu.h: No such file or directory. in pcpu.h (kgdb) up #1 0xc07a5e27 in boot (howto=3D260) at = /usr2/src/sys/kern/kern_shutdown.c:418 418 /usr2/src/sys/kern/kern_shutdown.c: No such file or directory. in /usr2/src/sys/kern/kern_shutdown.c (kgdb) down #0 doadump () at pcpu.h:196 196 pcpu.h: No such file or directory. in pcpu.h (kgdb) where #0 doadump () at pcpu.h:196 #1 0xc07a5e27 in boot (howto=3D260) at = /usr2/src/sys/kern/kern_shutdown.c:418 #2 0xc07a60f9 in panic (fmt=3DVariable "fmt" is not available. ) at /usr2/src/sys/kern/kern_shutdown.c:574 #3 0xc0aa763b in dblfault_handler () at = /usr2/src/sys/i386/i386/trap.c:972 #4 0xc0e21715 in acpi_cpu_c1 () at = /usr2/src/sys/modules/acpi/acpi/../../../i386/acpica/acpi_machdep.c:550 #5 0xc0e1a594 in acpi_cpu_idle () at = /usr2/src/sys/modules/acpi/acpi/../../../dev/acpica/acpi_cpu.c:943 #6 0xc0a97f78 in cpu_idle () at /usr2/src/sys/i386/i386/machdep.c:1183 #7 0xc07c7904 in sched_idletd (dummy=3D0x0) at /usr2/src/sys/kern/sched_ule.c:2681 #8 0xc07808d9 in fork_exit (callout=3D0xc07c7640 , = arg=3D0x0,=20 frame=3D0xc4ff8d38) at /usr2/src/sys/kern/kern_fork.c:810 #9 0xc0a8d6a0 in fork_trampoline () at = /usr2/src/sys/i386/i386/exception.s:264 (kgdb)=20