Date: Tue, 20 Feb 2001 16:48:55 -0800 (PST) From: scott@vicor-nb.com (Scott Macy) To: freebsd-questions@FreeBSD.ORG Cc: milt@vicor-nb.com (milt), cayford@vicor-nb.com (Cayford Burrell), jpl@vicor-nb.com (John Lynch), jrh@vicor-nb.com (Josh Howard), scott@vicor-nb.com (Scott Macy), julian@vicor-nb.com (Julian Elischer) Subject: Help with panic: page fault in FBSD 4.1.1 Message-ID: <20010221004855.83D4729A@muir.vicor-nb.com>
next in thread | raw e-mail | index | archive | help
We have been upgrading about 300 PCs at our clients production sites
from FreeBSD 2.2.6 to FreeBSD 4.1.1. On two of the oldest machines we
have had frequent (daily or so) Kernel crashes with "panic: page fault".
We have two kernel core dumps which show the exact same stack trace.
We are not seeing these crashes on the newer machines with 4.1.1 running
the same application, and the machines that are crashing were stable
under 2.2.6.
I've looked though the mail archives, and the GNATS database and not
found a match to our problem.
What's going on?? What can we do to elminate the crashes?? Is there
some Kernel config setting that might be causing the problem? Is there
another option than "buy new hw?"
Some Details:
The machines that are crashing are "server" machines, with 6x60GB
RAIDs, mostly running our own "QFT" (Quick File Transfer)
daemons that do *lots* of network and disk IO. The crashing machines
are Pentuim (Pro?) 200's (see dmesg below). We are not seeing the same
crash problem on user workstations that are of the same vintage,
same OS, but not as heavily loaded.
We swapped hardware on one of these older machines with a newer one, and
the problem went away when the same disk was used with Pentium II 350
CPU.
We now have trapped two kernel core dumps and both give exactly the
same information. Below is Kernel Debug Stack Trace info, then Dmesg
info.
The process that is running at the time of the panic is qftListener,
which is the QFT daemon mentioned above. We have two core
dumps, both time's it's in a qftListener "open()" call.
Thanks,
-Scott Macy
Gory Details:
===================================================================
bigwoop Feb 20 10:54am ~cayford/oos0b_crash 107: gdb -k kern* vm*
GNU gdb 4.18
Copyright 1998 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-unknown-freebsd"...
(no debugging symbols found)...
IdlePTD 4358144
initial pcb at 387100
panicstr: page fault
panic messages:
---
Fatal trap 12: page fault while in kernel mode
fault virtual address = 0x14
fault code = supervisor read, page not present
instruction pointer = 0x8:0xc015129b
stack pointer = 0x10:0xc8f58a08
frame pointer = 0x10:0xc8f58a70
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 22304 (qftListener)
interrupt mask =
trap number = 12
panic: page fault
syncing disks... 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30
giving up on 26 buffers
Uptime: 1d9h32m48s
(da1:ahc1:0:0:0): Synchronize cache failed, status == 0x34, scsi status == 0x0
(da2:ahc1:0:1:0): Synchronize cache failed, status == 0xb, scsi status == 0x0
(da3:ahc1:0:2:0): Synchronize cache failed, status == 0xb, scsi status == 0x0
dumping to dev #da/0x20001, offset 1310720
dump 128 127 126 125 124 123 122 121 120 119 118 117 116 115 114 113 112 111 110
109 108 107 106 105 104 103 102 101 100 99 98 97 96 95 94 93 92 91 90 89 88 87
86 85 84 83 82 81 80 79 78 77 76 75 74 73 72 71 70 69 68 67 66 65 64 63 62 61 60
59 58 57 56 55 54 53 52 51 50 49 48 47 46 45 44 43 42 41 40 39 38 37 36 35 34 3
3 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5
4 3 2 1
---
#0 0xc0193748 in boot ()
(kgdb) bt
#0 0xc0193748 in boot ()
#1 0xc0193acc in poweroff_wait ()
#2 0xc02ede85 in trap_fatal ()
#3 0xc02edb5d in trap_pfault ()
#4 0xc02ed717 in trap ()
#5 0xc015129b in ahc_action ()
#6 0xc012650f in xpt_run_dev_sendq ()
#7 0xc01258ff in xpt_action ()
#8 0xc012cd60 in dastart ()
#9 0xc01262b8 in xpt_run_dev_allocq ()
#10 0xc01261e7 in xpt_schedule ()
#11 0xc012c2c8 in dastrategy ()
#12 0xc019c9e9 in diskstrategy ()
#13 0xc01c8ed0 in spec_strategy ()
#14 0xc01c89a5 in spec_vnoperate ()
#15 0xc0292ddd in ufs_vnoperatespec ()
#16 0xc0292845 in ufs_strategy ()
#17 0xc0292dad in ufs_vnoperate ()
#18 0xc01b61e6 in bread ()
#19 0xc0289653 in ffs_blkatoff ()
#20 0xc028dfe9 in ufs_lookup ()
#21 0xc0292dad in ufs_vnoperate ()
#22 0xc01b9bf9 in vfs_cache_lookup ()
---Type <return> to continue, or q <return> to quit---
#23 0xc0292dad in ufs_vnoperate ()
#24 0xc01bc988 in lookup ()
#25 0xc01bc484 in namei ()
#26 0xc01c490a in vn_open ()
#27 0xc01c0c85 in open ()
#28 0xc02ee131 in syscall2 ()
#29 0xc02dfca5 in Xint0x80_syscall ()
#30 0x8050a69 in ?? ()
#31 0x8051a98 in ?? ()
#32 0x804a46d in ?? ()
(kgdb) h
===================================================================
Dmesg:
Copyright (c) 1992-2000 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD 4.1.1-RELEASE #0: Tue Sep 26 00:46:59 GMT 2000
jkh@narf.osd.bsdi.com:/usr/src/sys/compile/GENERIC
Timecounter "i8254" frequency 1193182 Hz
CPU: Pentium/P55C (200.46-MHz 586-class CPU)
Origin = "GenuineIntel" Id = 0x544 Stepping = 4
Features=0x8001bf<FPU,VME,DE,PSE,TSC,MSR,MCE,CX8,MMX>
real memory = 134217728 (131072K bytes)
avail memory = 126521344 (123556K bytes)
Preloaded elf kernel "kernel" at 0xc0416000.
Intel Pentium detected, installing workaround for F00F bug
md0: Malloc disk
npx0: <math processor> on motherboard
npx0: INT 16 interface
pcib0: <Host to PCI bridge> on motherboard
pci0: <PCI bus> on pcib0
isab0: <Intel 82371AB PCI to ISA bridge> at device 7.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PIIX4 ATA33 controller> port 0xf000-0xf00f at device 7.1 on pci0
ata0: at 0x1f0 irq 14 on atapci0
ata1: at 0x170 irq 15 on atapci0
uhci0: <Intel 82371AB/EB (PIIX4) USB controller> port 0x6400-0x641f irq 11 at device 7.2 on pci0
usb0: <Intel 82371AB/EB (PIIX4) USB controller> on uhci0
usb0: USB revision 1.0
uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
chip1: <Intel 82371AB Power management controller> port 0x5f00-0x5f0f at device 7.3 on pci0
ahc0: <Adaptec 2940 Ultra SCSI adapter> port 0x6800-0x68ff mem 0xe4100000-0xe4100fff irq 10 at device 9.0 on pci0
aic7880: Wide Channel A, SCSI Id=7, 16/255 SCBs
ahc1: <Adaptec 2940 Ultra SCSI adapter> port 0x6c00-0x6cff mem 0xe4102000-0xe4102fff irq 5 at device 10.0 on pci0
aic7880: Wide Channel A, SCSI Id=7, 16/255 SCBs
pci0: <S3 ViRGE DX/GX graphics accelerator> at 11.0 irq 9
fxp0: <Intel Pro 10/100B/100+ Ethernet> port 0x7000-0x701f mem 0xe4000000-0xe40fffff,0xe4101000-0xe4101fff irq 11 at device 12.0 on pci0
fxp0: Ethernet address 00:a0:c9:5b:f7:78
eisa0: <EISA bus> on motherboard
eisa0: unknown card ADP7881 (0x04907881) at slot 6
fdc0: <NEC 72065B or clone> at port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on isa0
fdc0: FIFO enabled, 8 bytes threshold
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> flags 0x1 irq 1 on atkbdc0
kbd0 at atkbd0
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: model Generic PS/2 mouse, device ID 0
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
sio0: type 16550A
sio1 at port 0x2f8-0x2ff irq 3 on isa0
sio1: type 16550A
ppc0: <Parallel port> at port 0x378-0x37f irq 7 on isa0
ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
plip0: <PLIP network interface> on ppbus0
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
Waiting 15 seconds for SCSI devices to settle
da0 at ahc0 bus 0 target 0 lun 0
da0: <SEAGATE ST19171N 0023> Fixed Direct Access SCSI-2 device
da0: 10.000MB/s transfers (10.000MHz, offset 15)
da0: 8683MB (17783112 512 byte sectors: 255H 63S/T 1106C)
da1 at ahc1 bus 0 target 0 lun 0
da1: <Seek seek5x18a-P0 0204> Fixed Direct Access SCSI-2 device
da1: 20.000MB/s transfers (10.000MHz, offset 8, 16bit)
da1: 61220MB (125380096 512 byte sectors: 255H 63S/T 7804C)
da2 at ahc1 bus 0 target 1 lun 0
da2: <Seek seek5x18b-P0 0204> Fixed Direct Access SCSI-2 device
da2: 20.000MB/s transfers (10.000MHz, offset 8, 16bit)
da2: 61220MB (125380096 512 byte sectors: 255H 63S/T 7804C)
da3 at ahc1 bus 0 target 2 lun 0
da3: <Seek seek5x18c-P0 0204> Fixed Direct Access SCSI-2 device
da3: 20.000MB/s transfers (10.000MHz, offset 8, 16bit)
da3: 61220MB (125380096 512 byte sectors: 255H 63S/T 7804C)
Mounting root from ufs:/dev/da0s1a
WARNING: / was not properly dismounted
===================================================================
And /var/log/messages give no hit to the problem:
===================================================================
Feb 15 20:12:26 oos0b inetd[149]: ntalk/udp: no such user 'tty', service ignored
Feb 15 20:19:49 oos0b ntpd[105]: time reset -6.935659 s
Feb 15 20:19:49 oos0b ntpd[105]: kernel pll status change 2041
Feb 16 22:46:37 oos0b mountd[112]: umountall request from 192.168.40.39 from unprivileged port
Feb 16 22:46:40 oos0b mountd[112]: umountall request from 192.168.40.39 from unprivileged port
Feb 16 22:50:30 oos0b mountd[112]: umountall request from 192.168.40.39 from unprivileged port
Feb 16 22:50:34 oos0b mountd[112]: umountall request from 192.168.40.39 from unprivileged port
Feb 16 23:41:17 oos0b mountd[112]: umountall request from 192.168.40.39 from unprivileged port
Feb 16 23:50:57 oos0b last message repeated 3 times
Feb 16 23:54:51 oos0b mountd[112]: umountall request from 192.168.40.39 from unprivileged port
Feb 17 07:29:58 oos0b /kernel: Copyright (c) 1992-2000 The FreeBSD Project.
Feb 17 07:29:58 oos0b /kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
Feb 17 07:29:58 oos0b /kernel: The Regents of the University of California. All rights reserved.
Feb 17 07:29:58 oos0b /kernel: FreeBSD 4.1.1-RELEASE #0: Tue Sep 26 00:46:59 GMT 2000
Feb 17 07:29:58 oos0b /kernel: jkh@narf.osd.bsdi.com:/usr/src/sys/compile/GENERIC
Feb 17 07:29:58 oos0b /kernel: Timecounter "i8254" frequency 1193182 Hz
Feb 17 07:29:58 oos0b /kernel: CPU: Pentium/P55C (200.46-MHz 586-class CPU)
Feb 17 07:29:58 oos0b /kernel: Origin = "GenuineIntel" Id = 0x544 Stepping = 4
Feb 17 07:29:58 oos0b /kernel: Features=0x8001bf<FPU,VME,DE,PSE,TSC,MSR,MCE,CX8,MMX>
Feb 17 07:29:58 oos0b /kernel: real memory = 134217728 (131072K bytes)
Feb 17 07:29:58 oos0b /kernel: avail memory = 126521344 (123556K bytes)
Feb 17 07:29:58 oos0b /kernel: Preloaded elf kernel "kernel" at 0xc0416000.
Feb 17 07:29:58 oos0b /kernel: Intel Pentium detected, installing workaround for F00F bug
Feb 17 07:29:58 oos0b /kernel: md0: Malloc disk
.... etc with reboot.
Feb 17 07:29:59 oos0b /kernel: Mounting root from ufs:/dev/da0s1a
Feb 17 07:29:59 oos0b savecore: reboot after panic: page fault
Feb 17 07:29:59 oos0b savecore: /var/crash/bounds: No such file or directory
Feb 17 07:29:59 oos0b savecore: writing core to /var/crash/vmcore.0
Feb 17 07:30:48 oos0b savecore: writing kernel to /var/crash/kernel.0
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-questions" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20010221004855.83D4729A>