Date: Mon, 24 Aug 2009 07:41:10 GMT From: Zete Tresor <zetinja@gmail.com> To: freebsd-gnats-submit@FreeBSD.org Subject: i386/138126: Kernel panic trap 12 on bigger load Message-ID: <200908240741.n7O7fAwi043291@www.freebsd.org> Resent-Message-ID: <200908240750.n7O7o1q8042657@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 138126
>Category: i386
>Synopsis: Kernel panic trap 12 on bigger load
>Confidential: no
>Severity: critical
>Priority: high
>Responsible: freebsd-i386
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Mon Aug 24 07:50:01 UTC 2009
>Closed-Date:
>Last-Modified:
>Originator: Zete Tresor
>Release: 7.2
>Organization:
>Environment:
FreeBSD webhosting3 7.2-RELEASE FreeBSD 7.2-RELEASE #0: Fri May 1 08:49:13 UTC 2009 root@walker.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC i386
>Description:
Server crashes on bigger load with Kernel Trap 12 page fault panic. I've vmware virtual server that has 25 ezjail jails on it. jails contain various stuff, mostly apache, proftpd. Sometimes machine works up to 7 - 8 hours, sometime it does not even boot properly - it crashes upon enabling jails. it worked fine for 2 months, but started to crash when more and more jails where added. Works fine on other kinds of load, crashes only when many jails are up. It's multiprocessor machine w/ 4 CPU's and 4GB of RAM. Most times but not always it just freezes and does not reboot. Environment shows os w/ no pathes, at the moment of the first crash there were all patches until august 20th applied.
Here are errors I see on crash:
kernel trap 12 with interrupts disabled
Fatal trap 12: page fault while in kernel mode
cpuid = 2; apic id = 02
fault virtual address = 0x10
fault code = supervisor read, page not present
instruction pointer = 0x20:0xc07d4a21
stack pointer = 0x28:0xc72dec1c
frame pointer = 0x28:0xc72dec30
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = resume, IOPL = 0
current process = 36 (sctp_iterator)
trap number = 12
panic: page fault
cpuid = 2
Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 01
fault virtual address = 0x6e727598
fault code = supervisor read, page not present
instruction pointer = 0x20:0xc08bf945
stack pointer = 0x28:0xc72bfaa8
frame pointer = 0x28:0xc72bfacc
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 30 (em1 taskq)
trap number = 12
kernel trap 12 with interrupts disabled
Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address = 0x10
fault code = supervisor read, page not present
instruction pointer = 0x20:0xc07d4a21
stack pointer = 0x28:0xc73e29c4
frame pointer = 0x28:0xc73e29d8
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = resume, IOPL = 0
current process = 9020 (libssl.so)
trap number = 12
panic: page fault
cpuid = 0
Uptime: 25m37s
I run kgdb on kernel crash dump file, and this is the result:
# kgdb kernel.debug /var/crash/vmcore.2
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd"...
Unread portion of the kernel message buffer:
kernel trap 12 with interrupts disabled
Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 01
fault virtual address = 0x10
fault code = supervisor read, page not present
instruction pointer = 0x20:0xc07d4a21
stack pointer = 0x28:0xe9c879c4
frame pointer = 0x28:0xe9c879d8
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = resume, IOPL = 0
current process = 18223 (libssl.so)
trap number = 12
panic: page fault
cpuid = 1
Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 01
fault virtual address = 0x6e727598
fault code = supervisor read, page not present
instruction pointer = 0x20:0xc08bf945
stack pointer = 0x28:0xc72bfaa8
frame pointer = 0x28:0xc72bfacc
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 30 (em1 taskq)
trap number = 12
panic: page fault
cpuid = 1
Uptime: 12h20m45s
Physical memory: 3827 MB
Dumping 295 MB: 280 264 248 232 216 200 184 168 152 136 120 104 88 72 56 40 24 8
Reading symbols from /boot/kernel/acpi.ko...Reading symbols from /boot/kernel/acpi.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/acpi.ko
Reading symbols from /boot/kernel/linux.ko...Reading symbols from /boot/kernel/linux.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/linux.ko
Reading symbols from /boot/kernel/nullfs.ko...Reading symbols from /boot/kernel/nullfs.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/nullfs.ko
Reading symbols from /boot/kernel/fdescfs.ko...Reading symbols from /boot/kernel/fdescfs.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/fdescfs.ko
#0 doadump () at pcpu.h:196
196 __asm __volatile("movl %%fs:0,%0" : "=r" (td));
(kgdb)
>How-To-Repeat:
Install freebsd 7.2 on vmware, patch it, make >20 apache jails. load the server. aait for crash.
>Fix:
>Release-Note:
>Audit-Trail:
>Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200908240741.n7O7fAwi043291>