Date: Sat, 13 Jan 2007 09:39:39 GMT From: Mikle Davidkin<skylord@vt.net.ru> To: freebsd-gnats-submit@FreeBSD.org Subject: kern/107864: kernel panic in 6.2-RC2 on heavy network load Message-ID: <200701130939.l0D9ddeu041069@www.freebsd.org> Resent-Message-ID: <200701130940.l0D9eDER027047@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 107864
>Category: kern
>Synopsis: kernel panic in 6.2-RC2 on heavy network load
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Sat Jan 13 09:40:13 GMT 2007
>Closed-Date:
>Last-Modified:
>Originator: Mikle Davidkin
>Release: 6.2-RC2
>Organization:
Vector Telecom
>Environment:
FreeBSD router05.vt.net.ru 6.2-RC2 FreeBSD 6.2-RC2 #0: Mon Jan 8 13:17:00 MSK 2007 root@router05.vt.net.ru:/usr/src/sys/i386/compile/skykernel i386
>Description:
Kernel panics almost every evening when the most heavy network load is on the net. Machine is just a software router for 3 subnets with routed, ipfw, pf (for binat) and no more...
Similar simptoms I have on other software router - will report in other PR and add link to it here.
Kernel is GENERIC with some devices disabled (SCSI/RAID) and few options added:
options DEVICE_POLLING
options HZ=1000
options IPFIREWALL
options DUMMYNET
options IPFIREWALL_FORWARD
dmesg
==============
Copyright (c) 1992-2007 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 6.2-RC2 #0: Mon Jan 8 13:17:00 MSK 2007
root@router05.vt.net.ru:/usr/src/sys/i386/compile/skykernel
ACPI APIC Table: <KT600 AWRDACPI>
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: AMD Athlon(tm) XP 2000+ (1661.94-MHz 686-class CPU)
Origin = "AuthenticAMD" Id = 0x681 Stepping = 1
Features=0x383fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE>
AMD Features=0xc0400800<SYSCALL,MMX+,3DNow+,3DNow>
real memory = 268369920 (255 MB)
avail memory = 252968960 (241 MB)
ioapic0 <Version 0.3> irqs 0-23 on motherboard
kbd1 at kbdmux0
acpi0: <KT600 AWRDACPI> on motherboard
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x4008-0x400b on acpi0
cpu0: <ACPI CPU> on acpi0
acpi_button0: <Power Button> on acpi0
acpi_button1: <Sleep Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
agp0: <VIA 8377 (Apollo KT400/KT400A/KT600) host to PCI bridge> mem 0xe5000000-0xe53fffff at device 0.0 on pci0
pcib1: <PCI-PCI bridge> at device 1.0 on pci0
pci1: <PCI bus> on pcib1
pci1: <display, VGA> at device 0.0 (no driver attached)
stge0: <Sundance ST-1023 Gigabit Ethernet> port 0xc000-0xc0ff mem 0xe5703000-0xe57030ff irq 17 at device 9.0 on pci0
miibus0: <MII bus> on stge0
ip1000phy0: <IC Plus 10/100/1000 media interface> on miibus0
ip1000phy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX, 1000baseTX-FDX, auto
stge0: Ethernet address: 00:15:f2:ef:de:a9
fxp0: <Intel 82559 Pro/100 Ethernet> port 0xc400-0xc43f mem 0xe5701000-0xe5701fff,0xe5400000-0xe54fffff irq 18 at device 10.0 on pci0
miibus1: <MII bus> on fxp0
inphy0: <i82555 10/100 media interface> on miibus1
inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
fxp0: Ethernet address: 00:d0:43:7a:fd:77
fxp1: <Intel 82559 Pro/100 Ethernet> port 0xc800-0xc83f mem 0xe5700000-0xe5700fff,0xe5500000-0xe55fffff irq 19 at device 11.0 on pci0
miibus2: <MII bus> on fxp1
inphy1: <i82555 10/100 media interface> on miibus2
inphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
fxp1: Ethernet address: 00:02:b3:bc:f2:60
fxp2: <Intel 82559 Pro/100 Ethernet> port 0xcc00-0xcc3f mem 0xe5702000-0xe5702fff,0xe5600000-0xe56fffff irq 16 at device 12.0 on pci0
miibus3: <MII bus> on fxp2
inphy2: <i82555 10/100 media interface> on miibus3
inphy2: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
fxp2: Ethernet address: 00:d0:43:7a:fa:99
atapci0: <VIA 8237 UDMA133 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xd000-0xd00f at device 15.0 on pci0
ata0: <ATA channel 0> on atapci0
ata1: <ATA channel 1> on atapci0
uhci0: <VIA 83C572 USB controller> port 0xd400-0xd41f at device 16.0 on pci0
uhci0: [GIANT-LOCKED]
usb0: <VIA 83C572 USB controller> on uhci0
usb0: USB revision 1.0
uhub0: VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhci1: <VIA 83C572 USB controller> port 0xd800-0xd81f at device 16.1 on pci0
uhci1: [GIANT-LOCKED]
usb1: <VIA 83C572 USB controller> on uhci1
usb1: USB revision 1.0
uhub1: VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
uhci2: <VIA 83C572 USB controller> port 0xdc00-0xdc1f at device 16.2 on pci0
uhci2: [GIANT-LOCKED]
usb2: <VIA 83C572 USB controller> on uhci2
usb2: USB revision 1.0
uhub2: VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub2: 2 ports with 2 removable, self powered
uhci3: <VIA 83C572 USB controller> port 0xe000-0xe01f at device 16.3 on pci0
uhci3: [GIANT-LOCKED]
usb3: <VIA 83C572 USB controller> on uhci3
usb3: USB revision 1.0
uhub3: VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub3: 2 ports with 2 removable, self powered
ehci0: <VIA VT6202 USB 2.0 controller> mem 0xe5704000-0xe57040ff at device 16.4 on pci0
ehci0: [GIANT-LOCKED]
usb4: EHCI version 1.0
usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3
usb4: <VIA VT6202 USB 2.0 controller> on ehci0
usb4: USB revision 2.0
uhub4: VIA EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
uhub4: 8 ports with 8 removable, self powered
isab0: <PCI-ISA bridge> at device 17.0 on pci0
isa0: <ISA bus> on isab0
vr0: <VIA VT6102 Rhine II 10/100BaseTX> port 0xe400-0xe4ff mem 0xe5705000-0xe57050ff irq 23 at device 18.0 on pci0
miibus4: <MII bus> on vr0
ukphy0: <Generic IEEE 802.3u media interface> on miibus4
ukphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
vr0: Ethernet address: 00:11:5b:93:c2:3d
acpi_tz0: <Thermal Zone> on acpi0
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: [FAST]
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
sio0: type 16550A
sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0
sio1: type 16550A
ppc0: <Standard parallel printer port> port 0x378-0x37f irq 7 on acpi0
ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
ppbus0: <Parallel port bus> on ppc0
plip0: <PLIP network interface> on ppbus0
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
pmtimer0 on isa0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xcffff,0xd0000-0xd0fff,0xd1000-0xd27ff,0xd3000-0xd3fff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounter "TSC" frequency 1661943183 Hz quality 800
Timecounters tick every 1.000 msec
ipfw2 (+ipv6) initialized, divert loadable, rule-based forwarding enabled, default to deny, logging disabled
ad0: 38166MB <WDC WD400JB-32ENA0 05.03E05> at ata0-master UDMA100
=================
backtrace
============================
1) Jan 8 20:26
# kgdb /boot/kernel/kernel.debug /var/crash/vmcore.6
kgdb: kvm_nlist(_stopped_cpus):
kgdb: kvm_nlist(_stoppcbs):
[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd".
Unread portion of the kernel message buffer:
kernel trap 12 with interrupts disabled
Fatal trap 12: page fault while in kernel mode
fault virtual address = 0x78
fault code = supervisor read, page not present
instruction pointer = 0x20:0xc0578abd
stack pointer = 0x28:0xd40b8ab4
frame pointer = 0x28:0xd40b8ab8
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = resume, IOPL = 0
current process = 11 (swi1: net)
trap number = 12
panic: page fault
Uptime: 6h42m36s
Physical memory: 247 MB
Dumping 34 MB: 19 3
#0 doadump () at pcpu.h:165
165 pcpu.h: No such file or directory.
in pcpu.h
(kgdb) bt
#0 doadump () at pcpu.h:165
#1 0xc05584ee in boot (howto=260) at ../../../kern/kern_shutdown.c:409
#2 0xc0558784 in panic (fmt=0xc0738cc7 "%s") at ../../../kern/kern_shutdown.c:565
#3 0xc070fa60 in trap_fatal (frame=0xd40b8a74, eva=120) at ../../../i386/i386/trap.c:837
#4 0xc070f242 in trap (frame=
{tf_fs = 8, tf_es = 40, tf_ds = 40, tf_edi = -737440936, tf_esi = -1038653568, tf_ebp = -737441096, tf_isp = -737441120, tf_ebx = -1038686208, tf_edx = -1038686208, tf_ecx = 4, tf_eax = -1038653536, tf_trapno = 12, tf_err = 0, tf_eip = -1068004675, tf_cs = 32, tf_eflags = 589831, tf_esp = -1038653568, tf_ss = -737441060}) at ../../../i386/i386/trap.c:270
#5 0xc06fe3fa in calltrap () at ../../../i386/i386/exception.s:139
#6 0xc0578abd in turnstile_setowner (ts=0xc216e800, owner=0x4) at ../../../kern/subr_turnstile.c:432
#7 0xc0578db4 in turnstile_wait (lock=0xc28de168, owner=0x4) at ../../../kern/subr_turnstile.c:591
#8 0xc054ec70 in _mtx_lock_sleep (m=0xc28de168, tid=3256313728, opts=0, file=0x0, line=0) at ../../../kern/kern_mutex.c:579
#9 0xc05d5434 in rt_check (lrt=0xc21767a0, lrt0=0xd40b8b58, dst=0xc288e210) at ../../../net/route.c:1280
#10 0xc05d8432 in arpresolve (ifp=0xc2245400, rt0=0xc25fd420, m=0xc24baa00, dst=0xc288e210, desten=0xd40b8b78 "")
at ../../../netinet/if_ether.c:404
#11 0xc05c7633 in ether_output (ifp=0xc2245400, m=0xc24baa00, dst=0xc288e210, rt0=0xc216e800) at ../../../net/if_ethersubr.c:174
#12 0xc05ed765 in ip_output (m=0xc24baa00, opt=0xc2245400, ro=0xd40b8bd4, flags=1, imo=0x0, inp=0x0)
at ../../../netinet/ip_output.c:777
#13 0xc05ecb8c in ip_forward (m=0xc24baa00, srcrt=0) at ../../../netinet/ip_input.c:1917
#14 0xc05eb6bb in ip_input (m=0xc24baa00) at ../../../netinet/ip_input.c:688
#15 0xc05cff6b in netisr_processqueue (ni=0xc07fc7b8) at ../../../net/netisr.c:236
#16 0xc05d0166 in swi_net (dummy=0x0) at ../../../net/netisr.c:349
#17 0xc05428fd in ithread_execute_handlers (p=0xc2175a78, ie=0xc21a9300) at ../../../kern/kern_intr.c:682
#18 0xc0542a08 in ithread_loop (arg=0xc2143720) at ../../../kern/kern_intr.c:765
#19 0xc0541884 in fork_exit (callout=0xc05429b4 <ithread_loop>, arg=0xc2143720, frame=0xd40b8d38) at ../../../kern/kern_fork.c:821
#20 0xc06fe45c in fork_trampoline () at ../../../i386/i386/exception.s:208
(kgdb) list *0xc0578abd
0xc0578abd is in turnstile_setowner (../../../kern/subr_turnstile.c:433).
428
429 mtx_assert(&td_contested_lock, MA_OWNED);
430 MPASS(owner->td_proc->p_magic == P_MAGIC);
431 MPASS(ts->ts_owner == NULL);
432 ts->ts_owner = owner;
433 LIST_INSERT_HEAD(&owner->td_contested, ts, ts_link);
434 }
435
436 /*
437 * Malloc a turnstile for a new thread, initialize it and return it.
2) Jan 9 15:26
# kgdb /boot/kernel/kernel.debug /var/crash/vmcore.7
kgdb: kvm_nlist(_stopped_cpus):
kgdb: kvm_nlist(_stoppcbs):
[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd".
Unread portion of the kernel message buffer:
#0 doadump () at pcpu.h:165
165 pcpu.h: No such file or directory.
in pcpu.h
(kgdb) bt
#0 doadump () at pcpu.h:165
#1 0xc05584ee in boot (howto=260) at ../../../kern/kern_shutdown.c:409
#2 0xc0558784 in panic (fmt=0xc0738cc7 "%s") at ../../../kern/kern_shutdown.c:565
#3 0xc070fa60 in trap_fatal (frame=0xd40b8a74, eva=120) at ../../../i386/i386/trap.c:837
#4 0xc070f242 in trap (frame=
{tf_fs = 8, tf_es = 40, tf_ds = 40, tf_edi = -737440936, tf_esi = -1038653568, tf_ebp = -737441096, tf_isp = -737441120, tf_ebx = -1038686208, tf_edx = -1038686208, tf_ecx = 4, tf_eax = -1038653536, tf_trapno = 12, tf_err = 0, tf_eip = -1068004675, tf_cs = 32, tf_eflags = 589831, tf_esp = -1038653568, tf_ss = -737441060}) at ../../../i386/i386/trap.c:270
#5 0xc06fe3fa in calltrap () at ../../../i386/i386/exception.s:139
#6 0xc0578abd in turnstile_setowner (ts=0xc216e800, owner=0x4) at ../../../kern/subr_turnstile.c:432
#7 0xc0578db4 in turnstile_wait (lock=0xc2806378, owner=0x4) at ../../../kern/subr_turnstile.c:591
#8 0xc054ec70 in _mtx_lock_sleep (m=0xc2806378, tid=3256313728, opts=0, file=0x0, line=0) at ../../../kern/kern_mutex.c:579
#9 0xc05d5434 in rt_check (lrt=0xc21767a0, lrt0=0xd40b8b58, dst=0xc266ad50) at ../../../net/route.c:1280
#10 0xc05d8432 in arpresolve (ifp=0xc2245400, rt0=0xc2806000, m=0xc25b4100, dst=0xc266ad50, desten=0xd40b8b78 "")
at ../../../netinet/if_ether.c:404
#11 0xc05c7633 in ether_output (ifp=0xc2245400, m=0xc25b4100, dst=0xc266ad50, rt0=0xc216e800) at ../../../net/if_ethersubr.c:174
#12 0xc05ed765 in ip_output (m=0xc25b4100, opt=0xc2245400, ro=0xd40b8bd4, flags=1, imo=0x0, inp=0x0)
at ../../../netinet/ip_output.c:777
#13 0xc05ecb8c in ip_forward (m=0xc25b4100, srcrt=0) at ../../../netinet/ip_input.c:1917
#14 0xc05eb6bb in ip_input (m=0xc25b4100) at ../../../netinet/ip_input.c:688
#15 0xc05cff6b in netisr_processqueue (ni=0xc07fc7b8) at ../../../net/netisr.c:236
#16 0xc05d0166 in swi_net (dummy=0x0) at ../../../net/netisr.c:349
#17 0xc05428fd in ithread_execute_handlers (p=0xc2175a78, ie=0xc21a9300) at ../../../kern/kern_intr.c:682
#18 0xc0542a08 in ithread_loop (arg=0xc2143720) at ../../../kern/kern_intr.c:765
#19 0xc0541884 in fork_exit (callout=0xc05429b4 <ithread_loop>, arg=0xc2143720, frame=0xd40b8d38) at ../../../kern/kern_fork.c:821
#20 0xc06fe45c in fork_trampoline () at ../../../i386/i386/exception.s:208
(kgdb) quit
>How-To-Repeat:
panic occurs in 1-3 days uptume
>Fix:
>Release-Note:
>Audit-Trail:
>Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200701130939.l0D9ddeu041069>