Date: Tue, 29 Mar 2011 08:54:28 GMT From: Sergey Kandaurov <pluknet@gmail.com> To: freebsd-gnats-submit@FreeBSD.org Subject: kern/156026: arpintr()->in_lltable_lookup() 8.1 bce(4) crash Message-ID: <201103290854.p2T8sSBZ077241@red.freebsd.org> Resent-Message-ID: <201103290900.p2T90MDA051426@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 156026
>Category: kern
>Synopsis: arpintr()->in_lltable_lookup() 8.1 bce(4) crash
>Confidential: no
>Severity: non-critical
>Priority: medium
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Tue Mar 29 09:00:22 UTC 2011
>Closed-Date:
>Last-Modified:
>Originator: Sergey Kandaurov
>Release: 8.1-RELEASE-p2
>Organization:
>Environment:
FreeBSD 8.1-RELEASE-p2 amd64
>Description:
It's GENERIC w/ :
options KDB
options DDB
options BREAK_TO_DEBUGGER
options IPFIREWALL
options IPFIREWALL_FORWARD
options IPFIREWALL_VERBOSE
options IPFIREWALL_VERBOSE_LIMIT=500
options IPFIREWALL_DEFAULT_TO_ACCEPT
options LIBALIAS
options IPFIREWALL_NAT
options QUOTA
device carp
There are 3 crashes seen between February and March
on different boxes but identical h/w.
Backtrace of crashed proc is the same.
db> bt
Tracing pid 12 tid 100039 td 0xffffff00029a23e0
propagate_priority() at propagate_priority+0x72
turnstile_wait() at turnstile_wait+0x1aa
_rw_wlock_hard() at _rw_wlock_hard+0xfa
in_lltable_lookup() at in_lltable_lookup+0x12b
arpintr() at arpintr+0x9d6
netisr_dispatch_src() at netisr_dispatch_src+0x7e
ether_demux() at ether_demux+0x14d
ether_input() at ether_input+0x17b
ether_demux() at ether_demux+0x6f
ether_input() at ether_input+0x17b
bce_intr() at bce_intr+0x3b0
intr_event_execute_handlers() at intr_event_execute_handlers+0xfd
ithread_loop() at ithread_loop+0x8e
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff82b155dd30, rbp = 0 ---
Other ddb data available per each crash (attached in shar).
>How-To-Repeat:
Happens by accident on a moderately loaded box. Usual numbers:
2799 processes:12 running, 2750 sleeping, 37 zombie
packets errs idrops bytes packets errs bytes colls
1723 0 0 198578 2166 0 2415553 0
interrupt total rate
irq4: uart0 309128 6
irq15: ata1 35 0
irq17: aac0 13404675 293
cpu0: timer 90914103 1993
irq256: bce0 2914454 63
irq257: bce1 109935240 2410
cpu1: timer 90974578 1994
cpu3: timer 91076473 1997
cpu2: timer 91059038 1996
cpu4: timer 90930409 1993
cpu5: timer 91033366 1996
cpu6: timer 91085723 1997
cpu7: timer 91111107 1997
Total 854748329 18743
>Fix:
Patch attached with submission follows:
# This is a shell archive. Save it in a file, remove anything before
# this line, and then unpack it by entering "sh file". Note, it may
# create directories; files and directories will be owned by you and
# have default permissions.
#
# This archive contains:
#
# crash1.txt
# crash2.txt
# crash3.txt
#
echo x - crash1.txt
sed 's/^X//' >crash1.txt << '24f218c6efe2375d7c9a281e6cc61910'
Xdb> x/s *panicstr
X0: *** error reading from address 0 ***
Xdb> bt
XTracing pid 12 tid 100039 td 0xffffff00029a23e0
Xpropagate_priority() at propagate_priority+0x72
Xturnstile_wait() at turnstile_wait+0x1aa
X_rw_wlock_hard() at _rw_wlock_hard+0xfa
Xin_lltable_lookup() at in_lltable_lookup+0x12b
Xarpintr() at arpintr+0x9d6
Xnetisr_dispatch_src() at netisr_dispatch_src+0x7e
Xether_demux() at ether_demux+0x14d
Xether_input() at ether_input+0x17b
Xether_demux() at ether_demux+0x6f
Xether_input() at ether_input+0x17b
Xbce_intr() at bce_intr+0x3b0
Xintr_event_execute_handlers() at intr_event_execute_handlers+0xfd
Xithread_loop() at ithread_loop+0x8e
Xfork_exit() at fork_exit+0x118
Xfork_trampoline() at fork_trampoline+0xe
X--- trap 0, rip = 0, rsp = 0xffffff82b155dd30, rbp = 0 ---
Xdb> show proc 12
XProcess 12 (intr) at 0xffffff00026e7000:
X state: NORMAL
X uid: 0 gids: 0
X parent: pid 0 at 0xffffffff80c7e3e0
X ABI: null
X threads: 24
X100065 I [irq1: atkbd0]
X100064 I [swi0: uart]
X100063 I [irq15: ata1]
X100062 I [irq14: ata0]
X100045 I [irq22: uhci1 uhci3]
X100040 I [irq23: uhci0 uhci2+]
X100039 L *lle 0xffffff00094123c0 [irq257: bce1]
X100038 I [irq256: bce0]
X100036 I [irq17: aac0]
X100035 I [irq9: acpi0]
X100034 I [swi5: +]
X100032 I [swi2: cambio]
X100028 I [swi6: task queue]
X100027 I [swi6: Giant taskq]
X100020 I [swi3: vm]
X100019 I [swi4: clock]
X100018 I [swi4: clock]
X100017 Run CPU 3 [swi4: clock]
X100016 I [swi4: clock]
X100015 Run CPU 4 [swi4: clock]
X100014 I [swi4: clock]
X100013 I [swi4: clock]
X100012 I [swi4: clock]
X100011 I [swi1: netisr 0]
Xdb> ps
X pid ppid pgrp uid state wmesg wchan cmd
X32064 13554 13554 31888 RL CPU 5 httpd
X32063 15369 15369 33268 S sbwait 0xffffff0054d33234 httpd
X32062 15369 15369 33268 S select 0xffffff0245521bc0 httpd
X32061 15369 15369 33268 S select 0xffffff00286cc0c0 httpd
X32060 15369 15369 33268 S select 0xffffff01e78db640 httpd
X32059 14231 14231 43024 S select 0xffffff0501ab61c0 httpd
X32058 45079 45079 33717 S sbwait 0xffffff0397c31be4 httpd
X32057 45079 45079 33717 S select 0xffffff035d057bc0 httpd
X32056 32055 32055 22 S select 0xffffff0102867640 sshd
X32055 1881 32055 0 Ss sbwait 0xffffff06011a3694 sshd
X32054 13522 13522 33665 S accept 0xffffff0028c7430e httpd
X32053 39869 39869 28053 S accept 0xffffff0397c20dae httpd
X32052 15280 15280 32867 S select 0xffffff002c0dbbc0 httpd
X32051 4253 4253 39534 S select 0xffffff02df7c57c0 httpd
X32050 11560 11560 11477 S select 0xffffff0501aca3c0 httpd
X32049 11560 11560 11477 S select 0xffffff00751ec8c0 httpd
X32048 15369 15369 33268 S select 0xffffff0037190c40 httpd
X32047 15369 15369 33268 S select 0xffffff0037ea24c0 httpd
X32046 45079 45079 33717 S select 0xffffff01e73a17c0 httpd
Xdb> show msgbuf
Xmsgbufp = 0xffffffff81020fe0
Xmagic = 63062, size = 65504, r= 820871, w = 821375, ptr = 0xffffffff81011000, cksum= 5524040
Xkernel trap 12 with interrupts disabled
X
X
XFatal trap 12: page fault while in kernel mode
Xcpuid = 1; apic id = 01
Xfault virtual address = 0xc
Xfault code = supervisor read data, page not present
Xinstruction pointer = 0x20:0xffffffff805f30d2
Xstack pointer = 0x28:0xffffff82b155d830
Xframe pointer = 0x28:0xffffff82b155d860
Xcode segment = base 0x0, limit 0xfffff, type 0x1b
X = DPL 0, pres 1, long 1, def32 0, gran 1
Xprocessor eflags = resume, IOPL = 0
Xcurrent process = 12 (irq257: bce1)
X
Xdb> show allpcpu
XCurrent CPU: 1
X
Xcpuid = 0
Xdynamic pcpu = 0x2a3f80
Xcurthread = 0xffffff04937443e0: pid 31741 "gtar"
Xcurpcb = 0xffffff82b9fead40
Xfpcurthread = 0xffffff04937443e0: pid 31741 "gtar"
Xidlethread = 0xffffff00026ea7c0: pid 11 "idle: cpu0"
Xcurpmap = 0
Xtssp = 0xffffffff80cf0080
Xcommontssp = 0xffffffff80cf0080
Xrsp0 = 0xffffff82b9fead40
Xgs32p = 0xffffffff80ceeeb8
Xldt = 0xffffffff80ceeef8
Xtss = 0xffffffff80ceeee8
X
Xcpuid = 1
Xdynamic pcpu = 0xffffff807f418f80
Xcurthread = 0xffffff00029a23e0: pid 12 "irq257: bce1"
Xcurpcb = 0xffffff82b155dd40
Xfpcurthread = none
Xidlethread = 0xffffff00026eaba0: pid 11 "idle: cpu1"
Xcurpmap = 0
Xtssp = 0xffffffff80cf00e8
Xcommontssp = 0xffffffff80cf00e8
Xrsp0 = 0xffffff82b155dd40
Xgs32p = 0xffffffff80ceef20
Xldt = 0xffffffff80ceef60
Xtss = 0xffffffff80ceef50
X
Xcpuid = 2
Xdynamic pcpu = 0xffffff807f41ff80
Xcurthread = 0xffffff02df2d57c0: pid 73366 "sendbackup"
Xcurpcb = 0xffffff82b7bfad40
Xfpcurthread = none
Xidlethread = 0xffffff00026f6000: pid 11 "idle: cpu2"
Xcurpmap = 0
Xtssp = 0xffffffff80cf0150
Xcommontssp = 0xffffffff80cf0150
Xrsp0 = 0xffffff82b7bfad40
Xgs32p = 0xffffffff80ceef88
Xldt = 0xffffffff80ceefc8
Xtss = 0xffffffff80ceefb8
X
Xcpuid = 3
Xdynamic pcpu = 0xffffff807f426f80
Xcurthread = 0xffffff00026fb000: pid 12 "swi4: clock"
Xcurpcb = 0xffffff80000e2d40
Xfpcurthread = none
Xidlethread = 0xffffff00026f63e0: pid 11 "idle: cpu3"
Xcurpmap = 0
Xtssp = 0xffffffff80cf01b8
Xcommontssp = 0xffffffff80cf01b8
Xrsp0 = 0xffffff80000e2d40
Xgs32p = 0xffffffff80ceeff0
Xldt = 0xffffffff80cef030
Xtss = 0xffffffff80cef020
X
Xcpuid = 4
Xdynamic pcpu = 0xffffff807f42df80
Xcurthread = 0xffffff00026fb7c0: pid 12 "swi4: clock"
Xcurpcb = 0xffffff80000d8d40
Xfpcurthread = none
Xidlethread = 0xffffff00026f67c0: pid 11 "idle: cpu4"
Xcurpmap = 0
Xtssp = 0xffffffff80cf0220
Xcommontssp = 0xffffffff80cf0220
Xrsp0 = 0xffffff80000d8d40
Xgs32p = 0xffffffff80cef058
Xldt = 0xffffffff80cef098
Xtss = 0xffffffff80cef088
X
Xcpuid = 5
Xdynamic pcpu = 0xffffff807f434f80
Xcurthread = 0xffffff008f1b67c0: pid 32064 "httpd"
Xcurpcb = 0xffffff82b6ca0d40
Xfpcurthread = none
Xidlethread = 0xffffff00026e9000: pid 11 "idle: cpu5"
Xcurpmap = 0
Xtssp = 0xffffffff80cf0288
Xcommontssp = 0xffffffff80cf0288
Xrsp0 = 0xffffff82b6ca0d40
Xgs32p = 0xffffffff80cef0c0
Xldt = 0xffffffff80cef100
Xtss = 0xffffffff80cef0f0
X
Xcpuid = 6
Xdynamic pcpu = 0xffffff807f43bf80
Xcurthread = 0xffffff009cc0c000: pid 3649 "gtar"
Xcurpcb = 0xffffff82b8b27d40
Xfpcurthread = 0xffffff009cc0c000: pid 3649 "gtar"
Xidlethread = 0xffffff00026e93e0: pid 11 "idle: cpu6"
Xcurpmap = 0
Xtssp = 0xffffffff80cf02f0
Xcommontssp = 0xffffffff80cf02f0
Xrsp0 = 0xffffff82b8b27d40
Xgs32p = 0xffffffff80cef128
Xldt = 0xffffffff80cef168
Xtss = 0xffffffff80cef158
X
Xcpuid = 7
Xdynamic pcpu = 0xffffff807f442f80
Xcurthread = 0xffffff00026e97c0: pid 11 "idle: cpu7"
Xcurpcb = 0xffffff800009cd40
Xfpcurthread = none
Xidlethread = 0xffffff00026e97c0: pid 11 "idle: cpu7"
Xcurpmap = 0
Xtssp = 0xffffffff80cf0358
Xcommontssp = 0xffffffff80cf0358
Xrsp0 = 0xffffff800009cd40
Xgs32p = 0xffffffff80cef190
Xldt = 0xffffffff80cef1d0
Xtss = 0xffffffff80cef1c0
X
Xdb> show lockedvnods
XLocked vnodes
X
X0xffffff01ced5f3b0: tag ufs, type VREG
X usecount 1, writecount 0, refcount 39883 mountedhere 0
X flags ()
X v_object 0xffffff008f8ff948 ref 0 pages 159524
X lock type ufs: SHARED (count 1)
X ino 108758910, on dev aacd0s1g
X
X0xffffff0397e37ce8: tag ufs, type VDIR
X usecount 1, writecount 0, refcount 2 mountedhere 0
X flags ()
X v_object 0xffffff0397e3de58 ref 0 pages 0
X lock type ufs: SHARED (count 1)
X ino 172089992, on dev aacd0s1g
X
X0xffffff04647aa588: tag ufs, type VREG
X usecount 4, writecount 0, refcount 5 mountedhere 0
X flags ()
X v_object 0xffffff04d13a55e8 ref 2 pages 6
X lock type ufs: SHARED (count 1)
X ino 44308534, on dev aacd0s1g
Xdb> bt 31741
XTracing pid 31741 tid 104937 td 0xffffff04937443e0
Xcpustop_handler() at cpustop_handler+0x40
Xipi_nmi_handler() at ipi_nmi_handler+0x30
Xtrap() at trap+0x175
Xnmi_calltrap() at nmi_calltrap+0x8
X--- trap 0x13, rip = 0xffffffff808c4614, rsp = 0xffffffff80cf7e40, rbp = 0xffffff82b9fea550 ---
XDELAY() at DELAY+0x64
X_thread_lock_flags() at _thread_lock_flags+0xb0
Xintr_event_schedule_thread() at intr_event_schedule_thread+0x72
Xcallout_tick() at callout_tick+0xf8
Xhardclock_cpu() at hardclock_cpu+0xd2
Xhardclock() at hardclock+0x23
Xlapic_handle_timer() at lapic_handle_timer+0xd2
XXtimerint() at Xtimerint+0x8c
X--- interrupt, rip = 0xffffffff806472f3, rsp = 0xffffff82b9fea770, rbp = 0xffffff82b9fea7d0 ---
Xvputx() at vputx+0x183
Xlookup() at lookup+0x280
Xnamei() at namei+0x52b
Xkern_statat_vnhook() at kern_statat_vnhook+0x8f
Xkern_statat() at kern_statat+0x15
Xlstat() at lstat+0x2a
Xsyscall() at syscall+0x1e7
XXfast_syscall() at Xfast_syscall+0xe1
X--- syscall (190, FreeBSD ELF64, lstat), rip = 0x800a7c10c, rsp = 0x7fffffffe6c8, rbp = 0x801295cf9 ---
Xdb> bt 11
XTracing pid 11 tid 100010 td 0xffffff00026ea7c0
Xsched_switch() at sched_switch+0xea
Xmi_switch() at mi_switch+0x16f
Xsched_preempt() at sched_preempt+0xb5
Xipi_bitmap_handler() at ipi_bitmap_handler+0x70
XXipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x9b
X--- interrupt, rip = 0xffffffff805a4209, rsp = 0xffffff80000bfb60, rbp = 0xffffff80000bfba0 ---
X_thread_lock_flags() at _thread_lock_flags+0x69
Xsched_idletd() at sched_idletd+0xde
Xfork_exit() at fork_exit+0x118
Xfork_trampoline() at fork_trampoline+0xe
X--- trap 0, rip = 0, rsp = 0xffffff80000bfd30, rbp = 0 ---
Xdb> bt 12
XTracing pid 12 tid 100065 td 0xffffff0009001ba0
Xfork_trampoline() at fork_trampoline
Xdb> bt 73366
XTracing pid 73366 tid 103097 td 0xffffff02df2d57c0
Xcpustop_handler() at cpustop_handler+0x40
Xipi_nmi_handler() at ipi_nmi_handler+0x30
Xtrap() at trap+0x175
Xnmi_calltrap() at nmi_calltrap+0x8
X--- trap 0x13, rip = 0xffffffff808c4614, rsp = 0xffffff8000059fe0, rbp = 0xffffff82b7bfa930 ---
XDELAY() at DELAY+0x64
X_mtx_lock_spin() at _mtx_lock_spin+0x6a
Xsched_add() at sched_add+0x117
Xsetrunnable() at setrunnable+0x78
Xsleepq_broadcast() at sleepq_broadcast+0x7c
Xwakeup() at wakeup+0x27
Xpipe_read() at pipe_read+0x3ff
Xdofileread() at dofileread+0xa1
Xkern_readv() at kern_readv+0x60
Xread() at read+0x55
Xsyscall() at syscall+0x1e7
XXfast_syscall() at Xfast_syscall+0xe1
X--- syscall (3, FreeBSD ELF64, read), rip = 0x8019033ec, rsp = 0x7fffffff6058, rbp = 0x400 ---
Xdb> bt 32064
XTracing pid 32064 tid 102311 td 0xffffff008f1b67c0
Xcpustop_handler() at cpustop_handler+0x40
Xipi_nmi_handler() at ipi_nmi_handler+0x30
Xtrap() at trap+0x175
Xnmi_calltrap() at nmi_calltrap+0x8
X--- trap 0x13, rip = 0xffffffff808c4614, rsp = 0xffffff800006efe0, rbp = 0xffffff82b6ca01f0 ---
XDELAY() at DELAY+0x64
X_mtx_lock_spin() at _mtx_lock_spin+0x6a
Xsmp_tlb_shootdown() at smp_tlb_shootdown+0xfa
Xpmap_invalidate_range() at pmap_invalidate_range+0xae
Xvnode_pager_generic_getpages() at vnode_pager_generic_getpages+0xa69
Xvnode_pager_getpages() at vnode_pager_getpages+0x91
Xvm_fault() at vm_fault+0x644
Xtrap_pfault() at trap_pfault+0x132
Xtrap() at trap+0x35e
Xcalltrap() at calltrap+0x8
X--- trap 0xc, rip = 0xffffffff808bfe0d, rsp = 0xffffff82b6ca08f0, rbp = 0xffffff82b6ca0970 ---
Xcopyin() at copyin+0x3d
Xm_uiotombuf() at m_uiotombuf+0xcb
Xsosend_generic() at sosend_generic+0x328
Xsoo_write() at soo_write+0x5e
Xdofilewrite() at dofilewrite+0x85
Xkern_writev() at kern_writev+0x60
Xfreebsd32_writev() at freebsd32_writev+0x41
Xia32_syscall() at ia32_syscall+0x1eb
XXint0x80_syscall() at Xint0x80_syscall+0x95
X--- syscall (121, FreeBSD ELF32, freebsd32_writev), rip = 0x281a092f, rsp = 0xffffd83c, rbp = 0xffffd868 ---
Xdb> bt 3649
XTracing pid 3649 tid 103874 td 0xffffff009cc0c000
Xcpustop_handler() at cpustop_handler+0x40
Xipi_nmi_handler() at ipi_nmi_handler+0x30
Xtrap() at trap+0x175
Xnmi_calltrap() at nmi_calltrap+0x8
X--- trap 0x13, rip = 0xffffffff808b5348, rsp = 0xffffff8000075fe0, rbp = 0xffffff82b8b27650 ---
Xsmp_tlb_shootdown() at smp_tlb_shootdown+0x98
Xpmap_invalidate_range() at pmap_invalidate_range+0xae
Xvfs_vmio_release() at vfs_vmio_release+0x143
Xgetnewbuf() at getnewbuf+0x9be
Xgetblk() at getblk+0x368
Xcluster_rbuild() at cluster_rbuild+0x761
Xcluster_read() at cluster_read+0x502
Xffs_read() at ffs_read+0x3bc
Xvn_read() at vn_read+0x2d9
Xdofileread() at dofileread+0xa1
Xkern_readv() at kern_readv+0x60
Xread() at read+0x55
Xsyscall() at syscall+0x1e7
XXfast_syscall() at Xfast_syscall+0xe1
X--- syscall (3, FreeBSD ELF64, read), rip = 0x800a8b3ec, rsp = 0x7fffffffe6c8, rbp = 0x131d3ea4 ---
Xdb>
X
24f218c6efe2375d7c9a281e6cc61910
echo x - crash2.txt
sed 's/^X//' >crash2.txt << '67367b08c0869beabc309c4d0c66c9a9'
Xdb> bt
XTracing pid 12 tid 100039 td 0xffffff00029a23e0
Xpropagate_priority() at propagate_priority+0x72
Xturnstile_wait() at turnstile_wait+0x1aa
X_rw_wlock_hard() at _rw_wlock_hard+0xfa
Xin_lltable_lookup() at in_lltable_lookup+0x12b
Xarpintr() at arpintr+0x9d6
Xnetisr_dispatch_src() at netisr_dispatch_src+0x7e
Xether_demux() at ether_demux+0x14d
Xether_input() at ether_input+0x17b
Xether_demux() at ether_demux+0x6f
Xether_input() at ether_input+0x17b
Xbce_intr() at bce_intr+0x3b0
Xintr_event_execute_handlers() at intr_event_execute_handlers+0xfd
Xithread_loop() at ithread_loop+0x8e
Xfork_exit() at fork_exit+0x118
Xfork_trampoline() at fork_trampoline+0xe
X--- trap 0, rip = 0, rsp = 0xffffff82b155dd30, rbp = 0 ---
Xdb> show proc 12
XProcess 12 (intr) at 0xffffff00026e7000:
X state: NORMAL
X uid: 0 gids: 0
X parent: pid 0 at 0xffffffff80c7e3e0
X ABI: null
X threads: 24
X100065 I [irq1: atkbd0]
X100064 I [swi0: uart uart]
X100063 I [irq15: ata1]
X100062 I [irq14: ata0]
X100045 I [irq22: uhci1 uhci3]
X100040 I [irq23: uhci0 uhci2+]
X100039 L *lle 0xffffff032af863c0 [irq257: bce1]
X100038 I [irq256: bce0]
X100036 I [irq17: aac0]
X100035 I [irq9: acpi0]
X100034 I [swi5: +]
X100032 I [swi2: cambio]
X100028 I [swi6: task queue]
X100027 I [swi6: Giant taskq]
X100020 I [swi3: vm]
X100019 Run CPU 1 [swi4: clock]
X100018 Run CPU 5 [swi4: clock]
X100017 Run CPU 6 [swi4: clock]
X100016 I [swi4: clock]
X100015 Run CPU 3 [swi4: clock]
X100014 I [swi4: clock]
X100013 CanRun [swi4: clock]
X100012 I [swi4: clock]
X100011 I [swi1: netisr 0]
Xdb> ps
X pid ppid pgrp uid state wmesg wchan cmd
X53746 53743 51388 0 R CPU 4 head
X53745 53743 51388 0 S piperd 0xffffff050c5ea5b0 sort
X53744 53743 51388 0 R find
X53743 51397 51388 0 S wait 0xffffff04bfd5b460 sh
X53626 49623 49623 37465 S accept 0xffffff022ab955b6 httpd
X53625 49623 49623 37465 S accept 0xffffff022ab955b6 httpd
X53575 65122 65122 5597 S accept 0xffffff04e8ef030e httpd
X53560 68225 68225 44526 S accept 0xffffff0156956066 httpd
X53552 98656 98656 45454 S accept 0xffffff02e9975066 httpd
X53550 9163 9163 45695 S accept 0xffffff041026cb06 httpd
X53549 9163 9163 45695 S accept 0xffffff041026cb06 httpd
X53543 62632 62632 42270 S sbwait 0xffffff022a8d44dc httpd
X53541 62632 62632 42270 S select 0xffffff01a2fb6440 httpd
X53530 67266 67266 36958 S sbwait 0xffffff04c29243ec httpd
X53509 22498 65331 10395 R perl5.8.8
X53410 49623 49623 37465 S select 0xffffff015c6afbc0 httpd
X53357 65545 65545 10900 S select 0xffffff02f289ddc0 httpd
X53335 66113 66113 16234 S lockf 0xffffff03d23c1100 httpd
X53334 65251 65251 8335 S accept 0xffffff060b393b06 httpd
X53309 9163 9163 45695 S select 0xffffff01a2cb1240 httpd
X53306 62632 62632 42270 S select 0xffffff01a2628a40 httpd
X53305 86977 86977 45532 S select 0xffffff010971bb40 httpd
X53201 65429 65429 9400 S accept 0xffffff05aeea5dae httpd
X53200 65429 65429 9400 S accept 0xffffff05aeea5dae httpd
X53199 65429 65429 9400 S accept 0xffffff05aeea5dae httpd
X53110 66113 66113 16234 S select 0xffffff045cabcac0 httpd
X53104 65604 65604 13307 S select 0xffffff01a295b540 httpd
X53103 88924 88924 11904 S accept 0xffffff0009a3d85e httpd
X53101 65141 65141 10330 S accept 0xffffff01dce31dae httpd
X53097 67266 67266 36958 S select 0xffffff0135c45e40 httpd
X53096 68233 68233 47018 S accept 0xffffff012194e30e httpd
X53095 67266 67266 36958 S select 0xffffff01a295e440 httpd
X53094 68233 68233 47018 S accept 0xffffff012194e30e httpd
X53041 65429 65429 9400 S sbwait 0xffffff0153601e8c httpd
X53040 65429 65429 9400 S sbwait 0xffffff02db91d144 httpd
X53037 66752 66752 25536 S select 0xffffff01a2bd7640 httpd
X53036 66752 66752 25536 S lockf 0xffffff0153cfa300 httpd
X53034 66752 66752 25536 S lockf 0xffffff01a29efa80 httpd
X53033 66752 66752 25536 S lockf 0xffffff0135cf0d80 httpd
X53032 66752 66752 25536 S lockf 0xffffff017f231200 httpd
X53030 66752 66752 25536 S lockf 0xffffff028d2f6380 httpd
Xdb> show msgbuf
Xmsgbufp = 0xffffffff81020fe0
Xmagic = 63062, size = 65504, r= 554532, w = 555036, ptr = 0xffffffff81011000, cksum= 5605281
Xkernel trap 12 with interrupts disabled
X
X
XFatal trap 12: page fault while in kernel mode
Xcpuid = 2; apic id = 02
Xfault virtual address = 0xc
Xfault code = supervisor read data, page not present
Xinstruction pointer = 0x20:0xffffffff805f30c2
Xstack pointer = 0x28:0xffffff82b155d830
Xframe pointer = 0x28:0xffffff82b155d860
Xcode segment = base 0x0, limit 0xfffff, type 0x1b
X = DPL 0, pres 1, long 1, def32 0, gran 1
Xprocessor eflags = resume, IOPL = 0
Xcurrent process = 12 (irq257: bce1)
X
Xdb> show allpcpu
XCurrent CPU: 2
X
Xcpuid = 0
Xdynamic pcpu = 0x2a3f80
Xcurthread = 0xffffff0061aca000: pid 52989 "httpd"
Xcurpcb = 0xffffff82b7e2fd40
Xfpcurthread = none
Xidlethread = 0xffffff00026ea7c0: pid 11 "idle: cpu0"
Xcurpmap = 0
Xtssp = 0xffffffff80cf0080
Xcommontssp = 0xffffffff80cf0080
Xrsp0 = 0xffffff82b7e2fd40
Xgs32p = 0xffffffff80ceeeb8
Xldt = 0xffffffff80ceeef8
Xtss = 0xffffffff80ceeee8
X
Xcpuid = 1
Xdynamic pcpu = 0xffffff807f418f80
Xcurthread = 0xffffff00027003e0: pid 12 "swi4: clock"
Xcurpcb = 0xffffff80000ecd40
Xfpcurthread = none
Xidlethread = 0xffffff00026eaba0: pid 11 "idle: cpu1"
Xcurpmap = 0
Xtssp = 0xffffffff80cf00e8
Xcommontssp = 0xffffffff80cf00e8
Xrsp0 = 0xffffff80000ecd40
Xgs32p = 0xffffffff80ceef20
Xldt = 0xffffffff80ceef60
Xtss = 0xffffffff80ceef50
X
Xcpuid = 2
Xdynamic pcpu = 0xffffff807f41ff80
Xcurthread = 0xffffff00029a23e0: pid 12 "irq257: bce1"
Xcurpcb = 0xffffff82b155dd40
Xfpcurthread = none
Xidlethread = 0xffffff00026f6000: pid 11 "idle: cpu2"
Xcurpmap = 0
Xtssp = 0xffffffff80cf0150
Xcommontssp = 0xffffffff80cf0150
Xrsp0 = 0xffffff82b155dd40
Xgs32p = 0xffffffff80ceef88
Xldt = 0xffffffff80ceefc8
Xtss = 0xffffffff80ceefb8
X
Xcpuid = 3
Xdynamic pcpu = 0xffffff807f426f80
Xcurthread = 0xffffff00026fb7c0: pid 12 "swi4: clock"
Xcurpcb = 0xffffff80000d8d40
Xfpcurthread = none
Xidlethread = 0xffffff00026f63e0: pid 11 "idle: cpu3"
Xcurpmap = 0
Xtssp = 0xffffffff80cf01b8
Xcommontssp = 0xffffffff80cf01b8
Xrsp0 = 0xffffff80000d8d40
Xgs32p = 0xffffffff80ceeff0
Xldt = 0xffffffff80cef030
Xtss = 0xffffffff80cef020
X
Xcpuid = 4
Xdynamic pcpu = 0xffffff807f42df80
Xcurthread = 0xffffff014bbbbba0: pid 53746 "head"
Xcurpcb = 0xffffff82b92c0d40
Xfpcurthread = none
Xidlethread = 0xffffff00026f67c0: pid 11 "idle: cpu4"
Xcurpmap = 0
Xtssp = 0xffffffff80cf0220
Xcommontssp = 0xffffffff80cf0220
Xrsp0 = 0xffffff82b92c0d40
Xgs32p = 0xffffffff80cef058
Xldt = 0xffffffff80cef098
Xtss = 0xffffffff80cef088
X
Xcpuid = 5
Xdynamic pcpu = 0xffffff807f434f80
Xcurthread = 0xffffff00026f6ba0: pid 12 "swi4: clock"
Xcurpcb = 0xffffff80000e7d40
Xfpcurthread = none
Xidlethread = 0xffffff00026e9000: pid 11 "idle: cpu5"
Xcurpmap = 0
Xtssp = 0xffffffff80cf0288
Xcommontssp = 0xffffffff80cf0288
Xrsp0 = 0xffffff80000e7d40
Xgs32p = 0xffffffff80cef0c0
Xldt = 0xffffffff80cef100
Xtss = 0xffffffff80cef0f0
X
Xcpuid = 6
Xdynamic pcpu = 0xffffff807f43bf80
Xcurthread = 0xffffff00026fb000: pid 12 "swi4: clock"
Xcurpcb = 0xffffff80000e2d40
Xfpcurthread = none
Xidlethread = 0xffffff00026e93e0: pid 11 "idle: cpu6"
Xcurpmap = 0
Xtssp = 0xffffffff80cf02f0
Xcommontssp = 0xffffffff80cf02f0
Xrsp0 = 0xffffff80000e2d40
Xgs32p = 0xffffffff80cef128
Xldt = 0xffffffff80cef168
Xtss = 0xffffffff80cef158
X
Xcpuid = 7
Xdynamic pcpu = 0xffffff807f442f80
Xcurthread = 0xffffff008976eba0: pid 45168 "httpd"
Xcurpcb = 0xffffff82b71b4d40
Xfpcurthread = 0xffffff008976eba0: pid 45168 "httpd"
Xidlethread = 0xffffff00026e97c0: pid 11 "idle: cpu7"
Xcurpmap = 0
Xtssp = 0xffffffff80cf0358
Xcommontssp = 0xffffffff80cf0358
Xrsp0 = 0xffffff82b71b4d40
Xgs32p = 0xffffffff80cef190
Xldt = 0xffffffff80cef1d0
Xtss = 0xffffffff80cef1c0
X
Xdb> show lockedvnods
XLocked vnodes
X
X0xffffff0109771938: tag ufs, type VDIR
X usecount 3, writecount 0, refcount 11 mountedhere 0
X flags ()
X v_object 0xffffff01096ad948 ref 0 pages 23
X lock type ufs: SHARED (count 1)
X ino 4752, on dev aacd0s1f
Xdb> bt 52989
XTracing pid 52989 tid 103210 td 0xffffff0061aca000
Xcpustop_handler() at cpustop_handler+0x40
Xipi_nmi_handler() at ipi_nmi_handler+0x30
Xtrap() at trap+0x175
Xnmi_calltrap() at nmi_calltrap+0x8
X--- trap 0x13, rip = 0xffffffff808c45c2, rsp = 0xffffffff80cf7e40, rbp = 0xffffff82b7e2f8d0 ---
XDELAY() at DELAY+0x62
X_thread_lock_flags() at _thread_lock_flags+0xb0
Xsleepq_add() at sleepq_add+0x8c
X_sleep() at _sleep+0x159
Xsoreceive_generic() at soreceive_generic+0xeba
Xdofileread() at dofileread+0xa1
Xkern_readv() at kern_readv+0x60
Xread() at read+0x55
Xia32_syscall() at ia32_syscall+0x1eb
XXint0x80_syscall() at Xint0x80_syscall+0x95
X--- syscall (3, FreeBSD ELF32, read), rip = 0x281b834f, rsp = 0xffffb89c, rbp = 0xffffb8b8 ---
Xdb> bt 53746
XTracing pid 53746 tid 104263 td 0xffffff014bbbbba0
Xcpustop_handler() at cpustop_handler+0x40
Xipi_nmi_handler() at ipi_nmi_handler+0x30
Xtrap() at trap+0x175
Xnmi_calltrap() at nmi_calltrap+0x8
X--- trap 0x13, rip = 0xffffffff808c45c4, rsp = 0xffffff8000067fe0, rbp = 0xffffff82b92c05e0 ---
XDELAY() at DELAY+0x64
X_thread_lock_flags() at _thread_lock_flags+0xb0
Xintr_event_schedule_thread() at intr_event_schedule_thread+0x72
Xcallout_tick() at callout_tick+0xf8
Xhardclock_cpu() at hardclock_cpu+0xd2
Xlapic_handle_timer() at lapic_handle_timer+0xff
XXtimerint() at Xtimerint+0x8c
X--- interrupt, rip = 0xffffffff808be298, rsp = 0xffffff82b92c07e0, rbp = 0xffffff82b92c0840 ---
Xpmap_remove() at pmap_remove+0x2a8
Xvm_map_delete() at vm_map_delete+0xf4
Xvm_map_fixed() at vm_map_fixed+0x78
Xvm_mmap() at vm_mmap+0x51d
Xmmap() at mmap+0x219
Xsyscall() at syscall+0x1e7
XXfast_syscall() at Xfast_syscall+0xe1
X--- syscall (477, FreeBSD ELF64, mmap), rip = 0x80050c86c, rsp = 0x7fffffffe278, rbp = 0x7fffffffe370 ---
Xdb> bt 45168
XTracing pid 45168 tid 102571 td 0xffffff008976eba0
Xcpustop_handler() at cpustop_handler+0x40
Xipi_nmi_handler() at ipi_nmi_handler+0x30
Xtrap() at trap+0x175
Xnmi_calltrap() at nmi_calltrap+0x8
X--- trap 0x13, rip = 0xffffffff805a3d8d, rsp = 0xffffff800007cfe0, rbp = 0xffffff82b71b4860 ---
X_mtx_lock_sleep() at _mtx_lock_sleep+0xdd
Xpmap_enter() at pmap_enter+0x2f7
Xvm_fault() at vm_fault+0x14dc
Xtrap_pfault() at trap_pfault+0x132
Xtrap() at trap+0x4dc
Xcalltrap() at calltrap+0x8
X--- trap 0xc, rip = 0x281b7b96, rsp = 0xfffe2424, rbp = 0xfffe24e8 ---
Xdb> bt 11
XTracing pid 11 tid 100010 td 0xffffff00026ea7c0
Xsched_switch() at sched_switch+0xea
Xmi_switch() at mi_switch+0x16f
Xsched_preempt() at sched_preempt+0xb5
Xipi_bitmap_handler() at ipi_bitmap_handler+0x70
XXipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x9b
X--- interrupt, rip = 0xffffffff805a41f4, rsp = 0xffffff80000bfb60, rbp = 0xffffff80000bfba0 ---
X_thread_lock_flags() at _thread_lock_flags+0x64
Xsched_idletd() at sched_idletd+0xde
Xfork_exit() at fork_exit+0x118
Xfork_trampoline() at fork_trampoline+0xe
X--- trap 0, rip = 0, rsp = 0xffffff80000bfd30, rbp = 0 ---
Xdb> bt 12
XTracing pid 12 tid 100065 td 0xffffff0009001ba0
Xfork_trampoline() at fork_trampoline
X
67367b08c0869beabc309c4d0c66c9a9
echo x - crash3.txt
sed 's/^X//' >crash3.txt << 'd8e1b7c43219a397cc6f0ec53abfcb51'
Xdb> bt
XTracing pid 12 tid 100038 td 0xffffff00029a17c0
Xpropagate_priority() at propagate_priority+0x72
Xturnstile_wait() at turnstile_wait+0x1aa
X_rw_wlock_hard() at _rw_wlock_hard+0xfa
Xin_lltable_lookup() at in_lltable_lookup+0x12b
Xarpintr() at arpintr+0x9d6
Xnetisr_dispatch_src() at netisr_dispatch_src+0x7e
Xether_demux() at ether_demux+0x14d
Xether_input() at ether_input+0x17b
Xbce_intr() at bce_intr+0x3b0
Xintr_event_execute_handlers() at intr_event_execute_handlers+0xfd
Xithread_loop() at ithread_loop+0x8e
Xfork_exit() at fork_exit+0x118
Xfork_trampoline() at fork_trampoline+0xe
X--- trap 0, rip = 0, rsp = 0xffffff82b1554d30, rbp = 0 ---
Xdb> show proc 12
XProcess 12 (intr) at 0xffffff00026e7000:
X state: NORMAL
X uid: 0 gids: 0
X parent: pid 0 at 0xffffffff80c7e3e0
X ABI: null
X threads: 24
X100065 I [irq1: atkbd0]
X100064 I [swi0: uart]
X100063 I [irq15: ata1]
X100062 I [irq14: ata0]
X100045 I [irq22: uhci1 uhci3]
X100040 I [irq23: uhci0 uhci2+]
X100039 I [irq257: bce1]
X100038 L *lle 0xffffff05ad99b9c0 [irq256: bce0]
X100036 Run CPU 4 [irq17: aac0]
X100035 I [irq9: acpi0]
X100034 I [swi5: +]
X100032 I [swi2: cambio]
X100028 I [swi6: task queue]
X100027 I [swi6: Giant taskq]
X100020 I [swi3: vm]
X100019 CanRun [swi4: clock]
X100018 I [swi4: clock]
X100017 CanRun [swi4: clock]
X100016 I [swi4: clock]
X100015 I [swi4: clock]
X100014 Run CPU 2 [swi4: clock]
X100013 Run CPU 3 [swi4: clock]
X100012 I [swi4: clock]
X100011 I [swi1: netisr 0]
Xdb> ps
X pid ppid pgrp uid state wmesg wchan cmd
X22640 13700 13700 27830 S select 0xffffff018dc0d640 httpd
X22639 13700 13700 27830 S select 0xffffff00a2555440 httpd
X22638 13700 13700 27830 S select 0xffffff0208c336c0 httpd
X22637 13700 13700 27830 S select 0xffffff038073c7c0 httpd
X22636 13700 13700 27830 S select 0xffffff0142b5c340 httpd
X22635 13700 13700 27830 S select 0xffffff038082c6c0 httpd
X22634 13700 13700 27830 S select 0xffffff0137773040 httpd
X22633 13700 13700 27830 S select 0xffffff04925374c0 httpd
X22632 10373 10373 11233 S accept 0xffffff00451bdb06 httpd
X22631 10433 10433 12845 S sbwait 0xffffff01b40fff7c httpd
X22630 10120 10120 10980 R httpd
X22629 13700 13700 27830 S select 0xffffff0009492640 httpd
X22628 13700 13700 27830 S select 0xffffff04924a9840 httpd
X22627 13700 13700 27830 S select 0xffffff0492737b40 httpd
X22626 13700 13700 27830 S select 0xffffff00a2823340 httpd
X22624 13700 13700 27830 S select 0xffffff0492c13cc0 httpd
X22623 13700 13700 27830 S select 0xffffff0124394140 httpd
X22621 12104 12104 45628 S accept 0xffffff00b22c55b6 httpd
X22620 10099 10099 11299 S accept 0xffffff0009afcb06 httpd
Xdb> show msgbuf
Xmsgbufp = 0xffffffff81020fe0
Xmagic = 63062, size = 65504, r= 972658, w = 973162, ptr = 0xffffffff81011000, cksum= 5556648
Xkernel trap 12 with interrupts disabled
X
X
XFatal trap 12: page fault while in kernel mode
Xcpuid = 0; apic id = 00
Xfault virtual address = 0xc
Xfault code = supervisor read data, page not present
Xinstruction pointer = 0x20:0xffffffff805f30d2
Xstack pointer = 0x28:0xffffff82b1554890
Xframe pointer = 0x28:0xffffff82b15548c0
Xcode segment = base 0x0, limit 0xfffff, type 0x1b
X = DPL 0, pres 1, long 1, def32 0, gran 1
Xprocessor eflags = resume, IOPL = 0
Xcurrent process = 12 (irq256: bce0)
X
Xdb> show allpcpu
XCurrent CPU: 0
X
Xcpuid = 0
Xdynamic pcpu = 0x2a3f80
Xcurthread = 0xffffff00029a17c0: pid 12 "irq256: bce0"
Xcurpcb = 0xffffff82b1554d40
Xfpcurthread = none
Xidlethread = 0xffffff00026ea7c0: pid 11 "idle: cpu0"
Xcurpmap = 0
Xtssp = 0xffffffff80cf0080
Xcommontssp = 0xffffffff80cf0080
Xrsp0 = 0xffffff82b1554d40
Xgs32p = 0xffffffff80ceeeb8
Xldt = 0xffffffff80ceeef8
Xtss = 0xffffffff80ceeee8
X
Xcpuid = 1
Xdynamic pcpu = 0xffffff807f418f80
Xcurthread = 0xffffff0380f1d000: pid 11389 "httpd"
Xcurpcb = 0xffffff82b81f9d40
Xfpcurthread = none
Xidlethread = 0xffffff00026eaba0: pid 11 "idle: cpu1"
Xcurpmap = 0
Xtssp = 0xffffffff80cf00e8
Xcommontssp = 0xffffffff80cf00e8
Xrsp0 = 0xffffff82b81f9d40
Xgs32p = 0xffffffff80ceef20
Xldt = 0xffffffff80ceef60
Xtss = 0xffffffff80ceef50
X
Xcpuid = 2
Xdynamic pcpu = 0xffffff807f41ff80
Xcurthread = 0xffffff00026fbba0: pid 12 "swi4: clock"
Xcurpcb = 0xffffff80000d3d40
Xfpcurthread = none
Xidlethread = 0xffffff00026f6000: pid 11 "idle: cpu2"
Xcurpmap = 0
Xtssp = 0xffffffff80cf0150
Xcommontssp = 0xffffffff80cf0150
Xrsp0 = 0xffffff80000d3d40
Xgs32p = 0xffffffff80ceef88
Xldt = 0xffffffff80ceefc8
Xtss = 0xffffffff80ceefb8
X
Xcpuid = 3
Xdynamic pcpu = 0xffffff807f426f80
Xcurthread = 0xffffff00026fc000: pid 12 "swi4: clock"
Xcurpcb = 0xffffff80000ced40
Xfpcurthread = none
Xidlethread = 0xffffff00026f63e0: pid 11 "idle: cpu3"
Xcurpmap = 0
Xtssp = 0xffffffff80cf01b8
Xcommontssp = 0xffffffff80cf01b8
Xrsp0 = 0xffffff80000ced40
Xgs32p = 0xffffffff80ceeff0
Xldt = 0xffffffff80cef030
Xtss = 0xffffffff80cef020
X
Xcpuid = 4
Xdynamic pcpu = 0xffffff807f42df80
Xcurthread = 0xffffff00029a3000: pid 12 "irq17: aac0"
Xcurpcb = 0xffffff82b1546d40
Xfpcurthread = none
Xidlethread = 0xffffff00026f67c0: pid 11 "idle: cpu4"
Xcurpmap = 0
Xtssp = 0xffffffff80cf0220
Xcommontssp = 0xffffffff80cf0220
Xrsp0 = 0xffffff82b1546d40
Xgs32p = 0xffffffff80cef058
Xldt = 0xffffffff80cef098
Xtss = 0xffffffff80cef088
X
Xcpuid = 5
Xdynamic pcpu = 0xffffff807f434f80
Xcurthread = 0xffffff0020b46000: pid 21095 "httpd"
Xcurpcb = 0xffffff82b5193d40
Xfpcurthread = 0xffffff0020b46000: pid 21095 "httpd"
Xidlethread = 0xffffff00026e9000: pid 11 "idle: cpu5"
Xcurpmap = 0
Xtssp = 0xffffffff80cf0288
Xcommontssp = 0xffffffff80cf0288
Xrsp0 = 0xffffff82b5193d40
Xgs32p = 0xffffffff80cef0c0
Xldt = 0xffffffff80cef100
Xtss = 0xffffffff80cef0f0
X
Xcpuid = 6
Xdynamic pcpu = 0xffffff807f43bf80
Xcurthread = 0xffffff00026e93e0: pid 11 "idle: cpu6"
Xcurpcb = 0xffffff80000a1d40
Xfpcurthread = none
Xidlethread = 0xffffff00026e93e0: pid 11 "idle: cpu6"
Xcurpmap = 0
Xtssp = 0xffffffff80cf02f0
Xcommontssp = 0xffffffff80cf02f0
Xrsp0 = 0xffffff80000a1d40
Xgs32p = 0xffffffff80cef128
Xldt = 0xffffffff80cef168
Xtss = 0xffffffff80cef158
Xcpuid = 7
Xdynamic pcpu = 0xffffff807f442f80
Xcurthread = 0xffffff00026e97c0: pid 11 "idle: cpu7"
Xcurpcb = 0xffffff800009cd40
Xfpcurthread = none
Xidlethread = 0xffffff00026e97c0: pid 11 "idle: cpu7"
Xcurpmap = 0
Xtssp = 0xffffffff80cf0358
Xcommontssp = 0xffffffff80cf0358
Xrsp0 = 0xffffff800009cd40
Xgs32p = 0xffffffff80cef190
Xldt = 0xffffffff80cef1d0
Xtss = 0xffffffff80cef1c0
X
Xdb> show lockedvnods
XLocked vnodes
X
X0xffffff0515307760: tag ufs, type VREG
X usecount 1, writecount 0, refcount 3 mountedhere 0
X flags ()
X v_object 0xffffff03effa7870 ref 0 pages 1
X lock type ufs: SHARED (count 1)
X ino 115366410, on dev aacd0s1g
X
d8e1b7c43219a397cc6f0ec53abfcb51
exit
>Release-Note:
>Audit-Trail:
>Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201103290854.p2T8sSBZ077241>