Date: Sun, 26 Jul 2009 11:34:12 +0900 From: Norikatsu Shigemura <nork@FreeBSD.org> To: freebsd-current@FreeBSD.org Cc: Norikatsu Shigemura <nork@FreeBSD.org> Subject: panic on _vn_lock Message-ID: <20090726113412.52901133.nork@FreeBSD.org>
next in thread | raw e-mail | index | archive | help
This is a multi-part message in MIME format.
--Multipart=_Sun__26_Jul_2009_11_34_12_+0900_UYZmooU7snDxtqGN
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Hi.
I got a panic on _vn_lock@/usr/src/sys/kern/vfs_vnops.c:883.
So I enabled 8-current debug flags (WITNESS, INVARIANTS,
DIAGNOSTIC), and retried.
Please see also attached files. I'll keep to have crash dump.
So I'll be able to report any data. What should I report?
--Multipart=_Sun__26_Jul_2009_11_34_12_+0900_UYZmooU7snDxtqGN
Content-Type: text/plain;
name="kernel.txt"
Content-Disposition: attachment;
filename="kernel.txt"
Content-Transfer-Encoding: 7bit
FreeBSD/amd64 (nadesico.ninth-nine.com) (dcons)
login: Kernel page fault with the following non-sleepable locks held:
exclusive sleep mutex pmap (pmap) r = 0 (0xffffff0010794a60) locked @ /usr/src/sys/amd64/amd64/pmap.c:3819
exclusive sleep mutex vm page queue mutex (vm page queue mutex) r = 0 (0xffffffff8099b100) locked @ /usr/src/sys/amd64/amd64/pmap.c:3818
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x65
witness_warn() at witness_warn+0x2d3
trap() at trap+0x2e5
calltrap() at calltrap+0x8
--- trap 0xc, rip = 0xffffffff804f3dab, rsp = 0xffffff80ec36d9f0, rbp = 0xffffff80ec36da80 ---
pmap_remove_pages() at pmap_remove_pages+0x29b
vmspace_exit() at vmspace_exit+0x96
exit1() at exit1+0x376
sys_exit() at sys_exit+0xe
syscall() at syscall+0x1d5
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (1, FreeBSD ELF64, sys_exit), rip = 0x84047c, rsp = 0x7fffffffe2a8, rbp = 0x7fffffffe2e0 ---
Fatal trap 12: page fault while in kernel mode
cpuid = 2; apic id = 02
fault virtual address = 0x10010
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff804f3dab
stack pointer = 0x28:0xffffff80ec36d9f0
frame pointer = 0x28:0xffffff80ec36da80
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 52423 (cc1)
trap number = 12
panic: page fault
VNASSERT failed
cpuid = 1
0xffffff0010f8b938: KDB: stack backtrace:
tag zfs, type VDIR
usecount 2, writecount 0, refcount 3 mountedhere 0
flags ()
v_object 0xffffff003e96b288 ref 0 pages 0
d block type zfs: UNLOCKED
_trace_self_wrapper() at db_trace_self_wrapper+0x2a
panic() at panic+0x182
trap_fatal() at trap_fatal+0x2ad
trap() at trap+0x2fc
calltrap() at calltrap+0x8
--- trap 0xc, rip = 0xffffffff804f3dab, rsp = 0xffffff80ec36d9f0, rbp = 0xffffff80ec36da80 ---
pmap_remove_pages() at pmap_remove_pages+0x29b
vmspace_exit() at vmspace_exit+0x96
exit1() at exit1+0x376
sys_exit() at sys_exit+0xe
syscall() at syscall+0x1d5
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (1, FreeBSD ELF64, sys_exit), rip = 0x84047c, rsp = 0x7fffffffe2a8, rbp = 0x7fffffffe2e0 ---
interrupt total
irq1: atkbd0 3
irq16: hdac1 ohci+ 57
irq17: ehci0 2
irq18: ohci2 ohci+ 6
irq19: fwohci0+ 100
irq21: pcm0 1
irq22: ahci0 27491
cpu0: timer 736892
irq257: hdac0 1
irq258: re0 32033
cpu2: timer 710963
cpu1: timer 710471
cpu3: timer 711037
Total 2929057
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
hardclock() at hardclock+0x117
lapic_handle_timer() at lapic_handle_timer+0xc9
Xtimerint() at Xtimerint+0x8c
--- interrupt, rip = 0xffffffff80305870, rsp = 0xffffff80ec3193d0, rbp = 0xffffff80ec3194d0 ---
panic() at panic+0x90
_vn_lock() at _vn_lock+0xec
cache_lookup() at cache_lookup+0x4fc
vfs_cache_lookup() at vfs_cache_lookup+0xc0
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x95
lookup() at lookup+0x414
namei() at namei+0x4d3
vn_open_cred() at vn_open_cred+0x1e8
kern_openat() at kern_openat+0x179
syscall() at syscall+0x1d5
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (5, FreeBSD ELF64, open), rip = 0x84eaac, rsp = 0x7fffffffdd78, rbp = 0x800b62a10 ---
KDB: enter: watchdog timeout
[thread pid 52424 tid 100195 ]
Stopped at kdb_enter+0x3d: movq $0,0x4b33f0(%rip)
db> call doadump
Physical memory: 8178 MB
Dumping 2063 MB: 2048 2032 2016 2000 1984 1968 1952 1936 1920 1904 1888 1872 1856 1840 1824 1808 1792 1776 1760 1744 1728 1712 1696 1680 1664 1648 1632 1616 1600 1584 1568 1552 1536 1520 1504 1488 1472 1456 1440 1424 1408 1392 1376 1360 1344 1328 1312 1296 1280 1264 1248 1232 1216 1200 1184 1168 1152 1136 1120 1104 1088 1072 1056 1040 1024 1008 992 976 960 944 928 912 896 880 864 848 832 816 800 784 768 752 736 720 704 688 672 656 640 624 608 592 576 560 544 528 512 496 480 464 448 432 416 400 384 368 352 336 320 304 288 272 256 240 224 208 192 176 160 144 128 112 96 80 64 48 32 16
Dump complete
= 0
db> reset
--Multipart=_Sun__26_Jul_2009_11_34_12_+0900_UYZmooU7snDxtqGN
Content-Type: text/plain;
name="kgdb.txt"
Content-Disposition: attachment;
filename="kgdb.txt"
Content-Transfer-Encoding: 7bit
# kgdb -n 0
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...
Unread portion of the kernel message buffer:
Kernel page fault with the following non-sleepable locks held:
exclusive sleep mutex pmap (pmap) r = 0 (0xffffff0010794a60) locked @ /usr/src/sys/amd64/amd64/pmap.c:3819
exclusive sleep mutex vm page queue mutex (vm page queue mutex) r = 0 (0xffffffff8099b100) locked @ /usr/src/sys/amd64/amd64/pmap.c:3818
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x65
witness_warn() at witness_warn+0x2d3
trap() at trap+0x2e5
calltrap() at calltrap+0x8
--- trap 0xc, rip = 0xffffffff804f3dab, rsp = 0xffffff80ec36d9f0, rbp = 0xffffff80ec36da80 ---
pmap_remove_pages() at pmap_remove_pages+0x29b
vmspace_exit() at vmspace_exit+0x96
exit1() at exit1+0x376
sys_exit() at sys_exit+0xe
syscall() at syscall+0x1d5
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (1, FreeBSD ELF64, sys_exit), rip = 0x84047c, rsp = 0x7fffffffe2a8, rbp = 0x7fffffffe2e0 ---
Fatal trap 12: page fault while in kernel mode
cpuid = 2; apic id = 02
fault virtual address = 0x10010
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff804f3dab
stack pointer = 0x28:0xffffff80ec36d9f0
frame pointer = 0x28:0xffffff80ec36da80
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 52423 (cc1)
trap number = 12
panic: page faultVNASSERT failed
cpuid = 1
0xffffff0010f8b938:
KDB: stack backtrace:tag zfs, type VDIR
usecount 2, writecount 0, refcount 3 mountedhere 0
flags ()
v_object 0xffffff003e96b288 ref 0 pages 0d
block type zfs: UNLOCKED_
trace_self_wrapper() at db_trace_self_wrapper+0x2a
panic() at panic+0x182
trap_fatal() at trap_fatal+0x2ad
trap() at trap+0x2fc
calltrap() at calltrap+0x8
--- trap 0xc, rip = 0xffffffff804f3dab, rsp = 0xffffff80ec36d9f0, rbp = 0xffffff80ec36da80 ---
pmap_remove_pages() at pmap_remove_pages+0x29b
vmspace_exit() at vmspace_exit+0x96
exit1() at exit1+0x376
sys_exit() at sys_exit+0xe
syscall() at syscall+0x1d5
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (1, FreeBSD ELF64, sys_exit), rip = 0x84047c, rsp = 0x7fffffffe2a8, rbp = 0x7fffffffe2e0 ---
interrupt total
irq1: atkbd0 3
irq16: hdac1 ohci+ 57
irq17: ehci0 2
irq18: ohci2 ohci+ 6
irq19: fwohci0+ 100
irq21: pcm0 1
irq22: ahci0 27491
cpu0: timer 736892
irq257: hdac0 1
irq258: re0 32033
cpu2: timer 710963
cpu1: timer 710471
cpu3: timer 711037
Total 2929057
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
hardclock() at hardclock+0x117
lapic_handle_timer() at lapic_handle_timer+0xc9
Xtimerint() at Xtimerint+0x8c
--- interrupt, rip = 0xffffffff80305870, rsp = 0xffffff80ec3193d0, rbp = 0xffffff80ec3194d0 ---
panic() at panic+0x90
_vn_lock() at _vn_lock+0xec
cache_lookup() at cache_lookup+0x4fc
vfs_cache_lookup() at vfs_cache_lookup+0xc0
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x95
lookup() at lookup+0x414
namei() at namei+0x4d3
vn_open_cred() at vn_open_cred+0x1e8
kern_openat() at kern_openat+0x179
syscall() at syscall+0x1d5
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (5, FreeBSD ELF64, open), rip = 0x84eaac, rsp = 0x7fffffffdd78, rbp = 0x800b62a10 ---
KDB: enter: watchdog timeout
Physical memory: 8178 MB
Dumping 2063 MB: 2048 2032 2016 2000 1984 1968 1952 1936 1920 1904 1888 1872 1856 1840 1824 1808 1792 1776 1760 1744 1728 1712 1696 1680 1664 1648 1632 1616 1600 1584 1568 1552 1536 1520 1504 1488 1472 1456 1440 1424 1408 1392 1376 1360 1344 1328 1312 1296 1280 1264 1248 1232 1216 1200 1184 1168 1152 1136 1120 1104 1088 1072 1056 1040 1024 1008 992 976 960 944 928 912 896 880 864 848 832 816 800 784 768 752 736 720 704 688 672 656 640 624 608 592 576 560 544 528 512 496 480 464 448 432 416 400 384 368 352 336 320 304 288 272 256 240 224 208 192 176 160 144 128 112 96 80 64 48 32 16
Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /boot/kernel/zfs.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/zfs.ko
Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from /boot/kernel/opensolaris.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/opensolaris.ko
Reading symbols from /boot/kernel/krpc.ko...Reading symbols from /boot/kernel/krpc.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/krpc.ko
Reading symbols from /boot/kernel/snd_maestro3.ko...Reading symbols from /boot/kernel/snd_maestro3.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/snd_maestro3.ko
Reading symbols from /boot/modules/vboxdrv.ko...done.
Loaded symbols for /boot/modules/vboxdrv.ko
Reading symbols from /boot/kernel/nfsclient.ko...Reading symbols from /boot/kernel/nfsclient.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/nfsclient.ko
Reading symbols from /boot/kernel/nfslockd.ko...Reading symbols from /boot/kernel/nfslockd.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/nfslockd.ko
Reading symbols from /boot/kernel/nullfs.ko...Reading symbols from /boot/kernel/nullfs.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/nullfs.ko
#0 doadump () at pcpu.h:223
223 __asm __volatile("movq %%gs:0,%0" : "=r" (td));
(kgdb) bt
#0 doadump () at pcpu.h:223
#1 0xffffffff801a995c in db_fncall (dummy1=Variable "dummy1" is not available.
) at /usr/src/sys/ddb/db_command.c:548
#2 0xffffffff801a9c91 in db_command (last_cmdp=0xffffffff807a5460, cmd_table=Variable "cmd_table" is not available.
) at /usr/src/sys/ddb/db_command.c:445
#3 0xffffffff801a9ee0 in db_command_loop () at /usr/src/sys/ddb/db_command.c:498
#4 0xffffffff801abec9 in db_trap (type=Variable "type" is not available.
) at /usr/src/sys/ddb/db_main.c:229
#5 0xffffffff80333f90 in kdb_trap (type=3, code=0, tf=0xffffff80ec319200) at /usr/src/sys/kern/subr_kdb.c:534
#6 0xffffffff804fbad6 in trap (frame=0xffffff80ec319200) at /usr/src/sys/amd64/amd64/trap.c:613
#7 0xffffffff804e0373 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:224
#8 0xffffffff8033416d in kdb_enter (why=0xffffffff8055aba7 "watchdog", msg=0xa <Address 0xa out of bounds>) at cpufunc.h:63
#9 0xffffffff804e6f09 in lapic_handle_timer (frame=0xffffff80ec319320) at /usr/src/sys/amd64/amd64/local_apic.c:703
#10 0xffffffff804e0eac in Xtimerint () at apic_vector.S:103
#11 0xffffffff80305870 in panic (fmt=0xffffffff805dafd0 "vn_lock called with no locktype.")
at /usr/src/sys/kern/kern_shutdown.c:526
#12 0xffffffff803a3aec in _vn_lock (vp=Variable "vp" is not available.
) at /usr/src/sys/kern/vfs_vnops.c:883
#13 0xffffffff8038580c in cache_lookup (dvp=0xffffff0010f8b938, vpp=0xffffff80ec319a30, cnp=0xffffff80ec319a58)
at /usr/src/sys/kern/vfs_cache.c:567
#14 0xffffffff80385ab0 in vfs_cache_lookup (ap=Variable "ap" is not available.
) at /usr/src/sys/kern/vfs_cache.c:881
#15 0xffffffff80542ca5 in VOP_LOOKUP_APV (vop=0xffffffff80be8b80, a=0xffffff80ec319690) at vnode_if.c:123
#16 0xffffffff8038c6d4 in lookup (ndp=0xffffff80ec319a00) at vnode_if.h:54
#17 0xffffffff8038d623 in namei (ndp=0xffffff80ec319a00) at /usr/src/sys/kern/vfs_lookup.c:259
#18 0xffffffff803a31d8 in vn_open_cred (ndp=0xffffff80ec319a00, flagp=0xffffff80ec31994c, cmode=420, vn_open_flags=Variable "vn_open_flags" is not available.
)
at /usr/src/sys/kern/vfs_vnops.c:188
#19 0xffffffff803a0f99 in kern_openat (td=0xffffff0005b96390, fd=-100,
path=0x800b62a10 <Address 0x800b62a10 out of bounds>, pathseg=Variable "pathseg" is not available.
) at /usr/src/sys/kern/vfs_syscalls.c:1084
#20 0xffffffff804fb495 in syscall (frame=0xffffff80ec319c80) at /usr/src/sys/amd64/amd64/trap.c:984
#21 0xffffffff804e0651 in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:373
#22 0x000000000084eaac in ?? ()
Previous frame inner to this frame (corrupt stack?)
--Multipart=_Sun__26_Jul_2009_11_34_12_+0900_UYZmooU7snDxtqGN--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20090726113412.52901133.nork>