Date: Wed, 7 Dec 2005 08:47:04 +0100 From: Yuri Khotyaintsev <yuri@irfu.se> To: John Baldwin <jhb@freebsd.org> Cc: freebsd-hackers@freebsd.org, freebsd-stable@freebsd.org Subject: Re: Fatal trap 12: page fault while in kernel mode Message-ID: <200512070847.04980.yuri@irfu.se> In-Reply-To: <200512020854.20959.jhb@freebsd.org> References: <200512021100.03167.yuri@irfu.se> <200512020854.20959.jhb@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Friday 02 December 2005 14.54, John Baldwin wrote:
> On Friday 02 December 2005 05:00 am, Yuri Khotyaintsev wrote:
> > I have the following panic occurring several times a week. The machine =
is
> > an NFS server, and it usually panics early in the morning, when first
> > people try to access it. After reboot it may work OK for 1-2 days, and
> > then panics again. I have tried changing memory and replacing disk which
> > was exported via NFS, but nothing helped :(
> >
> > Any suggestion on how to fix this panic will be very much appreciated !
>
> This panic (in propagate_priority) is usually caused when a thread goes to
> sleep while holding a mutex (which is forbidden). If you enable INVARIAN=
TS
> and/or WITNESS you should get a better panic, and with WITNESS you will
> even be warned when a thread goes to sleep while holding a mutex. Howeve=
r,
> these options do introduce considerable execution overhead, and sometimes
> that overhead changes the timing enough to hide the race. :(
Here are the two panics which I got with INVARIANTS and WITNESS enabled.
# kgdb /usr/obj/usr/src/sys/HEM.DEBUG/kernel.debug vmcore.8=20
[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so:=
=20
Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain condition=
s.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd".
Unread portion of the kernel message buffer:
Memory modified after free 0xc4759e00(508) val=3D0 @ 0xc4759e00
panic: Most recently used by UFS dirhash
Uptime: 11h8m36s
Dumping 511 MB (2 chunks)
chunk 0: 1MB (160 pages) ... ok
chunk 1: 511MB (130800 pages) 495 479 463 447 431 415 399 383 367 351 335=
=20
319 303 287 271 255 239 223 207 191 175 159 143 127 111 95 79 63 47 31 15
#0 doadump () at pcpu.h:165
165 pcpu.h: No such file or directory.
in pcpu.h
(kgdb) where
#0 doadump () at pcpu.h:165
#1 0xc050fd4f in boot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:3=
99
#2 0xc0510043 in panic (fmt=3D0xc06dccbb "Most recently used by %s\n")
at /usr/src/sys/kern/kern_shutdown.c:555
#3 0xc0648ccf in mtrash_ctor (mem=3D0xc4759e00, size=3D0, arg=3D0x0, flags=
=3D2)
at /usr/src/sys/vm/uma_dbg.c:137
#4 0xc06469c1 in uma_zalloc_arg (zone=3D0xc104d980, udata=3D0x0, flags=3D2)
at /usr/src/sys/vm/uma_core.c:1850
#5 0xc05043cd in malloc (size=3D400, mtp=3D0xc06fb700, flags=3D2) at uma.h=
:275
#6 0xc063fba9 in ufs_readdir (ap=3D0xd56eaaec)
at /usr/src/sys/ufs/ufs/ufs_vnops.c:1846
#7 0xc06a61cc in VOP_READDIR_APV (vop=3D0x0, a=3D0xd56eaaec) at vnode_if.c=
:1427
#8 0xc0607716 in nfsrv_readdir (nfsd=3D0xc4368c00, slp=3D0x0, td=3D0xc3326=
780,=20
mrq=3D0xd56eac80) at vnode_if.h:746
#9 0xc060fa5b in nfssvc_nfsd (td=3D0x0)
at /usr/src/sys/nfsserver/nfs_syscalls.c:472
#10 0xc060f280 in nfssvc (td=3D0xc3326780, uap=3D0xd56ead04)
at /usr/src/sys/nfsserver/nfs_syscalls.c:181
#11 0xc069b6b0 in syscall (frame=3D
=2D--Type <return> to continue, or q <return> to quit---
{tf_fs =3D 59, tf_es =3D 59, tf_ds =3D 59, tf_edi =3D 0, tf_esi =3D 0=
, tf_ebp =3D=20
=2D1077941464, tf_isp =3D -714166940, tf_ebx =3D 0, tf_edx =3D -1077936144,=
tf_ecx =3D=20
1, tf_eax =3D 155, tf_trapno =3D 12, tf_err =3D 2, tf_eip =3D 671852067, tf=
_cs =3D 51,=20
tf_eflags =3D 582, tf_esp =3D -1077941492, tf_ss =3D 59})=20
at /usr/src/sys/i386/i386/trap.c:981
#12 0xc068947f in Xint0x80_syscall ()=20
at /usr/src/sys/i386/i386/exception.s:200
#13 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) quit
# kgdb /usr/obj/usr/src/sys/HEM.DEBUG/kernel.debug vmcore.9
[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so:=
=20
Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain condition=
s.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd".
Unread portion of the kernel message buffer:
Memory modified after free 0xc5172800(508) val=3D0 @ 0xc5172800
panic: Most recently used by UFS dirhash
Uptime: 1d1h7m17s
Dumping 511 MB (2 chunks)
chunk 0: 1MB (160 pages) ... ok
chunk 1: 511MB (130800 pages) 495 479 463 447 431 415 399 383 367 351 335=
=20
319 303 287 271 255 239 223 207 191 175 159 143 127 111 95 79 63 47 31 15
#0 doadump () at pcpu.h:165
165 pcpu.h: No such file or directory.
in pcpu.h
(kgdb) where
#0 doadump () at pcpu.h:165
#1 0xc050fd4f in boot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:3=
99
#2 0xc0510043 in panic (fmt=3D0xc06dccbb "Most recently used by %s\n")
at /usr/src/sys/kern/kern_shutdown.c:555
#3 0xc0648ccf in mtrash_ctor (mem=3D0xc5172800, size=3D0, arg=3D0x0, flags=
=3D257)
at /usr/src/sys/vm/uma_dbg.c:137
#4 0xc06469c1 in uma_zalloc_arg (zone=3D0xc104d980, udata=3D0x0, flags=3D2=
57)
at /usr/src/sys/vm/uma_core.c:1850
#5 0xc05043cd in malloc (size=3D368, mtp=3D0xc070eb60, flags=3D257) at uma=
=2Eh:275
#6 0xc063729b in ufsdirhash_build (ip=3D0xc55664a4)
at /usr/src/sys/ufs/ufs/ufs_dirhash.c:184
#7 0xc0639441 in ufs_lookup (ap=3D0xd57c283c)
at /usr/src/sys/ufs/ufs/ufs_lookup.c:192
#8 0xc06a4e0a in VOP_CACHEDLOOKUP_APV (vop=3D0x0, a=3D0xd57c283c)
at vnode_if.c:150
#9 0xc0565e3b in vfs_cache_lookup (ap=3D0x0) at vnode_if.h:82
#10 0xc06a4d2f in VOP_LOOKUP_APV (vop=3D0xc070eee0, a=3D0xd57c28e4)
at vnode_if.c:99
#11 0xc056a8d0 in lookup (ndp=3D0xd57c2bec) at vnode_if.h:56
=2D--Type <return> to continue, or q <return> to quit---
#12 0xc060df58 in nfs_namei (ndp=3D0xd57c2bec, fhp=3D0x0, len=3D0, slp=3D0x=
0, nam=3D0x0,=20
mdp=3D0xd57c2a04, dposp=3D0xd57c2a08, retdirp=3D0xd57c29f0, v3=3D8,=20
retdirattrp=3D0x0, retdirattr_retp=3D0x0, td=3D0xc350a780, pubflag=3D0)
at /usr/src/sys/nfsserver/nfs_srvsubs.c:780
#13 0xc05fd284 in nfsrv_lookup (nfsd=3D0xc5764100, slp=3D0x0, td=3D0xc350a7=
80,=20
mrq=3D0xd57c2c80) at /usr/src/sys/nfsserver/nfs_serv.c:517
#14 0xc060fa5b in nfssvc_nfsd (td=3D0x0)
at /usr/src/sys/nfsserver/nfs_syscalls.c:472
#15 0xc060f280 in nfssvc (td=3D0xc350a780, uap=3D0xd57c2d04)
at /usr/src/sys/nfsserver/nfs_syscalls.c:181
#16 0xc069b6b0 in syscall (frame=3D
{tf_fs =3D 59, tf_es =3D 59, tf_ds =3D 59, tf_edi =3D 0, tf_esi =3D 0=
, tf_ebp =3D=20
=2D1077941464, tf_isp =3D -713282204, tf_ebx =3D 0, tf_edx =3D -1077936144,=
tf_ecx =3D=20
1, tf_eax =3D 155, tf_trapno =3D 12, tf_err =3D 2, tf_eip =3D 671852067, tf=
_cs =3D 51,=20
tf_eflags =3D 582, tf_esp =3D -1077941492, tf_ss =3D 59})=20
at /usr/src/sys/i386/i386/trap.c:981
#17 0xc068947f in Xint0x80_syscall ()=20
at /usr/src/sys/i386/i386/exception.s:200
#18 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) exit
Undefined command: "exit". Try "help".
(kgdb) quit
=2D-=20
Dr. Yuri Khotyaintsev
Institutet f=F6r rymdfysik (IRF), Uppsala
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200512070847.04980.yuri>