Date: Tue, 11 Dec 2012 17:23:18 GMT From: Derek Kulinski <takeda@takeda.tk> To: freebsd-gnats-submit@FreeBSD.org Subject: kern/174372: Pagefault appears to be related to ZFS Message-ID: <201212111723.qBBHNIOB021284@red.freebsd.org> Resent-Message-ID: <201212111730.qBBHU1N8081981@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 174372
>Category: kern
>Synopsis: Pagefault appears to be related to ZFS
>Confidential: no
>Severity: non-critical
>Priority: low
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Tue Dec 11 17:30:01 UTC 2012
>Closed-Date:
>Last-Modified:
>Originator: Derek Kulinski
>Release: FreeBSD 9-STABLE
>Organization:
>Environment:
FreeBSD chinatsu.takeda.tk 9.1-RC3 FreeBSD 9.1-RC3 #8: Mon Dec 3 21:34:28 PST 2012 root@chinatsu.takeda.tk:/usr/obj/usr/src/sys/CHINATSU amd64
>Description:
My FreeBSD box recently started to crash from the logs it looks like around 3:01am - 3:05am. My guess is that it happens when daily periodic tasks are running, but I was unable to reproduce it by running those tasks by hand. Looks like it is due to missing page, so perhaps when I'm trying it the page is not fully swapped.
I encountered 3 crashes so far (I got them on Dec 7, 9 and 11), two last ones fail at list_remove here is backtrace of last one (the one I'm reporting):
(kgdb) #0 doadump (textdump=1) at /usr/src/sys/kern/kern_shutdown.c:265
#1 0xffffffff8056c61b in kern_reboot (howto=260)
at /usr/src/sys/kern/kern_shutdown.c:448
#2 0xffffffff8056cb0e in panic (fmt=0x104 <Address 0x104 out of bounds>)
at /usr/src/sys/kern/kern_shutdown.c:636
#3 0xffffffff8080b6c5 in trap_fatal (frame=0xffffff82311040d0, eva=0)
at /usr/src/sys/amd64/amd64/trap.c:857
#4 0xffffffff8080ba26 in trap_pfault (frame=0xffffff82311040d0, usermode=0)
at /usr/src/sys/amd64/amd64/trap.c:773
#5 0xffffffff8080c023 in trap (frame=0xffffff82311040d0)
at /usr/src/sys/amd64/amd64/trap.c:456
#6 0xffffffff807f64e3 in calltrap ()
at /usr/src/sys/amd64/amd64/exception.S:228
#7 0xffffffff8101f48f in list_remove (list=0xfffffe01e7b835e8, object=Variable "object" is not available.
)
at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/os/list.c:130
#8 0xffffffff81031fad in dbuf_destroy (db=0xfffffe007ce41000)
at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:1776
#9 0xffffffff81064671 in sa_handle_destroy (hdl=0xfffffe007fa5d370)
at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/sa.c:1333
#10 0xffffffff81090da8 in zfs_znode_dmu_fini (zp=0xfffffe0189b66c80)
at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_znode.c:627
#11 0xffffffff810bf2ce in zfs_freebsd_reclaim (ap=Variable "ap" is not available.
)
at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:6172
#12 0xffffffff8085965a in VOP_RECLAIM_APV (vop=0xffffffff8112d2a0,
a=0xffffff8231104270) at vnode_if.c:1926
#13 0xffffffff806094f1 in vgonel (vp=0xfffffe00629dd780) at vnode_if.h:830
#14 0xffffffff8060e922 in vnlru_free (count=1)
at /usr/src/sys/kern/vfs_subr.c:930
#15 0xffffffff8060ebb1 in getnewvnode (tag=0xffffffff808c6071 "ufs",
mp=0xfffffe00066d0c60, vops=0xffffffff80bc9fc0, vpp=0xffffff82311043e8)
at /usr/src/sys/kern/vfs_subr.c:952
#16 0xffffffff807ae22f in ffs_vgetf (mp=0xfffffe00066d0c60, ino=2517042,
flags=524288, vpp=0xffffff82311044e0, ffs_flags=0)
at /usr/src/sys/ufs/ffs/ffs_vfsops.c:1681
#17 0xffffffff807bb4b3 in ufs_lookup_ino (vdp=0xfffffe001417a960,
vpp=0xffffff82311047f0, cnp=0xffffff8231104818, dd_ino=0x0)
at /usr/src/sys/ufs/ufs/ufs_lookup.c:749
#18 0xffffffff80858914 in VOP_CACHEDLOOKUP_APV (vop=0xffffffff80bca880,
a=0xffffff8231104560) at vnode_if.c:187
#19 0xffffffff805f5c3f in vfs_cache_lookup (ap=Variable "ap" is not available.
) at vnode_if.h:80
#20 0xffffffff8085ba8c in VOP_LOOKUP_APV (vop=0xffffffff80bca880,
a=0xffffff8231104660) at vnode_if.c:123
#21 0xffffffff805fd2b1 in lookup (ndp=0xffffff82311047b0) at vnode_if.h:54
#22 0xffffffff805fe535 in namei (ndp=0xffffff82311047b0)
at /usr/src/sys/kern/vfs_lookup.c:297
#23 0xffffffff80610ca6 in kern_statat_vnhook (td=0xfffffe00781ad000, flag=Variable "flag" is not available.
)
at /usr/src/sys/kern/vfs_syscalls.c:2432
#24 0xffffffff80610ee5 in kern_statat (td=Variable "td" is not available.
)
at /usr/src/sys/kern/vfs_syscalls.c:2413
#25 0xffffffff80610faa in sys_lstat (td=Variable "td" is not available.
)
at /usr/src/sys/kern/vfs_syscalls.c:2476
#26 0xffffffff8080ae88 in amd64_syscall (td=0xfffffe00781ad000, traced=0)
at subr_syscall.c:135
#27 0xffffffff807f67c7 in Xfast_syscall ()
at /usr/src/sys/amd64/amd64/exception.S:387
#28 0x0000000800917e8c in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb)
The first one looks a bit different, so it might be something else:
(kgdb) #0 doadump (textdump=1) at /usr/src/sys/kern/kern_shutdown.c:265
#1 0xffffffff8056c61b in kern_reboot (howto=260)
at /usr/src/sys/kern/kern_shutdown.c:448
#2 0xffffffff8056cb0e in panic (fmt=0x104 <Address 0x104 out of bounds>)
at /usr/src/sys/kern/kern_shutdown.c:636
#3 0xffffffff8080b6c5 in trap_fatal (frame=0xffffff8231545bb0,
eva=6597069767528) at /usr/src/sys/amd64/amd64/trap.c:857
#4 0xffffffff8080ba26 in trap_pfault (frame=0xffffff8231545bb0, usermode=0)
at /usr/src/sys/amd64/amd64/trap.c:773
#5 0xffffffff8080c023 in trap (frame=0xffffff8231545bb0)
at /usr/src/sys/amd64/amd64/trap.c:456
#6 0xffffffff807f64e3 in calltrap ()
at /usr/src/sys/amd64/amd64/exception.S:228
#7 0xffffffff8057500b in _sx_xlock_hard (sx=0xfffffe0091dbb43b,
tid=18446741874948425952, opts=0, file=Variable "file" is not available.
)
at /usr/src/sys/kern/kern_sx.c:544
#8 0xffffffff80575a85 in _sx_xlock (sx=Variable "sx" is not available.
) at sx.h:161
#9 0xffffffff81032e1e in dbuf_rele (db=0xfffffe0091dbb3e3,
tag=0xfffffe013a1923b0)
at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:2021
#10 0xffffffff81064671 in sa_handle_destroy (hdl=0xfffffe0072bc1aa0)
at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/sa.c:1333
#11 0xffffffff81090da8 in zfs_znode_dmu_fini (zp=0xfffffe0070887af0)
at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_znode.c:627
#12 0xffffffff810bf2ce in zfs_freebsd_reclaim (ap=Variable "ap" is not available.
)
at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:6172
#13 0xffffffff8085965a in VOP_RECLAIM_APV (vop=0xffffffff8112d2a0,
a=0xffffff8231545e00) at vnode_if.c:1926
#14 0xffffffff806094f1 in vgonel (vp=0xfffffe0070fa41e0) at vnode_if.h:830
#15 0xffffffff8060e922 in vnlru_free (count=1)
at /usr/src/sys/kern/vfs_subr.c:930
#16 0xffffffff8060ebb1 in getnewvnode (tag=0xffffffff81125dba "zfs",
mp=0xfffffe000f0d6000, vops=0xffffffff8112d2a0, vpp=0xffffff8231545f48)
at /usr/src/sys/kern/vfs_subr.c:952
#17 0xffffffff81090df5 in zfs_znode_cache_constructor (
buf=0xfffffe008322c000, arg=Variable "arg" is not available.
)
at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_znode.c:134
#18 0xffffffff81092d1c in zfs_znode_alloc (zfsvfs=0xfffffe0006fb6000,
db=0xfffffe00a8a69380, blksz=2048, obj_type=DMU_OT_SA, hdl=0x0)
at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_znode.c:662
#19 0xffffffff810933af in zfs_zget (zfsvfs=0xfffffe0006fb6000, obj_num=1802,
zpp=0xffffff8231546318)
at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_znode.c:1247
#20 0xffffffff810a8508 in zfs_dirent_lock (dlpp=0xffffff8231546320,
dzp=0xfffffe006e6e0960, name=0xffffff8231546400 ".depend",
zpp=0xffffff8231546318, flag=Variable "flag" is not available.
)
at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_dir.c:317
#21 0xffffffff810a8799 in zfs_dirlook (dzp=0xfffffe006e6e0960,
name=0xffffff8231546400 ".depend", vpp=0xffffff82315467f0, flags=Variable "flags" is not available.
)
at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_dir.c:411
#22 0xffffffff810b90fb in zfs_lookup (dvp=0xfffffe0056ddd780,
nm=0xffffff8231546400 ".depend", vpp=0xffffff82315467f0,
cnp=0xffffff8231546818, nameiop=0, cr=0xfffffe007f6ad300,
td=0xfffffe000f9fc8e0, flags=0)
at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1417
#23 0xffffffff810b9933 in zfs_freebsd_lookup (ap=0xffffff8231546560)
at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:5768
#24 0xffffffff80858914 in VOP_CACHEDLOOKUP_APV (vop=0xffffffff8112d2a0,
a=0xffffff8231546560) at vnode_if.c:187
#25 0xffffffff805f5c3f in vfs_cache_lookup (ap=Variable "ap" is not available.
) at vnode_if.h:80
#26 0xffffffff8085ba8c in VOP_LOOKUP_APV (vop=0xffffffff8112d2a0,
a=0xffffff8231546660) at vnode_if.c:123
#27 0xffffffff805fd2b1 in lookup (ndp=0xffffff82315467b0) at vnode_if.h:54
#28 0xffffffff805fe535 in namei (ndp=0xffffff82315467b0)
at /usr/src/sys/kern/vfs_lookup.c:297
#29 0xffffffff80610ca6 in kern_statat_vnhook (td=0xfffffe000f9fc8e0, flag=Variable "flag" is not available.
)
at /usr/src/sys/kern/vfs_syscalls.c:2432
#30 0xffffffff80610ee5 in kern_statat (td=Variable "td" is not available.
)
at /usr/src/sys/kern/vfs_syscalls.c:2413
#31 0xffffffff80610faa in sys_lstat (td=Variable "td" is not available.
)
at /usr/src/sys/kern/vfs_syscalls.c:2476
#32 0xffffffff8080ae88 in amd64_syscall (td=0xfffffe000f9fc8e0, traced=0)
at subr_syscall.c:135
#33 0xffffffff807f67c7 in Xfast_syscall ()
at /usr/src/sys/amd64/amd64/exception.S:387
#34 0x0000000800917e8c in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb)
>How-To-Repeat:
No idea, appears to happen when periodic daily is run, but the crash does not always happen.
>Fix:
>Release-Note:
>Audit-Trail:
>Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201212111723.qBBHNIOB021284>