Date: Wed, 16 May 2007 13:16:27 -0400 From: "Andrew Edwards" <aedwards@sandvine.com> To: <freebsd-performance@freebsd.org> Cc: freebsd-fs@freebsd.org Subject: RE: Ufs dead-locks on freebsd 6.2 Message-ID: <5230D3C40B842D4F9FB3CD368021BEF72F08F3@exchange-2.sandvine.com> In-Reply-To: <20070516163305.GA73495@xor.obsecurity.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Here's the backtrace from the last crash along with the output from show
alllocks when the system was deadlocked. I have been running
6.2-release and compliled with makeoptions debug=3D-g, invariants,
invariant_support and witness. I will update to 6-STABLE add
diagnositc, debug_locks and debug_vfs_locks as per the handbook
recommendation and retry.
Yes, when the system was un-usable I was still able to ping it. I have
the serial console setup as the default console so I can remotely access
the box and break into the debugger etc.
(kgdb) bt
#0 doadump () at pcpu.h:165
#1 0xc059b480 in boot (howto=3D260) at
/usr/src/sys/kern/kern_shutdown.c:409
#2 0xc059b795 in panic (fmt=3D0xc0787b04 "Most recently used by %s\n")
at /usr/src/sys/kern/kern_shutdown.c:565
#3 0xc06c4a4d in mtrash_ctor (mem=3D0xce74fa00, size=3D0, arg=3D0x0,
flags=3D258)
at /usr/src/sys/vm/uma_dbg.c:137
#4 0xc06c2c07 in uma_zalloc_arg (zone=3D0xc10615a0, udata=3D0x0, =
flags=3D258)
at /usr/src/sys/vm/uma_core.c:1850
#5 0xc0591416 in malloc (size=3D272, mtp=3D0xc07c32c0, flags=3D258) at
uma.h:275
#6 0xc05edfab in __mnt_vnode_first (mvp=3D0xf3741c48, mp=3D0xcaa14cf8)
at /usr/src/sys/kern/vfs_mount.c:1813
#7 0xc05f2467 in vfs_msync (mp=3D0xcaa14cf8, flags=3D2)
at /usr/src/sys/kern/vfs_subr.c:2874
#8 0xc05f2bbd in sync_fsync (ap=3D0x0) at
/usr/src/sys/kern/vfs_subr.c:3119
#9 0xc072f4ee in VOP_FSYNC_APV (vop=3D0x0, a=3D0xf3741cbc) at
vnode_if.c:1020
#10 0xc05f097c in sync_vnode (bo=3D0xca854e90, td=3D0xca435000) at
vnode_if.h:537
#11 0xc05f0bf1 in sched_sync () at /usr/src/sys/kern/vfs_subr.c:1698
#12 0xc0587248 in fork_exit (callout=3D0xc05f0a04 <sched_sync>, =
arg=3D0x0,
frame=3D0xf3741d38) at /usr/src/sys/kern/kern_fork.c:821
#13 0xc070712c in fork_trampoline () at
/usr/src/sys/i386/i386/exception.s:208
db> show alllocks
Process 36596 (sshd) thread 0xd1238c00 (102406)
exclusive sleep mutex vm object (standard object) r =3D 0 (0xce2c87bc)
locked @ /usr/src/sys/vm/vm_object.c:446
exclusive sx user map r =3D 0 (0xd128060c) locked @
/usr/src/sys/vm/vm_map.c:307
Process 887 (sshd) thread 0xca7d2000 (100056)
exclusive sleep mutex vm object (standard object) r =3D 0 (0xcb713ad4)
locked @ /usr/src/sys/vm/vm_fault.c:297
exclusive sx user map r =3D 0 (0xcaae4734) locked @
/usr/src/sys/vm/vm_map.c:3074
db> show lockedvnods
Locked vnodes
0xcaa78660: tag ufs, type VREG
usecount 2, writecount 1, refcount 3 mountedhere 0
flags ()
v_object 0xc1046738 ref 0 pages 1596
lock type ufs: EXCL (count 1) by thread 0xca689c00 (pid 536) with 1
pending
ino 494620, on dev amrd0s1d
0xcaa86110: tag ufs, type VREG
usecount 1, writecount 1, refcount 3 mountedhere 0
flags ()
v_object 0xca85f738 ref 0 pages 44
lock type ufs: EXCL (count 1) by thread 0xca7d2780 (pid 715)
ino 494633, on dev amrd0s1d
0xcabe4110: tag ufs, type VDIR
usecount 12, writecount 0, refcount 14 mountedhere 0
flags ()
v_object 0xcab28840 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xca9bbd80 (pid 14253) with
3 pending
ino 423947, on dev amrd0s1d
0xcb437990: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcb3c3dec ref 0 pages 4100
lock type ufs: EXCL (count 1) by thread 0xcaffac00 (pid 20868)
ino 282640, on dev amrd0s1d
0xcb99e550: tag ufs, type VDIR
usecount 2, writecount 0, refcount 4 mountedhere 0
flags ()
v_object 0xcef979cc ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xca9bb600 (pid 881) with 1
pending
ino 423987, on dev amrd0s1d
0xcfc97dd0: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcd4975ac ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcb748900 (pid 2518)
ino 424275, on dev amrd0s1d
0xccad9aa0: tag ufs, type VREG
usecount 1, writecount 1, refcount 3 mountedhere 0
flags ()
v_object 0xcf0c539c ref 0 pages 5
lock type ufs: EXCL (count 1) by thread 0xca7d1c00 (pid 600)
ino 188446, on dev amrd0s1d
0xccb0f110: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcb4609cc ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcafbf480 (pid 11054)
ino 424100, on dev amrd0s1d
0xcc501bb0: tag ufs, type VREG
usecount 1, writecount 1, refcount 3 mountedhere 0
flags ()
v_object 0xcc7d19cc ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcb76d600 (pid 13743) with
1 pending
ino 424279, on dev amrd0s1d
0xcf96b220: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcf135c60 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcafbf900 (pid 29458)
ino 424374, on dev amrd0s1d
0xcc5bbbb0: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcdd5b318 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcaad9900 (pid 50782)
ino 424276, on dev amrd0s1d
0xcec1d000: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcd3d7108 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcb76dc00 (pid 59514)
ino 424500, on dev amrd0s1d
0xcebe5110: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xccee95ac ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcb650780 (pid 59975)
ino 424509, on dev amrd0s1d
0xce0c1880: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xca8b64a4 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcb768a80 (pid 69466)
ino 424555, on dev amrd0s1d
0xcf652110: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcf4a318c ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcaff8600 (pid 75577)
ino 424579, on dev amrd0s1d
0xce282550: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcf261318 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xd0235a80 (pid 81734)
ino 424927, on dev amrd0s1d
0xcc1d4dd0: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcd4a6630 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcaccb900 (pid 81772)
ino 424928, on dev amrd0s1d
0xcb820bb0: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcb251ad4 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcadaf480 (pid 84037)
ino 424935, on dev amrd0s1d
0xced5aaa0: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcb784210 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xd0236000 (pid 202)
ino 425039, on dev amrd0s1d
0xcbe45220: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xce55de70 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcaad9a80 (pid 230)
ino 425043, on dev amrd0s1d
0xcc098220: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xce4a9dec ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcafbfd80 (pid 9902)
ino 425093, on dev amrd0s1d
0xcd585110: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcf8c1e70 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcaccba80 (pid 24017)
ino 425144, on dev amrd0s1d
0xceeac000: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcb1225ac ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcaff5a80 (pid 24775)
ino 425149, on dev amrd0s1d
0xcc549aa0: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcab2d318 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcaff8a80 (pid 42358)
ino 425227, on dev amrd0s1d
0xcc6f7000: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcfd4139c ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcafbf600 (pid 43117)
ino 425230, on dev amrd0s1d
0xccc44bb0: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcfd18d68 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcadafd80 (pid 42859)
ino 425234, on dev amrd0s1d
0xcc7a7220: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcfedf420 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcb76c600 (pid 48968)
ino 425264, on dev amrd0s1d
0xcc693aa0: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcf92f738 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcb655300 (pid 55381)
ino 425286, on dev amrd0s1d
0xcbabf220: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcfe297bc ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xd0234a80 (pid 63802)
ino 425322, on dev amrd0s1d
0xcd760220: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcb11ac60 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcadb0480 (pid 69938)
ino 425348, on dev amrd0s1d
0xcc044990: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcaaff084 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcadeda80 (pid 70418)
ino 425360, on dev amrd0s1d
0xcc190660: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcfed9108 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xca7d2900 (pid 76803)
ino 425378, on dev amrd0s1d
0xcc676330: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcf8c14a4 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcaff8900 (pid 76841)
ino 425384, on dev amrd0s1d
0xcf0ad110: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xce53b5ac ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcadb0900 (pid 79849)
ino 425394, on dev amrd0s1d
0xce4f6aa0: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xce3cc4a4 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcb767300 (pid 79620)
ino 425402, on dev amrd0s1d
0xce80d110: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcf502108 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcaff5d80 (pid 98225)
ino 425478, on dev amrd0s1d
0xcd218990: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcf08e18c ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcaff5000 (pid 98241)
ino 425482, on dev amrd0s1d
0xcbcb3440: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcf8a8738 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcaff8180 (pid 1341)
ino 425505, on dev amrd0s1d
0xcf6fe440: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcf88f39c ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcaad7900 (pid 4512)
ino 425512, on dev amrd0s1d
0xcdd07aa0: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcf7e9a50 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcadaf600 (pid 4464)
ino 425513, on dev amrd0s1d
0xcc18eaa0: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xce2fa108 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcafbf300 (pid 13669)
ino 425549, on dev amrd0s1d
0xcb9e1440: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcfbea39c ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcb64e480 (pid 13656)
ino 425555, on dev amrd0s1d
0xcb8a8bb0: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcf6565ac ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcadb0180 (pid 22845)
ino 425596, on dev amrd0s1d
0xccc47660: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcf551b58 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcb64e180 (pid 22870)
ino 425597, on dev amrd0s1d
0xcec8ecc0: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcab61948 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xd0234600 (pid 32036)
ino 425633, on dev amrd0s1d
0xcdca1bb0: tag ufs, type VREG
usecount 2, writecount 2, refcount 888 mountedhere 0
flags ()
v_object 0xcaf23294 ref 0 pages 41308
lock type ufs: EXCL (count 1) by thread 0xcadb0a80 (pid 5541)
ino 130855246, on dev amrd1s1d=20
> -----Original Message-----
> From: Kris Kennaway [mailto:kris@obsecurity.org]=20
> Sent: Wednesday, May 16, 2007 12:33 PM
> To: Andrew Edwards
> Cc: freebsd-performance@freebsd.org
> Subject: Re: Ufs dead-locks on freebsd 6.2
>=20
> On Wed, May 16, 2007 at 12:08:24PM -0400, Andrew Edwards wrote:
> > I have a system running a dual intel zeon 2.8Ghz with 4G of ram and=20
> > using an intel raid controller model SRCU42X which uses the=20
> amr driver.
> > I have had this server running 5.4 upgraded to 6.2 and was running=20
> > fine for several months and then after a normal reboot I've started=20
> > having all sorts of problems with what appears to be=20
> dead-locks in the=20
> > filesystem. This server is my backup server and I rsync files from=20
> > various servers onto this one fairly non-stop. If I stop=20
> the rsync's=20
> > the system appears to be stable although I did have a=20
> kernel core just=20
> > last night.
> >=20
> > When I have been able to observe the problem I ususally see one=20
> > filesystem become inaccessible, perhaps var but I'm not=20
> sure, and then=20
> > in a short period of time the whole system is inaccessible.=20
> Usually=20
> > if I startup just one of the rsync's within a couple of hours the=20
> > system will be un-usable.
> >=20
> > I did find this thread which seems to describe similar=20
> issues but this=20
> > is a different driver.
> >=20
> http://lists.freebsd.org/pipermail/freebsd-questions/2006-Augu
st/127835.
> > html
>=20
> Probably not relevant then. Deadlocks come in many=20
> varieties, all different.
>=20
> > Currently I'm running with debug.mpsafevfs=3D0, =
debug.mpsafenet=3D1and=20
> > debug.mpsafevm=3D0 but this doesn't seem to help.
> >=20
> > On perahps a related issue I have two other nearly=20
> identical systems=20
> > which were going to be upgrading to 6.2 as on 5.4 I am experiencing=20
> > deadlocks and when I hit ctrl-t I see the system is either stuck in=20
> > ufs or zoneinfo and I have not found very much information=20
> about zoneinfo.
> >=20
> > Does anyone have any suggestions on what I can look for or other=20
> > tuning options or had similar experiences?
>=20
> See the chapter on kernel debugging in the developers=20
> handbook for the information you need to provide before we=20
> can begin to debug your problem.
>=20
> Kris
>=20
>=20
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5230D3C40B842D4F9FB3CD368021BEF72F08F3>