Date: Thu, 3 Feb 2005 10:07:56 -0800 From: Kris Kennaway <kris@obsecurity.org> To: Mohan Srinivasan <mohan_srinivasan@yahoo.com> Cc: Kris Kennaway <kris@obsecurity.org> Subject: Re: Processes stuck in nfsreq Message-ID: <20050203180756.GA2103@xor.obsecurity.org> In-Reply-To: <20041225173524.33634.qmail@web80606.mail.yahoo.com> References: <20041225070022.GA93123@xor.obsecurity.org> <20041225173524.33634.qmail@web80606.mail.yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--y0ulUmNC+osPPQO6
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On Sat, Dec 25, 2004 at 09:35:24AM -0800, Mohan Srinivasan wrote:
> Hi,
>=20
> If the box is still in that state, can you force a core ?
>=20
> The trace below shows the process blocked on nfs_request()->msleep.
> If the process shows up as waiting on "nfsreq", it would be msleep'ing
> from nfs_reply(), not from nfs_request(). There's only one msleep()
> call in nfs_request() (nfs_socket.c:1125), but that is for "nqnfstry".
> The msleep("nfsreq") only happens from nfs_reply() (unless nfs_reply()=20
> has gotten inlined).
OK, I now have working online kgdb. Here's the trace from one stuck
cvs process:
#0 sched_switch (td=3D0xc3add2e0, newtd=3D0xc81c2a10, flags=3D1) at /usr/s=
rc/sys/kern/sched_4bsd.c:964
#1 0xc0517246 in mi_switch (flags=3D1, newtd=3D0x0) at /usr/src/sys/kern/k=
ern_synch.c:356
#2 0xc0534624 in sleepq_switch (wchan=3D0x0) at /usr/src/sys/kern/subr_sle=
epqueue.c:431
#3 0xc05348f2 in sleepq_wait_sig (wchan=3D0x0) at /usr/src/sys/kern/subr_s=
leepqueue.c:556
#4 0xc0516db6 in msleep (ident=3D0xc57ff500, mtx=3D0xc07a1040, priority=3D=
339, wmesg=3D0xc0700e35 "nfsreq",
timo=3D0) at /usr/src/sys/kern/kern_synch.c:225
#5 0xc05f186a in nfs_reply (rep=3D0xc57ff500) at /usr/src/sys/nfsclient/nf=
s_socket.c:600
#6 0xc05f2583 in nfs_request (vp=3D0xc8b3eac8, mrest=3D0xc3917200, procnum=
=3D4, td=3D0xc3add2e0,
cred=3D0xc512c380, mrp=3D0xee1a397c, mdp=3D0xee1a3980, dposp=3D0xee1a39=
84)
at /usr/src/sys/nfsclient/nfs_socket.c:1043
#7 0xc05f7681 in nfs3_access_otw (vp=3D0xc8b3eac8, wmode=3D-300271216, td=
=3D0xc3add2e0, cred=3D0xc512c380)
at /usr/src/sys/nfsclient/nfs_vnops.c:264
#8 0xc05f7f34 in nfs_getattr (ap=3D0xee1a3a88) at /usr/src/sys/nfsclient/n=
fs_vnops.c:592
#9 0xc06ce1a0 in VOP_GETATTR_AP (a=3D0x0) at vnode_if.c:484
#10 0xc05f89dc in nfs_lookup (ap=3D0x0) at vnode_if.h:275
#11 0xc06cdca0 in VOP_LOOKUP_AP (a=3D0x0) at vnode_if.c:89
#12 0xc0570489 in lookup (ndp=3D0xee1a3c28) at vnode_if.h:54
#13 0xc056fdb8 in namei (ndp=3D0xee1a3c28) at /usr/src/sys/kern/vfs_lookup.=
c:189
#14 0xc057e102 in stat (td=3D0xc3add2e0, uap=3D0xee1a3d14) at /usr/src/sys/=
kern/vfs_syscalls.c:2088
#15 0xc06b32d0 in syscall (frame=3D
{tf_fs =3D 47, tf_es =3D 674955311, tf_ds =3D -1078001617, tf_edi =3D=
3, tf_esi =3D 3, tf_ebp =3D -1077943176, tf_isp =3D -300270220, tf_ebx =3D=
674917772, tf_edx =3D 135074816, tf_ecx =3D 2, tf_eax =3D 188, tf_trapno =
=3D 12, tf_err =3D 2, tf_eip =3D 674368127, tf_cs =3D 31, tf_eflags =3D 514=
, tf_esp =3D -1077943860, tf_ss =3D 47})
at /usr/src/sys/i386/i386/trap.c:951
#16 0xc069dfcf in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s=
:200
#17 0x0000002f in ?? ()
(kgdb) frame 6
#6 0xc05f2583 in nfs_request (vp=3D0xc8b3eac8, mrest=3D0xc3917200, procnum=
=3D4, td=3D0xc3add2e0,
cred=3D0xc512c380, mrp=3D0xee1a397c, mdp=3D0xee1a3980, dposp=3D0xee1a39=
84)
at /usr/src/sys/nfsclient/nfs_socket.c:1043
1043 error =3D nfs_reply(rep);
(kgdb) print rep
$3 =3D (struct nfsreq *) 0xc57ff500
(kgdb) print *rep
$4 =3D {r_chain =3D {tqe_next =3D 0xc7b23200, tqe_prev =3D 0xc07a0ff0}, r_m=
req =3D 0xc37d8700, r_mrep =3D 0x0,
r_md =3D 0x0, r_dpos =3D 0xdeadc0de "", r_nmp =3D 0xc39d0000, r_vp =3D 0x=
c8b3eac8, r_xid =3D 2829996295,
r_flags =3D 1, r_retry =3D 10, r_rexmit =3D 0, r_timer =3D -559038242, r_=
procnum =3D 4, r_rtt =3D -1,
r_lastmsg =3D 4245, r_td =3D 0xc3add2e0}
(kgdb)
The deadc0de is interesting. Let me know if there's more debugging
info I can get for you.
Kris
--y0ulUmNC+osPPQO6
Content-Type: application/pgp-signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.0 (FreeBSD)
iD8DBQFCAmh8Wry0BWjoQKURAqcTAKDK9bL7eJ9jSpBw3nJaCch2U3YGbgCdHDda
E2sJMBNGanAlPJfxoa5XJGQ=
=NAbl
-----END PGP SIGNATURE-----
--y0ulUmNC+osPPQO6--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20050203180756.GA2103>