From owner-freebsd-current@FreeBSD.ORG Thu Feb 3 18:07:58 2005 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A54D616A4CE; Thu, 3 Feb 2005 18:07:58 +0000 (GMT) Received: from obsecurity.dyndns.org (CPE0050040655c8-CM00111ae02aac.cpe.net.cable.rogers.com [69.199.47.57]) by mx1.FreeBSD.org (Postfix) with ESMTP id CAD1B43D46; Thu, 3 Feb 2005 18:07:57 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id 01C4452546; Thu, 3 Feb 2005 10:07:56 -0800 (PST) Date: Thu, 3 Feb 2005 10:07:56 -0800 From: Kris Kennaway To: Mohan Srinivasan Message-ID: <20050203180756.GA2103@xor.obsecurity.org> References: <20041225070022.GA93123@xor.obsecurity.org> <20041225173524.33634.qmail@web80606.mail.yahoo.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="y0ulUmNC+osPPQO6" Content-Disposition: inline In-Reply-To: <20041225173524.33634.qmail@web80606.mail.yahoo.com> User-Agent: Mutt/1.4.2.1i cc: ps@FreeBSD.org cc: current@FreeBSD.org cc: mohans cc: Kris Kennaway Subject: Re: Processes stuck in nfsreq X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Feb 2005 18:07:58 -0000 --y0ulUmNC+osPPQO6 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Dec 25, 2004 at 09:35:24AM -0800, Mohan Srinivasan wrote: > Hi, >=20 > If the box is still in that state, can you force a core ? >=20 > The trace below shows the process blocked on nfs_request()->msleep. > If the process shows up as waiting on "nfsreq", it would be msleep'ing > from nfs_reply(), not from nfs_request(). There's only one msleep() > call in nfs_request() (nfs_socket.c:1125), but that is for "nqnfstry". > The msleep("nfsreq") only happens from nfs_reply() (unless nfs_reply()=20 > has gotten inlined). OK, I now have working online kgdb. Here's the trace from one stuck cvs process: #0 sched_switch (td=3D0xc3add2e0, newtd=3D0xc81c2a10, flags=3D1) at /usr/s= rc/sys/kern/sched_4bsd.c:964 #1 0xc0517246 in mi_switch (flags=3D1, newtd=3D0x0) at /usr/src/sys/kern/k= ern_synch.c:356 #2 0xc0534624 in sleepq_switch (wchan=3D0x0) at /usr/src/sys/kern/subr_sle= epqueue.c:431 #3 0xc05348f2 in sleepq_wait_sig (wchan=3D0x0) at /usr/src/sys/kern/subr_s= leepqueue.c:556 #4 0xc0516db6 in msleep (ident=3D0xc57ff500, mtx=3D0xc07a1040, priority=3D= 339, wmesg=3D0xc0700e35 "nfsreq", timo=3D0) at /usr/src/sys/kern/kern_synch.c:225 #5 0xc05f186a in nfs_reply (rep=3D0xc57ff500) at /usr/src/sys/nfsclient/nf= s_socket.c:600 #6 0xc05f2583 in nfs_request (vp=3D0xc8b3eac8, mrest=3D0xc3917200, procnum= =3D4, td=3D0xc3add2e0, cred=3D0xc512c380, mrp=3D0xee1a397c, mdp=3D0xee1a3980, dposp=3D0xee1a39= 84) at /usr/src/sys/nfsclient/nfs_socket.c:1043 #7 0xc05f7681 in nfs3_access_otw (vp=3D0xc8b3eac8, wmode=3D-300271216, td= =3D0xc3add2e0, cred=3D0xc512c380) at /usr/src/sys/nfsclient/nfs_vnops.c:264 #8 0xc05f7f34 in nfs_getattr (ap=3D0xee1a3a88) at /usr/src/sys/nfsclient/n= fs_vnops.c:592 #9 0xc06ce1a0 in VOP_GETATTR_AP (a=3D0x0) at vnode_if.c:484 #10 0xc05f89dc in nfs_lookup (ap=3D0x0) at vnode_if.h:275 #11 0xc06cdca0 in VOP_LOOKUP_AP (a=3D0x0) at vnode_if.c:89 #12 0xc0570489 in lookup (ndp=3D0xee1a3c28) at vnode_if.h:54 #13 0xc056fdb8 in namei (ndp=3D0xee1a3c28) at /usr/src/sys/kern/vfs_lookup.= c:189 #14 0xc057e102 in stat (td=3D0xc3add2e0, uap=3D0xee1a3d14) at /usr/src/sys/= kern/vfs_syscalls.c:2088 #15 0xc06b32d0 in syscall (frame=3D {tf_fs =3D 47, tf_es =3D 674955311, tf_ds =3D -1078001617, tf_edi =3D= 3, tf_esi =3D 3, tf_ebp =3D -1077943176, tf_isp =3D -300270220, tf_ebx =3D= 674917772, tf_edx =3D 135074816, tf_ecx =3D 2, tf_eax =3D 188, tf_trapno = =3D 12, tf_err =3D 2, tf_eip =3D 674368127, tf_cs =3D 31, tf_eflags =3D 514= , tf_esp =3D -1077943860, tf_ss =3D 47}) at /usr/src/sys/i386/i386/trap.c:951 #16 0xc069dfcf in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s= :200 #17 0x0000002f in ?? () (kgdb) frame 6 #6 0xc05f2583 in nfs_request (vp=3D0xc8b3eac8, mrest=3D0xc3917200, procnum= =3D4, td=3D0xc3add2e0, cred=3D0xc512c380, mrp=3D0xee1a397c, mdp=3D0xee1a3980, dposp=3D0xee1a39= 84) at /usr/src/sys/nfsclient/nfs_socket.c:1043 1043 error =3D nfs_reply(rep); (kgdb) print rep $3 =3D (struct nfsreq *) 0xc57ff500 (kgdb) print *rep $4 =3D {r_chain =3D {tqe_next =3D 0xc7b23200, tqe_prev =3D 0xc07a0ff0}, r_m= req =3D 0xc37d8700, r_mrep =3D 0x0, r_md =3D 0x0, r_dpos =3D 0xdeadc0de "", r_nmp =3D 0xc39d0000, r_vp =3D 0x= c8b3eac8, r_xid =3D 2829996295, r_flags =3D 1, r_retry =3D 10, r_rexmit =3D 0, r_timer =3D -559038242, r_= procnum =3D 4, r_rtt =3D -1, r_lastmsg =3D 4245, r_td =3D 0xc3add2e0} (kgdb) The deadc0de is interesting. Let me know if there's more debugging info I can get for you. Kris --y0ulUmNC+osPPQO6 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.0 (FreeBSD) iD8DBQFCAmh8Wry0BWjoQKURAqcTAKDK9bL7eJ9jSpBw3nJaCch2U3YGbgCdHDda E2sJMBNGanAlPJfxoa5XJGQ= =NAbl -----END PGP SIGNATURE----- --y0ulUmNC+osPPQO6--