From owner-freebsd-current@FreeBSD.ORG Tue Mar 21 15:37:20 2006 Return-Path: X-Original-To: current@freebsd.org Delivered-To: freebsd-current@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 056B516A400 for ; Tue, 21 Mar 2006 15:37:20 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from fw.zoral.com.ua (ll-227.216.82.212.sovam.net.ua [212.82.216.227]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1593A43D46 for ; Tue, 21 Mar 2006 15:37:18 +0000 (GMT) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by fw.zoral.com.ua (8.13.3/8.13.3) with ESMTP id k2LFb8hE068154 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 21 Mar 2006 17:37:08 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.13.4/8.13.4) with ESMTP id k2LFb7va075383; Tue, 21 Mar 2006 17:37:07 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.13.4/8.13.4/Submit) id k2LFb6dv075382; Tue, 21 Mar 2006 17:37:06 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 21 Mar 2006 17:37:06 +0200 From: Kostik Belousov To: Rong-En Fan Message-ID: <20060321153706.GA63595@deviant.kiev.zoral.com.ua> References: <200603161516.27081.mi+mx@aldan.algebra.com> <20060317094103.GD1203@deviant.kiev.zoral.com.ua> <6eb82e0603171026h4bc49a08kb0bda30e171feaec@mail.gmail.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="ZPt4rx8FFjLCG7dd" Content-Disposition: inline In-Reply-To: <6eb82e0603171026h4bc49a08kb0bda30e171feaec@mail.gmail.com> User-Agent: Mutt/1.4.2.1i X-Virus-Scanned: ClamAV version 0.88, clamav-milter version 0.87 on fw.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-1.4 required=5.0 tests=ALL_TRUSTED autolearn=failed version=3.1.0 X-Spam-Checker-Version: SpamAssassin 3.1.0 (2005-09-13) on fw.zoral.com.ua Cc: Mohan Srinivasan , current@freebsd.org Subject: [patches] [more problems] Re: unkillable process stuck in "nfsaio" X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Mar 2006 15:37:20 -0000 --ZPt4rx8FFjLCG7dd Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Mar 17, 2006 at 01:26:13PM -0500, Rong-En Fan wrote: > On 3/17/06, Kostik Belousov wrote: > > Just out of curiosity: > > > > could you, please, test this little patch: > > > > Index: sys/nfsclient/nfs_bio.c >=20 > I have been played around with intr/nointr with this patch + some in > nfs_bio.c, nfs_vnops.c. > Here are the results (note that, the revision # below means the > changes made in that > revision), tested on RELENG_6 as of yesterday, i386, SMP. kernel is built= with > INVARIANT on. >=20 > * nfs_vnops.c 1.262, nfs_bio.c 1.154 >=20 > - intr >=20 > $ dd if=3D/dev/zero of=3Db bs=3D1m count=3D50 > 50+0 records in > 50+0 records out > 52428800 bytes transferred in 4.410424 secs (11887474 bytes/sec) > $ dd if=3D/dev/zero of=3Db bs=3D1m count=3D50 > ^C^C^C^C^C^C^C^C^C^C7+0 records in > 6+0 records out > 6291456 bytes transferred in 291.017236 secs (21619 bytes/sec) > (stuck in nfsaio) >=20 > - nointr >=20 > $ dd if=3D/dev/zero of=3Db bs=3D1m count=3D50 > 50+0 records in > 50+0 records out > 52428800 bytes transferred in 4.264193 secs (12295128 bytes/sec) > $ dd if=3D/dev/zero of=3Db bs=3D1m count=3D50 > ^C12+0 records in > 11+0 records out > 11534336 bytes transferred in 0.990210 secs (11648373 bytes/sec) >=20 > * nfs_vnops.c 1.262 >=20 > - intr >=20 > $dd if=3D/dev/zero of=3Db bs=3D1m count=3D50 > 50+0 records in > 50+0 records out > 52428800 bytes transferred in 4.238704 secs (12369064 bytes/sec) > $ dd if=3D/dev/zero of=3Db bs=3D1m count=3D50 > ^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C > ^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^Ccc^C^C^C^C15+0 records = in > 14+0 records out > 14680064 bytes transferred in 677.578696 secs (21665 bytes/sec) > (stuck in nfsaio) >=20 > - nointr >=20 > $ dd if=3D/dev/zero of=3Db bs=3D1m count=3D50 > 50+0 records in > 50+0 records out > 52428800 bytes transferred in 4.255155 secs (12321244 bytes/sec) > $ dd if=3D/dev/zero of=3Db bs=3D1m count=3D50 > ^C11+0 records in > 10+0 records out > 10485760 bytes transferred in 0.899381 secs (11658864 bytes/sec) >=20 > * nfs_vnops.c 1.262, nfs_bio.c (remove slpflag =3D 0) >=20 > - intr >=20 > $ dd if=3D/dev/zero of=3Db bs=3D1m count=3D50 > 50+0 records in > 50+0 records out > 52428800 bytes transferred in 4.245185 secs (12350181 bytes/sec) > $ dd if=3D/dev/zero of=3Db bs=3D1m count=3D50 > ^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C > (top's state is changing between CPU0, CPU1, RUN, *Giant) > (11 minutes passed, I reboot this box) >=20 > - nointr > $ dd if=3D/dev/zero of=3Db bs=3D1m count=3D50 > 50+0 records in > 50+0 records out > 52428800 bytes transferred in 4.454680 secs (11769375 bytes/sec) > $ dd if=3D/dev/zero of=3Db bs=3D1m count=3D50 > ^C17+0 records in > 16+0 records out > 16777216 bytes transferred in 1.458180 secs (11505587 bytes/sec) >=20 > * nfs_vnops.c 1.262, nfs_bio.c 1.154 (remove slpflags =3D 0) >=20 > - intr > $ dd if=3D/dev/zero of=3Db bs=3D1m count=3D50 > 50+0 records in > 50+0 records out > 52428800 bytes transferred in 4.386083 secs (11953445 bytes/sec) > $ dd if=3D/dev/zero of=3Db bs=3D1m count=3D50 > ^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C > (top's state is changing between CPU0, CPU1, RUN, *Giant) > (44 minutes passwd, I rebooted) >=20 > - noitntr > $ dd if=3D/dev/zero of=3Db bs=3D1m count=3D50 > 50+0 records in > 50+0 records out > 52428800 bytes transferred in 4.370959 secs (11994805 bytes/sec) > $ dd if=3D/dev/zero of=3Db bs=3D1m count=3D50 > ^C25+0 records in > 24+0 records out > 25165824 bytes transferred in 2.122789 secs (11855076 bytes/sec) >=20 > Looks like that the changes to nfs_vnops.c last Nov by ps@, ^C can really= stop > the process, but it take too much time :( >=20 > Hope this helps, > Rong-En Fan Sure, removal of setting slpflag to 0 was wrong. It seems I found the reaso= n(s) for the process to stuck in nfsaio state: 1. It is wrong to call nfs_asyncio wih NULL td. This function assumes it running in the context of the real process and do the checks for signals delivered to it. Giving NULL td effectively ignored any signals. Patch: Index: sys/nfsclient/nfs_vnops.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D RCS file: /usr/local/arch/ncvs/src/sys/nfsclient/nfs_vnops.c,v retrieving revision 1.264 diff -u -r1.264 nfs_vnops.c --- sys/nfsclient/nfs_vnops.c 8 Mar 2006 01:43:01 -0000 1.264 +++ sys/nfsclient/nfs_vnops.c 21 Mar 2006 15:23:29 -0000 @@ -2588,7 +2588,7 @@ * otherwise just do it ourselves. */ if ((bp->b_flags & B_ASYNC) =3D=3D 0 || - nfs_asyncio(VFSTONFS(ap->a_vp->v_mount), bp, NOCRED, td)) + nfs_asyncio(VFSTONFS(ap->a_vp->v_mount), bp, NOCRED, curthread)) (void)nfs_doio(ap->a_vp, bp, cr, td); return (0); } 2. Signals delivered to the process may be actually put into the thread signal list. nfs_sigintr checked global process signal list only. Patch: Index: sys/nfsclient/nfs_socket.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D RCS file: /usr/local/arch/ncvs/src/sys/nfsclient/nfs_socket.c,v retrieving revision 1.135 diff -u -r1.135 nfs_socket.c --- sys/nfsclient/nfs_socket.c 20 Jan 2006 15:07:18 -0000 1.135 +++ sys/nfsclient/nfs_socket.c 21 Mar 2006 15:23:29 -0000 @@ -1513,11 +1513,13 @@ p =3D td->td_proc; PROC_LOCK(p); tmpset =3D p->p_siglist; + SIGSETOR(tmpset, td->td_siglist); SIGSETNAND(tmpset, td->td_sigmask); mtx_lock(&p->p_sigacts->ps_mtx); SIGSETNAND(tmpset, p->p_sigacts->ps_sigignore); mtx_unlock(&p->p_sigacts->ps_mtx); - if (SIGNOTEMPTY(p->p_siglist) && nfs_sig_pending(tmpset)) { + if ((SIGNOTEMPTY(p->p_siglist) || SIGNOTEMPTY(td->td_siglist)) + && nfs_sig_pending(tmpset)) { PROC_UNLOCK(p); return (EINTR); } But, this just reveals (at least) two another problems, for which I do not have patches. I will continue numeration. [Situations below where obtained by running dd as specified in quoted text, and, after some time, making nfs server unavailable by firewall rules.] 3. Sometimes, the system goes into livelock with processes hanged in the flswai state (AKA bwillwrite() function). 4. Sometimes, the dd process could not exit. It stucks in the exit() code, trying to close filedescriptor for nfs-located file. This nfs vnode is exclusively locked by bufdaemon, and close() path tries to get exclusive lock on it. So, it seems, that combination of intr mounting and large writes currently causes only problems. :( --ZPt4rx8FFjLCG7dd Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.2 (FreeBSD) iD8DBQFEIB2hC3+MBN1Mb4gRAn2EAJ4n/aK15gicB03ALjY8eGVZe2u/RwCg2hYI XHgW2y/ZUIHQvFeD6U9Tlig= =aAO6 -----END PGP SIGNATURE----- --ZPt4rx8FFjLCG7dd--