Skip site navigation (1)Skip section navigation (2)
Date:      Sat, 24 Aug 2013 20:41:03 +0300
From:      Konstantin Belousov <kostikbel@gmail.com>
To:        J David <j.david.lists@gmail.com>
Cc:        Rick Macklem <rmacklem@uoguelph.ca>, freebsd-stable <freebsd-stable@freebsd.org>
Subject:   Re: NFS deadlock on 9.2-Beta1
Message-ID:  <20130824174103.GQ4972@kib.kiev.ua>
In-Reply-To: <CABXB=RTYZRs2okKu%2B3eZA1twNx%2B2J849Efo-8t=8ZSO3K9sRHw@mail.gmail.com>
References:  <CABXB=RSGrpUr34EdDu_6kab4nDm4NmYAoJZR8MPJOitpVzSNrA@mail.gmail.com> <158996730.12825374.1377265437471.JavaMail.root@uoguelph.ca> <CABXB=RTYZRs2okKu%2B3eZA1twNx%2B2J849Efo-8t=8ZSO3K9sRHw@mail.gmail.com>

next in thread | previous in thread | raw e-mail | index | archive | help

--g1mh5Fh3W6jKnNId
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Sat, Aug 24, 2013 at 01:08:05PM -0400, J David wrote:
> The requested information about the deadlock was finally obtained and
> provided off-list to the requested parties due to size.

Thank you, the problem is clear now.

The problematic process backtrace is

Tracing command httpd pid 86383 tid 100138 td 0xfffffe000b7b2900
sched_switch() at sched_switch+0x234/frame 0xffffff834c442360
mi_switch() at mi_switch+0x15c/frame 0xffffff834c4423a0
sleepq_switch() at sleepq_switch+0x17d/frame 0xffffff834c4423e0
sleepq_wait() at sleepq_wait+0x43/frame 0xffffff834c442410
sleeplk() at sleeplk+0x11a/frame 0xffffff834c442460
__lockmgr_args() at __lockmgr_args+0x9a9/frame 0xffffff834c442580
nfs_lock1() at nfs_lock1+0x87/frame 0xffffff834c4425b0
VOP_LOCK1_APV() at VOP_LOCK1_APV+0xbe/frame 0xffffff834c4425e0
_vn_lock() at _vn_lock+0x63/frame 0xffffff834c442640
ncl_upgrade_vnlock() at ncl_upgrade_vnlock+0x5e/frame 0xffffff834c442670
ncl_bioread() at ncl_bioread+0x195/frame 0xffffff834c4427e0
VOP_READ_APV() at VOP_READ_APV+0xd1/frame 0xffffff834c442810
vn_rdwr() at vn_rdwr+0x2bc/frame 0xffffff834c4428d0
kern_sendfile() at kern_sendfile+0xa90/frame 0xffffff834c442ac0
do_sendfile() at do_sendfile+0x92/frame 0xffffff834c442b20
amd64_syscall() at amd64_syscall+0x259/frame 0xffffff834c442c30
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff834c442c30
--- syscall (393, FreeBSD ELF64, sys_sendfile), rip =3D 0x801b24f4c, rsp =
=3D 0x7fffffffce98, rbp =3D 0x7fffffffd1d0 ---

It tries to do the upgrade of the nfs vnode lock, and for this, the lock
is dropped and re-acquired. Since this happens with the vnode vm object'
page busied, we get a reversal between vnode lock and page busy state.
So effectively, my suspicion that NFS read path drops vnode lock was
true, and in fact I knew about the upgrade.

I think the easiest route is to a partial merge of the r253927 from HEAD.

Index: fs
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
--- fs	(revision 254800)
+++ fs	(working copy)

Property changes on: fs
___________________________________________________________________
Modified: svn:mergeinfo
   Merged /head/sys/fs:r253927
Index: kern/uipc_syscalls.c
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
--- kern/uipc_syscalls.c	(revision 254800)
+++ kern/uipc_syscalls.c	(working copy)
@@ -2124,11 +2124,6 @@
 			else {
 				ssize_t resid;
=20
-				/*
-				 * Ensure that our page is still around
-				 * when the I/O completes.
-				 */
-				vm_page_io_start(pg);
 				VM_OBJECT_UNLOCK(obj);
=20
 				/*
@@ -2144,10 +2139,8 @@
 				    IO_VMIO | ((MAXBSIZE / bsize) << IO_SEQSHIFT),
 				    td->td_ucred, NOCRED, &resid, td);
 				VFS_UNLOCK_GIANT(vfslocked);
-				VM_OBJECT_LOCK(obj);
-				vm_page_io_finish(pg);
-				if (!error)
-					VM_OBJECT_UNLOCK(obj);
+				if (error)
+					VM_OBJECT_LOCK(obj);
 				mbstat.sf_iocnt++;
 			}
 			if (error) {
Index: .
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
--- .	(revision 254800)
+++ .	(working copy)

Property changes on: .
___________________________________________________________________
Modified: svn:mergeinfo
   Merged /head/sys:r253927

--g1mh5Fh3W6jKnNId
Content-Type: application/pgp-signature

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.21 (FreeBSD)

iQIcBAEBAgAGBQJSGPAuAAoJEJDCuSvBvK1BnbgP/RDnR0CDCza8/lpHlqKrnIwD
cpVk+J+aJdUznwAGt+wuhQld2BaTXwnaQjy1fCQzjJ9kVjrTqqPeqD0DU9eD6i+V
6aCizKb3EB8BLjK6v5qzkdfficr/ojFX7a7x6pDEP7srOoNmoy+m/ZPy45HVzDbJ
YYXuMPW93bC927Tmbds8qdifhe3BzNXKIHH9NVpU1m4JsuhwR3reFjd4kW2B4FJk
+oMeKdASP2CYUzmniNdVqq55Q6RXc/SXZefcxVLTYwgC0TU456TlXr5dOLZCL+gO
wIBfBHi8/e0L0J/2xKe+5LT8F11jWupWoHKHWuyAFf77N7fMoeUeB9quiu6BaUaT
fZ/Pnvu1xzH2bQoeO0ncscF1xK13f9lz0dFKPnynGxKRdcwcQDACHKI8SJLJIN94
leuCV1EZRJdT8ZFqBUlyejagIKu2CnAYd96jLuloqytqksnBQxym2VPAuz2oNVZJ
m5b67brz7iUcHS1FH81htvAic4tSHcnboPo3uWjbZ1kaHyH7KLyuihPsva8pS0hx
GYBc+UrKVgW70r1/JacX0tNhlN6LKPgMNWIy268z8CBGjpU4Z5oY8RB5aArzwETv
ku+klJqbRj3IOByJZi5Vi3zM0galWrnInLlfOnebNqeiRUft2I7sbdvAsVALKZaS
kgOf13FX7LxmW5FV3nKB
=KIxM
-----END PGP SIGNATURE-----

--g1mh5Fh3W6jKnNId--



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20130824174103.GQ4972>