Date: Wed, 21 Aug 2013 16:10:32 +0300 From: Konstantin Belousov <kostikbel@gmail.com> To: Rick Macklem <rmacklem@uoguelph.ca> Cc: freebsd-stable <freebsd-stable@freebsd.org>, J David <j.david.lists@gmail.com>, scottl <scottl@freebsd.org>, Michael Tratz <michael@esosoft.com>, Steven Hartland <killing@multiplay.co.uk> Subject: Re: NFS deadlock on 9.2-Beta1 Message-ID: <20130821131032.GX4972@kib.kiev.ua> In-Reply-To: <937358501.11648801.1377037096794.JavaMail.root@uoguelph.ca> References: <CABXB=RTMV840C3dCiBa=tJ_ru9BS2mpFTGr5h6BoH5AbjL1kYQ@mail.gmail.com> <937358501.11648801.1377037096794.JavaMail.root@uoguelph.ca>
next in thread | previous in thread | raw e-mail | index | archive | help
--R+7SZZmxN2wT6E8p
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On Tue, Aug 20, 2013 at 06:18:16PM -0400, Rick Macklem wrote:
> J David wrote:
> > On Thu, Aug 15, 2013 at 5:39 PM, Rick Macklem <rmacklem@uoguelph.ca>
> > wrote:
> > > Have you been able to pass the debugging info on to Kostik?
> > >
> > > It would be really nice to get this fixed for FreeBSD9.2.
> >=20
> > You're probably not talking to me, but headway here is slow. At our
> > location, we have been continuing to test releng/9.2 extensively, but
> > with r250907 reverted. Since reverting it solves the issue, and
> > since
> > there haven't been any further changes to releng/9.2 that might also
> > resolve this issue, re-applying r250907 is perceived here as
> > un-fixing
> > a problem. Enthusiasm for doing so is correspondingly low, even if
> > the purpose is to gather debugging info. :(
> >=20
> > However, after finally having clearance to test releng/9.2 r254540
> > with r250907 included and with DDB on five nodes. The problem
> > cropped
> > up in about an hour. Two threads in one process deadlocked, was
> > perfect. Got it into DDB and saw the stack trace was scrolling off
> > so
> > there was no way to copy it by hand. Also, the machine's disk is
> > smaller than physical RAM, so no dump file. :(
> >=20
> > Here's what is available so far:
> >=20
> > db> show proc 33362
> >=20
> > Process 33362 (httpd) at 0xcd225b50:
> >=20
> > state: NORMAL
> >=20
> > uid: 25000 gids: 25000
> >=20
> > parent: pid 25104 at 0xc95f92d4
> >=20
> > ABI: FreeBSD ELF32
> >=20
> > arguments: /usr/local/libexec/httpd
> >=20
> > threads: 3
> >=20
> > 100405 D newnfs 0xc9b875e4 httpd
> >=20
> Ok, so this one is waiting for an NFS vnode lock.
>=20
> > 100393 D pgrbwt 0xc43a30c0 httpd
> >=20
> This one is sleeping in vm_page_grab() { which I suspect has
> been called from kern_sendfile() with a shared vnode lock held,
> from what I saw on the previous debug info }.
>=20
> > 100755 S uwait 0xc84b7c80 httpd
> >=20
> >=20
> > Not much to go on. :( Maybe these five can be configured with serial
> > consoles.
> >=20
> > So, inquiries are continuing, but the answer to "does this still
> > happen on 9.2-RC2?" is definitely yes.
> >=20
> Since r250027 moves a vn_lock() to before the vm_page_grab() call in
> kern_sendfile(), I suspect that is the cause of the deadlock. (r250027
> is one of the 3 commits MFC'd by r250907)
>=20
> I don't know if it would be safe to VOP_UNLOCK() the vnode after VOP_GETA=
TTR()
> and then put the vn_lock() call that comes after vm_page_grab() back in o=
r whether
> r250027 should be reverted (getting rid of the VOP_GETATTR() and going ba=
ck to
> using the size in the vm stuff).
>=20
> Hopefully Kostik will know what is best to do with it now, rick
I already described what to do with this. I need the debugging
information to see what is going on. Without the data, it is only
wasted time of everybody involved.
Some technical notes. The sendfile() uses shared lock for the duration
of vnode i/o, so any thread which is sleeping on the vnode lock cannot
be in the sendfile path, at least for UFS and NFS which do support true
shared locks.
The right lock order is vnode lock -> page busy wait. From this PoV,
the ordering in the sendfile is correct. Rick, are you aware of any
situation where the VOP_READ in nfs client could drop vnode lock
and then re-acquire it ? I was not able to find this from the code
inspection. But, if such situation exists, it would be problematic in 9.
Last note. The HEAD dropped pre-busying pages in the sendfile() syscall.
As I understand, this is because new Attilio' busy implementation cannot
support both busy and sbusy states simultaneously, and vfs_busy_pages()/
vfs_drain_busy_pages() actually created such situation. I think that
because the sbusy is removed from the sendfile(), and the vm object
lock is dropped, there is no sense to require vm_page_grab() to wait
for the busy state to clean. It is done by buffer cache or filesystem
code later. See the patch at the end.
Still, I do not know what happens in the supposedly reported deadlock.
diff --git a/sys/kern/uipc_syscalls.c b/sys/kern/uipc_syscalls.c
index 4797444..b974f53 100644
--- a/sys/kern/uipc_syscalls.c
+++ b/sys/kern/uipc_syscalls.c
@@ -2230,7 +2230,8 @@ retry_space:
pindex =3D OFF_TO_IDX(off);
VM_OBJECT_WLOCK(obj);
pg =3D vm_page_grab(obj, pindex, VM_ALLOC_NOBUSY |
- VM_ALLOC_NORMAL | VM_ALLOC_WIRED | VM_ALLOC_RETRY);
+ VM_ALLOC_IGN_SBUSY | VM_ALLOC_NORMAL |
+ VM_ALLOC_WIRED | VM_ALLOC_RETRY);
=20
/*
* Check if page is valid for what we need,
--R+7SZZmxN2wT6E8p
Content-Type: application/pgp-signature
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.21 (FreeBSD)
iQIcBAEBAgAGBQJSFLxHAAoJEJDCuSvBvK1Bys8P/3SIqT1N5qU0pjpooxusoCxM
esD8MxI+fO6s2hZgCMv3YoPL8YIm9qceKHm2wFsGIb6LBVO9+qFLgQrat+lMmrZ2
Hn57gqThdrDVZ15sM8nNTgHsc1TQ+6IJgwonPBm39BwHYrBnoHpDbp8p0aVlUZHN
m9wDIQtPi/aOxDRBJ93MmvdCopk3pwmvxMXilPT3OKH1dXyV99o7hxYzqVM2PwxF
taBKQbOhFKK4LqGEpPYW3QV3n7vWcs5GTM/Vez9M7ayVH3JOrABjT67oDuZH++9m
lcldRasiQ05lFfDjW2mSPSwf8sVsyFWfOo5I7RUaiXnGy2IovJ90hJAJC8k6c0dr
5kL3pk+1oPgtUMjt8+I2pqtipcvqoffwYDIb5KdCZdZ+Yw+DsfaRfOrv6k4XWjCX
bQY5Btn+ckXq5482oSNX+yh1h1WkeL1gnH2mR8Eog1GsYlWu10U5koX2H/1LySEz
sFRfrsVKeIDu+ly+7djFgtTo76AI/3nL4iEScDHkdhXs3/gOa7s2JZGz5BFD1226
j6hLRJLq+ZDtvXRZ0AhuVYe8EyA1FSB4Fc9IZJpOLGhJJbM6sokLFFr+Ax8tPySv
nPiBbqRNcnkcF9c8EfnQsR1BEyqG1RFepEC/NM78qOS/Y8YSuKyfj3+evmKiTQFN
LsofJq4TycXnXKRwyd2b
=YNpV
-----END PGP SIGNATURE-----
--R+7SZZmxN2wT6E8p--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20130821131032.GX4972>