Date: Tue, 23 Aug 2005 12:33:02 -0700 (PDT) From: Matthew Dillon <dillon@apollo.backplane.com> To: freebsd-hackers@freebsd.org Cc: Kirk McKusick <mckusick@mckusick.com>, Alan Cox <alc@cs.rice.edu> Subject: ffs_balloc() ordering deadlock against vm_fault Message-ID: <200508231933.j7NJX22j056873@apollo.backplane.com>
next in thread | raw e-mail | index | archive | help
I think this applies to FreeBSD, too and I could use some help in figuring
out how best to solve the problem.
I have a user reporting an interesting filesystem/VM deadlock in DragonFly
when running rtorrent. rtorrent has the particular effect of issuing
socket read()'s directly into mmap()'d files whos indirect blocks have
not yet been allocated.
What appears to be occuring is a deadlock between a VM page, data
block buffer, and indirect block buffer.
The first trace below is the process issuing the read() from a socket
into the mmap()'d file. It has VM faulted on the data page it is
trying to copy the data into. It is holding the VM page locked and
calling vnode_pager_haspage() which eventually goes down into the BMAP
code and tries to obtain the filesystem INDIRECT block (-12) related to
that portion of the file. That is where it deadlocks.
#2 0xc02aab43 in tsleep (ident=0x0, flags=0, wmesg=0x0, timo=0) at
/usr/src/sys/kern/kern_synch.c:334
#3 0xc02990bb in acquire (lkp=0xc1ab6000, extflags=33554464,
wanted=1536) at /usr/src/sys/kern/kern_lock.c:128
#4 0xc02994c4 in lockmgr (lkp=0xc1ab6000, flags=33620002,
interlkp=0x0, td=0xc9b65f00)
at /usr/src/sys/kern/kern_lock.c:355
#5 0xc02d8010 in getblk (vp=0xd38c4078, blkno=-12, size=8192,
slpflag=0, slptimeo=0) at thread.h:78
#6 0xc0428dec in ufs_bmaparray (vp=0xd38c4078, bn=924,
bnp=0xd38708e8, ap=0x0, nump=0x0, runp=0xd3870918,
runb=0xd387091c) at /usr/src/sys/vfs/ufs/ufs_bmap.c:178
#7 0xc0428ba6 in ufs_bmap (ap=0x0) at /usr/src/sys/vfs/ufs/ufs_bmap.c:77
#8 0xc042f409 in ufs_vnoperate (ap=0x0) at
/usr/src/sys/vfs/ufs/ufs_vnops.c:2366
#9 0xc02ef974 in vop_bmap (ops=0x0, vp=0x0, bn=0, vpp=0x0, bnp=0x0,
runp=0x0, runb=0x0)
at /usr/src/sys/kern/vfs_vopops.c:862
#10 0xc0443573 in vnode_pager_haspage (object=0xd3874420, pindex=1849,
before=0xd387091c, after=0xd3870918)
at /usr/src/sys/vm/vnode_pager.c:222
#11 0xc0434bf2 in vm_fault_additional_pages (m=0xc0ba727c, rbehind=7,
rahead=8, marray=0xd38709bc,
reqpage=0xd3870980) at vm_pager.h:168
#12 0xc0433f61 in vm_fault (map=0xd3862d00, vaddr=719491072,
The second trace below comes in through the syncer. It is attempting
to call PUTPAGES. This trace shows it getting stuck inside allocbuf.
In particular, ffs_balloc() at this point in the code is holding a
locked buffer representing the INDIRECT block and attempting to
obtain a buffer representing the DATA block. It is stuck trying to
resolve the VM pages within the DATA block (because the trace above
is holding those pages locked). It is doing this while holding the
indirect block buffer locked which the trace above is stuck on. Hence
we wind up with a deadlock.
#2 0xc02d89c1 in allocbuf (bp=0xc1af0600, size=8192) at vm_page.h:549
#3 0xc02d836a in getblk (vp=0x0, blkno=924, size=8192, slpflag=0,
slptimeo=0) at /usr/src/sys/kern/vfs_bio.c:2508
#4 0xc041c809 in ffs_balloc (ap=0xd12da8c0) at
/usr/src/sys/vfs/ufs/ffs_balloc.c:315
#5 0xc02efa85 in vop_balloc (ops=0x0, vp=0x0, startoffset=Unhandled
dwarf expression opcode 0x93
) at /usr/src/sys/kern/vfs_vopops.c:948
#6 0xc04275d4 in ffs_write (ap=0xd12da984) at ufs_readwrite.c:409
#7 0xc02ef524 in vop_write (ops=0x0, vp=0x0, uio=0x0, ioflag=0,
cred=0x0) at /usr/src/sys/kern/vfs_vopops.c:512
#8 0xc0444575 in vnode_pager_generic_putpages (vp=0xd38c4078,
m=0xd12dab10, bytecount=4096, flags=8,
rtvals=0xd12daad0) at /usr/src/sys/vm/vnode_pager.c:1009
#9 0xc0427ca3 in ffs_putpages (ap=0x0) at ufs_readwrite.c:646
#10 0xc02efb42 in vop_putpages (ops=0x0, vp=0x0, m=0x0, count=0,
sync=0, rtvals=0x0, offset=Unhandled dwarf expression opcode 0x93
)
at /usr/src/sys/kern/vfs_vopops.c:1003
#11 0xc0444392 in vnode_pager_putpages (object=0x0, m=0xd12dab10,
count=0, sync=8, rtvals=0xd12daad0)
at /usr/src/sys/vm/vnode_pager.c:902
#12 0xc0440d60 in vm_pageout_flush (mc=0xd12dab10, count=1, flags=8)
at vm_pager.h:146
In looking at the code, in particular the BMAP code, I believe that
what we want to do is to have ffs_balloc() obtain the data block buffer
BEFORE locking the indirect block buffer, and that doing this should
resolve the deadlock.
But this is very complex code and I need an expert opinion. also,
ffs_balloc() itself has softupdates interactions and, frankly,
manipulating that code looks a bit scary to me :-)
So my question is: (1) Is this the best solution to the problem and
(2) maybe Kirk or Alan or some other filesystem expert can have a go
at fixing it. I will attempt to fix it on my side as well, I'm just not
sure that I am approaching the solution correctly and I'm not sure how
the softupdates interactions are going to work out.
-Matt
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200508231933.j7NJX22j056873>