From owner-freebsd-hackers@FreeBSD.ORG Tue Aug 23 19:33:15 2005 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0494F16A420 for ; Tue, 23 Aug 2005 19:33:15 +0000 (GMT) (envelope-from dillon@apollo.backplane.com) Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3FD0143D45 for ; Tue, 23 Aug 2005 19:33:14 +0000 (GMT) (envelope-from dillon@apollo.backplane.com) Received: from apollo.backplane.com (localhost [127.0.0.1]) by apollo.backplane.com (8.12.9p2/8.12.9) with ESMTP id j7NJX2Yk056874; Tue, 23 Aug 2005 12:33:03 -0700 (PDT) (envelope-from dillon@apollo.backplane.com) Received: (from dillon@localhost) by apollo.backplane.com (8.12.9p2/8.12.9/Submit) id j7NJX22j056873; Tue, 23 Aug 2005 12:33:02 -0700 (PDT) (envelope-from dillon) Date: Tue, 23 Aug 2005 12:33:02 -0700 (PDT) From: Matthew Dillon Message-Id: <200508231933.j7NJX22j056873@apollo.backplane.com> To: freebsd-hackers@freebsd.org Cc: Kirk McKusick , Alan Cox Subject: ffs_balloc() ordering deadlock against vm_fault X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Aug 2005 19:33:15 -0000 I think this applies to FreeBSD, too and I could use some help in figuring out how best to solve the problem. I have a user reporting an interesting filesystem/VM deadlock in DragonFly when running rtorrent. rtorrent has the particular effect of issuing socket read()'s directly into mmap()'d files whos indirect blocks have not yet been allocated. What appears to be occuring is a deadlock between a VM page, data block buffer, and indirect block buffer. The first trace below is the process issuing the read() from a socket into the mmap()'d file. It has VM faulted on the data page it is trying to copy the data into. It is holding the VM page locked and calling vnode_pager_haspage() which eventually goes down into the BMAP code and tries to obtain the filesystem INDIRECT block (-12) related to that portion of the file. That is where it deadlocks. #2 0xc02aab43 in tsleep (ident=0x0, flags=0, wmesg=0x0, timo=0) at /usr/src/sys/kern/kern_synch.c:334 #3 0xc02990bb in acquire (lkp=0xc1ab6000, extflags=33554464, wanted=1536) at /usr/src/sys/kern/kern_lock.c:128 #4 0xc02994c4 in lockmgr (lkp=0xc1ab6000, flags=33620002, interlkp=0x0, td=0xc9b65f00) at /usr/src/sys/kern/kern_lock.c:355 #5 0xc02d8010 in getblk (vp=0xd38c4078, blkno=-12, size=8192, slpflag=0, slptimeo=0) at thread.h:78 #6 0xc0428dec in ufs_bmaparray (vp=0xd38c4078, bn=924, bnp=0xd38708e8, ap=0x0, nump=0x0, runp=0xd3870918, runb=0xd387091c) at /usr/src/sys/vfs/ufs/ufs_bmap.c:178 #7 0xc0428ba6 in ufs_bmap (ap=0x0) at /usr/src/sys/vfs/ufs/ufs_bmap.c:77 #8 0xc042f409 in ufs_vnoperate (ap=0x0) at /usr/src/sys/vfs/ufs/ufs_vnops.c:2366 #9 0xc02ef974 in vop_bmap (ops=0x0, vp=0x0, bn=0, vpp=0x0, bnp=0x0, runp=0x0, runb=0x0) at /usr/src/sys/kern/vfs_vopops.c:862 #10 0xc0443573 in vnode_pager_haspage (object=0xd3874420, pindex=1849, before=0xd387091c, after=0xd3870918) at /usr/src/sys/vm/vnode_pager.c:222 #11 0xc0434bf2 in vm_fault_additional_pages (m=0xc0ba727c, rbehind=7, rahead=8, marray=0xd38709bc, reqpage=0xd3870980) at vm_pager.h:168 #12 0xc0433f61 in vm_fault (map=0xd3862d00, vaddr=719491072, The second trace below comes in through the syncer. It is attempting to call PUTPAGES. This trace shows it getting stuck inside allocbuf. In particular, ffs_balloc() at this point in the code is holding a locked buffer representing the INDIRECT block and attempting to obtain a buffer representing the DATA block. It is stuck trying to resolve the VM pages within the DATA block (because the trace above is holding those pages locked). It is doing this while holding the indirect block buffer locked which the trace above is stuck on. Hence we wind up with a deadlock. #2 0xc02d89c1 in allocbuf (bp=0xc1af0600, size=8192) at vm_page.h:549 #3 0xc02d836a in getblk (vp=0x0, blkno=924, size=8192, slpflag=0, slptimeo=0) at /usr/src/sys/kern/vfs_bio.c:2508 #4 0xc041c809 in ffs_balloc (ap=0xd12da8c0) at /usr/src/sys/vfs/ufs/ffs_balloc.c:315 #5 0xc02efa85 in vop_balloc (ops=0x0, vp=0x0, startoffset=Unhandled dwarf expression opcode 0x93 ) at /usr/src/sys/kern/vfs_vopops.c:948 #6 0xc04275d4 in ffs_write (ap=0xd12da984) at ufs_readwrite.c:409 #7 0xc02ef524 in vop_write (ops=0x0, vp=0x0, uio=0x0, ioflag=0, cred=0x0) at /usr/src/sys/kern/vfs_vopops.c:512 #8 0xc0444575 in vnode_pager_generic_putpages (vp=0xd38c4078, m=0xd12dab10, bytecount=4096, flags=8, rtvals=0xd12daad0) at /usr/src/sys/vm/vnode_pager.c:1009 #9 0xc0427ca3 in ffs_putpages (ap=0x0) at ufs_readwrite.c:646 #10 0xc02efb42 in vop_putpages (ops=0x0, vp=0x0, m=0x0, count=0, sync=0, rtvals=0x0, offset=Unhandled dwarf expression opcode 0x93 ) at /usr/src/sys/kern/vfs_vopops.c:1003 #11 0xc0444392 in vnode_pager_putpages (object=0x0, m=0xd12dab10, count=0, sync=8, rtvals=0xd12daad0) at /usr/src/sys/vm/vnode_pager.c:902 #12 0xc0440d60 in vm_pageout_flush (mc=0xd12dab10, count=1, flags=8) at vm_pager.h:146 In looking at the code, in particular the BMAP code, I believe that what we want to do is to have ffs_balloc() obtain the data block buffer BEFORE locking the indirect block buffer, and that doing this should resolve the deadlock. But this is very complex code and I need an expert opinion. also, ffs_balloc() itself has softupdates interactions and, frankly, manipulating that code looks a bit scary to me :-) So my question is: (1) Is this the best solution to the problem and (2) maybe Kirk or Alan or some other filesystem expert can have a go at fixing it. I will attempt to fix it on my side as well, I'm just not sure that I am approaching the solution correctly and I'm not sure how the softupdates interactions are going to work out. -Matt