From owner-freebsd-stable@freebsd.org Sat Jul 2 18:31:24 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6179EB8FA81; Sat, 2 Jul 2016 18:31:24 +0000 (UTC) (envelope-from dcrosstech@gmail.com) Received: from mail-yw0-x241.google.com (mail-yw0-x241.google.com [IPv6:2607:f8b0:4002:c05::241]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 232A02FA5; Sat, 2 Jul 2016 18:31:24 +0000 (UTC) (envelope-from dcrosstech@gmail.com) Received: by mail-yw0-x241.google.com with SMTP id f75so1674076ywb.3; Sat, 02 Jul 2016 11:31:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to; bh=elkJZWGeMHtK533zurGlXYZV7HdKI6pS4Cy3ww05f6I=; b=kxRhP0sZbcBPgMm51A//ciu6ZeAaGwqIVe/x3i153xRw876qiIUYvNng9Z66liK7Xh bfHXLxKLJffOSs6aLmWAa5AgYK+TxfMzCJKbysj95smBXWRaeEgIuLJTqBjoEI/guUDt rHeqF2zWjXBPIYzsOg+p1RPAckhEo2xbLssRAFu7bETnD1IGYvEHUE9HQ4cArKxG8jm7 yt+A1F8vImRXLmPo8IzVT3SDU1jf3TDiajAXC9lp7CqdeT9BOUpGA8W4vAAlvDdzwIKK bP2TrSYpabDgmVQi2loQAaSLUXyUTpyNJhHlL3MIbFfwtwQK3hYeTKVq/TquDLbGXy6W TJeQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=elkJZWGeMHtK533zurGlXYZV7HdKI6pS4Cy3ww05f6I=; b=fXKat9QoarpTtO98M+RPMyS1mnxnaPg1K03bU7RU1NIStdVyjDt4fXsuM1RQx3zbuf T4obvi0pXMNkEcoJ6f32G6yJgivkvTBFIklAp950sXMuD447TNHZsExe2GrJSSlQJF/r DUSS0dHyc0fiZYkhVGTw4ChxsFJQZnj2cWVI48hEeuOoD/L20bAeai4zbgUkAqzI9mJH 2iKah0i352+equWFQbMZkM/+J3pqLGWiGxF9eACkggUKEk2/8BlU+lhs1IQYgzfVHzKs IP0bd+/duUsH99eSJGUf2kYYricEU7GooEHOkMx/vIPk+KX2LgbTuYG1syrq1XDki6Zh aHYw== X-Gm-Message-State: ALyK8tLBB08ru9zEPzZqs2nrdwDEfmn+TDlyi7YeOCCaTZD3qb9aP2dleLVgxoGxwKM/BvZUClCBujB5Wffm7Q== X-Received: by 10.129.80.131 with SMTP id e125mr2684540ywb.45.1467484283184; Sat, 02 Jul 2016 11:31:23 -0700 (PDT) MIME-Version: 1.0 Received: by 10.37.212.66 with HTTP; Sat, 2 Jul 2016 11:31:22 -0700 (PDT) From: David Cross Date: Sat, 2 Jul 2016 14:31:22 -0400 Message-ID: Subject: Reproducable panic in FFS with softupdates and no journaling (10.3-RELEASE-pLATEST) To: freebsd-stable@freebsd.org, freebsd-hackers@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Jul 2016 18:31:24 -0000 Ok, I have been trying to trace this down for awhile..I know quite a bit about it.. but there's a lot I don't know, or I would have a patch. I have been trying to solve this on my own, but bringing in some outside assistance will let me move on with my life. First up: The stacktrace (from a debugging kernel, with coredump #0 doadump (textdump=1) at /usr/src/sys/kern/kern_shutdown.c:298 #1 0xffffffff8071018a in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:486 #2 0xffffffff80710afc in vpanic ( fmt=0xffffffff80c7a325 "softdep_deallocate_dependencies: dangling deps b_ioflags: %d, b_bufsize: %ld, b_flags: %d, bo_flag: %d", ap=0xfffffe023ae5cf40) at /usr/src/sys/kern/kern_shutdown.c:889 #3 0xffffffff807108c0 in panic ( fmt=0xffffffff80c7a325 "softdep_deallocate_dependencies: dangling deps b_ioflags: %d, b_bufsize: %ld, b_flags: %d, bo_flag: %d") at /usr/src/sys/kern/kern_shutdown.c:818 #4 0xffffffff80a7c841 in softdep_deallocate_dependencies ( bp=0xfffffe01f030e148) at /usr/src/sys/ufs/ffs/ffs_softdep.c:14099 #5 0xffffffff807f793f in buf_deallocate (bp=0xfffffe01f030e148) at buf.h:428 #6 0xffffffff807f59c9 in brelse (bp=0xfffffe01f030e148) at /usr/src/sys/kern/vfs_bio.c:1599 #7 0xffffffff807f3132 in bufwrite (bp=0xfffffe01f030e148) at /usr/src/sys/kern/vfs_bio.c:1180 #8 0xffffffff80ab226a in bwrite (bp=0xfffffe01f030e148) at buf.h:395 #9 0xffffffff80aafb1b in ffs_write (ap=0xfffffe023ae5d2b8) at /usr/src/sys/ufs/ffs/ffs_vnops.c:800 #10 0xffffffff80bdf0ed in VOP_WRITE_APV (vop=0xffffffff80f15480, a=0xfffffe023ae5d2b8) at vnode_if.c:999 #11 0xffffffff80b1d02e in VOP_WRITE (vp=0xfffff80077e7a000, uio=0xfffffe023ae5d378, ioflag=8323232, cred=0xfffff80004235000) at vnode_if.h:413 #12 0xffffffff80b1ce97 in vnode_pager_generic_putpages (vp=0xfffff80077e7a000, ma=0xfffffe023ae5d660, bytecount=16384, flags=1, rtvals=0xfffffe023ae5d580) at /usr/src/sys/vm/vnode_pager.c:1138 #13 0xffffffff80805a57 in vop_stdputpages (ap=0xfffffe023ae5d478) at /usr/src/sys/kern/vfs_default.c:760 #14 0xffffffff80be201e in VOP_PUTPAGES_APV (vop=0xffffffff80f00218, a=0xfffffe023ae5d478) at vnode_if.c:2861 #15 0xffffffff80b1d7e3 in VOP_PUTPAGES (vp=0xfffff80077e7a000, m=0xfffffe023ae5d660, count=16384, sync=1, rtvals=0xfffffe023ae5d580, offset=0) at vnode_if.h:1189 #16 0xffffffff80b196f3 in vnode_pager_putpages (object=0xfffff8014a1fce00, m=0xfffffe023ae5d660, count=4, flags=1, rtvals=0xfffffe023ae5d580) at /usr/src/sys/vm/vnode_pager.c:1016 #17 0xffffffff80b0a605 in vm_pager_put_pages (object=0xfffff8014a1fce00, m=0xfffffe023ae5d660, count=4, flags=1, rtvals=0xfffffe023ae5d580) at vm_pager.h:144 #18 0xffffffff80b0a18c in vm_pageout_flush (mc=0xfffffe023ae5d660, count=4, flags=1, mreq=0, prunlen=0xfffffe023ae5d6f8, eio=0xfffffe023ae5d77c) at /usr/src/sys/vm/vm_pageout.c:533 #19 0xffffffff80afec76 in vm_object_page_collect_flush ( object=0xfffff8014a1fce00, p=0xfffff8023a882370, pagerflags=1, flags=1, clearobjflags=0xfffffe023ae5d780, eio=0xfffffe023ae5d77c) at /usr/src/sys/vm/vm_object.c:971 #20 0xffffffff80afe91e in vm_object_page_clean (object=0xfffff8014a1fce00, start=0, end=0, flags=1) at /usr/src/sys/vm/vm_object.c:897 #21 0xffffffff80afe1fa in vm_object_terminate (object=0xfffff8014a1fce00) at /usr/src/sys/vm/vm_object.c:735 #22 0xffffffff80b1a0f1 in vnode_destroy_vobject (vp=0xfffff80077e7a000) at /usr/src/sys/vm/vnode_pager.c:164 #23 0xffffffff80abb191 in ufs_prepare_reclaim (vp=0xfffff80077e7a000) at /usr/src/sys/ufs/ufs/ufs_inode.c:190 #24 0xffffffff80abb1f9 in ufs_reclaim (ap=0xfffffe023ae5d968) at /usr/src/sys/ufs/ufs/ufs_inode.c:219 #25 0xffffffff80be0ade in VOP_RECLAIM_APV (vop=0xffffffff80f15ec0, a=0xfffffe023ae5d968) at vnode_if.c:2019 #26 0xffffffff80827849 in VOP_RECLAIM (vp=0xfffff80077e7a000, td=0xfffff80008931960) at vnode_if.h:830 #27 0xffffffff808219a9 in vgonel (vp=0xfffff80077e7a000) at /usr/src/sys/kern/vfs_subr.c:2943 #28 0xffffffff808294e8 in vlrureclaim (mp=0xfffff80008b2e000) at /usr/src/sys/kern/vfs_subr.c:882 #29 0xffffffff80828ea9 in vnlru_proc () at /usr/src/sys/kern/vfs_subr.c:1000 #30 0xffffffff806b66c5 in fork_exit (callout=0xffffffff80828c50 , arg=0x0, frame=0xfffffe023ae5dc00) at /usr/src/sys/kern/kern_fork.c:1027 #31 0xffffffff80b21dce in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:611 #32 0x0000000000000000 in ?? () This is a kernel compiled -O -g, its "almost" GENERIC; the only difference is some removed drivers, I have reproduced this on a few different kernels, including a BHYVE one so I can poke at it and not take out the main machine. The reproduction as it currently stands needs to have jails running, but I don't believe this is a jail interaction, I think its just that the process that sets up the problem happens to be running in a jail. The step is "start jail; run "find /mountpoint -xdev >/dev/null" on the filesystem, when the vnlru forces the problem vnode out the system panics. I made a few modifications to the kernel to spit out information about the buf that causes the issue, but that is it. Information about the buf in question; it has a single softdependency worklist for direct allocation: (kgdb) print *bp->b_dep->lh_first $6 = {wk_list = {le_next = 0x0, le_prev = 0xfffffe01f030e378}, wk_mp = 0xfffff80008b2e000, wk_type = 4, wk_state = 163841} The file that maps to that buffer: ls -lh MOUNTPOINT/jails/mail/var/imap/db/__db.002 -rw------- 1 cyrus cyrus 24K Jul 1 20:32 MOUNTPOINT/jails/mail/var/imap/db/__db.002 Any help is appreciated, until then I will keep banging my head against the proverbial wall on this :)