From owner-freebsd-stable@freebsd.org Mon Oct 17 20:16:38 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 64E78AF4DBC for ; Mon, 17 Oct 2016 20:16:38 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-lf0-x234.google.com (mail-lf0-x234.google.com [IPv6:2a00:1450:4010:c07::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 709BE1BF for ; Mon, 17 Oct 2016 20:16:37 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-lf0-x234.google.com with SMTP id b75so306191039lfg.3 for ; Mon, 17 Oct 2016 13:16:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to; bh=KYNGngg2mkmVAmKqQ51ubtDcvZ3BVHtkfGBu4/WOrGo=; b=TlEDgF/HqOlZYJlcxbaLSE/YHSoUa9A1IRsmj1fbialC2JuISZ57LXEjtbUf0Iy0yy fgLbPtn5UN0rHJCU3j20XKNWdPUhUHnQxuuzIYuLdEEHF6/1nxz8n8khx8RZvjzO+pSD UdVGSTA7PvONjaSuLrbFGT7UhVC+XCn+/PAhp+oU1y5JnzqqyCj9AXQHWPLdTTqXNVpM Tf7MHh9gpkOdfwQgubNGCrcVCX0hf+0k0XfdGV31EKO+QwQNmJDdaCqVTGktZkuNzpRh ujtM2k1ynmSjLYkBs0J9owcqH5V23BOwPEw2JYRM/F915XZY+oZqkyqM8Gi5VbyEDrnP Qqxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to; bh=KYNGngg2mkmVAmKqQ51ubtDcvZ3BVHtkfGBu4/WOrGo=; b=PosNeHY5qwfUUFJ3VAPGCqLCdWyBJL8kHYgmWzqEZkMAUbhuRZyW587ZsugMdIKLr5 zSds9tmufFU7KE0pfjkFc9ki2wjLHuGGubaAErYMbJFrwTGPlOk0ZGBKVjp4vQdTdff9 /d0lJw8TelT4p7NJ5B5lkx8+yvK5g1CtRN3NWWH8JD7rC9ay+v+eDJoibzes/GieN5u+ 1GNHB3yRy3Pr44uTZs94vTHoOXHhVuN0RlA+gSj2EHtHSPldfQuI8m4jUxsPrLf1ZU/4 9XdtDIpNigFTQfabXlyvkgKe8ZICtVVR+5JPVinw8+xzxbgPIxPZB42dKDmQh2H40iea TWOg== X-Gm-Message-State: AA6/9RkjSqI0PY3w1dVUvXDt9EjswgezXwY68jYH/rRd+dLiksV29CyGpLDwcJjQRrR+q7Oj X-Received: by 10.28.37.70 with SMTP id l67mr3042wml.44.1476735393826; Mon, 17 Oct 2016 13:16:33 -0700 (PDT) Received: from [10.10.1.58] ([185.97.61.26]) by smtp.gmail.com with ESMTPSA id a5sm20396598wjv.6.2016.10.17.13.16.32 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 17 Oct 2016 13:16:32 -0700 (PDT) Subject: Re: Repeatable panic on ZFS filesystem (used for backups); 11.0-STABLE To: freebsd-stable@freebsd.org References: <3d4f25c9-a262-a373-ec7e-755325f8810b@denninger.net> <9adecd24-6659-0da5-5c05-d0d3957a2cb3@denninger.net> <0f58b11f-0bca-bc08-6f90-4e6e530f9956@denninger.net> <43a67287-f4f8-5d3e-6c5e-b3599c6adb4d@multiplay.co.uk> <76551fd6-0565-ee6c-b0f2-7d472ad6a4b3@denninger.net> From: Steven Hartland Message-ID: <25ff3a3e-77a9-063b-e491-8d10a06e6ae2@multiplay.co.uk> Date: Mon, 17 Oct 2016 21:16:59 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: <76551fd6-0565-ee6c-b0f2-7d472ad6a4b3@denninger.net> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Oct 2016 20:16:38 -0000 Be good to confirm its not an infinite loop by giving it a good bump first. On 17/10/2016 19:58, Karl Denninger wrote: > I can certainly attempt setting that higher.... but is that not just > hiding the problem rather than addressing it? > > > On 10/17/2016 13:54, Steven Hartland wrote: >> You're hitting stack exhaustion, have you tried increasing the kernel >> stack pages? >> It can be changed from /boot/loader.conf >> kern.kstack_pages="6" >> >> Default on amd64 is 4 IIRC >> >> On 17/10/2016 19:08, Karl Denninger wrote: >>> The target (and devices that trigger this) are a pair of 4Gb 7200RPM >>> SATA rotating rust drives (zmirror) with each provider geli-encrypted >>> (that is, the actual devices used for the pool create are the .eli's) >>> >>> The machine generating the problem has both rotating rust devices *and* >>> SSDs, so I can't simply shut TRIM off system-wide and call it a day as >>> TRIM itself is heavily-used; both the boot/root pools and a Postgresql >>> database pool are on SSDs, while several terabytes of lesser-used data >>> is on a pool of Raidz2 that is made up of spinning rust. >> snip... >>> NewFS.denninger.net dumped core - see /var/crash/vmcore.1 >>> >>> Mon Oct 17 09:02:33 CDT 2016 >>> >>> FreeBSD NewFS.denninger.net 11.0-STABLE FreeBSD 11.0-STABLE #13 >>> r307318M: Fri Oct 14 09:23:46 CDT 2016 >>> karl@NewFS.denninger.net:/usr/obj/usr/src/sys/KSD-SMP amd64 >>> >>> panic: double fault >>> >>> GNU gdb 6.1.1 [FreeBSD] >>> Copyright 2004 Free Software Foundation, Inc. >>> GDB is free software, covered by the GNU General Public License, and >>> you are >>> welcome to change it and/or distribute copies of it under certain >>> conditions. >>> Type "show copying" to see the conditions. >>> There is absolutely no warranty for GDB. Type "show warranty" for >>> details. >>> This GDB was configured as "amd64-marcel-freebsd"... >>> >>> Unread portion of the kernel message buffer: >>> >>> Fatal double fault >>> rip = 0xffffffff8220d9ec >>> rsp = 0xfffffe066821f000 >>> rbp = 0xfffffe066821f020 >>> cpuid = 6; apic id = 14 >>> panic: double fault >>> cpuid = 6 >>> KDB: stack backtrace: >>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame >>> 0xfffffe0649d78e30 >>> vpanic() at vpanic+0x182/frame 0xfffffe0649d78eb0 >>> panic() at panic+0x43/frame 0xfffffe0649d78f10 >>> dblfault_handler() at dblfault_handler+0xa2/frame 0xfffffe0649d78f30 >>> Xdblfault() at Xdblfault+0xac/frame 0xfffffe0649d78f30 >>> --- trap 0x17, rip = 0xffffffff8220d9ec, rsp = 0xfffffe066821f000, rbp = >>> 0xfffffe066821f020 --- >>> avl_rotation() at avl_rotation+0xc/frame 0xfffffe066821f020 >>> avl_remove() at avl_remove+0x1c8/frame 0xfffffe066821f070 >>> vdev_queue_io_to_issue() at vdev_queue_io_to_issue+0x87f/frame >>> 0xfffffe066821f530 >>> vdev_queue_io_done() at vdev_queue_io_done+0x83/frame 0xfffffe066821f570 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f5a0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f5f0 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821f650 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f6a0 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821f6e0 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f710 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f760 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821f7c0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f810 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821f850 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f880 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f8d0 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821f930 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f980 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821f9c0 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f9f0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fa40 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821faa0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821faf0 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821fb30 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821fb60 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fbb0 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821fc10 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fc60 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821fca0 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821fcd0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fd20 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821fd80 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fdd0 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821fe10 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821fe40 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fe90 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821fef0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821ff40 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821ff80 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821ffb0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220000 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220060 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682200b0 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682200f0 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220120 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220170 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682201d0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220220 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220260 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220290 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682202e0 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220340 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220390 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682203d0 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220400 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220450 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682204b0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220500 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220540 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220570 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682205c0 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220620 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220670 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682206b0 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682206e0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220730 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220790 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682207e0 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220820 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220850 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682208a0 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220900 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220950 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220990 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682209c0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220a10 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220a70 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220ac0 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220b00 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220b30 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220b80 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220be0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220c30 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220c70 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220ca0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220cf0 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220d50 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220da0 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220de0 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220e10 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220e60 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220ec0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220f10 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220f50 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220f80 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220fd0 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221030 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221080 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682210c0 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682210f0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221140 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682211a0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682211f0 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221230 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221260 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682212b0 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221310 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221360 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682213a0 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682213d0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221420 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221480 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682214d0 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221510 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221540 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221590 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682215f0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221640 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221680 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682216b0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221700 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221760 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682217b0 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682217f0 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221820 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221870 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682218d0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221920 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221960 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221990 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682219e0 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221a40 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221a90 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221ad0 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221b00 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221b50 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221bb0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221c00 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221c40 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221c70 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221cc0 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221d20 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221d70 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221db0 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221de0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221e30 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221e90 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221ee0 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221f20 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221f50 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221fa0 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668222000 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222050 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668222090 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682220c0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222110 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668222170 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682221c0 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668222200 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222230 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222280 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682222e0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222330 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668222370 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682223a0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682223f0 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668222450 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682224a0 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682224e0 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222510 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222560 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682225c0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222610 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668222650 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222680 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682226d0 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668222730 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222780 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682227c0 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682227f0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222840 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682228a0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682228f0 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668222930 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222960 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682229b0 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668222a10 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222a60 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668222aa0 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222ad0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222b20 >>> taskqueue_run_locked() at taskqueue_run_locked+0x14a/frame >>> 0xfffffe0668222b80 >>> taskqueue_thread_loop() at taskqueue_thread_loop+0xe8/frame >>> 0xfffffe0668222bb0 >>> fork_exit() at fork_exit+0x85/frame 0xfffffe0668222bf0 >>> fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0668222bf0 >>> --- trap 0, rip = 0, rsp = 0, rbp = 0 --- >>> Uptime: 4m29s >>> Dumping 1497 out of 24531 >>> MB:..2%..11%..21%..31%..41%..51%..61%..71%..81%..91% >>> >>> Reading symbols from /boot/kernel/zfs.ko...Reading symbols from >>> /usr/lib/debug//boot/kernel/zfs.ko.debug...done. >>> done. >>> Loaded symbols for /boot/kernel/zfs.ko >>> Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from >>> /usr/lib/debug//boot/kernel/opensolaris.ko.debug...done. >>> done. >>> Loaded symbols for /boot/kernel/opensolaris.ko >>> Reading symbols from /boot/kernel/geom_eli.ko...Reading symbols from >>> /usr/lib/debug//boot/kernel/geom_eli.ko.debug...done. >>> done. >>> Loaded symbols for /boot/kernel/geom_eli.ko >>> Reading symbols from /boot/kernel/geom_mirror.ko...Reading symbols from >>> /usr/lib/debug//boot/kernel/geom_mirror.ko.debug...done. >>> done. >>> Loaded symbols for /boot/kernel/geom_mirror.ko >>> Reading symbols from /boot/kernel/aesni.ko...Reading symbols from >>> /usr/lib/debug//boot/kernel/aesni.ko.debug...done. >>> done. >>> Loaded symbols for /boot/kernel/aesni.ko >>> Reading symbols from /boot/kernel/uhid.ko...Reading symbols from >>> /usr/lib/debug//boot/kernel/uhid.ko.debug...done. >>> done. >>> Loaded symbols for /boot/kernel/uhid.ko >>> Reading symbols from /boot/kernel/ums.ko...Reading symbols from >>> /usr/lib/debug//boot/kernel/ums.ko.debug...done. >>> done. >>> Loaded symbols for /boot/kernel/ums.ko >>> Reading symbols from /boot/kernel/ipfw.ko...Reading symbols from >>> /usr/lib/debug//boot/kernel/ipfw.ko.debug...done. >>> done. >>> Loaded symbols for /boot/kernel/ipfw.ko >>> Reading symbols from /boot/kernel/ipfw_nat.ko...Reading symbols from >>> /usr/lib/debug//boot/kernel/ipfw_nat.ko.debug...done. >>> done. >>> Loaded symbols for /boot/kernel/ipfw_nat.ko >>> Reading symbols from /boot/kernel/libalias.ko...Reading symbols from >>> /usr/lib/debug//boot/kernel/libalias.ko.debug...done. >>> done. >>> Loaded symbols for /boot/kernel/libalias.ko >>> Reading symbols from /boot/kernel/daemon_saver.ko...Reading symbols from >>> /usr/lib/debug//boot/kernel/daemon_saver.ko.debug...done. >>> done. >>> Loaded symbols for /boot/kernel/daemon_saver.ko >>> #0 doadump (textdump=1) at pcpu.h:221 >>> 221 pcpu.h: No such file or directory. >>> in pcpu.h >>> (kgdb) #0 doadump (textdump=1) at pcpu.h:221 >>> #1 0xffffffff80ae9b6e in kern_reboot (howto=260) >>> at /usr/src/sys/kern/kern_shutdown.c:366 >>> #2 0xffffffff80aea13b in vpanic (fmt=, >>> ap=) at /usr/src/sys/kern/kern_shutdown.c:759 >>> #3 0xffffffff80ae9f73 in panic (fmt=0x0) >>> at /usr/src/sys/kern/kern_shutdown.c:690 >>> #4 0xffffffff80fd18f2 in dblfault_handler (frame=) >>> at /usr/src/sys/amd64/amd64/trap.c:868 >>> #5 0xffffffff80fb406c in Xdblfault () >>> at /usr/src/sys/amd64/amd64/exception.S:292 >>> #6 0xffffffff8220d9ec in avl_rotation (tree=0xfffff8004dc54510, >>> node=0xfffff8056e45cdd0, balance=2) >>> at /usr/src/sys/cddl/contrib/opensolaris/common/avl/avl.c:304 >>> #7 0xffffffff8220de58 in avl_remove (tree=, >>> data=) >>> at /usr/src/sys/cddl/contrib/opensolaris/common/avl/avl.c:818 >>> #8 0xffffffff822a1eff in vdev_queue_io_to_issue (vq=>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:439 >>> #9 0xffffffff822a2123 in vdev_queue_io_done (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:907 >>> #10 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e3f33d8) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>> #11 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #12 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e3f33d8) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>> #13 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #14 0xffffffff822a216d in vdev_queue_io_done (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>> #15 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e578000) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>> #16 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #17 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e578000) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>> #18 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #19 0xffffffff822a216d in vdev_queue_io_done (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>> #20 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5ec3d8) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>> #21 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #22 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5ec3d8) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>> #23 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #24 0xffffffff822a216d in vdev_queue_io_done (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>> #25 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e48a000) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>> #26 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #27 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e48a000) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>> #28 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #29 0xffffffff822a216d in vdev_queue_io_done (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>> #30 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e57ab88) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>> #31 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #32 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e57ab88) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>> #33 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #34 0xffffffff822a216d in vdev_queue_io_done (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>> #35 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800a69a3000) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>> #36 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #37 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff800a69a3000) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>> #38 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #39 0xffffffff822a216d in vdev_queue_io_done (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>> #40 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5c8000) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>> #41 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #42 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5c8000) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>> #43 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #44 0xffffffff822a216d in vdev_queue_io_done (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>> #45 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a23f9000) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>> #46 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #47 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff802a23f9000) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>> #48 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #49 0xffffffff822a216d in vdev_queue_io_done (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>> #50 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5147b0) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>> #51 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #52 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5147b0) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>> #53 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #54 0xffffffff822a216d in vdev_queue_io_done (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>> #55 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e977000) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>> #56 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #57 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e977000) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>> #58 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #59 0xffffffff822a216d in vdev_queue_io_done (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>> #60 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e90b3d8) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>> #61 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #62 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e90b3d8) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>> #63 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #64 0xffffffff822a216d in vdev_queue_io_done (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>> #65 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800b1160b88) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>> #66 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #67 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff800b1160b88) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>> #68 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #69 0xffffffff822a216d in vdev_queue_io_done (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>> #70 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e4f53d8) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>> #71 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #72 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e4f53d8) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>> #73 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #74 0xffffffff822a216d in vdev_queue_io_done (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>> #75 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e8ebb88) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>> #76 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #77 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e8ebb88) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>> #78 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #79 0xffffffff822a216d in vdev_queue_io_done (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>> #80 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e8d9b88) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>> #81 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #82 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e8d9b88) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>> #83 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #84 0xffffffff822a216d in vdev_queue_io_done (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>> #85 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a21a03d8) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>> #86 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #87 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff802a21a03d8) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>> #88 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #89 0xffffffff822a216d in vdev_queue_io_done (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>> #90 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e6903d8) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>> #91 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #92 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e6903d8) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>> #93 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #94 0xffffffff822a216d in vdev_queue_io_done (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>> #95 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a22dc7b0) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>> #96 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #97 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff802a22dc7b0) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>> #98 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #99 0xffffffff822a216d in vdev_queue_io_done (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>> #100 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5fb7b0) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>> #101 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #102 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5fb7b0) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>> #103 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #104 0xffffffff822a216d in vdev_queue_io_done (zio=>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>> #105 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5ae3d8) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>> #106 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #107 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5ae3d8) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>> #108 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #109 0xffffffff822a216d in vdev_queue_io_done (zio=>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>> #110 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a2131b88) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>> #111 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #112 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff802a2131b88) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>> #113 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #114 0xffffffff822a216d in vdev_queue_io_done (zio=>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>> #115 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e80b7b0) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>> #116 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #117 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e80b7b0) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>> #118 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #119 0xffffffff822a216d in vdev_queue_io_done (zio=>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>> #120 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800b14b9000) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>> #121 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #122 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff800b14b9000) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>> #123 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #124 0xffffffff822a216d in vdev_queue_io_done (zio=>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>> #125 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a21fa3d8) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>> #126 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #127 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff802a21fa3d8) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>> #128 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #129 0xffffffff822a216d in vdev_queue_io_done (zio=>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>> #130 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5333d8) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>> #131 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #132 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5333d8) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>> #133 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #134 0xffffffff822a216d in vdev_queue_io_done (zio=>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>> #135 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e90ab88) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>> #136 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #137 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e90ab88) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>> #138 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #139 0xffffffff822a216d in vdev_queue_io_done (zio=>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>> #140 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e4047b0) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>> #141 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #142 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e4047b0) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>> #143 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #144 0xffffffff822a216d in vdev_queue_io_done (zio=>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>> #145 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e7083d8) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>> #146 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #147 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e7083d8) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>> #148 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #149 0xffffffff822a216d in vdev_queue_io_done (zio=>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>> #150 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a23917b0) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>> #151 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #152 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff802a23917b0) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>> #153 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #154 0xffffffff822a216d in vdev_queue_io_done (zio=>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>> #155 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e636000) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>> #156 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #157 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e636000) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>> #158 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #159 0xffffffff822a216d in vdev_queue_io_done (zio=>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>> #160 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5afb88) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>> #161 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #162 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5afb88) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>> #163 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #164 0xffffffff822a216d in vdev_queue_io_done (zio=>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>> #165 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800a69523d8) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>> #166 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #167 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff800a69523d8) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>> #168 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #169 0xffffffff822a216d in vdev_queue_io_done (zio=>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>> #170 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e47c000) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>> #171 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #172 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e47c000) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>> #173 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #174 0xffffffff822a216d in vdev_queue_io_done (zio=>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>> #175 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800b17b23d8) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>> #176 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #177 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff800b17b23d8) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>> #178 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #179 0xffffffff822a216d in vdev_queue_io_done (zio=>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>> #180 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800a6d367b0) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>> #181 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #182 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff800a6d367b0) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>> #183 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #184 0xffffffff822a216d in vdev_queue_io_done (zio=>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>> #185 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e99fb88) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>> #186 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #187 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e99fb88) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>> #188 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #189 0xffffffff822a216d in vdev_queue_io_done (zio=>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>> #190 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5227b0) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>> #191 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #192 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5227b0) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>> #193 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #194 0xffffffff822a216d in vdev_queue_io_done (zio=>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>> #195 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800a6d3c000) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>> #196 0xffffffff822bbefd in zio_execute (zio=) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>> #197 0xffffffff80b4895a in taskqueue_run_locked (queue=>> out>) >>> at /usr/src/sys/kern/subr_taskqueue.c:454 >>> #198 0xffffffff80b49b58 in taskqueue_thread_loop (arg=>> optimized out>) >>> at /usr/src/sys/kern/subr_taskqueue.c:724 >>> #199 0xffffffff80a9f255 in fork_exit ( >>> callout=0xffffffff80b49a70 , >>> arg=0xfffff8057caa72a0, frame=0xfffffe0668222c00) >>> at /usr/src/sys/kern/kern_fork.c:1040 >>> #200 0xffffffff80fb44ae in fork_trampoline () >>> at /usr/src/sys/amd64/amd64/exception.S:611 >>> #201 0x0000000000000000 in ?? () >>> Current language: auto; currently minimal >>> (kgdb) >>> >>> >>>>> -- >>>>> Karl Denninger >>>>> karl@denninger.net >>>>> /The Market Ticker/ >>>>> /[S/MIME encrypted email preferred]/ >> _______________________________________________ >> freebsd-stable@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-stable >> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"