Skip site navigation (1)Skip section navigation (2)
Date:      Mon, 17 Oct 2016 15:22:44 -0500
From:      Karl Denninger <karl@denninger.net>
To:        freebsd-stable@freebsd.org
Subject:   Re: Repeatable panic on ZFS filesystem (used for backups); 11.0-STABLE
Message-ID:  <26e092b2-17c6-8744-5035-d0853d733870@denninger.net>
In-Reply-To: <25ff3a3e-77a9-063b-e491-8d10a06e6ae2@multiplay.co.uk>
References:  <3d4f25c9-a262-a373-ec7e-755325f8810b@denninger.net> <9adecd24-6659-0da5-5c05-d0d3957a2cb3@denninger.net> <CANCZdfq5QCDNhLY5GOpmBoh5ONYy2VPteuaMhQ2=3v%2B0vcoM0g@mail.gmail.com> <0f58b11f-0bca-bc08-6f90-4e6e530f9956@denninger.net> <43a67287-f4f8-5d3e-6c5e-b3599c6adb4d@multiplay.co.uk> <76551fd6-0565-ee6c-b0f2-7d472ad6a4b3@denninger.net> <25ff3a3e-77a9-063b-e491-8d10a06e6ae2@multiplay.co.uk>

next in thread | previous in thread | raw e-mail | index | archive | help
This is a cryptographically signed message in MIME format.

--------------ms020209010902040701010306
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

Since I cleared it (by setting TRIM off on the test machine, rebooting,
importing the pool and noting that it did not panic -- pulled drives,
re-inserted into the production machine and ran backup routine -- all
was normal) it may be a while before I see it again (a week or so is usua=
l.)

It appears to be related to entropy in the filesystem that comes up as
"eligible" to be removed from the backup volume, which (not
surprisingly) tends to happen a few days after I do a new world build or
something similar (the daily and/or periodic snapshots roll off at about
that point.)

I don't happen to have a spare pair of high-performance SSDs I can stick
in the sandbox machine in an attempt to force the condition to assert
itself in test, unfortunately.

I *am* concerned that it's not "simple" stack exhaustion because setting
the max outstanding TRIMs on a per-vdev basis down quite-dramatically
did *not* prevent it from happening -- and if it was simply stack depth
related I would have expected that to put a stop to it.

On 10/17/2016 15:16, Steven Hartland wrote:
> Be good to confirm its not an infinite loop by giving it a good bump
> first.
>
> On 17/10/2016 19:58, Karl Denninger wrote:
>> I can certainly attempt setting that higher.... but is that not just
>> hiding the problem rather than addressing it?
>>
>>
>> On 10/17/2016 13:54, Steven Hartland wrote:
>>> You're hitting stack exhaustion, have you tried increasing the kernel=

>>> stack pages?
>>> It can be changed from /boot/loader.conf
>>> kern.kstack_pages=3D"6"
>>>
>>> Default on amd64 is 4 IIRC
>>>
>>> On 17/10/2016 19:08, Karl Denninger wrote:
>>>> The target (and devices that trigger this) are a pair of 4Gb 7200RPM=

>>>> SATA rotating rust drives (zmirror) with each provider geli-encrypte=
d
>>>> (that is, the actual devices used for the pool create are the .eli's=
)
>>>>
>>>> The machine generating the problem has both rotating rust devices
>>>> *and*
>>>> SSDs, so I can't simply shut TRIM off system-wide and call it a day =
as
>>>> TRIM itself is heavily-used; both the boot/root pools and a Postgres=
ql
>>>> database pool are on SSDs, while several terabytes of lesser-used da=
ta
>>>> is on a pool of Raidz2 that is made up of spinning rust.
>>> snip...
>>>> NewFS.denninger.net dumped core - see /var/crash/vmcore.1
>>>>
>>>> Mon Oct 17 09:02:33 CDT 2016
>>>>
>>>> FreeBSD NewFS.denninger.net 11.0-STABLE FreeBSD 11.0-STABLE #13
>>>> r307318M: Fri Oct 14 09:23:46 CDT 2016
>>>> karl@NewFS.denninger.net:/usr/obj/usr/src/sys/KSD-SMP  amd64
>>>>
>>>> panic: double fault
>>>>
>>>> GNU gdb 6.1.1 [FreeBSD]
>>>> Copyright 2004 Free Software Foundation, Inc.
>>>> GDB is free software, covered by the GNU General Public License, and=

>>>> you are
>>>> welcome to change it and/or distribute copies of it under certain
>>>> conditions.
>>>> Type "show copying" to see the conditions.
>>>> There is absolutely no warranty for GDB.  Type "show warranty" for
>>>> details.
>>>> This GDB was configured as "amd64-marcel-freebsd"...
>>>>
>>>> Unread portion of the kernel message buffer:
>>>>
>>>> Fatal double fault
>>>> rip =3D 0xffffffff8220d9ec
>>>> rsp =3D 0xfffffe066821f000
>>>> rbp =3D 0xfffffe066821f020
>>>> cpuid =3D 6; apic id =3D 14
>>>> panic: double fault
>>>> cpuid =3D 6
>>>> KDB: stack backtrace:
>>>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
>>>> 0xfffffe0649d78e30
>>>> vpanic() at vpanic+0x182/frame 0xfffffe0649d78eb0
>>>> panic() at panic+0x43/frame 0xfffffe0649d78f10
>>>> dblfault_handler() at dblfault_handler+0xa2/frame 0xfffffe0649d78f30=

>>>> Xdblfault() at Xdblfault+0xac/frame 0xfffffe0649d78f30
>>>> --- trap 0x17, rip =3D 0xffffffff8220d9ec, rsp =3D 0xfffffe066821f00=
0,
>>>> rbp =3D
>>>> 0xfffffe066821f020 ---
>>>> avl_rotation() at avl_rotation+0xc/frame 0xfffffe066821f020
>>>> avl_remove() at avl_remove+0x1c8/frame 0xfffffe066821f070
>>>> vdev_queue_io_to_issue() at vdev_queue_io_to_issue+0x87f/frame
>>>> 0xfffffe066821f530
>>>> vdev_queue_io_done() at vdev_queue_io_done+0x83/frame
>>>> 0xfffffe066821f570
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f5a0=

>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f5f0
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe066821f650
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f6a0
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe066821f6e0
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f710=

>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f760
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe066821f7c0
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f810
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe066821f850
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f880=

>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f8d0
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe066821f930
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f980
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe066821f9c0
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f9f0=

>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fa40
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe066821faa0
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821faf0
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe066821fb30
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821fb60=

>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fbb0
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe066821fc10
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fc60
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe066821fca0
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821fcd0=

>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fd20
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe066821fd80
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fdd0
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe066821fe10
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821fe40=

>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fe90
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe066821fef0
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821ff40
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe066821ff80
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821ffb0=

>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220000
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe0668220060
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682200b0
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe06682200f0
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220120=

>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220170
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe06682201d0
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220220
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe0668220260
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220290=

>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682202e0
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe0668220340
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220390
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe06682203d0
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220400=

>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220450
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe06682204b0
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220500
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe0668220540
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220570=

>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682205c0
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe0668220620
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220670
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe06682206b0
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682206e0=

>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220730
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe0668220790
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682207e0
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe0668220820
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220850=

>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682208a0
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe0668220900
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220950
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe0668220990
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682209c0=

>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220a10
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe0668220a70
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220ac0
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe0668220b00
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220b30=

>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220b80
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe0668220be0
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220c30
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe0668220c70
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220ca0=

>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220cf0
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe0668220d50
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220da0
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe0668220de0
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220e10=

>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220e60
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe0668220ec0
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220f10
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe0668220f50
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220f80=

>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220fd0
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe0668221030
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221080
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe06682210c0
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682210f0=

>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221140
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe06682211a0
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682211f0
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe0668221230
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221260=

>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682212b0
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe0668221310
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221360
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe06682213a0
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682213d0=

>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221420
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe0668221480
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682214d0
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe0668221510
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221540=

>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221590
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe06682215f0
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221640
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe0668221680
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682216b0=

>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221700
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe0668221760
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682217b0
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe06682217f0
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221820=

>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221870
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe06682218d0
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221920
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe0668221960
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221990=

>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682219e0
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe0668221a40
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221a90
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe0668221ad0
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221b00=

>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221b50
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe0668221bb0
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221c00
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe0668221c40
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221c70=

>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221cc0
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe0668221d20
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221d70
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe0668221db0
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221de0=

>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221e30
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe0668221e90
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221ee0
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe0668221f20
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221f50=

>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221fa0
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe0668222000
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222050
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe0668222090
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682220c0=

>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222110
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe0668222170
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682221c0
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe0668222200
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222230=

>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222280
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe06682222e0
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222330
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe0668222370
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682223a0=

>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682223f0
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe0668222450
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682224a0
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe06682224e0
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222510=

>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222560
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe06682225c0
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222610
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe0668222650
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222680=

>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682226d0
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe0668222730
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222780
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe06682227c0
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682227f0=

>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222840
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe06682228a0
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682228f0
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe0668222930
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222960=

>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682229b0
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe0668222a10
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222a60
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe0668222aa0
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222ad0=

>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222b20
>>>> taskqueue_run_locked() at taskqueue_run_locked+0x14a/frame
>>>> 0xfffffe0668222b80
>>>> taskqueue_thread_loop() at taskqueue_thread_loop+0xe8/frame
>>>> 0xfffffe0668222bb0
>>>> fork_exit() at fork_exit+0x85/frame 0xfffffe0668222bf0
>>>> fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0668222bf0
>>>> --- trap 0, rip =3D 0, rsp =3D 0, rbp =3D 0 ---
>>>> Uptime: 4m29s
>>>> Dumping 1497 out of 24531
>>>> MB:..2%..11%..21%..31%..41%..51%..61%..71%..81%..91%
>>>>
>>>> Reading symbols from /boot/kernel/zfs.ko...Reading symbols from
>>>> /usr/lib/debug//boot/kernel/zfs.ko.debug...done.
>>>> done.
>>>> Loaded symbols for /boot/kernel/zfs.ko
>>>> Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols
>>>> from
>>>> /usr/lib/debug//boot/kernel/opensolaris.ko.debug...done.
>>>> done.
>>>> Loaded symbols for /boot/kernel/opensolaris.ko
>>>> Reading symbols from /boot/kernel/geom_eli.ko...Reading symbols from=

>>>> /usr/lib/debug//boot/kernel/geom_eli.ko.debug...done.
>>>> done.
>>>> Loaded symbols for /boot/kernel/geom_eli.ko
>>>> Reading symbols from /boot/kernel/geom_mirror.ko...Reading symbols
>>>> from
>>>> /usr/lib/debug//boot/kernel/geom_mirror.ko.debug...done.
>>>> done.
>>>> Loaded symbols for /boot/kernel/geom_mirror.ko
>>>> Reading symbols from /boot/kernel/aesni.ko...Reading symbols from
>>>> /usr/lib/debug//boot/kernel/aesni.ko.debug...done.
>>>> done.
>>>> Loaded symbols for /boot/kernel/aesni.ko
>>>> Reading symbols from /boot/kernel/uhid.ko...Reading symbols from
>>>> /usr/lib/debug//boot/kernel/uhid.ko.debug...done.
>>>> done.
>>>> Loaded symbols for /boot/kernel/uhid.ko
>>>> Reading symbols from /boot/kernel/ums.ko...Reading symbols from
>>>> /usr/lib/debug//boot/kernel/ums.ko.debug...done.
>>>> done.
>>>> Loaded symbols for /boot/kernel/ums.ko
>>>> Reading symbols from /boot/kernel/ipfw.ko...Reading symbols from
>>>> /usr/lib/debug//boot/kernel/ipfw.ko.debug...done.
>>>> done.
>>>> Loaded symbols for /boot/kernel/ipfw.ko
>>>> Reading symbols from /boot/kernel/ipfw_nat.ko...Reading symbols from=

>>>> /usr/lib/debug//boot/kernel/ipfw_nat.ko.debug...done.
>>>> done.
>>>> Loaded symbols for /boot/kernel/ipfw_nat.ko
>>>> Reading symbols from /boot/kernel/libalias.ko...Reading symbols from=

>>>> /usr/lib/debug//boot/kernel/libalias.ko.debug...done.
>>>> done.
>>>> Loaded symbols for /boot/kernel/libalias.ko
>>>> Reading symbols from /boot/kernel/daemon_saver.ko...Reading symbols
>>>> from
>>>> /usr/lib/debug//boot/kernel/daemon_saver.ko.debug...done.
>>>> done.
>>>> Loaded symbols for /boot/kernel/daemon_saver.ko
>>>> #0  doadump (textdump=3D1) at pcpu.h:221
>>>> 221     pcpu.h: No such file or directory.
>>>>           in pcpu.h
>>>> (kgdb) #0  doadump (textdump=3D1) at pcpu.h:221
>>>> #1  0xffffffff80ae9b6e in kern_reboot (howto=3D260)
>>>>       at /usr/src/sys/kern/kern_shutdown.c:366
>>>> #2  0xffffffff80aea13b in vpanic (fmt=3D<value optimized out>,
>>>>       ap=3D<value optimized out>) at
>>>> /usr/src/sys/kern/kern_shutdown.c:759
>>>> #3  0xffffffff80ae9f73 in panic (fmt=3D0x0)
>>>>       at /usr/src/sys/kern/kern_shutdown.c:690
>>>> #4  0xffffffff80fd18f2 in dblfault_handler (frame=3D<value optimized=

>>>> out>)
>>>>       at /usr/src/sys/amd64/amd64/trap.c:868
>>>> #5  0xffffffff80fb406c in Xdblfault ()
>>>>       at /usr/src/sys/amd64/amd64/exception.S:292
>>>> #6  0xffffffff8220d9ec in avl_rotation (tree=3D0xfffff8004dc54510,
>>>>       node=3D0xfffff8056e45cdd0, balance=3D2)
>>>>       at /usr/src/sys/cddl/contrib/opensolaris/common/avl/avl.c:304
>>>> #7  0xffffffff8220de58 in avl_remove (tree=3D<value optimized out>,
>>>>       data=3D<value optimized out>)
>>>>       at /usr/src/sys/cddl/contrib/opensolaris/common/avl/avl.c:818
>>>> #8  0xffffffff822a1eff in vdev_queue_io_to_issue (vq=3D<value optimi=
zed
>>>> out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c=
:439
>>>>
>>>> #9  0xffffffff822a2123 in vdev_queue_io_done (zio=3D<value optimized=

>>>> out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c=
:907
>>>>
>>>> #10 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e3f33d8=
)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #11 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #12 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e3f33d=
8)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #13 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #14 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized=

>>>> out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c=
:913
>>>>
>>>> #15 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e578000=
)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #16 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #17 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e57800=
0)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #18 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #19 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized=

>>>> out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c=
:913
>>>>
>>>> #20 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e5ec3d8=
)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #21 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #22 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e5ec3d=
8)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #23 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #24 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized=

>>>> out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c=
:913
>>>>
>>>> #25 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e48a000=
)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #26 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #27 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e48a00=
0)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #28 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #29 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized=

>>>> out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c=
:913
>>>>
>>>> #30 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e57ab88=
)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #31 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #32 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e57ab8=
8)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #33 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #34 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized=

>>>> out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c=
:913
>>>>
>>>> #35 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff800a69a3000=
)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #36 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #37 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff800a69a300=
0)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #38 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #39 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized=

>>>> out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c=
:913
>>>>
>>>> #40 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e5c8000=
)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #41 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #42 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e5c800=
0)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #43 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #44 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized=

>>>> out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c=
:913
>>>>
>>>> #45 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff802a23f9000=
)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #46 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #47 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff802a23f900=
0)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #48 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #49 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized=

>>>> out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c=
:913
>>>>
>>>> #50 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e5147b0=
)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #51 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #52 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e5147b=
0)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #53 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #54 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized=

>>>> out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c=
:913
>>>>
>>>> #55 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e977000=
)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #56 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #57 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e97700=
0)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #58 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #59 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized=

>>>> out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c=
:913
>>>>
>>>> #60 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e90b3d8=
)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #61 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #62 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e90b3d=
8)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #63 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #64 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized=

>>>> out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c=
:913
>>>>
>>>> #65 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff800b1160b88=
)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #66 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #67 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff800b1160b8=
8)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #68 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #69 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized=

>>>> out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c=
:913
>>>>
>>>> #70 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e4f53d8=
)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #71 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #72 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e4f53d=
8)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #73 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #74 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized=

>>>> out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c=
:913
>>>>
>>>> #75 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e8ebb88=
)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #76 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #77 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e8ebb8=
8)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #78 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #79 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized=

>>>> out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c=
:913
>>>>
>>>> #80 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e8d9b88=
)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #81 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #82 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e8d9b8=
8)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #83 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #84 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized=

>>>> out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c=
:913
>>>>
>>>> #85 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff802a21a03d8=
)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #86 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #87 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff802a21a03d=
8)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #88 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #89 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized=

>>>> out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c=
:913
>>>>
>>>> #90 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e6903d8=
)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #91 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #92 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e6903d=
8)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #93 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #94 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized=

>>>> out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c=
:913
>>>>
>>>> #95 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff802a22dc7b0=
)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #96 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #97 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff802a22dc7b=
0)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #98 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #99 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized=

>>>> out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c=
:913
>>>>
>>>> #100 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e5fb7b=
0)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #101 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)=

>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #102 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e5fb7=
b0)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #103 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)=

>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #104 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimize=
d
>>>> out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c=
:913
>>>>
>>>> #105 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e5ae3d=
8)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #106 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)=

>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #107 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e5ae3=
d8)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #108 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)=

>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #109 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimize=
d
>>>> out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c=
:913
>>>>
>>>> #110 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff802a2131b8=
8)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #111 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)=

>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #112 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff802a2131b=
88)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #113 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)=

>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #114 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimize=
d
>>>> out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c=
:913
>>>>
>>>> #115 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e80b7b=
0)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #116 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)=

>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #117 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e80b7=
b0)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #118 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)=

>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #119 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimize=
d
>>>> out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c=
:913
>>>>
>>>> #120 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff800b14b900=
0)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #121 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)=

>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #122 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff800b14b90=
00)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #123 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)=

>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #124 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimize=
d
>>>> out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c=
:913
>>>>
>>>> #125 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff802a21fa3d=
8)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #126 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)=

>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #127 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff802a21fa3=
d8)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #128 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)=

>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #129 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimize=
d
>>>> out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c=
:913
>>>>
>>>> #130 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e5333d=
8)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #131 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)=

>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #132 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e5333=
d8)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #133 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)=

>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #134 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimize=
d
>>>> out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c=
:913
>>>>
>>>> #135 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e90ab8=
8)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #136 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)=

>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #137 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e90ab=
88)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #138 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)=

>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #139 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimize=
d
>>>> out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c=
:913
>>>>
>>>> #140 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e4047b=
0)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #141 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)=

>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #142 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e4047=
b0)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #143 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)=

>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #144 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimize=
d
>>>> out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c=
:913
>>>>
>>>> #145 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e7083d=
8)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #146 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)=

>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #147 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e7083=
d8)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #148 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)=

>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #149 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimize=
d
>>>> out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c=
:913
>>>>
>>>> #150 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff802a23917b=
0)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #151 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)=

>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #152 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff802a23917=
b0)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #153 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)=

>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #154 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimize=
d
>>>> out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c=
:913
>>>>
>>>> #155 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e63600=
0)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #156 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)=

>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #157 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e6360=
00)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #158 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)=

>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #159 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimize=
d
>>>> out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c=
:913
>>>>
>>>> #160 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e5afb8=
8)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #161 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)=

>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #162 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e5afb=
88)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #163 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)=

>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #164 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimize=
d
>>>> out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c=
:913
>>>>
>>>> #165 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff800a69523d=
8)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #166 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)=

>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #167 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff800a69523=
d8)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #168 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)=

>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #169 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimize=
d
>>>> out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c=
:913
>>>>
>>>> #170 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e47c00=
0)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #171 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)=

>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #172 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e47c0=
00)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #173 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)=

>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #174 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimize=
d
>>>> out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c=
:913
>>>>
>>>> #175 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff800b17b23d=
8)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #176 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)=

>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #177 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff800b17b23=
d8)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #178 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)=

>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #179 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimize=
d
>>>> out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c=
:913
>>>>
>>>> #180 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff800a6d367b=
0)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #181 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)=

>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #182 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff800a6d367=
b0)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #183 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)=

>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #184 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimize=
d
>>>> out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c=
:913
>>>>
>>>> #185 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e99fb8=
8)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #186 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)=

>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #187 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e99fb=
88)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #188 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)=

>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #189 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimize=
d
>>>> out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c=
:913
>>>>
>>>> #190 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e5227b=
0)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #191 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)=

>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #192 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e5227=
b0)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #193 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)=

>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #194 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimize=
d
>>>> out>)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c=
:913
>>>>
>>>> #195 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff800a6d3c00=
0)
>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #196 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>)=

>>>>       at
>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #197 0xffffffff80b4895a in taskqueue_run_locked (queue=3D<value
>>>> optimized
>>>> out>)
>>>>       at /usr/src/sys/kern/subr_taskqueue.c:454
>>>> #198 0xffffffff80b49b58 in taskqueue_thread_loop (arg=3D<value
>>>> optimized out>)
>>>>       at /usr/src/sys/kern/subr_taskqueue.c:724
>>>> #199 0xffffffff80a9f255 in fork_exit (
>>>>       callout=3D0xffffffff80b49a70 <taskqueue_thread_loop>,
>>>>       arg=3D0xfffff8057caa72a0, frame=3D0xfffffe0668222c00)
>>>>       at /usr/src/sys/kern/kern_fork.c:1040
>>>> #200 0xffffffff80fb44ae in fork_trampoline ()
>>>>       at /usr/src/sys/amd64/amd64/exception.S:611
>>>> #201 0x0000000000000000 in ?? ()
>>>> Current language:  auto; currently minimal
>>>> (kgdb)
>>>>
>>>>
>>>>>> --=20
>>>>>> Karl Denninger
>>>>>> karl@denninger.net <mailto:karl@denninger.net>
>>>>>> /The Market Ticker/
>>>>>> /[S/MIME encrypted email preferred]/
>>> _______________________________________________
>>> freebsd-stable@freebsd.org mailing list
>>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
>>> To unsubscribe, send any mail to
>>> "freebsd-stable-unsubscribe@freebsd.org"
>
> _______________________________________________
> freebsd-stable@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.or=
g"

--=20
Karl Denninger
karl@denninger.net <mailto:karl@denninger.net>
/The Market Ticker/
/[S/MIME encrypted email preferred]/

--------------ms020209010902040701010306
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature

MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC
Bl8wggZbMIIEQ6ADAgECAgEpMA0GCSqGSIb3DQEBCwUAMIGQMQswCQYDVQQGEwJVUzEQMA4G
A1UECBMHRmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3Rl
bXMgTExDMRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhND
dWRhIFN5c3RlbXMgTExDIENBMB4XDTE1MDQyMTAyMjE1OVoXDTIwMDQxOTAyMjE1OVowWjEL
MAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM
TEMxHjAcBgNVBAMTFUthcmwgRGVubmluZ2VyIChPQ1NQKTCCAiIwDQYJKoZIhvcNAQEBBQAD
ggIPADCCAgoCggIBALmEWPhAdphrWd4K5VTvE5pxL3blRQPyGF3ApjUjgtavqU1Y8pbI3Byg
XDj2/Uz9Si8XVj/kNbKEjkRh5SsNvx3Fc0oQ1uVjyCq7zC/kctF7yLzQbvWnU4grAPZ3IuAp
3/fFxIVaXpxEdKmyZAVDhk9az+IgHH43rdJRIMzxJ5vqQMb+n2EjadVqiGPbtG9aZEImlq7f
IYDTnKyToi23PAnkPwwT+q1IkI2DTvf2jzWrhLR5DTX0fUYC0nxlHWbjgpiapyJWtR7K2YQO
aevQb/3vN9gSojT2h+cBem7QIj6U69rEYcEDvPyCMXEV9VcXdcmW42LSRsPvZcBHFkWAJqMZ
Myiz4kumaP+s+cIDaXitR/szoqDKGSHM4CPAZV9Yh8asvxQL5uDxz5wvLPgS5yS8K/o7zDR5
vNkMCyfYQuR6PAJxVOk5Arqvj9lfP3JSVapwbr01CoWDBkpuJlKfpQIEeC/pcCBKknllbMYq
yHBO2TipLyO5Ocd1nhN/nOsO+C+j31lQHfOMRZaPQykXVPWG5BbhWT7ttX4vy5hOW6yJgeT/
o3apynlp1cEavkQRS8uJHoQszF6KIrQMID/JfySWvVQ4ksnfzwB2lRomrdrwnQ4eG/HBS+0l
eozwOJNDIBlAP+hLe8A5oWZgooIIK/SulUAsfI6Sgd8dTZTTYmlhAgMBAAGjgfQwgfEwNwYI
KwYBBQUHAQEEKzApMCcGCCsGAQUFBzABhhtodHRwOi8vY3VkYXN5c3RlbXMubmV0Ojg4ODgw
CQYDVR0TBAIwADARBglghkgBhvhCAQEEBAMCBaAwCwYDVR0PBAQDAgXgMCwGCWCGSAGG+EIB
DQQfFh1PcGVuU1NMIEdlbmVyYXRlZCBDZXJ0aWZpY2F0ZTAdBgNVHQ4EFgQUxRyULenJaFwX
RtT79aNmIB/u5VkwHwYDVR0jBBgwFoAUJHGbnYV9/N3dvbDKkpQDofrTbTUwHQYDVR0RBBYw
FIESa2FybEBkZW5uaW5nZXIubmV0MA0GCSqGSIb3DQEBCwUAA4ICAQBPf3cYtmKowmGIYsm6
eBinJu7QVWvxi1vqnBz3KE+HapqoIZS8/PolB/hwiY0UAE1RsjBJ7yEjihVRwummSBvkoOyf
G30uPn4yg4vbJkR9lTz8d21fPshWETa6DBh2jx2Qf13LZpr3Pj2fTtlu6xMYKzg7cSDgd2bO
sJGH/rcvva9Spkx5Vfq0RyOrYph9boshRN3D4tbWgBAcX9POdXCVfJONDxhfBuPHsJ6vEmPb
An+XL5Yl26XYFPiODQ+Qbk44Ot1kt9s7oS3dVUrh92Qv0G3J3DF+Vt6C15nED+f+bk4gScu+
JHT7RjEmfa18GT8DcT//D1zEke1Ymhb41JH+GyZchDRWtjxsS5OBFMzrju7d264zJUFtX7iJ
3xvpKN7VcZKNtB6dLShj3v/XDsQVQWXmR/1YKWZ93C3LpRs2Y5nYdn6gEOpL/WfQFThtfnat
HNc7fNs5vjotaYpBl5H8+VCautKbGOs219uQbhGZLYTv6okuKcY8W+4EJEtK0xB08vqr9Jd0
FS9MGjQE++GWo+5eQxFt6nUENHbVYnsr6bYPQsZH0CRNycgTG9MwY/UIXOf4W034UpR82TBG
1LiMsYfb8ahQJhs3wdf1nzipIjRwoZKT1vGXh/cj3gwSr64GfenURBxaFZA5O1acOZUjPrRT
n3ci4McYW/0WVVA3lDGCBRMwggUPAgEBMIGWMIGQMQswCQYDVQQGEwJVUzEQMA4GA1UECBMH
RmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3RlbXMgTExD
MRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhNDdWRhIFN5
c3RlbXMgTExDIENBAgEpMA0GCWCGSAFlAwQCAwUAoIICTTAYBgkqhkiG9w0BCQMxCwYJKoZI
hvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNjEwMTcyMDIyNDRaME8GCSqGSIb3DQEJBDFCBEBD
SxPkkVliNyJn5MJyl2y0Dd6m5ZhzAGpkqyjAImE47rnEdCi0PrUs4RmErcgrsRegUIapZoy0
PspD3M7ml3alMGwGCSqGSIb3DQEJDzFfMF0wCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBAjAK
BggqhkiG9w0DBzAOBggqhkiG9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYI
KoZIhvcNAwICASgwgacGCSsGAQQBgjcQBDGBmTCBljCBkDELMAkGA1UEBhMCVVMxEDAOBgNV
BAgTB0Zsb3JpZGExEjAQBgNVBAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1z
IExMQzEcMBoGA1UEAxMTQ3VkYSBTeXN0ZW1zIExMQyBDQTEiMCAGCSqGSIb3DQEJARYTQ3Vk
YSBTeXN0ZW1zIExMQyBDQQIBKTCBqQYLKoZIhvcNAQkQAgsxgZmggZYwgZAxCzAJBgNVBAYT
AlVTMRAwDgYDVQQIEwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1
ZGEgU3lzdGVtcyBMTEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExIjAgBgkqhkiG
9w0BCQEWE0N1ZGEgU3lzdGVtcyBMTEMgQ0ECASkwDQYJKoZIhvcNAQEBBQAEggIAiXOqnbPF
eFhqqO1oWvvvGhgRNUTxfwz33KiFTwW974YH40XzSAoMmgUVyq0GADp87h18R2snGkkWfa0a
24K7tVHAeGhoQvvMFLsaVeaSj9+ajSvP//4/Tl1u+8qPLynjyCmVkub3ORZCjFbgrE70vklm
57M1ylMjvFZQkOPU2U3UlQytiCrJbs8XVVdJfser0ckNkk81ZMq8ubZ/Ojx+8+dFK4Vn8p1E
uXaKewjpylySMBqXF7U3Aua8mnMdOTYUSlqdAvM6ooL9NPcwdIfaCckAgIhY0PJvprIAJcmw
7q4oqCqyOXhZfGiS1fKpyMjD2x9C4crk5mTo4RtHnGw9tA5lRo3e/cYLwQPvp7ii1NlmGRHk
CwWt4h9l8XAIX4G16CwHRFJtQ46ECjU/JGk3NBUQlPdLeGYeZUTKnOjc2X2nJfsNszlZPbHq
rGF35En9MfhgmGb2W6DIei9fXcxgib+Xgt9J1YgPFQqMMHgesl5DW7hGpFx8MJuqaDyZBDv5
ZqzDN38DE4SOlSXYtn2eJYdpDt5mMM9cNNNUGRl3Lbc3J9DaIWucAAZCvI5uA4N1tKuYbaOe
wOIu49GLec4SS+8KIsxKgrOp8ZOxboqRcq6odTMCO/7R1jAQP7tOFI+qXHG4Pm0CTPUNW7Rv
fEpdbICwKGvtGRuCnX9WZnBvu30AAAAAAAA=
--------------ms020209010902040701010306--



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?26e092b2-17c6-8744-5035-d0853d733870>