Date: Mon, 17 Oct 2016 11:02:10 -0500 From: Karl Denninger <karl@denninger.net> To: freebsd-stable@freebsd.org Subject: Re: Repeatable panic on ZFS filesystem (used for backups); 11.0-STABLE Message-ID: <9adecd24-6659-0da5-5c05-d0d3957a2cb3@denninger.net> In-Reply-To: <3d4f25c9-a262-a373-ec7e-755325f8810b@denninger.net> References: <3d4f25c9-a262-a373-ec7e-755325f8810b@denninger.net>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a cryptographically signed message in MIME format. --------------ms090805080502070508080105 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Update from my test system: Setting vfs.zfs.vdev_trim_max_active to 10 (from default 64) does *not* stop the panics. Setting vfs.zfs.vdev.trim.enabled =3D 0 (which requires a reboot) DOES stop the panics. I am going to run a scrub on the pack, but I suspect the pack itself (now that I can actually mount it without the machine blowing up!) is fin= e. THIS (OBVIOUSLY) NEEDS ATTENTION! On 10/17/2016 09:17, Karl Denninger wrote: > This is a situation I've had happen before, and reported -- it appeared= > to be a kernel stack overflow, and it has gotten materially worse on > 11.0-STABLE. > > The issue occurs after some period of time (normally a week or so.) Th= e > system has a mirrored pair of large drives used for backup purposes to > which ZFS snapshots are written using a script that iterates over the > system. > > The panic /only /happens when the root filesystem is being sent, and it= > appears that the panic itself is being triggered by an I/O pattern on > the /backup /drive -- not the source drives. Zpool scrubs on the sourc= e > are clean; I am going to run one now on the backup, but in the past tha= t > has been clean as well. > > I now have a *repeatable* panic in that if I attempt a "zfs list -rt al= l > backup" on the backup volume I get the below panic. A "zfs list" > does*__*not panic the system. > > The operating theory previously (after digging through the passed > structures in the dump) was that the ZFS system was attempting to issue= > TRIMs on a device that can't do them before the ZFS system realizes thi= s > and stops asking (the backup volume is comprised of spinning rust) but > the appearance of the panic now on the volume when I simply do a "zfs > list -rt all backup" appears to negate that theory since no writes are > performed by that operation, and thus no TRIM calls should be issued. > > I can leave the backup volume in the state that causes this for a short= > period of time in an attempt to find and fix this. > > > NewFS.denninger.net dumped core - see /var/crash/vmcore.1 > > Mon Oct 17 09:02:33 CDT 2016 > > FreeBSD NewFS.denninger.net 11.0-STABLE FreeBSD 11.0-STABLE #13 > r307318M: Fri Oct 14 09:23:46 CDT 2016 =20 > karl@NewFS.denninger.net:/usr/obj/usr/src/sys/KSD-SMP amd64 > > panic: double fault > > GNU gdb 6.1.1 [FreeBSD] > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and yo= u are > welcome to change it and/or distribute copies of it under certain > conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for deta= ils. > This GDB was configured as "amd64-marcel-freebsd"... > > Unread portion of the kernel message buffer: > > Fatal double fault > rip =3D 0xffffffff8220d9ec > rsp =3D 0xfffffe066821f000 > rbp =3D 0xfffffe066821f020 > cpuid =3D 6; apic id =3D 14 > panic: double fault > cpuid =3D 6 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame > 0xfffffe0649d78e30 > vpanic() at vpanic+0x182/frame 0xfffffe0649d78eb0 > panic() at panic+0x43/frame 0xfffffe0649d78f10 > dblfault_handler() at dblfault_handler+0xa2/frame 0xfffffe0649d78f30 > Xdblfault() at Xdblfault+0xac/frame 0xfffffe0649d78f30 > --- trap 0x17, rip =3D 0xffffffff8220d9ec, rsp =3D 0xfffffe066821f000, = rbp =3D > 0xfffffe066821f020 --- > avl_rotation() at avl_rotation+0xc/frame 0xfffffe066821f020 > avl_remove() at avl_remove+0x1c8/frame 0xfffffe066821f070 > vdev_queue_io_to_issue() at vdev_queue_io_to_issue+0x87f/frame > 0xfffffe066821f530 > vdev_queue_io_done() at vdev_queue_io_done+0x83/frame 0xfffffe066821f57= 0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f5a0 > zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f5f0 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821f650= > zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f6a0 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821f6e= 0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f710 > zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f760 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821f7c0= > zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f810 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821f85= 0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f880 > zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f8d0 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821f930= > zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f980 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821f9c= 0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f9f0 > zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fa40 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821faa0= > zio_execute() at zio_execute+0x23d/frame 0xfffffe066821faf0 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821fb3= 0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821fb60 > zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fbb0 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821fc10= > zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fc60 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821fca= 0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821fcd0 > zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fd20 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821fd80= > zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fdd0 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821fe1= 0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821fe40 > zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fe90 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821fef0= > zio_execute() at zio_execute+0x23d/frame 0xfffffe066821ff40 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821ff8= 0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821ffb0 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220000 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220060= > zio_execute() at zio_execute+0x23d/frame 0xfffffe06682200b0 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682200f= 0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220120 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220170 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682201d0= > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220220 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066822026= 0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220290 > zio_execute() at zio_execute+0x23d/frame 0xfffffe06682202e0 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220340= > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220390 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682203d= 0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220400 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220450 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682204b0= > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220500 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066822054= 0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220570 > zio_execute() at zio_execute+0x23d/frame 0xfffffe06682205c0 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220620= > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220670 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682206b= 0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682206e0 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220730 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220790= > zio_execute() at zio_execute+0x23d/frame 0xfffffe06682207e0 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066822082= 0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220850 > zio_execute() at zio_execute+0x23d/frame 0xfffffe06682208a0 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220900= > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220950 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066822099= 0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682209c0 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220a10 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220a70= > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220ac0 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220b0= 0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220b30 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220b80 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220be0= > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220c30 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220c7= 0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220ca0 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220cf0 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220d50= > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220da0 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220de= 0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220e10 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220e60 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220ec0= > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220f10 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220f5= 0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220f80 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220fd0 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221030= > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221080 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682210c= 0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682210f0 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221140 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682211a0= > zio_execute() at zio_execute+0x23d/frame 0xfffffe06682211f0 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066822123= 0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221260 > zio_execute() at zio_execute+0x23d/frame 0xfffffe06682212b0 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221310= > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221360 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682213a= 0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682213d0 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221420 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221480= > zio_execute() at zio_execute+0x23d/frame 0xfffffe06682214d0 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066822151= 0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221540 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221590 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682215f0= > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221640 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066822168= 0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682216b0 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221700 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221760= > zio_execute() at zio_execute+0x23d/frame 0xfffffe06682217b0 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682217f= 0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221820 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221870 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682218d0= > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221920 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066822196= 0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221990 > zio_execute() at zio_execute+0x23d/frame 0xfffffe06682219e0 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221a40= > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221a90 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221ad= 0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221b00 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221b50 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221bb0= > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221c00 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221c4= 0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221c70 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221cc0 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221d20= > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221d70 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221db= 0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221de0 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221e30 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221e90= > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221ee0 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221f2= 0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221f50 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221fa0 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668222000= > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222050 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066822209= 0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682220c0 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222110 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668222170= > zio_execute() at zio_execute+0x23d/frame 0xfffffe06682221c0 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066822220= 0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222230 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222280 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682222e0= > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222330 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066822237= 0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682223a0 > zio_execute() at zio_execute+0x23d/frame 0xfffffe06682223f0 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668222450= > zio_execute() at zio_execute+0x23d/frame 0xfffffe06682224a0 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682224e= 0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222510 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222560 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682225c0= > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222610 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066822265= 0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222680 > zio_execute() at zio_execute+0x23d/frame 0xfffffe06682226d0 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668222730= > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222780 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682227c= 0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682227f0 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222840 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682228a0= > zio_execute() at zio_execute+0x23d/frame 0xfffffe06682228f0 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066822293= 0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222960 > zio_execute() at zio_execute+0x23d/frame 0xfffffe06682229b0 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668222a10= > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222a60 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668222aa= 0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222ad0 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222b20 > taskqueue_run_locked() at taskqueue_run_locked+0x14a/frame > 0xfffffe0668222b80 > taskqueue_thread_loop() at taskqueue_thread_loop+0xe8/frame > 0xfffffe0668222bb0 > fork_exit() at fork_exit+0x85/frame 0xfffffe0668222bf0 > fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0668222bf0 > --- trap 0, rip =3D 0, rsp =3D 0, rbp =3D 0 --- > Uptime: 4m29s > Dumping 1497 out of 24531 > MB:..2%..11%..21%..31%..41%..51%..61%..71%..81%..91% > > Reading symbols from /boot/kernel/zfs.ko...Reading symbols from > /usr/lib/debug//boot/kernel/zfs.ko.debug...done. > done. > Loaded symbols for /boot/kernel/zfs.ko > Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from= > /usr/lib/debug//boot/kernel/opensolaris.ko.debug...done. > done. > Loaded symbols for /boot/kernel/opensolaris.ko > Reading symbols from /boot/kernel/geom_eli.ko...Reading symbols from > /usr/lib/debug//boot/kernel/geom_eli.ko.debug...done. > done. > Loaded symbols for /boot/kernel/geom_eli.ko > Reading symbols from /boot/kernel/geom_mirror.ko...Reading symbols from= > /usr/lib/debug//boot/kernel/geom_mirror.ko.debug...done. > done. > Loaded symbols for /boot/kernel/geom_mirror.ko > Reading symbols from /boot/kernel/aesni.ko...Reading symbols from > /usr/lib/debug//boot/kernel/aesni.ko.debug...done. > done. > Loaded symbols for /boot/kernel/aesni.ko > Reading symbols from /boot/kernel/uhid.ko...Reading symbols from > /usr/lib/debug//boot/kernel/uhid.ko.debug...done. > done. > Loaded symbols for /boot/kernel/uhid.ko > Reading symbols from /boot/kernel/ums.ko...Reading symbols from > /usr/lib/debug//boot/kernel/ums.ko.debug...done. > done. > Loaded symbols for /boot/kernel/ums.ko > Reading symbols from /boot/kernel/ipfw.ko...Reading symbols from > /usr/lib/debug//boot/kernel/ipfw.ko.debug...done. > done. > Loaded symbols for /boot/kernel/ipfw.ko > Reading symbols from /boot/kernel/ipfw_nat.ko...Reading symbols from > /usr/lib/debug//boot/kernel/ipfw_nat.ko.debug...done. > done. > Loaded symbols for /boot/kernel/ipfw_nat.ko > Reading symbols from /boot/kernel/libalias.ko...Reading symbols from > /usr/lib/debug//boot/kernel/libalias.ko.debug...done. > done. > Loaded symbols for /boot/kernel/libalias.ko > Reading symbols from /boot/kernel/daemon_saver.ko...Reading symbols fro= m > /usr/lib/debug//boot/kernel/daemon_saver.ko.debug...done. > done. > Loaded symbols for /boot/kernel/daemon_saver.ko > #0 doadump (textdump=3D1) at pcpu.h:221 > 221 pcpu.h: No such file or directory. > in pcpu.h > (kgdb) #0 doadump (textdump=3D1) at pcpu.h:221 > #1 0xffffffff80ae9b6e in kern_reboot (howto=3D260) > at /usr/src/sys/kern/kern_shutdown.c:366 > #2 0xffffffff80aea13b in vpanic (fmt=3D<value optimized out>, > ap=3D<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:75= 9 > #3 0xffffffff80ae9f73 in panic (fmt=3D0x0) > at /usr/src/sys/kern/kern_shutdown.c:690 > #4 0xffffffff80fd18f2 in dblfault_handler (frame=3D<value optimized ou= t>) > at /usr/src/sys/amd64/amd64/trap.c:868 > #5 0xffffffff80fb406c in Xdblfault () > at /usr/src/sys/amd64/amd64/exception.S:292 > #6 0xffffffff8220d9ec in avl_rotation (tree=3D0xfffff8004dc54510, > node=3D0xfffff8056e45cdd0, balance=3D2) > at /usr/src/sys/cddl/contrib/opensolaris/common/avl/avl.c:304 > #7 0xffffffff8220de58 in avl_remove (tree=3D<value optimized out>, > data=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/common/avl/avl.c:818 > #8 0xffffffff822a1eff in vdev_queue_io_to_issue (vq=3D<value optimized= out>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:43= 9 > #9 0xffffffff822a2123 in vdev_queue_io_done (zio=3D<value optimized ou= t>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:90= 7 > #10 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e3f33d8) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 37 > #11 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #12 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e3f33d8) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 12 > #13 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #14 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized ou= t>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:91= 3 > #15 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e578000) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 37 > #16 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #17 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e578000) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 12 > #18 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #19 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized ou= t>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:91= 3 > #20 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e5ec3d8) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 37 > #21 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #22 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e5ec3d8) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 12 > #23 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #24 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized ou= t>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:91= 3 > #25 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e48a000) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 37 > #26 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #27 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e48a000) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 12 > #28 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #29 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized ou= t>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:91= 3 > #30 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e57ab88) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 37 > #31 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #32 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e57ab88) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 12 > #33 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #34 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized ou= t>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:91= 3 > #35 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff800a69a3000) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 37 > #36 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #37 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff800a69a3000) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 12 > #38 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #39 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized ou= t>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:91= 3 > #40 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e5c8000) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 37 > #41 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #42 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e5c8000) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 12 > #43 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #44 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized ou= t>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:91= 3 > #45 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff802a23f9000) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 37 > #46 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #47 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff802a23f9000) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 12 > #48 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #49 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized ou= t>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:91= 3 > #50 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e5147b0) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 37 > #51 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #52 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e5147b0) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 12 > #53 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #54 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized ou= t>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:91= 3 > #55 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e977000) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 37 > #56 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #57 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e977000) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 12 > #58 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #59 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized ou= t>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:91= 3 > #60 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e90b3d8) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 37 > #61 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #62 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e90b3d8) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 12 > #63 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #64 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized ou= t>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:91= 3 > #65 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff800b1160b88) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 37 > #66 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #67 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff800b1160b88) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 12 > #68 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #69 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized ou= t>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:91= 3 > #70 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e4f53d8) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 37 > #71 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #72 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e4f53d8) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 12 > #73 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #74 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized ou= t>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:91= 3 > #75 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e8ebb88) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 37 > #76 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #77 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e8ebb88) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 12 > #78 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #79 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized ou= t>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:91= 3 > #80 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e8d9b88) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 37 > #81 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #82 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e8d9b88) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 12 > #83 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #84 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized ou= t>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:91= 3 > #85 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff802a21a03d8) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 37 > #86 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #87 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff802a21a03d8) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 12 > #88 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #89 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized ou= t>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:91= 3 > #90 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e6903d8) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 37 > #91 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #92 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e6903d8) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 12 > #93 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #94 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized ou= t>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:91= 3 > #95 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff802a22dc7b0) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 37 > #96 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #97 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff802a22dc7b0) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 12 > #98 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #99 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized ou= t>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:91= 3 > #100 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e5fb7b0) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 37 > #101 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #102 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e5fb7b0)= > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 12 > #103 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #104 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized o= ut>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:91= 3 > #105 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e5ae3d8) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 37 > #106 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #107 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e5ae3d8)= > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 12 > #108 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #109 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized o= ut>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:91= 3 > #110 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff802a2131b88) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 37 > #111 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #112 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff802a2131b88)= > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 12 > #113 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #114 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized o= ut>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:91= 3 > #115 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e80b7b0) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 37 > #116 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #117 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e80b7b0)= > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 12 > #118 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #119 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized o= ut>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:91= 3 > #120 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff800b14b9000) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 37 > #121 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #122 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff800b14b9000)= > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 12 > #123 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #124 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized o= ut>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:91= 3 > #125 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff802a21fa3d8) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 37 > #126 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #127 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff802a21fa3d8)= > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 12 > #128 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #129 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized o= ut>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:91= 3 > #130 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e5333d8) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 37 > #131 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #132 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e5333d8)= > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 12 > #133 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #134 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized o= ut>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:91= 3 > #135 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e90ab88) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 37 > #136 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #137 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e90ab88)= > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 12 > #138 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #139 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized o= ut>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:91= 3 > #140 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e4047b0) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 37 > #141 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #142 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e4047b0)= > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 12 > #143 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #144 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized o= ut>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:91= 3 > #145 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e7083d8) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 37 > #146 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #147 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e7083d8)= > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 12 > #148 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #149 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized o= ut>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:91= 3 > #150 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff802a23917b0) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 37 > #151 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #152 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff802a23917b0)= > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 12 > #153 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #154 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized o= ut>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:91= 3 > #155 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e636000) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 37 > #156 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #157 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e636000)= > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 12 > #158 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #159 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized o= ut>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:91= 3 > #160 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e5afb88) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 37 > #161 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #162 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e5afb88)= > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 12 > #163 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #164 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized o= ut>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:91= 3 > #165 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff800a69523d8) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 37 > #166 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #167 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff800a69523d8)= > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 12 > #168 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #169 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized o= ut>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:91= 3 > #170 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e47c000) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 37 > #171 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #172 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e47c000)= > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 12 > #173 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #174 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized o= ut>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:91= 3 > #175 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff800b17b23d8) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 37 > #176 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #177 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff800b17b23d8)= > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 12 > #178 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #179 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized o= ut>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:91= 3 > #180 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff800a6d367b0) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 37 > #181 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #182 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff800a6d367b0)= > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 12 > #183 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #184 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized o= ut>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:91= 3 > #185 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e99fb88) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 37 > #186 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #187 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e99fb88)= > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 12 > #188 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #189 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized o= ut>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:91= 3 > #190 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e5227b0) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 37 > #191 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #192 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e5227b0)= > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 12 > #193 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #194 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized o= ut>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:91= 3 > #195 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff800a6d3c000) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:31= 37 > #196 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:16= 51 > #197 0xffffffff80b4895a in taskqueue_run_locked (queue=3D<value optimiz= ed > out>) > at /usr/src/sys/kern/subr_taskqueue.c:454 > #198 0xffffffff80b49b58 in taskqueue_thread_loop (arg=3D<value optimize= d out>) > at /usr/src/sys/kern/subr_taskqueue.c:724 > #199 0xffffffff80a9f255 in fork_exit ( > callout=3D0xffffffff80b49a70 <taskqueue_thread_loop>, > arg=3D0xfffff8057caa72a0, frame=3D0xfffffe0668222c00) > at /usr/src/sys/kern/kern_fork.c:1040 > #200 0xffffffff80fb44ae in fork_trampoline () > at /usr/src/sys/amd64/amd64/exception.S:611 > #201 0x0000000000000000 in ?? () > Current language: auto; currently minimal > (kgdb) > > --=20 Karl Denninger karl@denninger.net <mailto:karl@denninger.net> /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms090805080502070508080105 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC Bl8wggZbMIIEQ6ADAgECAgEpMA0GCSqGSIb3DQEBCwUAMIGQMQswCQYDVQQGEwJVUzEQMA4G A1UECBMHRmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3Rl bXMgTExDMRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhND dWRhIFN5c3RlbXMgTExDIENBMB4XDTE1MDQyMTAyMjE1OVoXDTIwMDQxOTAyMjE1OVowWjEL MAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM TEMxHjAcBgNVBAMTFUthcmwgRGVubmluZ2VyIChPQ1NQKTCCAiIwDQYJKoZIhvcNAQEBBQAD ggIPADCCAgoCggIBALmEWPhAdphrWd4K5VTvE5pxL3blRQPyGF3ApjUjgtavqU1Y8pbI3Byg XDj2/Uz9Si8XVj/kNbKEjkRh5SsNvx3Fc0oQ1uVjyCq7zC/kctF7yLzQbvWnU4grAPZ3IuAp 3/fFxIVaXpxEdKmyZAVDhk9az+IgHH43rdJRIMzxJ5vqQMb+n2EjadVqiGPbtG9aZEImlq7f IYDTnKyToi23PAnkPwwT+q1IkI2DTvf2jzWrhLR5DTX0fUYC0nxlHWbjgpiapyJWtR7K2YQO aevQb/3vN9gSojT2h+cBem7QIj6U69rEYcEDvPyCMXEV9VcXdcmW42LSRsPvZcBHFkWAJqMZ Myiz4kumaP+s+cIDaXitR/szoqDKGSHM4CPAZV9Yh8asvxQL5uDxz5wvLPgS5yS8K/o7zDR5 vNkMCyfYQuR6PAJxVOk5Arqvj9lfP3JSVapwbr01CoWDBkpuJlKfpQIEeC/pcCBKknllbMYq yHBO2TipLyO5Ocd1nhN/nOsO+C+j31lQHfOMRZaPQykXVPWG5BbhWT7ttX4vy5hOW6yJgeT/ o3apynlp1cEavkQRS8uJHoQszF6KIrQMID/JfySWvVQ4ksnfzwB2lRomrdrwnQ4eG/HBS+0l eozwOJNDIBlAP+hLe8A5oWZgooIIK/SulUAsfI6Sgd8dTZTTYmlhAgMBAAGjgfQwgfEwNwYI KwYBBQUHAQEEKzApMCcGCCsGAQUFBzABhhtodHRwOi8vY3VkYXN5c3RlbXMubmV0Ojg4ODgw CQYDVR0TBAIwADARBglghkgBhvhCAQEEBAMCBaAwCwYDVR0PBAQDAgXgMCwGCWCGSAGG+EIB DQQfFh1PcGVuU1NMIEdlbmVyYXRlZCBDZXJ0aWZpY2F0ZTAdBgNVHQ4EFgQUxRyULenJaFwX RtT79aNmIB/u5VkwHwYDVR0jBBgwFoAUJHGbnYV9/N3dvbDKkpQDofrTbTUwHQYDVR0RBBYw FIESa2FybEBkZW5uaW5nZXIubmV0MA0GCSqGSIb3DQEBCwUAA4ICAQBPf3cYtmKowmGIYsm6 eBinJu7QVWvxi1vqnBz3KE+HapqoIZS8/PolB/hwiY0UAE1RsjBJ7yEjihVRwummSBvkoOyf G30uPn4yg4vbJkR9lTz8d21fPshWETa6DBh2jx2Qf13LZpr3Pj2fTtlu6xMYKzg7cSDgd2bO sJGH/rcvva9Spkx5Vfq0RyOrYph9boshRN3D4tbWgBAcX9POdXCVfJONDxhfBuPHsJ6vEmPb An+XL5Yl26XYFPiODQ+Qbk44Ot1kt9s7oS3dVUrh92Qv0G3J3DF+Vt6C15nED+f+bk4gScu+ JHT7RjEmfa18GT8DcT//D1zEke1Ymhb41JH+GyZchDRWtjxsS5OBFMzrju7d264zJUFtX7iJ 3xvpKN7VcZKNtB6dLShj3v/XDsQVQWXmR/1YKWZ93C3LpRs2Y5nYdn6gEOpL/WfQFThtfnat HNc7fNs5vjotaYpBl5H8+VCautKbGOs219uQbhGZLYTv6okuKcY8W+4EJEtK0xB08vqr9Jd0 FS9MGjQE++GWo+5eQxFt6nUENHbVYnsr6bYPQsZH0CRNycgTG9MwY/UIXOf4W034UpR82TBG 1LiMsYfb8ahQJhs3wdf1nzipIjRwoZKT1vGXh/cj3gwSr64GfenURBxaFZA5O1acOZUjPrRT n3ci4McYW/0WVVA3lDGCBRMwggUPAgEBMIGWMIGQMQswCQYDVQQGEwJVUzEQMA4GA1UECBMH RmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3RlbXMgTExD MRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhNDdWRhIFN5 c3RlbXMgTExDIENBAgEpMA0GCWCGSAFlAwQCAwUAoIICTTAYBgkqhkiG9w0BCQMxCwYJKoZI hvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNjEwMTcxNjAyMTBaME8GCSqGSIb3DQEJBDFCBEBu 1dvvxNl8w5wVJn1OQk8dVZRZxlDaPQqkSrFaNICZZ5K3d4EE8sxzrzqWdSAuGFajua9O56wF VW5mRGK0uYa9MGwGCSqGSIb3DQEJDzFfMF0wCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBAjAK BggqhkiG9w0DBzAOBggqhkiG9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYI KoZIhvcNAwICASgwgacGCSsGAQQBgjcQBDGBmTCBljCBkDELMAkGA1UEBhMCVVMxEDAOBgNV BAgTB0Zsb3JpZGExEjAQBgNVBAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1z IExMQzEcMBoGA1UEAxMTQ3VkYSBTeXN0ZW1zIExMQyBDQTEiMCAGCSqGSIb3DQEJARYTQ3Vk YSBTeXN0ZW1zIExMQyBDQQIBKTCBqQYLKoZIhvcNAQkQAgsxgZmggZYwgZAxCzAJBgNVBAYT AlVTMRAwDgYDVQQIEwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1 ZGEgU3lzdGVtcyBMTEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExIjAgBgkqhkiG 9w0BCQEWE0N1ZGEgU3lzdGVtcyBMTEMgQ0ECASkwDQYJKoZIhvcNAQEBBQAEggIAJ/Ga2khE yO1WAOja0vqp/gyZ0faxzQwp4J6JMxp3N8cHwyXd8F906ZsoqAqxJPWh2pS0vapSWmO7Mvtf zQ7eNwRr5H1LboId046HHZaGGjptP7J7LPcP9PM35f/xobk0PzWTOi25+6/DseTNFmlwg8rX M7j2vZNvPZGHAT3HwCIHIht9Zook7tOobDGgmvjwEHaTE0eHFB5wHHaZ0czT/3fKMXV7x3gd m2+rMWteNzd2xIaaXksGZiqSTlHtoK7I7bUMfJpYx3cdOf9+Vop+4CKEWpTwuPBz596Jmi8N 4LsGAtJXznbC65FzaIiZ5bA+P6fUroaG+n8qoY6raE9hHUqj+Blc2yz41m2p3D7O4V7drbbb AEb/KH3cpwpjWXoLMpzqOHpoJSqEp8+n1eYKIzAlJ1vEMfkWp3arwrt/b5hxIP5oVfVzcRON Rm6J8LzpNzybHNvcPgz90VKWpekQOG5Js2GSWdfBbDNmWUjpS7c9jobyK7Uq1Epc2oaJopKd 88DcHzrQ23e4d56Tq1gO8MGutXayayavAUyOdcPfJRgIdjtwhMn+q1WPjmCaum2PctxFs9TO Hf5Y1IyWshCntX5ELxdnZiHomJrHrifjk/Cab+P+megCQYJTtmbfhayv6Keui+HCMTqur7W5 cV0XF7ya7rvz8uUTZMPMI0OL1NwAAAAAAAA= --------------ms090805080502070508080105--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?9adecd24-6659-0da5-5c05-d0d3957a2cb3>