Date: Thu, 18 Jul 2019 15:13:37 -0500 From: Karl Denninger <karl@denninger.net> To: freebsd-stable@freebsd.org Subject: Kernel panic in zfs code; 12-STABLE Message-ID: <61e5debd-b440-16c9-2a70-0912634e52aa@denninger.net>
next in thread | raw e-mail | index | archive | help
This is a cryptographically signed message in MIME format. --------------ms020904040100020602000304 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable FreeBSD 12.0-STABLE #2 r349024M: Thu Jun 13 18:01:16 CDT 2019=C2=A0=C2=A0= =C2=A0=C2=A0 karl@NewFS.denninger.net:/usr/obj/usr/src/amd64.amd64/sys/KSD-SMP Note -- no patches of any sort in the ZFS code; I am NOT running any of my former patch set. NewFS.denninger.net dumped core - see /var/crash/vmcore.8 Thu Jul 18 15:02:54 CDT 2019 FreeBSD NewFS.denninger.net 12.0-STABLE FreeBSD 12.0-STABLE #2 r349024M: Thu Jun 13 18:01:16 CDT 2019=C2=A0=C2=A0=C2=A0=C2=A0 karl@NewFS.denninger.net:/usr/obj/usr/src/amd64.amd64/sys/KSD-SMP=C2=A0 a= md64 panic: double fault GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you = are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB.=C2=A0 Type "show warranty" for d= etails. This GDB was configured as "amd64-marcel-freebsd"... Unread portion of the kernel message buffer: Fatal double fault rip 0xffffffff8271eeec rsp 0xfffffe009d4e7f60 rbp 0xfffffe009d4e8450 rax 0xfffff801b5b68000 rdx 0xababe19 rbx 0xfffff801ac399000 rcx 0x6f598 rsi 0xfffff801b5b68740 rdi 0xfffff801ac2a2668 r8 0xfffff801ac2a2668 r9 0 r10 0xfffff801ac7cf250 r11 0 r12 0xfffff801b5b685b8 r13 0xfffff801b5b68000 r14 0xfffffe0082dfb000 r15 0xfffff801b5b685b8 rflags 0x10286 cs 0x20 ss 0x28 ds 0x3b es 0x3b fs 0x13 gs 0x1b fsbase 0x8002328d0 gsbase 0xffffffff8202a100 kgsbase 0 cpuid =3D 11; apic id =3D 35 panic: double fault cpuid =3D 11 time =3D 1563479881 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe000338edb0 vpanic() at vpanic+0x19d/frame 0xfffffe000338ee00 panic() at panic+0x43/frame 0xfffffe000338ee60 dblfault_handler() at dblfault_handler+0x1de/frame 0xfffffe000338ef30 Xdblfault() at Xdblfault+0xc3/frame 0xfffffe000338ef30 --- trap 0x17, rip =3D 0xffffffff8271eeec, rsp =3D 0xfffffe009d4e7f60, rb= p =3D 0xfffffe009d4e8450 --- vdev_queue_io_to_issue() at vdev_queue_io_to_issue+0x2c/frame 0xfffffe009d4e8450 vdev_queue_io_done() at vdev_queue_io_done+0xc8/frame 0xfffffe009d4e84a0 zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e84e0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e8530 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4e8590 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e85e0 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4e8630= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e8670 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e86c0 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4e8720 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e8770 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4e87c0= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e8800 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e8850 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4e88b0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e8900 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4e8950= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e8990 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e89e0 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4e8a40 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e8a90 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4e8ae0= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e8b20 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e8b70 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4e8bd0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e8c20 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4e8c70= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e8cb0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e8d00 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4e8d60 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e8db0 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4e8e00= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e8e40 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e8e90 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4e8ef0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e8f40 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4e8f90= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e8fd0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e9020 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4e9080 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e90d0 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4e9120= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e9160 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e91b0 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4e9210 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e9260 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4e92b0= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e92f0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e9340 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4e93a0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e93f0 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4e9440= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e9480 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e94d0 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4e9530 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e9580 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4e95d0= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e9610 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e9660 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4e96c0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e9710 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4e9760= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e97a0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e97f0 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4e9850 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e98a0 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4e98f0= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e9930 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e9980 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4e99e0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e9a30 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4e9a80= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e9ac0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e9b10 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4e9b70 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e9bc0 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4e9c10= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e9c50 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e9ca0 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4e9d00 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e9d50 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4e9da0= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e9de0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e9e30 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4e9e90 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e9ee0 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4e9f30= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e9f70 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e9fc0 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ea020 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ea070 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ea0c0= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ea100 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ea150 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ea1b0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ea200 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ea250= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ea290 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ea2e0 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ea340 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ea390 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ea3e0= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ea420 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ea470 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ea4d0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ea520 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ea570= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ea5b0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ea600 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ea660 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ea6b0 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ea700= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ea740 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ea790 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ea7f0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ea840 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ea890= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ea8d0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ea920 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ea980 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ea9d0 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4eaa20= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4eaa60 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eaab0 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4eab10 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eab60 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4eabb0= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4eabf0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eac40 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4eaca0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eacf0 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ead40= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ead80 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eadd0 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4eae30 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eae80 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4eaed0= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4eaf10 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eaf60 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4eafc0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eb010 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4eb060= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4eb0a0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eb0f0 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4eb150 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eb1a0 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4eb1f0= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4eb230 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eb280 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4eb2e0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eb330 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4eb380= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4eb3c0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eb410 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4eb470 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eb4c0 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4eb510= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4eb550 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eb5a0 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4eb600 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eb650 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4eb6a0= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4eb6e0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eb730 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4eb790 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eb7e0 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4eb830= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4eb870 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eb8c0 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4eb920 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eb970 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4eb9c0= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4eba00 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eba50 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ebab0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ebb00 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ebb50= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ebb90 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ebbe0 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ebc40 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ebc90 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ebce0= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ebd20 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ebd70 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ebdd0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ebe20 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ebe70= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ebeb0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ebf00 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ebf60 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ebfb0 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ec000= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ec040 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ec090 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ec0f0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ec140 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ec190= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ec1d0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ec220 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ec280 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ec2d0 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ec320= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ec360 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ec3b0 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ec410 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ec460 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ec4b0= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ec4f0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ec540 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ec5a0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ec5f0 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ec640= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ec680 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ec6d0 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ec730 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ec780 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ec7d0= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ec810 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ec860 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ec8c0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ec910 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ec960= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ec9a0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ec9f0 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4eca50 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ecaa0 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ecaf0= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ecb30 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ecb80 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ecbe0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ecc30 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ecc80= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4eccc0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ecd10 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ecd70 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ecdc0 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ece10= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ece50 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ecea0 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ecf00 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ecf50 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ecfa0= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ecfe0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ed030 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ed090 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ed0e0 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ed130= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ed170 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ed1c0 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ed220 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ed270 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ed2c0= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ed300 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ed350 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ed3b0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ed400 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ed450= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ed490 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ed4e0 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ed540 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ed590 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ed5e0= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ed620 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ed670 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ed6d0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ed720 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ed770= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ed7b0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ed800 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ed860 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ed8b0 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ed900= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ed940 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ed990 zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ed9f0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eda40 vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4eda90= zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4edad0 zio_execute() at zio_execute+0xac/frame 0xfffffe009d4edb20 taskqueue_run_locked() at taskqueue_run_locked+0x154/frame 0xfffffe009d4edb80 taskqueue_thread_loop() at taskqueue_thread_loop+0x98/frame 0xfffffe009d4edbb0 fork_exit() at fork_exit+0x83/frame 0xfffffe009d4edbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009d4edbf0 --- trap 0, rip =3D 0, rsp =3D 0, rbp =3D 0 --- Uptime: 15m13s Dumping 1284 out of 12237 MB:..2%..12%..22%..32%..42%..52%..62%..72%..81%..91% (kgdb) #0=C2=A0 doadump (textdump=3D1) at src/sys/amd64/include/pcpu.h:23= 4 #1=C2=A0 0xffffffff80bdb75d in kern_reboot (howto=3D260) =C2=A0=C2=A0=C2=A0 at /usr/src/sys/kern/kern_shutdown.c:451 #2=C2=A0 0xffffffff80bdbbe9 in vpanic (fmt=3D<value optimized out>, =C2=A0=C2=A0=C2=A0 ap=3D<value optimized out>) at /usr/src/sys/kern/kern_= shutdown.c:877 #3=C2=A0 0xffffffff80bdb9e3 in panic (fmt=3D<value optimized out>) =C2=A0=C2=A0=C2=A0 at /usr/src/sys/kern/kern_shutdown.c:804 #4=C2=A0 0xffffffff810d904e in dblfault_handler (frame=3D<value optimized= out>) =C2=A0=C2=A0=C2=A0 at /usr/src/sys/amd64/amd64/trap.c:1008 #5=C2=A0 0xffffffff810b27b3 in Xdblfault () =C2=A0=C2=A0=C2=A0 at /usr/src/sys/amd64/amd64/exception.S:294 #6=C2=A0 0xffffffff8271eeec in vdev_queue_io_to_issue (vq=3D0xfffff801b5b= 685b8) =C2=A0=C2=A0=C2=A0 at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:638 #7=C2=A0 0xffffffff8271fb48 in vdev_queue_io_done (zio=3D0xfffff801ac3990= 00) =C2=A0=C2=A0=C2=A0 at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:969 #8=C2=A0 0xffffffff82748d91 in zio_vdev_io_done (zio=3D0xfffff801ac399000= ) =C2=A0=C2=A0=C2=A0 at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs= /zfs/zio.c:3376 #9=C2=A0 0xffffffff82744eac in zio_execute (zio=3D0xfffff801ac399000) =C2=A0=C2=A0=C2=A0 at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs= /zfs/zio.c:1786 #10 0xffffffff82748ac7 in zio_vdev_io_start (zio=3D0xfffff801ac399000) =C2=A0=C2=A0=C2=A0 at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs= /zfs/zio.c:3350 #11 0xffffffff82744eac in zio_execute (zio=3D0xfffff801ac399000) =C2=A0=C2=A0=C2=A0 at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs= /zfs/zio.c:1786 #12 0xffffffff8271fb95 in vdev_queue_io_done (zio=3D<value optimized out>= ) =C2=A0=C2=A0=C2=A0 at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:975 #13 0xffffffff82748d91 in zio_vdev_io_done (zio=3D0xfffff8000b376830) =C2=A0=C2=A0=C2=A0 at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs= /zfs/zio.c:3376 #14 0xffffffff82744eac in zio_execute (zio=3D0xfffff8000b376830) =C2=A0=C2=A0=C2=A0 at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs= /zfs/zio.c:1786 #15 0xffffffff82748ac7 in zio_vdev_io_start (zio=3D0xfffff8000b376830) =C2=A0=C2=A0=C2=A0 at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs= /zfs/zio.c:3350 #16 0xffffffff82744eac in zio_execute (zio=3D0xfffff8000b376830) =C2=A0=C2=A0=C2=A0 at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs= /zfs/zio.c:1786 #17 0xffffffff8271fb95 in vdev_queue_io_done (zio=3D<value optimized out>= ) =C2=A0=C2=A0=C2=A0 at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:975 #18 0xffffffff82748d91 in zio_vdev_io_done (zio=3D0xfffff801ac744830) =C2=A0=C2=A0=C2=A0 at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs= /zfs/zio.c:3376 #19 0xffffffff82744eac in zio_execute (zio=3D0xfffff801ac744830) =C2=A0=C2=A0=C2=A0 at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs= /zfs/zio.c:1786 #20 0xffffffff82748ac7 in zio_vdev_io_start (zio=3D0xfffff801ac744830) =C2=A0=C2=A0=C2=A0 at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs= /zfs/zio.c:3350 #21 0xffffffff82744eac in zio_execute (zio=3D0xfffff801ac744830) =C2=A0=C2=A0=C2=A0 at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs= /zfs/zio.c:1786 #22 0xffffffff8271fb95 in vdev_queue_io_done (zio=3D<value optimized out>= ) =2E.... #275 0xffffffff82748ac7 in zio_vdev_io_start (zio=3D0xfffff8000b17e418) =C2=A0=C2=A0=C2=A0 at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs= /zfs/zio.c:3350 #276 0xffffffff82744eac in zio_execute (zio=3D0xfffff8000b17e418) =C2=A0=C2=A0=C2=A0 at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs= /zfs/zio.c:1786 #277 0xffffffff8271fb95 in vdev_queue_io_done (zio=3D<value optimized out= >) =C2=A0=C2=A0=C2=A0 at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:975 #278 0xffffffff82748d91 in zio_vdev_io_done (zio=3D0xfffff801ac355000) =C2=A0=C2=A0=C2=A0 at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs= /zfs/zio.c:3376 #279 0xffffffff82744eac in zio_execute (zio=3D0xfffff801ac355000) =C2=A0=C2=A0=C2=A0 at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs= /zfs/zio.c:1786 #280 0xffffffff82748ac7 in zio_vdev_io_start (zio=3D0xfffff801ac355000) =C2=A0=C2=A0=C2=A0 at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs= /zfs/zio.c:3350 #281 0xffffffff82744eac in zio_execute (zio=3D0xfffff801ac355000) =C2=A0=C2=A0=C2=A0 at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs= /zfs/zio.c:1786 #282 0xffffffff8271fb95 in vdev_queue_io_done (zio=3D<value optimized out= >) =C2=A0=C2=A0=C2=A0 at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:975 #283 0xffffffff82748d91 in zio_vdev_io_done (zio=3D0xfffff8000b8b8000) =C2=A0=C2=A0=C2=A0 at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs= /zfs/zio.c:3376 #284 0xffffffff82744eac in zio_execute (zio=3D0xfffff8000b8b8000) =C2=A0=C2=A0=C2=A0 at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs= /zfs/zio.c:1786 #285 0xffffffff80c3b7f4 in taskqueue_run_locked (queue=3D0xfffff801a8b351= 00) =C2=A0=C2=A0=C2=A0 at /usr/src/sys/kern/subr_taskqueue.c:467 #286 0xffffffff80c3cb28 in taskqueue_thread_loop (arg=3D<value optimized = out>) =C2=A0=C2=A0=C2=A0 at /usr/src/sys/kern/subr_taskqueue.c:773 #287 0xffffffff80b9ab23 in fork_exit ( =C2=A0=C2=A0=C2=A0 callout=3D0xffffffff80c3ca90 <taskqueue_thread_loop>, =C2=A0=C2=A0=C2=A0 arg=3D0xfffff801a0577520, frame=3D0xfffffe009d4edc00) =C2=A0=C2=A0=C2=A0 at /usr/src/sys/kern/kern_fork.c:1063 #288 0xffffffff810b367e in fork_trampoline () =C2=A0=C2=A0=C2=A0 at /usr/src/sys/amd64/amd64/exception.S:996 #289 0x0000000000000000 in ?? () Current language:=C2=A0 auto; currently minimal (kgdb) This is currently repeatable.=C2=A0 What was going on at the instant in t= ime was: root@NewFS:~ # /root/backup-zfs/run-backup Begin local ZFS backup by SEND Run backups of default [zsr/R/12.STABLE-2019-06-14 zsr/home zs/archive zs/colo-archive zs/disk zsr/dbms/pgsql zs/work zs/dbms/ticker-9.6] Thu Jul 18 14:57:57 CDT 2019 Import backup pool Imported; ready to proceed Processing zsr/R/12.STABLE-2019-06-14 Bring incremental backup up to date attempting destroy backup/R/12.STABLE-2019-06-14@zfs-auto-snap_daily-2019-07-10-00h07 success attempting destroy backup/R/12.STABLE-2019-06-14@zfs-auto-snap_daily-2019-07-11-00h07 success It destroyed the snapshot on the backup volume, and panic'd immediately thereafter.=C2=A0 This is an incremental send. If I reboot the machine and re-start the backup job it will blow up when a couple more of the incremental deletes get done. Given the depth of the callback stack is this simply a kernel stack exhaustion problem?=C2=A0 I wouldn't THINK it would be, but...... I do have this in /boot/loader.conf: # Try to avoid kernel stack exhaustion due to TRIM storms. kern.kstack_pages=3D"6" The backup volumes are spinning rust, so there should be no "TRIM" attempts to them.=C2=A0 In theory. I have the dump if someone wants me to run anything specific against it in terms of stack frames, etc. --=20 Karl Denninger karl@denninger.net <mailto:karl@denninger.net> /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms020904040100020602000304 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC DdgwggagMIIEiKADAgECAhMA5EiKghDOXrvfxYxjITXYDdhIMA0GCSqGSIb3DQEBCwUAMIGL MQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJTmljZXZpbGxlMRkw FwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExITAf BgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQTAeFw0xNzA4MTcxNjQyMTdaFw0yNzA4 MTUxNjQyMTdaMHsxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9yaWRhMRkwFwYDVQQKDBBD dWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExJTAjBgNVBAMMHEN1 ZGEgU3lzdGVtcyBMTEMgMjAxNyBJbnQgQ0EwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIK AoICAQC1aJotNUI+W4jP7xQDO8L/b4XiF4Rss9O0B+3vMH7Njk85fZ052QhZpMVlpaaO+sCI KqG3oNEbuOHzJB/NDJFnqh7ijBwhdWutdsq23Ux6TvxgakyMPpT6TRNEJzcBVQA0kpby1DVD 0EKSK/FrWWBiFmSxg7qUfmIq/mMzgE6epHktyRM3OGq3dbRdOUgfumWrqHXOrdJz06xE9NzY vc9toqZnd79FUtE/nSZVm1VS3Grq7RKV65onvX3QOW4W1ldEHwggaZxgWGNiR/D4eosAGFxn uYeWlKEC70c99Mp1giWux+7ur6hc2E+AaTGh+fGeijO5q40OGd+dNMgK8Es0nDRw81lRcl24 SWUEky9y8DArgIFlRd6d3ZYwgc1DMTWkTavx3ZpASp5TWih6yI8ACwboTvlUYeooMsPtNa9E 6UQ1nt7VEi5syjxnDltbEFoLYcXBcqhRhFETJe9CdenItAHAtOya3w5+fmC2j/xJz29og1KH YqWHlo3Kswi9G77an+zh6nWkMuHs+03DU8DaOEWzZEav3lVD4u76bKRDTbhh0bMAk4eXriGL h4MUoX3Imfcr6JoyheVrAdHDL/BixbMH1UUspeRuqQMQ5b2T6pabXP0oOB4FqldWiDgJBGRd zWLgCYG8wPGJGYgHibl5rFiI5Ix3FQncipc6SdUzOQIDAQABo4IBCjCCAQYwHQYDVR0OBBYE FF3AXsKnjdPND5+bxVECGKtc047PMIHABgNVHSMEgbgwgbWAFBu1oRhUMNEzjODolDka5k4Q EDBioYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJ TmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5 c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQYIJAKxAy1WBo2kY MBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgGGMA0GCSqGSIb3DQEBCwUAA4IC AQCB5686UCBVIT52jO3sz9pKuhxuC2npi8ZvoBwt/IH9piPA15/CGF1XeXUdu2qmhOjHkVLN gO7XB1G8CuluxofOIUce0aZGyB+vZ1ylHXlMeB0R82f5dz3/T7RQso55Y2Vog2Zb7PYTC5B9 oNy3ylsnNLzanYlcW3AAfzZcbxYuAdnuq0Im3EpGm8DoItUcf1pDezugKm/yKtNtY6sDyENj tExZ377cYA3IdIwqn1Mh4OAT/Rmh8au2rZAo0+bMYBy9C11Ex0hQ8zWcvPZBDn4v4RtO8g+K uQZQcJnO09LJNtw94W3d2mj4a7XrsKMnZKvm6W9BJIQ4Nmht4wXAtPQ1xA+QpxPTmsGAU0Cv HmqVC7XC3qxFhaOrD2dsvOAK6Sn3MEpH/YrfYCX7a7cz5zW3DsJQ6o3pYfnnQz+hnwLlz4MK 17NIA0WOdAF9IbtQqarf44+PEyUbKtz1r0KGeGLs+VGdd2FLA0e7yuzxJDYcaBTVwqaHhU2/ Fna/jGU7BhrKHtJbb/XlLeFJ24yvuiYKpYWQSSyZu1R/gvZjHeGb344jGBsZdCDrdxtQQcVA 6OxsMAPSUPMrlg9LWELEEYnVulQJerWxpUecGH92O06wwmPgykkz//UmmgjVSh7ErNvL0lUY UMfunYVO/O5hwhW+P4gviCXzBFeTtDZH259O7TCCBzAwggUYoAMCAQICEwCg0WvVwekjGFiO 62SckFwepz0wDQYJKoZIhvcNAQELBQAwezELMAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3Jp ZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBMTEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBD QTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExMQyAyMDE3IEludCBDQTAeFw0xNzA4MTcyMTIx MjBaFw0yMjA4MTYyMTIxMjBaMFcxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9yaWRhMRkw FwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRswGQYDVQQDDBJrYXJsQGRlbm5pbmdlci5uZXQw ggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQC+HVSyxVtJhy3Ohs+PAGRuO//Dha9A 16l5FPATr6wude9zjX5f2lrkRyU8vhCXTZW7WbvWZKpcZ8r0dtZmiK9uF58Ec6hhvfkxJzbg 96WHBw5Fumd5ahZzuCJDtCAWW8R7/KN+zwzQf1+B3MVLmbaXAFBuKzySKhKMcHbK3/wjUYTg y+3UK6v2SBrowvkUBC+jxNg3Wy12GsTXcUS/8FYIXgVVPgfZZrbJJb5HWOQpvvhILpPCD3xs YJFNKEPltXKWHT7Qtc2HNqikgNwj8oqOb+PeZGMiWapsatKm8mxuOOGOEBhAoTVTwUHlMNTg 6QUCJtuWFCK38qOCyk9Haj+86lUU8RG6FkRXWgMbNQm1mWREQhw3axgGLSntjjnznJr5vsvX SYR6c+XKLd5KQZcS6LL8FHYNjqVKHBYM+hDnrTZMqa20JLAF1YagutDiMRURU23iWS7bA9tM cXcqkclTSDtFtxahRifXRI7Epq2GSKuEXe/1Tfb5CE8QsbCpGsfSwv2tZ/SpqVG08MdRiXxN 5tmZiQWo15IyWoeKOXl/hKxA9KPuDHngXX022b1ly+5ZOZbxBAZZMod4y4b4FiRUhRI97r9l CxsP/EPHuuTIZ82BYhrhbtab8HuRo2ofne2TfAWY2BlA7ExM8XShMd9bRPZrNTokPQPUCWCg CdIATQIDAQABo4IBzzCCAcswPAYIKwYBBQUHAQEEMDAuMCwGCCsGAQUFBzABhiBodHRwOi8v b2NzcC5jdWRhc3lzdGVtcy5uZXQ6ODg4ODAJBgNVHRMEAjAAMBEGCWCGSAGG+EIBAQQEAwIF oDAOBgNVHQ8BAf8EBAMCBeAwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMEMDMGCWCG SAGG+EIBDQQmFiRPcGVuU1NMIEdlbmVyYXRlZCBDbGllbnQgQ2VydGlmaWNhdGUwHQYDVR0O BBYEFLElmNWeVgsBPe7O8NiBzjvjYnpRMIHKBgNVHSMEgcIwgb+AFF3AXsKnjdPND5+bxVEC GKtc047PoYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UE BwwJTmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRh IFN5c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQYITAORIioIQ zl6738WMYyE12A3YSDAdBgNVHREEFjAUgRJrYXJsQGRlbm5pbmdlci5uZXQwDQYJKoZIhvcN AQELBQADggIBAJXboPFBMLMtaiUt4KEtJCXlHO/3ZzIUIw/eobWFMdhe7M4+0u3te0sr77QR dcPKR0UeHffvpth2Mb3h28WfN0FmJmLwJk+pOx4u6uO3O0E1jNXoKh8fVcL4KU79oEQyYkbu 2HwbXBU9HbldPOOZDnPLi0whi/sbFHdyd4/w/NmnPgzAsQNZ2BYT9uBNr+jZw4SsluQzXG1X lFL/qCBoi1N2mqKPIepfGYF6drbr1RnXEJJsuD+NILLooTNf7PMgHPZ4VSWQXLNeFfygoOOK FiO0qfxPKpDMA+FHa8yNjAJZAgdJX5Mm1kbqipvb+r/H1UAmrzGMbhmf1gConsT5f8KU4n3Q IM2sOpTQe7BoVKlQM/fpQi6aBzu67M1iF1WtODpa5QUPvj1etaK+R3eYBzi4DIbCIWst8MdA 1+fEeKJFvMEZQONpkCwrJ+tJEuGQmjoQZgK1HeloepF0WDcviiho5FlgtAij+iBPtwMuuLiL shAXA5afMX1hYM4l11JXntle12EQFP1r6wOUkpOdxceCcMVDEJBBCHW2ZmdEaXgAm1VU+fnQ qS/wNw/S0X3RJT1qjr5uVlp2Y0auG/eG0jy6TT0KzTJeR9tLSDXprYkN2l/Qf7/nT6Q03qyE QnnKiBXWAZXveafyU/zYa7t3PTWFQGgWoC4w6XqgPo4KV44OMYIFBzCCBQMCAQEwgZIwezEL MAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBM TEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExM QyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTANBglghkgBZQMEAgMFAKCCAkUw GAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTkwNzE4MjAxMzM3 WjBPBgkqhkiG9w0BCQQxQgRAQE4dWrJ8FP0wAfoJuVYJ4y12HL8HMYZBkc28zbTWNqVKYy4l 0TtKfQqrxINzvOegHaP8/2gyG+gK05Jo2EwecDBsBgkqhkiG9w0BCQ8xXzBdMAsGCWCGSAFl AwQBKjALBglghkgBZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3 DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMIGjBgkrBgEEAYI3EAQxgZUwgZIwezEL MAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBM TEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExM QyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTCBpQYLKoZIhvcNAQkQAgsxgZWg gZIwezELMAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lz dGVtcyBMTEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0 ZW1zIExMQyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTANBgkqhkiG9w0BAQEF AASCAgAkrnbn2VjePQh4vyokXbCCRpuwkAesAsY7f2mVsydWS2A28xhp3QKKvO+QihH+HQvz XEtpXzRwY7QB/uk+DLsevHT1/KEDRK+FtI9bHJ7Qe6qmqKPxD1bymw6cG2AApFPX3BdQinRi NOGI5tuTaiZLjIJEbxmCBpzBLNqAslTRa8ZYulaNsJtHfSeqLQ15zoEbLVQK1yXL+Htky1EH ch1dosXeALHwOFwsv3qO2lK3nBmiPXf1CmRbQcGrdCT8pE4bIFa42NVwu3k8dX+S7jkimCUG tHJ6SIdILOvFnUtbNHmWIZhAnoxkWu6hWxKASYu0hJmuIHN76ItQ61r5nx9w6qBxw8ircWsC kk035iAjQ7n3IXqrxHmzC5J8UvQQwlIE6wtS51lDh4TvVyQsmpXPtRXF5nAHwY1dNX/VqdaF 9yZU7NO8pOIlxuTKtzag6gI3omF4mfsuRJ2FkIG2Z02AIY8iUFwlcGBWjr2oTgT78fdeRXw7 1dknLieDfLxbh0NMfGyn8GACdFYgg/GtOAyCHsv3bBmNXnRKuyVkUAX4JtV6OZo2iwJzncb2 yLrx1XNEb4+oU0xiLoFU/efjybL5g1SYnYJSp91KOVDJyeeaOy2i3i0BDrw4RBoZsYgwR6S2 0friaN2bSPhLlOqlCorzEbnz57g5tOpg2jXz0B8biQAAAAAAAA== --------------ms020904040100020602000304--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?61e5debd-b440-16c9-2a70-0912634e52aa>