Date: Mon, 02 Jun 2014 13:46:43 -0700 From: Mike Carlson <mike@bayphoto.com> To: Steven Hartland <killing@multiplay.co.uk>, freebsd-fs@freebsd.org Subject: Re: ZFS Kernel Panic on 10.0-RELEASE Message-ID: <538CE2B3.8090008@bayphoto.com> In-Reply-To: <88B3A7562A5F4F9B9EEF0E83BCAD2FB0@multiplay.co.uk> References: <5388D64D.4030400@bayphoto.com> <EC2EA442-56FC-46B4-A1E2-97523029B7B3@mail.turbofuzz.com> <5388E5B4.3030002@bayphoto.com> <538BBEB7.4070008@bayphoto.com> <782C34792E95484DBA631A96FE3BEF20@multiplay.co.uk> <538C9CF3.6070208@bayphoto.com> <16ADD4D9DC73403C9669D8F34FDBD316@multiplay.co.uk> <538CB3EA.9010807@bayphoto.com> <6C6FB182781541CEBF627998B73B1DB4@multiplay.co.uk> <538CC16A.6060207@bayphoto.com> <F959477921CD4552A94BF932A55961F4@multiplay.co.uk> <538CDB7F.2060408@bayphoto.com> <88B3A7562A5F4F9B9EEF0E83BCAD2FB0@multiplay.co.uk>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a cryptographically signed message in MIME format.
--------------ms050206090208010102000709
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: quoted-printable
On 6/2/2014 1:44 PM, Steven Hartland wrote:
>
> ----- Original Message ----- From: "Mike Carlson" <mike@bayphoto.com>
> To: "Steven Hartland" <killing@multiplay.co.uk>; <freebsd-fs@freebsd.or=
g>
> Sent: Monday, June 02, 2014 9:15 PM
> Subject: Re: ZFS Kernel Panic on 10.0-RELEASE
>
>
>> On 6/2/2014 1:06 PM, Steven Hartland wrote:
>
>>> I don't have a core.0.txt, I only have:
>>>
>>> ~/p/z/dump> ls -al
>>> total 347690
>>> drwxr-xr-x 3 mikec wheel 8 Jun 2 03:25 .
>>> drwxr-xr-x 4 mikec wheel 5 Jun 2 10:44 ..
>>> drwxrwxr-x 2 mikec operator 2 Jun 2 03:07 .snap
>>> -rw-r--r-- 1 mikec wheel 2 Jun 2 03:24 bounds
>>> -rw------- 1 mikec wheel 446 Jun 2 03:24 info.0
>>> lrwxr-xr-x 1 mikec wheel 6 Jun 2 03:25 info.last -=
>
>>> info.0
>>> -rw------- 1 mikec wheel 3469885440 Jun 2 03:25 vmcore.0
>>> lrwxr-xr-x 1 mikec wheel 8 Jun 2 03:25 vmcore.last=
>>> -> vmcore.0
>>>
>>> But, here is the kgdb output (with backtrace):
>>>
>>> ~/p/z/dump> cat ../kgdb_backtrace.txt
>>> <118>root@:/ # zfs set canmount=3Don zroot/data/working
>>> <118>root@:/ # zfs mount zroot/data/working
>>>
>>>
>>> Fatal trap 12: page fault while in kernel mode
>>> cpuid =3D 14; apic id =3D 22
>>> fault virtual address =3D 0x4a0
>>> fault code =3D supervisor read data, page not present=
>>> instruction pointer =3D 0x20:0xffffffff8185a39f
>>> stack pointer =3D 0x28:0xfffffe1834608570
>>> frame pointer =3D 0x28:0xfffffe18346085b0
>>> code segment =3D base 0x0, limit 0xfffff, type 0x1b
>>> =3D DPL 0, pres 1, long 1, def32 0, gran =
1
>>> processor eflags =3D interrupt enabled, resume, IOPL =3D 0
>>> current process =3D 2 (txg_thread_enter)
>>> trap number =3D 12
>>> panic: page fault
>>> cpuid =3D 14
>>> KDB: stack backtrace:
>>> #0 0xffffffff808e7ee0 at kdb_backtrace+0x60
>>> #1 0xffffffff808af9c5 at panic+0x155
>>> #2 0xffffffff80c8e7b2 at trap_fatal+0x3a2
>>> #3 0xffffffff80c8ea89 at trap_pfault+0x2c9
>>> #4 0xffffffff80c8e216 at trap+0x5e6
>>> #5 0xffffffff80c754b2 at calltrap+0x8
>>> #6 0xffffffff8182eb5a at dsl_dataset_block_kill+0x3a
>>> #7 0xffffffff8182b967 at dnode_sync+0x237
>>> #8 0xffffffff81823fcb at dmu_objset_sync_dnodes+0x2b
>>> #9 0xffffffff81823e4d at dmu_objset_sync+0x1ed
>>> #10 0xffffffff8183829a at dsl_pool_sync+0xca
>>> #11 0xffffffff81853a4e at spa_sync+0x52e
>>> #12 0xffffffff8185c925 at txg_sync_thread+0x375
>>> #13 0xffffffff80881a9a at fork_exit+0x9a
>>> #14 0xffffffff80c759ee at fork_trampoline+0xe
>>> Uptime: 26m15s
>>> Dumping 3309 out of 98234
>>> MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%
>>>
>>> Reading symbols from /boot/kernel/zfs.ko.symbols...done.
>>> Loaded symbols for /boot/kernel/zfs.ko.symbols
>>> Reading symbols from /boot/kernel/opensolaris.ko.symbols...done.
>>> Loaded symbols for /boot/kernel/opensolaris.ko.symbols
>>> #0 doadump (textdump=3D<value optimized out>) at pcpu.h:219
>>> 219 __asm("movq %%gs:%1,%0" : "=3Dr" (td)
>>> (kgdb) backtrace
>>> #0 doadump (textdump=3D<value optimized out>) at pcpu.h:219
>>> #1 0xffffffff808af640 in kern_reboot (howto=3D260) at
>>> /usr/src/sys/kern/kern_shutdown.c:447
>>> #2 0xffffffff808afa04 in panic (fmt=3D<value optimized out>) at
>>> /usr/src/sys/kern/kern_shutdown.c:754
>>> #3 0xffffffff80c8e7b2 in trap_fatal (frame=3D<value optimized out=
>,
>>> eva=3D<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:88=
2
>>> #4 0xffffffff80c8ea89 in trap_pfault (frame=3D0xfffffe18346084c0,=
>>> usermode=3D0) at /usr/src/sys/amd64/amd64/trap.c:699
>>> #5 0xffffffff80c8e216 in trap (frame=3D0xfffffe18346084c0) at
>>> /usr/src/sys/amd64/amd64/trap.c:463
>>> #6 0xffffffff80c754b2 in calltrap () at
>>> /usr/src/sys/amd64/amd64/exception.S:232
>>> #7 0xffffffff8185a39f in bp_get_dsize_sync (spa=3D0xfffff80041835=
000,
>>> bp=3D0xfffffe001b8a1780)
>>> at
>>> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs=
/zfs/spa_misc.c:1635=20
>>>
>>> #8 0xffffffff8182eb5a in dsl_dataset_block_kill
>>> (ds=3D0xfffff800410fec00, bp=3D0xfffffe001b8a1780,
>>> tx=3D0xfffff8004faa0600, async=3D0)
>>> at
>>> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs=
/zfs/dsl_dataset.c:129=20
>>>
>>> #9 0xffffffff8182b967 in dnode_sync (dn=3D0xfffff8004fe626c0,
>>> tx=3D0xfffff8004faa0600) at
>>> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs=
/zfs/dnode_sync.c:128=20
>>>
>>> #10 0xffffffff81823fcb in dmu_objset_sync_dnodes
>>> (list=3D0xfffff80041956b10, newlist=3D<value optimized out>, tx=3D=
<value
>>> optimized out>)
>>> at
>>> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs=
/zfs/dmu_objset.c:945=20
>>>
>>> #11 0xffffffff81823e4d in dmu_objset_sync (os=3D0xfffff80041956800=
,
>>> pio=3D0xfffff800418c43b0, tx=3D0xfffff8004faa0600)
>>> at
>>> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs=
/zfs/dmu_objset.c:1062=20
>>>
>>> #12 0xffffffff8183829a in dsl_pool_sync (dp=3D0xfffff8004183c000,
>>> txg=3D<value optimized out>)
>>> at
>>> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs=
/zfs/dsl_pool.c:413=20
>>>
>>> #13 0xffffffff81853a4e in spa_sync (spa=3D0xfffff80041835000,
>>> txg=3D3373534) at
>>> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs=
/zfs/spa.c:6410=20
>>>
>>> #14 0xffffffff8185c925 in txg_sync_thread (arg=3D0xfffff8004183c00=
0)
>>> at
>>> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs=
/zfs/txg.c:515=20
>>>
>>> #15 0xffffffff80881a9a in fork_exit (callout=3D0xffffffff8185c5b0
>>> <txg_sync_thread>, arg=3D0xfffff8004183c000, frame=3D0xfffffe18346=
08ac0)
>>> at /usr/src/sys/kern/kern_fork.c:995
>>> #16 0xffffffff80c759ee in fork_trampoline () at
>>> /usr/src/sys/amd64/amd64/exception.S:606
>>> #17 0x0000000000000000 in ?? ()
>>> Current language: auto; currently minimal
>>>
>>>
>>> If anyone wants to help out and check out the vmcore file, email me=20
>>> off the list and I'll provide a S3 url of the tar'd + xz file.
>>>
>>>
>> Output of "frame 7":
>>
>> (kgdb) frame 7
>> #7 0xffffffff8185a39f in bp_get_dsize_sync (spa=3D0xfffff800418350=
00,
>> bp=3D0xfffffe001b8a1780)
>> at
>> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/=
zfs/spa_misc.c:1635
>> 1635 dsize =3D (asize >> SPA_MINBLOCKSHIFT) *
>> vd->vdev_deflate_ratio;
>>
>> Is that what you were looking for?
>
> Thats the line I gathered it was on but no I need to know what the valu=
e
> of vd is, so what you need to do is:
> print vd
>
> If thats valid then:
> print *vd
>
It reports:
(kgdb) print *vd
No symbol "vd" in current context.
Should I rebuild the kernel with additional options?
> Given the panic I'm kind of expecting garbage or null (0x00)
>
>> I'm not familar with this process, so I hope this does not become too =
>> painful in pulling the details out.
>
> No problem, everyone has to learn some time ;-)
>
Thanks Steve :)
> Regards
> Steve
>
>
--------------ms050206090208010102000709
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature
MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIITIjCC
BhwwggQEoAMCAQICCGWP9DPVdj3ZMA0GCSqGSIb3DQEBCwUAMEsxDzANBgNVBAMMBlJvb3RD
QTEWMBQGA1UECgwNQmF5IFBob3RvIExhYjETMBEGA1UECAwKQ2FsaWZvcm5pYTELMAkGA1UE
BhMCVVMwHhcNMTIxMDIzMTczMjE4WhcNMjcxMDIzMTczMjE4WjBYMRwwGgYDVQQDDBNCYXkg
UGhvdG8gUGVvcGxlIENBMRYwFAYDVQQKDA1CYXkgUGhvdG8gTGFiMRMwEQYDVQQIDApDYWxp
Zm9ybmlhMQswCQYDVQQGEwJVUzCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBALuW
vZE7FFSO77jEhdT/dZzu0XnPS35aepuvMvCNTagnNMbaf+AKrLZFaVRqBt4f6yl5TDXNEAMI
+CJrg7T9dsoGN1Vygm4bIFzOIRtTZ1A7seemeonyqMto8gYMPhEFicuKavEAFcCFprVcVu5v
yXbEG+tYkx08oUzeZ9LWZnh4a0wxAucbH0PcZFmdXFMbgjt6GSiR8jVUT1t/Dik1+mJ1A4K3
3VyM+G1CaqGIKg2UbvHiVQVoJrpgUc6vO5aWWgDreH+CAKLpyJwCj4BGphfHBPENu5LUp0CL
fa4pA4r+OH0HNBp/I7utZMrIevh3pya34lDJXj0CQWSuVBR9Kp80IHHYGuyPU/sMXkUplsyI
Y4KgQSRYRFNdWoboLwsRu1/2iDUaTR+XYH7Tu5zvCOedUm8ns/wfE7TW6kax5nR36VzuZS6B
kUcuETO1QMxt2SJcHZIftiyTuxC8Y9Z7J6igR39pZPWGywJ2+oPRKFQD/u9RWZqjh9k5euVi
4bVwwog5DqnsymMTuKb0I1nCs5js+rfEVo+mc4gAfEmzk/dmpwkOnGV3oTfASbGtwMIlAEe5
HHJhdTA3jpVo3eQAZi479xZ7SozKvnjt1y9SMQ4uTJXjER/IVH3YC9S/BSFrqWLLDfWEGW84
SBn+ogldDIy3PX3SU7nOiN4Vse2Jg+r7AgMBAAGjgfYwgfMwHQYDVR0OBBYEFC34wnK9ZmLm
B7vV52IeLHaapJ33MA8GA1UdEwEB/wQFMAMBAf8wHwYDVR0jBBgwFoAU00bJTnGPaRmhJAUZ
t4x4J3soVyswgY8GA1UdHwSBhzCBhDCBgaB/oH2Ge2h0dHA6Ly9iYXljYS5iYXlob3RvLmxv
Y2FsL2VqYmNhL3B1YmxpY3dlYi93ZWJkaXN0L2NlcnRkaXN0P2NtZD1jcmwmaXNzdWVyPUNO
PVJvb3RDQSxPPUJheSUyMFBob3RvJTIwTGFiLFNUPUNhbGlmb3JuaWEsQz1VUzAOBgNVHQ8B
Af8EBAMCAYYwDQYJKoZIhvcNAQELBQADggIBAEoDTeb1wVXz2VqFPrc3uaRnbVt6DCB9uBft
L8YRgi5+FF5KHjvTqYHGiS38yVFfXM6MaN6yI7WSwtS+wVhM7Tdw6miwKNBApmCDKzjorNVX
iZUmv88fjGm652LeIfT/oFGJg2oruoad9OfIofjT/jFpiFT7KOgjg4xeKCC0f+9naVo5uWOJ
HTyFm1Kq6ryT+g7mZb7kLvndmJRW0aybn8Dls7/bhThEaOYkwn4dbZ6q0W2I5ZWg9bbh66V+
J9P1XObZA/TkRVTlpSYJ1hAgYX1yTbL360vuTAAwdSWQSFm4TAoXC4BsPdlgy86lDjNrW9id
WX0RJCBzk/FzHhI4Aj9+SVhLl/Vkf9nbz8VhPPK2Az0PPYKy5ARtgOqGTKZbgVKut9Kgc/fn
vUgPhObHQsD1Ui+NKsIGYBFKBhNmVXqMQSn4JzC9x0oDEmv37UGrcut2cCP3ZS17p137VaUM
lQ0RWomju+sPCPFgyCPa/TLPoMZ2334uIxkRbDefvOoXIosORMQ9Jh50XqktUesbhuBfH9Q5
8h8bTWm1Cn/LxXW9qdSSbnta0OAH1G4hwVUlcusSM0o7Ude8tszw6kRpmEDDE8BQjE5nXSY7
wOf1eXfQfDkVQiouTD3l5ElqB98tnCnL/y9dphWoBLmiJwgb/4yWZ/Zewc1V65UFr7LmvcQM
MIIGIzCCBAugAwIBAgIIYpSXgZOT7j0wDQYJKoZIhvcNAQELBQAwWDEcMBoGA1UEAwwTQmF5
IFBob3RvIFBlb3BsZSBDQTEWMBQGA1UECgwNQmF5IFBob3RvIExhYjETMBEGA1UECAwKQ2Fs
aWZvcm5pYTELMAkGA1UEBhMCVVMwHhcNMTIxMDIzMTgwMDAzWhcNMTQxMDIzMTgwMDAzWjBg
MRUwEwYKCZImiZPyLGQBAQwFMTMwNjkxFTATBgNVBAMMDE1pa2UgQ2FybHNvbjELMAkGA1UE
CwwCSVQxFjAUBgNVBAoMDUJheSBQaG90byBMYWIxCzAJBgNVBAYTAlVTMIICIjANBgkqhkiG
9w0BAQEFAAOCAg8AMIICCgKCAgEAoTzIvF66A3wYPWXQFzmjBUuIrHSQGhOcE7rb0KQtar+H
rkmHX56bCAklW5P/pd+yJ08lMwb3CxbTOz0q47JuBv2kKJO1xCgCua26Uvz3VAmfirmWwpXq
zZBDqy/bEIt/XFfiVUC4jriGSEPrtx9q9nJJsb2JVRgtsbcHaaJFu8u8s8p8cLbcYdKobS00
g6+7it2IpIJhxc5tEMa1Yku3kCQiHVVFa9b4H5pFDHpkCrKZ43cuCneiR5kgr47z/3U66kLt
J7Q5IT/i7nThjGQMa/f1JSWet8yeTomKvqkuEAA4o/IWQzEbtxzeps6vWxaCDULjEq69s//S
6PtqiQSmG9ZGFoPYD8/GGd4CMBqgjKopintD5sGTlJ851yZwl9VY/hRuxInp8gTjWrt1gQIB
zlgSgSKnKTN6f+e85XMPU1y/wVz8RJWl8Tr11kzo6vrM14+ruNUxo1Ea3PJ9MUcWenoRKGSU
I/IP94kZVjPkZlJv6tTF0Yi2Gclet/ZDu8vgvkxmUZYdQMGGlgZTCAsvHr37/ov6g51Tf+im
7410EsdYCmSINRGzWQAzlH9NscsW1TAd1Znog1H6NRDExY3ksjvFcKYOjUmkyWT8Vl2oJmT2
IzI23/C3esGL9OZzZ6K84MRNrH1y/yNp75vQnP3JfDMpbb5kkDp95Bu365qBluECAwEAAaOB
6DCB5TBSBggrBgEFBQcBAQRGMEQwQgYIKwYBBQUHMAGGNmh0dHA6Ly9iYXljYS5iYXlob3Rv
LmxvY2FsL2VqYmNhL3B1YmxpY3dlYi9zdGF0dXMvb2NzcDAdBgNVHQ4EFgQUzeso+31hmtp3
soKHShXXtAEo+iMwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBQt+MJyvWZi5ge71ediHix2
mqSd9zAOBgNVHQ8BAf8EBAMCBDAwEwYDVR0lBAwwCgYIKwYBBQUHAwQwHAYDVR0RBBUwE4ER
bWlrZUBiYXlwaG90by5jb20wDQYJKoZIhvcNAQELBQADggIBADnQfCasVgMsKsxIZAOZCbsU
xo9BfsbpoM02p2aP+vPNDLXYRmcnH6ReeeUKSfIn0HmS9XkeHizMEXaC5sV9g4dasdQJQOGU
mDcBnlxGn5fzNVFBM7/RHL83waYq4MCeyP9M7lSiNFZTrnSLVL9lIO0FLrBE06c9bn09kExc
zkXI6Qm+e/MNrnoC3vw3GbH3a7tZCPsQcyNSok99jgPTRb1g9uVPg25M+ScYMU0wv2BE24u1
Dfzwcq52h64TllbzdVg/qOQH1HM96wmU+CtPuzA6eYnWRao/80LfQcyhNZ/jfMB/9xwFwsam
o3Bw7SrSPEatw/tMyEEVMzas5/wZm2uMtab7642d5mr5OWLVPYgmKUscSlNt87vKkFhvn0Cz
Z7O8O79WNMJA0sx1aomn1/ZrWDkd8X/ACUC2Fa3cV4AAzmjytiNu7r2z+GwdXPmvWSlBDXKX
wLSoRkdq5hmYAP3GwXF0dsZo63WJLuCU1bPyERNLKdZM//eX832WgomPs4FA4xg0MUH0S7vJ
eo7K1cTutZEmyLT623p0GcOINs2ir/ZqPTDLKszI7ytAltYaATt4kYUXbmMGGYItDf1X/caj
DoLv2hjBTM5HORZYABC/Kfo9iL4KeYDqAvblJc7qyw+QXdHOUbwc9gQXQJvlQlfjDYvJLKme
zoZ1sMzRBOl0MIIG1zCCBL+gAwIBAgIIGE38aUOyx8EwDQYJKoZIhvcNAQELBQAwWDEcMBoG
A1UEAwwTQmF5IFBob3RvIFBlb3BsZSBDQTEWMBQGA1UECgwNQmF5IFBob3RvIExhYjETMBEG
A1UECAwKQ2FsaWZvcm5pYTELMAkGA1UEBhMCVVMwHhcNMTIxMDIzMTc1NzQ1WhcNMTQxMDIz
MTc1NzQ1WjBgMRUwEwYKCZImiZPyLGQBAQwFMTMwNjkxFTATBgNVBAMMDE1pa2UgQ2FybHNv
bjELMAkGA1UECwwCSVQxFjAUBgNVBAoMDUJheSBQaG90byBMYWIxCzAJBgNVBAYTAlVTMIIC
IjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAuECpC3YUm7GV0xz/DMmMQZ3EvMfvVhFB
77TcVxY1OoBlp2jk0ST0Hel/vp9uPhhi/eAlH89rC9fhNwORUHfeofWhoT3ZXrnjisNDQnb5
MgBV6wVM58PrikwU13FuNrIrPuUeuUE659BQhfgj2j2Pv9GVgBib6NBbHJAIOFo8H3wmd4b9
Yv6RKM145qSJCrJp96jSkoQSwni+jraHGMs55kgFP/3f0X6RpY7GhvItFI95Xauf7R5qTuW0
oamhvjFnlWVBoMuHd5yqzUgL7gQ0wfB31wfBP2ghFPAv715Qv9DYleFmgWE1LRkrt8clCzzk
huj00CrEL+rSK9bDYNpa5AdCQ8aA/bG2x24ApOBvfDYnem+Ytu9lIY7qKZiP+9pASOiXsSSt
g8CH9HoG+9GpK/rDyQ2nuNxTWEQEetwofkLdrGU/VgpcwIZqETsugu+l/FCu3Ogslezpiolb
SqjCF5CU3aZravNZo4+HDriFKm7jyKEb+zUcXcwNH2iwbGt6uTMILslXbWXuhi8Lu3SKyacj
1gn9OMPQTCUKxaiTJXqIcJP0CV9wKcgTn7vanChDgz2fTVm6HOzb2RZlyhYzU4ofuMk+VGbg
tBSDns+B9D1ACZMd58D2XdqRvmGPlnbJEZPkAZyDiR4mp/Aw09uXzOCIHzvqLhiysGon7lkV
a18CAwEAAaOCAZswggGXMFIGCCsGAQUFBwEBBEYwRDBCBggrBgEFBQcwAYY2aHR0cDovL2Jh
eWNhLmJheWhvdG8ubG9jYWwvZWpiY2EvcHVibGljd2ViL3N0YXR1cy9vY3NwMB0GA1UdDgQW
BBRGT4gPK1JkpmK4YAsQgD82sskTGDAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFC34wnK9
ZmLmB7vV52IeLHaapJ33MIGlBgNVHR8EgZ0wgZowgZeggZSggZGGgY5odHRwOi8vYmF5Y2Eu
YmF5aG90by5sb2NhbC9lamJjYS9wdWJsaWN3ZWIvd2ViZGlzdC9jZXJ0ZGlzdD9jbWQ9Y3Js
Jmlzc3Vlcj1DTj1CYXklMjBQaG90byUyMFBlb3BsZSUyMENBLE89QmF5JTIwUGhvdG8lMjBM
YWIsU1Q9Q2FsaWZvcm5pYSxDPVVTMA4GA1UdDwEB/wQEAwIGwDAdBgNVHSUEFjAUBggrBgEF
BQcDAgYIKwYBBQUHAwQwHAYDVR0RBBUwE4ERbWlrZUBiYXlwaG90by5jb20wDQYJKoZIhvcN
AQELBQADggIBAC+1BfV1bmcHZtgM4HN5rZek1vzY2UBLTHevnLDsLmNNASYMuOrm1TYMCI0/
LVkGltk0kq3lAP3hECsrSeH+SlmoRAmSCEOAwqOBU5oTqV/SMiQcZb6K2o/QUAVV6Si2KMyW
U35hnqi9/vVNq4y28RjvMAydcrJpmn5qrWsy1ZLbW+Zu3jneBYjUcm4mQnqaCyjRyE2Z3byi
/4m1sxISSdyqoRsWKsixSdb4bXUdNeNscltRYDMEDODtzYjnO7WebHtaMAEOyfE3n2gkPmGr
dBopA6/Z53GVkG+MXF1wSlc3g38dHSpbY6qJiMkl8/wKCnkcMRPC3hRGQikYprh/oAbpHAyS
q9P0jv/3cB0yzZ5b3n49PxqjwyH93lcRZJQ5pAMMWFk1B7IuYk9LVbJEVsPAmxFbWoS87Tn0
/zi6lkUNXoSgm4fJ+NBY5jnk1m48nPFIs7dpr8wW8qT2C0DiQz/tSKMramzsBduXY67djCZ5
cam1H5pRuaiEPEmrafC91i+wCsmjpMoqykKkuiGOlmY8LhRSZZ7M8qwtpo/0yur94z1Z5gyG
3yo/vZQtNDt8jaJ2ajGR50C48CtJufrckT3o3UrtjvQ3jCUnak2VbXIV6QNTio9NQKVH4+fL
VhoGvc/PC/CifBs6Qyfk3a6nHdVfzwGx8Uytj/Br0AV0p8Y2MYIERjCCBEICAQEwZDBYMRww
GgYDVQQDDBNCYXkgUGhvdG8gUGVvcGxlIENBMRYwFAYDVQQKDA1CYXkgUGhvdG8gTGFiMRMw
EQYDVQQIDApDYWxpZm9ybmlhMQswCQYDVQQGEwJVUwIIGE38aUOyx8EwCQYFKw4DAhoFAKCC
AbcwGAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTQwNjAyMjA0
NjQzWjAjBgkqhkiG9w0BCQQxFgQU9DRFRFD+ueornXIePCgh6LilaM8wbAYJKoZIhvcNAQkP
MV8wXTALBglghkgBZQMEASowCwYJYIZIAWUDBAECMAoGCCqGSIb3DQMHMA4GCCqGSIb3DQMC
AgIAgDANBggqhkiG9w0DAgIBQDAHBgUrDgMCBzANBggqhkiG9w0DAgIBKDBzBgkrBgEEAYI3
EAQxZjBkMFgxHDAaBgNVBAMME0JheSBQaG90byBQZW9wbGUgQ0ExFjAUBgNVBAoMDUJheSBQ
aG90byBMYWIxEzARBgNVBAgMCkNhbGlmb3JuaWExCzAJBgNVBAYTAlVTAghilJeBk5PuPTB1
BgsqhkiG9w0BCRACCzFmoGQwWDEcMBoGA1UEAwwTQmF5IFBob3RvIFBlb3BsZSBDQTEWMBQG
A1UECgwNQmF5IFBob3RvIExhYjETMBEGA1UECAwKQ2FsaWZvcm5pYTELMAkGA1UEBhMCVVMC
CGKUl4GTk+49MA0GCSqGSIb3DQEBAQUABIICAJ2iy7R2vLDhGuSGMooD5cEnuuwkkyyyMODK
6T6Y6wpRte/zTrCuo+t+v0GNj5eDCfO2C7+P9OXEqSIi+SPf8vW+TmeM8E6GFQ1vE52Jo+5v
YpgtPx6dL/9yKihHigVuLACMNr701EHlbhstsM0VVsE4EzETv1JM1Bl4oaWX5s7n+4H9wuvI
3p4Tg1WmUaypv1YrBQUNDTWefovww14WFwMPFxInMQUkAGYHZ2f2HrkkbtNo9FmoopexajbM
Drkm5UrsH8cSCCyWHLB1Zvx+Pdr3R1/CUYi91DuFo20vPCSQBuElwIBd/KV/D7TrqdHkGH9R
csVVHv5U3Vkp2bdvdy34taL3HgAM+vp23Xtv3c0UMz2ZrL+5DFmaOHQ5ZncFbfVe+d8oya5+
hWgndUnarAY8lC07lheOhcOe61nNe8idN7zEa4BnIQot5lXEm8ocscv81eAHiG/cmE99S9ax
LfT7Q7j7SRzbYLwaHNGCsfiNZHE186GJxBtGXay4hC4FcAhdofM75/Gm6TJSm9sGLX/ZHmjP
h+5uzfa0yE1yP7950u7/9ZxgUripr3HoVBXgngJIFN7vLI2QZapBid2LAch4tmVDLuTxeu/m
3sxQQyxvU5Gfs5PgG3afABiT4d8uqOIzOJzwAhe2IfDBWhCQHRS2vlNPSMxVy4eQe2OHzFCc
AAAAAAAA
--------------ms050206090208010102000709--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?538CE2B3.8090008>