Date: Thu, 12 Jun 2025 15:14:28 -0400 From: Karl Denninger <karl@denninger.net> To: stable@freebsd.org Subject: Re: 14.3R zfs panic VERIFY3(zfs_btree_find(tree, node, &where) == NULL) failed (0xfffff80090dd93a0x == 0x) Message-ID: <d44a2a6c-67c1-4dfb-b281-dbcfdd1a4971@denninger.net> In-Reply-To: <6ff367d2-7684-4080-bf2d-dfb2b280d48c@sentex.net> References: <0c989fde-2b33-442b-9e39-680f584baa7c@sentex.net> <19db4360-1846-4c6c-b8d1-ead546defc16@sentex.net> <6ff367d2-7684-4080-bf2d-dfb2b280d48c@sentex.net>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a cryptographically signed message in MIME format.
--------------ms060103010105020003090105
Content-Type: multipart/alternative;
boundary="------------a3De5p6tL1jUlaeVEnQw5v05"
--------------a3De5p6tL1jUlaeVEnQw5v05
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: base64
T24gNi8xMi8yMDI1IDE1OjA0LCBtaWtlIHRhbmNzYSB3cm90ZToNCj4gT24gNi8xMi8yMDI1
IDExOjQ3IEFNLCBtaWtlIHRhbmNzYSB3cm90ZToNCj4+IE9uIDYvMTIvMjAyNSA3OjEyIEFN
LCBtaWtlIHRhbmNzYSB3cm90ZToNCj4+PiBOb3Qgc3VyZSBpZiB0aGlzIGlzIG9wZXJhdG9y
IGVycm9yIG9yIHRyaWdnZXJpbmcgYSBidWcuwqDCoCBCdXQgbXkgDQo+Pj4gcHJvY2VzcyB3
b3JrZWQgZmluZSB3aXRoIGEgMTQuMlIgYmFzZSBpbWFnZS4NCj4+Pg0KPj4gUFIgc3VibWl0
dGVkDQo+Pg0KPj4gaHR0cHM6Ly9idWdzLmZyZWVic2Qub3JnL2J1Z3ppbGxhL3Nob3dfYnVn
LmNnaT9pZD0yODc0ODINCj4+DQo+DQo+IEl0IHNlZW1zIHRvIGJlIGp1c3QgdGhlIC5yYXcg
aW1hZ2UgdGhhdCBJIGNhbiB0cmlnZ2VyIHRoZSBwYW5pYyBvbi4gSWYgDQo+IEkgZG93bmxv
YWQgdGhlIC5xY293MiB2ZXJzaW9uIGl0cyBmaW5lLiBJIGFsc28gdGVzdGVkIHRoZSBzdGFi
bGUgaW1hZ2UgDQo+IGZyb20gSnVuZSA2IGFuZCBpdHMgZmluZSB0b28NCj4NCj4gSXRzIGVh
c3kgZW5vdWdoIHRvIHRlc3QuIE9uIGEgbWFjaGluZSwganVzdA0KPg0KPiBmZXRjaCANCj4g
aHR0cHM6Ly9kb3dubG9hZC5mcmVlYnNkLm9yZy9yZWxlYXNlcy9WTS1JTUFHRVMvMTQuMy1S
RUxFQVNFL2FtZDY0L0xhdGVzdC9GcmVlQlNELTE0LjMtUkVMRUFTRS1hbWQ2NC16ZnMucmF3
Lnh6DQo+DQo+IHh6IC1kIEZyZWVCU0QtMTQuMy1SRUxFQVNFLWFtZDY0LXpmcy5yYXcueHoN
Cj4NCj4gc2ggL3Vzci9zaGFyZS9leGFtcGxlcy9iaHl2ZS92bXJ1bi5zaCAtZCANCj4gLi9G
cmVlQlNELTE0LjMtUkVMRUFTRS1hbWQ2NC16ZnMucmF3IC1FIC1IIHZtMC9ob3N0IHZtMA0K
Pg0KPg0KPiBMb2dpbiBhcyByb290IChubyBwYXNzKSBhbmQgaXNzdWUgdGhlIGNvbW1hbmQN
Cj4NCj4gcGVyaW9kaWMgZGFpbHkNCj4NCj4gcm9vdEBmcmVlYnNkOn4gIyBwZXJpb2RpYyBk
YWlseQ0KPiBwYW5pYzogVkVSSUZZMyh6ZnNfYnRyZWVfZmluZCh0cmVlLCBub2RlLCAmd2hl
cmUpID09IE5VTEwpIGZhaWxlZCANCj4gKDB4ZmZmZmY4MDAxOTMwMDlhMHggPT0gMHgpDQo+
DQo+IGNwdWlkID0gMA0KPiB0aW1lID0gMTc0OTc1NDg2OQ0KPiBLREI6IHN0YWNrIGJhY2t0
cmFjZToNCj4gIzAgMHhmZmZmZmZmZjgwYmE4ZjFkIGF0IGtkYl9iYWNrdHJhY2UrMHg1ZA0K
PiAjMSAweGZmZmZmZmZmODBiNWFhMTEgYXQgdnBhbmljKzB4MTYxDQo+ICMyIDB4ZmZmZmZm
ZmY4MjE4MWM0YSBhdCBzcGxfcGFuaWMrMHgzYQ0KPiAjMyAweGZmZmZmZmZmODIxZDRhYWUg
YXQgemZzX2J0cmVlX2FkZCsweDZlDQo+ICM0IDB4ZmZmZmZmZmY4MjMxNTkwNSBhdCB6YXBf
bG9ja2Rpcl9pbXBsKzB4MzI1DQo+ICM1IDB4ZmZmZmZmZmY4MjMxNTVhNCBhdCB6YXBfbG9j
a2RpcisweDg0DQo+ICM2IDB4ZmZmZmZmZmY4MjMxOGIxNSBhdCB6YXBfY3Vyc29yX3JldHJp
ZXZlKzB4MTc1DQo+ICM3IDB4ZmZmZmZmZmY4MjFhMDFhMyBhdCB6ZnNfZnJlZWJzZF9yZWFk
ZGlyKzB4MzgzDQo+ICM4IDB4ZmZmZmZmZmY4MTExMjA1MCBhdCBWT1BfUkVBRERJUl9BUFYr
MHgyMA0KPiAjOSAweGZmZmZmZmZmODBjNTZhMTkgYXQga2Vybl9nZXRkaXJlbnRyaWVzKzB4
MjI5DQo+ICMxMCAweGZmZmZmZmZmODBjNTZkYzkgYXQgc3lzX2dldGRpcmVudHJpZXMrMHgy
OQ0KPiAjMTEgMHhmZmZmZmZmZjgxMDRlNGM3IGF0IGFtZDY0X3N5c2NhbGwrMHgxMTcNCj4g
IzEyIDB4ZmZmZmZmZmY4MTAyNDY5YiBhdCBmYXN0X3N5c2NhbGxfY29tbW9uKzB4ZjgNCj4g
VXB0aW1lOiA0NXMNCj4gQXV0b21hdGljIHJlYm9vdCBpbiAxNSBzZWNvbmRzIC0gcHJlc3Mg
YSBrZXkgb24gdGhlIGNvbnNvbGUgdG8gYWJvcnQNCj4gLS0+IFByZXNzIGEga2V5IG9uIHRo
ZSBjb25zb2xlIHRvIHJlYm9vdCwNCj4gLS0+IG9yIHN3aXRjaCBvZmYgdGhlIHN5c3RlbSBu
b3cuDQo+IGZidWYgZnJhbWUgYnVmZmVyIGJhc2U6IDB4M2VkMjIyNDAwMDAwIFtzeiAxNjc3
NzIxNl0NCj4NClNvdW5kcyBsaWtlIHRoZSBmaWxlc3lzdGVtIChlLmcuIGxvYWQgZmlsZSkg
aXMgYm9ya2VkIGludGVybmFsbHkuDQoNClNjcnViIGNoZWNrcyBibG9jayBpbnRlZ3JpdHkg
YnV0IEkgZG9uJ3QgYmVsaWV2ZSBpdCB3aWxsIC9uZWNlc3NhcmlseSANCi9kZXRlY3QgYSBm
aWxlc3lzdGVtIHRoYXQgaGFzIHByb3BlciBjaGVja3N1bXMgYnV0IGlzIGJvcmtlZCB3aXRo
aW4gdGhlIA0Kc3RydWN0dXJlIG9mIHRoZSBtZXRhZGF0YSBpdHNlbGYuDQoNCi0tIA0KS2Fy
bCBEZW5uaW5nZXINCmthcmxAZGVubmluZ2VyLm5ldA0KL1RoZSBNYXJrZXQgVGlja2VyLw0K
L1tTL01JTUUgZW5jcnlwdGVkIGVtYWlsIHByZWZlcnJlZF0vDQo=
--------------a3De5p6tL1jUlaeVEnQw5v05
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
<!DOCTYPE html>
<html>
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3DUTF=
-8">
</head>
<body>
<div class=3D"moz-cite-prefix">On 6/12/2025 15:04, mike tancsa wrote:=
<br>
</div>
<blockquote type=3D"cite"
cite=3D"mid:6ff367d2-7684-4080-bf2d-dfb2b280d48c@sentex.net">On
6/12/2025 11:47 AM, mike tancsa wrote:
<br>
<blockquote type=3D"cite">On 6/12/2025 7:12 AM, mike tancsa wrote:
<br>
<blockquote type=3D"cite">Not sure if this is operator error or
triggering a bug.=C2=A0=C2=A0 But my process worked fine with a=
14.2R
base image.
<br>
<br>
</blockquote>
PR submitted
<br>
<br>
<a class=3D"moz-txt-link-freetext" href=3D"https://bugs.freebsd.o=
rg/bugzilla/show_bug.cgi?id=3D287482">https://bugs.freebsd.org/bugzilla/s=
how_bug.cgi?id=3D287482</a>
<br>
<br>
</blockquote>
<br>
It seems to be just the .raw image that I can trigger the panic
on. If I download the .qcow2 version its fine. I also tested the
stable image from June 6 and its fine too
<br>
<br>
Its easy enough to test. On a machine, just
<br>
<br>
fetch
<a class=3D"moz-txt-link-freetext" href=3D"https://download.freebsd.org/r=
eleases/VM-IMAGES/14.3-RELEASE/amd64/Latest/FreeBSD-14.3-RELEASE-amd64-zf=
s.raw.xz">https://download.freebsd.org/releases/VM-IMAGES/14.3-RELEASE/am=
d64/Latest/FreeBSD-14.3-RELEASE-amd64-zfs.raw.xz</a><br>
<br>
xz -d FreeBSD-14.3-RELEASE-amd64-zfs.raw.xz
<br>
<br>
sh /usr/share/examples/bhyve/vmrun.sh -d
./FreeBSD-14.3-RELEASE-amd64-zfs.raw -E -H vm0/host vm0
<br>
<br>
<br>
Login as root (no pass) and issue the command
<br>
<br>
periodic daily
<br>
<br>
root@freebsd:~ # periodic daily
<br>
panic: VERIFY3(zfs_btree_find(tree, node, &where) =3D=3D NULL)
failed (0xfffff800193009a0x =3D=3D 0x)
<br>
<br>
cpuid =3D 0
<br>
time =3D 1749754869
<br>
KDB: stack backtrace:
<br>
#0 0xffffffff80ba8f1d at kdb_backtrace+0x5d
<br>
#1 0xffffffff80b5aa11 at vpanic+0x161
<br>
#2 0xffffffff82181c4a at spl_panic+0x3a
<br>
#3 0xffffffff821d4aae at zfs_btree_add+0x6e
<br>
#4 0xffffffff82315905 at zap_lockdir_impl+0x325
<br>
#5 0xffffffff823155a4 at zap_lockdir+0x84
<br>
#6 0xffffffff82318b15 at zap_cursor_retrieve+0x175
<br>
#7 0xffffffff821a01a3 at zfs_freebsd_readdir+0x383
<br>
#8 0xffffffff81112050 at VOP_READDIR_APV+0x20
<br>
#9 0xffffffff80c56a19 at kern_getdirentries+0x229
<br>
#10 0xffffffff80c56dc9 at sys_getdirentries+0x29
<br>
#11 0xffffffff8104e4c7 at amd64_syscall+0x117
<br>
#12 0xffffffff8102469b at fast_syscall_common+0xf8
<br>
Uptime: 45s
<br>
Automatic reboot in 15 seconds - press a key on the console to
abort
<br>
--> Press a key on the console to reboot,
<br>
--> or switch off the system now.
<br>
fbuf frame buffer base: 0x3ed222400000 [sz 16777216]
<br>
<br>
</blockquote>
<p>Sounds like the filesystem (e.g. load file) is borked internally.<=
/p>
<p>Scrub checks block integrity but I don't believe it will <i>necess=
arily
</i>detect a filesystem that has proper checksums but is borked
within the structure of the metadata itself.<br>
</p>
<div class=3D"moz-signature">-- <br>
Karl Denninger<br>
<a href=3D"mailto:karl@denninger.net" class=3D"moz-txt-link-freetex=
t">karl@denninger.net</a><br>
<i>The Market Ticker</i><br>
<font size=3D"-2"><i>[S/MIME encrypted email preferred]</i></font><=
/div>
</body>
</html>
--------------a3De5p6tL1jUlaeVEnQw5v05--
--------------ms060103010105020003090105
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature
MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC
C4owggWZMIIDgaADAgECAhRZU8dKdMneRI1Vq5kv0k54Q5rQuDANBgkqhkiG9w0BAQsFADB2
MQswCQYDVQQGEwJVUzESMBAGA1UECAwJVGVubmVzc2VlMRYwFAYDVQQKDA1EZW5uaW5nZXIu
TmV0MRcwFQYDVQQDDA5EZW5uaW5nZXIgUm9vdDEiMCAGCSqGSIb3DQEJARYTYWRtaW5AZGVu
bmluZ2VyLm5ldDAeFw0yNDA1MDkyMTA4MDNaFw00NDA1MDQyMTA4MDNaMF0xCzAJBgNVBAYT
AlVTMRIwEAYDVQQIDAlUZW5uZXNzZWUxFjAUBgNVBAoMDURlbm5pbmdlci5uZXQxIjAgBgNV
BAMMGURlbm5pbmdlci5OZXQgU2lnbmluZyBJbnQwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw
ggEKAoIBAQDbR0tSiuLG5HPfo+cWtdeYQ8jc8Bjfuo0GTcNRT0glHnH1apUtInIktUknEZDH
ohahInN+mMBdKg54FCHOiYZrJbyxBIo9FwX7hRmOc+spxmSYWnOd2E/YcGInMK4ZpjPzldzB
Yt1n3zygkhx2bssxTJS3x4nv1qAXfLSZd1VwqoQufifEoPyTtymkkvHLv86vLgqAqooM/cXc
4LSIQ5u2uM308n42r8RkKtp7X1v9fJW8oRZN2XnFZtiUPH44YY2rHqyN2Hea9Y3+TXbldXjo
xhPHTA+JYVFq8KTmbQBqU7YcMhlIG0cSxPeFLMxnP6pqPcIVTAlK+a6YGRFppfjZAgMBAAGj
ggE2MIIBMjAdBgNVHQ4EFgQUH+VuxXhBxaJAQrvDekwkH91hBi4wgbMGA1UdIwSBqzCBqIAU
RFYC4p6L6KITnEvrpx2cyt+PcMmheqR4MHYxCzAJBgNVBAYTAlVTMRIwEAYDVQQIDAlUZW5u
ZXNzZWUxFjAUBgNVBAoMDURlbm5pbmdlci5OZXQxFzAVBgNVBAMMDkRlbm5pbmdlciBSb290
MSIwIAYJKoZIhvcNAQkBFhNhZG1pbkBkZW5uaW5nZXIubmV0ghQZE7NBItWtQsCouuwU6jZ+
HPPwnjAPBgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB/wQEAwIBBjA6BgNVHR8EMzAxMC+gLaAr
hilodHRwOi8vd3d3LmRlbm5pbmdlci5uZXQvcm9vdC1yZXZva2VkLmNybDANBgkqhkiG9w0B
AQsFAAOCAgEAfFbhPc82AfhyUqONs7IccYD36w+OP4nQgwfC4IWf3y/aQAZ2Zk6IITzYqwf7
PFM0bJRT3zi7xyetolqHDhfMJvnOQWpITZiyM/FSKwIvuBsy/uJUqPuqui4XQMYoSbAA1qmI
MW/z7VZZHwaRFoeWE40UirYcf0fNcooBZ72bmd+iBaVyjtZvky0Vgcz0eC6e6LR5kNb23yC6
TkyQIlGyQkK5/afXUYFzk49rOHVbVyxW3oXRfq8Ow6HCrpDGAS8p84S04MFwBVAUfbe4aXs3
bampaI2LzKgkVywyFP14LSvvdjCfLYfnLy1Z9hm2EHMqNHA2tCGdRhWp2d7aZC1MYFqng0ZS
fjPJjqHrI1qPU0p6k9A1GxAtrQlL2v/IUzUnMZkiawFV3qlxMGZf/kTYTUOcJhx1KU4zSLHu
80qO7ldRpp5gHssCAGFbeTu2gp6LxfmaFhLPDBJ1VGfdPx9lUrU/9OcoHczcl5x2Rb8IUZyX
9elzP5WdAU8p5R/DLlOAq24VcabhFtYBCA2dOESLupSfWKNQuJCN/1gz7ysSc+mjnnPV77IO
mpszJfkFFJEDNJlGIVKX1vwwygtC/9Ulox8frgbZlRAYAgDc/YbOBFxticVVre0Y3Ujx6Kzb
tkgZRlgfdZWbT1W5smncqJxg5qAL8e/yTb3fCe2nJ0jhiP4wggXpMIIE0aADAgECAhMAmNFt
CiCF3j+FwQLYtBTmGjzkMA0GCSqGSIb3DQEBCwUAMF0xCzAJBgNVBAYTAlVTMRIwEAYDVQQI
DAlUZW5uZXNzZWUxFjAUBgNVBAoMDURlbm5pbmdlci5uZXQxIjAgBgNVBAMMGURlbm5pbmdl
ci5OZXQgU2lnbmluZyBJbnQwHhcNMjQwNTEwMTkyNjU5WhcNMjkwNTA5MTkyNjU5WjBXMQsw
CQYDVQQGEwJVUzESMBAGA1UECAwJVGVubmVzc2VlMRcwFQYDVQQKDA5LYXJsIERlbm5pbmdl
cjEbMBkGA1UEAwwSa2FybEBkZW5uaW5nZXIubmV0MIICIjANBgkqhkiG9w0BAQEFAAOCAg8A
MIICCgKCAgEAvh1UssVbSYctzobPjwBkbjv/w4WvQNepeRTwE6+sLnXvc41+X9pa5EclPL4Q
l02Vu1m71mSqXGfK9HbWZoivbhefBHOoYb35MSc24PelhwcORbpneWoWc7giQ7QgFlvEe/yj
fs8M0H9fgdzFS5m2lwBQbis8kioSjHB2yt/8I1GE4Mvt1Cur9kga6ML5FAQvo8TYN1stdhrE
13FEv/BWCF4FVT4H2Wa2ySW+R1jkKb74SC6Twg98bGCRTShD5bVylh0+0LXNhzaopIDcI/KK
jm/j3mRjIlmqbGrSpvJsbjjhjhAYQKE1U8FB5TDU4OkFAibblhQit/KjgspPR2o/vOpVFPER
uhZEV1oDGzUJtZlkREIcN2sYBi0p7Y4585ya+b7L10mEenPlyi3eSkGXEuiy/BR2DY6lShwW
DPoQ5602TKmttCSwBdWGoLrQ4jEVEVNt4lku2wPbTHF3KpHJU0g7RbcWoUYn10SOxKathkir
hF3v9U32+QhPELGwqRrH0sL9rWf0qalRtPDHUYl8TebZmYkFqNeSMlqHijl5f4SsQPSj7gx5
4F19Ntm9ZcvuWTmW8QQGWTKHeMuG+BYkVIUSPe6/ZQsbD/xDx7rkyGfNgWIa4W7Wm/B7kaNq
H53tk3wFmNgZQOxMTPF0oTHfW0T2azU6JD0D1AlgoAnSAE0CAwEAAaOCAaYwggGiMDoGCCsG
AQUFBwEBBC4wLDAqBggrBgEFBQcwAYYeaHR0cDovL29jc3AuZGVubmluZ2VyLm5ldDo3Nzc3
MAwGA1UdEwEB/wQCMAAwDgYDVR0PAQH/BAQDAgXgMB0GA1UdJQQWMBQGCCsGAQUFBwMCBggr
BgEFBQcDBDAzBglghkgBhvhCAQ0EJhYkT3BlblNTTCBHZW5lcmF0ZWQgQ2xpZW50IENlcnRp
ZmljYXRlMB0GA1UdDgQWBBSxJZjVnlYLAT3uzvDYgc4742J6UTCBswYDVR0jBIGrMIGogBQf
5W7FeEHFokBCu8N6TCQf3WEGLqF6pHgwdjELMAkGA1UEBhMCVVMxEjAQBgNVBAgMCVRlbm5l
c3NlZTEWMBQGA1UECgwNRGVubmluZ2VyLk5ldDEXMBUGA1UEAwwORGVubmluZ2VyIFJvb3Qx
IjAgBgkqhkiG9w0BCQEWE2FkbWluQGRlbm5pbmdlci5uZXSCFFlTx0p0yd5EjVWrmS/STnhD
mtC4MB0GA1UdEQQWMBSBEmthcmxAZGVubmluZ2VyLm5ldDANBgkqhkiG9w0BAQsFAAOCAQEA
TrQ45/tBN3SiuqItFv/V+CF3h7Hxe0YLsL+A/P+q9ZhxIscaNjaclgQhPA+rUr+l8DGoXJ/w
yAl1E0SSBK+9phIc/9xFOBg3rCy4ngubzP+lHS1t03nMCBSUNsu5qPzqLBPiKaPabUu3Gr9o
koRezSszgM3/zNJfr8cMO93csCK/fBccsMx5q+3nxB5XeT7UciicjfEzUA4m2mQxBmGk9SSU
147Gy8UmdSq57Tw82KqUrQ1pJ6IOzVPLREpwlqGbHykSU3MwtPYPtfQeFVjvO/XcWvoFQjbV
UyhzAqMMYFudxoVLlJQiAgU38OScTLDgKxCO41h7VOjb2mss0zHndzGCBZUwggWRAgEBMHQw
XTELMAkGA1UEBhMCVVMxEjAQBgNVBAgMCVRlbm5lc3NlZTEWMBQGA1UECgwNRGVubmluZ2Vy
Lm5ldDEiMCAGA1UEAwwZRGVubmluZ2VyLk5ldCBTaWduaW5nIEludAITAJjRbQoghd4/hcEC
2LQU5ho85DANBglghkgBZQMEAgMFAKCCAvIwGAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAc
BgkqhkiG9w0BCQUxDxcNMjUwNjEyMTkxNDI5WjBPBgkqhkiG9w0BCQQxQgRAp0cNOuo61xcE
BouzmiMG5LrvFb1rfdJGBw59OEkx2ouzInVvkwgcZ6Q/LpmDfy4QLFbrjWS1rw1b2ntWXjGo
JzCBgwYJKwYBBAGCNxAEMXYwdDBdMQswCQYDVQQGEwJVUzESMBAGA1UECAwJVGVubmVzc2Vl
MRYwFAYDVQQKDA1EZW5uaW5nZXIubmV0MSIwIAYDVQQDDBlEZW5uaW5nZXIuTmV0IFNpZ25p
bmcgSW50AhMAmNFtCiCF3j+FwQLYtBTmGjzkMIGFBgsqhkiG9w0BCRACCzF2oHQwXTELMAkG
A1UEBhMCVVMxEjAQBgNVBAgMCVRlbm5lc3NlZTEWMBQGA1UECgwNRGVubmluZ2VyLm5ldDEi
MCAGA1UEAwwZRGVubmluZ2VyLk5ldCBTaWduaW5nIEludAITAJjRbQoghd4/hcEC2LQU5ho8
5DCCAVcGCSqGSIb3DQEJDzGCAUgwggFEMAsGCWCGSAFlAwQBKjALBglghkgBZQMEAQIwCgYI
KoZIhvcNAwcwDQYIKoZIhvcNAwICAQUwDQYIKoZIhvcNAwICAQUwBwYFKw4DAgcwDQYIKoZI
hvcNAwICAQUwBwYFKw4DAhowCwYJYIZIAWUDBAIBMAsGCWCGSAFlAwQCAjALBglghkgBZQME
AgMwCwYJYIZIAWUDBAIEMAsGCWCGSAFlAwQCBzALBglghkgBZQMEAggwCwYJYIZIAWUDBAIJ
MAsGCWCGSAFlAwQCCjALBgkqhkiG9w0BAQEwCwYJK4EFEIZIPwACMAgGBiuBBAELADAIBgYr
gQQBCwEwCAYGK4EEAQsCMAgGBiuBBAELAzALBgkrgQUQhkg/AAMwCAYGK4EEAQ4AMAgGBiuB
BAEOATAIBgYrgQQBDgIwCAYGK4EEAQ4DMA0GCSqGSIb3DQEBAQUABIICALur0SZhZpCKPQxU
YO6Tirskm6EerDDY+dtaWKLtgcReLoDK1JgGVe736mRqsEFvE8ArMVMmoEPHQbyIsg6jYHEN
XIi1omX7FFqlsd4fY1xyGc/fDhOs9hwMlZtN7448+CI5P4EMILinjRmdAvzZOdFF2aRmzz0y
bnGzQUMVzmixDtOZGxYAL9ptWhQG+vmYPRtWggC926iqDjduBleFX1jztQjI8pZnSnvxsWbq
mEjyAjFzm04/cKjM3p9sa0p/I0eUf2/Y8D63WZsXVRTVTmvezhBnlORx8/7P2LD37gmZsuR2
iaDH0NRLZD49WaIITr8GAzg8Rl597pX8036hTa5otpszlaKHYeInmXZ3vMsQyHbSgqwTdjsh
bJNLRGk0iG6zyxZI2kyXLE9p+BM4pKULo+eD5E6rtjPMbh9TzGvWixBMo8VQQSwtsRQ0fj/y
oTclEz3dxklE6zxEIvo6jRlVdRJG/laVcQbJ7+jb4b1Bc7G0eXdPGIoCK4LaIauRp4N69Lrs
Ol1rL4Ov7HkNh2JIREvwjZwEzECr/7rDiPWXZAp8cGS4iTd6+w4ONOZzVJAX6Xu9EjqBz2NO
MbIbdHKyGCKrIZRs2DffGhdLCC+Cyb0QYZZ2LBWk3drPnI4OjeiuEQXJfAeg86TzKe/Msq4L
dsHpOV+9NLGlcBEbF6JKAAAAAAAA
--------------ms060103010105020003090105--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?d44a2a6c-67c1-4dfb-b281-dbcfdd1a4971>