Date: Fri, 30 May 2014 12:04:45 -0700 From: Mike Carlson <mike@bayphoto.com> To: freebsd-fs@FreeBSD.org Subject: ZFS Kernel Panic on 10.0-RELEASE Message-ID: <5388D64D.4030400@bayphoto.com>
next in thread | raw e-mail | index | archive | help
This is a cryptographically signed message in MIME format. --------------ms070200060301070101080704 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: quoted-printable Hey FS@ Over the weekend, we had upgraded one of our servers from 9.1-RELEASE to = 10.0-RELEASE, and then the zpool was upgraded (from 28 to 5000) Tuesday afternoon, the server suddenly rebooted (kernel panic), and as=20 soon as it tried to remount all of its ZFS volumes, it panic'd again. We have a zfs on root install, the total raidz is around 9TB. The volume it is panic-ing on is zroot/data/working, which has (or had=20 at this point) 4TB of data. I can boot off of the 10.0-RELEASE usb=20 image, and mount: zroot zroot/usr zroot/var zroot/data Just not the large volume that had 4TB of data. I've set the volume to readonly, and that still causes a panic upon=20 mount, I was able to snapshot the problematic volume, and even do a send = | receive, but that panics when the transfer is nearly complete (4.64TB=20 out of 4.639TB) Now, that data is not super critical, its basically scratch storage=20 where archives are extracted and shuffled around, and then moved off to=20 another location. We just like to keep about 60 days worth in case we=20 need to re-process something. The more important issue is, why did this happen, and what can we=20 recover from situations like this. It looks pretty bad when any data is=20 lost. See url's for the kernel panic message (used my phone to take a picture) https://drive.google.com/file/d/0B0i2JyKe_ya2RnRYT3A1Qk5ldkk/edit?usp=3Ds= haring https://drive.google.com/file/d/0B0i2JyKe_ya2YWNlbVl3MVFlTGc/edit?usp=3Ds= haring Here is the zpool history for that storage: History for 'zroot': 2013-11-19.11:31:37 zpool create -o altroot=3D/mnt -o cachefile=3D/var/tmp/zpool.cache -f zroot raidz2 /dev/gpt/disk0 /dev/gpt/disk1 /dev/gpt/disk2 /dev/gpt/disk3 /dev/gpt/disk4 /dev/gpt/disk5 /dev/gpt/disk6 /dev/gpt/disk7 /dev/gpt/disk8 /dev/gpt/disk9 /dev/gpt/disk11 spare /dev/gpt/disk12 2013-11-19.11:31:37 zpool export zroot 2013-11-19.11:31:38 zpool import -o altroot=3D/mnt -o cachefile=3D/var/tmp/zpool.cache zroot 2013-11-19.11:31:38 zpool set bootfs=3Dzroot zroot 2013-11-19.11:31:43 zfs set checksum=3Dfletcher4 zroot 2013-11-19.11:34:11 zfs create zroot/usr 2013-11-19.11:34:11 zfs create zroot/home 2013-11-19.11:34:11 zfs create zroot/var 2013-11-19.11:34:11 zfs create zroot/data 2013-11-19.11:34:11 zfs create -o compression=3Don -o exec=3Don -o setuid=3Doff zroot/tmp 2013-11-19.11:34:11 zfs create -o compression=3Dlzjb -o setuid=3Doff zroot/usr/ports 2013-11-19.11:34:11 zfs create -o compression=3Doff -o exec=3Doff -o setuid=3Doff zroot/usr/ports/distfiles 2013-11-19.11:34:11 zfs create -o compression=3Doff -o exec=3Doff -o setuid=3Doff zroot/usr/ports/packages 2013-11-19.11:34:11 zfs create -o compression=3Dlzjb -o exec=3Doff -o= setuid=3Doff zroot/usr/src 2013-11-19.11:34:11 zfs create -o compression=3Dlzjb -o exec=3Doff -o= setuid=3Doff zroot/var/crash 2013-11-19.11:34:11 zfs create -o exec=3Doff -o setuid=3Doff zroot/va= r/db 2013-11-19.11:34:11 zfs create -o compression=3Dlzjb -o exec=3Don -o setuid=3Doff zroot/var/db/pkg 2013-11-19.11:34:11 zfs create -o exec=3Doff -o setuid=3Doff zroot/va= r/empty 2013-11-19.11:34:11 zfs create -o compression=3Dlzjb -o exec=3Doff -o= setuid=3Doff zroot/var/log 2013-11-19.11:34:11 zfs create -o compression=3Dgzip -o exec=3Doff -o= setuid=3Doff zroot/var/mail 2013-11-19.11:34:11 zfs create -o exec=3Doff -o setuid=3Doff zroot/va= r/run 2013-11-19.11:34:11 zfs create -o compression=3Dlzjb -o exec=3Don -o setuid=3Doff zroot/var/tmp 2013-11-19.11:34:11 zfs create -V 4G zroot/swap 2013-11-19.11:34:11 zfs set org.freebsd:swap=3Don zroot/swap 2013-11-19.11:34:11 zfs set checksum=3Doff zroot/swap 2013-11-19.11:43:24 zfs set readonly=3Don zroot/var/empty 2013-11-19.11:43:40 zfs set mountpoint=3Dlegacy zroot 2013-11-19.11:43:50 zfs set mountpoint=3D/tmp zroot/tmp 2013-11-19.11:43:58 zfs set mountpoint=3D/usr zroot/usr 2013-11-19.11:44:05 zfs set mountpoint=3D/var zroot/var 2013-11-19.11:44:12 zfs set mountpoint=3D/home zroot/home 2013-11-19.11:44:18 zfs set mountpoint=3D/data zroot/data 2013-11-19.20:11:53 zfs create zroot/data/working 2013-11-19.20:17:59 zpool scrub zroot 2013-11-19.21:21:23 zfs set aclmode=3Dpassthrough zroot/data 2013-11-19.21:21:33 zfs set aclinherit=3Dpassthrough zroot/data 2013-11-21.00:58:57 zfs set compression=3Dlzjb zroot/data/working 2014-05-24.14:24:40 zfs set readonly=3Doff zroot/var/empty 2014-05-24.15:37:15 zpool upgrade zroot 2014-05-27.15:32:41 zfs set mountpoint=3D/mnt zroot 2014-05-27.15:33:55 zfs set mountpoint=3D/mnt/tmp zroot/tmp 2014-05-27.15:34:03 zfs set mountpoint=3D/mnt/var zroot/var 2014-05-27.15:34:13 zfs set mountpoint=3D/mnt/crash zroot/var/crash 2014-05-27.15:34:22 zfs set mountpoint=3D/mnt/db zroot/var/db 2014-05-27.15:34:35 zfs set mountpoint=3D/mnt/db/pkg zroot/var/db/pkg= 2014-05-27.15:34:47 zfs set mountpoint=3D/mnt/db/empty zroot/var/empt= y 2014-05-27.15:35:22 zfs set mountpoint=3D/mnt/var/db zroot/var/db 2014-05-27.15:35:29 zfs set mountpoint=3D/mnt/var/db/pkg zroot/var/db= /pkg 2014-05-27.15:35:38 zfs set mountpoint=3D/mnt/var/empty zroot/var/emp= ty 2014-05-27.15:35:45 zfs set mountpoint=3D/mnt/var/log zroot/var/log 2014-05-27.15:35:54 zfs set mountpoint=3D/mnt/var/mail zroot/var/mail= 2014-05-27.15:36:02 zfs set mountpoint=3D/mnt/var/run zroot/var/run 2014-05-27.15:36:09 zfs set mountpoint=3D/mnt/var/tmp zroot/var/tmp 2014-05-27.15:36:34 zfs set mountpoint=3D/mnt/usr zroot/usr 2014-05-27.15:36:40 zfs set mountpoint=3D/mnt/usr/ports zroot/usr/por= ts 2014-05-27.15:36:54 zfs set mountpoint=3D/mnt/usr/distfiles zroot/usr/ports/distfiles 2014-05-27.15:37:12 zfs set mountpoint=3D/mnt/usr/ports/packages zroot/usr/ports/packages 2014-05-27.15:37:20 zfs set mountpoint=3D/mnt/usr/ports/distfiles zroot/usr/ports/distfiles 2014-05-27.15:37:35 zfs set mountpoint=3D/mnt/usr/src zroot/usr/src 2014-05-27.15:37:53 zfs set mountpoint=3D/mnt/home zroot/home 2014-05-27.15:38:39 zfs set mountpoint=3D/mnt/data zroot/data 2014-05-27.15:38:47 zfs set mountpoint=3D/mnt/data/working zroot/data/working 2014-05-27.15:57:53 zpool scrub zroot 2014-05-28.09:34:16 zfs snapshot zroot/data/working@1 2014-05-28.18:55:12 zfs set readonly=3Don zroot/data/working The full zfs attributes for that particular volume: NAME PROPERTY VALUE SOURCE zroot/data/working type filesystem - zroot/data/working creation Tue Nov 19 20:11 2013 - zroot/data/working used 4.64T - zroot/data/working available 2.49T - zroot/data/working referenced 4.64T - zroot/data/working compressratio 1.00x - zroot/data/working mounted no - zroot/data/working quota none default zroot/data/working reservation none default zroot/data/working recordsize 128K default zroot/data/working mountpoint /mnt/data/working local zroot/data/working sharenfs off default zroot/data/working checksum fletcher4 inherited from zr= oot zroot/data/working compression lzjb local zroot/data/working atime on default zroot/data/working devices on default zroot/data/working exec on default zroot/data/working setuid on default zroot/data/working readonly on local zroot/data/working jailed off default zroot/data/working snapdir hidden default zroot/data/working aclmode passthrough inherited from zroot/data zroot/data/working aclinherit passthrough inherited from zroot/data zroot/data/working canmount on default zroot/data/working xattr on default zroot/data/working copies 1 default zroot/data/working version 5 - zroot/data/working utf8only off - zroot/data/working normalization none - zroot/data/working casesensitivity sensitive - zroot/data/working vscan off default zroot/data/working nbmand off default zroot/data/working sharesmb off default zroot/data/working refquota none default zroot/data/working refreservation none default zroot/data/working primarycache all default zroot/data/working secondarycache all default zroot/data/working usedbysnapshots 0 - zroot/data/working usedbydataset 4.64T - zroot/data/working usedbychildren 0 - zroot/data/working usedbyrefreservation 0 - zroot/data/working logbias latency default zroot/data/working dedup off default zroot/data/working mlslabel - zroot/data/working sync standard default zroot/data/working refcompressratio 1.00x - zroot/data/working written 0 - zroot/data/working logicalused 4.65T - zroot/data/working logicalreferenced 4.65T - Was this a zfs 10.0-RELEASE issue? Or, did our Dell PERC H710 controller = just happen to become an issue and the timing is coincidental? Any pointers on either restoring the data or preventing this in the=20 future would be great Mike C --------------ms070200060301070101080704 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIITIjCC BhwwggQEoAMCAQICCGWP9DPVdj3ZMA0GCSqGSIb3DQEBCwUAMEsxDzANBgNVBAMMBlJvb3RD QTEWMBQGA1UECgwNQmF5IFBob3RvIExhYjETMBEGA1UECAwKQ2FsaWZvcm5pYTELMAkGA1UE BhMCVVMwHhcNMTIxMDIzMTczMjE4WhcNMjcxMDIzMTczMjE4WjBYMRwwGgYDVQQDDBNCYXkg UGhvdG8gUGVvcGxlIENBMRYwFAYDVQQKDA1CYXkgUGhvdG8gTGFiMRMwEQYDVQQIDApDYWxp Zm9ybmlhMQswCQYDVQQGEwJVUzCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBALuW vZE7FFSO77jEhdT/dZzu0XnPS35aepuvMvCNTagnNMbaf+AKrLZFaVRqBt4f6yl5TDXNEAMI +CJrg7T9dsoGN1Vygm4bIFzOIRtTZ1A7seemeonyqMto8gYMPhEFicuKavEAFcCFprVcVu5v yXbEG+tYkx08oUzeZ9LWZnh4a0wxAucbH0PcZFmdXFMbgjt6GSiR8jVUT1t/Dik1+mJ1A4K3 3VyM+G1CaqGIKg2UbvHiVQVoJrpgUc6vO5aWWgDreH+CAKLpyJwCj4BGphfHBPENu5LUp0CL fa4pA4r+OH0HNBp/I7utZMrIevh3pya34lDJXj0CQWSuVBR9Kp80IHHYGuyPU/sMXkUplsyI Y4KgQSRYRFNdWoboLwsRu1/2iDUaTR+XYH7Tu5zvCOedUm8ns/wfE7TW6kax5nR36VzuZS6B kUcuETO1QMxt2SJcHZIftiyTuxC8Y9Z7J6igR39pZPWGywJ2+oPRKFQD/u9RWZqjh9k5euVi 4bVwwog5DqnsymMTuKb0I1nCs5js+rfEVo+mc4gAfEmzk/dmpwkOnGV3oTfASbGtwMIlAEe5 HHJhdTA3jpVo3eQAZi479xZ7SozKvnjt1y9SMQ4uTJXjER/IVH3YC9S/BSFrqWLLDfWEGW84 SBn+ogldDIy3PX3SU7nOiN4Vse2Jg+r7AgMBAAGjgfYwgfMwHQYDVR0OBBYEFC34wnK9ZmLm B7vV52IeLHaapJ33MA8GA1UdEwEB/wQFMAMBAf8wHwYDVR0jBBgwFoAU00bJTnGPaRmhJAUZ t4x4J3soVyswgY8GA1UdHwSBhzCBhDCBgaB/oH2Ge2h0dHA6Ly9iYXljYS5iYXlob3RvLmxv Y2FsL2VqYmNhL3B1YmxpY3dlYi93ZWJkaXN0L2NlcnRkaXN0P2NtZD1jcmwmaXNzdWVyPUNO PVJvb3RDQSxPPUJheSUyMFBob3RvJTIwTGFiLFNUPUNhbGlmb3JuaWEsQz1VUzAOBgNVHQ8B Af8EBAMCAYYwDQYJKoZIhvcNAQELBQADggIBAEoDTeb1wVXz2VqFPrc3uaRnbVt6DCB9uBft L8YRgi5+FF5KHjvTqYHGiS38yVFfXM6MaN6yI7WSwtS+wVhM7Tdw6miwKNBApmCDKzjorNVX iZUmv88fjGm652LeIfT/oFGJg2oruoad9OfIofjT/jFpiFT7KOgjg4xeKCC0f+9naVo5uWOJ HTyFm1Kq6ryT+g7mZb7kLvndmJRW0aybn8Dls7/bhThEaOYkwn4dbZ6q0W2I5ZWg9bbh66V+ J9P1XObZA/TkRVTlpSYJ1hAgYX1yTbL360vuTAAwdSWQSFm4TAoXC4BsPdlgy86lDjNrW9id WX0RJCBzk/FzHhI4Aj9+SVhLl/Vkf9nbz8VhPPK2Az0PPYKy5ARtgOqGTKZbgVKut9Kgc/fn vUgPhObHQsD1Ui+NKsIGYBFKBhNmVXqMQSn4JzC9x0oDEmv37UGrcut2cCP3ZS17p137VaUM lQ0RWomju+sPCPFgyCPa/TLPoMZ2334uIxkRbDefvOoXIosORMQ9Jh50XqktUesbhuBfH9Q5 8h8bTWm1Cn/LxXW9qdSSbnta0OAH1G4hwVUlcusSM0o7Ude8tszw6kRpmEDDE8BQjE5nXSY7 wOf1eXfQfDkVQiouTD3l5ElqB98tnCnL/y9dphWoBLmiJwgb/4yWZ/Zewc1V65UFr7LmvcQM MIIGIzCCBAugAwIBAgIIYpSXgZOT7j0wDQYJKoZIhvcNAQELBQAwWDEcMBoGA1UEAwwTQmF5 IFBob3RvIFBlb3BsZSBDQTEWMBQGA1UECgwNQmF5IFBob3RvIExhYjETMBEGA1UECAwKQ2Fs aWZvcm5pYTELMAkGA1UEBhMCVVMwHhcNMTIxMDIzMTgwMDAzWhcNMTQxMDIzMTgwMDAzWjBg MRUwEwYKCZImiZPyLGQBAQwFMTMwNjkxFTATBgNVBAMMDE1pa2UgQ2FybHNvbjELMAkGA1UE CwwCSVQxFjAUBgNVBAoMDUJheSBQaG90byBMYWIxCzAJBgNVBAYTAlVTMIICIjANBgkqhkiG 9w0BAQEFAAOCAg8AMIICCgKCAgEAoTzIvF66A3wYPWXQFzmjBUuIrHSQGhOcE7rb0KQtar+H rkmHX56bCAklW5P/pd+yJ08lMwb3CxbTOz0q47JuBv2kKJO1xCgCua26Uvz3VAmfirmWwpXq zZBDqy/bEIt/XFfiVUC4jriGSEPrtx9q9nJJsb2JVRgtsbcHaaJFu8u8s8p8cLbcYdKobS00 g6+7it2IpIJhxc5tEMa1Yku3kCQiHVVFa9b4H5pFDHpkCrKZ43cuCneiR5kgr47z/3U66kLt J7Q5IT/i7nThjGQMa/f1JSWet8yeTomKvqkuEAA4o/IWQzEbtxzeps6vWxaCDULjEq69s//S 6PtqiQSmG9ZGFoPYD8/GGd4CMBqgjKopintD5sGTlJ851yZwl9VY/hRuxInp8gTjWrt1gQIB zlgSgSKnKTN6f+e85XMPU1y/wVz8RJWl8Tr11kzo6vrM14+ruNUxo1Ea3PJ9MUcWenoRKGSU I/IP94kZVjPkZlJv6tTF0Yi2Gclet/ZDu8vgvkxmUZYdQMGGlgZTCAsvHr37/ov6g51Tf+im 7410EsdYCmSINRGzWQAzlH9NscsW1TAd1Znog1H6NRDExY3ksjvFcKYOjUmkyWT8Vl2oJmT2 IzI23/C3esGL9OZzZ6K84MRNrH1y/yNp75vQnP3JfDMpbb5kkDp95Bu365qBluECAwEAAaOB 6DCB5TBSBggrBgEFBQcBAQRGMEQwQgYIKwYBBQUHMAGGNmh0dHA6Ly9iYXljYS5iYXlob3Rv LmxvY2FsL2VqYmNhL3B1YmxpY3dlYi9zdGF0dXMvb2NzcDAdBgNVHQ4EFgQUzeso+31hmtp3 soKHShXXtAEo+iMwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBQt+MJyvWZi5ge71ediHix2 mqSd9zAOBgNVHQ8BAf8EBAMCBDAwEwYDVR0lBAwwCgYIKwYBBQUHAwQwHAYDVR0RBBUwE4ER bWlrZUBiYXlwaG90by5jb20wDQYJKoZIhvcNAQELBQADggIBADnQfCasVgMsKsxIZAOZCbsU xo9BfsbpoM02p2aP+vPNDLXYRmcnH6ReeeUKSfIn0HmS9XkeHizMEXaC5sV9g4dasdQJQOGU mDcBnlxGn5fzNVFBM7/RHL83waYq4MCeyP9M7lSiNFZTrnSLVL9lIO0FLrBE06c9bn09kExc zkXI6Qm+e/MNrnoC3vw3GbH3a7tZCPsQcyNSok99jgPTRb1g9uVPg25M+ScYMU0wv2BE24u1 Dfzwcq52h64TllbzdVg/qOQH1HM96wmU+CtPuzA6eYnWRao/80LfQcyhNZ/jfMB/9xwFwsam o3Bw7SrSPEatw/tMyEEVMzas5/wZm2uMtab7642d5mr5OWLVPYgmKUscSlNt87vKkFhvn0Cz Z7O8O79WNMJA0sx1aomn1/ZrWDkd8X/ACUC2Fa3cV4AAzmjytiNu7r2z+GwdXPmvWSlBDXKX wLSoRkdq5hmYAP3GwXF0dsZo63WJLuCU1bPyERNLKdZM//eX832WgomPs4FA4xg0MUH0S7vJ eo7K1cTutZEmyLT623p0GcOINs2ir/ZqPTDLKszI7ytAltYaATt4kYUXbmMGGYItDf1X/caj DoLv2hjBTM5HORZYABC/Kfo9iL4KeYDqAvblJc7qyw+QXdHOUbwc9gQXQJvlQlfjDYvJLKme zoZ1sMzRBOl0MIIG1zCCBL+gAwIBAgIIGE38aUOyx8EwDQYJKoZIhvcNAQELBQAwWDEcMBoG A1UEAwwTQmF5IFBob3RvIFBlb3BsZSBDQTEWMBQGA1UECgwNQmF5IFBob3RvIExhYjETMBEG A1UECAwKQ2FsaWZvcm5pYTELMAkGA1UEBhMCVVMwHhcNMTIxMDIzMTc1NzQ1WhcNMTQxMDIz MTc1NzQ1WjBgMRUwEwYKCZImiZPyLGQBAQwFMTMwNjkxFTATBgNVBAMMDE1pa2UgQ2FybHNv bjELMAkGA1UECwwCSVQxFjAUBgNVBAoMDUJheSBQaG90byBMYWIxCzAJBgNVBAYTAlVTMIIC IjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAuECpC3YUm7GV0xz/DMmMQZ3EvMfvVhFB 77TcVxY1OoBlp2jk0ST0Hel/vp9uPhhi/eAlH89rC9fhNwORUHfeofWhoT3ZXrnjisNDQnb5 MgBV6wVM58PrikwU13FuNrIrPuUeuUE659BQhfgj2j2Pv9GVgBib6NBbHJAIOFo8H3wmd4b9 Yv6RKM145qSJCrJp96jSkoQSwni+jraHGMs55kgFP/3f0X6RpY7GhvItFI95Xauf7R5qTuW0 oamhvjFnlWVBoMuHd5yqzUgL7gQ0wfB31wfBP2ghFPAv715Qv9DYleFmgWE1LRkrt8clCzzk huj00CrEL+rSK9bDYNpa5AdCQ8aA/bG2x24ApOBvfDYnem+Ytu9lIY7qKZiP+9pASOiXsSSt g8CH9HoG+9GpK/rDyQ2nuNxTWEQEetwofkLdrGU/VgpcwIZqETsugu+l/FCu3Ogslezpiolb SqjCF5CU3aZravNZo4+HDriFKm7jyKEb+zUcXcwNH2iwbGt6uTMILslXbWXuhi8Lu3SKyacj 1gn9OMPQTCUKxaiTJXqIcJP0CV9wKcgTn7vanChDgz2fTVm6HOzb2RZlyhYzU4ofuMk+VGbg tBSDns+B9D1ACZMd58D2XdqRvmGPlnbJEZPkAZyDiR4mp/Aw09uXzOCIHzvqLhiysGon7lkV a18CAwEAAaOCAZswggGXMFIGCCsGAQUFBwEBBEYwRDBCBggrBgEFBQcwAYY2aHR0cDovL2Jh eWNhLmJheWhvdG8ubG9jYWwvZWpiY2EvcHVibGljd2ViL3N0YXR1cy9vY3NwMB0GA1UdDgQW BBRGT4gPK1JkpmK4YAsQgD82sskTGDAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFC34wnK9 ZmLmB7vV52IeLHaapJ33MIGlBgNVHR8EgZ0wgZowgZeggZSggZGGgY5odHRwOi8vYmF5Y2Eu YmF5aG90by5sb2NhbC9lamJjYS9wdWJsaWN3ZWIvd2ViZGlzdC9jZXJ0ZGlzdD9jbWQ9Y3Js Jmlzc3Vlcj1DTj1CYXklMjBQaG90byUyMFBlb3BsZSUyMENBLE89QmF5JTIwUGhvdG8lMjBM YWIsU1Q9Q2FsaWZvcm5pYSxDPVVTMA4GA1UdDwEB/wQEAwIGwDAdBgNVHSUEFjAUBggrBgEF BQcDAgYIKwYBBQUHAwQwHAYDVR0RBBUwE4ERbWlrZUBiYXlwaG90by5jb20wDQYJKoZIhvcN AQELBQADggIBAC+1BfV1bmcHZtgM4HN5rZek1vzY2UBLTHevnLDsLmNNASYMuOrm1TYMCI0/ LVkGltk0kq3lAP3hECsrSeH+SlmoRAmSCEOAwqOBU5oTqV/SMiQcZb6K2o/QUAVV6Si2KMyW U35hnqi9/vVNq4y28RjvMAydcrJpmn5qrWsy1ZLbW+Zu3jneBYjUcm4mQnqaCyjRyE2Z3byi /4m1sxISSdyqoRsWKsixSdb4bXUdNeNscltRYDMEDODtzYjnO7WebHtaMAEOyfE3n2gkPmGr dBopA6/Z53GVkG+MXF1wSlc3g38dHSpbY6qJiMkl8/wKCnkcMRPC3hRGQikYprh/oAbpHAyS q9P0jv/3cB0yzZ5b3n49PxqjwyH93lcRZJQ5pAMMWFk1B7IuYk9LVbJEVsPAmxFbWoS87Tn0 /zi6lkUNXoSgm4fJ+NBY5jnk1m48nPFIs7dpr8wW8qT2C0DiQz/tSKMramzsBduXY67djCZ5 cam1H5pRuaiEPEmrafC91i+wCsmjpMoqykKkuiGOlmY8LhRSZZ7M8qwtpo/0yur94z1Z5gyG 3yo/vZQtNDt8jaJ2ajGR50C48CtJufrckT3o3UrtjvQ3jCUnak2VbXIV6QNTio9NQKVH4+fL VhoGvc/PC/CifBs6Qyfk3a6nHdVfzwGx8Uytj/Br0AV0p8Y2MYIERjCCBEICAQEwZDBYMRww GgYDVQQDDBNCYXkgUGhvdG8gUGVvcGxlIENBMRYwFAYDVQQKDA1CYXkgUGhvdG8gTGFiMRMw EQYDVQQIDApDYWxpZm9ybmlhMQswCQYDVQQGEwJVUwIIGE38aUOyx8EwCQYFKw4DAhoFAKCC AbcwGAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTQwNTMwMTkw NDQ1WjAjBgkqhkiG9w0BCQQxFgQUSO2i2KfMX9DonqJuEdiTuurfdGswbAYJKoZIhvcNAQkP MV8wXTALBglghkgBZQMEASowCwYJYIZIAWUDBAECMAoGCCqGSIb3DQMHMA4GCCqGSIb3DQMC AgIAgDANBggqhkiG9w0DAgIBQDAHBgUrDgMCBzANBggqhkiG9w0DAgIBKDBzBgkrBgEEAYI3 EAQxZjBkMFgxHDAaBgNVBAMME0JheSBQaG90byBQZW9wbGUgQ0ExFjAUBgNVBAoMDUJheSBQ aG90byBMYWIxEzARBgNVBAgMCkNhbGlmb3JuaWExCzAJBgNVBAYTAlVTAghilJeBk5PuPTB1 BgsqhkiG9w0BCRACCzFmoGQwWDEcMBoGA1UEAwwTQmF5IFBob3RvIFBlb3BsZSBDQTEWMBQG A1UECgwNQmF5IFBob3RvIExhYjETMBEGA1UECAwKQ2FsaWZvcm5pYTELMAkGA1UEBhMCVVMC CGKUl4GTk+49MA0GCSqGSIb3DQEBAQUABIICACIkDSeYND7Z5p3UjJHSujTaXPZiCEoYYpFu q7Y6o+bk6GNxnopwdwwyYkhJH5gqjKoRn+Dge7Nn0a3qYSjZ4AIE6MW7iTZmAGjdzB9cJy8L f6yNnIkIPIT04lF7ifdFA7hRbgj+Df+pmCeSaoB0zIKFOFlPXYzbjyuhidNgAJH+OwG1XFPZ uxNCK5hZFADHitQapdSRj8SLxj5qP4kKsoVcFdK29XBQ1Y0rM/3htMea83UcDjJJICBP6975 QSTbkLtZmQvFnHNTqRqcsWSvoa1woLDG/L407cs+49qCBCnzsoOt/VfIgffXshypWw5ThtKr 8ywBOF6PxWUhibo+ZDTKrn3unwbvqGgLmi8BwJi6Fgji8C1ZTEKge8e+LKo8L+mBO75dsJKX 7auvKf4Fs9afswC0eWYIhy9UnAVe7pmnQPjay+43xTcWrnl0DRKAAWIvtedLP0XTbhMEGYwW kieVJYG9RckzYBgFDJdLNWd0OyRKDCiT1iP0MpK0JyFBOBBKzBsPX1xs5we1TTEbGEI5idVW 8t0/NvfuLwd5oqm+kCpxAOJ+9YgMIwXJQTabuP326eRdonOrtgQnyFmh9jxBAEcKmAVaST36 kdy9VqIgENLCu1+xLjT0Ip2udQqPsdKWQiPtJCL2c0T+PdsFa6f4tC/iKJFO2t7yeeSCZJ5M AAAAAAAA --------------ms070200060301070101080704--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5388D64D.4030400>