Date: Wed, 04 Jun 2014 11:46:54 -0700 From: Mike Carlson <mike@bayphoto.com> To: freebsd-fs@freebsd.org Subject: Re: ZFS Kernel Panic on 10.0-RELEASE Message-ID: <538F699E.4060802@bayphoto.com> In-Reply-To: <538DF082.3030407@bayphoto.com> References: <5388D64D.4030400@bayphoto.com> <EC2EA442-56FC-46B4-A1E2-97523029B7B3@mail.turbofuzz.com> <5388E5B4.3030002@bayphoto.com> <538BBEB7.4070008@bayphoto.com> <782C34792E95484DBA631A96FE3BEF20@multiplay.co.uk> <538C9CF3.6070208@bayphoto.com> <16ADD4D9DC73403C9669D8F34FDBD316@multiplay.co.uk> <538CB3EA.9010807@bayphoto.com> <6C6FB182781541CEBF627998B73B1DB4@multiplay.co.uk> <538CC16A.6060207@bayphoto.com> <F959477921CD4552A94BF932A55961F4@multiplay.co.uk> <538CDB7F.2060408@bayphoto.com> <88B3A7562A5F4F9B9EEF0E83BCAD2FB0@multiplay.co.uk> <538CE2B3.8090008@bayphoto.com> <85184EB23AA84607A360E601D03E1741@multiplay.co.uk> <538D0174.6000906@bayphoto.com> <F445995D86AA44FB8497296E0D41AC8F@multiplay.co.uk> <538D18CB.5020906@bayphoto.com> <538D1CD5.5070902@bayphoto.com> <C7A3A403F308403B99A4BD39D48521D8@multiplay.co.uk> <538DF082.3030407@bayphoto.com>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a cryptographically signed message in MIME format. --------------ms090905050307090206000305 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: quoted-printable Top-posting... sorry I'm going to have to roll this particular server back into production,=20 so I'll be rebuilding it from scratch That is okay with this particular system, the other server that=20 exhibited the same issue will have to have all 19TB of its usable data=20 streamed off to temp storage (if we can get it) and rebuilt as well. Thank you Steve for being so helpful, and patient with me stumbling=20 through kgdb :) I have some lingering questions about the entire situation: First, these servers perform regular zpool scrubs (once a month), and=20 have ECC memory. According the the additional logging information I was=20 able to get from Steve's patch, it seems that even with these safeguards = data was still corrupted. A scub after the initial panic did not report=20 any errors. Second, these two servers had an extra anomaly, and that was the missing = zpool.cache. I say missing, because zdb was unable to access the zpool,=20 it was not until I ran "zpool set cachefile=3D/boot/zfs/zpool.cache=20 <pool>". This was previously not an issue. The two servers were upgraded fro 9.1 to 10 on the same morning, within=20 minutes of each other. That is about it as far as commonalities. Both=20 have different drive types (900GB SAS vs 2TB SATA), different=20 controllers (Dell PERC (mfi) vs LSI (mps)), Dell vs SuperMicro boards... We do use the aio kernel module, and as well as some sysctl and=20 loader.conf tuning. I've backed all of those out, so we're just running=20 a stock OS. Ideally, I would like to never run into this situation again. However, I = don't have any evidence to point to an upgrade misstep or some=20 catastrophic configuration error (kernel parameters, zpool create). Thank everyone, Mike C On 6/3/2014 8:57 AM, Mike Carlson wrote: > On 6/3/2014 2:24 AM, Steven Hartland wrote: >> >> ----- Original Message ----- From: "Mike Carlson" <mike@bayphoto.com> >> >>> Scratch that last one, the cachefile had to be reset on the pool to=20 >>> /boot/zfs/zpool.cache >>> >>> So I'm running it now, and its taking so long to traverse all=20 >>> blocks, that it is telling me its going to take around 5400 HOURS >>> >>> I guess I'll report back 90 days? >> >> Try with just the following should be quicker: >> zdb -uuuC zroot >> >> Regards >> Steve >> > > zdb -uuumcD eventually segfaulted: > > Uberblock: > magic =3D 0000000000bab10c > version =3D 5000 > txg =3D 3378596 > guid_sum =3D 1996697515446579069 > timestamp =3D 1401756315 UTC =3D Mon Jun 2 17:45:15 2014 > rootbp =3D DVA[0]=3D<0:3f08b7fd000:600> > DVA[1]=3D<0:5500f66f000:600> DVA[2]=3D<0:86001fb9c00:600> [L0 DMU > objset] fletcher4 lzjb LE contiguous unique triple size=3D800L/200P > birth=3D3378596L/3378596P fill=3D326 > cksum=3D10553d553d:65de1705c49:1445be46ea217:2bed6cb4bc5e02 > > All DDTs are empty > > Metaslabs: > vdev 0 > metaslabs 143 offset spacemap free > --------------- ------------------- --------------- =20 > ------------- > metaslab 0 offset 0 spacemap 34 =20 > free 12.8G > metaslab 1 offset 1000000000 spacemap 162 =20 > free 21.0G > metaslab 2 offset 2000000000 spacemap 170 =20 > free 4.20G > metaslab 3 offset 3000000000 spacemap 182 =20 > free 26.8G > metaslab 4 offset 4000000000 spacemap 183 =20 > free 18.7G > metaslab 5 offset 5000000000 spacemap 184 =20 > free 27.9G > metaslab 6 offset 6000000000 spacemap 185 =20 > free 19.9G > metaslab 7 offset 7000000000 spacemap 187 =20 > free 30.8G > metaslab 8 offset 8000000000 spacemap 188 =20 > free 24.4G > metaslab 9 offset 9000000000 spacemap 189 =20 > free 2.73G > metaslab 10 offset a000000000 spacemap 190 =20 > free 17.4G > metaslab 11 offset b000000000 spacemap 193 =20 > free 20.5G > metaslab 12 offset c000000000 spacemap 194 =20 > free 10.0G > metaslab 13 offset d000000000 spacemap 195 =20 > free 15.0G > metaslab 14 offset e000000000 spacemap 196 =20 > free 19.8G > metaslab 15 offset f000000000 spacemap 197 =20 > free 22.6G > metaslab 16 offset 10000000000 spacemap 198 =20 > free 11.8G > metaslab 17 offset 11000000000 spacemap 199 =20 > free 18.3G > metaslab 18 offset 12000000000 spacemap 200 =20 > free 3.35G > metaslab 19 offset 13000000000 spacemap 201 =20 > free 24.2G > metaslab 20 offset 14000000000 spacemap 202 =20 > free 9.8G > metaslab 21 offset 15000000000 spacemap 205 =20 > free 16.1G > metaslab 22 offset 16000000000 spacemap 206 =20 > free 31.4G > metaslab 23 offset 17000000000 spacemap 207 =20 > free 10.6G > metaslab 24 offset 18000000000 spacemap 208 =20 > free 29.9G > metaslab 25 offset 19000000000 spacemap 209 =20 > free 13.0G > metaslab 26 offset 1a000000000 spacemap 210 =20 > free 15.2G > metaslab 27 offset 1b000000000 spacemap 33 =20 > free 35.3G > metaslab 28 offset 1c000000000 spacemap 186 =20 > free 3.40G > metaslab 29 offset 1d000000000 spacemap 211 =20 > free 17.9G > metaslab 30 offset 1e000000000 spacemap 212 =20 > free 11.2G > metaslab 31 offset 1f000000000 spacemap 213 =20 > free 7.69G > metaslab 32 offset 20000000000 spacemap 214 =20 > free 21.2G > metaslab 33 offset 21000000000 spacemap 215 =20 > free 7.66G > metaslab 34 offset 22000000000 spacemap 216 =20 > free 15.6G > metaslab 35 offset 23000000000 spacemap 217 =20 > free 28.2G > metaslab 36 offset 24000000000 spacemap 218 =20 > free 20.8G > metaslab 37 offset 25000000000 spacemap 221 =20 > free 14.5G > metaslab 38 offset 26000000000 spacemap 192 =20 > free 14.1G > metaslab 39 offset 27000000000 spacemap 222 =20 > free 23.5G > metaslab 40 offset 28000000000 spacemap 223 =20 > free 22.8G > metaslab 41 offset 29000000000 spacemap 224 =20 > free 16.2G > metaslab 42 offset 2a000000000 spacemap 225 =20 > free 16.7G > metaslab 43 offset 2b000000000 spacemap 226 =20 > free 18.3G > metaslab 44 offset 2c000000000 spacemap 227 =20 > free 3.63G > metaslab 45 offset 2d000000000 spacemap 228 =20 > free 6.13G > metaslab 46 offset 2e000000000 spacemap 229 =20 > free 22.8G > metaslab 47 offset 2f000000000 spacemap 230 =20 > free 31.2G > metaslab 48 offset 30000000000 spacemap 204 =20 > free 5.64G > metaslab 49 offset 31000000000 spacemap 232 =20 > free 4.14G > metaslab 50 offset 32000000000 spacemap 233 =20 > free 22.0G > metaslab 51 offset 33000000000 spacemap 234 =20 > free 21.1G > metaslab 52 offset 34000000000 spacemap 235 =20 > free 10.9G > metaslab 53 offset 35000000000 spacemap 236 =20 > free 28.6G > metaslab 54 offset 36000000000 spacemap 32 =20 > free 24.2G > metaslab 55 offset 37000000000 spacemap 237 =20 > free 6.30G > metaslab 56 offset 38000000000 spacemap 238 =20 > free 22.6G > metaslab 57 offset 39000000000 spacemap 239 =20 > free 12.9G > metaslab 58 offset 3a000000000 spacemap 242 =20 > free 22.8G > metaslab 59 offset 3b000000000 spacemap 243 =20 > free 22.0G > metaslab 60 offset 3c000000000 spacemap 244 =20 > free 26.4G > metaslab 61 offset 3d000000000 spacemap 245 =20 > free 9.6G > metaslab 62 offset 3e000000000 spacemap 246 =20 > free 22.1G > metaslab 63 offset 3f000000000 spacemap 247 =20 > free 59.1G > metaslab 64 offset 40000000000 spacemap 220 =20 > free 61.8G > metaslab 65 offset 41000000000 spacemap 191 =20 > free 17.7G > metaslab 66 offset 42000000000 spacemap 248 =20 > free 13.1G > metaslab 67 offset 43000000000 spacemap 249 =20 > free 22.5G > metaslab 68 offset 44000000000 spacemap 250 =20 > free 4.39G > metaslab 69 offset 45000000000 spacemap 251 =20 > free 16.2G > metaslab 70 offset 46000000000 spacemap 252 =20 > free 3.88G > metaslab 71 offset 47000000000 spacemap 253 =20 > free 8.96G > metaslab 72 offset 48000000000 spacemap 254 =20 > free 25.2G > metaslab 73 offset 49000000000 spacemap 255 =20 > free 15.2G > metaslab 74 offset 4a000000000 spacemap 257 =20 > free 26.1G > metaslab 75 offset 4b000000000 spacemap 203 =20 > free 5.36G > metaslab 76 offset 4c000000000 spacemap 258 =20 > free 59.4G > metaslab 77 offset 4d000000000 spacemap 259 =20 > free 15.9G > metaslab 78 offset 4e000000000 spacemap 260 =20 > free 62.1G > metaslab 79 offset 4f000000000 spacemap 261 =20 > free 19.4G > metaslab 80 offset 50000000000 spacemap 262 =20 > free 4.07G > metaslab 81 offset 51000000000 spacemap 263 =20 > free 31.0G > metaslab 82 offset 52000000000 spacemap 264 =20 > free 32.1G > metaslab 83 offset 53000000000 spacemap 265 =20 > free 21.9G > metaslab 84 offset 54000000000 spacemap 266 =20 > free 26.2G > metaslab 85 offset 55000000000 spacemap 241 =20 > free 58.9G > metaslab 86 offset 56000000000 spacemap 267 =20 > free 22.3G > metaslab 87 offset 57000000000 spacemap 268 =20 > free 8.49G > metaslab 88 offset 58000000000 spacemap 269 =20 > free 17.5G > metaslab 89 offset 59000000000 spacemap 270 =20 > free 24.2G > metaslab 90 offset 5a000000000 spacemap 271 =20 > free 6.78G > metaslab 91 offset 5b000000000 spacemap 219 =20 > free 12.7G > metaslab 92 offset 5c000000000 spacemap 274 =20 > free 27.4G > metaslab 93 offset 5d000000000 spacemap 275 =20 > free 21.5G > metaslab 94 offset 5e000000000 spacemap 276 =20 > free 25.2G > metaslab 95 offset 5f000000000 spacemap 277 =20 > free 27.8G > metaslab 96 offset 60000000000 spacemap 278 =20 > free 6.67G > metaslab 97 offset 61000000000 spacemap 279 =20 > free 26.3G > metaslab 98 offset 62000000000 spacemap 280 =20 > free 12.0G > metaslab 99 offset 63000000000 spacemap 281 =20 > free 18.1G > metaslab 100 offset 64000000000 spacemap 282 =20 > free 23.3G > metaslab 101 offset 65000000000 spacemap 256 =20 > free 25.0G > metaslab 102 offset 66000000000 spacemap 231 =20 > free 16.8G > metaslab 103 offset 67000000000 spacemap 284 =20 > free 16.2G > metaslab 104 offset 68000000000 spacemap 285 =20 > free 20.0G > metaslab 105 offset 69000000000 spacemap 286 =20 > free 30.6G > metaslab 106 offset 6a000000000 spacemap 287 =20 > free 24.5G > metaslab 107 offset 6b000000000 spacemap 288 =20 > free 19.6G > metaslab 108 offset 6c000000000 spacemap 289 =20 > free 16.8G > metaslab 109 offset 6d000000000 spacemap 290 =20 > free 22.7G > metaslab 110 offset 6e000000000 spacemap 291 =20 > free 22.0G > metaslab 111 offset 6f000000000 spacemap 292 =20 > free 16.6G > metaslab 112 offset 70000000000 spacemap 240 =20 > free 14.8G > metaslab 113 offset 71000000000 spacemap 293 =20 > free 20.9G > metaslab 114 offset 72000000000 spacemap 294 =20 > free 53.7G > metaslab 115 offset 73000000000 spacemap 295 =20 > free 17.9G > metaslab 116 offset 74000000000 spacemap 296 =20 > free 19.1G > metaslab 117 offset 75000000000 spacemap 297 =20 > free 32.7G > metaslab 118 offset 76000000000 spacemap 298 =20 > free 17.8G > metaslab 119 offset 77000000000 spacemap 273 =20 > free 55.0G > metaslab 120 offset 78000000000 spacemap 299 =20 > free 20.7G > metaslab 121 offset 79000000000 spacemap 300 =20 > free 16.8G > metaslab 122 offset 7a000000000 spacemap 301 =20 > free 16.8G > metaslab 123 offset 7b000000000 spacemap 302 =20 > free 22.7G > metaslab 124 offset 7c000000000 spacemap 303 =20 > free 14.8G > metaslab 125 offset 7d000000000 spacemap 304 =20 > free 22.1G > metaslab 126 offset 7e000000000 spacemap 305 =20 > free 15.3G > metaslab 127 offset 7f000000000 spacemap 306 =20 > free 17.1G > metaslab 128 offset 80000000000 spacemap 307 =20 > free 20.2G > metaslab 129 offset 81000000000 spacemap 283 =20 > free 58.2G > metaslab 130 offset 82000000000 spacemap 308 =20 > free 24.5G > metaslab 131 offset 83000000000 spacemap 309 =20 > free 4.19G > metaslab 132 offset 84000000000 spacemap 310 =20 > free 15.0G > metaslab 133 offset 85000000000 spacemap 311 =20 > free 19.9G > metaslab 134 offset 86000000000 spacemap 312 =20 > free 60.9G > metaslab 135 offset 87000000000 spacemap 313 =20 > free 60.6G > metaslab 136 offset 88000000000 spacemap 314 =20 > free 60.9G > metaslab 137 offset 89000000000 spacemap 315 =20 > free 59.8G > metaslab 138 offset 8a000000000 spacemap 316 =20 > free 60.9G > metaslab 139 offset 8b000000000 spacemap 272 =20 > free 61.6G > metaslab 140 offset 8c000000000 spacemap 317 =20 > free 62.4G > metaslab 141 offset 8d000000000 spacemap 318 =20 > free 61.2G > metaslab 142 offset 8e000000000 spacemap 319 =20 > free 61.5G > > > Traversing all blocks to verify metadata checksums and verify > nothing leaked ... > > load: 1.59 cmd: zdb 54160 [physrd] 31.13r 3.05u 1.15s 4% 142544k > load: 0.45 cmd: zdb 54160 [physrd] 105.37r 6.69u 2.33s 4% 263428k > 5.64T completed ( 119MB/s) estimated time remaining: 0hr 12min > 55sec Assertion failed: (bp->blk_pad[0] =3D=3D 0), file > /usr/src/cddl/lib/libzpool/../../../sys/cddl/contrib/opensolaris/uts/co= mmon/fs/zfs/zio.c, > line 2978. > Abort (core dumped) > > The second command you suggested returned: > > # zdb -uuuC zroot > > MOS Configuration: > version: 5000 > name: 'zroot' > state: 0 > txg: 3377279 > pool_guid: 9132288035431788388 > hostid: 2783470193 > hostname: 'working-1.discdrive.bayphoto.com' > vdev_children: 1 > vdev_tree: > type: 'root' > id: 0 > guid: 9132288035431788388 > children[0]: > type: 'raidz' > id: 0 > guid: 15520162542638044402 > nparity: 2 > metaslab_array: 31 > metaslab_shift: 36 > ashift: 9 > asize: 9894744555520 > is_log: 0 > create_txg: 4 > children[0]: > type: 'disk' > id: 0 > guid: 4289437176706222104 > path: '/dev/mfid0p2' > devid:=20 > 'id1,sd@n6b8ca3a0f13870001a02310703f4b791/b' > phys_path: '/dev/mfid0p2' > whole_disk: 1 > DTL: 181 > create_txg: 4 > children[1]: > type: 'disk' > id: 1 > guid: 5369387862706621015 > path: '/dev/mfid1p2' > devid:=20 > 'id1,sd@n6b8ca3a0f13870001a02311604ce1965/b' > phys_path: '/dev/mfid1p2' > whole_disk: 1 > DTL: 180 > create_txg: 4 > children[2]: > type: 'disk' > id: 2 > guid: 456749962069636782 > path: '/dev/mfid2p2' > devid:=20 > 'id1,sd@n6b8ca3a0f13870001a02312105778eef/b' > phys_path: '/dev/mfid2p2' > whole_disk: 1 > DTL: 179 > create_txg: 4 > children[3]: > type: 'disk' > id: 3 > guid: 3809413300177228462 > path: '/dev/mfid3p2' > devid:=20 > 'id1,sd@n6b8ca3a0f13870001a02312905f430b5/b' > phys_path: '/dev/mfid3p2' > whole_disk: 1 > DTL: 178 > create_txg: 4 > children[4]: > type: 'disk' > id: 4 > guid: 4978694931676882497 > path: '/dev/mfid4p2' > devid:=20 > 'id1,sd@n6b8ca3a0f13870001a02313606b73c4a/b' > phys_path: '/dev/mfid4p2' > whole_disk: 1 > DTL: 177 > create_txg: 4 > children[5]: > type: 'disk' > id: 5 > guid: 17831739822150458220 > path: '/dev/mfid5p2' > devid:=20 > 'id1,sd@n6b8ca3a0f13870001a023142077914f5/b' > phys_path: '/dev/mfid5p2' > whole_disk: 1 > DTL: 176 > create_txg: 4 > children[6]: > type: 'disk' > id: 6 > guid: 1286918567594965543 > path: '/dev/mfid6p2' > devid:=20 > 'id1,sd@n6b8ca3a0f13870001a02314c080cb066/b' > phys_path: '/dev/mfid6p2' > whole_disk: 1 > DTL: 175 > create_txg: 4 > children[7]: > type: 'disk' > id: 7 > guid: 7958718879588658810 > path: '/dev/mfid7p2' > devid:=20 > 'id1,sd@n6b8ca3a0f13870001a02315608a7f0a2/b' > phys_path: '/dev/mfid7p2' > whole_disk: 1 > DTL: 174 > create_txg: 4 > children[8]: > type: 'disk' > id: 8 > guid: 18392960683862755998 > path: '/dev/mfid8p2' > devid:=20 > 'id1,sd@n6b8ca3a0f13870001a023160093a9190/b' > phys_path: '/dev/mfid8p2' > whole_disk: 1 > DTL: 173 > create_txg: 4 > children[9]: > type: 'disk' > id: 9 > guid: 13046629036569375198 > path: '/dev/mfid9p2' > devid:=20 > 'id1,sd@n6b8ca3a0f13870001a02316909c8894c/b' > phys_path: '/dev/mfid9p2' > whole_disk: 1 > DTL: 172 > create_txg: 4 > children[10]: > type: 'disk' > id: 10 > guid: 10604061156531251346 > path: '/dev/mfid11p2' > devid:=20 > 'id1,sd@n6b8ca3a0ef7a7a0019cc18e30bbfa11e/b' > phys_path: '/dev/mfid11p2' > whole_disk: 1 > DTL: 171 > create_txg: 4 > features_for_read: > > Uberblock: > magic =3D 0000000000bab10c > version =3D 5000 > txg =3D 3389469 > guid_sum =3D 1996697515446579069 > timestamp =3D 1401810802 UTC =3D Tue Jun 3 08:53:22 2014 > rootbp =3D DVA[0]=3D<0:3f0bf445c00:c00> > DVA[1]=3D<0:55027e77200:c00> DVA[2]=3D<0:86003a4c400:c00> [L0 DMU > objset] fletcher4 uncompressed LE contiguous unique triple > size=3D800L/800P birth=3D3389469L/3389469P fill=3D326 > cksum=3D389487e40:6aa058451f9:64bbaf298ba16:3f9bfc58017be5d > > Any reason why I would have to manually re-import the cache file? I=20 > had performed that task during the initial install (this was before=20 > bsdinstall had a zfs on root option, so it was done manually, where=20 > you have to export the cachefile, then at the end of the install cp it = > to /boot/zfs/zpool.cache and re-import it) > > Mike C --------------ms090905050307090206000305 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIITIjCC BhwwggQEoAMCAQICCGWP9DPVdj3ZMA0GCSqGSIb3DQEBCwUAMEsxDzANBgNVBAMMBlJvb3RD QTEWMBQGA1UECgwNQmF5IFBob3RvIExhYjETMBEGA1UECAwKQ2FsaWZvcm5pYTELMAkGA1UE BhMCVVMwHhcNMTIxMDIzMTczMjE4WhcNMjcxMDIzMTczMjE4WjBYMRwwGgYDVQQDDBNCYXkg UGhvdG8gUGVvcGxlIENBMRYwFAYDVQQKDA1CYXkgUGhvdG8gTGFiMRMwEQYDVQQIDApDYWxp Zm9ybmlhMQswCQYDVQQGEwJVUzCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBALuW vZE7FFSO77jEhdT/dZzu0XnPS35aepuvMvCNTagnNMbaf+AKrLZFaVRqBt4f6yl5TDXNEAMI +CJrg7T9dsoGN1Vygm4bIFzOIRtTZ1A7seemeonyqMto8gYMPhEFicuKavEAFcCFprVcVu5v yXbEG+tYkx08oUzeZ9LWZnh4a0wxAucbH0PcZFmdXFMbgjt6GSiR8jVUT1t/Dik1+mJ1A4K3 3VyM+G1CaqGIKg2UbvHiVQVoJrpgUc6vO5aWWgDreH+CAKLpyJwCj4BGphfHBPENu5LUp0CL fa4pA4r+OH0HNBp/I7utZMrIevh3pya34lDJXj0CQWSuVBR9Kp80IHHYGuyPU/sMXkUplsyI Y4KgQSRYRFNdWoboLwsRu1/2iDUaTR+XYH7Tu5zvCOedUm8ns/wfE7TW6kax5nR36VzuZS6B kUcuETO1QMxt2SJcHZIftiyTuxC8Y9Z7J6igR39pZPWGywJ2+oPRKFQD/u9RWZqjh9k5euVi 4bVwwog5DqnsymMTuKb0I1nCs5js+rfEVo+mc4gAfEmzk/dmpwkOnGV3oTfASbGtwMIlAEe5 HHJhdTA3jpVo3eQAZi479xZ7SozKvnjt1y9SMQ4uTJXjER/IVH3YC9S/BSFrqWLLDfWEGW84 SBn+ogldDIy3PX3SU7nOiN4Vse2Jg+r7AgMBAAGjgfYwgfMwHQYDVR0OBBYEFC34wnK9ZmLm B7vV52IeLHaapJ33MA8GA1UdEwEB/wQFMAMBAf8wHwYDVR0jBBgwFoAU00bJTnGPaRmhJAUZ t4x4J3soVyswgY8GA1UdHwSBhzCBhDCBgaB/oH2Ge2h0dHA6Ly9iYXljYS5iYXlob3RvLmxv Y2FsL2VqYmNhL3B1YmxpY3dlYi93ZWJkaXN0L2NlcnRkaXN0P2NtZD1jcmwmaXNzdWVyPUNO PVJvb3RDQSxPPUJheSUyMFBob3RvJTIwTGFiLFNUPUNhbGlmb3JuaWEsQz1VUzAOBgNVHQ8B Af8EBAMCAYYwDQYJKoZIhvcNAQELBQADggIBAEoDTeb1wVXz2VqFPrc3uaRnbVt6DCB9uBft L8YRgi5+FF5KHjvTqYHGiS38yVFfXM6MaN6yI7WSwtS+wVhM7Tdw6miwKNBApmCDKzjorNVX iZUmv88fjGm652LeIfT/oFGJg2oruoad9OfIofjT/jFpiFT7KOgjg4xeKCC0f+9naVo5uWOJ HTyFm1Kq6ryT+g7mZb7kLvndmJRW0aybn8Dls7/bhThEaOYkwn4dbZ6q0W2I5ZWg9bbh66V+ J9P1XObZA/TkRVTlpSYJ1hAgYX1yTbL360vuTAAwdSWQSFm4TAoXC4BsPdlgy86lDjNrW9id WX0RJCBzk/FzHhI4Aj9+SVhLl/Vkf9nbz8VhPPK2Az0PPYKy5ARtgOqGTKZbgVKut9Kgc/fn vUgPhObHQsD1Ui+NKsIGYBFKBhNmVXqMQSn4JzC9x0oDEmv37UGrcut2cCP3ZS17p137VaUM lQ0RWomju+sPCPFgyCPa/TLPoMZ2334uIxkRbDefvOoXIosORMQ9Jh50XqktUesbhuBfH9Q5 8h8bTWm1Cn/LxXW9qdSSbnta0OAH1G4hwVUlcusSM0o7Ude8tszw6kRpmEDDE8BQjE5nXSY7 wOf1eXfQfDkVQiouTD3l5ElqB98tnCnL/y9dphWoBLmiJwgb/4yWZ/Zewc1V65UFr7LmvcQM MIIGIzCCBAugAwIBAgIIYpSXgZOT7j0wDQYJKoZIhvcNAQELBQAwWDEcMBoGA1UEAwwTQmF5 IFBob3RvIFBlb3BsZSBDQTEWMBQGA1UECgwNQmF5IFBob3RvIExhYjETMBEGA1UECAwKQ2Fs aWZvcm5pYTELMAkGA1UEBhMCVVMwHhcNMTIxMDIzMTgwMDAzWhcNMTQxMDIzMTgwMDAzWjBg MRUwEwYKCZImiZPyLGQBAQwFMTMwNjkxFTATBgNVBAMMDE1pa2UgQ2FybHNvbjELMAkGA1UE CwwCSVQxFjAUBgNVBAoMDUJheSBQaG90byBMYWIxCzAJBgNVBAYTAlVTMIICIjANBgkqhkiG 9w0BAQEFAAOCAg8AMIICCgKCAgEAoTzIvF66A3wYPWXQFzmjBUuIrHSQGhOcE7rb0KQtar+H rkmHX56bCAklW5P/pd+yJ08lMwb3CxbTOz0q47JuBv2kKJO1xCgCua26Uvz3VAmfirmWwpXq zZBDqy/bEIt/XFfiVUC4jriGSEPrtx9q9nJJsb2JVRgtsbcHaaJFu8u8s8p8cLbcYdKobS00 g6+7it2IpIJhxc5tEMa1Yku3kCQiHVVFa9b4H5pFDHpkCrKZ43cuCneiR5kgr47z/3U66kLt J7Q5IT/i7nThjGQMa/f1JSWet8yeTomKvqkuEAA4o/IWQzEbtxzeps6vWxaCDULjEq69s//S 6PtqiQSmG9ZGFoPYD8/GGd4CMBqgjKopintD5sGTlJ851yZwl9VY/hRuxInp8gTjWrt1gQIB zlgSgSKnKTN6f+e85XMPU1y/wVz8RJWl8Tr11kzo6vrM14+ruNUxo1Ea3PJ9MUcWenoRKGSU I/IP94kZVjPkZlJv6tTF0Yi2Gclet/ZDu8vgvkxmUZYdQMGGlgZTCAsvHr37/ov6g51Tf+im 7410EsdYCmSINRGzWQAzlH9NscsW1TAd1Znog1H6NRDExY3ksjvFcKYOjUmkyWT8Vl2oJmT2 IzI23/C3esGL9OZzZ6K84MRNrH1y/yNp75vQnP3JfDMpbb5kkDp95Bu365qBluECAwEAAaOB 6DCB5TBSBggrBgEFBQcBAQRGMEQwQgYIKwYBBQUHMAGGNmh0dHA6Ly9iYXljYS5iYXlob3Rv LmxvY2FsL2VqYmNhL3B1YmxpY3dlYi9zdGF0dXMvb2NzcDAdBgNVHQ4EFgQUzeso+31hmtp3 soKHShXXtAEo+iMwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBQt+MJyvWZi5ge71ediHix2 mqSd9zAOBgNVHQ8BAf8EBAMCBDAwEwYDVR0lBAwwCgYIKwYBBQUHAwQwHAYDVR0RBBUwE4ER bWlrZUBiYXlwaG90by5jb20wDQYJKoZIhvcNAQELBQADggIBADnQfCasVgMsKsxIZAOZCbsU xo9BfsbpoM02p2aP+vPNDLXYRmcnH6ReeeUKSfIn0HmS9XkeHizMEXaC5sV9g4dasdQJQOGU mDcBnlxGn5fzNVFBM7/RHL83waYq4MCeyP9M7lSiNFZTrnSLVL9lIO0FLrBE06c9bn09kExc zkXI6Qm+e/MNrnoC3vw3GbH3a7tZCPsQcyNSok99jgPTRb1g9uVPg25M+ScYMU0wv2BE24u1 Dfzwcq52h64TllbzdVg/qOQH1HM96wmU+CtPuzA6eYnWRao/80LfQcyhNZ/jfMB/9xwFwsam o3Bw7SrSPEatw/tMyEEVMzas5/wZm2uMtab7642d5mr5OWLVPYgmKUscSlNt87vKkFhvn0Cz Z7O8O79WNMJA0sx1aomn1/ZrWDkd8X/ACUC2Fa3cV4AAzmjytiNu7r2z+GwdXPmvWSlBDXKX wLSoRkdq5hmYAP3GwXF0dsZo63WJLuCU1bPyERNLKdZM//eX832WgomPs4FA4xg0MUH0S7vJ eo7K1cTutZEmyLT623p0GcOINs2ir/ZqPTDLKszI7ytAltYaATt4kYUXbmMGGYItDf1X/caj DoLv2hjBTM5HORZYABC/Kfo9iL4KeYDqAvblJc7qyw+QXdHOUbwc9gQXQJvlQlfjDYvJLKme zoZ1sMzRBOl0MIIG1zCCBL+gAwIBAgIIGE38aUOyx8EwDQYJKoZIhvcNAQELBQAwWDEcMBoG A1UEAwwTQmF5IFBob3RvIFBlb3BsZSBDQTEWMBQGA1UECgwNQmF5IFBob3RvIExhYjETMBEG A1UECAwKQ2FsaWZvcm5pYTELMAkGA1UEBhMCVVMwHhcNMTIxMDIzMTc1NzQ1WhcNMTQxMDIz MTc1NzQ1WjBgMRUwEwYKCZImiZPyLGQBAQwFMTMwNjkxFTATBgNVBAMMDE1pa2UgQ2FybHNv bjELMAkGA1UECwwCSVQxFjAUBgNVBAoMDUJheSBQaG90byBMYWIxCzAJBgNVBAYTAlVTMIIC IjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAuECpC3YUm7GV0xz/DMmMQZ3EvMfvVhFB 77TcVxY1OoBlp2jk0ST0Hel/vp9uPhhi/eAlH89rC9fhNwORUHfeofWhoT3ZXrnjisNDQnb5 MgBV6wVM58PrikwU13FuNrIrPuUeuUE659BQhfgj2j2Pv9GVgBib6NBbHJAIOFo8H3wmd4b9 Yv6RKM145qSJCrJp96jSkoQSwni+jraHGMs55kgFP/3f0X6RpY7GhvItFI95Xauf7R5qTuW0 oamhvjFnlWVBoMuHd5yqzUgL7gQ0wfB31wfBP2ghFPAv715Qv9DYleFmgWE1LRkrt8clCzzk huj00CrEL+rSK9bDYNpa5AdCQ8aA/bG2x24ApOBvfDYnem+Ytu9lIY7qKZiP+9pASOiXsSSt g8CH9HoG+9GpK/rDyQ2nuNxTWEQEetwofkLdrGU/VgpcwIZqETsugu+l/FCu3Ogslezpiolb SqjCF5CU3aZravNZo4+HDriFKm7jyKEb+zUcXcwNH2iwbGt6uTMILslXbWXuhi8Lu3SKyacj 1gn9OMPQTCUKxaiTJXqIcJP0CV9wKcgTn7vanChDgz2fTVm6HOzb2RZlyhYzU4ofuMk+VGbg tBSDns+B9D1ACZMd58D2XdqRvmGPlnbJEZPkAZyDiR4mp/Aw09uXzOCIHzvqLhiysGon7lkV a18CAwEAAaOCAZswggGXMFIGCCsGAQUFBwEBBEYwRDBCBggrBgEFBQcwAYY2aHR0cDovL2Jh eWNhLmJheWhvdG8ubG9jYWwvZWpiY2EvcHVibGljd2ViL3N0YXR1cy9vY3NwMB0GA1UdDgQW BBRGT4gPK1JkpmK4YAsQgD82sskTGDAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFC34wnK9 ZmLmB7vV52IeLHaapJ33MIGlBgNVHR8EgZ0wgZowgZeggZSggZGGgY5odHRwOi8vYmF5Y2Eu YmF5aG90by5sb2NhbC9lamJjYS9wdWJsaWN3ZWIvd2ViZGlzdC9jZXJ0ZGlzdD9jbWQ9Y3Js Jmlzc3Vlcj1DTj1CYXklMjBQaG90byUyMFBlb3BsZSUyMENBLE89QmF5JTIwUGhvdG8lMjBM YWIsU1Q9Q2FsaWZvcm5pYSxDPVVTMA4GA1UdDwEB/wQEAwIGwDAdBgNVHSUEFjAUBggrBgEF BQcDAgYIKwYBBQUHAwQwHAYDVR0RBBUwE4ERbWlrZUBiYXlwaG90by5jb20wDQYJKoZIhvcN AQELBQADggIBAC+1BfV1bmcHZtgM4HN5rZek1vzY2UBLTHevnLDsLmNNASYMuOrm1TYMCI0/ LVkGltk0kq3lAP3hECsrSeH+SlmoRAmSCEOAwqOBU5oTqV/SMiQcZb6K2o/QUAVV6Si2KMyW U35hnqi9/vVNq4y28RjvMAydcrJpmn5qrWsy1ZLbW+Zu3jneBYjUcm4mQnqaCyjRyE2Z3byi /4m1sxISSdyqoRsWKsixSdb4bXUdNeNscltRYDMEDODtzYjnO7WebHtaMAEOyfE3n2gkPmGr dBopA6/Z53GVkG+MXF1wSlc3g38dHSpbY6qJiMkl8/wKCnkcMRPC3hRGQikYprh/oAbpHAyS q9P0jv/3cB0yzZ5b3n49PxqjwyH93lcRZJQ5pAMMWFk1B7IuYk9LVbJEVsPAmxFbWoS87Tn0 /zi6lkUNXoSgm4fJ+NBY5jnk1m48nPFIs7dpr8wW8qT2C0DiQz/tSKMramzsBduXY67djCZ5 cam1H5pRuaiEPEmrafC91i+wCsmjpMoqykKkuiGOlmY8LhRSZZ7M8qwtpo/0yur94z1Z5gyG 3yo/vZQtNDt8jaJ2ajGR50C48CtJufrckT3o3UrtjvQ3jCUnak2VbXIV6QNTio9NQKVH4+fL VhoGvc/PC/CifBs6Qyfk3a6nHdVfzwGx8Uytj/Br0AV0p8Y2MYIERjCCBEICAQEwZDBYMRww GgYDVQQDDBNCYXkgUGhvdG8gUGVvcGxlIENBMRYwFAYDVQQKDA1CYXkgUGhvdG8gTGFiMRMw EQYDVQQIDApDYWxpZm9ybmlhMQswCQYDVQQGEwJVUwIIGE38aUOyx8EwCQYFKw4DAhoFAKCC AbcwGAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTQwNjA0MTg0 NjU0WjAjBgkqhkiG9w0BCQQxFgQUG8m07b7FQ/fq6SgP2xsIQROvmyYwbAYJKoZIhvcNAQkP MV8wXTALBglghkgBZQMEASowCwYJYIZIAWUDBAECMAoGCCqGSIb3DQMHMA4GCCqGSIb3DQMC AgIAgDANBggqhkiG9w0DAgIBQDAHBgUrDgMCBzANBggqhkiG9w0DAgIBKDBzBgkrBgEEAYI3 EAQxZjBkMFgxHDAaBgNVBAMME0JheSBQaG90byBQZW9wbGUgQ0ExFjAUBgNVBAoMDUJheSBQ aG90byBMYWIxEzARBgNVBAgMCkNhbGlmb3JuaWExCzAJBgNVBAYTAlVTAghilJeBk5PuPTB1 BgsqhkiG9w0BCRACCzFmoGQwWDEcMBoGA1UEAwwTQmF5IFBob3RvIFBlb3BsZSBDQTEWMBQG A1UECgwNQmF5IFBob3RvIExhYjETMBEGA1UECAwKQ2FsaWZvcm5pYTELMAkGA1UEBhMCVVMC CGKUl4GTk+49MA0GCSqGSIb3DQEBAQUABIICAJrxrvhxDGvQTG10aI6+vM3TQPcKyDslQYdm 8NHaREYl0Jh7M61kNEHG+dmcYTjIXu6PbF3wzYgw26xvnFPEBAM7AP1TtCZip/XVdQrDZvBU lGOWLx8d0zUSLa6yZIthQ71Pt+etjc4qaf/8epmtEi7Uo4uMRt0koRkrdagJeT4cQwyDYisn 9T62aVLoPtEZ7ecdG9Z0+E09DonAk7+MpuEDxMOKXCRUeKAjPffrXnCWzusrv8KmzNUp8Wwq AGceYmyJmQcCpTlGWYJ0fyTpNcUMtEDlK2sAN/BugFG3gdytGcnd1X8KQuRwHxyxMWbIX1+v plXHBAlbAUlCgd4UNNOstkeukak53ZDuvVk84GwSSdlZ00bVOBixUUrc6e+F+8c/066zWPVK 5QfV3rzFyDrZ+F31sJgVZ5/8hNuavtWepDaSt2knC/EGjnm8qECuT+GgzmvSAfFTDAwUUCy/ d1vP90XxyojqbyvG4Ns1g0Lp7/b1JMOEEzMkXdI+41ygkO3VI84CZ62+9nqnnt4u8KX7VzmO 12TtWZ8mHMi9xs0Io5j3trIcl4vOphrQB9WgTYXf6+xiMcZxsvAoXYc0jlQmIzdcnS+IIaBX KqAkFulkiD0r486uNFbplrckrY1+rfjrKx9pCYDQb/XOV2670xSj/GeRJmnQMr2X9IpA/40P AAAAAAAA --------------ms090905050307090206000305--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?538F699E.4060802>