Date: Sat, 04 May 2013 16:33:13 -0700 From: Mike Carlson <mike@bayphoto.com> To: freebsd-fs@freebsd.org Subject: Re: zfs issue - disappearing data Message-ID: <51859AB9.6040804@bayphoto.com> In-Reply-To: <51845054.3020302@bayphoto.com> References: <5183F739.2040908@bayphoto.com> <51843D0E.2020907@platinum.linux.pl> <51845054.3020302@bayphoto.com>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a cryptographically signed message in MIME format. --------------ms000105080501020404010507 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: quoted-printable Just to update the list and close this thread, this issue as it turns=20 out was not ZFS related, but the application that read/writes to the=20 CIFS share. Good to know about the leaking extended attributes though, and of=20 course, the great responses from Adam and Jeremy. Thanks again, Mike C On 5/3/2013 5:03 PM, Mike Carlson wrote: > Interesting. > > Is that why zdb shows so many objects? > > Is this a configuration mistake, and would it lead to data loss? > > Can I provide any addition information ? > > Mike C > > On 5/3/2013 3:41 PM, Adam Nowacki wrote: >> Looks like we have a leak with extended attributes: >> >> # zfs create -o mountpoint=3D/test root/test >> # touch /test/file1 >> # setextattr user test abc /test/file1 >> # zdb root/test >> Object lvl iblk dblk dsize lsize %full type >> 8 1 16K 512 0 512 0.00 ZFS plain file >> 9 1 16K 512 1K 512 100.00 ZFS directory >> 10 1 16K 512 512 512 100.00 ZFS plain file >> >> object 8 - the file, >> object 9 - extended attributes directory, >> object 10 - value of the 'test' extended attribute >> >> # rm /test/file1 >> # zdb root/test >> >> Object lvl iblk dblk dsize lsize %full type >> 10 1 16K 512 512 512 100.00 ZFS plain file >> >> objects 8 and 9 are deleted, object 10 is still there (leaked). >> >> On 2013-05-03 19:43, Mike Carlson wrote: >>> We had a critical issue with a zfs server that exports shares via sam= ba >>> (3.5) last night >>> >>> system info: >>> uname -a >>> >>> FreeBSD zfs-1.discdrive.bayphoto.com 9.1-RELEASE FreeBSD=20 >>> 9.1-RELEASE >>> #0 r243825: Tue Dec 4 09:23:10 UTC 2012 >>> root@farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64 >>> >>> zpool history: >>> >>> History for 'data': >>> 2013-02-25.17:11:37 zpool create data raidz /dev/gpt/disk1.nop >>> /dev/gpt/disk2.nop /dev/gpt/disk3.nop /dev/gpt/disk4.nop >>> 2013-02-25.17:11:41 zpool add data raidz /dev/gpt/disk5.nop >>> /dev/gpt/disk6.nop /dev/gpt/disk7.nop /dev/gpt/disk8.nop >>> 2013-02-25.17:11:47 zpool add data raidz /dev/gpt/disk9.nop >>> /dev/gpt/disk10.nop /dev/gpt/disk11.nop /dev/gpt/disk12.nop >>> 2013-02-25.17:11:53 zpool add data raidz /dev/gpt/disk13.nop >>> /dev/gpt/disk14.nop /dev/gpt/disk15.nop /dev/gpt/disk16.nop >>> 2013-02-25.17:11:57 zpool add data raidz /dev/gpt/disk17.nop >>> /dev/gpt/disk18.nop /dev/gpt/disk19.nop /dev/gpt/disk20.nop >>> 2013-02-25.17:12:02 zpool add data raidz /dev/gpt/disk21.nop >>> /dev/gpt/disk22.nop /dev/gpt/disk23.nop /dev/gpt/disk24.nop >>> 2013-02-25.17:12:08 zpool add data spare /dev/gpt/disk25.nop >>> /dev/gpt/disk26.nop >>> 2013-02-25.17:12:15 zpool add data log /dev/gpt/log.nop >>> 2013-02-25.17:12:19 zfs set checksum=3Dfletcher4 data >>> 2013-02-25.17:12:22 zfs set compression=3Dlzjb data >>> 2013-02-25.17:12:25 zfs set aclmode=3Dpassthrough data >>> 2013-02-25.17:12:30 zfs set aclinherit=3Dpassthrough data >>> 2013-02-25.17:13:25 zpool export data >>> 2013-02-25.17:15:33 zpool import -d /dev/gpt data >>> 2013-03-01.12:31:58 zpool add data cache /dev/gpt/cache.nop >>> 2013-03-15.12:22:22 zfs create data/XML_WORKFLOW >>> 2013-03-27.12:05:42 zfs create data/IMAGEQUIX >>> 2013-03-27.13:32:54 zfs create data/ROES_ORDERS >>> 2013-03-27.13:32:59 zfs create data/ROES_PRINTABLES >>> 2013-03-27.13:33:21 zfs destroy data/ROES_PRINTABLES >>> 2013-03-27.13:33:26 zfs create data/ROES_PRINTABLE >>> >>> We had a file structure drop off: >>> >>> /data/XML_WORKFLOW/XML_ORDERS/ >>> >>> around 5/2/2012 @ 17:00 >>> >>> In that directory, there were a few thousand directories (containing >>> images and a couple metadata text/xml files) >>> >>> What is odd, is doing a du -h in the parent XML_WORKFLOW directory,=20 >>> only >>> reports ~150MB: >>> >>> # find . -type f |wc -l >>> 86 >>> # du -sh . >>> 130M . >>> >>> >>> however, df reports 1.5GB: >>> >>> # df -h . >>> Filesystem Size Used Avail Capacity Mounted on >>> data/XML_WORKFLOW 28T 1.5G 28T 0% /data/XML_WORKFLOW >>> >>> zdb -d shows: >>> >>> # zdb -d data/XML_WORKFLOW >>> Dataset data/XML_WORKFLOW [ZPL], ID 139, cr_txg 339633, 1.53G, >>> 212812 objects >>> >>> Digging further into zdb, the path is missing for most of those=20 >>> objects: >>> >>> # zdb -ddddd data/XML_WORKFLOW 635248 >>> Dataset data/XML_WORKFLOW [ZPL], ID 139, cr_txg 339633, 1.53G, >>> 212812 objects, rootbp DVA[0]=3D<5:b274264000:2000> >>> DVA[1]=3D<0:b4d81a8000:2000> [L0 DMU objset] fletcher4 lzjb LE >>> contiguous unique double size=3D800L/200P birth=3D1202311L/120231= 1P >>> fill=3D212812=20 >>> cksum=3D16d24fb5aa:6c2e0aff6bc:129af90fe2eff:2612f938c5292b >>> >>> Object lvl iblk dblk dsize lsize %full type >>> 635248 1 16K 512 6.00K 512 100.00 ZFS plain file >>> 168 bonus System attributes >>> dnode flags: USED_BYTES USERUSED_ACCOUNTED >>> dnode maxblkid: 0 >>> path ???<object#635248> >>> uid 11258 >>> gid 10513 >>> atime Thu May 2 17:31:26 2013 >>> mtime Thu May 2 17:31:26 2013 >>> ctime Thu May 2 17:31:26 2013 >>> crtime Thu May 2 17:13:58 2013 >>> gen 1197180 >>> mode 100600 >>> size 52 >>> parent 635247 >>> links 1 >>> pflags 40800000005 >>> Indirect blocks: >>> 0 L0 3:a9da05a000:2000 200L/200P F=3D1 B=3D1197391/1197391 >>> >>> segment [0000000000000000, 0000000000000200) size 512 >>> >>> The application that writes to this volume runs on a windows client, = so >>> far, it has exhibited identical behavior across two zfs servers, but = >>> not >>> on a generic windows server 2003 network share. >>> >>> The question is, what is happening to the data. Is it a samba issue? = Is >>> it ZFS? I've enabled the samba full_audit module to track file >>> deletions, so I should have more information on that side. >>> >>> If anyone has seen similar behavior please let me know >>> >>> Mike C >> > > --------------ms000105080501020404010507 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIITIjCC BhwwggQEoAMCAQICCGWP9DPVdj3ZMA0GCSqGSIb3DQEBCwUAMEsxDzANBgNVBAMMBlJvb3RD QTEWMBQGA1UECgwNQmF5IFBob3RvIExhYjETMBEGA1UECAwKQ2FsaWZvcm5pYTELMAkGA1UE BhMCVVMwHhcNMTIxMDIzMTczMjE4WhcNMjcxMDIzMTczMjE4WjBYMRwwGgYDVQQDDBNCYXkg UGhvdG8gUGVvcGxlIENBMRYwFAYDVQQKDA1CYXkgUGhvdG8gTGFiMRMwEQYDVQQIDApDYWxp Zm9ybmlhMQswCQYDVQQGEwJVUzCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBALuW vZE7FFSO77jEhdT/dZzu0XnPS35aepuvMvCNTagnNMbaf+AKrLZFaVRqBt4f6yl5TDXNEAMI +CJrg7T9dsoGN1Vygm4bIFzOIRtTZ1A7seemeonyqMto8gYMPhEFicuKavEAFcCFprVcVu5v yXbEG+tYkx08oUzeZ9LWZnh4a0wxAucbH0PcZFmdXFMbgjt6GSiR8jVUT1t/Dik1+mJ1A4K3 3VyM+G1CaqGIKg2UbvHiVQVoJrpgUc6vO5aWWgDreH+CAKLpyJwCj4BGphfHBPENu5LUp0CL fa4pA4r+OH0HNBp/I7utZMrIevh3pya34lDJXj0CQWSuVBR9Kp80IHHYGuyPU/sMXkUplsyI Y4KgQSRYRFNdWoboLwsRu1/2iDUaTR+XYH7Tu5zvCOedUm8ns/wfE7TW6kax5nR36VzuZS6B kUcuETO1QMxt2SJcHZIftiyTuxC8Y9Z7J6igR39pZPWGywJ2+oPRKFQD/u9RWZqjh9k5euVi 4bVwwog5DqnsymMTuKb0I1nCs5js+rfEVo+mc4gAfEmzk/dmpwkOnGV3oTfASbGtwMIlAEe5 HHJhdTA3jpVo3eQAZi479xZ7SozKvnjt1y9SMQ4uTJXjER/IVH3YC9S/BSFrqWLLDfWEGW84 SBn+ogldDIy3PX3SU7nOiN4Vse2Jg+r7AgMBAAGjgfYwgfMwHQYDVR0OBBYEFC34wnK9ZmLm B7vV52IeLHaapJ33MA8GA1UdEwEB/wQFMAMBAf8wHwYDVR0jBBgwFoAU00bJTnGPaRmhJAUZ t4x4J3soVyswgY8GA1UdHwSBhzCBhDCBgaB/oH2Ge2h0dHA6Ly9iYXljYS5iYXlob3RvLmxv Y2FsL2VqYmNhL3B1YmxpY3dlYi93ZWJkaXN0L2NlcnRkaXN0P2NtZD1jcmwmaXNzdWVyPUNO PVJvb3RDQSxPPUJheSUyMFBob3RvJTIwTGFiLFNUPUNhbGlmb3JuaWEsQz1VUzAOBgNVHQ8B Af8EBAMCAYYwDQYJKoZIhvcNAQELBQADggIBAEoDTeb1wVXz2VqFPrc3uaRnbVt6DCB9uBft L8YRgi5+FF5KHjvTqYHGiS38yVFfXM6MaN6yI7WSwtS+wVhM7Tdw6miwKNBApmCDKzjorNVX iZUmv88fjGm652LeIfT/oFGJg2oruoad9OfIofjT/jFpiFT7KOgjg4xeKCC0f+9naVo5uWOJ HTyFm1Kq6ryT+g7mZb7kLvndmJRW0aybn8Dls7/bhThEaOYkwn4dbZ6q0W2I5ZWg9bbh66V+ J9P1XObZA/TkRVTlpSYJ1hAgYX1yTbL360vuTAAwdSWQSFm4TAoXC4BsPdlgy86lDjNrW9id WX0RJCBzk/FzHhI4Aj9+SVhLl/Vkf9nbz8VhPPK2Az0PPYKy5ARtgOqGTKZbgVKut9Kgc/fn vUgPhObHQsD1Ui+NKsIGYBFKBhNmVXqMQSn4JzC9x0oDEmv37UGrcut2cCP3ZS17p137VaUM lQ0RWomju+sPCPFgyCPa/TLPoMZ2334uIxkRbDefvOoXIosORMQ9Jh50XqktUesbhuBfH9Q5 8h8bTWm1Cn/LxXW9qdSSbnta0OAH1G4hwVUlcusSM0o7Ude8tszw6kRpmEDDE8BQjE5nXSY7 wOf1eXfQfDkVQiouTD3l5ElqB98tnCnL/y9dphWoBLmiJwgb/4yWZ/Zewc1V65UFr7LmvcQM MIIGIzCCBAugAwIBAgIIYpSXgZOT7j0wDQYJKoZIhvcNAQELBQAwWDEcMBoGA1UEAwwTQmF5 IFBob3RvIFBlb3BsZSBDQTEWMBQGA1UECgwNQmF5IFBob3RvIExhYjETMBEGA1UECAwKQ2Fs aWZvcm5pYTELMAkGA1UEBhMCVVMwHhcNMTIxMDIzMTgwMDAzWhcNMTQxMDIzMTgwMDAzWjBg MRUwEwYKCZImiZPyLGQBAQwFMTMwNjkxFTATBgNVBAMMDE1pa2UgQ2FybHNvbjELMAkGA1UE CwwCSVQxFjAUBgNVBAoMDUJheSBQaG90byBMYWIxCzAJBgNVBAYTAlVTMIICIjANBgkqhkiG 9w0BAQEFAAOCAg8AMIICCgKCAgEAoTzIvF66A3wYPWXQFzmjBUuIrHSQGhOcE7rb0KQtar+H rkmHX56bCAklW5P/pd+yJ08lMwb3CxbTOz0q47JuBv2kKJO1xCgCua26Uvz3VAmfirmWwpXq zZBDqy/bEIt/XFfiVUC4jriGSEPrtx9q9nJJsb2JVRgtsbcHaaJFu8u8s8p8cLbcYdKobS00 g6+7it2IpIJhxc5tEMa1Yku3kCQiHVVFa9b4H5pFDHpkCrKZ43cuCneiR5kgr47z/3U66kLt J7Q5IT/i7nThjGQMa/f1JSWet8yeTomKvqkuEAA4o/IWQzEbtxzeps6vWxaCDULjEq69s//S 6PtqiQSmG9ZGFoPYD8/GGd4CMBqgjKopintD5sGTlJ851yZwl9VY/hRuxInp8gTjWrt1gQIB zlgSgSKnKTN6f+e85XMPU1y/wVz8RJWl8Tr11kzo6vrM14+ruNUxo1Ea3PJ9MUcWenoRKGSU I/IP94kZVjPkZlJv6tTF0Yi2Gclet/ZDu8vgvkxmUZYdQMGGlgZTCAsvHr37/ov6g51Tf+im 7410EsdYCmSINRGzWQAzlH9NscsW1TAd1Znog1H6NRDExY3ksjvFcKYOjUmkyWT8Vl2oJmT2 IzI23/C3esGL9OZzZ6K84MRNrH1y/yNp75vQnP3JfDMpbb5kkDp95Bu365qBluECAwEAAaOB 6DCB5TBSBggrBgEFBQcBAQRGMEQwQgYIKwYBBQUHMAGGNmh0dHA6Ly9iYXljYS5iYXlob3Rv LmxvY2FsL2VqYmNhL3B1YmxpY3dlYi9zdGF0dXMvb2NzcDAdBgNVHQ4EFgQUzeso+31hmtp3 soKHShXXtAEo+iMwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBQt+MJyvWZi5ge71ediHix2 mqSd9zAOBgNVHQ8BAf8EBAMCBDAwEwYDVR0lBAwwCgYIKwYBBQUHAwQwHAYDVR0RBBUwE4ER bWlrZUBiYXlwaG90by5jb20wDQYJKoZIhvcNAQELBQADggIBADnQfCasVgMsKsxIZAOZCbsU xo9BfsbpoM02p2aP+vPNDLXYRmcnH6ReeeUKSfIn0HmS9XkeHizMEXaC5sV9g4dasdQJQOGU mDcBnlxGn5fzNVFBM7/RHL83waYq4MCeyP9M7lSiNFZTrnSLVL9lIO0FLrBE06c9bn09kExc zkXI6Qm+e/MNrnoC3vw3GbH3a7tZCPsQcyNSok99jgPTRb1g9uVPg25M+ScYMU0wv2BE24u1 Dfzwcq52h64TllbzdVg/qOQH1HM96wmU+CtPuzA6eYnWRao/80LfQcyhNZ/jfMB/9xwFwsam o3Bw7SrSPEatw/tMyEEVMzas5/wZm2uMtab7642d5mr5OWLVPYgmKUscSlNt87vKkFhvn0Cz Z7O8O79WNMJA0sx1aomn1/ZrWDkd8X/ACUC2Fa3cV4AAzmjytiNu7r2z+GwdXPmvWSlBDXKX wLSoRkdq5hmYAP3GwXF0dsZo63WJLuCU1bPyERNLKdZM//eX832WgomPs4FA4xg0MUH0S7vJ eo7K1cTutZEmyLT623p0GcOINs2ir/ZqPTDLKszI7ytAltYaATt4kYUXbmMGGYItDf1X/caj DoLv2hjBTM5HORZYABC/Kfo9iL4KeYDqAvblJc7qyw+QXdHOUbwc9gQXQJvlQlfjDYvJLKme zoZ1sMzRBOl0MIIG1zCCBL+gAwIBAgIIGE38aUOyx8EwDQYJKoZIhvcNAQELBQAwWDEcMBoG A1UEAwwTQmF5IFBob3RvIFBlb3BsZSBDQTEWMBQGA1UECgwNQmF5IFBob3RvIExhYjETMBEG A1UECAwKQ2FsaWZvcm5pYTELMAkGA1UEBhMCVVMwHhcNMTIxMDIzMTc1NzQ1WhcNMTQxMDIz MTc1NzQ1WjBgMRUwEwYKCZImiZPyLGQBAQwFMTMwNjkxFTATBgNVBAMMDE1pa2UgQ2FybHNv bjELMAkGA1UECwwCSVQxFjAUBgNVBAoMDUJheSBQaG90byBMYWIxCzAJBgNVBAYTAlVTMIIC IjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAuECpC3YUm7GV0xz/DMmMQZ3EvMfvVhFB 77TcVxY1OoBlp2jk0ST0Hel/vp9uPhhi/eAlH89rC9fhNwORUHfeofWhoT3ZXrnjisNDQnb5 MgBV6wVM58PrikwU13FuNrIrPuUeuUE659BQhfgj2j2Pv9GVgBib6NBbHJAIOFo8H3wmd4b9 Yv6RKM145qSJCrJp96jSkoQSwni+jraHGMs55kgFP/3f0X6RpY7GhvItFI95Xauf7R5qTuW0 oamhvjFnlWVBoMuHd5yqzUgL7gQ0wfB31wfBP2ghFPAv715Qv9DYleFmgWE1LRkrt8clCzzk huj00CrEL+rSK9bDYNpa5AdCQ8aA/bG2x24ApOBvfDYnem+Ytu9lIY7qKZiP+9pASOiXsSSt g8CH9HoG+9GpK/rDyQ2nuNxTWEQEetwofkLdrGU/VgpcwIZqETsugu+l/FCu3Ogslezpiolb SqjCF5CU3aZravNZo4+HDriFKm7jyKEb+zUcXcwNH2iwbGt6uTMILslXbWXuhi8Lu3SKyacj 1gn9OMPQTCUKxaiTJXqIcJP0CV9wKcgTn7vanChDgz2fTVm6HOzb2RZlyhYzU4ofuMk+VGbg tBSDns+B9D1ACZMd58D2XdqRvmGPlnbJEZPkAZyDiR4mp/Aw09uXzOCIHzvqLhiysGon7lkV a18CAwEAAaOCAZswggGXMFIGCCsGAQUFBwEBBEYwRDBCBggrBgEFBQcwAYY2aHR0cDovL2Jh eWNhLmJheWhvdG8ubG9jYWwvZWpiY2EvcHVibGljd2ViL3N0YXR1cy9vY3NwMB0GA1UdDgQW BBRGT4gPK1JkpmK4YAsQgD82sskTGDAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFC34wnK9 ZmLmB7vV52IeLHaapJ33MIGlBgNVHR8EgZ0wgZowgZeggZSggZGGgY5odHRwOi8vYmF5Y2Eu YmF5aG90by5sb2NhbC9lamJjYS9wdWJsaWN3ZWIvd2ViZGlzdC9jZXJ0ZGlzdD9jbWQ9Y3Js Jmlzc3Vlcj1DTj1CYXklMjBQaG90byUyMFBlb3BsZSUyMENBLE89QmF5JTIwUGhvdG8lMjBM YWIsU1Q9Q2FsaWZvcm5pYSxDPVVTMA4GA1UdDwEB/wQEAwIGwDAdBgNVHSUEFjAUBggrBgEF BQcDAgYIKwYBBQUHAwQwHAYDVR0RBBUwE4ERbWlrZUBiYXlwaG90by5jb20wDQYJKoZIhvcN AQELBQADggIBAC+1BfV1bmcHZtgM4HN5rZek1vzY2UBLTHevnLDsLmNNASYMuOrm1TYMCI0/ LVkGltk0kq3lAP3hECsrSeH+SlmoRAmSCEOAwqOBU5oTqV/SMiQcZb6K2o/QUAVV6Si2KMyW U35hnqi9/vVNq4y28RjvMAydcrJpmn5qrWsy1ZLbW+Zu3jneBYjUcm4mQnqaCyjRyE2Z3byi /4m1sxISSdyqoRsWKsixSdb4bXUdNeNscltRYDMEDODtzYjnO7WebHtaMAEOyfE3n2gkPmGr dBopA6/Z53GVkG+MXF1wSlc3g38dHSpbY6qJiMkl8/wKCnkcMRPC3hRGQikYprh/oAbpHAyS q9P0jv/3cB0yzZ5b3n49PxqjwyH93lcRZJQ5pAMMWFk1B7IuYk9LVbJEVsPAmxFbWoS87Tn0 /zi6lkUNXoSgm4fJ+NBY5jnk1m48nPFIs7dpr8wW8qT2C0DiQz/tSKMramzsBduXY67djCZ5 cam1H5pRuaiEPEmrafC91i+wCsmjpMoqykKkuiGOlmY8LhRSZZ7M8qwtpo/0yur94z1Z5gyG 3yo/vZQtNDt8jaJ2ajGR50C48CtJufrckT3o3UrtjvQ3jCUnak2VbXIV6QNTio9NQKVH4+fL VhoGvc/PC/CifBs6Qyfk3a6nHdVfzwGx8Uytj/Br0AV0p8Y2MYIERjCCBEICAQEwZDBYMRww GgYDVQQDDBNCYXkgUGhvdG8gUGVvcGxlIENBMRYwFAYDVQQKDA1CYXkgUGhvdG8gTGFiMRMw EQYDVQQIDApDYWxpZm9ybmlhMQswCQYDVQQGEwJVUwIIGE38aUOyx8EwCQYFKw4DAhoFAKCC AbcwGAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTMwNTA0MjMz MzEzWjAjBgkqhkiG9w0BCQQxFgQUMOaJuYr1zqRRis9UygH/n2BV/HcwbAYJKoZIhvcNAQkP MV8wXTALBglghkgBZQMEASowCwYJYIZIAWUDBAECMAoGCCqGSIb3DQMHMA4GCCqGSIb3DQMC AgIAgDANBggqhkiG9w0DAgIBQDAHBgUrDgMCBzANBggqhkiG9w0DAgIBKDBzBgkrBgEEAYI3 EAQxZjBkMFgxHDAaBgNVBAMME0JheSBQaG90byBQZW9wbGUgQ0ExFjAUBgNVBAoMDUJheSBQ aG90byBMYWIxEzARBgNVBAgMCkNhbGlmb3JuaWExCzAJBgNVBAYTAlVTAghilJeBk5PuPTB1 BgsqhkiG9w0BCRACCzFmoGQwWDEcMBoGA1UEAwwTQmF5IFBob3RvIFBlb3BsZSBDQTEWMBQG A1UECgwNQmF5IFBob3RvIExhYjETMBEGA1UECAwKQ2FsaWZvcm5pYTELMAkGA1UEBhMCVVMC CGKUl4GTk+49MA0GCSqGSIb3DQEBAQUABIICAB7HJxHc5viiC+moreff63icWktFInk+jDvA wtHdqZ+u8aNFBSf6sY92yIEHuAmqL59aTRP3ovxFvg1j/WTpoySMFBv4eZ7E33u171EHO/un YensoBlQ8Miea6AlZfEcr9eh8Ehw5I0Mfz0Z3PNWi50LVJLEl8aXtMtwwQSWV5IPssY7EkFO TOf0yPJIFH+frwv0JE6i0enjWS+7RDirWG7kLuQiDQLi2J4T86P/sZpuvrit3mNyEnrFmm7W 1hp8SgcGZyT2uLwcMnf8OY09MJV/LJIbjOcFIfYqocHe8SfNreXfT4p/qVSiInfllnPl5J0r JadmoXoFIMW+pw20qkmvNhYut+93RhWJm8FnD98zUQqYx2bzv+1rtOvlp7lLD4f7QsN9TdSB LsmQH4JR9HCRA1BhK4UM8yh7FbETEG7Z9PyAHAj5c8oDPUeBt1YvymkgMVv9twyWAHFiRM3l GMwJXEraVtE5RYbRtfHgFg7TVgWgMXggW13UvWEX72iytC1tn9IzEe17a7EMqeVm/kB79AK5 1bMx/eDxZ1k4wmRociC1922yhzjadyPYZdq/1WGMU2FzcTYljZs3v9rOig+W0qmt5c2B+b0v +NLmzDjBbZ9bRjVFYrUhwAAJpsulyAX7hKPfwMLY+t2DRchfU82VtggP6j1qcNMM5I72N0H2 AAAAAAAA --------------ms000105080501020404010507--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?51859AB9.6040804>