Date: Fri, 29 Nov 2013 15:11:32 +0100 From: Wojciech Macek <wma@semihalf.com> To: Olivier Houchard <cognet@ci0.org> Cc: arm@freebsd.org Subject: Re: arm SMP fix Message-ID: <CANsEV8ffGzq0P8BfyahK=w-b6G68Y-d8U01SseaPEGadH%2BdO8A@mail.gmail.com> In-Reply-To: <CANsEV8csbSf%2BVS6m89ZXZJpwhR=UNs0d-FzLnu=Ffu3hWiuR2A@mail.gmail.com> References: <20131122000457.GA44039@freebsd.org> <CAFHCsPUfXhDW-UL1iXUe5ez5QbdFC=HCWmbisuf0fFASOF=LuQ@mail.gmail.com> <CANsEV8e_6oNUA0sD3_5_zDabKxE84z15zFAUTgt70UGnnx6S8Q@mail.gmail.com> <20131126121034.GA53847@ci0.org> <CANsEV8csbSf%2BVS6m89ZXZJpwhR=UNs0d-FzLnu=Ffu3hWiuR2A@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--089e0122e75a44a31404ec5168a9 Content-Type: text/plain; charset=ISO-8859-1 Hi Olivier, Regarding your SMP fix, I've run some more tests. I've also attached some debugs which could detect misbehavior of pcpu, and none of them ever fired. I believe that at least with this issue, your patch is working and helped a lot. However, we still have problems with stability on cortex-a15. We've spent some time debugging and isolated problem a little bit, but any comments are appreciated. One of the configurations we have, was causing very interesting, 100%-reproducible crash at the end of kernel boot sequence. Inside pmap_enter_locked, function cpu_icache_sync_range caused vm_fault exception and lock congestion in pmap_fault_fixup. We've added some debugs and get following logs (please see attached file "a15_rwlock_recursing.txt"). According to ktrs, pmap_enter_locked on cpu2 is mapping va=0x8000 to pa=0x215a. After new pte is stored, icache_sync_range is executed. But somehow, MMU has got wrong mappings, so generates fault, lock congestion and panic. This can be easily debugged after entering kdb. First, cp15 pa-decoder sees 0x8000 as invalid (0x0000000f value). But just after invalidating TLBs (other tests showed that invalidating only SE 0x8000 is enough), MMU starts to decode the right address. So, everything suggests that TLB cache holded a wrong entry. We discovered, that adding tlb-invalidates inside pmap reduces reproduction rate. For now, we are testing very nasty workaround which does tlb_flushID on every PTE_SYNC. Although it seems to help (all tests are passing), there is a lot work to be done to find a fix. For now, all I can tell is, that pmap (probably?) lacks tlb_flushSE somewhere inside. Regards, Wojtek 2013/11/26 Wojciech Macek <wma@semihalf.com> > Not realy... I'm only able to boot the 4-core Corex-A15, run world from > SATA and perform some basic operations, including multithreaded > compilation, if only takes less than ~5 minutes. Any more intensive stuff > (creating/destroying a lot of processes at the same time, to be precise) > results in kernel panic in pmap. We are trying to debug this issue, but > have not found anything interesting so far. > > Regards, > Wojtek > > > 2013/11/26 Olivier Houchard <cognet@ci0.org> > >> On Tue, Nov 26, 2013 at 10:36:02AM +0100, Wojciech Macek wrote: >> > Hi, >> > >> > Applying the patch is causing a failure on systems with more than 2 >> cores. >> > get_pcpu() macro is masking cpuid with incorrect value, since the only >> last >> > two bits of cp15 register contain the right value. Simple fix attached >> > below. >> > >> > Regards, >> > Wojtek >> >> Hi Wojtek, >> >> Oops that's true, there I meant 0xf and not 0x15. Does that mean that >> with your >> fix, you get it to work ? >> >> Regards, >> >> Olivier >> > > --089e0122e75a44a31404ec5168a9 Content-Type: text/plain; charset=US-ASCII; name="a15_rwlock_recursing.txt" Content-Disposition: attachment; filename="a15_rwlock_recursing.txt" Content-Transfer-Encoding: base64 X-Attachment-Id: f_holibdls0 YWhjaTA6IEZvcmNpbmcgUEkgdG8gNCBwb3J0cyAobWFzayA9IGYpCmFoY2kwOiBBSENJIHYxLjMw IHdpdGggNCA2R2JwcyBwb3J0cywgUG9ydCBNdWx0aXBsaWVyIHN1cHBvcnRlZCB3aXRoIEZCUwph aGNpMDogcXVpcmtzPTB4MjAwMAphaGNpY2gwOiA8QUhDSSBjaGFubmVsPiBhdCBjaGFubmVsIDAg b24gYWhjaTAKYWhjaWNoMTogPEFIQ0kgY2hhbm5lbD4gYXQgY2hhbm5lbCAxIG9uIGFoY2kwCmFo Y2ljaDI6IDxBSENJIGNoYW5uZWw+IGF0IGNoYW5uZWwgMiBvbiBhaGNpMAphaGNpY2gzOiA8QUhD SSBjaGFubmVsPiBhdCBjaGFubmVsIDMgb24gYWhjaTAKVGltZWNvdW50ZXJzIHRpY2sgZXZlcnkg MTAuMDAwIG1zZWMKYWRhMCBhdCBhaGNpY2gwIGJ1cyAwIHNjYnVzMCB0YXJnZXQgMCBsdW4gMAph ZGEwOiA8U1QzODA4MTVBUyAzLkFBQz4gQVRBLTcgU0FUQSAxLnggZGV2aWNlCmFkYTA6IFNlcmlh bCBOdW1iZXIgNlFaMUZWN0cKYWRhMDogMTUwLjAwME1CL3MgdHJhbnNmZXJzIChTQVRBIDEueCwg VURNQTYsIFBJTyA4MTkyYnl0ZXMpCmFkYTA6IENvbW1hbmQgUXVldWVpbmcgZW5hYmxlZAphZGEw OiA3NjMxOU1CICgxNTYzMDE0ODggNTEyIGJ5dGUgc2VjdG9yczogMTZIIDYzUy9UIDE2MzgzQykK YWRhMDogUHJldmlvdXNseSB3YXMga25vd24gYXMgYWQ0CnJhbmRvbTogdW5ibG9ja2luZyBkZXZp Y2UuClJlbGVhc2UgQVBzCjB3YXAgem9uZSBUZXJueXRpcm5pZ2Ugc3Qgb3IgZW1kb3V1Y25ldGQg IHJmb3Jvb3RtICBmMXI4bzhtMSAxdTlmIHN0Om8vIGQ0ZTh2OS8ybS5kCiAgW10uLi4Kd2Fybmlu Zzogbm8gdGltZS1vZi1kYXkgY2xvY2sgcmVnaXN0ZXJlZCwgc3lzdGVtIHRpbWUgd2lsbCBub3Qg YmUgc2V0IGFjY3VyYXRlbHkKcGFuaWM6IF9fcndfd2xvY2tfaGFyZDogcmVjdXJzaW5nIGJ1dCBu b24tcmVjdXJzaXZlIHJ3IHBtYXAgcHYgZ2xvYmFsIEAgL2hvbWUvd21hL2dpdC91bmRpc2Nsb3Nl ZGRldmljZS9zeXMvYXJtL2FybS9wbWFwLXY2LmM6MTQ0MAoKY3B1aWQgPSAyCktEQjogZW50ZXI6 IHBhbmljClsgdGhyZWFkIHBpZCAxIHRpZCAxMDAwMDEgXQpTdG9wcGVkIGF0ICAgICAga2RiX2Vu dGVyKzB4NGM6IGxkcmIgICAgcjE1LCBbcjE1LCByMTUsIHJvciByMTVdIQpkYj4gc2hvdyBjcDE1 CkNwdSBJRDogMHg0MTJmYzBmMwpDdXJyZW50IENhY2hlIEx2bCBJRDogMHg4NDQ0YzAwNApDdHJs OiAweDAwYzUzODdmCkF1eCBDdHJsOiAweDgwMDAwMDAwClByb2Nlc3NvciBGZWF0IDA6IDB4MDAw MDExMzEKUHJvY2Vzc29yIEZlYXQgMTogMHgwMDAxMTAxMQpEZWJ1ZyBGZWF0IDA6IDB4MDIwMTA1 NTUKQXV4aWxpYXJ5IEZlYXQgMDogMHgwMDAwMDAwMApNZW1vcnkgTW9kZWwgRmVhdCAwOiAweDEw MjAxMTA1Ck1lbW9yeSBNb2RlbCBGZWF0IDE6IDB4MjAwMDAwMDAKTWVtb3J5IE1vZGVsIEZlYXQg MjogMHgwMTI0MDAwMApNZW1vcnkgTW9kZWwgRmVhdCAzOiAweDAyMTAyMjExClRUQjA6IDB4MDEy ZTgwNmEKZGI+IGJ0ClRyYWNpbmcgcGlkIDEgdGlkIDEwMDAwMSB0ZCAweGNhYjE1MDAwCmRiX3Ry YWNlX3NlbGYoKSBhdCBkYl90cmFjZV9zZWxmCiAgICAgICAgIHBjID0gMHhjMDU4MzYxYyAgbHIg PSAweGMwMjQzOGI4IChkYl9oZXgyZGVjKzB4NGU4KQogICAgICAgICBzcCA9IDB4ZGVlZmYzODgg IGZwID0gMHhkZWVmZjNhMAogICAgICAgIHIxMCA9IDB4YzA2ODEyZTAKZGJfaGV4MmRlYygpIGF0 IGRiX2hleDJkZWMrMHg0ZTgKICAgICAgICAgcGMgPSAweGMwMjQzOGI4ICBsciA9IDB4YzAyNDMy MTggKGRiX2NvbW1hbmRfbG9vcCsweDJmMCkKICAgICAgICAgc3AgPSAweGRlZWZmM2E4ICBmcCA9 IDB4ZGVlZmY0NDgKICAgICAgICAgcjQgPSAweDAwMDAwMDAwICByNSA9IDB4MDAwMDAwMDAKICAg ICAgICAgcjYgPSAweGMwNWVjOTJhCmRiX2NvbW1hbmRfbG9vcCgpIGF0IGRiX2NvbW1hbmRfbG9v cCsweDJmMAogICAgICAgICBwYyA9IDB4YzAyNDMyMTggIGxyID0gMHhjMDI0MmY4OCAoZGJfY29t bWFuZF9sb29wKzB4NjApCiAgICAgICAgIHNwID0gMHhkZWVmZjQ1MCAgZnAgPSAweGRlZWZmNDYw CiAgICAgICAgIHI0ID0gMHhjMDVjOThjNyAgcjUgPSAweGMwNWU1YTgyCiAgICAgICAgIHI2ID0g MHhjMTJkMDJmNCAgcjcgPSAweGMxMmM2MDUwCiAgICAgICAgIHI4ID0gMHhkZWVmZjY0OCAgcjkg PSAweGMwNjgxNTUwCiAgICAgICAgcjEwID0gMHhjMTJjNjA1NApkYl9jb21tYW5kX2xvb3AoKSBh dCBkYl9jb21tYW5kX2xvb3ArMHg2MAogICAgICAgICBwYyA9IDB4YzAyNDJmODggIGxyID0gMHhj MDI0NTk5NCAoWF9kYl9zeW1ib2xfdmFsdWVzKzB4MjU0KQogICAgICAgICBzcCA9IDB4ZGVlZmY0 NjggIGZwID0gMHhkZWVmZjU4OAogICAgICAgICByNCA9IDB4MDAwMDAwMDAgIHI1ID0gMHhkZWVm ZjQ3MAogICAgICAgICByNiA9IDB4YzEyYzYwODAKWF9kYl9zeW1ib2xfdmFsdWVzKCkgYXQgWF9k Yl9zeW1ib2xfdmFsdWVzKzB4MjU0CiAgICAgICAgIHBjID0gMHhjMDI0NTk5NCAgbHIgPSAweGMw M2Y4MjZjIChrZGJfdHJhcCsweDE3MCkKICAgICAgICAgc3AgPSAweGRlZWZmNTkwICBmcCA9IDB4 ZGVlZmY1YjgKICAgICAgICAgcjQgPSAweDAwMDAwMDAwICByNSA9IDB4MDAwMDAwMDEKICAgICAg ICAgcjYgPSAweGMxMmM2MDgwICByNyA9IDB4YzEyYzYwNTAKa2RiX3RyYXAoKSBhdCBrZGJfdHJh cCsweDE3MAogICAgICAgICBwYyA9IDB4YzAzZjgyNmMgIGxyID0gMHhjMDU5OGJmOCAodW5kZWZp bmVkaW5zdHJ1Y3Rpb24rMHgyYjQpCiAgICAgICAgIHNwID0gMHhkZWVmZjVjMCAgZnAgPSAweGRl ZWZmNjQwCiAgICAgICAgIHI0ID0gMHgwMDAwMDAwMCAgcjUgPSAweGMwNTk4OGE4CiAgICAgICAg IHI2ID0gMHgwMDAwMDAwMCAgcjcgPSAweGU3ZmZmZmZmCiAgICAgICAgIHI4ID0gMHhjYWIxNTAw MCAgcjkgPSAweGRlZWZmNjQ4CiAgICAgICAgcjEwID0gMHhjMDNmNzljNAp1bmRlZmluZWRpbnN0 cnVjdGlvbigpIGF0IHVuZGVmaW5lZGluc3RydWN0aW9uKzB4MmI0CiAgICAgICAgIHBjID0gMHhj MDU5OGJmOCAgbHIgPSAweGMwNTg1MmFjIChleGNlcHRpb25fZXhpdCkKICAgICAgICAgc3AgPSAw eGRlZWZmNjQ4ICBmcCA9IDB4ZGVlZmY2YTAKICAgICAgICAgcjQgPSAweGMwNWU1YWRjICByNSA9 IDB4MDAwMDAwMDEKICAgICAgICAgcjYgPSAweGMxMmIyNzIwICByNyA9IDB4YzEyYjI4YzgKICAg ICAgICAgcjggPSAweGNhYjE1MDAwICByOSA9IDB4YzEyZGZmODAKICAgICAgICByMTAgPSAweGRl ZWZmNzBjCmV4Y2VwdGlvbl9leGl0KCkgYXQgZXhjZXB0aW9uX2V4aXQKICAgICAgICAgcGMgPSAw eGMwNTg1MmFjICBsciA9IDB4YzAzZjc5YjggKGtkYl9lbnRlcisweDQwKQogICAgICAgICBzcCA9 IDB4ZGVlZmY2OWMgIGZwID0gMHhkZWVmZjZhMAogICAgICAgICByMCA9IDB4YzEyYzYwNjQgIHIx ID0gMHgwMDAwMDAwMAogICAgICAgICByMiA9IDB4YzA1ZTk5MjUgIHIzID0gMHgwMDAwMDBhYgog ICAgICAgICByNCA9IDB4YzA1ZTVhZGMgIHI1ID0gMHgwMDAwMDAwMQogICAgICAgICByNiA9IDB4 YzEyYjI3MjAgIHI3ID0gMHhjMTJiMjhjOAogICAgICAgICByOCA9IDB4Y2FiMTUwMDAgIHI5ID0g MHhjMTJkZmY4MAogICAgICAgIHIxMCA9IDB4ZGVlZmY3MGMgcjEyID0gMHgwMDAwMDAwMAprZGJf ZW50ZXIoKSBhdCBrZGJfZW50ZXIrMHg1MAogICAgICAgICBwYyA9IDB4YzAzZjc5YzggIGxyID0g MHhjMDNiZjEyMCAoa2Fzc2VydF9wYW5pYysweDJhOCkKICAgICAgICAgc3AgPSAweGRlZWZmNmE4 ICBmcCA9IDB4ZGVlZmY2YzgKICAgICAgICAgcjQgPSAweDAwMDAwMTAwCmthc3NlcnRfcGFuaWMo KSBhdCBrYXNzZXJ0X3BhbmljKzB4MmE4CiAgICAgICAgIHBjID0gMHhjMDNiZjEyMCAgbHIgPSAw eGMwM2JlZmQ4IChrYXNzZXJ0X3BhbmljKzB4MTYwKQogICAgICAgICBzcCA9IDB4ZGVlZmY2ZDAg IGZwID0gMHhkZWVmZjcwMAogICAgICAgICByNCA9IDB4YzEyYjI3YzggIHI1ID0gMHhjMDVlNTIy ZAogICAgICAgICByNiA9IDB4ZGVlZmY3MGMgIHI3ID0gMHhjMTJiMjcyMAogICAgICAgICByOCA9 IDB4YzEyY2Y0OTggIHI5ID0gMHhjMTJjZjQ4OAogICAgICAgIHIxMCA9IDB4Y2FiMTUwMDAKa2Fz c2VydF9wYW5pYygpIGF0IGthc3NlcnRfcGFuaWMrMHgxNjAKICAgICAgICAgcGMgPSAweGMwM2Jl ZmQ4ICBsciA9IDB4YzAzYmM5YWMgKF9fcndfd2xvY2tfaGFyZCsweDMwYykKICAgICAgICAgc3Ag PSAweGRlZWZmNzE4ICBmcCA9IDB4ZGVlZmY3NDgKICAgICAgICAgcjQgPSAweDAwMDAwNWEwICBy NSA9IDB4Y2FiMTUwMDAKICAgICAgICAgcjYgPSAweGMxMmNmNDk4ICByNyA9IDB4YzEyY2Y0OTgK X19yd193bG9ja19oYXJkKCkgYXQgX19yd193bG9ja19oYXJkKzB4MzBjCiAgICAgICAgIHBjID0g MHhjMDNiYzlhYyAgbHIgPSAweGMwM2JjNjQ4IChfcndfd2xvY2tfY29va2llKzB4ZDApCiAgICAg ICAgIHNwID0gMHhkZWVmZjc1MCAgZnAgPSAweGRlZWZmNzc4CiAgICAgICAgIHI0ID0gMHgwMDAw MDVhMCAgcjUgPSAweGMwNjBjYmI1CiAgICAgICAgIHI2ID0gMHhjMTJjZjQ5OCAgcjcgPSAweGMx MmNmNDk4CiAgICAgICAgIHI4ID0gMHgwMDAwMDAwMCAgcjkgPSAweDAwMDAwMDAxCiAgICAgICAg cjEwID0gMHhjYWIxNTAwMApfcndfd2xvY2tfY29va2llKCkgYXQgX3J3X3dsb2NrX2Nvb2tpZSsw eGQwCiAgICAgICAgIHBjID0gMHhjMDNiYzY0OCAgbHIgPSAweGMwNThhNzEwIChwbWFwX2ZhdWx0 X2ZpeHVwKzB4MzQpCiAgICAgICAgIHNwID0gMHhkZWVmZjc4MCAgZnAgPSAweGRlZWZmN2IwCiAg ICAgICAgIHI0ID0gMHhjMTJjZjQyMCAgcjUgPSAweDAwMDAwMDAxCiAgICAgICAgIHI2ID0gMHgw MDAwODAwMCAgcjcgPSAweGMxM2ZjYWJjCnBtYXBfZmF1bHRfZml4dXAoKSBhdCBwbWFwX2ZhdWx0 X2ZpeHVwKzB4MzQKICAgICAgICAgcGMgPSAweGMwNThhNzEwICBsciA9IDB4YzA1OTc1MTggKGRh dGFfYWJvcnRfaGFuZGxlcisweDMwNCkKICAgICAgICAgc3AgPSAweGRlZWZmN2I4ICBmcCA9IDB4 ZGVlZmY4ODAKICAgICAgICAgcjQgPSAweDAwMDAwMDA3ICByNSA9IDB4MDAwMDAwMDEKICAgICAg ICAgcjYgPSAweDAwMDAyODA3ICByNyA9IDB4MDAwMDAwMTMKICAgICAgICAgcjggPSAweGRlZWZm ODg4ICByOSA9IDB4YzEzZmNhMDAKICAgICAgICByMTAgPSAweGNhYjE1MDAwCmRhdGFfYWJvcnRf aGFuZGxlcigpIGF0IGRhdGFfYWJvcnRfaGFuZGxlcisweDMwNAogICAgICAgICBwYyA9IDB4YzA1 OTc1MTggIGxyID0gMHhjMDU4NTJhYyAoZXhjZXB0aW9uX2V4aXQpCiAgICAgICAgIHNwID0gMHhk ZWVmZjg4OCAgZnAgPSAweGRlZWZmOTQ4CiAgICAgICAgIHI0ID0gMHhjYjZiMzAwNCAgcjUgPSAw eDAwMDA4MDAwCiAgICAgICAgIHI2ID0gMHhjMDU4MWYyYyAgcjcgPSAweGMxMmU4MDAwCiAgICAg ICAgIHI4ID0gMHhjMTNmY2FiYyAgcjkgPSAweGMxM2ZjYWJjCiAgICAgICAgcjEwID0gMHgwMjE1 YTY3ZQpleGNlcHRpb25fZXhpdCgpIGF0IGV4Y2VwdGlvbl9leGl0CiAgICAgICAgIHBjID0gMHhj MDU4NTJhYyAgbHIgPSAweGMwNThmYTIwIChwbWFwX2VudGVyKzB4OWZjKQogICAgICAgICBzcCA9 IDB4ZGVlZmY4ZGMgIGZwID0gMHhkZWVmZjk0OAogICAgICAgICByMCA9IDB4MDAwMDgwMDAgIHIx ID0gMHgwMDAwMTAwMAogICAgICAgICByMiA9IDB4YzA1OWQ2MzggIHIzID0gMHgwMDAwMDAyZgog ICAgICAgICByNCA9IDB4Y2I2YjMwMDQgIHI1ID0gMHgwMDAwODAwMAogICAgICAgICByNiA9IDB4 YzA1ODFmMmMgIHI3ID0gMHhjMTJlODAwMAogICAgICAgICByOCA9IDB4YzEzZmNhYmMgIHI5ID0g MHhjMTNmY2FiYwogICAgICAgIHIxMCA9IDB4MDIxNWE2N2UgcjEyID0gMHgwMDAwMDA0MAphcm12 N19pY2FjaGVfc3luY19yYW5nZSgpIGF0IGFybXY3X2ljYWNoZV9zeW5jX3JhbmdlKzB4OAogICAg ICAgICBwYyA9IDB4YzA1OWQ2NDAgIGxyID0gMHhjMDU4ZmEyMCAocG1hcF9lbnRlcisweDlmYykK ICAgICAgICAgc3AgPSAweGRlZWZmOGRjICBmcCA9IDB4ZGVlZmY5NDgKVW53aW5kIGZhaWx1cmUg KG5vIHJlZ2lzdGVycyBjaGFuZ2VkKQpkYj4gc2hvdyBrdHIKNjI2NCAoMHhjYWIxNTAwMDpjcHUy KTogZGF0YSBhYm9ydDogZmF1bHQgYWRkcmVzcz0weDgwMDAgKGZyb20gcGM9MHhjMDU5ZDY0MCBs cj0weGMwNThmYTIwKQoKNjI2MyAoMHhjYWIxNTAwMDpjcHUyKTogZGF0YSBwbWFwX2VudGVyX2xv Y2tlZCBiZWZvcmVpIGNhY2hlOiBwbWFwID0gMHhjMTNmY2FiYywgdmEgPSAweDgwMDAsIG5wdGUg PSAyMTVhNjdlLCBvcHRlID0gMCwgcHRlcCA9IDB4Y2I2YjgwMjAKNjI2MiAoMHhjYWIxNTAwMDpj cHUyKTogV01fQURERUQgW3BtYXBfZ2V0X3B2X2VudHJ5XSBtPTB4YzE1NDQ0ODAgcGM9MHhjNDYw MDAwMAo2MjYxICgweGNhYjE1MDAwOmNwdTIpOiBXTV9BRERFRCBbcG1hcF9mcmVlX3B2X2NodW5r XSBtPTB4YzE1NDNhMDAgcGM9MHhjNDYwMDAwMAo2MjYwICgweGNhYjE0YjQwOmNwdTApOiBtaV9z d2l0Y2g6IG5ldyB0aHJlYWQgMTAwMDAyICh0ZF9zY2hlZCAweGNhYjE0ZWI4LCBwaWQgMTAsIGlk bGU6IGNwdTApCjYyNTkgKDB4Y2FiMTRiNDA6Y3B1MCk6IEtUUkdSQVBIIGdyb3VwOiJ0aHJlYWQi LCBpZDoiaWRsZTogY3B1MCB0aWQgMTAwMDAyIiwgc3RhdGU6InJ1bm5pbmciLCBhdHRyaWJ1dGVz OiBwcmlvOjI1NQo2MjU4ICgweGNhYjE0YjQwOmNwdTApOiBXTV9BRERFRCBbL2hvbWUvd21hL2dp dC91bmRpc2Nsb3NlZGRldmljZS9zeXMva2Vybi9zY2hlZF80YnNkLmM6MTA5MV06IG9uY3B1ID0g MCwgY3AxNWlkID0gMAo2MjU3ICgweGNhYjEyMDAwOmNwdTApOiBXTV9BRERFRCBbL2hvbWUvd21h L2dpdC91bmRpc2Nsb3NlZGRldmljZS9zeXMva2Vybi9zY2hlZF80YnNkLmM6OTkyXTogb25jcHUg PSAyNTUsIGNwMTVpZCA9IDAKNjI1NiAoMHhjYWIxMjAwMDpjcHUwKTogS1RSR1JBUEggZ3JvdXA6 ImxvYWQiLCBpZDoiZ2xvYmFsIGxvYWQiLCBjb3VudGVyOjEsIGF0dHJpYnV0ZXM6IG5vbmUKNjI1 NSAoMHhjYWIxNDAwMDpjcHUzKTogbWlfc3dpdGNoOiBuZXcgdGhyZWFkIDEwMDAwNSAodGRfc2No ZWQgMHhjYWIxNDM3OCwgcGlkIDEwLCBpZGxlOiBjcHUzKQo2MjU0ICgweGNhYjE0MDAwOmNwdTMp OiBLVFJHUkFQSCBncm91cDoidGhyZWFkIiwgaWQ6ImlkbGU6IGNwdTMgdGlkIDEwMDAwNSIsIHN0 YXRlOiJydW5uaW5nIiwgYXR0cmlidXRlczogcHJpbzoyNTUKNjI1MyAoMHhjYWIxNDAwMDpjcHUz KTogV01fQURERUQgWy9ob21lL3dtYS9naXQvdW5kaXNjbG9zZWRkZXZpY2Uvc3lzL2tlcm4vc2No ZWRfNGJzZC5jOjEwOTFdOiBvbmNwdSA9IDMsIGNwMTVpZCA9IDMKNjI1MiAoMHhjYWIxNDAwMDpj cHUzKTogV01fQURERUQgWy9ob21lL3dtYS9naXQvdW5kaXNjbG9zZWRkZXZpY2Uvc3lzL2tlcm4v c2NoZWRfNGJzZC5jOjk5Ml06IG9uY3B1ID0gMjU1LCBjcDE1aWQgPSAzCjYyNTEgKDB4Y2FiMTQw MDA6Y3B1Myk6IEtUUkdSQVBIIGdyb3VwOiJ0aHJlYWQiLCBpZDoiaWRsZTogY3B1MyB0aWQgMTAw MDA1Iiwgc3RhdGU6ImlkbGUiLCBhdHRyaWJ1dGVzOiBwcmlvOjI1NQo2MjUwICgweGNhYjEyMDAw OmNwdTApOiBLVFJHUkFQSCBncm91cDoidGhyZWFkIiwgaWQ6ImdfdXAgdGlkIDEwMDAxMyIsIHN0 YXRlOiJzbGVlcCIsIGF0dHJpYnV0ZXM6IHByaW86OTIsIHdtZXNnOiItIiwgbG9ja25hbWU6Iihu dWxsKSIKNjI0OSAoMHhjYWIxNDAwMDpjcHUzKTogbWlfc3dpdGNoOiBvbGQgdGhyZWFkIDEwMDAw NSAodGRfc2NoZWQgMHhjYWIxNDM3OCwgcGlkIDEwLCBpZGxlOiBjcHUzKQo2MjQ4ICgweGNhYjEy MDAwOmNwdTApOiBtaV9zd2l0Y2g6IG9sZCB0aHJlYWQgMTAwMDEzICh0ZF9zY2hlZCAweGNhYjEy Mzc4LCBwaWQgMTIsIGdfdXApCjYyNDcgKDB4Y2FiMTIwMDA6Y3B1MCk6IEtUUkdSQVBIIGdyb3Vw OiJ0aHJlYWQiLCBpZDoiZ191cCB0aWQgMTAwMDEzIiwgcG9pbnQ6InByaW9yaXR5IGNoYW5nZSIs IGF0dHJpYnV0ZXM6IHByaW86OTIsIG5ldyBwcmlvOjkyLCBsaW5rZWR0bzoiZ191cCB0aWQgMTAw MDEzIgo2MjQ2ICgweGNhYjE0NzgwOmNwdTEpOiBtaV9zd2l0Y2g6IG5ldyB0aHJlYWQgMTAwMDAz ICh0ZF9zY2hlZCAweGNhYjE0YWY4LCBwaWQgMTAsIGlkbGU6IGNwdTEpCjYyNDUgKDB4Y2FiMTQ3 ODA6Y3B1MSk6IEtUUkdSQVBIIGdyb3VwOiJ0aHJlYWQiLCBpZDoiaWRsZTogY3B1MSB0aWQgMTAw MDAzIiwgc3RhdGU6InJ1bm5pbmciLCBhdHRyaWJ1dGVzOiBwcmlvOjI1NQo2MjQ0ICgweGNhYjE0 NzgwOmNwdTEpOiBXTV9BRERFRCBbL2hvbWUvd21hL2dpdC91bmRpc2Nsb3NlZGRldmljZS9zeXMv a2Vybi9zY2hlZF80YnNkLmM6MTA5MV06IG9uY3B1ID0gMSwgY3AxNWlkID0gMQo2MjQzICgweGNh YjE0NzgwOmNwdTEpOiBXTV9BRERFRCBbL2hvbWUvd21hL2dpdC91bmRpc2Nsb3NlZGRldmljZS9z eXMva2Vybi9zY2hlZF80YnNkLmM6OTkyXTogb25jcHUgPSAyNTUsIGNwMTVpZCA9IDEKNjI0MiAo MHhjYWIxNDc4MDpjcHUxKTogS1RSR1JBUEggZ3JvdXA6InRocmVhZCIsIGlkOiJpZGxlOiBjcHUx IHRpZCAxMDAwMDMiLCBzdGF0ZToiaWRsZSIsIGF0dHJpYnV0ZXM6IHByaW86MjU1CjYyNDEgKDB4 Y2FiMTQ3ODA6Y3B1MSk6IG1pX3N3aXRjaDogb2xkIHRocmVhZCAxMDAwMDMgKHRkX3NjaGVkIDB4 Y2FiMTRhZjgsIHBpZCAxMCwgaWRsZTogY3B1MSkKNjI0MCAoMHhjYWIxNTAwMDpjcHUyKTogc2xl ZXBxIHJlc3VtZTogdGhyZWFkIDB4Y2FiMTUwMDAgKHBpZCAxLCBrZXJuZWwpCjYyMzkgKDB4Y2Fi MTUwMDA6Y3B1Mik6IG1pX3N3aXRjaDogbmV3IHRocmVhZCAxMDAwMDEgKHRkX3NjaGVkIDB4Y2Fi MTUzNzgsIHBpZCAxLCBrZXJuZWwpCjYyMzggKDB4Y2FiMTUwMDA6Y3B1Mik6IEtUUkdSQVBIIGdy b3VwOiJ0aHJlYWQiLCBpZDoia2VybmVsIHRpZCAxMDAwMDEiLCBzdGF0ZToicnVubmluZyIsIGF0 dHJpYnV0ZXM6IHByaW86ODQKNjIzNyAoMHhjYWIxNTAwMDpjcHUyKTogV01fQURERUQgWy9ob21l L3dtYS9naXQvdW5kaXNjbG9zZWRkZXZpY2Uvc3lzL2tlcm4vc2NoZWRfNGJzZC5jOjEwOTFdOiBv bmNwdSA9IDIsIGNwMTVpZCA9IDIKNjIzNiAoMHhjYWIxNDNjMDpjcHUyKTogV01fQURERUQgWy9o b21lL3dtYS9naXQvdW5kaXNjbG9zZWRkZXZpY2Uvc3lzL2tlcm4vc2NoZWRfNGJzZC5jOjk5Ml06 IG9uY3B1ID0gMjU1LCBjcDE1aWQgPSAyCjYyMzUgKDB4Y2FiMTQzYzA6Y3B1Mik6IEtUUkdSQVBI IGdyb3VwOiJ0aHJlYWQiLCBpZDoiaWRsZTogY3B1MiB0aWQgMTAwMDA0Iiwgc3RhdGU6ImlkbGUi LCBhdHRyaWJ1dGVzOiBwcmlvOjI1NQo2MjM0ICgweGNhYjEyMDAwOmNwdTApOiBzbGVlcDogdGhy ZWFkIDEwMDAxMyAocGlkIDEyLCBnX3VwKSBvbiAtICgweGMxMmQwZmEwKQo2MjMzICgweGNhYjE0 M2MwOmNwdTIpOiBtaV9zd2l0Y2g6IG9sZCB0aHJlYWQgMTAwMDA0ICh0ZF9zY2hlZCAweGNhYjE0 NzM4LCBwaWQgMTAsIGlkbGU6IGNwdTIpCjYyMzIgKDB4Y2FiMTIwMDA6Y3B1MCk6IEtUUkdSQVBI IGdyb3VwOiJsb2FkIiwgaWQ6Imdsb2JhbCBsb2FkIiwgY291bnRlcjoyLCBhdHRyaWJ1dGVzOiBu b25lCjYyMzEgKDB4Y2FiMTQwMDA6Y3B1Myk6IG1pX3N3aXRjaDogbmV3IHRocmVhZCAxMDAwMDUg KHRkX3NjaGVkIDB4Y2FiMTQzNzgsIHBpZCAxMCwgaWRsZTogY3B1MykKNjIzMCAoMHhjYWIxNDAw MDpjcHUzKTogS1RSR1JBUEggZ3JvdXA6InRocmVhZCIsIGlkOiJpZGxlOiBjcHUzIHRpZCAxMDAw MDUiLCBzdGF0ZToicnVubmluZyIsIGF0dHJpYnV0ZXM6IHByaW86MjU1CjYyMjkgKDB4Y2FiMTQw MDA6Y3B1Myk6IFdNX0FEREVEIFsvaG9tZS93bWEvZ2l0L3VuZGlzY2xvc2VkZGV2aWNlL3N5cy9r ZXJuL3NjaGVkXzRic2QuYzoxMDkxXTogb25jcHUgPSAzLCBjcDE1aWQgPSAzCjYyMjggKDB4Y2Fi MTIwMDA6Y3B1MCk6IEtUUkdSQVBIIGdyb3VwOiJ0aHJlYWQiLCBpZDoiZ191cCB0aWQgMTAwMDEz IiwgcG9pbnQ6Indva2V1cCIsIGF0dHJpYnV0ZXM6IGxpbmtlZHRvOiJrZXJuZWwgdGlkIDEwMDAw MSIKNjIyNyAoMHhjYWIxMjAwMDpjcHUwKTogS1RSR1JBUEggZ3JvdXA6InRocmVhZCIsIGlkOiJr ZXJuZWwgdGlkIDEwMDAwMSIsIHN0YXRlOiJydW5xIGFkZCIsIGF0dHJpYnV0ZXM6IHByaW86ODQs IGxpbmtlZHRvOiJnX3VwIHRpZCAxMDAwMTMiCjYyMjYgKDB4Y2FiMTIwMDA6Y3B1MCk6IHNsZWVw cV93YWtldXA6IHRocmVhZCAweGNhYjE1MDAwIChwaWQgMSwga2VybmVsKQo2MjI1ICgweGNhYjEy MDAwOmNwdTApOiBzbGVlcHFfYnJvYWRjYXN0KDB4ZGVmMTg2NzgsIDApCjYyMjQgKDB4Y2I2MWY3 ODA6Y3B1Myk6IFdNX0FEREVEIFsvaG9tZS93bWEvZ2l0L3VuZGlzY2xvc2VkZGV2aWNlL3N5cy9r ZXJuL3NjaGVkXzRic2QuYzo5OTJdOiBvbmNwdSA9IDI1NSwgY3AxNWlkID0gMwo2MjIzICgweGNi NjFmNzgwOmNwdTMpOiBLVFJHUkFQSCBncm91cDoibG9hZCIsIGlkOiJnbG9iYWwgbG9hZCIsIGNv dW50ZXI6MSwgYXR0cmlidXRlczogbm9uZQo2MjIyICgweGNiNjFmNzgwOmNwdTMpOiBLVFJHUkFQ SCBncm91cDoidGhyZWFkIiwgaWQ6Im1kMCB0aWQgMTAwMDI3Iiwgc3RhdGU6InNsZWVwIiwgYXR0 cmlidXRlczogcHJpbzo5Miwgd21lc2c6Im1kd2FpdCIsIGxvY2tuYW1lOiIobnVsbCkiCjYyMjEg KDB4Y2I2MWY3ODA6Y3B1Myk6IG1pX3N3aXRjaDogb2xkIHRocmVhZCAxMDAwMjcgKHRkX3NjaGVk IDB4Y2I2MWZhZjgsIHBpZCAzLCBtZDApCjYyMjAgKDB4Y2I2MWY3ODA6Y3B1Myk6IEtUUkdSQVBI IGdyb3VwOiJ0aHJlYWQiLCBpZDoibWQwIHRpZCAxMDAwMjciLCBwb2ludDoicHJpb3JpdHkgY2hh bmdlIiwgYXR0cmlidXRlczogcHJpbzo5MiwgbmV3IHByaW86OTIsIGxpbmtlZHRvOiJtZDAgdGlk IDEwMDAyNyIKNjIxOSAoMHhjYWIxNDc4MDpjcHUxKTogbWlfc3dpdGNoOiBuZXcgdGhyZWFkIDEw MDAwMyAodGRfc2NoZWQgMHhjYWIxNGFmOCwgcGlkIDEwLCBpZGxlOiBjcHUxKQo2MjE4ICgweGNh YjE0NzgwOmNwdTEpOiBLVFJHUkFQSCBncm91cDoidGhyZWFkIiwgaWQ6ImlkbGU6IGNwdTEgdGlk IDEwMDAwMyIsIHN0YXRlOiJydW5uaW5nIiwgYXR0cmlidXRlczogcHJpbzoyNTUKNjIxNyAoMHhj YWIxNDc4MDpjcHUxKTogV01fQURERUQgWy9ob21lL3dtYS9naXQvdW5kaXNjbG9zZWRkZXZpY2Uv c3lzL2tlcm4vc2NoZWRfNGJzZC5jOjEwOTFdOiBvbmNwdSA9IDEsIGNwMTVpZCA9IDEKNjIxNiAo MHhjYWIxNDc4MDpjcHUxKTogV01fQURERUQgWy9ob21lL3dtYS9naXQvdW5kaXNjbG9zZWRkZXZp Y2Uvc3lzL2tlcm4vc2NoZWRfNGJzZC5jOjk5Ml06IG9uY3B1ID0gMjU1LCBjcDE1aWQgPSAxCjYy MTUgKDB4Y2FiMTQ3ODA6Y3B1MSk6IEtUUkdSQVBIIGdyb3VwOiJ0aHJlYWQiLCBpZDoiaWRsZTog Y3B1MSB0aWQgMTAwMDAzIiwgc3RhdGU6ImlkbGUiLCBhdHRyaWJ1dGVzOiBwcmlvOjI1NQo2MjE0 ICgweGNhYjE0NzgwOmNwdTEpOiBtaV9zd2l0Y2g6IG9sZCB0aHJlYWQgMTAwMDAzICh0ZF9zY2hl ZCAweGNhYjE0YWY4LCBwaWQgMTAsIGlkbGU6IGNwdTEpCjYyMTMgKDB4Y2FiMTIwMDA6Y3B1MCk6 IHNsZWVwcSByZXN1bWU6IHRocmVhZCAweGNhYjEyMDAwIChwaWQgMTIsIGdfdXApCjYyMTIgKDB4 Y2FiMTIwMDA6Y3B1MCk6IG1pX3N3aXRjaDogbmV3IHRocmVhZCAxMDAwMTMgKHRkX3NjaGVkIDB4 Y2FiMTIzNzgsIHBpZCAxMiwgZ191cCkKNjIxMSAoMHhjYWIxMjAwMDpjcHUwKTogS1RSR1JBUEgg Z3JvdXA6InRocmVhZCIsIGlkOiJnX3VwIHRpZCAxMDAwMTMiLCBzdGF0ZToicnVubmluZyIsIGF0 dHJpYnV0ZXM6IHByaW86OTIKNjIxMCAoMHhjYWIxMjAwMDpjcHUwKTogV01fQURERUQgWy9ob21l L3dtYS9naXQvdW5kaXNjbG9zZWRkZXZpY2Uvc3lzL2tlcm4vc2NoZWRfNGJzZC5jOjEwOTFdOiBv bmNwdSA9IDAsIGNwMTVpZCA9IDAKNjIwOSAoMHhjYWIxNGI0MDpjcHUwKTogV01fQURERUQgWy9o b21lL3dtYS9naXQvdW5kaXNjbG9zZWRkZXZpY2Uvc3lzL2tlcm4vc2NoZWRfNGJzZC5jOjk5Ml06 IG9uY3B1ID0gMjU1LCBjcDE1aWQgPSAwCjYyMDggKDB4Y2FiMTRiNDA6Y3B1MCk6IEtUUkdSQVBI IGdyb3VwOiJ0aHJlYWQiLCBpZDoiaWRsZTogY3B1MCB0aWQgMTAwMDAyIiwgc3RhdGU6ImlkbGUi LCBhdHRyaWJ1dGVzOiBwcmlvOjI1NQo2MjA3ICgweGNhYjE0YjQwOmNwdTApOiBtaV9zd2l0Y2g6 IG9sZCB0aHJlYWQgMTAwMDAyICh0ZF9zY2hlZCAweGNhYjE0ZWI4LCBwaWQgMTAsIGlkbGU6IGNw dTApCjYyMDYgKDB4Y2I2MWY3ODA6Y3B1Myk6IHNsZWVwOiB0aHJlYWQgMTAwMDI3IChwaWQgMywg bWQwKSBvbiBtZHdhaXQgKDB4Y2I2MTgwMDApCmRiPiBzaG93IHZ0b3AgMHg4MDAwClBoeXNpY2Fs IGFkZHJlc3MgcmVnOiAweDAwMDAwMDBmCmRiPiBzaG93IGNwMTUKQ3B1IElEOiAweDQxMmZjMGYz CkN1cnJlbnQgQ2FjaGUgTHZsIElEOiAweDg0NDRjMDA0CkN0cmw6IDB4MDBjNTM4N2YKQXV4IEN0 cmw6IDB4ODAwMDAwMDAKUHJvY2Vzc29yIEZlYXQgMDogMHgwMDAwMTEzMQpQcm9jZXNzb3IgRmVh dCAxOiAweDAwMDExMDExCkRlYnVnIEZlYXQgMDogMHgwMjAxMDU1NQpBdXhpbGlhcnkgRmVhdCAw OiAweDAwMDAwMDAwCk1lbW9yeSBNb2RlbCBGZWF0IDA6IDB4MTAyMDExMDUKTWVtb3J5IE1vZGVs IEZlYXQgMTogMHgyMDAwMDAwMApNZW1vcnkgTW9kZWwgRmVhdCAyOiAweDAxMjQwMDAwCk1lbW9y eSBNb2RlbCBGZWF0IDM6IDB4MDIxMDIyMTEKVFRCMDogMHgwMTJlODA2YQpkYj4gc2hvdyBwcm9j IDEKUHJvY2VzcyAxIChrZXJuZWwpIGF0IDB4Y2FiMTEwMDA6CiBzdGF0ZTogTk9STUFMCiB1aWQ6 IDAgIGdpZHM6IDAKIHBhcmVudDogcGlkIDAgYXQgMHhjMTJkMGZjOAogQUJJOiBGcmVlQlNEIEVM RjMyCiBwbWFwID0gMHhjMTNmY2FiYwogdGhyZWFkczogMQoxMDAwMDEgICAgICAgICAgICAgICAg ICAgUnVuICAgICBDUFUgMiAgICAgICAgICAgICAgIFtrZXJuZWxdIFRUQj0weDEyZTgwMDAgREFD Uj0weDUKZGI+IHNob3cgdnRvcCAweDgwMDAKUGh5c2ljYWwgYWRkcmVzcyByZWc6IDB4MDAwMDAw MGYKZGI+IGFybXY3X3RsYl9mbHVzaElECk5vIHN1Y2ggY29tbWFuZApkYj4gY2FsbCBhcm12N190 bGJfZmx1c2hJRAo9IDAKZGI+IHNob3cgdnRvcCAweDgwMDAgICAgICAKUGh5c2ljYWwgYWRkcmVz cyByZWc6IDB4MDIxNWEyZDQKZGI+IAo= --089e0122e75a44a31404ec5168a9--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CANsEV8ffGzq0P8BfyahK=w-b6G68Y-d8U01SseaPEGadH%2BdO8A>