Date: Sat, 30 Nov 2002 19:56:42 +0100 (CET) From: Michal Mertl <mime@traveller.cz> To: current@freebsd.org Subject: Re: system locks with vnode backed md(4) Message-ID: <Pine.BSF.4.41.0211301949200.88595-200000@prg.traveller.cz>
next in thread | raw e-mail | index | archive | help
This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. Send mail to mime@docserver.cac.washington.edu for more info. --0-1318642693-1038682138=:88595 Content-Type: TEXT/PLAIN; CHARSET=US-ASCII Content-ID: <Pine.BSF.4.41.0211301949202.88595@prg.traveller.cz> Ok, I got another one. DDB output attached. I did all kinds of operations to trigger it - had 3 md mounted from the same dir, in 2 of them doing my ports.tgz torture test and in root file system I had 'find . -inum 1231231' running. One find finished succesfully but then it finally locked-up. > Yeah, vnode locks and other lockmgr locks don't show up in 'show locks', > since only SMPng locking primitives are tracked by WITNESS. I see. > There are a fair number of vnode locking deadlock scenarios that are > unavoidable where we rely on grabbing vnode locks out of the directory > structure lock order. This occurs for vnode-backed md devices, quotas, > and UFS1 extended attributes, and probably some other situations. I > suspect that Terry is correct that operations on the vnode backing file > storage directory are triggering the problem, since that increases the > chances that a vnode lock "race to root" will occur from both the file > system backed into the md device, and for the md backing vnodes during > blocking I/O. If you can avoid directory operations on the md backing > directory, that would probably be one way to avoid triggering the bug. I'm afraid this doesn't sound too good for me. > Seeing it reproduced would probably confirm that this is the case. On the > other hand, there may be other deadlocks in the vnode/ufs/md code that can > be more easily corrected than this general VFS problem, so details there > would be very useful. May be the attached one will allow someone to track something down. PS: Sorry if you have problems with attachment, I myself find them difficult to read (I'm receivind digest of this list - isn't there a possibility to have it in mime form (like Buqtraq and others)? -- Michal Mertl mime@traveller.cz --0-1318642693-1038682138=:88595 Content-Type: TEXT/PLAIN; charset=US-ASCII; name="md-deadlock.txt" Content-Transfer-Encoding: BASE64 Content-ID: <Pine.BSF.4.41.0211301956420.88595@prg.traveller.cz> Content-Description: Content-Disposition: attachment; filename="md-deadlock.txt" ZGI+IHNob3cgbG9ja2Vkdm5vZHMNCkxvY2tlZCB2bm9kZXMNCjB4YzNlZjhi OTA6IHRhZyB1ZnMsIHR5cGUgVkRJUiwgdXNlY291bnQgMSwgd3JpdGVjb3Vu dCAwLCByZWZjb3VudCAxLCBmbGFncyAoVlZfT0JKQlVGKSwgbG9jayB0eXBl IHVmczogRVhDTCAoY291bnQgMSkgYnkgcGlkIDE5OTMNCiAgICAgICAgaW5v IDY1OTMsIG9uIGRldiBhZDBzMWEgKDQsIDQpDQoweGMzZWZlYTY4OiB0YWcg dWZzLCB0eXBlIFZSRUcsIHVzZWNvdW50IDIsIHdyaXRlY291bnQgMCwgcmVm Y291bnQgMCwgZmxhZ3MgKFZWX09CSkJVRiksIGxvY2sgdHlwZSB1ZnM6IEVY Q0wgKGNvdW50IDEpIGJ5IHBpZCA1OTkNCiAgICAgICAgaW5vIDY1OTYsIG9u IGRldiBhZDBzMWEgKDQsIDQpDQoweGM0YWM3ODE4OiB0YWcgZGV2ZnMsIHR5 cGUgVkNIUiwgdXNlY291bnQgMTI5Nywgd3JpdGVjb3VudCAwLCByZWZjb3Vu dCAzMCwgZmxhZ3MgKFZWX09CSkJVRiksIGxvY2sgdHlwZSBkZXZmczogRVhD TCAoY291bnQgMSkgYnkgcGlkIDgNCiAgICAgICAgMHhjNTc0ZjI1MDogdGFn IHVmcywgdHlwZSBWUkVHLCB1c2Vjb3VudCAyLCB3cml0ZWNvdW50IDEsIHJl ZmNvdW50IDgsIGZsYWdzIChWVl9PQkpCVUYpLCBsb2NrIHR5cGUgdWZzOiBF WENMIChjb3VudCAxKSBieSBwaWQgMTkxOQ0KICAgICAgICBpbm8gNywgb24g ZGV2IGFkMHMyZSAoNCwgMTApDQoweGMzZjNjOTQwOiB0YWcgdWZzLCB0eXBl IFZSRUcsIHVzZWNvdW50IDIsIHdyaXRlY291bnQgMSwgcmVmY291bnQgMjI1 LCBmbGFncyAoVlZfT0JKQlVGKSwgbG9jayB0eXBlIHVmczogRVhDTCAoY291 bnQgMSkgYnkgcGlkIDQzOQ0KICAgICAgICBpbm8gMywgb24gZGV2IGFkMHMy ZSAoNCwgMTApDQoweGM1NTlkOTQwOiB0YWcgdWZzLCB0eXBlIFZSRUcsIHVz ZWNvdW50IDAsIHdyaXRlY291bnQgMCwgcmVmY291bnQgMCwgZmxhZ3MgKFZW X09CSkJVRiksIGxvY2sgdHlwZSB1ZnM6IEVYQ0wgKGNvdW50IDEpIGJ5IHBp ZCAxOTY1DQogICAgICAgIGlubyA0LCBvbiBkZXYgYWQwczJlICg0LCAxMCkN CjB4YzVjODRjYjg6IHRhZyB1ZnMsIHR5cGUgVlJFRywgdXNlY291bnQgMiwg d3JpdGVjb3VudCAxLCByZWZjb3VudCA0MjYsIGZsYWdzIChWVl9PQkpCVUYp LCBsb2NrIHR5cGUgdWZzOiBFWENMIChjb3VudCAxKSBieSBwaWQgODEwDQog ICAgICAgIGlubyA1LCBvbiBkZXYgYWQwczJlICg0LCAxMCkNCjB4YzRhNWEw MDA6IHRhZyB1ZnMsIHR5cGUgVkRJUiwgdXNlY291bnQgMCwgd3JpdGVjb3Vu dCAwLCByZWZjb3VudCAxLCBmbGFncyAoVlZfT0JKQlVGKSwgbG9jayB0eXBl IHVmczogRVhDTCAoY291bnQgMSkgYnkgcGlkIDE5OTANCiAgICAgICAgaW5v IDk2OTg4LCBvbiBkZXYgbWQwICg0LCAxMikNCjB4YzQ0NGYxMjg6IHRhZyB1 ZnMsIHR5cGUgVkRJUiwgdXNlY291bnQgMSwgd3JpdGVjb3VudCAwLCByZWZj b3VudCAyLCBsb2NrIHR5cGUgdWZzOiBFWENMIChjb3VudCAxKSBieSBwaWQg MTk2NA0KICAgICAgICBpbm8gMTQzMzAsIG9uIGRldiBtZDAgKDQsIDEyKQ0K MHhjNDQ1NDgxODogdGFnIHVmcywgdHlwZSBWTk9OLCB1c2Vjb3VudCAxLCB3 cml0ZWNvdW50IDAsIHJlZmNvdW50IDAsIGxvY2sgdHlwZSB1ZnM6IEVYQ0wg KGNvdW50IDEpIGJ5IHBpZCAxOTY0DQogICAgICAgIGlubyAxNDMzNiwgb24g ZGV2IG1kMCAoNCwgMTIpDQoweGM1MTgzZGUwOiB0YWcgdWZzLCB0eXBlIFZE SVIsIHVzZWNvdW50IDAsIHdyaXRlY291bnQgMCwgcmVmY291bnQgMiwgZmxh Z3MgKFZWX09CSkJVRiksIGxvY2sgdHlwZSB1ZnM6IEVYQ0wgKGNvdW50IDEp IGJ5IHBpZCAxOTkyDQogICAgICAgIGlubyA1MDkwLCBvbiBkZXYgbWQxICg0 LCAxNCkNCmRiPiBzaG93IGxvY2tzDQpleGNsdXNpdmUgc2xlZXAgbXV0ZXgg R2lhbnQgciA9IDAgKDB4YzAzOTk2NjApIGxvY2tlZCBAIC4uLy4uLy4uL2tl cm4va2Vybl9pbnRyLmM6NTM0DQpkYj4gcHMNCiAgcGlkICAgcHJvYyAgICAg YWRkciAgICB1aWQgIHBwaWQgIHBncnAgIGZsYWcgICBzdGF0ICB3bWVzZyAg ICB3Y2hhbiAgY21kDQogMTk5MyBjNDgxOTkzOCBlMTc5NzAwMCAgICAwICAx NzM3ICAxOTkzIDAwMDQwMDIgbm9ybVtTTFBRICBuZXdidWYgYzAzZWRlMTBd W1NMUF0gZmluZA0KIDE5OTIgYzQ4MWIzYjAgZTE3OWMwMDAgICAgMCAgIDU0 NyAgMTk5MiAwMDA0MDAyIG5vcm1bU0xQUSAgZ2V0YmxrIGNlM2VhYmI4XVtT TFBdIHJtDQogMTk5MCBjNDgxOWIxMCBlMTc5ODAwMCAgICAwICAxOTYyICAx OTkwIDAwMDQwMDIgbm9ybVtTTFBRICBnZXRibGsgY2UyZDFiYTRdW1NMUF0g Y3ANCiAxOTY1IGM0ODE5MDAwIGUxNzRlMDAwICAgIDAgIDE5NjQgIDE5NjQg MDAwNjAwMiBub3JtW1NMUFEgIG5ld2J1ZiBjMDNlZGUxMF1bU0xQXSBnemlw DQogMTk2NCBjNDgxYjkzOCBlMTc5ZjAwMCAgICAwICAgNDM2ICAxOTY0IDAw MDQwMDIgbm9ybVtTTFBRICAgYmlvcmQgY2UyZjc0M2NdW1NMUF0gdGFyDQog MTk2MiBjNGI1NzAwMCBlMTdjMzAwMCAgICAwICAxOTU4ICAxOTYyIDIwMDQw MDIgbm9ybVtTTFBRICAgcGF1c2UgZTE3YzMwMDBdW1NMUF0gY3NoDQogMTk1 OCBjNGI1NzFkOCBlMTdjNDAwMCAxMDAxICAxNzc5ICAxOTU4IDAwMDQxMDIg bm9ybVtTTFBRICAgIHdhaXQgYzRiNTcxZDhdW1NMUF0gc3UNCiAxOTE5IGM0 ODE5M2IwIGUxNzVlMDAwICAgIDAgICAgIDAgICAgIDAgMDAwMDIwNCBub3Jt W1NMUFEgIG5ld2J1ZiBjMDNlZGUxMF1bU0xQXSBtZDINCiAxNzc5IGM0ODE5 NTg4IGUxNzVmMDAwIDEwMDEgIDE3NzggIDE3NzkgMjAwNDAwMiBub3JtW1NM UFEgICBwYXVzZSBlMTc1ZjAwMF1bU0xQXSB0Y3NoDQogMTc3OCBjNGI1OTAw MCBlMTdjYjAwMCAxMDAxICAxNzc1ICAgMzYwIDAwMDAxMDAgbm9ybVtDVlEg IHNlbGVjdCBjMDM5Y2JlNF1bU0xQXSBzc2hkDQogMTc3NSBjNGI1OTU4OCBl MTgwNDAwMCAgICAwICAgMzYwICAgMzYwIDAwMDAxMDAgbm9ybVtTTFBRICBz YndhaXQgYzNmMDBlNjRdW1NMUF0gc3NoZA0KIDE3MzcgYzRiNTkxZDggZTE3 Y2MwMDAgICAgMCAgMTczNiAgMTczNyAyMDA0MDAyIG5vcm1bU0xQUSAgIHBh dXNlIGUxN2NjMDAwXVtTTFBdIGNzaA0KIDE3MzYgYzQ4MTkxZDggZTE3NGYw MDAgMTAwMSAgIDQ1OCAgMTczNiAwMDA0MTAyIG5vcm1bU0xQUSAgICB3YWl0 IGM0ODE5MWQ4XVtTTFBdIHN1DQogIDgxMCBjNGI1N2NlOCBlMTdjYTAwMCAg ICAwICAgICAwICAgICAwIDAwMDAyMDQgbm9ybVtTTFBRICBuZXdidWYgYzAz ZWRlMTBdW1NMUF0gbWQxDQogIDU5OSBjNGI1N2IxMCBlMTdjOTAwMCAgICAw ICAgNTk3ICAgNTk5IDAwMDQwMDIgbm9ybVtTTFBRICBuZXdidWYgYzAzZWRl MTBdW1NMUF0gc3lzdGF0DQogIDU5NyBjNGI1Nzc2MCBlMTdjNzAwMCAgICAw ICAgNDE1ICAgNTk3IDIwMDQwMDIgbm9ybVtTTFBRICAgcGF1c2UgZTE3Yzcw MDBdW1NMUF0gY3NoDQogIDU0NyBjNDgxOTc2MCBlMTc2MDAwMCAgICAwICAg NTQ1ICAgNTQ3IDIwMDQwMDIgbm9ybVtTTFBRICAgcGF1c2UgZTE3NjAwMDBd W1NMUF0gY3NoDQogIDU0NSBjNDAzYWNlOCBlMTcxMTAwMCAxMDAxICAgNTEy ICAgNTQ1IDAwMDQxMDIgbm9ybVtTTFBRICAgIHdhaXQgYzQwM2FjZThdW1NM UF0gc3UNCiAgNTEyIGM0ODFiNzYwIGUxNzllMDAwIDEwMDEgICA1MTEgICA1 MTIgMjAwNDAwMiBub3JtW1NMUFEgICBwYXVzZSBlMTc5ZTAwMF1bU0xQXSB0 Y3NoDQogIDUxMSBjM2U0YTFkOCBlMDRkZDAwMCAxMDAxICAgNTA4ICAgMzYw IDAwMDAxMDAgbm9ybVtDVlEgIHNlbGVjdCBjMDM5Y2JlNF1bU0xQXSBzc2hk DQogIDUwOCBjNDgxYjAwMCBlMTc5YTAwMCAgICAwICAgMzYwICAgMzYwIDAw MDAxMDAgbm9ybVtTTFBRICBzYndhaXQgYzNlZmY3NjRdW1NMUF0gc3NoZA0K ICA0NTggYzNlNGQ3NjAgZTA1MWUwMDAgMTAwMSAgIDQ1NyAgIDQ1OCAyMDA0 MDAyIG5vcm1bU0xQUSAgIHBhdXNlIGUwNTFlMDAwXVtTTFBdIHRjc2gNCiAg NDU3IGMzZTRhM2IwIGUwNGRlMDAwIDEwMDEgICA0NTQgICAzNjAgMDAwMDEw MCBub3JtW0NWUSAgc2VsZWN0IGMwMzljYmU0XVtTTFBdIHNzaGQNCiAgNDU0 IGMzZDI3YjEwIGUwNGQyMDAwICAgIDAgICAzNjAgICAzNjAgMDAwMDEwMCBu b3JtW1NMUFEgIHNid2FpdCBjM2VmZjk2NF1bU0xQXSBzc2hkDQogIDQzOSBj M2U0ZDNiMCBlMDUxYzAwMCAgICAwICAgICAwICAgICAwIDAwMDAyMDQgbm9y bVtTTFBRICBuZXdidWYgYzAzZWRlMTBdW1NMUF0gbWQwDQogIDQzNiBjNDAz YTAwMCBlMTcwYTAwMCAgICAwICAgNDM1ICAgNDM2IDIwMDQwMDIgbm9ybVtT TFBRICAgcGF1c2UgZTE3MGEwMDBdW1NMUF0gY3NoDQogIDQzNSBjM2U0ZGNl OCBlMDUyMTAwMCAxMDAxICAgNDI3ICAgNDM1IDAwMDQxMDIgbm9ybVtTTFBR ICAgIHdhaXQgYzNlNGRjZThdW1NMUF0gc3UNCiAgNDI3IGMzZTRkOTM4IGUw NTFmMDAwIDEwMDEgICA0MjYgICA0MjcgMjAwNDAwMiBub3JtW1NMUFEgICBw YXVzZSBlMDUxZjAwMF1bU0xQXSB0Y3NoDQogIDQyNiBjM2U0ZGIxMCBlMDUy MDAwMCAxMDAxICAgNDIzICAgMzYwIDAwMDAxMDAgbm9ybVtDVlEgIHNlbGVj dCBjMDM5Y2JlNF1bU0xQXSBzc2hkDQogIDQyMyBjNDAzYTFkOCBlMTcwYjAw MCAgICAwICAgMzYwICAgMzYwIDAwMDAxMDAgbm9ybVtTTFBRICBzYndhaXQg YzNlZmYyNjRdW1NMUF0gc3NoZA0KICA0MjIgYzQwM2EzYjAgZTE3MGMwMDAg ICAgMCAgICAgMSAgIDQyMiAwMDA0MDAyIG5vcm1bU0xQUSAgIHR0eWluIGMz ZmQ4NjEwXVtTTFBdIGdldHR5DQogIDQyMSBjNDAzYTU4OCBlMTcwZDAwMCAg ICAwICAgICAxICAgNDIxIDAwMDQwMDIgbm9ybVtTTFBRICAgdHR5aW4gYzNm ZDhhMTBdW1NMUF0gZ2V0dHkNCiAgNDIwIGM0MDNhNzYwIGUxNzBlMDAwICAg IDAgICAgIDEgICA0MjAgMDAwNDAwMiBub3JtW1NMUFEgICB0dHlpbiBjM2Zk OGUxMF1bU0xQXSBnZXR0eQ0KICA0MTkgYzQwM2E5MzggZTE3MGYwMDAgICAg MCAgICAgMSAgIDQxOSAwMDA0MDAyIG5vcm1bU0xQUSAgIHR0eWluIGMzZmQ5 MjEwXVtTTFBdIGdldHR5DQogIDQxOCBjNDAzYWIxMCBlMTcxMDAwMCAgICAw ICAgICAxICAgNDE4IDAwMDQwMDIgbm9ybVtTTFBRICAgdHR5aW4gYzNlZmEy MTBdW1NMUF0gZ2V0dHkNCiAgNDE3IGMzZTRhNTg4IGUwNGRmMDAwICAgIDAg ICAgIDEgICA0MTcgMDAwNDAwMiBub3JtW1NMUFEgICB0dHlpbiBjM2U0N2Mx MF1bU0xQXSBnZXR0eQ0KICA0MTYgYzNlNGFiMTAgZTA0ZTIwMDAgICAgMCAg ICAgMSAgIDQxNiAwMDA0MDAyIG5vcm1bU0xQUSAgIHR0eWluIGMzZTQ3ODEw XVtTTFBdIGdldHR5DQogIDQxNSBjM2U0YTc2MCBlMDRlMDAwMCAgICAwICAg ICAxICAgNDE1IDAwMDQxMDIgbm9ybVtTTFBRICAgIHdhaXQgYzNlNGE3NjBd W1NMUF0gbG9naW4NCiAgNDA1IGMzZTRhMDAwIGUwNGRjMDAwICAgIDAgICAg IDEgICA0MDUgMDAwMDAwMCBub3JtW1NMUFEgIG5hbnNscCBjMDNjOGUzNF1b U0xQXSBjcm9uDQogIDM2MCBjM2U0YTkzOCBlMDRlMTAwMCAgICAwICAgICAx ICAgMzYwIDAwMDAxMDAgbm9ybVtDVlEgIHNlbGVjdCBjMDM5Y2JlNF1bU0xQ XSBzc2hkDQogIDIxMCBjM2QyN2NlOCBlMDRkMzAwMCAgICAwICAgICAxICAg MjEwIDAwMDAwMDAgbm9ybVtDVlEgIHNlbGVjdCBjMDM5Y2JlNF1bU0xQXSBz eXNsb2dkDQogICAzMyBjM2U0YWNlOCBlMDRlMzAwMCAgICAwICAgICAwICAg ICAwIDAwMDAyMDQgbm9ybVtTTFBRICB2bHJ1d3QgYzNlNGFjZThdW1NMUF0g dm5scnUNCiAgICA5IGMzZTRkMDAwIGUwNTFhMDAwICAgIDAgICAgIDAgICAg IDAgMDAwMDIwNCBub3JtW1NMUFEgIG5ld2J1ZiBjMDNlZGUxMF1bU0xQXSBz eW5jZXINCiAgICA4IGMzZTRkMWQ4IGUwNTFiMDAwICAgIDAgICAgIDAgICAg IDAgMDAwMDIwNCBub3JtW1NMUFEgIHdkcmFpbiBjMDNlZGUwY11bU0xQXSBi dWZkYWVtb24NCiAgICA3IGMzY2MwNTg4IGQ3ODFiMDAwICAgIDAgICAgIDAg ICAgIDAgMDAwMDIwYyBub3JtW1NMUFEgIHBnemVybyBjMDNlZjJhNF1bU0xQ XSBwYWdlemVybw0KICAgIDYgYzNjYzA3NjAgZDc4NTIwMDAgICAgMCAgICAg MCAgICAgMCAwMDAwMjA0IG5vcm1bU0xQUSAgcHNsZWVwIGMwM2VmMmJjXVtT TFBdIHZtZGFlbW9uDQogICAgNSBjM2NjMDkzOCBkNzg1MzAwMCAgICAwICAg ICAwICAgICAwIDAwMDAyMDQgbm9ybVtTTFBRICBwc2xlZXAgYzAzYTk3MThd W1NMUF0gcGFnZWRhZW1vbg0KICAgMzIgYzNjYzBiMTAgZDc4NTQwMDAgICAg MCAgICAgMCAgICAgMCAwMDAwMjA0IG5ldyBbSVdBSVRdIGlycTg6IHJ0Yw0K ICAgMzEgYzNjYzBjZTggZDc4NTUwMDAgICAgMCAgICAgMCAgICAgMCAwMDAw MjA0IG5ldyBbSVdBSVRdIGlycTA6IGNsaw0KICAgMzAgYzNkMjcwMDAgZTA0 Y2MwMDAgICAgMCAgICAgMCAgICAgMCAwMDAwMjA0IG5ldyBbSVdBSVRdIGly cTQ6IHNpbzANCiAgIDI5IGMzZDI3MWQ4IGUwNGNkMDAwICAgIDAgICAgIDAg ICAgIDAgMDAwMDIwNCBub3JtW0lXQUlUXSBzd2kwOiB0dHk6c2lvDQogICAy OCBjM2QyNzNiMCBlMDRjZTAwMCAgICAwICAgICAwICAgICAwIDAwMDAyMDQg bm9ybVtDUFUgMF0gaXJxMTogYXRrYmQwDQogICAyNyBjM2QyNzU4OCBlMDRj ZjAwMCAgICAwICAgICAwICAgICAwIDAwMDAyMDQgbmV3IFtJV0FJVF0gaXJx MTU6IGF0YTENCiAgIDI2IGMzZDI3NzYwIGUwNGQwMDAwICAgIDAgICAgIDAg ICAgIDAgMDAwMDIwNCBub3JtW0lXQUlUXSBpcnExNDogYXRhMA0KICAgMjUg YzNkMjc5MzggZTA0ZDEwMDAgICAgMCAgICAgMCAgICAgMCAwMDAwMjA0IG5l dyBbSVdBSVRdIGlycTU6IHBjbTANCiAgIDI0IGMxMjBlMWQ4IGQ2NjQwMDAw ICAgIDAgICAgIDAgICAgIDAgMDAwMDIwNCBub3JtW0lXQUlUXSBpcnExMDog cmwwIGJrdHIzKysrDQogICAyMyBjMTIwZTNiMCBkNjY0MTAwMCAgICAwICAg ICAwICAgICAwIDAwMDAyMDQgbmV3IFtJV0FJVF0gaXJxNjogYmt0cjIgYmt0 cjUrKw0KICAgMjIgYzEyMGU1ODggZDY2NDIwMDAgICAgMCAgICAgMCAgICAg MCAwMDAwMjA0IG5ldyBbSVdBSVRdIGlycTc6IGJrdHIxIGJrdHI0KysNCiAg IDIxIGMxMjBlNzYwIGQ2NjQzMDAwICAgIDAgICAgIDAgICAgIDAgMDAwMDIw NCBuZXcgW0lXQUlUXSBpcnExMTogYmt0cjAgYmt0cjcqDQogICAyMCBjMTIw ZTkzOCBkNjY0NDAwMCAgICAwICAgICAwICAgICAwIDAwMDAyMDQgbmV3IFtJ V0FJVF0gaXJxMTM6DQogICAxOSBjMTIwZWIxMCBkNjY0NTAwMCAgICAwICAg ICAwICAgICAwIDAwMDAyMDQgbmV3IFtJV0FJVF0gc3dpNTogYWNwaXRhc2tx DQogICAxOCBjMTIwZWNlOCBkNjY0NjAwMCAgICAwICAgICAwICAgICAwIDAw MDAyMDQgbmV3IFtJV0FJVF0gc3dpMzogY2FtYmlvDQogICAxNyBjM2NjMDAw MCBkNzgxODAwMCAgICAwICAgICAwICAgICAwIDAwMDAyMDQgbmV3IFtJV0FJ VF0gc3dpMjogY2FtbmV0DQogICAxNiBjM2NjMDFkOCBkNzgxOTAwMCAgICAw ICAgICAwICAgICAwIDAwMDAyMDQgbmV3IFtJV0FJVF0gc3dpNTogdGFzayBx dWV1ZQ0KICAgMTUgYzNjYzAzYjAgZDc4MWEwMDAgICAgMCAgICAgMCAgICAg MCAwMDAwMjA0IG5vcm1bU0xQUSAgIHNsZWVwIGMwM2I1MDQwXVtTTFBdIHJh bmRvbQ0KICAgIDQgYzEyMDcwMDAgZDY1Y2IwMDAgICAgMCAgICAgMCAgICAg MCAwMDAwMjA0IG5vcm1bU0xQUSAgZ19kb3duIGMwMzkzNGIwXVtTTFBdIGdf ZG93bg0KICAgIDMgYzEyMDcxZDggZDY2MzgwMDAgICAgMCAgICAgMCAgICAg MCAwMDAwMjA0IG5vcm1bU0xQUSAgICBnX3VwIGMwMzkzNGFjXVtTTFBdIGdf dXANCiAgICAyIGMxMjA3M2IwIGQ2NjM5MDAwICAgIDAgICAgIDAgICAgIDAg MDAwMDIwNCBub3JtW1NMUFEgIGdfZXZlbnRzIGMwMzkzNGE0XVtTTFBdIGdf ZXZlbnQNCiAgIDE0IGMxMjA3NTg4IGQ2NjNhMDAwICAgIDAgICAgIDAgICAg IDAgMDAwMDIwNCBuZXcgW0lXQUlUXSBzd2k0OiB2bQ0KICAgMTMgYzEyMDc3 NjAgZDY2M2IwMDAgICAgMCAgICAgMCAgICAgMCAwMDAwMjBjIG5vcm1bUlVO UV0gc3dpNjogdHR5OnNpbyBjbG9jaw0KICAgMTIgYzEyMDc5MzggZDY2M2Mw MDAgICAgMCAgICAgMCAgICAgMCAwMDAwMjA0IG5vcm1bSVdBSVRdIHN3aTE6 IG5ldA0KICAgMTEgYzEyMDdiMTAgZDY2M2QwMDAgICAgMCAgICAgMCAgICAg MCAwMDAwMjBjIG5vcm1bQ2FuIHJ1bl0gaWRsZQ0KICAgIDEgYzEyMDdjZTgg ZDY2M2UwMDAgICAgMCAgICAgMCAgICAgMSAwMDA0MjAwIG5vcm1bU0xQUSAg ICB3YWl0IGMxMjA3Y2U4XVtTTFBdIGluaXQNCiAgIDEwIGMxMjBlMDAwIGQ2 NjNmMDAwICAgIDAgICAgIDAgICAgIDAgMDAwMDIwNCBub3JtW0NWUSAga3Ry YWNlIGMwM2M1MGM0XVtTTFBdIGt0cmFjZQ0KICAgIDAgYzAzOTQ3MjAgYzA0 ZmYwMDAgICAgMCAgICAgMCAgICAgMCAwMDAwMjAwIG5vcm1bU0xQUSAgIHNj aGVkIGMwMzk0NzIwXVtTTFBdIHN3YXBwZXINCg0K --0-1318642693-1038682138=:88595-- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-current" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?Pine.BSF.4.41.0211301949200.88595-200000>