Date: Tue, 27 Sep 2005 15:12:31 -0400 (EDT) From: Rob Watt <rob@hudson-trading.com> To: Robert Watson <rwatson@FreeBSD.org> Cc: Rob Watt <rob@hudson-trading.com>, freebsd-hackers@FreeBSD.org, mikep@hudson-trading.com, freebsd-amd64@FreeBSD.org, Jason Carroll <jason@hudson-trading.com> Subject: Re: freebsd-5.4-stable panics Message-ID: <20050927140535.G50334@daemon.mistermishap.net> In-Reply-To: <20050925115912.H11229@fledge.watson.org> References: <da4a53d805092310237d732554@mail.gmail.com> <20050925115912.H11229@fledge.watson.org>
next in thread | previous in thread | raw e-mail | index | archive | help
This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. Send mail to mime@docserver.cac.washington.edu for more info. --0-779721010-1127845675=:50334 Content-Type: TEXT/PLAIN; CHARSET=US-ASCII On Sun, 25 Sep 2005, Robert Watson wrote: > > On Fri, 23 Sep 2005, Jason Carroll wrote: > 5B > > There seem to be 2 types of crashes we see with pretty different stack > > traces. What I'll call a type 1 crash, I believe, is often caused by > > one of the triggers I mention above. A type 2 crash appears to happen > > spontaneously after the machine has been running for a while. > > > > I poked around using kgdb in a core file from a type 2 crash, and it > > appeared the system hung closing sockets (specifically cleaning up > > multicast state i think) while cleaning up one of our multicast > > applications (note the trace through sys_exit). There's no reason this > > application should have been exiting unless it encountered some kind of > > error. > > Sounds nasty. It's possible the two panics are related, especially if > they involve a race in the multicast code, which could result in treading > on other kernel memory, potentially leading to the thread related panic. > My leaning would be that they are unrelated, but since we may be able to > eliminate the multicast one (see below), that would be a good starting > point. > > There are some other known stability nits in 6.x which are being worked > on, but in general the network stack stability is higher in 6.x than 5.x > when it comes to multicast due to the work I reference above. If you run > into any stability problems relating to the file system, set > debug.mpsafevfs=0 in loader.conf -- there are a few bug fixes relating to > running out of disk space or hitting quota limits that are fixed in HEAD, > but not yet backported to 6.x. Robert, Thanks for your quick response and suggestions. We have now experienced an additional type of crash. Type 3 is from 6.0-BETA5, it did not enter the debugger at all and we could not generate a core. Unfortunately the 6-BETA crash was completely different from everything we've seen so far. The panic was related to a page fault and 'top' was the active process. We are trying again to run our tests on 6.0, but if we keep encountering other bugs, then those other bugs may prevent us from determining if multicast is the problem. We also ran our applications in 5-STABLE without reading from or writing to disk (ie we ran the multicast data streams on a remote machine, and we told our listener/rebroadcaster apps not to write to disk). In this configuration we were able to run for 4 days without crashing. A few hours before the crash we had introduced disk activity (bonnie in a constant loop with 1G test file size). This crash was a type 1, and we were not able to save a core. The longest we had gone before without a crash was 6 hours, so it is possible that either load, or disk activity help trigger the bugs we have seen. files attached: kernel-conf.txt (6.0 kernel) type3-core.txt (copy of panic output to console) We will update you with more info from our 6.0 tests when we have it. We are in a bind right now. All modern hardware (ie emt64/amd64) only seems to work with versions of freebsd that aren't stable when running our applications. Many vendors do not even sell server hardware that is purely i386. We never encountered these types of problems on freebsd 4.x, and many of our 120+ i386 class machines that are running 4.x are showing their age and need to be replaced. Assuming that the problems we are experiencing are purely related to ths OS, we now don't have an OS to run on the newer hardware we've been buying. We really need to find a way to patch these problems or find a version of freebsd that supports our platform and is stable. Obviously we appreciate the hard work that all of you on the freebsd team do, and we are happy to do whatever we can to help squash these bugs. - Rob Watt --0-779721010-1127845675=:50334 Content-Type: TEXT/PLAIN; CHARSET=US-ASCII; NAME="kernel-conf.txt" Content-Transfer-Encoding: BASE64 Content-Description: Content-Disposition: ATTACHMENT; FILENAME="kernel-conf.txt" Iw0KIyBHRU5FUklDIC0tIEdlbmVyaWMga2VybmVsIGNvbmZpZ3VyYXRpb24g ZmlsZSBmb3IgRnJlZUJTRC9hbWQ2NA0KIw0KIyBGb3IgbW9yZSBpbmZvcm1h dGlvbiBvbiB0aGlzIGZpbGUsIHBsZWFzZSByZWFkIHRoZSBoYW5kYm9vayBz ZWN0aW9uIG9uDQojIEtlcm5lbCBDb25maWd1cmF0aW9uIEZpbGVzOg0KIw0K IyAgICBodHRwOi8vd3d3LkZyZWVCU0Qub3JnL2RvYy9lbl9VUy5JU084ODU5 LTEvYm9va3MvaGFuZGJvb2sva2VybmVsY29uZmlnLWNvbmZpZy5odG1sDQoj DQojIFRoZSBoYW5kYm9vayBpcyBhbHNvIGF2YWlsYWJsZSBsb2NhbGx5IGlu IC91c3Ivc2hhcmUvZG9jL2hhbmRib29rDQojIGlmIHlvdSd2ZSBpbnN0YWxs ZWQgdGhlIGRvYyBkaXN0cmlidXRpb24sIG90aGVyd2lzZSBhbHdheXMgc2Vl IHRoZQ0KIyBGcmVlQlNEIFdvcmxkIFdpZGUgV2ViIHNlcnZlciAoaHR0cDov L3d3dy5GcmVlQlNELm9yZy8pIGZvciB0aGUNCiMgbGF0ZXN0IGluZm9ybWF0 aW9uLg0KIw0KIyBBbiBleGhhdXN0aXZlIGxpc3Qgb2Ygb3B0aW9ucyBhbmQg bW9yZSBkZXRhaWxlZCBleHBsYW5hdGlvbnMgb2YgdGhlDQojIGRldmljZSBs aW5lcyBpcyBhbHNvIHByZXNlbnQgaW4gdGhlIC4uLy4uL2NvbmYvTk9URVMg YW5kIE5PVEVTIGZpbGVzLg0KIyBJZiB5b3UgYXJlIGluIGRvdWJ0IGFzIHRv IHRoZSBwdXJwb3NlIG9yIG5lY2Vzc2l0eSBvZiBhIGxpbmUsIGNoZWNrIGZp cnN0DQojIGluIE5PVEVTLg0KIw0KIyAkRnJlZUJTRDogc3JjL3N5cy9hbWQ2 NC9jb25mL0dFTkVSSUMsdiAxLjQyMS4yLjExLjIuMSAyMDA1LzA0LzA5IDE3 OjI4OjM3IGtlbnNtaXRoIEV4cCAkDQoNCm1hY2hpbmUJCWFtZDY0DQpjcHUJ CUhBTU1FUg0KaWRlbnQJCUNVU1RPTQ0KDQojIFRvIHN0YXRpY2FsbHkgY29t cGlsZSBpbiBkZXZpY2Ugd2lyaW5nIGluc3RlYWQgb2YgL2Jvb3QvZGV2aWNl LmhpbnRzDQojaGludHMJCSJHRU5FUklDLmhpbnRzIgkJIyBEZWZhdWx0IHBs YWNlcyB0byBsb29rIGZvciBkZXZpY2VzLg0KbWFrZW9wdGlvbnMgICAgIERF QlVHPS1nDQpvcHRpb25zICAgICAgICAgS0RCDQpvcHRpb25zICAgICAgICAg RERCDQpvcHRpb25zICAgICAgICAgQlJFQUtfVE9fREVCVUdHRVINCm9wdGlv bnMgICAgICAgICBJTlZBUklBTlRTDQpvcHRpb25zICAgICAgICAgSU5WQVJJ QU5UX1NVUFBPUlQNCm9wdGlvbnMgICAgICAgICBXSVRORVNTDQpvcHRpb25z ICAgICAgICAgV0lUTkVTU19TS0lQU1BJTg0KI21ha2VvcHRpb25zICAgICBD T1BURkxBR1M9Ii1PIC1mcmVuYW1lLXJlZ2lzdGVycyAtcGlwZSINCg0KI29w dGlvbnMgICAgICAgIFNDSEVEX1VMRSAgICAgICAgICAgICAgICMgVUxFIHNj aGVkdWxlcg0Kb3B0aW9ucyAJU0NIRURfNEJTRAkJIyA0QlNEIHNjaGVkdWxl cg0Kb3B0aW9ucyAJSU5FVAkJCSMgSW50ZXJORVR3b3JraW5nDQpvcHRpb25z IAlJTkVUNgkJCSMgSVB2NiBjb21tdW5pY2F0aW9ucyBwcm90b2NvbHMNCm9w dGlvbnMgCUZGUwkJCSMgQmVya2VsZXkgRmFzdCBGaWxlc3lzdGVtDQpvcHRp b25zIAlTT0ZUVVBEQVRFUwkJIyBFbmFibGUgRkZTIHNvZnQgdXBkYXRlcyBz dXBwb3J0DQpvcHRpb25zIAlVRlNfQUNMCQkJIyBTdXBwb3J0IGZvciBhY2Nl c3MgY29udHJvbCBsaXN0cw0Kb3B0aW9ucyAJVUZTX0RJUkhBU0gJCSMgSW1w cm92ZSBwZXJmb3JtYW5jZSBvbiBiaWcgZGlyZWN0b3JpZXMNCm9wdGlvbnMg CU1EX1JPT1QJCQkjIE1EIGlzIGEgcG90ZW50aWFsIHJvb3QgZGV2aWNlDQpv cHRpb25zIAlORlNDTElFTlQJCSMgTmV0d29yayBGaWxlc3lzdGVtIENsaWVu dA0Kb3B0aW9ucyAJTkZTU0VSVkVSCQkjIE5ldHdvcmsgRmlsZXN5c3RlbSBT ZXJ2ZXINCm9wdGlvbnMgCU5GU19ST09UCQkjIE5GUyB1c2FibGUgYXMgLywg cmVxdWlyZXMgTkZTQ0xJRU5UDQpvcHRpb25zIAlOVEZTCQkJIyBOVCBGaWxl IFN5c3RlbQ0Kb3B0aW9ucyAJTVNET1NGUwkJCSMgTVNET1MgRmlsZXN5c3Rl bQ0Kb3B0aW9ucyAJQ0Q5NjYwCQkJIyBJU08gOTY2MCBGaWxlc3lzdGVtDQpv cHRpb25zIAlQUk9DRlMJCQkjIFByb2Nlc3MgZmlsZXN5c3RlbSAocmVxdWly ZXMgUFNFVURPRlMpDQpvcHRpb25zIAlQU0VVRE9GUwkJIyBQc2V1ZG8tZmls ZXN5c3RlbSBmcmFtZXdvcmsNCm9wdGlvbnMgCUdFT01fR1BUCQkjIEdVSUQg UGFydGl0aW9uIFRhYmxlcy4NCm9wdGlvbnMgCUNPTVBBVF80MwkJIyBOZWVk ZWQgYnkgQ09NUEFUX0xJTlVYMzINCm9wdGlvbnMgCUNPTVBBVF9JQTMyCQkj IENvbXBhdGlibGUgd2l0aCBpMzg2IGJpbmFyaWVzDQpvcHRpb25zIAlDT01Q QVRfRlJFRUJTRDQJCSMgQ29tcGF0aWJsZSB3aXRoIEZyZWVCU0Q0DQpvcHRp b25zICAgICAgICAgQ09NUEFUX0ZSRUVCU0Q1ICAgICAgICAgIyBDb21wYXRp YmxlIHdpdGggRnJlZUJTRDUNCm9wdGlvbnMgCUNPTVBBVF9MSU5VWDMyCQkj IENvbXBhdGlibGUgd2l0aCBpMzg2IGxpbnV4IGJpbmFyaWVzIA0Kb3B0aW9u cyAJU0NTSV9ERUxBWT01MDAwCSAgICAgICAgIyBEZWxheSAoaW4gbXMpIGJl Zm9yZSBwcm9iaW5nIFNDU0kNCm9wdGlvbnMgCUtUUkFDRQkJCSMga3RyYWNl KDEpIHN1cHBvcnQNCm9wdGlvbnMgCVNZU1ZTSE0JCQkjIFNZU1Ytc3R5bGUg c2hhcmVkIG1lbW9yeQ0Kb3B0aW9ucyAJU1lTVk1TRwkJCSMgU1lTVi1zdHls ZSBtZXNzYWdlIHF1ZXVlcw0Kb3B0aW9ucyAJU1lTVlNFTQkJCSMgU1lTVi1z dHlsZSBzZW1hcGhvcmVzDQpvcHRpb25zIAlfS1BPU0lYX1BSSU9SSVRZX1ND SEVEVUxJTkcgIyBQT1NJWCBQMTAwM18xQiByZWFsLXRpbWUgZXh0ZW5zaW9u cw0Kb3B0aW9ucyAJS0JEX0lOU1RBTExfQ0RFVgkjIGluc3RhbGwgYSBDREVW IGVudHJ5IGluIC9kZXYNCm9wdGlvbnMgCUFIQ19SRUdfUFJFVFRZX1BSSU5U CSMgUHJpbnQgcmVnaXN0ZXIgYml0ZmllbGRzIGluIGRlYnVnDQoJCQkJCSMg b3V0cHV0LiAgQWRkcyB+MTI4ayB0byBkcml2ZXIuDQpvcHRpb25zIAlBSERf UkVHX1BSRVRUWV9QUklOVAkjIFByaW50IHJlZ2lzdGVyIGJpdGZpZWxkcyBp biBkZWJ1Zw0KCQkJCQkjIG91dHB1dC4gIEFkZHMgfjIxNWsgdG8gZHJpdmVy Lg0Kb3B0aW9ucyAJQURBUFRJVkVfR0lBTlQJCSMgR2lhbnQgbXV0ZXggaXMg YWRhcHRpdmUuDQpvcHRpb25zICAgICAgICAgUFJFRU1QVElPTiAgICAgICAg ICAgICAgIyBFbmFibGUga2VybmVsIHRocmVhZCBwcmVlbXB0aW9uDQoNCg0K b3B0aW9ucyAJU01QDQoNCiMgV29ya2Fyb3VuZHMgZm9yIHNvbWUga25vd24t dG8tYmUtYnJva2VuIGNoaXBzZXRzIChuVmlkaWEgbkZvcmNlMy1Qcm8xNTAp DQpkZXZpY2UJCWF0cGljCQkjIDgyNTlBIGNvbXBhdGFiaWxpdHkNCg0KIyBF bmFibGluZyBOT19NSVhFRF9NT0RFIGdpdmVzIGEgcGVyZm9ybWFuY2UgaW1w cm92ZW1lbnQgb24gc29tZSBtb3RoZXJib2FyZHMNCiMgYnV0IGRvZXMgbm90 IHdvcmsgd2l0aCBzb21lIGJvYXJkcyAobW9zdGx5IG5WaWRpYSBjaGlwc2V0 IGJhc2VkKS4NCiNvcHRpb25zIAlOT19NSVhFRF9NT0RFCSMgRG9uJ3QgcGVu YWxpemUgd29ya2luZyBjaGlwc2V0cw0KDQojIExpbnV4IDMyLWJpdCBBQkkg c3VwcG9ydA0Kb3B0aW9ucyAJTElOUFJPQ0ZTCQkjIENhbm5vdCBiZSBhIG1v ZHVsZSB5ZXQuDQoNCiMgQnVzIHN1cHBvcnQuICBEbyBub3QgcmVtb3ZlIGlz YSwgZXZlbiBpZiB5b3UgaGF2ZSBubyBpc2Egc2xvdHMNCmRldmljZQkJYWNw aQ0KZGV2aWNlCQlpc2ENCmRldmljZQkJcGNpDQoNCiMgRmxvcHB5IGRyaXZl cw0KZGV2aWNlCQlmZGMNCg0KIyBBVEEgYW5kIEFUQVBJIGRldmljZXMNCmRl dmljZQkJYXRhDQpkZXZpY2UJCWF0YWRpc2sJCSMgQVRBIGRpc2sgZHJpdmVz DQpkZXZpY2UJCWF0YXJhaWQJCSMgQVRBIFJBSUQgZHJpdmVzDQpkZXZpY2UJ CWF0YXBpY2QJCSMgQVRBUEkgQ0RST00gZHJpdmVzDQpkZXZpY2UJCWF0YXBp ZmQJCSMgQVRBUEkgZmxvcHB5IGRyaXZlcw0KZGV2aWNlCQlhdGFwaXN0CQkj IEFUQVBJIHRhcGUgZHJpdmVzDQpvcHRpb25zIAlBVEFfU1RBVElDX0lECSMg U3RhdGljIGRldmljZSBudW1iZXJpbmcNCg0KIyBTQ1NJIENvbnRyb2xsZXJz DQpkZXZpY2UJCWFoYwkJIyBBSEEyOTQwIGFuZCBvbmJvYXJkIEFJQzd4eHgg ZGV2aWNlcw0KZGV2aWNlCQlhaGQJCSMgQUhBMzkzMjAvMjkzMjAgYW5kIG9u Ym9hcmQgQUlDNzl4eCBkZXZpY2VzDQojZGV2aWNlCQlhbWQJCSMgQU1EIDUz Qzk3NCAoVGVrcmFtIERDLTM5MChUKSkNCiNkZXZpY2UJCWlzcAkJIyBRbG9n aWMgZmFtaWx5DQojZGV2aWNlIAlpc3BmdwkJIyBGaXJtd2FyZSBmb3IgUUxv Z2ljIEhCQXMtIG5vcm1hbGx5IGEgbW9kdWxlDQojZGV2aWNlCQltcHQJCSMg TFNJLUxvZ2ljIE1QVC1GdXNpb24NCiNkZXZpY2UJCW5jcgkJIyBOQ1IvU3lt YmlvcyBMb2dpYw0KI2RldmljZQkJc3ltCQkjIE5DUi9TeW1iaW9zIExvZ2lj IChuZXdlciBjaGlwc2V0cyArIHRob3NlIG9mIGBuY3InKQ0KI2RldmljZQkJ dHJtCQkjIFRla3JhbSBEQzM5NVUvVVcvRiBEQzMxNVUgYWRhcHRlcnMNCg0K I2RldmljZQkJYWR2CQkjIEFkdmFuc3lzIFNDU0kgYWRhcHRlcnMNCiNkZXZp Y2UJCWFkdwkJIyBBZHZhbnN5cyB3aWRlIFNDU0kgYWRhcHRlcnMNCmRldmlj ZQkJYWljCQkjIEFkYXB0ZWMgMTVbMDEyXXggU0NTSSBhZGFwdGVycywgQUlD LTZbMjNdNjAuDQojZGV2aWNlCQlidAkJIyBCdXNsb2dpYy9NeWxleCBNdWx0 aU1hc3RlciBTQ1NJIGFkYXB0ZXJzDQoNCg0KIyBTQ1NJIHBlcmlwaGVyYWxz DQpkZXZpY2UJCXNjYnVzCQkjIFNDU0kgYnVzIChyZXF1aXJlZCBmb3IgU0NT SSkNCmRldmljZQkJY2gJCSMgU0NTSSBtZWRpYSBjaGFuZ2Vycw0KZGV2aWNl CQlkYQkJIyBEaXJlY3QgQWNjZXNzIChkaXNrcykNCmRldmljZQkJc2EJCSMg U2VxdWVudGlhbCBBY2Nlc3MgKHRhcGUgZXRjKQ0KZGV2aWNlCQljZAkJIyBD RA0KZGV2aWNlCQlwYXNzCQkjIFBhc3N0aHJvdWdoIGRldmljZSAoZGlyZWN0 IFNDU0kgYWNjZXNzKQ0KZGV2aWNlCQlzZXMJCSMgU0NTSSBFbnZpcm9ubWVu dGFsIFNlcnZpY2VzIChhbmQgU0FGLVRFKQ0KDQojIFJBSUQgY29udHJvbGxl cnMgaW50ZXJmYWNlZCB0byB0aGUgU0NTSSBzdWJzeXN0ZW0NCiNkZXZpY2UJ CWFtcgkJIyBBTUkgTWVnYVJBSUQNCiNkZXZpY2UJCWFyY21zcgkJIyBBcmVj YSBTQVRBIElJIFJBSUQNCiNkZXZpY2UJCWNpc3MJCSMgQ29tcGFxIFNtYXJ0 IFJBSUQgNSoNCiNkZXZpY2UJCWRwdAkJIyBEUFQgU21hcnRjYWNoZSBJSUks IElWIC0gU2VlIE5PVEVTIGZvciBvcHRpb25zDQojZGV2aWNlCQlpaXIJCSMg SW50ZWwgSW50ZWdyYXRlZCBSQUlEDQojZGV2aWNlCQlpcHMJCSMgSUJNIChB ZGFwdGVjKSBTZXJ2ZVJBSUQNCiNkZXZpY2UJCW1seQkJIyBNeWxleCBBY2Nl bGVSQUlEL2VYdHJlbWVSQUlEDQojZGV2aWNlCQl0d2EJCSMgM3dhcmUgOTAw MCBzZXJpZXMgUEFUQS9TQVRBIFJBSUQNCg0KIyBSQUlEIGNvbnRyb2xsZXJz DQpkZXZpY2UJCWFhYwkJIyBBZGFwdGVjIEZTQSBSQUlEDQpkZXZpY2UJCWFh Y3AJCSMgU0NTSSBwYXNzdGhyb3VnaCBmb3IgYWFjIChyZXF1aXJlcyBDQU0p DQojZGV2aWNlCQlpZGEJCSMgQ29tcGFxIFNtYXJ0IFJBSUQNCiNkZXZpY2UJ CW1seAkJIyBNeWxleCBEQUM5NjAgZmFtaWx5DQojWFhYIHBvaW50ZXIvaW50 IHdhcm5pbmdzDQojZGV2aWNlCQlwc3QJCSMgUHJvbWlzZSBTdXBlcnRyYWsg U1g2MDAwDQojZGV2aWNlCQl0d2UJCSMgM3dhcmUgQVRBIFJBSUQNCg0KIyBh dGtiZGMwIGNvbnRyb2xzIGJvdGggdGhlIGtleWJvYXJkIGFuZCB0aGUgUFMv MiBtb3VzZQ0KZGV2aWNlCQlhdGtiZGMJCSMgQVQga2V5Ym9hcmQgY29udHJv bGxlcg0KZGV2aWNlCQlhdGtiZAkJIyBBVCBrZXlib2FyZA0KZGV2aWNlCQlw c20JCSMgUFMvMiBtb3VzZQ0KDQpkZXZpY2UJCXZnYQkJIyBWR0EgdmlkZW8g Y2FyZCBkcml2ZXINCg0KZGV2aWNlCQlzcGxhc2gJCSMgU3BsYXNoIHNjcmVl biBhbmQgc2NyZWVuIHNhdmVyIHN1cHBvcnQNCg0KIyBzeXNjb25zIGlzIHRo ZSBkZWZhdWx0IGNvbnNvbGUgZHJpdmVyLCByZXNlbWJsaW5nIGFuIFNDTyBj b25zb2xlDQpkZXZpY2UJCXNjDQoNCiMgUENDQVJEIChQQ01DSUEpIHN1cHBv cnQNCiMgUENNQ0lBIGFuZCBjYXJkYnVzIGJyaWRnZSBzdXBwb3J0DQojZGV2 aWNlCQljYmIJCSMgY2FyZGJ1cyAoeWVudGEpIGJyaWRnZQ0KI2RldmljZQkJ cGNjYXJkCQkjIFBDIENhcmQgKDE2LWJpdCkgYnVzDQojZGV2aWNlCQljYXJk YnVzCQkjIENhcmRCdXMgKDMyLWJpdCkgYnVzDQoNCiMgU2VyaWFsIChDT00p IHBvcnRzDQpkZXZpY2UJCXNpbwkJIyA4MjUwLCAxNls0NV01MCBiYXNlZCBz ZXJpYWwgcG9ydHMNCg0KIyBQYXJhbGxlbCBwb3J0DQpkZXZpY2UJCXBwYw0K ZGV2aWNlCQlwcGJ1cwkJIyBQYXJhbGxlbCBwb3J0IGJ1cyAocmVxdWlyZWQp DQpkZXZpY2UJCWxwdAkJIyBQcmludGVyDQojZGV2aWNlCQlwbGlwCQkjIFRD UC9JUCBvdmVyIHBhcmFsbGVsDQpkZXZpY2UJCXBwaQkJIyBQYXJhbGxlbCBw b3J0IGludGVyZmFjZSBkZXZpY2UNCiNkZXZpY2UJCXZwbwkJIyBSZXF1aXJl cyBzY2J1cyBhbmQgZGENCg0KIyBJZiB5b3UndmUgZ290IGEgImR1bWIiIHNl cmlhbCBvciBwYXJhbGxlbCBQQ0kgY2FyZCB0aGF0IGlzDQojIHN1cHBvcnRl ZCBieSB0aGUgcHVjKDQpIGdsdWUgZHJpdmVyLCB1bmNvbW1lbnQgdGhlIGZv bGxvd2luZw0KIyBsaW5lIHRvIGVuYWJsZSBpdCAoY29ubmVjdHMgdG8gdGhl IHNpbyBhbmQvb3IgcHBjIGRyaXZlcnMpOg0KI2RldmljZQkJcHVjDQoNCiMg UENJIEV0aGVybmV0IE5JQ3MuDQojZGV2aWNlCQlkZQkJIyBERUMvSW50ZWwg REMyMXg0eCAoYGBUdWxpcCcnKQ0KZGV2aWNlCQllbQkJIyBJbnRlbCBQUk8v MTAwMCBhZGFwdGVyIEdpZ2FiaXQgRXRoZXJuZXQgQ2FyZA0KI2RldmljZQkJ aXhnYgkJIyBJbnRlbCBQUk8vMTBHYkUgRXRoZXJuZXQgQ2FyZA0KI2Rldmlj ZQkJdHhwCQkjIDNDb20gM2NSOTkwIChgYFR5cGhvb24nJykNCiNkZXZpY2UJ CXZ4CQkjIDNDb20gM2M1OTAsIDNjNTk1IChgYFZvcnRleCcnKQ0KDQojIFBD SSBFdGhlcm5ldCBOSUNzIHRoYXQgdXNlIHRoZSBjb21tb24gTUlJIGJ1cyBj b250cm9sbGVyIGNvZGUuDQojIE5PVEU6IEJlIHN1cmUgdG8ga2VlcCB0aGUg J2RldmljZSBtaWlidXMnIGxpbmUgaW4gb3JkZXIgdG8gdXNlIHRoZXNlIE5J Q3MhDQpkZXZpY2UJCW1paWJ1cwkJIyBNSUkgYnVzIHN1cHBvcnQNCiNkZXZp Y2UJCWJmZQkJIyBCcm9hZGNvbSBCQ000NDB4IDEwLzEwMCBFdGhlcm5ldA0K ZGV2aWNlCQliZ2UJCSMgQnJvYWRjb20gQkNNNTcweHggR2lnYWJpdCBFdGhl cm5ldA0KI2RldmljZQkJZGMJCSMgREVDL0ludGVsIDIxMTQzIGFuZCB2YXJp b3VzIHdvcmthbGlrZXMNCmRldmljZQkJZnhwCQkjIEludGVsIEV0aGVyRXhw cmVzcyBQUk8vMTAwQiAoODI1NTcsIDgyNTU4KQ0KI2RldmljZQkJbGdlCQkj IExldmVsIDEgTFhUMTAwMSBnaWdhYml0IEV0aGVybmV0DQojZGV2aWNlCQlu Z2UJCSMgTmF0U2VtaSBEUDgzODIwIGdpZ2FiaXQgRXRoZXJuZXQNCiNkZXZp Y2UJCXBjbgkJIyBBTUQgQW03OUM5N3ggUENJIDEwLzEwMCAocHJlY2VkZW5j ZSBvdmVyICdsbmMnKQ0KI2RldmljZQkJcmUJCSMgUmVhbFRlayA4MTM5Qysv ODE2OS84MTY5Uy84MTEwUw0KI2RldmljZQkJcmwJCSMgUmVhbFRlayA4MTI5 LzgxMzkNCiNkZXZpY2UJCXNmCQkjIEFkYXB0ZWMgQUlDLTY5MTUgKGBgU3Rh cmZpcmUnJykNCiNkZXZpY2UJCXNpcwkJIyBTaWxpY29uIEludGVncmF0ZWQg U3lzdGVtcyBTaVMgOTAwL1NpUyA3MDE2DQojZGV2aWNlCQlzawkJIyBTeXNL b25uZWN0IFNLLTk4NHggJiBTSy05ODJ4IGdpZ2FiaXQgRXRoZXJuZXQNCiNk ZXZpY2UJCXN0ZQkJIyBTdW5kYW5jZSBTVDIwMSAoRC1MaW5rIERGRS01NTBU WCkNCiNkZXZpY2UJCXRpCQkjIEFsdGVvbiBOZXR3b3JrcyBUaWdvbiBJL0lJ IGdpZ2FiaXQgRXRoZXJuZXQNCiNkZXZpY2UJCXRsCQkjIFRleGFzIEluc3Ry dW1lbnRzIFRodW5kZXJMQU4NCiNkZXZpY2UJCXR4CQkjIFNNQyBFdGhlclBv d2VyIElJICg4M2MxNzAgYGBFUElDJycpDQojZGV2aWNlCQl2Z2UJCSMgVklB IFZUNjEyeCBnaWdhYml0IEV0aGVybmV0DQojZGV2aWNlCQl2cgkJIyBWSUEg UmhpbmUsIFJoaW5lIElJDQojZGV2aWNlCQl3YgkJIyBXaW5ib25kIFc4OUM4 NDBGDQojZGV2aWNlCQl4bAkJIyAzQ29tIDNjOTB4IChgYEJvb21lcmFuZycn LCBgYEN5Y2xvbmUnJykNCg0KIyBJU0EgRXRoZXJuZXQgTklDcy4gIHBjY2Fy ZCBOSUNzIGluY2x1ZGVkLg0KI2RldmljZQkJY3MJCSMgQ3J5c3RhbCBTZW1p Y29uZHVjdG9yIENTODl4MCBOSUMNCiMgJ2RldmljZSBlZCcgcmVxdWlyZXMg J2RldmljZSBtaWlidXMnDQojIFhYWCBrdnRvcCBicm9rZW5uZXNzLCBwb2lu dGVyL2ludCB3YXJuaW5ncw0KI2RldmljZQkJZWQJCSMgTkVbMTJdMDAwLCBT TUMgVWx0cmEsIDNjNTAzLCBEUzgzOTAgY2FyZHMNCiNkZXZpY2UJCWV4CQkj IEludGVsIEV0aGVyRXhwcmVzcyBQcm8vMTAgYW5kIFByby8xMCsNCiNkZXZp Y2UJCWVwCQkjIEV0aGVybGluayBJSUkgYmFzZWQgY2FyZHMNCiNkZXZpY2UJ CWZlCQkjIEZ1aml0c3UgTUI4Njk2eCBiYXNlZCBjYXJkcw0KIyBYWFgga3Z0 b3AgYnJva2VubmVzcywgcG9pbnRlci9pbnQgd2FybmluZ3MNCiNkZXZpY2UJ CWxuYwkJIyBORTIxMDAsIE5FMzItVkwgTGFuY2UgRXRoZXJuZXQgY2FyZHMN CiNkZXZpY2UJCXNuCQkjIFNNQydzIDkwMDAgc2VyaWVzIG9mIEV0aGVybmV0 IGNoaXBzDQojZGV2aWNlCQl4ZQkJIyBYaXJjb20gcGNjYXJkIEV0aGVybmV0 DQoNCiMgV2lyZWxlc3MgTklDIGNhcmRzDQojZGV2aWNlCQl3bGFuCQkjIDgw Mi4xMSBzdXBwb3J0DQojZGV2aWNlCQlhbgkJIyBBaXJvbmV0IDQ1MDAvNDgw MCA4MDIuMTEgd2lyZWxlc3MgTklDcy4NCiNkZXZpY2UJCWF3aQkJIyBCYXlT dGFjayA2NjAgYW5kIG90aGVycw0KI2RldmljZQkJd2kJCSMgV2F2ZUxBTi9J bnRlcnNpbC9TeW1ib2wgODAyLjExIHdpcmVsZXNzIE5JQ3MuDQoNCiMgUHNl dWRvIGRldmljZXMuDQpkZXZpY2UJCWxvb3AJCSMgTmV0d29yayBsb29wYmFj aw0KZGV2aWNlCQltZW0JCSMgTWVtb3J5IGFuZCBrZXJuZWwgbWVtb3J5IGRl dmljZXMNCmRldmljZQkJaW8JCSMgSS9PIGRldmljZQ0KZGV2aWNlCQlyYW5k b20JCSMgRW50cm9weSBkZXZpY2UNCmRldmljZQkJZXRoZXIJCSMgRXRoZXJu ZXQgc3VwcG9ydA0KZGV2aWNlCQlzbAkJIyBLZXJuZWwgU0xJUA0KZGV2aWNl CQlwcHAJCSMgS2VybmVsIFBQUA0KZGV2aWNlCQl0dW4JCSMgUGFja2V0IHR1 bm5lbC4NCmRldmljZQkJcHR5CQkjIFBzZXVkby10dHlzICh0ZWxuZXQgZXRj KQ0KZGV2aWNlCQltZAkJIyBNZW1vcnkgImRpc2tzIg0KZGV2aWNlCQlnaWYJ CSMgSVB2NiBhbmQgSVB2NCB0dW5uZWxpbmcNCmRldmljZQkJZmFpdGgJCSMg SVB2Ni10by1JUHY0IHJlbGF5aW5nICh0cmFuc2xhdGlvbikNCg0KIyBUaGUg YGJwZicgZGV2aWNlIGVuYWJsZXMgdGhlIEJlcmtlbGV5IFBhY2tldCBGaWx0 ZXIuDQojIEJlIGF3YXJlIG9mIHRoZSBhZG1pbmlzdHJhdGl2ZSBjb25zZXF1 ZW5jZXMgb2YgZW5hYmxpbmcgdGhpcyENCiMgTm90ZSB0aGF0ICdicGYnIGlz IHJlcXVpcmVkIGZvciBESENQLg0KZGV2aWNlCQlicGYJCSMgQmVya2VsZXkg cGFja2V0IGZpbHRlcg0KDQojIFVTQiBzdXBwb3J0DQpkZXZpY2UJCXVoY2kJ CSMgVUhDSSBQQ0ktPlVTQiBpbnRlcmZhY2UNCmRldmljZQkJb2hjaQkJIyBP SENJIFBDSS0+VVNCIGludGVyZmFjZQ0KI2RldmljZQkJZWhjaQkJIyBFSENJ IFBDSS0+VVNCIGludGVyZmFjZSAoVVNCIDIuMCkNCmRldmljZQkJdXNiCQkj IFVTQiBCdXMgKHJlcXVpcmVkKQ0KI2RldmljZQkJdWRicAkJIyBVU0IgRG91 YmxlIEJ1bGsgUGlwZSBkZXZpY2VzDQpkZXZpY2UJCXVnZW4JCSMgR2VuZXJp Yw0KZGV2aWNlCQl1aGlkCQkjICJIdW1hbiBJbnRlcmZhY2UgRGV2aWNlcyIN CmRldmljZQkJdWtiZAkJIyBLZXlib2FyZA0KZGV2aWNlCQl1bHB0CQkjIFBy aW50ZXINCmRldmljZQkJdW1hc3MJCSMgRGlza3MvTWFzcyBzdG9yYWdlIC0g UmVxdWlyZXMgc2NidXMgYW5kIGRhDQpkZXZpY2UJCXVtcwkJIyBNb3VzZQ0K I2RldmljZQkJdXJpbwkJIyBEaWFtb25kIFJpbyA1MDAgTVAzIHBsYXllcg0K I2RldmljZQkJdXNjYW5uZXIJIyBTY2FubmVycw0KIyBVU0IgRXRoZXJuZXQs IHJlcXVpcmVzIG1paQ0KI2RldmljZQkJYXVlCQkjIEFETXRlayBVU0IgRXRo ZXJuZXQNCiNkZXZpY2UJCWF4ZQkJIyBBU0lYIEVsZWN0cm9uaWNzIFVTQiBF dGhlcm5ldA0KI2RldmljZQkJY2RjZQkJIyBHZW5lcmljIFVTQiBvdmVyIEV0 aGVybmV0DQojZGV2aWNlCQljdWUJCSMgQ0FUQyBVU0IgRXRoZXJuZXQNCiNk ZXZpY2UJCWt1ZQkJIyBLYXdhc2FraSBMU0kgVVNCIEV0aGVybmV0DQojZGV2 aWNlCQlydWUJCSMgUmVhbFRlayBSVEw4MTUwIFVTQiBFdGhlcm5ldA0KDQoj IEZpcmVXaXJlIHN1cHBvcnQNCiNkZXZpY2UJCWZpcmV3aXJlCSMgRmlyZVdp cmUgYnVzIGNvZGUNCiNkZXZpY2UJCXNicAkJIyBTQ1NJIG92ZXIgRmlyZVdp cmUgKFJlcXVpcmVzIHNjYnVzIGFuZCBkYSkNCiNkZXZpY2UJCWZ3ZQkJIyBF dGhlcm5ldCBvdmVyIEZpcmVXaXJlIChub24tc3RhbmRhcmQhKQ0KDQpvcHRp b25zICAgICAgICAgSVBGSVJFV0FMTA0Kb3B0aW9ucyAgICAgICAgIElQRklS RVdBTExfVkVSQk9TRQ0KDQo= --0-779721010-1127845675=:50334 Content-Type: TEXT/PLAIN; charset=US-ASCII; name="type3-crash.txt" Content-Transfer-Encoding: BASE64 Content-Description: Content-Disposition: attachment; filename="type3-crash.txt" a2VybmVsIHRyYXAgMTIgd2l0aCBpbnRlcnJ1cHRzIGRpc2FibGVkDQoNCmZh dGFsIHRyYXAgMTI6IHBhZ2UgZmF1bHQgd2hpbGUgaW4ga2VybmVsIG1vZGUN CmNwdWlkPTM7IGFwaWNpZD0wMw0KZmF1bHQgdmlydHVhbCBhZGRyZXNzICAg PSAwMw0KZmF1bHQgY29kZSAgICAgICAgICAgICAgPSBzdXBlcnZpc29yIHJl YWQsIHBhZ2Ugbm90IHByZXNlbnQNCmluc3RydWN0aW9uIHBvaW50ZXIgICAg ID0gMHg4OmZmZmZmZmZmODAzYjg4Y2ENCnN0YWNrIHBvaW50ZXIgICAgICAg ICAgID0gMHgxMDpmZmZmZmZmZmI2NjM5NDkwDQpmcmFtZSBwb2ludGVyICAg ICAgICAgICA9IDB4MTA6ZmZmZmZmZmZiNjYzOTRmMA0KY29kZSBzZWdtZW50 ICAgICAgICAgICAgPSBiYXNlIDB4MDsgbGltaXQgMHhmZmZmZiwgdHlwZT0w eDFiDQogICAgICAgICAgICAgICAgICAgICAgICA9IERQTD0wLCBwcmVzIDEs IGxvbmcgMSwgZGVmMzI9MCwgZ3JhbiAxDQpwcm9jZXNzb3IgZWZsYWdzICAg ICAgICA9IHJlc3VtZSwgSU9QTD0wDQpjdXJyZW50IHByb2Nlc3MgICAgICAg ICA9IDQ4NjI4ICh0b3ApDQoNCmRpZCBub3QgZW50ZXIgRERCIG9yIGdlbmVy YXRlIGNvcmUgZmlsZQ0K --0-779721010-1127845675=:50334--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20050927140535.G50334>