Date: Thu, 14 Feb 2008 15:54:59 -0500 From: Jim Pingle <lists@pingle.org> To: stable@freebsd.org Subject: 7.0-PRERELEASE Fatal Trap 12 with sysctl and acpi Message-ID: <47B4AAA3.6060501@pingle.org>
next in thread | raw e-mail | index | archive | help
This is a multi-part message in MIME format.
--------------060204090303000808020609
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
I'm having some trouble with a SuperMicro SuperServer 6022L-6 that
previously ran 7.0-BETA4 without problems. Today, I updated this machine to
7.0-PRERELEASE and now it will not fully boot unless I disable ACPI. A quick
search of the PR database didn't turn up anything similar with sysctl and ACPI.
I can boot to single user mode, but if I issue sysctl -a while there, it
also crashes.
I have two vmcore files, one where I booted to single user mode and issued
sysctl -a, the other when it was attempting to boot normally.
When running sysctl -a in single user mode, the last three lines before the
crash are (transcribed by hand, no serial console available):
dev.pcib.3.%location: handle=\_SB_.PCI3
dev.pcib.3.%pnpinfo: _HID=PNP0A03 UID=3
dev.pcib.3.%parent: acpi0
=========================================================================
Kernel config is GENERIC, with ULE scheduler and "options ASR_COMPAT"
=========================================================================
[root@test1 /usr/obj/usr/src/sys/TEST]# uname -a
FreeBSD test1.hpcisp.com 7.0-PRERELEASE FreeBSD 7.0-PRERELEASE #1: Thu Feb
14 14:08:02 EST 2008 root@test1.hpcisp.com:/usr/obj/usr/src/sys/TEST i386
=========================================================================
[root@test1 /usr/obj/usr/src/sys/TEST]# kgdb kernel.debug
/var/crash/vmcore.1
Fatal trap 12: page fault while in kernel mode
cpuid = 3; apic id = 03
fault virtual address = 0x2043455c
fault code = supervisor read, page not present
instruction pointer = 0x20:0xc0743036
stack pointer = 0x28:0xe8cb3a0c
frame pointer = 0x28:0xe8cb3a38
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 67 (sysctl)
trap number = 12
panic: page fault
cpuid = 3
Uptime: 6s
Physical memory: 2035 MB
Dumping 65 MB: 50 34 18 2
#0 doadump () at pcpu.h:195
195 __asm __volatile("movl %%fs:0,%0" : "=r" (td));
(kgdb) bt
#0 doadump () at pcpu.h:195
#1 0xc073aa38 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409
#2 0xc073acf1 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:563
#3 0xc0a1fd00 in trap_fatal (frame=0xe8cb39cc, eva=541279580) at
/usr/src/sys/i386/i386/trap.c:899
#4 0xc0a1ff70 in trap_pfault (frame=0xe8cb39cc, usermode=0,
eva=541279580) at /usr/src/sys/i386/i386/trap.c:812
#5 0xc0a208ed in trap (frame=0xe8cb39cc) at
/usr/src/sys/i386/i386/trap.c:490
#6 0xc0a07bdb in calltrap () at /usr/src/sys/i386/i386/exception.s:139
#7 0xc0743036 in sysctl_sysctl_next_ls (lsp=Variable "lsp" is not
available.
) at /usr/src/sys/kern/kern_sysctl.c:630
#8 0xc07430f6 in sysctl_sysctl_next_ls (lsp=Variable "lsp" is not
available.
) at /usr/src/sys/kern/kern_sysctl.c:618
#9 0xc0743133 in sysctl_sysctl_next_ls (lsp=Variable "lsp" is not
available.
) at /usr/src/sys/kern/kern_sysctl.c:630
#10 0xc0743133 in sysctl_sysctl_next_ls (lsp=Variable "lsp" is not
available.
) at /usr/src/sys/kern/kern_sysctl.c:630
#11 0xc0743196 in sysctl_sysctl_next (oidp=0xc0b53280, arg1=0xe8cb3c1c,
arg2=4, req=0xe8cb3ba4)
at /usr/src/sys/kern/kern_sysctl.c:651
#12 0xc0743aa2 in sysctl_root (oidp=Variable "oidp" is not available.
) at /usr/src/sys/kern/kern_sysctl.c:1306
#13 0xc0743bde in userland_sysctl (td=0xc5479660, name=0xe8cb3c14,
namelen=6, old=0xbfbfe4e8, oldlenp=0xbfbfe598, inkernel=0,
new=0x0, newlen=0, retval=0xe8cb3c10, flags=0) at
/usr/src/sys/kern/kern_sysctl.c:1401
#14 0xc0744812 in __sysctl (td=0xc5479660, uap=0xe8cb3cfc) at
/usr/src/sys/kern/kern_sysctl.c:1336
#15 0xc0a202b8 in syscall (frame=0xe8cb3d38) at
/usr/src/sys/i386/i386/trap.c:1035
#16 0xc0a07c40 in Xint0x80_syscall () at
/usr/src/sys/i386/i386/exception.s:196
#17 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
=========================================================================
dmesg is attached, but it is from a non-acpi boot.
=========================================================================
Anyone have any ideas on what might be the cause or a possible fix?
I'll keep the crash dumps around. This is a test box that I'm researching
7.0 on for possible production use on similar hardware. There is no planned
usage yet, and no other plans for this box, so anything goes in terms of
possible debugging.
If I get some time next week I might try a binary search of commits between
BETA4 and now, to pinpoint where it stopped working.
Thanks,
Jim
--------------060204090303000808020609
Content-Type: text/plain;
name="dmesg-testbox-noacpi.txt"
Content-Transfer-Encoding: base64
Content-Disposition: inline;
filename="dmesg-testbox-noacpi.txt"
Q29weXJpZ2h0IChjKSAxOTkyLTIwMDggVGhlIEZyZWVCU0QgUHJvamVjdC4NCkNvcHlyaWdo
dCAoYykgMTk3OSwgMTk4MCwgMTk4MywgMTk4NiwgMTk4OCwgMTk4OSwgMTk5MSwgMTk5Miwg
MTk5MywgMTk5NA0KICAgICAgICBUaGUgUmVnZW50cyBvZiB0aGUgVW5pdmVyc2l0eSBvZiBD
YWxpZm9ybmlhLiBBbGwgcmlnaHRzIHJlc2VydmVkLg0KRnJlZUJTRCBpcyBhIHJlZ2lzdGVy
ZWQgdHJhZGVtYXJrIG9mIFRoZSBGcmVlQlNEIEZvdW5kYXRpb24uDQpGcmVlQlNEIDcuMC1Q
UkVSRUxFQVNFICMxOiBUaHUgRmViIDE0IDE0OjA4OjAyIEVTVCAyMDA4DQogICAgcm9vdEB0
ZXN0MS5ocGNpc3AuY29tOi91c3Ivb2JqL3Vzci9zcmMvc3lzL1RFU1QNClRpbWVjb3VudGVy
ICJpODI1NCIgZnJlcXVlbmN5IDExOTMxODIgSHogcXVhbGl0eSAwDQpDUFU6IEludGVsKFIp
IFhFT04oVE0pIENQVSAyLjAwR0h6ICgxOTk5Ljk1LU1IeiA2ODYtY2xhc3MgQ1BVKQ0KICBP
cmlnaW4gPSAiR2VudWluZUludGVsIiAgSWQgPSAweGYyNCAgU3RlcHBpbmcgPSA0DQogIEZl
YXR1cmVzPTB4M2ZlYmZiZmY8RlBVLFZNRSxERSxQU0UsVFNDLE1TUixQQUUsTUNFLENYOCxB
UElDLFNFUCxNVFJSLFBHRSxNQ0EsQ01PVixQQVQsUFNFMzYsQ0xGTFVTSCxEVFMsQUNQSSxN
TVgsRlhTUixTU0UsU1NFMixTUyxIVFQsVE0+DQogIExvZ2ljYWwgQ1BVcyBwZXIgY29yZTog
Mg0KcmVhbCBtZW1vcnkgID0gMjE0NzQxODExMiAoMjA0NyBNQikNCmF2YWlsIG1lbW9yeSA9
IDIwOTE4OTI3MzYgKDE5OTQgTUIpDQpNUFRhYmxlOiA8QU1JICAgICAgR0NIRSAgICAgICAg
Pg0KRnJlZUJTRC9TTVA6IE11bHRpcHJvY2Vzc29yIFN5c3RlbSBEZXRlY3RlZDogNCBDUFVz
DQogY3B1MCAoQlNQKTogQVBJQyBJRDogIDANCiBjcHUxIChBUCk6IEFQSUMgSUQ6ICAxDQog
Y3B1MiAoQVApOiBBUElDIElEOiAgMg0KIGNwdTMgKEFQKTogQVBJQyBJRDogIDMNCmlvYXBp
YzA6IEFzc3VtaW5nIGludGJhc2Ugb2YgMA0KaW9hcGljMTogQXNzdW1pbmcgaW50YmFzZSBv
ZiAxNg0KaW9hcGljMjogQXNzdW1pbmcgaW50YmFzZSBvZiAzMg0KaW9hcGljMzogQXNzdW1p
bmcgaW50YmFzZSBvZiA0OA0KaW9hcGljMCA8VmVyc2lvbiAxLjE+IGlycXMgMC0xNSBvbiBt
b3RoZXJib2FyZA0KaW9hcGljMSA8VmVyc2lvbiAxLjE+IGlycXMgMTYtMzEgb24gbW90aGVy
Ym9hcmQNCmlvYXBpYzIgPFZlcnNpb24gMS4xPiBpcnFzIDMyLTQ3IG9uIG1vdGhlcmJvYXJk
DQppb2FwaWMzIDxWZXJzaW9uIDEuMT4gaXJxcyA0OC02MyBvbiBtb3RoZXJib2FyZA0Ka2Jk
MSBhdCBrYmRtdXgwDQphdGhfaGFsOiAwLjkuMjAuMyAoQVI1MjEwLCBBUjUyMTEsIEFSNTIx
MiwgUkY1MTExLCBSRjUxMTIsIFJGMjQxMywgUkY1NDEzKQ0KY3B1MCBvbiBtb3RoZXJib2Fy
ZA0KcDR0Y2MwOiA8Q1BVIEZyZXF1ZW5jeSBUaGVybWFsIENvbnRyb2w+IG9uIGNwdTANCmNw
dTEgb24gbW90aGVyYm9hcmQNCnA0dGNjMTogPENQVSBGcmVxdWVuY3kgVGhlcm1hbCBDb250
cm9sPiBvbiBjcHUxDQpjcHUyIG9uIG1vdGhlcmJvYXJkDQpwNHRjYzI6IDxDUFUgRnJlcXVl
bmN5IFRoZXJtYWwgQ29udHJvbD4gb24gY3B1Mg0KY3B1MyBvbiBtb3RoZXJib2FyZA0KcDR0
Y2MzOiA8Q1BVIEZyZXF1ZW5jeSBUaGVybWFsIENvbnRyb2w+IG9uIGNwdTMNCnBjaWIwOiA8
TVBUYWJsZSBIb3N0LVBDSSBicmlkZ2U+IHBjaWJ1cyAwIG9uIG1vdGhlcmJvYXJkDQpwY2kw
OiA8UENJIGJ1cz4gb24gcGNpYjANCnZnYXBjaTA6IDxWR0EtY29tcGF0aWJsZSBkaXNwbGF5
PiBwb3J0IDB4YTgwMC0weGE4ZmYgbWVtIDB4ZmQwMDAwMDAtMHhmZGZmZmZmZiwweGZlNWZm
MDAwLTB4ZmU1ZmZmZmYgaXJxIDE4IGF0IGRldmljZSAyLjAgb24gcGNpMA0KZnhwMDogPElu
dGVsIDgyNTUwIFByby8xMDAgRXRoZXJuZXQ+IHBvcnQgMHhhZTgwLTB4YWViZiBtZW0gMHhm
ZTVmYzAwMC0weGZlNWZjZmZmLDB4ZmU1ODAwMDAtMHhmZTU5ZmZmZiBpcnEgMTcgYXQgZGV2
aWNlIDQuMCBvbiBwY2kwDQptaWlidXMwOiA8TUlJIGJ1cz4gb24gZnhwMA0KaW5waHkwOiA8
aTgyNTU1IDEwLzEwMCBtZWRpYSBpbnRlcmZhY2U+IFBIWSAxIG9uIG1paWJ1czANCmlucGh5
MDogIDEwYmFzZVQsIDEwYmFzZVQtRkRYLCAxMDBiYXNlVFgsIDEwMGJhc2VUWC1GRFgsIGF1
dG8NCmZ4cDA6IEV0aGVybmV0IGFkZHJlc3M6IDAwOjMwOjQ4OjIwOmEzOjllDQpmeHAwOiBb
SVRIUkVBRF0NCmZ4cDE6IDxJbnRlbCA4MjU1MCBQcm8vMTAwIEV0aGVybmV0PiBwb3J0IDB4
YWYwMC0weGFmM2YgbWVtIDB4ZmU1ZmQwMDAtMHhmZTVmZGZmZiwweGZlNWEwMDAwLTB4ZmU1
YmZmZmYgaXJxIDE5IGF0IGRldmljZSA1LjAgb24gcGNpMA0KbWlpYnVzMTogPE1JSSBidXM+
IG9uIGZ4cDENCmlucGh5MTogPGk4MjU1NSAxMC8xMDAgbWVkaWEgaW50ZXJmYWNlPiBQSFkg
MSBvbiBtaWlidXMxDQppbnBoeTE6ICAxMGJhc2VULCAxMGJhc2VULUZEWCwgMTAwYmFzZVRY
LCAxMDBiYXNlVFgtRkRYLCBhdXRvDQpmeHAxOiBFdGhlcm5ldCBhZGRyZXNzOiAwMDozMDo0
ODoyMDphMzo5Zg0KZnhwMTogW0lUSFJFQURdDQppc2FiMDogPFBDSS1JU0EgYnJpZGdlPiBh
dCBkZXZpY2UgMTUuMCBvbiBwY2kwDQppc2EwOiA8SVNBIGJ1cz4gb24gaXNhYjANCmF0YXBj
aTA6IDxTZXJ2ZXJXb3JrcyBDU0I1IFVETUExMDAgY29udHJvbGxlcj4gcG9ydCAweDFmMC0w
eDFmNywweDNmNiwweDE3MC0weDE3NywweDM3NiwweGZmYTAtMHhmZmFmIGF0IGRldmljZSAx
NS4xIG9uIHBjaTANCmF0YTA6IDxBVEEgY2hhbm5lbCAwPiBvbiBhdGFwY2kwDQphdGEwOiBb
SVRIUkVBRF0NCmF0YTE6IDxBVEEgY2hhbm5lbCAxPiBvbiBhdGFwY2kwDQphdGExOiBbSVRI
UkVBRF0NCm9oY2kwOiA8T0hDSSAoZ2VuZXJpYykgVVNCIGNvbnRyb2xsZXI+IG1lbSAweGZl
NWZlMDAwLTB4ZmU1ZmVmZmYgaXJxIDEwIGF0IGRldmljZSAxNS4yIG9uIHBjaTANCm9oY2kw
OiBbR0lBTlQtTE9DS0VEXQ0Kb2hjaTA6IFtJVEhSRUFEXQ0KdXNiMDogT0hDSSB2ZXJzaW9u
IDEuMCwgbGVnYWN5IHN1cHBvcnQNCnVzYjA6IFNNTSBkb2VzIG5vdCByZXNwb25kLCByZXNl
dHRpbmcNCnVzYjA6IDxPSENJIChnZW5lcmljKSBVU0IgY29udHJvbGxlcj4gb24gb2hjaTAN
CnVzYjA6IFVTQiByZXZpc2lvbiAxLjANCnVodWIwOiA8KDB4MTE2NikgT0hDSSByb290IGh1
YiwgY2xhc3MgOS8wLCByZXYgMS4wMC8xLjAwLCBhZGRyIDE+IG9uIHVzYjANCnVodWIwOiA0
IHBvcnRzIHdpdGggNCByZW1vdmFibGUsIHNlbGYgcG93ZXJlZA0KcGNpYjE6IDxTZXJ2ZXJX
b3JrcyBob3N0IHRvIFBDSSBicmlkZ2UodW5rbm93biBjaGlwc2V0KT4gcGNpYnVzIDEgb24g
bW90aGVyYm9hcmQNCnBpcjA6IDxQQ0kgSW50ZXJydXB0IFJvdXRpbmcgVGFibGU6IDkgRW50
cmllcz4gb24gbW90aGVyYm9hcmQNCiRQSVI6IElnbm9yaW5nIGludmFsaWQgQklPUyBJUlEg
MTggZnJvbSAwLjIuSU5UQSBmb3IgbGluayAweDEyDQokUElSOiBJZ25vcmluZyBpbnZhbGlk
IEJJT1MgSVJRIDE3IGZyb20gMC40LklOVEEgZm9yIGxpbmsgMHgxMQ0KJFBJUjogSWdub3Jp
bmcgaW52YWxpZCBCSU9TIElSUSAxOSBmcm9tIDAuNS5JTlRBIGZvciBsaW5rIDB4MTMNCnBj
aTE6IDxQQ0kgYnVzPiBvbiBwY2liMQ0KcGNpYjI6IDxTZXJ2ZXJXb3JrcyBob3N0IHRvIFBD
SSBicmlkZ2UodW5rbm93biBjaGlwc2V0KT4gcGNpYnVzIDIgb24gbW90aGVyYm9hcmQNCnBj
aTI6IDxQQ0kgYnVzPiBvbiBwY2liMg0KcGNpYjM6IDxTZXJ2ZXJXb3JrcyBob3N0IHRvIFBD
SSBicmlkZ2UodW5rbm93biBjaGlwc2V0KT4gcGNpYnVzIDMgb24gbW90aGVyYm9hcmQNCnBj
aTM6IDxQQ0kgYnVzPiBvbiBwY2liMw0KcGNpYjQ6IDxNUFRhYmxlIEhvc3QtUENJIGJyaWRn
ZT4gcGNpYnVzIDQgb24gbW90aGVyYm9hcmQNCnBjaTQ6IDxQQ0kgYnVzPiBvbiBwY2liNA0K
YXNyMDogPEFkYXB0ZWMgQ2FjaGluZyBTQ1NJIFJBSUQ+IG1lbSAweGZlYjAwMDAwLTB4ZmVi
ZmZmZmYsMHhmYjAwMDAwMC0weGZiZmZmZmZmLDB4ZjgwMDAwMDAtMHhmOWZmZmZmZiBpcnEg
MjkgYXQgZGV2aWNlIDMuMCBvbiBwY2k0DQphc3IwOiBbR0lBTlQtTE9DS0VEXQ0KYXNyMDog
W0lUSFJFQURdDQphc3IwOiAgIEFEQVBURUMgMjAwNVMgRlcgUmV2LiAzODBFLCAyIGNoYW5u
ZWwsIDIwMDAgQ0NCcywgUHJvdG9jb2wgSTJPDQpwbXRpbWVyMCBvbiBpc2EwDQpvcm0wOiA8
SVNBIE9wdGlvbiBST01zPiBhdCBpb21lbSAweGMwMDAwLTB4YzdmZmYsMHhjODAwMC0weGNk
ZmZmLDB4Y2UwMDAtMHhjZWZmZiwweGNmMDAwLTB4Y2ZmZmYgcG5waWQgT1JNMDAwMCBvbiBp
c2EwDQphdGtiZGMwOiA8S2V5Ym9hcmQgY29udHJvbGxlciAoaTgwNDIpPiBhdCBwb3J0IDB4
NjAsMHg2NCBvbiBpc2EwDQphdGtiZDA6IDxBVCBLZXlib2FyZD4gaXJxIDEgb24gYXRrYmRj
MA0Ka2JkMCBhdCBhdGtiZDANCmF0a2JkMDogW0dJQU5ULUxPQ0tFRF0NCmF0a2JkMDogW0lU
SFJFQURdDQpwc20wOiA8UFMvMiBNb3VzZT4gaXJxIDEyIG9uIGF0a2JkYzANCnBzbTA6IFtH
SUFOVC1MT0NLRURdDQpwc20wOiBbSVRIUkVBRF0NCnBzbTA6IG1vZGVsIE5ldE1vdXNlL05l
dFNjcm9sbCBPcHRpY2FsLCBkZXZpY2UgSUQgMA0KZmRjMDogPEVuaGFuY2VkIGZsb3BweSBj
b250cm9sbGVyPiBhdCBwb3J0IDB4M2YwLTB4M2Y1LDB4M2Y3IGlycSA2IGRycSAyIG9uIGlz
YTANCmZkYzA6IFtGSUxURVJdDQpmZDA6IDwxNDQwLUtCIDMuNSIgZHJpdmU+IG9uIGZkYzAg
ZHJpdmUgMA0KcHBjMDogPFBhcmFsbGVsIHBvcnQ+IGF0IHBvcnQgMHgzNzgtMHgzN2YgaXJx
IDcgb24gaXNhMA0KcHBjMDogR2VuZXJpYyBjaGlwc2V0IChFQ1AvUFMyL05JQkJMRSkgaW4g
Q09NUEFUSUJMRSBtb2RlDQpwcGMwOiBGSUZPIHdpdGggMTYvMTYvOCBieXRlcyB0aHJlc2hv
bGQNCnBwYnVzMDogPFBhcmFsbGVsIHBvcnQgYnVzPiBvbiBwcGMwDQpwcGJ1czA6IFtJVEhS
RUFEXQ0KcGxpcDA6IDxQTElQIG5ldHdvcmsgaW50ZXJmYWNlPiBvbiBwcGJ1czANCmxwdDA6
IDxQcmludGVyPiBvbiBwcGJ1czANCmxwdDA6IEludGVycnVwdC1kcml2ZW4gcG9ydA0KcHBp
MDogPFBhcmFsbGVsIEkvTz4gb24gcHBidXMwDQpwcGMwOiBbR0lBTlQtTE9DS0VEXQ0KcHBj
MDogW0lUSFJFQURdDQpzYzA6IDxTeXN0ZW0gY29uc29sZT4gYXQgZmxhZ3MgMHgxMDAgb24g
aXNhMA0Kc2MwOiBWR0EgPDE2IHZpcnR1YWwgY29uc29sZXMsIGZsYWdzPTB4MzAwPg0Kc2lv
MCBhdCBwb3J0IDB4M2Y4LTB4M2ZmIGlycSA0IGZsYWdzIDB4MTAgb24gaXNhMA0Kc2lvMDog
dHlwZSAxNjU1MEENCnNpbzA6IFtGSUxURVJdDQpzaW8xIGF0IHBvcnQgMHgyZjgtMHgyZmYg
aXJxIDMgb24gaXNhMA0Kc2lvMTogdHlwZSAxNjU1MEENCnNpbzE6IFtGSUxURVJdDQp2Z2Ew
OiA8R2VuZXJpYyBJU0EgVkdBPiBhdCBwb3J0IDB4M2MwLTB4M2RmIGlvbWVtIDB4YTAwMDAt
MHhiZmZmZiBvbiBpc2EwDQp1bmtub3duOiA8UE5QMDMwMz4gY2FuJ3QgYXNzaWduIHJlc291
cmNlcyAocG9ydCkNCnVua25vd246IDxQTlAwZjEzPiBjYW4ndCBhc3NpZ24gcmVzb3VyY2Vz
IChpcnEpDQp1bmtub3duOiA8UE5QMDUwMT4gY2FuJ3QgYXNzaWduIHJlc291cmNlcyAocG9y
dCkNCnVua25vd246IDxQTlAwNTAxPiBjYW4ndCBhc3NpZ24gcmVzb3VyY2VzIChwb3J0KQ0K
dW5rbm93bjogPFBOUDA0MDE+IGNhbid0IGFzc2lnbiByZXNvdXJjZXMgKHBvcnQpDQp1bmtu
b3duOiA8UE5QMDcwMD4gY2FuJ3QgYXNzaWduIHJlc291cmNlcyAocG9ydCkNClRpbWVjb3Vu
dGVycyB0aWNrIGV2ZXJ5IDEuMDAwIG1zZWMNCmFjZDA6IENEUk9NIDxNQVRTSElUQSBDUi0x
NzcvN1QwRD4gYXQgYXRhMS1tYXN0ZXIgVURNQTMzDQpzZXMwIGF0IGFzcjAgYnVzIDAgdGFy
Z2V0IDYgbHVuIDANCnNlczA6IDxTVVBFUiBHRU0zMTggMD4gRml4ZWQgUHJvY2Vzc29yIFND
U0ktMiBkZXZpY2UgDQpTTVA6IEFQIENQVSAjMSBMYXVuY2hlZCENClNNUDogQVAgQ1BVICMy
IExhdW5jaGVkIQ0KU01QOiBBUCBDUFUgIzMgTGF1bmNoZWQhDQpkYTAgYXQgYXNyMCBidXMg
MCB0YXJnZXQgMCBsdW4gMA0KZGEwOiA8QURBUFRFQyBSQUlELTUgMzgwRT4gRml4ZWQgRGly
ZWN0IEFjY2VzcyBTQ1NJLTIgZGV2aWNlIA0KVHJ5aW5nIHRvIG1vdW50IHJvb3QgZnJvbSB1
ZnM6L2Rldi9kYTBzMWENCldBUk5JTkc6IC8gd2FzIG5vdCBwcm9wZXJseSBkaXNtb3VudGVk
DQo=
--------------060204090303000808020609--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?47B4AAA3.6060501>