Date: Tue, 10 Aug 2004 20:29:29 +0200 From: "Terrence Koeman" <root@mediamonks.net> To: <freebsd-current@freebsd.org> Subject: Lock order reversal in 5.2-CURRENT Message-ID: <20040810203073.SM01804@manrikigusari>
next in thread | raw e-mail | index | archive | help
This is a multi-part message in MIME format. ------=_NextPart_000_001E_01C47F18.BCC07A50 Content-Type: multipart/mixed; boundary="----=_NextPart_001_001F_01C47F18.BCC07A50" ------=_NextPart_001_001F_01C47F18.BCC07A50 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Hi, In addition to my compiler not working, I've encountered another problem, the system just hung and dropped to the debugger on the serial port. There's nothing in the logfiles, the output on the serial console was: lock order reversal 1st 0xc0645aa0 sched lock (sched lock) @ /usr/src/sys/vm/vm_zeroidle.c:156 2nd 0xc06748c0 sio (sio) @ /usr/src/sys/dev/sio/sio.c:3039 KDB: stack backtrace: kdb_backtrace(c05fe96f,c06748c0,c0636660,c0636660,c060eede) at kdb_backtrace+0x2e witness_checkorder(c06748c0,9,c060eede,bdf,da7a) at witness_checkorder+0x6a6 _mtx_lock_spin_flags(c06748c0,0,c060eede,bdf,a) at _mtx_lock_spin_flags+0x8d siocnputc(c0636840,6b,5,d4dd7bf0,6b) at siocnputc+0x7a cnputc(6b,da7a,1,c15f39a0,c0611cff) at cnputc+0x6a putchar(6b,d4dd7bf0,c064ab40,0,c0686634) at putchar+0x5c kvprintf(c0611cfe,c04c9930,d4dd7bf0,a,d4dd7c10) at kvprintf+0x8d printf(c0611cfe,c,d4dd7c30,c04b2c9e,c0645aa0) at printf+0x55 trap(18,c0640010,10,c15f39a0,d4dd7cfc) at trap+0xc2 calltrap() at calltrap+0x5 --- trap 0xc, eip = 0xc04b3ed2, esp = 0xd4dd7c90, ebp = 0xd4dd7ce0 --- mi_switch(1,0,c060d2be,9c,dbba0) at mi_switch+0x102 vm_pagezero(0,d4dd7d48,c05f8b5a,32b,e209f103) at vm_pagezero+0xe9 fork_exit(c059b140,0,d4dd7d48) at fork_exit+0xc7 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xd4dd7d7c, ebp = 0 --- KDB: enter: witness_checkorder [thread 100051] Stopped at kdb_enter+0x30: leave db> This LOR is not on http://sources.zabbadoz.net/freebsd/lor.html, so I'm guessing this one locked up my system. I have ruled out broken hardware by swapping raid arrays with an identical working system earier because of my compiler problem (see my earlier mail on that). My dmesg and kernel config file are attached. Please advice. -- Regards, Terrence Koeman MediaMonks B.V. (www.mediamonks.com) Please quote all replies in correspondence. ------=_NextPart_001_001F_01C47F18.BCC07A50 Content-Type: application/octet-stream; name="dmesg.dhammapada" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="dmesg.dhammapada" Copyright (c) 1992-2004 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights = reserved. FreeBSD 5.2-CURRENT #8: Thu Aug 5 14:04:45 CEST 2004 terrence@dhammapada.mediamonks.net:/usr/obj/usr/src/sys/DHAMMAPADA WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. WARNING: Kernel preemption is disabled, expect reduced performance. Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Celeron(R) CPU 2.60GHz (2600.45-MHz 686-class CPU) Origin =3D "GenuineIntel" Id =3D 0xf29 Stepping =3D 9 = Features=3D0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PG= E,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> real memory =3D 528416768 (503 MB) avail memory =3D 511713280 (488 MB) npx0: [FAST] npx0: <math processor> on motherboard npx0: INT 16 interface acpi0: <IntelR AWRDACPI> on motherboard acpi0: [GIANT-LOCKED] acpi0: Power Button (fixed) Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 cpu0: <ACPI CPU> on acpi0 acpi_tz0: <Thermal Zone> on acpi0 acpi_button0: <Power Button> on acpi0 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 pci0: <ACPI PCI bus> on pcib0 agp0: <Intel 82845G (845G GMCH) SVGA controller> mem = 0xec100000-0xec17ffff,0xe0000000-0xe7ffffff irq 5 at device 2.0 on pci0 agp0: detected 8060k stolen memory agp0: aperture size is 128M pcib1: <ACPI PCI-PCI bridge> at device 30.0 on pci0 pci1: <ACPI PCI bus> on pcib1 atapci0: <Promise PDC20271 UDMA133 controller> port = 0xa000-0xa00f,0x9c00-0x9c03,0x9800-0x9807,0x9400-0x9403,0x9000-0x9007 = mem 0xec040000-0xec04ffff irq 11 at device 0.0 on pci1 ata2: at 0x9000 on atapci0 ata3: at 0x9800 on atapci0 em0: <Intel(R) PRO/1000 Network Connection, Version - 1.7.25> port = 0xa400-0xa43f mem 0xec000000-0xec01ffff irq 10 at device 5.0 on pci1 em0: [GIANT-LOCKED] em0: Ethernet address: 00:30:48:42:96:0e em0: Speed:N/A Duplex:N/A em1: <Intel(R) PRO/1000 Network Connection, Version - 1.7.25> port = 0xa800-0xa83f mem 0xec020000-0xec03ffff irq 12 at device 6.0 on pci1 em1: [GIANT-LOCKED] em1: Ethernet address: 00:30:48:42:96:0f em1: Speed:N/A Duplex:N/A isab0: <PCI-ISA bridge> at device 31.0 on pci0 isa0: <ISA bus> on isab0 atapci1: <Intel ICH4 UDMA100 controller> port = 0xcc00-0xcc0f,0x376,0x170-0x177,0x3f6,0x1f0-0x1f7 at device 31.1 on pci0 ata0: at 0x1f0 irq 14 on atapci1 ata1: at 0x170 irq 15 on atapci1 pci0: <serial bus, SMBus> at device 31.3 (no driver attached) fdc0: <floppy drive controller> port 0x3f7,0x3f0-0x3f5 irq 6 drq 2 on = acpi0 fdc0: FIFO enabled, 8 bytes threshold fd0: <1440-KB 3.5" drive> on fdc0 drive 0 sio0 port 0x3f8-0x3ff irq 4 on acpi0 sio0: type 16550A, console sio1 port 0x2f8-0x2ff irq 3 on acpi0 sio1: type 16550A atkbdc0: <Keyboard controller (i8042)> port 0x64,0x60 irq 1 on acpi0 orm0: <ISA Option ROM> at iomem 0xcc000-0xd57ff on isa0 pmtimer0 on isa0 sc0: <System console> at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=3D0x100> vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on = isa0 Timecounter "TSC" frequency 2600450072 Hz quality 800 Timecounters tick every 1.000 msec ipfw2 initialized, divert disabled, rule-based forwarding enabled, = default to accept, logging limited to 20 packets/entry by default ATAPI_RESET time =3D 320us acd0: CDROM <CD-224E> at ata0-master UDMA33 ad4: 156334MB <Maxtor 6Y160P0> [317632/16/63] at ata2-master UDMA133 ad6: 156334MB <Maxtor 6Y160P0> [317632/16/63] at ata3-master UDMA133 ar0: 156334MB <ATA RAID1 array> [19929/255/63] status: READY subdisks: disk0 READY on ad4 at ata2-master disk1 READY on ad6 at ata3-master Mounting root from ufs:/dev/ar0s1a em0: Link is up 100 Mbps Full Duplex Expensive timeout(9) function: 0xc05341e0(0) 0.008006324 s em1: Link is up 100 Mbps Full Duplex ------=_NextPart_001_001F_01C47F18.BCC07A50 Content-Type: application/octet-stream; name="DHAMMAPADA" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="DHAMMAPADA" machine i386 cpu I686_CPU ident DHAMMAPADA #To statically compile in device wiring instead of /boot/device.hints #hints "GENERIC.hints" #Default places to look for = devices. #makeoptions DEBUG=3D-g #Build kernel with gdb(1) = debug symbols options SW_WATCHDOG options DIAGNOSTIC options SCHED_4BSD #4BSD scheduler #options SCHED_ULE # ULE scheduler options INET #InterNETworking options FFS #Berkeley Fast Filesystem options SOFTUPDATES #Enable FFS soft updates support options UFS_DIRHASH #Improve performance on big = directories options CD9660 #ISO 9660 Filesystem options PROCFS #Process filesystem (requires = PSEUDOFS) options PSEUDOFS #Pseudo-filesystem framework options COMPAT_43 #Compatible with BSD 4.3 [KEEP = THIS!] #options COMPAT_FREEBSD4 #Compatible with FreeBSD4 #options SCSI_DELAY=3D15000 #Delay (in ms) before probing = SCSI options KTRACE #ktrace(1) support options SYSVSHM #SYSV-style shared memory options SYSVMSG #SYSV-style message queues options SYSVSEM #SYSV-style semaphores options _KPOSIX_PRIORITY_SCHEDULING #Posix P1003_1B real-time = extensions #options KBD_INSTALL_CDEV # install a CDEV entry in /dev options AHC_REG_PRETTY_PRINT # Print register bitfields in = debug # output. Adds ~128k to driver. options AHD_REG_PRETTY_PRINT # Print register bitfields in = debug # output. Adds ~215k to driver. #options PFIL_HOOKS # pfil(9) framework #options QUOTA #options COMPAT_LINUX options DDB_UNATTENDED=20 options KDB options DDB # Enable the kernel debugger options WITNESS_KDB options KDB_TRACE options INVARIANTS # Enable calls of extra sanity = checking options INVARIANT_SUPPORT # Extra sanity checks of = internal structures, required by INVARIANTS options WITNESS # Enable checks to detect = deadlocks and cycles #options WITNESS_SKIPSPIN # Don't run witness on = spinlocks for speed #options SMP # Symmetric MultiProcessor Kernel #device apic # I/O APIC options IPFIREWALL options IPFIREWALL_VERBOSE options IPFIREWALL_VERBOSE_LIMIT=3D20 options IPFIREWALL_DEFAULT_TO_ACCEPT options HZ=3D3000 options DUMMYNET options RANDOM_IP_ID options CPU_ENABLE_SSE options CPU_FASTER_5X86_FPU options NO_F00F_HACK options ACCEPT_FILTER_HTTP #options ACCEPT_FILTER_DATA device isa #device eisa device pci # Floppy drives device fdc # ATA and ATAPI devices device ata device atadisk # ATA disk drives device ataraid # ATA RAID drives device atapicd # ATAPI CDROM drives #device atapifd # ATAPI floppy drives #device atapist # ATAPI tape drives options ATA_STATIC_ID #Static device numbering # atkbdc0 controls both the keyboard and the PS/2 mouse device atkbdc # AT keyboard controller device atkbd # AT keyboard #device psm # PS/2 mouse device vga # VGA video card driver #device splash # Splash screen and screen saver support # syscons is the default console driver, resembling an SCO console device sc device agp # support several AGP chipsets # Floating point support - do not disable. device npx # Power management support (see NOTES for more options) #device apm # Add suspend/resume support for the i8254. device pmtimer # Serial (COM) ports device sio # 8250, 16[45]50 based serial ports #device ppbus #device ppc # parallel port # PCI Ethernet NICs. #device de # DEC/Intel DC21x4x (``Tulip'') device em # Intel PRO/1000 adapter Gigabit = Ethernet Card #device txp # 3Com 3cR990 (``Typhoon'') #device vx # 3Com 3c590, 3c595 (``Vortex'') # Pseudo devices - the number indicates how many units to allocate. device random # Entropy device device loop # Network loopback device ether # Ethernet support device pty # Pseudo-ttys (telnet etc) # The `bpf' device enables the Berkeley Packet Filter. # Be aware of the administrative consequences of enabling this! device bpf # Berkeley packet filte ------=_NextPart_001_001F_01C47F18.BCC07A50-- ------=_NextPart_000_001E_01C47F18.BCC07A50 Content-Type: application/x-pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIKUzCCAj0w ggGmAhEAzbp/VvDf5LxU/iKss3KqVTANBgkqhkiG9w0BAQIFADBfMQswCQYDVQQGEwJVUzEXMBUG A1UEChMOVmVyaVNpZ24sIEluYy4xNzA1BgNVBAsTLkNsYXNzIDEgUHVibGljIFByaW1hcnkgQ2Vy dGlmaWNhdGlvbiBBdXRob3JpdHkwHhcNOTYwMTI5MDAwMDAwWhcNMjgwODAxMjM1OTU5WjBfMQsw CQYDVQQGEwJVUzEXMBUGA1UEChMOVmVyaVNpZ24sIEluYy4xNzA1BgNVBAsTLkNsYXNzIDEgUHVi bGljIFByaW1hcnkgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwgZ8wDQYJKoZIhvcNAQEBBQADgY0A MIGJAoGBAOUZv22jVmEtmUhx9mfeuY3rt56GgAqRDvo4Ja9GiILlc6igmyRdDR/MZW4MsNBWhBiH mgabEKFz37RYOWtuwfYV1aioP6oSBo0xrH+wNNePNGeICc0UEeJORVZpH3gCgNrcR5EpuzbJY1zF 4Ncth3uhtzKwezC6Ki8xqu6jZ9rbAgMBAAEwDQYJKoZIhvcNAQECBQADgYEATD+4i8Zo3+5DMw5d 6abLB4RNejP/khv0Nq3YlSI2aBFsfELM85wuxAc/FLAPT/+Qknb54rxK6Y/NoIAK98Up8YIiXbix 3YEjo3slFUYweRb46gVLlH8dwhzI47f0EEA8E8NfH1PoSOSGtHuhNbB7Jbq4046rPzidADQAmPPR cZQwggNmMIICz6ADAgECAhANi0/uqtIYW/R1ap0p4X/7MA0GCSqGSIb3DQEBAgUAMF8xCzAJBgNV BAYTAlVTMRcwFQYDVQQKEw5WZXJpU2lnbiwgSW5jLjE3MDUGA1UECxMuQ2xhc3MgMSBQdWJsaWMg UHJpbWFyeSBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eTAeFw05ODA1MTIwMDAwMDBaFw0wODA1MTIy MzU5NTlaMIHMMRcwFQYDVQQKEw5WZXJpU2lnbiwgSW5jLjEfMB0GA1UECxMWVmVyaVNpZ24gVHJ1 c3QgTmV0d29yazFGMEQGA1UECxM9d3d3LnZlcmlzaWduLmNvbS9yZXBvc2l0b3J5L1JQQSBJbmNv cnAuIEJ5IFJlZi4sTElBQi5MVEQoYyk5ODFIMEYGA1UEAxM/VmVyaVNpZ24gQ2xhc3MgMSBDQSBJ bmRpdmlkdWFsIFN1YnNjcmliZXItUGVyc29uYSBOb3QgVmFsaWRhdGVkMIGfMA0GCSqGSIb3DQEB AQUAA4GNADCBiQKBgQC7WkSKBBa7Vf0DeootlE8VeDa4DUqyb5xUv7zodyqdufBou5XZMUFweoFL uUgTVi3HCOGEQqvAopKrRFyqQvCCDgLpL/vCO7u+yScKXbawNkIztW5UiE+HSr8Z2vkV6A+Hthzj zMaajn9qJJLj/OBluqexfu/J2zdqyErICQbkmQIDAQABo4G0MIGxMBEGCWCGSAGG+EIBAQQEAwIB BjA1BgNVHR8ELjAsMCqgKKAmhiRodHRwOi8vY3JsLnZlcmlzaWduLmNvbS9wY2ExLjEuMS5jcmww RwYDVR0gBEAwPjA8BgtghkgBhvhFAQcBATAtMCsGCCsGAQUFBwIBFh93d3cudmVyaXNpZ24uY29t L3JlcG9zaXRvcnkvUlBBMA8GA1UdEwQIMAYBAf8CAQAwCwYDVR0PBAQDAgEGMA0GCSqGSIb3DQEB AgUAA4GBAEJ8Dt+MeUysvwjsTVUvUImgxV5OLl6VMpt5rWURCxxKUsTVqDEhjt4Qm2wIxQfmA7nn yDR4CQnyvAZC+FqMg9GK3qoi9dnjIdLPZYwGM7DNILIzzQq9PuGdwTWpZLCnpSRb6fFo6xPEfDf0 lGQNmsW9MxfvgzOgPuWqPq7Ycx+tMIIEpDCCBA2gAwIBAgIQZx6EJ4oHSmQocGtEpogzPzANBgkq hkiG9w0BAQQFADCBzDEXMBUGA1UEChMOVmVyaVNpZ24sIEluYy4xHzAdBgNVBAsTFlZlcmlTaWdu IFRydXN0IE5ldHdvcmsxRjBEBgNVBAsTPXd3dy52ZXJpc2lnbi5jb20vcmVwb3NpdG9yeS9SUEEg SW5jb3JwLiBCeSBSZWYuLExJQUIuTFREKGMpOTgxSDBGBgNVBAMTP1ZlcmlTaWduIENsYXNzIDEg Q0EgSW5kaXZpZHVhbCBTdWJzY3JpYmVyLVBlcnNvbmEgTm90IFZhbGlkYXRlZDAeFw0wNDAyMDQw MDAwMDBaFw0wNTAyMDMyMzU5NTlaMIIBFjEXMBUGA1UEChMOVmVyaVNpZ24sIEluYy4xHzAdBgNV BAsTFlZlcmlTaWduIFRydXN0IE5ldHdvcmsxRjBEBgNVBAsTPXd3dy52ZXJpc2lnbi5jb20vcmVw b3NpdG9yeS9SUEEgSW5jb3JwLiBieSBSZWYuLExJQUIuTFREKGMpOTgxHjAcBgNVBAsTFVBlcnNv bmEgTm90IFZhbGlkYXRlZDE0MDIGA1UECxMrRGlnaXRhbCBJRCBDbGFzcyAxIC0gTWljcm9zb2Z0 IEZ1bGwgU2VydmljZTEYMBYGA1UEAxQPVGVycmVuY2UgS29lbWFuMSIwIAYJKoZIhvcNAQkBFhNy b290QG1lZGlhbW9ua3MubmV0MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDKUqZ0P8RUaCCe aemq1V8OtPimM3jxXZY6yuWtQz74POz82LhrkctttQoqujz5SubFWsQ15KsVnK4Nt+gTmtUELQVa I4uUebypbChhf1fxFerA7Rx8SYL4Ez9Fvco0tn89FRht6C/xoc/Ms/1YkQQcfnu2sqohx53S5Pm0 Ffo/ZQIDAQABo4IBODCCATQwCQYDVR0TBAIwADCBrAYDVR0gBIGkMIGhMIGeBgtghkgBhvhFAQcB ATCBjjAoBggrBgEFBQcCARYcaHR0cHM6Ly93d3cudmVyaXNpZ24uY29tL0NQUzBiBggrBgEFBQcC AjBWMBUWDlZlcmlTaWduLCBJbmMuMAMCAQEaPVZlcmlTaWduJ3MgQ1BTIGluY29ycC4gYnkgcmVm ZXJlbmNlIGxpYWIuIGx0ZC4gKGMpOTcgVmVyaVNpZ24wEQYJYIZIAYb4QgEBBAQDAgeAMDAGCmCG SAGG+EUBBgcEIhYgMWE5NjkyOTM3Y2MyOTFhMzZkZjAxN2Q4NDBjNDRiMTYwMwYDVR0fBCwwKjAo oCagJIYiaHR0cDovL2NybC52ZXJpc2lnbi5jb20vY2xhc3MxLmNybDANBgkqhkiG9w0BAQQFAAOB gQAmpan0cUGODHHRIbkVFBHbTOumbMmEO3TR6d9z3LVO9cU5YDC/8BI9e5DZ7Kv43p/ldaGVo2ua ZwtvwL6o/iDcP8UDw+WmjSwzp6tck5cVRDEi+q9nCe3JAYBeRgSJlP5JGOirFpLuDTwL4UPdE0em p9ELBC+sXACm2C3u19hlKTGCBD4wggQ6AgEBMIHhMIHMMRcwFQYDVQQKEw5WZXJpU2lnbiwgSW5j LjEfMB0GA1UECxMWVmVyaVNpZ24gVHJ1c3QgTmV0d29yazFGMEQGA1UECxM9d3d3LnZlcmlzaWdu LmNvbS9yZXBvc2l0b3J5L1JQQSBJbmNvcnAuIEJ5IFJlZi4sTElBQi5MVEQoYyk5ODFIMEYGA1UE AxM/VmVyaVNpZ24gQ2xhc3MgMSBDQSBJbmRpdmlkdWFsIFN1YnNjcmliZXItUGVyc29uYSBOb3Qg VmFsaWRhdGVkAhBnHoQnigdKZChwa0SmiDM/MAkGBSsOAwIaBQCgggKyMBgGCSqGSIb3DQEJAzEL BgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTA0MDgxMDE4MjkyOVowIwYJKoZIhvcNAQkEMRYE FJLw3mAUAkftqneRILtknUw+jKhuMGcGCSqGSIb3DQEJDzFaMFgwCgYIKoZIhvcNAwcwDgYIKoZI hvcNAwICAgCAMA0GCCqGSIb3DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMAcGBSsOAwIa MAoGCCqGSIb3DQIFMIHyBgkrBgEEAYI3EAQxgeQwgeEwgcwxFzAVBgNVBAoTDlZlcmlTaWduLCBJ bmMuMR8wHQYDVQQLExZWZXJpU2lnbiBUcnVzdCBOZXR3b3JrMUYwRAYDVQQLEz13d3cudmVyaXNp Z24uY29tL3JlcG9zaXRvcnkvUlBBIEluY29ycC4gQnkgUmVmLixMSUFCLkxURChjKTk4MUgwRgYD VQQDEz9WZXJpU2lnbiBDbGFzcyAxIENBIEluZGl2aWR1YWwgU3Vic2NyaWJlci1QZXJzb25hIE5v dCBWYWxpZGF0ZWQCEGcehCeKB0pkKHBrRKaIMz8wgfQGCyqGSIb3DQEJEAILMYHkoIHhMIHMMRcw FQYDVQQKEw5WZXJpU2lnbiwgSW5jLjEfMB0GA1UECxMWVmVyaVNpZ24gVHJ1c3QgTmV0d29yazFG MEQGA1UECxM9d3d3LnZlcmlzaWduLmNvbS9yZXBvc2l0b3J5L1JQQSBJbmNvcnAuIEJ5IFJlZi4s TElBQi5MVEQoYyk5ODFIMEYGA1UEAxM/VmVyaVNpZ24gQ2xhc3MgMSBDQSBJbmRpdmlkdWFsIFN1 YnNjcmliZXItUGVyc29uYSBOb3QgVmFsaWRhdGVkAhBnHoQnigdKZChwa0SmiDM/MA0GCSqGSIb3 DQEBAQUABIGARMXYUGH0ShLKROXu2revI7qPPFbrcFg5XG5dNYfN7qAwDgF/2kOHPOU0aTZg3OrX hfAq6FgQxbuU0pIOYprN5XdGhMhx05VGaoXpn6v4zIhA0atcwpk6fCcGtDu2eqWZ+zqp9ivsyFpk 6VdadVPPJdHUb0IVbeaWtws/bEZLp9sAAAAAAAA= ------=_NextPart_000_001E_01C47F18.BCC07A50--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20040810203073.SM01804>