Date: Fri, 14 Jan 2005 11:47:26 +1000 From: Matthew Sullivan <matthew@uq.edu.au> To: freebsd-current@freebsd.org Subject: Re: Fatal Trap 12: Page fault while in kernel mode (racoon/amd64/5.3-RELEASE-p4) Message-ID: <41E724AE.3040809@uq.edu.au> In-Reply-To: <41E6B56B.6020207@gmx.de> References: <41E44CD0.1000008@uq.edu.au> <41E5F22A.6010607@uq.edu.au> <20050113093955.P12838@carver.gumbysoft.com> <41E6B56B.6020207@gmx.de>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a cryptographically signed message in MIME format. --------------ms080104050604090205060905 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Phil Schulz wrote: > [lost the original mail...] > > On 01/13/05 18:41, Doug White wrote: > >> On Thu, 13 Jan 2005, Matthew Sullivan wrote: >> >>> I'm going to have to put this machine into production within the next 7 >>> days so any help would be really great, also any extra info anyone >>> requires is available. As I said in my last this is 100% reproducable. >>> Dumps are not available - calling panic will lock the system solid. >>> Calling boot(0) seems to work fine though... >> > > When you are in the debugger, can you type "call doadump"? Yes thanks, between you mailing and my last post I spotted this call in someone elses woes an took immediate advantage ;-) Results are at: http://www.au.sorbs.net/~matthew/freebsd/ along with the core, the kernel, the symbols etc etc etc.. ;-) > Last time I tried to debug a panic, I've had problems with panics not > generating a core dump as well. Calling the doadump() function > manually worked, though. Gets me here: Fatal trap 12: page fault while in kernel mode fault virtual address = 0x39 fault code = supervisor write, page not present instruction pointer = 0x8:0xffffffff80307a70 stack pointer = 0x10:0xffffffff93cc0860 frame pointer = 0x10:0xffffffff93cc0960 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 480 (racoon) [thread 100068] Stopped at keydb_newsecasvar+0x100: decl %ecx db> w Nothing written. db> where keydb_newsecasvar() at keydb_newsecasvar+0x100 raw_usend() at raw_usend+0x60 key_send() at key_send+0xa sosend() at sosend+0x626 kern_sendit() at kern_sendit+0x113 sendit() at sendit+0x5f sendto() at sendto+0x4d syscall() at syscall+0x50c Xfast_syscall() at Xfast_syscall+0xa8 --- syscall (133, FreeBSD ELF64, sendto), rip = 0x800a63da8, rsp = 0x7fffffffec38, rbp = 0x2 --- db> call doadump Dumping 479 MB 16 32 48 64 80 96 112 128 144 160 176 192 208 224 240 256 272 288 304 320 336 352 368 384 400 416 432 448 464 Dump complete 0xf Then we can proceed ;-) (kgdb) file /usr/obj/usr/src/sys/DESPERADO/kernel.debug Reading symbols from /usr/obj/usr/src/sys/DESPERADO/kernel.debug...done. (kgdb) where #0 doadump () at pcpu.h:167 #1 0xffffffff80172736 in db_fncall (dummy1=0, dummy2=0, dummy3=0, dummy4=0x0) at /usr/src/sys/ddb/db_command.c:531 #2 0xffffffff80172bc5 in db_command_loop () at /usr/src/sys/ddb/db_command.c:349 #3 0xffffffff80174a53 in db_trap (type=-1815345680, code=0) at /usr/src/sys/ddb/db_main.c:221 #4 0xffffffff8023070b in kdb_trap (type=12, code=0, tf=0xffffffff93cc07b0) at /usr/src/sys/kern/subr_kdb.c:418 #5 0xffffffff80371dae in trap_fatal (frame=0xffffffff93cc07b0, eva=18446742974681318688) at /usr/src/sys/amd64/amd64/trap.c:626 #6 0xffffffff80372143 in trap_pfault (frame=0xffffffff93cc07b0, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:554 #7 0xffffffff803723a4 in trap (frame= {tf_rdi = -1099028463104, tf_rsi = 640, tf_rdx = -1815344784, tf_rcx = -1815344833, tf_r8 = 160, tf_r9 = -1099028232928 , tf_rax = -1815345008, tf_rbx = -2144306579, tf_rbp = -1815344800, tf_r10 = -2142160512, tf_r11 = -1815344624, tf_r12 = 57, tf_r13 = 0, tf_r14 = 0, tf_r15 = -1099140303240, tf_trapno = 12, tf_addr = 57, tf_flags = -1099132378656, tf_err = 2, tf_rip = -2144306576, tf_cs = 8, tf_rflags = 66054, tf_rsp = -1815345040, tf_ss = 16}) at /usr/src/sys/amd64/amd64/trap.c:333 #8 0xffffffff80361bab in calltrap () at /usr/src/sys/amd64/amd64/exception.S:171 #9 0xffffff001ccc8200 in ?? () #10 0x0000000000000280 in ?? () #11 0xffffffff93cc0970 in ?? () #12 0xffffffff93cc093f in ?? () #13 0x00000000000000a0 in ?? () #14 0xffffff001cd00520 in ?? () #15 0xffffffff93cc0890 in ?? () #16 0xffffffff80307a6d in keydb_newsecasvar () at /usr/src/sys/netkey/keydb.c:187 #17 0xffffffff8029cfc0 in raw_usend (so=0x0, flags=0, m=0x0, nam=0x0, control=0x0, td=0x0) at /usr/src/sys/net/raw_usrreq.c:263 #18 0xffffffff8030845a in key_send (so=0x0, flags=0, m=0x0, nam=0x0, control=0x0, td=0x0) at /usr/src/sys/netkey/keysock.c:442 #19 0xffffffff80253bc6 in sosend (so=0xffffff001621f678, addr=0x0, uio=0xffffffff93cc0a80, top=0xffffff001ccc8200, control=0x0, flags=0, td=0xffffff001cd00520) at /usr/src/sys/kern/uipc_socket.c:815 #20 0xffffffff8025ba73 in kern_sendit (td=0xffffff001cd00520, s=4, mp=0xffffffff93cc0b50, flags=0, control=0x0) at /usr/src/sys/kern/uipc_syscalls.c:738 #21 0xffffffff8025ca5f in sendit (td=0xffffff001cd00520, s=4, mp=0xffffffff93cc0b50, flags=0) at /usr/src/sys/kern/uipc_syscalls.c:682 #22 0xffffffff8025cbed in sendto (td=0x0, uap=0x0) at /usr/src/sys/kern/uipc_syscalls.c:795 #23 0xffffffff80372b6c in syscall (frame= {tf_rdi = 4, tf_rsi = 5660864, tf_rdx = 16, tf_rcx = 0, tf_r8 = 0, tf_r9 = 0, tf_rax = 133, tf_rbx = 5660880, tf_rbp = 2, tf_r10 = -2141993928, tf_r11 = 514, tf_r12 = 5660864, tf_r13 = 16, tf_r14 = 7, tf_r15 = 4, tf_trapno = 12, tf_addr = 42840 00, tf_flags = 0, tf_err = 2, tf_rip = 34370633128, tf_cs = 43, tf_rflags = 514, tf_rsp = 140737488350264, tf_ss = 35}) at /usr/src/sys/amd64/amd64/trap.c:763 #24 0xffffffff80361ce8 in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:248 . . . (kgdb) frame 16 #16 0xffffffff80307a6d in keydb_newsecasvar () at /usr/src/sys/netkey/keydb.c:187 187 p->id = said; (kgdb) info loc p = (struct secasvar *) 0x39 q = (struct secasvar *) 0xffffffff80307a6d said = 0 (kgdb) l *0xffffffff80307a70 0xffffffff80307a70 is in keydb_newsecasvar (/usr/src/sys/netkey/keydb.c:191). 186 bzero(p, sizeof(*p)); 187 p->id = said; 188 if (q) 189 TAILQ_INSERT_AFTER(&satailq, q, p, tailq); 190 else 191 TAILQ_INSERT_TAIL(&satailq, p, tailq); 192 return p; 193 } 194 195 void (kgdb) l 156,193 156 /* 157 * secasvar management (reference counted) 158 */ 159 struct secasvar * 160 keydb_newsecasvar() 161 { 162 struct secasvar *p, *q; 163 static u_int32_t said = 0; 164 165 p = (struct secasvar *)malloc(sizeof(*p), M_SECA, M_NOWAIT); 166 if (!p) 167 return p; 168 169 again: 170 said++; 171 if (said == 0) 172 said++; 173 TAILQ_FOREACH(q, &satailq, tailq) { 174 if (q->id == said) 175 goto again; 176 if (TAILQ_NEXT(q, tailq)) { 177 if (q->id < said && said < TAILQ_NEXT(q, tailq)->id) 178 break; 179 if (q->id + 1 < TAILQ_NEXT(q, tailq)->id) { 180 said = q->id + 1; 181 break; 182 } 183 } 184 } 185 186 bzero(p, sizeof(*p)); 187 p->id = said; 188 if (q) 189 TAILQ_INSERT_AFTER(&satailq, q, p, tailq); 190 else 191 TAILQ_INSERT_TAIL(&satailq, p, tailq); 192 return p; 193 } Without a clue on the FreeBSD kernel and not enough time to get stuck in (atm) I'm not geing to be able to get further without help. Regards, -- Matthew Sullivan Specialist Systems Programmer Information Technology Services The University of Queensland --------------ms080104050604090205060905 Content-Type: application/x-pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIG7DCC A3IwggJaoAMCAQICASowDQYJKoZIhvcNAQEEBQAwgaMxCzAJBgNVBAYTAkFVMRMwEQYDVQQI EwpRdWVlbnNsYW5kMREwDwYDVQQHEwhCcmlzYmFuZTElMCMGA1UEChMcVGhlIFVuaXZlcnNp dHkgb2YgUXVlZW5zbGFuZDEoMCYGA1UECxMfSW5mb3JtYXRpb24gVGVjaG5vbG9neSBTZXJ2 aWNlczEbMBkGA1UEAxMSQ2VydGlmaWNhdGUgU2VydmVyMB4XDTA0MDEyMTIzMzYyMVoXDTA2 MDEyMTIzMzYyMVowgbIxCzAJBgNVBAYTAkFVMSUwIwYDVQQKExxUaGUgVW5pdmVyc2l0eSBv ZiBRdWVlbnNsYW5kMScwJQYDVQQLEx5JbmZvcm1hdGlvbiBUZWNub2xvZ3kgU2VydmljZXMx FjAUBgoJkiaJk/IsZAEBEwZjY21hdHQxGTAXBgNVBAMTEE1hdHRoZXcgU3VsbGl2YW4xIDAe BgkqhkiG9w0BCQEWEW1hdHRoZXdAdXEuZWR1LmF1MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJB AJsUfrw/QUqKIzDverWc2F4GFFRZmIeO+bAl+7BM6x/9frMzOtygx4QGb4oQwtOE8Sda1aIs v+yJF3Di9EuUyvMCAwEAAaNoMGYwDgYDVR0PAQH/BAQDAgXgMBEGCWCGSAGG+EIBAQQEAwIF oDAfBgNVHSMEGDAWgBQmqtoyueiWTYZBinvsnzeOWLtUuzAgBgNVHREEGTAXgRVtYXR0aGV3 QGl0cy51cS5lZHUuYXUwDQYJKoZIhvcNAQEEBQADggEBAF2gZrkqZsZlHd4K/+yBN6qrpD61 hctDf7/Eg4jk6DMknEs6nvHMFUMZ4SXvkqPLnHBygTARKAs7qBSLd7mUUBOOQEgk6ovQVY6S 1CDSt3P9O6wjG0K1igtk8v6u7lkQ8p2STXqrOePVINdaucUgBO/IpeUtt9ATl1qvPTWyM/fz oUZsIKeYjNQVEQsuimrZjdbIAFxdl1fggSngUv64wBn8wCssGrPZIZA2lpBBEW1wejoWrDOH IIr+SspGd0i8MovDTMRSvgTERLki17FU/ANilcrSXiODKeIvpXhnQqVScnsoMSZmBmN2QIoG SnBjNK5mYxx5E3v20VOwtP1hVdEwggNyMIICWqADAgECAgEqMA0GCSqGSIb3DQEBBAUAMIGj MQswCQYDVQQGEwJBVTETMBEGA1UECBMKUXVlZW5zbGFuZDERMA8GA1UEBxMIQnJpc2JhbmUx JTAjBgNVBAoTHFRoZSBVbml2ZXJzaXR5IG9mIFF1ZWVuc2xhbmQxKDAmBgNVBAsTH0luZm9y bWF0aW9uIFRlY2hub2xvZ3kgU2VydmljZXMxGzAZBgNVBAMTEkNlcnRpZmljYXRlIFNlcnZl cjAeFw0wNDAxMjEyMzM2MjFaFw0wNjAxMjEyMzM2MjFaMIGyMQswCQYDVQQGEwJBVTElMCMG A1UEChMcVGhlIFVuaXZlcnNpdHkgb2YgUXVlZW5zbGFuZDEnMCUGA1UECxMeSW5mb3JtYXRp b24gVGVjbm9sb2d5IFNlcnZpY2VzMRYwFAYKCZImiZPyLGQBARMGY2NtYXR0MRkwFwYDVQQD ExBNYXR0aGV3IFN1bGxpdmFuMSAwHgYJKoZIhvcNAQkBFhFtYXR0aGV3QHVxLmVkdS5hdTBc MA0GCSqGSIb3DQEBAQUAA0sAMEgCQQCbFH68P0FKiiMw73q1nNheBhRUWZiHjvmwJfuwTOsf /X6zMzrcoMeEBm+KEMLThPEnWtWiLL/siRdw4vRLlMrzAgMBAAGjaDBmMA4GA1UdDwEB/wQE AwIF4DARBglghkgBhvhCAQEEBAMCBaAwHwYDVR0jBBgwFoAUJqraMrnolk2GQYp77J83jli7 VLswIAYDVR0RBBkwF4EVbWF0dGhld0BpdHMudXEuZWR1LmF1MA0GCSqGSIb3DQEBBAUAA4IB AQBdoGa5KmbGZR3eCv/sgTeqq6Q+tYXLQ3+/xIOI5OgzJJxLOp7xzBVDGeEl75Kjy5xwcoEw ESgLO6gUi3e5lFATjkBIJOqL0FWOktQg0rdz/TusIxtCtYoLZPL+ru5ZEPKdkk16qznj1SDX WrnFIATvyKXlLbfQE5darz01sjP386FGbCCnmIzUFRELLopq2Y3WyABcXZdX4IEp4FL+uMAZ /MArLBqz2SGQNpaQQRFtcHo6FqwzhyCK/krKRndIvDKLw0zEUr4ExES5ItexVPwDYpXK0l4j gyniL6V4Z0KlUnJ7KDEmZgZjdkCKBkpwYzSuZmMceRN79tFTsLT9YVXRMYIDQDCCAzwCAQEw gakwgaMxCzAJBgNVBAYTAkFVMRMwEQYDVQQIEwpRdWVlbnNsYW5kMREwDwYDVQQHEwhCcmlz YmFuZTElMCMGA1UEChMcVGhlIFVuaXZlcnNpdHkgb2YgUXVlZW5zbGFuZDEoMCYGA1UECxMf SW5mb3JtYXRpb24gVGVjaG5vbG9neSBTZXJ2aWNlczEbMBkGA1UEAxMSQ2VydGlmaWNhdGUg U2VydmVyAgEqMAkGBSsOAwIaBQCgggItMBgGCSqGSIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJ KoZIhvcNAQkFMQ8XDTA1MDExNDAxNDcyNlowIwYJKoZIhvcNAQkEMRYEFHw1D9hCgiQVJpeY tVYR7YG48m0OMFIGCSqGSIb3DQEJDzFFMEMwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCA MA0GCCqGSIb3DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMIG6BgkrBgEEAYI3EAQx gawwgakwgaMxCzAJBgNVBAYTAkFVMRMwEQYDVQQIEwpRdWVlbnNsYW5kMREwDwYDVQQHEwhC cmlzYmFuZTElMCMGA1UEChMcVGhlIFVuaXZlcnNpdHkgb2YgUXVlZW5zbGFuZDEoMCYGA1UE CxMfSW5mb3JtYXRpb24gVGVjaG5vbG9neSBTZXJ2aWNlczEbMBkGA1UEAxMSQ2VydGlmaWNh dGUgU2VydmVyAgEqMIG8BgsqhkiG9w0BCRACCzGBrKCBqTCBozELMAkGA1UEBhMCQVUxEzAR BgNVBAgTClF1ZWVuc2xhbmQxETAPBgNVBAcTCEJyaXNiYW5lMSUwIwYDVQQKExxUaGUgVW5p dmVyc2l0eSBvZiBRdWVlbnNsYW5kMSgwJgYDVQQLEx9JbmZvcm1hdGlvbiBUZWNobm9sb2d5 IFNlcnZpY2VzMRswGQYDVQQDExJDZXJ0aWZpY2F0ZSBTZXJ2ZXICASowDQYJKoZIhvcNAQEB BQAEQBgM4n0c4dLW7KaHLpmLubtt7i79lsSOjHzLLO/O/zEbl0VkxCTmmZNg8WGdzbRTgrh6 sq8dxxNh3k5ncR1+gYUAAAAAAAA= --------------ms080104050604090205060905--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?41E724AE.3040809>