From owner-freebsd-current@FreeBSD.ORG Wed Nov 19 14:31:14 2003 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id ACA9816A4CE; Wed, 19 Nov 2003 14:31:14 -0800 (PST) Received: from transport.cksoft.de (transport.cksoft.de [62.111.66.27]) by mx1.FreeBSD.org (Postfix) with ESMTP id D883344015; Wed, 19 Nov 2003 14:31:12 -0800 (PST) (envelope-from bzeeb-lists@lists.zabbadoz.net) Received: from localhost (localhost [127.0.0.1]) by transport.cksoft.de (Postfix) with ESMTP id EB1561FF90C; Wed, 19 Nov 2003 23:31:11 +0100 (CET) Received: by transport.cksoft.de (Postfix, from userid 66) id 5EEF61FF908; Wed, 19 Nov 2003 23:31:10 +0100 (CET) Received: by mail.int.zabbadoz.net (Postfix, from userid 1060) id 61B67154DF; Wed, 19 Nov 2003 22:30:43 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mail.int.zabbadoz.net (Postfix) with ESMTP id 563FF154D7; Wed, 19 Nov 2003 22:30:44 +0000 (UTC) Date: Wed, 19 Nov 2003 22:30:44 +0000 (UTC) From: "Bjoern A. Zeeb" X-X-Sender: bz@e0-0.zab2.int.zabbadoz.net To: Robert Watson , Hajimu UMEMOTO In-Reply-To: Message-ID: References: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Virus-Scanned: by AMaViS snapshot-20020300 cc: current@freebsd.org Subject: Re: another trap 12 while in kernel mode (now with trace) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Nov 2003 22:31:14 -0000 On Wed, 19 Nov 2003, Robert Watson wrote: > You should still be able to turn the bad instruction pointer into at least > a function name using nm on your compiled kernel. Run nm on the kernel > binary, and search through it until you find the symbols just before > (lower) and just after (higher) than the value listed for the instruction > pointer below. That would at least tell us what function the fault is > occuring in. I suspect to get anywhere useful, we'll need the full > debugging results, but it's probably a useful start, and it will allow us > to be sure the panics you might get after you have debugging support > compiled in are the same failure mode. Ok, I could trigger it or s.th. very related by doing this: crisco# ps ax | grep rac | grep -v grep 348 ?? Is 0:04.50 /usr/local/sbin/racoon -f /usr/local/etc/racoon/racoon.conf crisco# kill -9 348 Fatal trap 12: page fault while in kernel mode fault virtual address = 0xdeadc0df fault code = supervisor read, page not present instruction pointer = 0x8:0xc055088c stack pointer = 0x10:0xc9b5c8dc frame pointer = 0x10:0xc9b5c8e4 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 485 (sshd) kernel: type 12 trap, code=0 Stopped at key_cmpspidx_withmask+0x2c: movb 0x1(%ebx),%al db> trace key_cmpspidx_withmask(deadc0de,c9b5c914) at key_cmpspidx_withmask+0x2c key_allocsp(0,c9b5c914,2,16000210,c1426f0a) at key_allocsp+0x8b ipsec4_getpolicybyaddr(c2fa7c00,2,0,c9b5ca88,0) at ipsec4_getpolicybyaddr+0x69 ip_output(c2fa7c00,0,c3a36b00,0,0) at ip_output+0x770 tcp_output(c3a386f4,0,c3a36ac4,0,30) at tcp_output+0xbf0 tcp_usr_send(c3a091e0,0,c2fa7700,0,0) at tcp_usr_send+0x18b sosend(c3a091e0,0,c9b5cc88,c2fa7700,0) at sosend+0x405 soo_write(c3a9b7b4,c9b5cc88,c3a58c80,0,c2f83a00) at soo_write+0x37 dofilewrite(c2f83a00,c3a9b7b4,5,807a000,30) at dofilewrite+0x7b write(c2f83a00,c9b5cd14,3,ab,286) at write+0x3e syscall(2f,2f,2f,806cc08,8076860) at syscall+0x202 Xint0x80_syscall() at Xint0x80_syscall+0x1d --- syscall (4, FreeBSD ELF32, write), eip = 0x28258b6f, esp = 0xbfbfe4cc, ebp = 0xbfbfe4f8 --- db> where key_cmpspidx_withmask(deadc0de,c9b5c914) at key_cmpspidx_withmask+0x2c key_allocsp(0,c9b5c914,2,16000210,c1426f0a) at key_allocsp+0x8b ipsec4_getpolicybyaddr(c2fa7c00,2,0,c9b5ca88,0) at ipsec4_getpolicybyaddr+0x69 ip_output(c2fa7c00,0,c3a36b00,0,0) at ip_output+0x770 tcp_output(c3a386f4,0,c3a36ac4,0,30) at tcp_output+0xbf0 tcp_usr_send(c3a091e0,0,c2fa7700,0,0) at tcp_usr_send+0x18b sosend(c3a091e0,0,c9b5cc88,c2fa7700,0) at sosend+0x405 soo_write(c3a9b7b4,c9b5cc88,c3a58c80,0,c2f83a00) at soo_write+0x37 dofilewrite(c2f83a00,c3a9b7b4,5,807a000,30) at dofilewrite+0x7b write(c2f83a00,c9b5cd14,3,ab,286) at write+0x3e syscall(2f,2f,2f,806cc08,8076860) at syscall+0x202 Xint0x80_syscall() at Xint0x80_syscall+0x1d --- syscall (4, FreeBSD ELF32, write), eip = 0x28258b6f, esp = 0xbfbfe4cc, ebp = 0xbfbfe4f8 --- db> show locks exclusive sleep mutex inp r = 0 (0xc3a36b74) locked @ HEAD/compile-20031119-1814/sys/netinet/tcp_usrreq.c:670 exclusive sleep mutex tcp r = 0 (0xc063bc6c) locked @ HEAD/compile-20031119-1814/sys/netinet/tcp_usrreq.c:653 exclusive sleep mutex Giant r = 0 (0xc0610680) locked @ HEAD/compile-20031119-1814/sys/vm/vm_fault.c:223 db> show reg cs 0x8 ds 0x10 es 0x10 fs 0x18 ss 0x10 eax 0 ecx 0 edx 0 ebx 0xdeadc0de esp 0xc9b5c8dc ebp 0xc9b5c8e4 esi 0xc9b5c914 edi 0x2 eip 0xc055088c key_cmpspidx_withmask+0x2c efl 0x10286 dr0 0 dr1 0 dr2 0 dr3 0 dr4 0xffff0ff0 dr5 0x400 dr6 0xffff0ff0 dr7 0x400 key_cmpspidx_withmask+0x2c: movb 0x1(%ebx),%al db> ps pid proc uarea uid ppid pgrp flag stat wmesg wchan cmd 804 c3adcc5c ca470000 0 455 804 0004002 [SLP]ttyin 0xc3976a10] csh 550 c3ae0388 ca474000 0 549 550 0004002 [RUNQ] csh 549 c3adc710 ca46d000 1060 486 549 0004102 [SLP]wait 0xc3adc710] su 486 c3936c5c ca06f000 1060 485 486 0004002 [SLP]pause 0xc39e5000] csh 485 c39af000 ca08f000 1060 482 482 0000100 [CPU 0] sshd 482 c3a3f388 ca407000 0 284 482 0000100 [SLP]sbwait 0xc3ac442c] sshd 460 c39b1e20 ca0c4000 25 458 458 0004100 [CV]select 0xc0638a04] qmgr 459 c39b18d4 ca0c1000 25 458 458 0004100 [CV]select 0xc0638a04] pickup 458 c39b1710 ca0c0000 0 1 458 0004100 [CV]select 0xc0638a04] master 455 c39afa98 ca095000 0 1 455 0004102 [SLP]wait 0xc39afa98] login 384 c39b1388 ca0be000 0 1 384 0000000 [CV]select 0xc0638a04] inetd 342 c39b154c ca0bf000 0 1 342 0000000 [CV]select 0xc0638a04] ppp 339 c39af388 ca091000 0 1 339 0000000 [CV]select 0xc0638a04] ppp 329 c3a3f710 ca409000 0 1 329 0000000 [CV]select 0xc0638a04] dhcrelay 305 c39b1c5c ca0c3000 0 1 305 0000000 [SLP]nanslp 0xc061251c] cron 284 c3936a98 ca06e000 0 1 284 0000100 [CV]select 0xc0638a04] sshd 273 c3936e20 ca070000 0 1 273 0000000 [CV]select 0xc0638a04] ntpd 231 c39af54c ca092000 0 1 231 0000000 [CV]select 0xc0638a04] syslogd 158 c39af710 ca093000 0 1 158 0000000 [CV]select 0xc0638a04] natd 36 c39afc5c ca0ba000 0 0 0 0000204 [SLP]mdwait 0xc3986c00] md0 35 c39afe20 ca0bb000 0 0 0 0000204 [SLP]vlruwt 0xc39afe20] vnlru 34 c39b1000 ca0bc000 0 0 0 0000204 [SLP]syncer 0xc0611ee0] syncer 9 c392754c ca03e000 0 0 0 0000204 [SLP]psleep 0xc0638e6c] bufdaemon 8 c3927710 ca03f000 0 0 0 000020c [SLP]pgzero 0xc0649268] pagezero 7 c39278d4 ca040000 0 0 0 0000204 [SLP]psleep 0xc06492c0] vmdaemon 6 c3927a98 ca041000 0 0 0 0000204 [SLP]psleep 0xc06492ac] pagedaemon 33 c3927c5c ca042000 0 0 0 0000204 [IWAIT] swi0: tty:sio 5 c3927e20 ca043000 0 0 0 0000204 [SLP]tqthr 0xc0613108] taskqueue 32 c3936000 ca044000 0 0 0 0000204 new [IWAIT] swi6:+ 31 c39361c4 ca069000 0 0 0 0000204 new [IWAIT] swi7: task queue 30 c3936388 ca06a000 0 0 0 0000204 new [IWAIT] swi5:+ 29 c393654c ca06b000 0 0 0 0000204 [SLP]- 0xc06bd500] random 4 c3936710 ca06c000 0 0 0 0000204 [SLP]- 0xc060cc60] g_down 3 c39368d4 ca06d000 0 0 0 0000204 [SLP]- 0xc060cc5c] g_up 2 c2f821c4 c9b3e000 0 0 0 0000204 [SLP]- 0xc060cc54] g_event 28 c2f82388 c9b3f000 0 0 0 0000204 [IWAIT] swi1: net 27 c2f8254c c9b40000 0 0 0 0000204 new [IWAIT] swi4: vm 26 c2f82710 c9b41000 0 0 0 000020c [RUNQ] swi8: tty:sio clock 25 c2f828d4 c9b66000 0 0 0 0000204 [IWAIT] irq15: fxp2 ata1 24 c2f82a98 c9b67000 0 0 0 0000204 new [IWAIT] irq14: ata0 23 c2f82c5c c9b68000 0 0 0 0000204 new [IWAIT] irq13: 22 c2f82e20 c9b69000 0 0 0 0000204 [IWAIT] irq12: fxp1 21 c3927000 ca03b000 0 0 0 0000204 [IWAIT] irq11: ed0 20 c39271c4 ca03c000 0 0 0 0000204 new [IWAIT] irq10: 19 c3927388 ca03d000 0 0 0 0000204 new [IWAIT] irq9: 18 c2f7a000 c9aec000 0 0 0 0000204 new [IWAIT] irq8: rtc 17 c2f7a1c4 c9b35000 0 0 0 0000204 [IWAIT] irq7: fxp0 16 c2f7a388 c9b36000 0 0 0 0000204 new [IWAIT] irq6: 15 c2f7a54c c9b37000 0 0 0 0000204 new [IWAIT] irq5: 14 c2f7a710 c9b38000 0 0 0 0000204 new [IWAIT] irq4: sio0 13 c2f7a8d4 c9b39000 0 0 0 0000204 new [IWAIT] irq3: sio1 12 c2f7aa98 c9b3a000 0 0 0 0000204 new [IWAIT] irq1: 11 c2f7ac5c c9b3b000 0 0 0 0000204 new [IWAIT] irq0: clk 10 c2f7ae20 c9b3c000 0 0 0 000020c [Can run] idle 1 c2f82000 c9b3d000 0 0 1 0004200 [SLP]wait 0xc2f82000] init 0 c060cd00 c281f000 0 0 0 0000200 [SLP]sched 0xc060cd00] swapper bz@noc:...obj.../HEAD/compile-20031119-1814/sys/ZAB1-2003111901-CF-DEBUG> gdb -k kernel.debug GNU gdb 5.2.1 (FreeBSD) Copyright 2002 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-undermydesk-freebsd"... (kgdb) l *0xc055088c 0xc055088c is in key_cmpspidx_withmask (HEAD/compile-20031119-1814/sys/netkey/key.c:4105). 4100 return 1; 4101 4102 if (spidx0 == NULL || spidx1 == NULL) 4103 return 0; 4104 4105 if (spidx0->src.ss_family != spidx1->src.ss_family || 4106 spidx0->dst.ss_family != spidx1->dst.ss_family || 4107 spidx0->src.ss_len != spidx1->src.ss_len || 4108 spidx0->dst.ss_len != spidx1->dst.ss_len) 4109 return 0; -- Bjoern A. Zeeb bzeeb at Zabbadoz dot NeT 56 69 73 69 74 http://www.zabbadoz.net/