From owner-freebsd-current@FreeBSD.ORG Mon Aug 27 15:48:30 2007 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C026716A41A for ; Mon, 27 Aug 2007 15:48:30 +0000 (UTC) (envelope-from pawel.worach@gmail.com) Received: from ug-out-1314.google.com (ug-out-1314.google.com [66.249.92.170]) by mx1.freebsd.org (Postfix) with ESMTP id 1F01A13C45B for ; Mon, 27 Aug 2007 15:48:29 +0000 (UTC) (envelope-from pawel.worach@gmail.com) Received: by ug-out-1314.google.com with SMTP id a2so51146ugf for ; Mon, 27 Aug 2007 08:48:28 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:user-agent:mime-version:to:subject:content-type:content-transfer-encoding; b=AxwdNwvJIs3dXMGHUzOvwXFN8mqTsO0XwFmHawWP35Iq44eUYyFyq1+EUcAs6UUaVpbahvv1bkMYJz8cT5xoYmFcpY8qNKYJKLLu9IoJeP1ozWMhTI+vRU9Q3TLbqtRNMQ8h+0nl3vHTd4wGoHWBX5r4aH9/8tKCa1LIU1PmkF0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:user-agent:mime-version:to:subject:content-type:content-transfer-encoding; b=dEpl11+wSvBCslxSTeXGmNxbFUTEyjLULy5udeyeCtGlVvWViz11IE5RduEzJOsX8ek1W3DRQBDJUc++MV7olXo9INr/yhByBohTrg1YA4gRdT5p14QMyTSlIFuNNrlb7xjcjG8XDpoBwZLLF3kqc8qfJ4Vf8HzovOWRNRAoGrg= Received: by 10.78.138.6 with SMTP id l6mr4049162hud.1188227985020; Mon, 27 Aug 2007 08:19:45 -0700 (PDT) Received: from ibm-se82151.se.ibm.com ( [195.212.29.163]) by mx.google.com with ESMTPS id b35sm5352049ugd.2007.08.27.08.19.42 (version=TLSv1/SSLv3 cipher=RC4-MD5); Mon, 27 Aug 2007 08:19:44 -0700 (PDT) Message-ID: <46D2EB88.7020905@gmail.com> Date: Mon, 27 Aug 2007 17:19:36 +0200 From: Pawel Worach User-Agent: Thunderbird 2.0.0.7pre (X11/20070820) MIME-Version: 1.0 To: current@freebsd.org Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit Cc: Subject: IPSec panics X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2007 15:48:30 -0000 Hi, While testing IPSec I got this panic on two different -CURRENT systems. I think they happened when racoon was updating the SAD. kernel.debug and vmcore is still available if more info needed. FreeBSD 7.0-CURRENT #0: Fri Aug 24 22:31:26 CEST 2007 Script started on Sun Aug 26 02:21:17 2007 kgdb: kvm_nlist(_stopped_cpus): kgdb: kvm_nlist(_stoppcbs): [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd". Unread portion of the kernel message buffer: kernel trap 12 with interrupts disabled Fatal trap 12: page fault while in kernel mode fault virtual address = 0x18 fault code = supervisor read, page not present instruction pointer = 0x20:0xc059ba74 stack pointer = 0x28:0xe40be9f8 frame pointer = 0x28:0xe40bea04 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = resume, IOPL = 0 current process = 32 (ath0 taskq) trap number = 12 panic: page fault KDB: stack backtrace: db_trace_self_wrapper(c07d4c94,e40be8d8,c056b7da,c07d308a,c0849280,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c07d308a,c0849280,c07c639b,e40be8e4,e40be8e4,...) at kdb_backtrace+0x29 panic(c07c639b,c07f1dac,c3bd9a28,1,1,...) at panic+0xaa trap_fatal(c07f1cae,c,0,14,c,...) at trap_fatal+0x353 trap(e40be9b8) at trap+0x10a calltrap() at calltrap+0x6 --- trap 0xc, eip = 0xc059ba74, esp = 0xe40be9f8, ebp = 0xe40bea04 --- turnstile_broadcast(0,0,18,c3fe72a0,e40beac8,...) at turnstile_broadcast+0x34 _mtx_unlock_sleep(c3fe7330,0,0,0,49c6,...) at _mtx_unlock_sleep+0x52 tcp_input(c3e6ae00,14,0,c3ea281a,800,...) at tcp_input+0xe29 ip_input(c3e6ae00,c3e6ae00,800,c3ba5c00,800,...) at ip_input+0x6ff netisr_dispatch(2,c3e6ae00,10,3,0,...) at netisr_dispatch+0x52 ether_demux(c3ba5c00,c3e6ae00,3,0,3,...) at ether_demux+0x1c1 ether_input(c3ba5c00,c3e6ae00,18,c055ca7a,c3fc4000,...) at ether_input+0x34f ieee80211_deliver_data(c3bda22c,c3fc4000,c3e6ae00,18,c05b9a42,...) at ieee80211_deliver_data+0x137 ieee80211_input(c3bda22c,c3e6ae00,c3fc4000,1d,ffffffa2,...) at ieee80211_input+0x10f6 ath_rx_proc(c3bda000,1,c07c96a3,0,0,...) at ath_rx_proc+0x3cd taskqueue_run(c3bb3700,c3bb371c,0,c07c96a3,0,...) at taskqueue_run+0x14f taskqueue_thread_loop(c3bdb65c,e40bed38,0,0,0,...) at taskqueue_thread_loop+0x98 fork_exit(c0599d70,c3bdb65c,e40bed38) at fork_exit+0xa1 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe40bed70, ebp = 0 --- Uptime: 16m47s Physical memory: 1014 MB Dumping 95 MB: 80 64 48 32 16 #0 doadump () at pcpu.h:195 195 pcpu.h: No such file or directory. in pcpu.h (kgdb) bt #0 doadump () at pcpu.h:195 #1 0xc056b5e3 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409 #2 0xc056b81a in panic (fmt=Variable "fmt" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:563 #3 0xc07903b3 in trap_fatal (frame=0xe40be9b8, eva=24) at /usr/src/sys/i386/i386/trap.c:872 #4 0xc0790d5a in trap (frame=0xe40be9b8) at /usr/src/sys/i386/i386/trap.c:277 #5 0xc077f4cb in calltrap () at /usr/src/sys/i386/i386/exception.s:139 #6 0xc059ba74 in turnstile_broadcast (ts=0x0, queue=0) at /usr/src/sys/kern/subr_turnstile.c:834 #7 0xc055f542 in _mtx_unlock_sleep (m=0xc3fe7330, opts=0, file=0x0, line=0) at /usr/src/sys/kern/kern_mutex.c:593 #8 0xc069e9c9 in tcp_input (m=0xc3e6ae00, off0=20) at /usr/src/sys/netinet/tcp_input.c:854 #9 0xc0641c1f in ip_input (m=0xc3e6ae00) at /usr/src/sys/netinet/ip_input.c:663 #10 0xc06043a2 in netisr_dispatch (num=2, m=0xc3e6ae00) at /usr/src/sys/net/netisr.c:185 #11 0xc06030a1 in ether_demux (ifp=0xc3ba5c00, m=0xc3e6ae00) at /usr/src/sys/net/if_ethersubr.c:848 #12 0xc06034cf in ether_input (ifp=0xc3ba5c00, m=0xc3e6ae00) at /usr/src/sys/net/if_ethersubr.c:706 #13 0xc061ba57 in ieee80211_deliver_data (ic=0xc3bda22c, ni=0xc3fc4000, m=0xc3e6ae00) at /usr/src/sys/net80211/ieee80211_input.c:771 ---Type to continue, or q to quit--- #14 0xc0620df6 in ieee80211_input (ic=0xc3bda22c, m=0xc3e6ae00, ni=0xc3fc4000, rssi=29, noise=-94, rstamp=894) at /usr/src/sys/net80211/ieee80211_input.c:518 #15 0xc090fa7d in ?? () #16 0xc3bda22c in ?? () #17 0xc3e6ae00 in ?? () #18 0xc3fc4000 in ?? () #19 0x0000001d in ?? () #20 0xffffffa2 in ?? () #21 0x0000037e in ?? () #22 0xc3be3b98 in ?? () #23 0x014e22a0 in ?? () #24 0xc3bdb9dc in ?? () #25 0xc3bdb6b4 in ?? () #26 0xc3bda22c in ?? () #27 0xc3ba5c00 in ?? () #28 0xc3bde000 in ?? () #29 0xc3be3b98 in ?? () #30 0xc3fc4000 in ?? () #31 0x00000000 in ?? () #32 0xffffffa2 in ?? () #33 0xc0d303a7 in ?? () #34 0x000000de in ?? () ---Type to continue, or q to quit--- #35 0x000000cc in ?? () #36 0xc3bdb9ec in ?? () #37 0xc3bb3700 in ?? () #38 0x00000001 in ?? () #39 0xe40becd0 in ?? () #40 0xc0599c0f in taskqueue_run (queue=0xc3be3b7c) at /usr/src/sys/kern/subr_taskqueue.c:255 Previous frame identical to this frame (corrupt stack?) (kgdb) f 8 #8 0xc069e9c9 in tcp_input (m=0xc3e6ae00, off0=20) at /usr/src/sys/netinet/tcp_input.c:854 854 INP_UNLOCK(inp); (kgdb) list 849 tcp_dropwithreset(m, th, tp, tlen, rstreason); 850 m = NULL; /* mbuf chain got consumed. */ 851 dropunlock: 852 INP_INFO_WLOCK_ASSERT(&tcbinfo); 853 if (inp != NULL) 854 INP_UNLOCK(inp); 855 INP_INFO_WUNLOCK(&tcbinfo); 856 drop: 857 INP_INFO_UNLOCK_ASSERT(&tcbinfo); 858 if (s != NULL) (kgdb) p *inp $1 = {inp_hash = {le_next = 0x0, le_prev = 0xc3bfb654}, inp_list = { le_next = 0xc3fe7bd0, le_prev = 0xc3fe7200}, inp_flow = 0, inp_inc = { inc_flags = 0 '\0', inc_len = 0 '\0', inc_pad = 0, inc_ie = { ie_fport = 5632, ie_lport = 18886, ie_dependfaddr = {ie46_foreign = { ia46_pad32 = {0, 0, 0}, ia46_addr4 = {s_addr = 17475776}}, ie6_foreign = {__u6_addr = { __u6_addr8 = '\0' , "Àš\n\001", __u6_addr16 = { 0, 0, 0, 0, 0, 0, 43200, 266}, __u6_addr32 = {0, 0, 0, 17475776}}}}, ie_dependladdr = {ie46_local = {ia46_pad32 = {0, 0, 0}, ia46_addr4 = {s_addr = 3356141760}}, ie6_local = { __u6_addr = {__u6_addr8 = '\0' , "Àš\nÈ", __u6_addr16 = {0, 0, 0, 0, 0, 0, 43200, 51210}, __u6_addr32 = {0, 0, 0, 3356141760}}}}}}, inp_ppcb = 0xc3fe9e10, inp_pcbinfo = 0xc0851e00, inp_socket = 0xc4b61c60, inp_label = 0x0, inp_flags = 8388672, inp_sp = 0xc44c5110, inp_vflag = 1 '\001', inp_ip_ttl = 64 '@', inp_ip_p = 0 '\0', inp_ip_minttl = 0 '\0', inp_depend4 = {inp4_ip_tos = 16 '\020', inp4_options = 0x0, inp4_moptions = 0x0}, inp_depend6 = {inp6_options = 0x0, inp6_outputopts = 0x0, inp6_moptions = 0x0, inp6_icmp6filt = 0x0, inp6_cksum = 0, inp6_hops = 0}, inp_portlist = {le_next = 0x0, le_prev = 0xc44c52c8}, inp_phd = 0xc44c52c0, inp_gencnt = 52, inp_mtx = { lock_object = {lo_name = 0xc07dedd3 "inp", lo_type = 0xc07e0b4d "tcpinp", lo_flags = 21692416, lo_witness_data = {lod_list = {stqe_next = 0x0}, ---Type to continue, or q to quit--- lod_witness = 0x0}}, mtx_lock = 4, mtx_recurse = 0}} (kgdb) Script done on Sun Aug 26 02:21:46 2007 And the other: Script started on Sun Aug 26 02:23:40 2007 kgdb: kvm_nlist(_stopped_cpus): kgdb: kvm_nlist(_stoppcbs): [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd". Unread portion of the kernel message buffer: kernel trap 12 with interrupts disabled Fatal trap 12: page fault while in kernel mode fault virtual address = 0x18 fault code = supervisor read, page not present instruction pointer = 0x20:0xc059ba74 stack pointer = 0x28:0xd4d86ac8 frame pointer = 0x28:0xd4d86ad4 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = resume, IOPL = 0 current process = 31 (em0 taskq) trap number = 12 panic: page fault KDB: stack backtrace: db_trace_self_wrapper(c07d4c94,d4d869a8,c056b7da,c07d308a,c0849280,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c07d308a,c0849280,c07c639b,d4d869b4,d4d869b4,...) at kdb_backtrace+0x29 panic(c07c639b,c07f1dac,c2a66cd4,1,1,...) at panic+0xaa trap_fatal(c07f1cae,c,14,14,c,...) at trap_fatal+0x353 trap(d4d86a88) at trap+0x10a calltrap() at calltrap+0x6 --- trap 0xc, eip = 0xc059ba74, esp = 0xd4d86ac8, ebp = 0xd4d86ad4 --- turnstile_broadcast(0,0,10,c2d7ba80,d4d86b98,...) at turnstile_broadcast+0x34 _mtx_unlock_sleep(c2d7bb10,0,0,0,1600,...) at _mtx_unlock_sleep+0x52 tcp_input(c2cfa300,14,c2a5c800,1,0,...) at tcp_input+0xe29 ip_input(c2cfa300,c2cfa300,800,c2a5c800,800,...) at ip_input+0x6ff netisr_dispatch(2,c2cfa300,10,3,0,...) at netisr_dispatch+0x52 ether_demux(c2a5c800,c2cfa300,3,0,3,...) at ether_demux+0x1c1 ether_input(c2a5c800,c2cfa300,c0570028,0,c2a62000,...) at ether_input+0x34f em_handle_rxtx(c29d7000,1,c0573862,c2a46b00,c2a46b1c,...) at em_handle_rxtx+0x43e taskqueue_run(c2a46b00,c2a46b1c,c07c96a3,0,d4d86cf4,...) at taskqueue_run+0x14f taskqueue_thread_loop(c29d72ec,d4d86d38,c0549050,c05489b0,c0548990,...) at taskqueue_thread_loop+0x98 fork_exit(c0599d70,c29d72ec,d4d86d38) at fork_exit+0xa1 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xd4d86d70, ebp = 0 --- Uptime: 9h39m42s Physical memory: 502 MB Dumping 46 MB: 31 15 #0 doadump () at pcpu.h:195 195 pcpu.h: No such file or directory. in pcpu.h (kgdb) bt #0 doadump () at pcpu.h:195 #1 0xc056b5e3 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409 #2 0xc056b81a in panic (fmt=Variable "fmt" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:563 #3 0xc07903b3 in trap_fatal (frame=0xd4d86a88, eva=24) at /usr/src/sys/i386/i386/trap.c:872 #4 0xc0790d5a in trap (frame=0xd4d86a88) at /usr/src/sys/i386/i386/trap.c:277 #5 0xc077f4cb in calltrap () at /usr/src/sys/i386/i386/exception.s:139 #6 0xc059ba74 in turnstile_broadcast (ts=0x0, queue=0) at /usr/src/sys/kern/subr_turnstile.c:834 #7 0xc055f542 in _mtx_unlock_sleep (m=0xc2d7bb10, opts=0, file=0x0, line=0) at /usr/src/sys/kern/kern_mutex.c:593 #8 0xc069e9c9 in tcp_input (m=0xc2cfa300, off0=20) at /usr/src/sys/netinet/tcp_input.c:854 #9 0xc0641c1f in ip_input (m=0xc2cfa300) at /usr/src/sys/netinet/ip_input.c:663 #10 0xc06043a2 in netisr_dispatch (num=2, m=0xc2cfa300) at /usr/src/sys/net/netisr.c:185 #11 0xc06030a1 in ether_demux (ifp=0xc2a5c800, m=0xc2cfa300) at /usr/src/sys/net/if_ethersubr.c:848 #12 0xc06034cf in ether_input (ifp=0xc2a5c800, m=0xc2cfa300) at /usr/src/sys/net/if_ethersubr.c:706 #13 0xc04bc25e in em_handle_rxtx (context=0xc29d7000, pending=1) at /usr/src/sys/dev/em/if_em.c:4308 ---Type to continue, or q to quit--- #14 0xc0599c0f in taskqueue_run (queue=0xc2a46b00) at /usr/src/sys/kern/subr_taskqueue.c:255 #15 0xc0599e08 in taskqueue_thread_loop (arg=0xc29d72ec) at /usr/src/sys/kern/subr_taskqueue.c:374 #16 0xc054eae1 in fork_exit (callout=0xc0599d70 , arg=0xc29d72ec, frame=0xd4d86d38) at /usr/src/sys/kern/kern_fork.c:797 #17 0xc077f540 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:205 (kgdb) f 8 #8 0xc069e9c9 in tcp_input (m=0xc2cfa300, off0=20) at /usr/src/sys/netinet/tcp_input.c:854 854 INP_UNLOCK(inp); (kgdb) list 849 tcp_dropwithreset(m, th, tp, tlen, rstreason); 850 m = NULL; /* mbuf chain got consumed. */ 851 dropunlock: 852 INP_INFO_WLOCK_ASSERT(&tcbinfo); 853 if (inp != NULL) 854 INP_UNLOCK(inp); 855 INP_INFO_WUNLOCK(&tcbinfo); 856 drop: 857 INP_INFO_UNLOCK_ASSERT(&tcbinfo); 858 if (s != NULL) (kgdb) p *inp $1 = {inp_hash = {le_next = 0x0, le_prev = 0xc29d58bc}, inp_list = { le_next = 0xc2d7bd20, le_prev = 0xc2d7b9e0}, inp_flow = 0, inp_inc = { inc_flags = 0 '\0', inc_len = 0 '\0', inc_pad = 0, inc_ie = { ie_fport = 62440, ie_lport = 5632, ie_dependfaddr = {ie46_foreign = { ia46_pad32 = {0, 0, 0}, ia46_addr4 = {s_addr = 3356141760}}, ie6_foreign = {__u6_addr = { __u6_addr8 = '\0' , "Àš\nÈ", __u6_addr16 = {0, 0, 0, 0, 0, 0, 43200, 51210}, __u6_addr32 = {0, 0, 0, 3356141760}}}}, ie_dependladdr = {ie46_local = {ia46_pad32 = {0, 0, 0}, ia46_addr4 = {s_addr = 17475776}}, ie6_local = { __u6_addr = {__u6_addr8 = '\0' , "Àš\n\001", __u6_addr16 = {0, 0, 0, 0, 0, 0, 43200, 266}, __u6_addr32 = {0, 0, 0, 17475776}}}}}}, inp_ppcb = 0xc2d7d5a0, inp_pcbinfo = 0xc0851e00, inp_socket = 0xc2d81948, inp_label = 0x0, inp_flags = 8388608, inp_sp = 0xc2aa8ca0, inp_vflag = 1 '\001', inp_ip_ttl = 64 '@', inp_ip_p = 0 '\0', inp_ip_minttl = 0 '\0', inp_depend4 = {inp4_ip_tos = 16 '\020', inp4_options = 0x0, inp4_moptions = 0x0}, inp_depend6 = {inp6_options = 0x0, inp6_outputopts = 0x0, inp6_moptions = 0x0, inp6_icmp6filt = 0x0, inp6_cksum = 0, inp6_hops = 0}, inp_portlist = {le_next = 0xc2d7bd20, le_prev = 0xc2a925f8}, inp_phd = 0xc2a925f0, inp_gencnt = 16, inp_mtx = { lock_object = {lo_name = 0xc07dedd3 "inp", lo_type = 0xc07e0b4d "tcpinp", lo_flags = 21692416, lo_witness_data = {lod_list = {stqe_next = 0x0}, ---Type to continue, or q to quit--- lod_witness = 0x0}}, mtx_lock = 4, mtx_recurse = 0}} (kgdb) Script done on Sun Aug 26 02:24:15 2007 ipsec.conf: flush; spdflush; spdadd 192.168.10.200 192.168.10.1 any -P out ipsec esp/transport//require; spdadd 192.168.10.1 192.168.10.200 any -P in ipsec esp/transport//require; spdadd -6 ::/0 ::/0 icmp6 -P out none; spdadd -6 ::/0 ::/0 icmp6 -P in none; spdadd -6 1ce:c01d:c0ca:c01a:205:4eff:fe4b:7613 1ce:c01d:c0ca:c01a::1 any -P out ipsec esp/transport//require; spdadd -6 1ce:c01d:c0ca:c01a::1 1ce:c01d:c0ca:c01a:205:4eff:fe4b:7613 any -P in ipsec esp/transport//require; add -6 1ce:c01d:c0ca:c01a::1 1ce:c01d:c0ca:c01a:205:4eff:fe4b:7613 esp 0x1001 -m transport -E rijndael-cbc "01234567890123456789012345678901" -A hmac-sha2-256 "01234567890123456789012345678901"; add -6 1ce:c01d:c0ca:c01a:205:4eff:fe4b:7613 1ce:c01d:c0ca:c01a::1 esp 0x1002 -m transport -E rijndael-cbc "01234567890123456789012345678901" -A hmac-sha2-256 "01234567890123456789012345678901"; (IPv6 uses static keying because racoon fails to find the policy for some reason). racoon.conf is a pretty basic rsasig authentication setup. -- Pawel