From owner-freebsd-bugs@FreeBSD.ORG Mon May 21 07:30:11 2012 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 94D191065672 for ; Mon, 21 May 2012 07:30:11 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 630E18FC12 for ; Mon, 21 May 2012 07:30:11 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q4L7U5s3026030 for ; Mon, 21 May 2012 07:30:05 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q4L7U5ZV026020; Mon, 21 May 2012 07:30:05 GMT (envelope-from gnats) Resent-Date: Mon, 21 May 2012 07:30:05 GMT Resent-Message-Id: <201205210730.q4L7U5ZV026020@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Joerg Pulz Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 8BA0A1065673; Mon, 21 May 2012 07:29:47 +0000 (UTC) (envelope-from Joerg.Pulz@frm2.tum.de) Received: from mailhost.frm2.tum.de (mailhost.frm2.tum.de [129.187.179.12]) by mx1.freebsd.org (Postfix) with ESMTP id BF6078FC14; Mon, 21 May 2012 07:29:46 +0000 (UTC) Received: from mailhost.frm2.tum.de (localhost [127.0.0.1]) by mailhost.frm2.tum.de (8.14.4/8.14.4) with ESMTP id q4L7Oa6q084013; Mon, 21 May 2012 09:26:08 +0200 (CEST) (envelope-from jpulz@frm2.tum.de) Received: from hades.admin.frm2 (hades.admin.frm2 [172.25.1.10]) by mailhost.frm2.tum.de (8.14.4/8.14.4) with ESMTP id q4L7Q6D9084047 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Mon, 21 May 2012 09:26:06 +0200 (CEST) (envelope-from jpulz@frm2.tum.de) Received: from hades.admin.frm2 (localhost [127.0.0.1]) by hades.admin.frm2 (8.14.4/8.14.3) with ESMTP id q4L7Q6ku064259; Mon, 21 May 2012 09:26:06 +0200 (CEST) (envelope-from jpulz@frm2.tum.de) Received: (from jpulz@localhost) by hades.admin.frm2 (8.14.4/8.14.3/Submit) id q4L7Q6m9064258; Mon, 21 May 2012 09:26:06 +0200 (CEST) (envelope-from jpulz) Message-Id: <201205210726.q4L7Q6m9064258@hades.admin.frm2> Date: Mon, 21 May 2012 09:26:06 +0200 (CEST) From: Joerg Pulz To: FreeBSD-gnats-submit@FreeBSD.org X-Send-Pr-Version: 3.113 Cc: freebsd-pf@FreeBSD.org Subject: kern/168190: panic when using pf and route-to (maybe: bad fragment handling?) X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Joerg Pulz List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 May 2012 07:30:11 -0000 >Number: 168190 >Category: kern >Synopsis: panic when using pf and route-to (maybe: bad fragment handling?) >Confidential: no >Severity: critical >Priority: high >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Mon May 21 07:30:05 UTC 2012 >Closed-Date: >Last-Modified: >Originator: Joerg Pulz >Release: FreeBSD 9.0-RELEASE-p1 >Organization: TU Muenchen / FRM II >Environment: System: FreeBSD charon 9.0-RELEASE-p1 FreeBSD 9.0-RELEASE-p1 #3: Sun May 20 08:42:19 CEST 2012 root@charon:/usr/obj/usr/src/sys/IPSEC amd64 >Description: We have a dual home VPN (IPSec) gateway running ipsec-tools. All packets coming from VPN clients have to hit the main router to generate and evaluate states. Therefor we use pf(4) and the route-to feature. Unfortunately the system panics (and it paniced with FreeBSD-8.x too) at irregular intervals with: "panic: m_copym, offset > size of mbuf chain" I'm unable to reproduce the network traffic to force the problem to appear. I recompiled the Kernel and configured the system to keep dumps to find the find the relevant place in the code for this issue. As i'm not that deep in network processing and pf code i can only guess that handling fragmented packets with pf(4) and route-to is buggy somewhere. Below is the kgdb(1) output for the last dump with some more information. If someone can give me some advise i can produce more and detailed output. If necessary i can upload all requiered stuff somewhere. >How-To-Repeat: >Fix: --- kgdb.out begins here --- GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Unread portion of the kernel message buffer: panic: m_copym, offset > size of mbuf chain cpuid = 1 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a kdb_backtrace() at kdb_backtrace+0x37 panic() at panic+0x182 m_copym() at m_copym+0x280 ip_fragment() at ip_fragment+0x1e5 pf_route() at pf_route+0x75c pf_test() at pf_test+0xc29 pf_route() at pf_route+0x30a pf_test() at pf_test+0xc29 pf_check_out() at pf_check_out+0x3a pfil_run_hooks() at pfil_run_hooks+0xd2 ip_output() at ip_output+0x655 ip_forward() at ip_forward+0x175 ip_input() at ip_input+0x5fd swi_net() at swi_net+0x15a intr_event_execute_handlers() at intr_event_execute_handlers+0x66 ithread_loop() at ithread_loop+0xaf fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000241d00, rbp = 0 --- KDB: enter: panic Dumping 649 out of 4077 MB:..3%..13%..23%..33%..42%..52%..62%..72%..82%..92% Reading symbols from /boot/kernel.old/geom_mirror.ko...Reading symbols from /boot/kernel.old/geom_mirror.ko.symbols...done. done. Loaded symbols for /boot/kernel.old/geom_mirror.ko Reading symbols from /boot/kernel.old/ipmi.ko...Reading symbols from /boot/kernel.old/ipmi.ko.symbols...done. done. Loaded symbols for /boot/kernel.old/ipmi.ko #0 doadump (textdump=0) at pcpu.h:224 224 __asm("movq %%gs:0,%0" : "=r" (td)); (kgdb) up 10 #10 0xffffffff806e9079 in m_copym (m=0x0, off0=1500, len=1480, wait=1) at /usr/src/sys/kern/uipc_mbuf.c:541 541 KASSERT(m != NULL, ("m_copym, offset > size of mbuf chain")); (kgdb) list 536 KASSERT(len >= 0, ("m_copym, negative len %d", len)); 537 MBUF_CHECKSLEEP(wait); 538 if (off == 0 && m->m_flags & M_PKTHDR) 539 copyhdr = 1; 540 while (off > 0) { 541 KASSERT(m != NULL, ("m_copym, offset > size of mbuf chain")); 542 if (off < m->m_len) 543 break; 544 off -= m->m_len; 545 m = m->m_next; (kgdb) p off $1 = 1166 (kgdb) up #11 0xffffffff8077fe1f in ip_fragment (ip=0xfffffe0005b7a974, m_frag=0xffffff8000241428, mtu=) at /usr/src/sys/netinet/ip_output.c:816 816 m->m_next = m_copym(m0, off, len, M_DONTWAIT); (kgdb) list 811 len = ip->ip_len - off; 812 m->m_flags |= M_LASTFRAG; 813 } else 814 mhip->ip_off |= IP_MF; 815 mhip->ip_len = htons((u_short)(len + mhlen)); 816 m->m_next = m_copym(m0, off, len, M_DONTWAIT); 817 if (m->m_next == NULL) { /* copy failed */ 818 m_free(m); 819 error = ENOBUFS; /* ??? */ 820 IPSTAT_INC(ips_odropped); (kgdb) p *m0 $2 = {m_hdr = {mh_next = 0xfffffe0052a2e200, mh_nextpkt = 0x0, mh_data = 0xfffffe0005b7a974 "E", mh_len = 60, mh_flags = 66, mh_type = 1, pad = "­ÞÞÀ­Þ"}, M_dat = {MH = {MH_pkthdr = {rcvif = 0xfffffe0003001800, header = 0x0, len = 334, flowid = 0, csum_flags = 1, csum_data = 27136, tso_segsz = 0, PH_vt = {vt_vtag = 0, vt_nrecs = 0}, tags = {slh_first = 0xfffffe00050a90c0}}, MH_dat = {MH_ext = { ext_buf = 0x493a017c0045
, ext_free = 0x493a014e0045, ext_arg1 = 0x4947cb4f287c0437, ext_arg2 = 0x4e01004557b3bb81, ext_size = 3586, ref_cnt = 0xc6be758202079b0a, ext_type = 89195267}, MH_databuf = "E\000|\001:I\000\000E\000N\001:I\000\0007\004|(OËGI\201»³WE\000\001N\002\016\000\000?\001$É\n\233\a\002\202u¾Æ\003\003Q\005\000\000\000\000E\000\0012ûS\000\0000\021;\217\202u¾Æ\n\233\a\002\aÑùÃ\001\036éQKS\000\000\001B\000\000\000\001\v\001ÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­Þ"}}, M_databuf = "\000\030\000\003\000þÿÿ\000\000\000\000\000\000\000\000N\001\000\000\000\000\000\000\001\000\000\000\000j\000\000\000\000\000\000ÞÀ­ÞÀ\220\n\005\000þÿÿE\000|\001:I\000\000E\000N\001:I\000\0007\004|(OËGI\201»³WE\000\001N\002\016\000\000?\001$É\n\233\a\002\202u¾Æ\003\003Q\005\000\000\000\000E\000\0012ûS\000\0000\021;\217\202u¾Æ\n\233\a\002\aÑùÃ\001\036éQKS\000\000\001B\000\000\000\001\v\001ÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­Þ"...}} (kgdb) p off $3 = 1500 (kgdb) p len $4 = 1480 (kgdb) p hlen $5 = 20 (kgdb) up #12 0xffffffff8032842a in pf_route (m=0xffffff8000241658, r=0xfffffe0005dc8af8, dir=) at /usr/src/sys/contrib/pf/net/pf.c:6138 6138 error = ip_fragment(ip, &m0, ifp->if_mtu, ifp->if_hwassist, sw_csum); (kgdb) list 6133 /* 6134 * XXX: is cheaper + less error prone than own function 6135 */ 6136 NTOHS(ip->ip_len); 6137 NTOHS(ip->ip_off); 6138 error = ip_fragment(ip, &m0, ifp->if_mtu, ifp->if_hwassist, sw_csum); 6139 #else 6140 error = ip_fragment(m0, ifp, ifp->if_mtu); 6141 #endif 6142 if (error) { (kgdb) p *ip $6 = {ip_hl = 5 '\005', ip_v = 4 '\004', ip_tos = 0 '\0', ip_len = 19969, ip_id = 3586, ip_off = 0, ip_ttl = 63 '?', ip_p = 1 '\001', ip_sum = 51492, ip_src = {s_addr = 34052874}, ip_dst = {s_addr = 3334370690}} (kgdb) p *m0 $7 = {m_hdr = {mh_next = 0xfffffe0052a2e200, mh_nextpkt = 0x0, mh_data = 0xfffffe0005b7a974 "E", mh_len = 60, mh_flags = 66, mh_type = 1, pad = "­ÞÞÀ­Þ"}, M_dat = {MH = {MH_pkthdr = {rcvif = 0xfffffe0003001800, header = 0x0, len = 334, flowid = 0, csum_flags = 1, csum_data = 27136, tso_segsz = 0, PH_vt = {vt_vtag = 0, vt_nrecs = 0}, tags = {slh_first = 0xfffffe00050a90c0}}, MH_dat = {MH_ext = { ext_buf = 0x493a017c0045
, ext_free = 0x493a014e0045, ext_arg1 = 0x4947cb4f287c0437, ext_arg2 = 0x4e01004557b3bb81, ext_size = 3586, ref_cnt = 0xc6be758202079b0a, ext_type = 89195267}, MH_databuf = "E\000|\001:I\000\000E\000N\001:I\000\0007\004|(OËGI\201»³WE\000\001N\002\016\000\000?\001$É\n\233\a\002\202u¾Æ\003\003Q\005\000\000\000\000E\000\0012ûS\000\0000\021;\217\202u¾Æ\n\233\a\002\aÑùÃ\001\036éQKS\000\000\001B\000\000\000\001\v\001ÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­Þ"}}, M_databuf = "\000\030\000\003\000þÿÿ\000\000\000\000\000\000\000\000N\001\000\000\000\000\000\000\001\000\000\000\000j\000\000\000\000\000\000ÞÀ­ÞÀ\220\n\005\000þÿÿE\000|\001:I\000\000E\000N\001:I\000\0007\004|(OËGI\201»³WE\000\001N\002\016\000\000?\001$É\n\233\a\002\202u¾Æ\003\003Q\005\000\000\000\000E\000\0012ûS\000\0000\021;\217\202u¾Æ\n\233\a\002\aÑùÃ\001\036éQKS\000\000\001B\000\000\000\001\v\001ÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­Þ"...}} (kgdb) p *m1 $8 = {m_hdr = {mh_next = 0xfffffe0052a2e200, mh_nextpkt = 0x0, mh_data = 0xfffffe0005b7a974 "E", mh_len = 60, mh_flags = 66, mh_type = 1, pad = "­ÞÞÀ­Þ"}, M_dat = {MH = {MH_pkthdr = {rcvif = 0xfffffe0003001800, header = 0x0, len = 334, flowid = 0, csum_flags = 1, csum_data = 27136, tso_segsz = 0, PH_vt = {vt_vtag = 0, vt_nrecs = 0}, tags = {slh_first = 0xfffffe00050a90c0}}, MH_dat = {MH_ext = { ext_buf = 0x493a017c0045
, ext_free = 0x493a014e0045, ext_arg1 = 0x4947cb4f287c0437, ext_arg2 = 0x4e01004557b3bb81, ext_size = 3586, ref_cnt = 0xc6be758202079b0a, ext_type = 89195267}, MH_databuf = "E\000|\001:I\000\000E\000N\001:I\000\0007\004|(OËGI\201»³WE\000\001N\002\016\000\000?\001$É\n\233\a\002\202u¾Æ\003\003Q\005\000\000\000\000E\000\0012ûS\000\0000\021;\217\202u¾Æ\n\233\a\002\aÑùÃ\001\036éQKS\000\000\001B\000\000\000\001\v\001ÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­Þ"}}, M_databuf = "\000\030\000\003\000þÿÿ\000\000\000\000\000\000\000\000N\001\000\000\000\000\000\000\001\000\000\000\000j\000\000\000\000\000\000ÞÀ­ÞÀ\220\n\005\000þÿÿE\000|\001:I\000\000E\000N\001:I\000\0007\004|(OËGI\201»³WE\000\001N\002\016\000\000?\001$É\n\233\a\002\202u¾Æ\003\003Q\005\000\000\000\000E\000\0012ûS\000\0000\021;\217\202u¾Æ\n\233\a\002\aÑùÃ\001\036éQKS\000\000\001B\000\000\000\001\v\001ÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­Þ"...}} (kgdb) --- kgdb.out ends here --- >Release-Note: >Audit-Trail: >Unformatted: