From owner-freebsd-current@FreeBSD.ORG Mon Nov 7 16:46:04 2011 Return-Path: Delivered-To: current@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4C9E0106566B; Mon, 7 Nov 2011 16:46:04 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: from mail.sippysoft.com (mail.sippysoft.com [4.59.13.245]) by mx1.freebsd.org (Postfix) with ESMTP id A7F9F8FC0A; Mon, 7 Nov 2011 16:46:01 +0000 (UTC) Received: from s0106005004e13421.vs.shawcable.net ([70.71.175.212] helo=[192.168.1.79]) by mail.sippysoft.com with esmtpsa (TLSv1:CAMELLIA256-SHA:256) (Exim 4.72 (FreeBSD)) (envelope-from ) id 1RNRu4-000GNl-TG; Mon, 07 Nov 2011 08:18:34 -0800 Message-ID: <4EB804D2.2090101@FreeBSD.org> Date: Mon, 07 Nov 2011 08:18:26 -0800 From: Maxim Sobolev Organization: Sippy Software, Inc. User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:7.0.1) Gecko/20110929 Thunderbird/7.0.1 MIME-Version: 1.0 To: "Bjoern A. Zeeb" , Robert Watson , "current@freebsd.org" , freebsd-net@FreeBSD.ORG Content-Type: multipart/mixed; boundary="------------010709080200050001090900" Sender: sobomax@sippysoft.com X-ssp-trusted: yes X-Mailman-Approved-At: Mon, 07 Nov 2011 17:09:10 +0000 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Subject: Panic in the udp_input() under heavy load X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Nov 2011 16:46:04 -0000 This is a multi-part message in MIME format. --------------010709080200050001090900 Content-Type: text/plain; charset=KOI8-U; format=flowed Content-Transfer-Encoding: 7bit Hi Gang, We are seeing repeatable panics under high PPS load on our production systems. It happens when the traffic gets into the range or 200MBps and 150-200K PPS. We have been managed to track it down to the following piece of code: (gdb) l *udp_input+0x5d2 0xffffffff806f6202 is in udp_input (/usr/src/sys/netinet/udp_usrreq.c:628). 623 if (inp->inp_ip_minttl && inp->inp_ip_minttl > ip->ip_ttl) { 624 INP_RUNLOCK(inp); 625 goto badunlocked; 626 } 627 up = intoudpcb(inp); 628 if (up->u_tun_func == NULL) { 629 udp_append(inp, ip, m, iphlen + sizeof(struct udphdr), &udp_in); 630 } else { 631 /* 632 * Engage the tunneling protocol. The faulty line appears to be 628, with up value is being NULL, attempt to deference it causes NULL pointer exception. I believe this particular piece of code has been introduced here: --- Author: bz Date: Thu Aug 13 15:16:30 2009 New Revision: 196192 URL: http://svn.freebsd.org/changeset/base/196192 Log: MFC: r192649 Implement UDP control block support. Add udpcb support with own fields and flags for UDP instead of further sticking things into in_pcb and flags fields. Attach the udpcb to the inp_ppcb in the kernel. Note: the udp tunneling parts are not (yet) existing in 7 and thus were not merged. Reviewed by: rwatson --- The screenshot of the panic message is attached. This is pretty recent 8.2-STABLE. Any help is greatly appreciated. This particular bug has haunted us for at least 4-5 months now. Thanks! -Maxim --------------010709080200050001090900--