From owner-freebsd-stable@FreeBSD.ORG Wed Apr 26 19:40:33 2006 Return-Path: X-Original-To: stable@freebsd.org Delivered-To: freebsd-stable@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id CFB4416A473 for ; Wed, 26 Apr 2006 19:40:33 +0000 (UTC) (envelope-from Stephen.Clark@seclark.us) Received: from smtpout07-04.prod.mesa1.secureserver.net (smtpout07-01.prod.mesa1.secureserver.net [64.202.165.230]) by mx1.FreeBSD.org (Postfix) with SMTP id 23AAA43D49 for ; Wed, 26 Apr 2006 19:40:33 +0000 (GMT) (envelope-from Stephen.Clark@seclark.us) Received: (qmail 26098 invoked from network); 26 Apr 2006 19:40:32 -0000 Received: from unknown (24.144.77.138) by smtpout07-04.prod.mesa1.secureserver.net (64.202.165.233) with ESMTP; 26 Apr 2006 19:40:32 -0000 Message-ID: <444FCCAF.8000006@seclark.us> Date: Wed, 26 Apr 2006 15:40:31 -0400 From: Stephen Clark User-Agent: Mozilla/5.0 (X11; U; Linux 2.2.16-22smp i686; en-US; m18) Gecko/20010110 Netscape6/6.5 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Robert Watson References: <4444EE93.9050003@seclark.us> <44459286.1000008@seclark.us> <20060424141346.O44099@fledge.watson.org> <444F75FE.3010101@seclark.us> <20060426180246.O16859@fledge.watson.org> In-Reply-To: <20060426180246.O16859@fledge.watson.org> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Cc: stable@freebsd.org Subject: Re: FreeBSD 4.9 losing mbufs!!! X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Stephen.Clark@seclark.us List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Apr 2006 19:40:34 -0000 Robert Watson wrote: >On Wed, 26 Apr 2006, Stephen Clark wrote: > > > >>>Sorry not to have caught this thread earlier; I've been on travel for the >>>last few weeks. My general suggestion would be to try to narrow the code >>>paths traversed to try to eliminate as much code as possible from the >>>search. It sounds like you've done that pretty effectively :-). >>> >>>Typically, memory leaks occur in edge error cases, where the memory is not >>>properly released, or ownership is unclear. My suggestion would be to add >>>counters (or look at existing counters where already present) and see if >>>there's an error case being triggered in about the same quantity that mbuf >>>leakage is occuring. Chances are, there's an error being returned and a >>>missing m_freem(). >>> >>>Based on your comments above, I might also pay attention to the routing >>>socket path -- the rate of leak could correspond to the routing daemons >>>talking to the network stack, rather than the rate of traffic. For >>>example, it could be that one of the routing messages is handled improperly >>>resulting in a leak. >>> >>>Unfortunately, tracking down memory leaks can be quite difficult, and tends >>>to require a combination of dogged persistence and luck... >>> >>> >>Good news and bad news. >> >>I managed to get enough of our system running on 6.x stable to test and it >>does not appear to lose mbufs. Bad news my ipsec transfer rate dropped from >>54mbits/sec to 39mbits/sec. We need to be able to handle a t3 (45mbits/sec). >>Any ideas as to why this drop off in 6.x? >> >> > >Well, that's good about the good news, but not so great about the bad news. >Are you using IPv6? If not, could you look at how FAST_IPSEC performs instead >of the default IPSEC package, if you're not already doing so? The KAME IPSEC >implementation is not multi-processor compatible, so when IPSEC support is >compiled into the kernel, execution of the network stack and related >components is limited to a single CPU (you'll see a warning about this at boot >if this is the case). This is something we hope to fix in a future release, >but in the mean time FAST_IPSEC offers improved performance and SMP >scalability, as well as support for hardware crypto acceleration. The >downside to FAST_IPSEC is that it doesn't currently support IPv6, which is >also something we'd like to fix in the future. > >Aside from the above, there are a number of other things we can look at to >decide what the source of the performance problem is. First, I'd encourage >you to make sure any system debugging features, such as INVARIANTS, >INVARIANT_SUPPORT, and WITNESS, are disabled. Next, I would recommend >generating a high level of your load on the system, and using systat -vmstat 1 >and top -S to grab some information about it in the steady state. I recommend >letting both run for a couple of minutes, then grabbing the output from both >of them. This will give us an idea of where CPU usage is going in the kernel, >which is where I assume most of your workload ends up being handled. > >Thanks, > >Robert N M Watson >_______________________________________________ >freebsd-stable@freebsd.org mailing list >http://lists.freebsd.org/mailman/listinfo/freebsd-stable >To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > > > Thanks Robert, I'll give that a try. Regards, Steve -- "They that give up essential liberty to obtain temporary safety, deserve neither liberty nor safety." (Ben Franklin) "The course of history shows that as a government grows, liberty decreases." (Thomas Jefferson)