From owner-freebsd-net@FreeBSD.ORG Thu Sep 23 18:40:46 2010 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E96151065696; Thu, 23 Sep 2010 18:40:46 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from mail-pw0-f54.google.com (mail-pw0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id ADC648FC1D; Thu, 23 Sep 2010 18:40:46 +0000 (UTC) Received: by pwi8 with SMTP id 8so761123pwi.13 for ; Thu, 23 Sep 2010 11:40:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:received:from:date:to:cc :subject:message-id:reply-to:references:mime-version:content-type :content-disposition:in-reply-to:user-agent; bh=nEe/5SBprdXgoZKUWZK/kP3rG8YEwn5n9vwlchlR5iQ=; b=TBxMK5JwfaTJLDzZtRzRAqJprxAbKKcfZdKWrEVPg3XOFastBIT0b12vucql4s8pfr eqijdxFYM3fdIS1/sw5TzumLfSAuMWz3J1tlIZYEkQj8F2WNxoq6zJEkgemQwuIRDvuV TrgENRYV75E3rNKnvByHJC7h6s6hcTaPKuAG0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:date:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; b=dDX39YtJf4n2vvaiqRqcScEN3j2YAjXnwRCS2zV5Snmx+ljHpsoUdNp4QwrW/aTJpZ lCF9g6vwbSphsifyl47+b5F/a7OHVYUcb4TmtXlv87QPQOzK9UWX5HEroTfUiL1e8X4Y XH5qGQIhdDNm3s2i8AW27gq7gfe2mwE0u1P9A= Received: by 10.114.92.16 with SMTP id p16mr2304736wab.210.1285267246151; Thu, 23 Sep 2010 11:40:46 -0700 (PDT) Received: from pyunyh@gmail.com ([174.35.1.224]) by mx.google.com with ESMTPS id q6sm1881932waj.10.2010.09.23.11.40.44 (version=TLSv1/SSLv3 cipher=RC4-MD5); Thu, 23 Sep 2010 11:40:44 -0700 (PDT) Received: by pyunyh@gmail.com (sSMTP sendmail emulation); Thu, 23 Sep 2010 11:39:54 -0700 From: Pyun YongHyeon Date: Thu, 23 Sep 2010 11:39:54 -0700 To: Tom Judge Message-ID: <20100923183954.GC15014@michelle.cdnetworks.com> References: <4C894A76.5040200@tomjudge.com> <20100910002439.GO7203@michelle.cdnetworks.com> <4C8E3D79.6090102@tomjudge.com> <20100913184833.GF1229@michelle.cdnetworks.com> <4C8E768E.7000003@tomjudge.com> <20100913193322.GG1229@michelle.cdnetworks.com> <4C8E8BD1.5090007@tomjudge.com> <20100913205348.GJ1229@michelle.cdnetworks.com> <4C9B6CBD.2030408@tomjudge.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4C9B6CBD.2030408@tomjudge.com> User-Agent: Mutt/1.4.2.3i Cc: freebsd-net@freebsd.org, davidch@broadcom.com, yongari@freebsd.org Subject: Re: bce(4) - com_no_buffers (Again) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Sep 2010 18:40:47 -0000 On Thu, Sep 23, 2010 at 10:05:33AM -0500, Tom Judge wrote: > On 09/13/2010 03:53 PM, Pyun YongHyeon wrote: > > On Mon, Sep 13, 2010 at 03:38:41PM -0500, Tom Judge wrote: > > > >> On 09/13/2010 02:33 PM, Pyun YongHyeon wrote: > >> > >>> On Mon, Sep 13, 2010 at 02:07:58PM -0500, Tom Judge wrote: > >>> > >>> > >>>>>> Without BCE_JUMBO_HDRSPLIT then we see no errors. With it we see number > >>>>>> of errors, however the rate seems to be reduced compaired to the > >>>>>> previous version of the driver. > >>>>>> > >>>>> It seems there are issues in header splitting and it was disabled > >>>>> by default. Header splitting reduces packet processing overhead in > >>>>> upper layer so it's normal to see better performance with header > >>>>> splitting. > >>>>> > >>>> The reason that we have had header splitting enabled in the past is that > >>>> historically there have been issues with memory fragmentation when using > >>>> 8k jumbo frames (resulting in 9k mbuf's). > >>>> > >>> Yes, if you use jumbo frames, header splitting would help to reduce > >>> memory fragmentation as header splitting wouldn't allocate jumbo > >>> clusters. > >>> > >>> > >> Under testing I have yet to see a memory fragmentation issue with this > >> driver. I follow up if/when I find a problem with this again. > >> > >> > So here we are again. The system is locking up again because of 9k mbuf > allocation failures. > > tj@pidge '14:12:25' '~' > > $ netstat -m > 514/4781/5295 mbufs in use (current/cache/total) > 0/2708/2708/25600 mbuf clusters in use (current/cache/total/max) > 0/1750 mbuf+clusters out of packet secondary zone in use (current/cache) > 0/2904/2904/12800 4k (page size) jumbo clusters in use > (current/cache/total/max) > 513/3274/3787/6400 9k jumbo clusters in use (current/cache/total/max) Number of 9k clusters didn't reach to the limit. > 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) > 4745K/47693K/52438K bytes allocated to network (current/cache/total) > 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > 0/2692655/0 requests for jumbo clusters denied (4k/9k/16k) I see large denied value for 9k jumbo clusters but it could be normal under hight network load. But it should not lock up the controller. Note, under these conditions(cluster allocation failure) driver would drop incoming frames which in turn will does not pass received frames to upper stack. The end result could be shown as locked up as upper stack does not receive frames. I think you can check MAC statistics whether the driver is still running or not. > 0/0/0 sfbufs in use (current/peak/max) > 0 requests for sfbufs denied > 0 requests for sfbufs delayed > 0 requests for I/O initiated by sendfile > 0 calls to protocol drain routines > > > >>>> I have a kernel with the following configuration in testing right now: > >>>> > >>>> * Flow control enabled. > >>>> * Jumbo header splitting turned off. > >>>> > >>>> > >>>> Is there any way that we can fix flow control with jumbo header > >>>> splitting turned on? > >>>> > >>>> > >>>> > >>> Flow control has nothing to do with header splitting(i.e. flow > >>> control is always enabled). > >>> > >>> > >>> > >> Sorry let me rephrase that: > >> > >> Is there a way to fix the RX buffer shortage issues (when header > >> splitting is turned on) so that they are guarded by flow control. Maybe > >> change the low watermark for flow control when its enabled? > >> > >> > > I'm not sure how much it would help but try changing RX low > > watermark. Default value is 32 which seems to be reasonable value. > > But it's only for 5709/5716 controllers and Linux seems to use > > different default value. > > > These are: NetXtreme II BCM5709 Gigabit Ethernet > > So my next task is to turn the watermark related defines into sysctls > and turn on header splitting so that I can try to tune them without > having to reboot. > > > > My next question is, is it possible to increase the size of the RX ring > without switching to RSS? > Yes but I doubt it would help in this case as you seem to suffer from 9K jumbo frame allocation failure.