From owner-freebsd-net@FreeBSD.ORG Thu Sep 23 15:05:51 2010 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 033281065673; Thu, 23 Sep 2010 15:05:51 +0000 (UTC) (envelope-from tom@tomjudge.com) Received: from eu1sys200aog101.obsmtp.com (eu1sys200aog101.obsmtp.com [207.126.144.111]) by mx1.freebsd.org (Postfix) with SMTP id 77B448FC26; Thu, 23 Sep 2010 15:05:49 +0000 (UTC) Received: from source ([63.174.175.251]) by eu1sys200aob101.postini.com ([207.126.147.11]) with SMTP ID DSNKTJtsy5LpR7kKMblyMA+3aPg2e2hQsq4Y@postini.com; Thu, 23 Sep 2010 15:05:50 UTC Received: from [172.17.10.53] (unknown [172.17.10.53]) by bbbx3.usdmm.com (Postfix) with ESMTP id 57572FD01F; Thu, 23 Sep 2010 15:05:46 +0000 (UTC) Message-ID: <4C9B6CBD.2030408@tomjudge.com> Date: Thu, 23 Sep 2010 10:05:33 -0500 From: Tom Judge User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.12) Gecko/20100915 Lightning/1.0b1 Thunderbird/3.0.8 MIME-Version: 1.0 To: pyunyh@gmail.com References: <4C894A76.5040200@tomjudge.com> <20100910002439.GO7203@michelle.cdnetworks.com> <4C8E3D79.6090102@tomjudge.com> <20100913184833.GF1229@michelle.cdnetworks.com> <4C8E768E.7000003@tomjudge.com> <20100913193322.GG1229@michelle.cdnetworks.com> <4C8E8BD1.5090007@tomjudge.com> <20100913205348.GJ1229@michelle.cdnetworks.com> In-Reply-To: <20100913205348.GJ1229@michelle.cdnetworks.com> X-Enigmail-Version: 1.0.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org, davidch@broadcom.com, yongari@freebsd.org Subject: Re: bce(4) - com_no_buffers (Again) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Sep 2010 15:05:51 -0000 On 09/13/2010 03:53 PM, Pyun YongHyeon wrote: > On Mon, Sep 13, 2010 at 03:38:41PM -0500, Tom Judge wrote: > >> On 09/13/2010 02:33 PM, Pyun YongHyeon wrote: >> >>> On Mon, Sep 13, 2010 at 02:07:58PM -0500, Tom Judge wrote: >>> >>> >>>>>> Without BCE_JUMBO_HDRSPLIT then we see no errors. With it we see number >>>>>> of errors, however the rate seems to be reduced compaired to the >>>>>> previous version of the driver. >>>>>> >>>>> It seems there are issues in header splitting and it was disabled >>>>> by default. Header splitting reduces packet processing overhead in >>>>> upper layer so it's normal to see better performance with header >>>>> splitting. >>>>> >>>> The reason that we have had header splitting enabled in the past is that >>>> historically there have been issues with memory fragmentation when using >>>> 8k jumbo frames (resulting in 9k mbuf's). >>>> >>> Yes, if you use jumbo frames, header splitting would help to reduce >>> memory fragmentation as header splitting wouldn't allocate jumbo >>> clusters. >>> >>> >> Under testing I have yet to see a memory fragmentation issue with this >> driver. I follow up if/when I find a problem with this again. >> >> So here we are again. The system is locking up again because of 9k mbuf allocation failures. tj@pidge '14:12:25' '~' > $ netstat -m 514/4781/5295 mbufs in use (current/cache/total) 0/2708/2708/25600 mbuf clusters in use (current/cache/total/max) 0/1750 mbuf+clusters out of packet secondary zone in use (current/cache) 0/2904/2904/12800 4k (page size) jumbo clusters in use (current/cache/total/max) 513/3274/3787/6400 9k jumbo clusters in use (current/cache/total/max) 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) 4745K/47693K/52438K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/2692655/0 requests for jumbo clusters denied (4k/9k/16k) 0/0/0 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 0 calls to protocol drain routines >>>> I have a kernel with the following configuration in testing right now: >>>> >>>> * Flow control enabled. >>>> * Jumbo header splitting turned off. >>>> >>>> >>>> Is there any way that we can fix flow control with jumbo header >>>> splitting turned on? >>>> >>>> >>>> >>> Flow control has nothing to do with header splitting(i.e. flow >>> control is always enabled). >>> >>> >>> >> Sorry let me rephrase that: >> >> Is there a way to fix the RX buffer shortage issues (when header >> splitting is turned on) so that they are guarded by flow control. Maybe >> change the low watermark for flow control when its enabled? >> >> > I'm not sure how much it would help but try changing RX low > watermark. Default value is 32 which seems to be reasonable value. > But it's only for 5709/5716 controllers and Linux seems to use > different default value. > These are: NetXtreme II BCM5709 Gigabit Ethernet So my next task is to turn the watermark related defines into sysctls and turn on header splitting so that I can try to tune them without having to reboot. My next question is, is it possible to increase the size of the RX ring without switching to RSS? Tom -- TJU13-ARIN