Date: Thu, 23 Sep 2010 10:05:33 -0500 From: Tom Judge <tom@tomjudge.com> To: pyunyh@gmail.com Cc: freebsd-net@freebsd.org, davidch@broadcom.com, yongari@freebsd.org Subject: Re: bce(4) - com_no_buffers (Again) Message-ID: <4C9B6CBD.2030408@tomjudge.com> In-Reply-To: <20100913205348.GJ1229@michelle.cdnetworks.com> References: <4C894A76.5040200@tomjudge.com> <20100910002439.GO7203@michelle.cdnetworks.com> <4C8E3D79.6090102@tomjudge.com> <20100913184833.GF1229@michelle.cdnetworks.com> <4C8E768E.7000003@tomjudge.com> <20100913193322.GG1229@michelle.cdnetworks.com> <4C8E8BD1.5090007@tomjudge.com> <20100913205348.GJ1229@michelle.cdnetworks.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 09/13/2010 03:53 PM, Pyun YongHyeon wrote: > On Mon, Sep 13, 2010 at 03:38:41PM -0500, Tom Judge wrote: > >> On 09/13/2010 02:33 PM, Pyun YongHyeon wrote: >> >>> On Mon, Sep 13, 2010 at 02:07:58PM -0500, Tom Judge wrote: >>> >>> >>>>>> Without BCE_JUMBO_HDRSPLIT then we see no errors. With it we see number >>>>>> of errors, however the rate seems to be reduced compaired to the >>>>>> previous version of the driver. >>>>>> >>>>> It seems there are issues in header splitting and it was disabled >>>>> by default. Header splitting reduces packet processing overhead in >>>>> upper layer so it's normal to see better performance with header >>>>> splitting. >>>>> >>>> The reason that we have had header splitting enabled in the past is that >>>> historically there have been issues with memory fragmentation when using >>>> 8k jumbo frames (resulting in 9k mbuf's). >>>> >>> Yes, if you use jumbo frames, header splitting would help to reduce >>> memory fragmentation as header splitting wouldn't allocate jumbo >>> clusters. >>> >>> >> Under testing I have yet to see a memory fragmentation issue with this >> driver. I follow up if/when I find a problem with this again. >> >> So here we are again. The system is locking up again because of 9k mbuf allocation failures. tj@pidge '14:12:25' '~' > $ netstat -m 514/4781/5295 mbufs in use (current/cache/total) 0/2708/2708/25600 mbuf clusters in use (current/cache/total/max) 0/1750 mbuf+clusters out of packet secondary zone in use (current/cache) 0/2904/2904/12800 4k (page size) jumbo clusters in use (current/cache/total/max) 513/3274/3787/6400 9k jumbo clusters in use (current/cache/total/max) 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) 4745K/47693K/52438K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/2692655/0 requests for jumbo clusters denied (4k/9k/16k) 0/0/0 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 0 calls to protocol drain routines >>>> I have a kernel with the following configuration in testing right now: >>>> >>>> * Flow control enabled. >>>> * Jumbo header splitting turned off. >>>> >>>> >>>> Is there any way that we can fix flow control with jumbo header >>>> splitting turned on? >>>> >>>> >>>> >>> Flow control has nothing to do with header splitting(i.e. flow >>> control is always enabled). >>> >>> >>> >> Sorry let me rephrase that: >> >> Is there a way to fix the RX buffer shortage issues (when header >> splitting is turned on) so that they are guarded by flow control. Maybe >> change the low watermark for flow control when its enabled? >> >> > I'm not sure how much it would help but try changing RX low > watermark. Default value is 32 which seems to be reasonable value. > But it's only for 5709/5716 controllers and Linux seems to use > different default value. > These are: NetXtreme II BCM5709 Gigabit Ethernet So my next task is to turn the watermark related defines into sysctls and turn on header splitting so that I can try to tune them without having to reboot. My next question is, is it possible to increase the size of the RX ring without switching to RSS? Tom -- TJU13-ARIN
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4C9B6CBD.2030408>