Date: Mon, 13 Sep 2010 15:38:41 -0500 From: Tom Judge <tom@tomjudge.com> To: pyunyh@gmail.com Cc: freebsd-net@freebsd.org, davidch@broadcom.com, yongari@freebsd.org Subject: Re: bce(4) - com_no_buffers (Again) Message-ID: <4C8E8BD1.5090007@tomjudge.com> In-Reply-To: <20100913193322.GG1229@michelle.cdnetworks.com> References: <4C894A76.5040200@tomjudge.com> <20100910002439.GO7203@michelle.cdnetworks.com> <4C8E3D79.6090102@tomjudge.com> <20100913184833.GF1229@michelle.cdnetworks.com> <4C8E768E.7000003@tomjudge.com> <20100913193322.GG1229@michelle.cdnetworks.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 09/13/2010 02:33 PM, Pyun YongHyeon wrote: > On Mon, Sep 13, 2010 at 02:07:58PM -0500, Tom Judge wrote: > >> On 09/13/2010 01:48 PM, Pyun YongHyeon wrote: >> >>> On Mon, Sep 13, 2010 at 10:04:25AM -0500, Tom Judge wrote: >>> >>> >>>> >> <SNIP/> >> >>>> Does this mean that these cards are going to perform badly? This is was >>>> what I gathered from the previous thread. >>>> >>>> >>>> >>> I mean there are still many rooms to be done in driver for better >>> performance. bce(4) controllers are one of best controllers for >>> servers and driver didn't take full advantage of it. >>> >>> >>> >> So far our experiences with bce(4) on FreeBSD have been very >> disappointing. Starting with when Dell switched to bce(4) based NIC's >> (around the time 6.2 was released and with the introduction of the Power >> Edge X9XX hardware) we have always had problems with the driver in every >> release we have used: 6.2, 7.0 and 7.1. Luckily David has been helpful >> and helped us fix the issues. >> >> <SNIP/> >> >>> >>> >>>> Without BCE_JUMBO_HDRSPLIT then we see no errors. With it we see number >>>> of errors, however the rate seems to be reduced compaired to the >>>> previous version of the driver. >>>> >>>> >>>> >>> It seems there are issues in header splitting and it was disabled >>> by default. Header splitting reduces packet processing overhead in >>> upper layer so it's normal to see better performance with header >>> splitting. >>> >>> >> The reason that we have had header splitting enabled in the past is that >> historically there have been issues with memory fragmentation when using >> 8k jumbo frames (resulting in 9k mbuf's). >> >> > Yes, if you use jumbo frames, header splitting would help to reduce > memory fragmentation as header splitting wouldn't allocate jumbo > clusters. > > Under testing I have yet to see a memory fragmentation issue with this driver. I follow up if/when I find a problem with this again. >> I have a kernel with the following configuration in testing right now: >> >> * Flow control enabled. >> * Jumbo header splitting turned off. >> >> >> Is there any way that we can fix flow control with jumbo header >> splitting turned on? >> >> > Flow control has nothing to do with header splitting(i.e. flow > control is always enabled). > > Sorry let me rephrase that: Is there a way to fix the RX buffer shortage issues (when header splitting is turned on) so that they are guarded by flow control. Maybe change the low watermark for flow control when its enabled? >> Thanks >> >> Tom >> >> PS. The following test was more than enough to trigger buffer shortages >> with header splitting on: >> >> ( while true; do ldapsearch -h ldap-server1 -b "ou=Some,o=Base" dn; done >> ) & >> ( while true; do ldapsearch -h ldap-server1 -b "ou=Some,o=Base" dn; done >> ) & >> ( while true; do ldapsearch -h ldap-server1 -b "ou=Some,o=Base" dn; done >> ) & >> >> The search in question returned about 1700 entries. >> >> > I can trigger this kind of buffer shortage with benchmark tools. > Actually fixing header splitting is on my TODO list as well as > other things but I don't know how long it would take. > Great to here, thanks for all the hard work. Tom -- TJU13-ARIN
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4C8E8BD1.5090007>