Date: Wed, 10 Mar 2010 15:02:20 -0800 From: Pyun YongHyeon <pyunyh@gmail.com> To: David Christensen <davidch@broadcom.com> Cc: Ian FREISLICH <ianf@clue.co.za>, "current@freebsd.org" <current@freebsd.org> Subject: Re: dev.bce.X.com_no_buffers increasing and packet loss Message-ID: <20100310230220.GI10657@michelle.cdnetworks.com> In-Reply-To: <5D267A3F22FD854F8F48B3D2B52381933AF90EF25A@IRVEXCHCCR01.corp.ad.broadcom.com> References: <E1Nnc4d-0003mB-6e@clue.co.za> <E1Nne0Q-0003uZ-OR@clue.co.za> <E1Noulp-0007Rc-Ro@clue.co.za> <20100309212139.GO1311@michelle.cdnetworks.com> <5D267A3F22FD854F8F48B3D2B52381933AF90EED69@IRVEXCHCCR01.corp.ad.broadcom.com> <20100309214012.GQ1311@michelle.cdnetworks.com> <E1NpHr0-000HWT-It@clue.co.za> <5D267A3F22FD854F8F48B3D2B52381933AF90EF16F@IRVEXCHCCR01.corp.ad.broadcom.com> <20100310195206.GB10657@michelle.cdnetworks.com> <5D267A3F22FD854F8F48B3D2B52381933AF90EF25A@IRVEXCHCCR01.corp.ad.broadcom.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Mar 10, 2010 at 02:45:47PM -0800, David Christensen wrote: > > > I successfully reproduced the issue with netperf on BCM5709. > > You can use UDP frame size 1 to trigger the issue. > > > > > Changing the high level design of bce_rx_intr() and > > > bce_rx_fill_chain() slightly to post a new buffer as each frame is > > > passed to the OS would likely avoid these gaps during > > bursts of small > > > frames but I'm not sure whether they'll have a negative > > impact on the > > > more common case of streams of MTU sized frames. I've > > considered this > > > in the past but never coded the change and tested the resulting > > > performance. > > > > > > > I guess this may slightly increase performance with additional > > bus_dma(9) overheads but I think one of reason of dropping > > frames under heavy UDP frames may come from lack of free RX > > descriptors. > > Because bce(4) just uses a single RX ring so the number of > > available RX buffers would be 512. However it seems it's not > > possible to increase the number of RX buffers per RX ring so > > the next possible approach would be switching to use multiple > > RX rings with RSS. Even though FreeBSD does not dynamically > > adjust loads among CPUs I guess using RSS would be the way to go. > > The bce(4) hardware supports a linked list of pages for RX > buffer descriptors. The stock build supports 2 pages (RX_PAGES) > with a total of 511 BD's per page. The hardware can support a > maximum of 64K BD's but that would be an unnecessarily large > amount of mbufs for an infrequent problem. > Thanks for the info. I guess 2048 or 4096 BDs would be necessary to get satisfactory Rx performance. I'll have to experiment this. > The middle road would probably involve changing RX_PAGES from a > #define to a sysctl variable to allow tuning for specific > environments along with a change in bce_rx_intr() to fill the > ring after all frames have been processed or when more than > 256 BDs have been consumed, whichever comes first. > > RSS would be great as well though it wouldn't make a dent in > this case since RSS is only supported for TCP, not UDP. > Even though UDP is not supported in RSS, RSS can handle IP. This wouldn't distribute UDP load coming from a single host but if source IP address is different it may help, I guess. > Dave
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20100310230220.GI10657>