From owner-freebsd-current@FreeBSD.ORG Wed Mar 10 23:02:31 2010 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D1E5F1065670 for ; Wed, 10 Mar 2010 23:02:31 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from fg-out-1718.google.com (fg-out-1718.google.com [72.14.220.159]) by mx1.freebsd.org (Postfix) with ESMTP id 4E8B78FC0C for ; Wed, 10 Mar 2010 23:02:30 +0000 (UTC) Received: by fg-out-1718.google.com with SMTP id 19so19038fgg.13 for ; Wed, 10 Mar 2010 15:02:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:received:from:date:to:cc :subject:message-id:reply-to:references:mime-version:content-type :content-disposition:in-reply-to:user-agent; bh=0413ZGRKWStMQxA26bu3JMKHFactAC5apKbmng7RpKg=; b=xPc2m4wvzgWvf274nIAqpl5lH/oBTACMvXGVGyaNMSWUfltr3QVMX8/b9Hd+1QtKND lUGYc6Fj3KDMcIrqAvV6MexO4W1Do0wud+z85kD1ChbRWm9ry21NtGCbrHhLDRZ3CCek vQSChoRkjK9YA+rl6GdiP3gxTNK30XKPTMbOU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:date:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; b=D5pAR0jJa3iPv1YHNeTuR7gJxFKaYyxNVwlXZ2fa3dLiR9qJcO3fZmUpZTN0XWABxN 5DMTIR4TNJRdQBm+msVR44fC+aJUdFct8z4CC+v4nrUk12/zrvSY+8Sx18vPCJX9PzAA Z5s54cq1wRREeURNAlZmINag86ePYdKI2cLwo= Received: by 10.87.73.13 with SMTP id a13mr4006052fgl.44.1268262146525; Wed, 10 Mar 2010 15:02:26 -0800 (PST) Received: from pyunyh@gmail.com ([174.35.1.224]) by mx.google.com with ESMTPS id d4sm1579440fga.6.2010.03.10.15.02.23 (version=TLSv1/SSLv3 cipher=RC4-MD5); Wed, 10 Mar 2010 15:02:25 -0800 (PST) Received: by pyunyh@gmail.com (sSMTP sendmail emulation); Wed, 10 Mar 2010 15:02:20 -0800 From: Pyun YongHyeon Date: Wed, 10 Mar 2010 15:02:20 -0800 To: David Christensen Message-ID: <20100310230220.GI10657@michelle.cdnetworks.com> References: <20100309212139.GO1311@michelle.cdnetworks.com> <5D267A3F22FD854F8F48B3D2B52381933AF90EED69@IRVEXCHCCR01.corp.ad.broadcom.com> <20100309214012.GQ1311@michelle.cdnetworks.com> <5D267A3F22FD854F8F48B3D2B52381933AF90EF16F@IRVEXCHCCR01.corp.ad.broadcom.com> <20100310195206.GB10657@michelle.cdnetworks.com> <5D267A3F22FD854F8F48B3D2B52381933AF90EF25A@IRVEXCHCCR01.corp.ad.broadcom.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <5D267A3F22FD854F8F48B3D2B52381933AF90EF25A@IRVEXCHCCR01.corp.ad.broadcom.com> User-Agent: Mutt/1.4.2.3i Cc: Ian FREISLICH , "current@freebsd.org" Subject: Re: dev.bce.X.com_no_buffers increasing and packet loss X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Mar 2010 23:02:32 -0000 On Wed, Mar 10, 2010 at 02:45:47PM -0800, David Christensen wrote: > > > I successfully reproduced the issue with netperf on BCM5709. > > You can use UDP frame size 1 to trigger the issue. > > > > > Changing the high level design of bce_rx_intr() and > > > bce_rx_fill_chain() slightly to post a new buffer as each frame is > > > passed to the OS would likely avoid these gaps during > > bursts of small > > > frames but I'm not sure whether they'll have a negative > > impact on the > > > more common case of streams of MTU sized frames. I've > > considered this > > > in the past but never coded the change and tested the resulting > > > performance. > > > > > > > I guess this may slightly increase performance with additional > > bus_dma(9) overheads but I think one of reason of dropping > > frames under heavy UDP frames may come from lack of free RX > > descriptors. > > Because bce(4) just uses a single RX ring so the number of > > available RX buffers would be 512. However it seems it's not > > possible to increase the number of RX buffers per RX ring so > > the next possible approach would be switching to use multiple > > RX rings with RSS. Even though FreeBSD does not dynamically > > adjust loads among CPUs I guess using RSS would be the way to go. > > The bce(4) hardware supports a linked list of pages for RX > buffer descriptors. The stock build supports 2 pages (RX_PAGES) > with a total of 511 BD's per page. The hardware can support a > maximum of 64K BD's but that would be an unnecessarily large > amount of mbufs for an infrequent problem. > Thanks for the info. I guess 2048 or 4096 BDs would be necessary to get satisfactory Rx performance. I'll have to experiment this. > The middle road would probably involve changing RX_PAGES from a > #define to a sysctl variable to allow tuning for specific > environments along with a change in bce_rx_intr() to fill the > ring after all frames have been processed or when more than > 256 BDs have been consumed, whichever comes first. > > RSS would be great as well though it wouldn't make a dent in > this case since RSS is only supported for TCP, not UDP. > Even though UDP is not supported in RSS, RSS can handle IP. This wouldn't distribute UDP load coming from a single host but if source IP address is different it may help, I guess. > Dave