From owner-freebsd-net@FreeBSD.ORG Mon Sep 13 20:54:40 2010 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6EA2A106564A; Mon, 13 Sep 2010 20:54:40 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from mail-pw0-f54.google.com (mail-pw0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id 33C608FC13; Mon, 13 Sep 2010 20:54:39 +0000 (UTC) Received: by pwi8 with SMTP id 8so2691867pwi.13 for ; Mon, 13 Sep 2010 13:54:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:received:from:date:to:cc :subject:message-id:reply-to:references:mime-version:content-type :content-disposition:in-reply-to:user-agent; bh=6hgzRo4KDQaADKUtPXsEtPxhCTCHc+Zpi2Buf34RVfk=; b=sAOjpbfvmIRiuMpTwLeISbASX7tGKg4GUfsBnMi92RfSyaKUE7sY3OnR/Wn6giabvj wTPuAcxvrh5BZ09/YpCW4GzJCtkTOOtkU4vzX1J7qM9bBO+vBD0+ekKb6VkTHM1SvXO3 zwHKgAmvvl4HZMFRQ/c2XX1aAQUvDOGKJZXho= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:date:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; b=a9NItv23efku/Wt5XwQmk2E6arItsVsgYqPKOJea4n5IT57gI29A9i+rmSKwA0fUZp HXrTjxPCEWm83nQAyIhtwERsbqzFqV2r7ultr99gk/R3HN3zS7MKBqtMvOtTyHgxlRZ5 lyIdu5sKOvIQCXaqLd/2bkIoCxlokJ2l7lzPc= Received: by 10.142.1.37 with SMTP id 37mr205015wfa.242.1284411279643; Mon, 13 Sep 2010 13:54:39 -0700 (PDT) Received: from pyunyh@gmail.com ([174.35.1.224]) by mx.google.com with ESMTPS id n35sm7147458wfa.3.2010.09.13.13.54.36 (version=TLSv1/SSLv3 cipher=RC4-MD5); Mon, 13 Sep 2010 13:54:37 -0700 (PDT) Received: by pyunyh@gmail.com (sSMTP sendmail emulation); Mon, 13 Sep 2010 13:53:48 -0700 From: Pyun YongHyeon Date: Mon, 13 Sep 2010 13:53:48 -0700 To: Tom Judge Message-ID: <20100913205348.GJ1229@michelle.cdnetworks.com> References: <4C894A76.5040200@tomjudge.com> <20100910002439.GO7203@michelle.cdnetworks.com> <4C8E3D79.6090102@tomjudge.com> <20100913184833.GF1229@michelle.cdnetworks.com> <4C8E768E.7000003@tomjudge.com> <20100913193322.GG1229@michelle.cdnetworks.com> <4C8E8BD1.5090007@tomjudge.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4C8E8BD1.5090007@tomjudge.com> User-Agent: Mutt/1.4.2.3i Cc: freebsd-net@freebsd.org, davidch@broadcom.com, yongari@freebsd.org Subject: Re: bce(4) - com_no_buffers (Again) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Sep 2010 20:54:40 -0000 On Mon, Sep 13, 2010 at 03:38:41PM -0500, Tom Judge wrote: > On 09/13/2010 02:33 PM, Pyun YongHyeon wrote: > > On Mon, Sep 13, 2010 at 02:07:58PM -0500, Tom Judge wrote: > > > >> On 09/13/2010 01:48 PM, Pyun YongHyeon wrote: > >> > >>> On Mon, Sep 13, 2010 at 10:04:25AM -0500, Tom Judge wrote: > >>> > >>> > >>>> > >> > >> > >>>> Does this mean that these cards are going to perform badly? This is was > >>>> what I gathered from the previous thread. > >>>> > >>>> > >>>> > >>> I mean there are still many rooms to be done in driver for better > >>> performance. bce(4) controllers are one of best controllers for > >>> servers and driver didn't take full advantage of it. > >>> > >>> > >>> > >> So far our experiences with bce(4) on FreeBSD have been very > >> disappointing. Starting with when Dell switched to bce(4) based NIC's > >> (around the time 6.2 was released and with the introduction of the Power > >> Edge X9XX hardware) we have always had problems with the driver in every > >> release we have used: 6.2, 7.0 and 7.1. Luckily David has been helpful > >> and helped us fix the issues. > >> > >> > >> > >>> > >>> > >>>> Without BCE_JUMBO_HDRSPLIT then we see no errors. With it we see number > >>>> of errors, however the rate seems to be reduced compaired to the > >>>> previous version of the driver. > >>>> > >>>> > >>>> > >>> It seems there are issues in header splitting and it was disabled > >>> by default. Header splitting reduces packet processing overhead in > >>> upper layer so it's normal to see better performance with header > >>> splitting. > >>> > >>> > >> The reason that we have had header splitting enabled in the past is that > >> historically there have been issues with memory fragmentation when using > >> 8k jumbo frames (resulting in 9k mbuf's). > >> > >> > > Yes, if you use jumbo frames, header splitting would help to reduce > > memory fragmentation as header splitting wouldn't allocate jumbo > > clusters. > > > > > > Under testing I have yet to see a memory fragmentation issue with this > driver. I follow up if/when I find a problem with this again. > > >> I have a kernel with the following configuration in testing right now: > >> > >> * Flow control enabled. > >> * Jumbo header splitting turned off. > >> > >> > >> Is there any way that we can fix flow control with jumbo header > >> splitting turned on? > >> > >> > > Flow control has nothing to do with header splitting(i.e. flow > > control is always enabled). > > > > > Sorry let me rephrase that: > > Is there a way to fix the RX buffer shortage issues (when header > splitting is turned on) so that they are guarded by flow control. Maybe > change the low watermark for flow control when its enabled? > I'm not sure how much it would help but try changing RX low watermark. Default value is 32 which seems to be reasonable value. But it's only for 5709/5716 controllers and Linux seems to use different default value.