From owner-freebsd-net@FreeBSD.ORG Thu Sep 23 18:22:49 2010 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D25C91065674; Thu, 23 Sep 2010 18:22:49 +0000 (UTC) (envelope-from davidch@broadcom.com) Received: from mms2.broadcom.com (mms2.broadcom.com [216.31.210.18]) by mx1.freebsd.org (Postfix) with ESMTP id A00778FC0A; Thu, 23 Sep 2010 18:22:49 +0000 (UTC) Received: from [10.9.200.131] by mms2.broadcom.com with ESMTP (Broadcom SMTP Relay (Email Firewall v6.3.2)); Thu, 23 Sep 2010 11:21:41 -0700 X-Server-Uuid: D3C04415-6FA8-4F2C-93C1-920E106A2031 Received: from IRVEXCHCCR01.corp.ad.broadcom.com ([10.252.49.30]) by IRVEXCHHUB01.corp.ad.broadcom.com ([10.9.200.131]) with mapi; Thu, 23 Sep 2010 11:21:36 -0700 From: "David Christensen" To: "Tom Judge" , "pyunyh@gmail.com" Date: Thu, 23 Sep 2010 11:21:25 -0700 Thread-Topic: bce(4) - com_no_buffers (Again) Thread-Index: ActbMNbcz5WrJs6wRPioWTxM0VHJgAAGauzg Message-ID: <5D267A3F22FD854F8F48B3D2B52381933B5A78B484@IRVEXCHCCR01.corp.ad.broadcom.com> References: <4C894A76.5040200@tomjudge.com> <20100910002439.GO7203@michelle.cdnetworks.com> <4C8E3D79.6090102@tomjudge.com> <20100913184833.GF1229@michelle.cdnetworks.com> <4C8E768E.7000003@tomjudge.com> <20100913193322.GG1229@michelle.cdnetworks.com> <4C8E8BD1.5090007@tomjudge.com> <20100913205348.GJ1229@michelle.cdnetworks.com> <4C9B6CBD.2030408@tomjudge.com> In-Reply-To: <4C9B6CBD.2030408@tomjudge.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-cr-hashedpuzzle: A5nR Bu7n CfqD EaQY JXHx XRv6 aZ+G eE9k fVFW mrUr qrfH q7eJ reMx s2fB uNYX vLS1; 4; ZgByAGUAZQBiAHMAZAAtAG4AZQB0AEAAZgByAGUAZQBiAHMAZAAuAG8AcgBnADsAcAB5AHUAbgB5AGgAQABnAG0AYQBpAGwALgBjAG8AbQA7AHQAbwBtAEAAdABvAG0AagB1AGQAZwBlAC4AYwBvAG0AOwB5AG8AbgBnAGEAcgBpAEAAZgByAGUAZQBiAHMAZAAuAG8AcgBnAA==; Sosha1_v1; 7; {2C203A59-FC76-4AEB-BE36-EFE4F6E8D7F5}; ZABhAHYAaQBkAGMAaABAAGIAcgBvAGEAZABjAG8AbQAuAGMAbwBtAA==; Thu, 23 Sep 2010 18:21:25 GMT; UgBFADoAIABiAGMAZQAoADQAKQAgAC0AIABjAG8AbQBfAG4AbwBfAGIAdQBmAGYAZQByAHMAIAAoAEEAZwBhAGkAbgApAA== x-cr-puzzleid: {2C203A59-FC76-4AEB-BE36-EFE4F6E8D7F5} acceptlanguage: en-US MIME-Version: 1.0 X-WSS-ID: 6085453F3N04757123-01-01 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Cc: "freebsd-net@freebsd.org" , "yongari@freebsd.org" Subject: RE: bce(4) - com_no_buffers (Again) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Sep 2010 18:22:50 -0000 > >> Under testing I have yet to see a memory fragmentation issue with > this > >> driver. I follow up if/when I find a problem with this again. > >> > >> > So here we are again. The system is locking up again because of 9k > mbuf > allocation failures. Failure to allocate a new buffer should cause the driver to drop the received frame and reuse the buffer, not lock up the system. Are you seeing the lockup come from bce(4) or does it come from somewhere else due to the dropped data? > >> Is there a way to fix the RX buffer shortage issues (when header > >> splitting is turned on) so that they are guarded by flow control. > Maybe > >> change the low watermark for flow control when its enabled? > >> > >> > > I'm not sure how much it would help but try changing RX low > > watermark. Default value is 32 which seems to be reasonable value. > > But it's only for 5709/5716 controllers and Linux seems to use > > different default value. > > > These are: NetXtreme II BCM5709 Gigabit Ethernet >=20 > So my next task is to turn the watermark related defines into sysctls > and turn on header splitting so that I can try to tune them without > having to reboot. > Do you have flow control enabled? There are arguments both for and against flow control. For bce(4), I haven't tested flow control for quite a while and it's behavior may have changed since it is controlled by firmware. Keep an eye on the hardware statistics to see that's it's actively generating pause frames. >=20 > My next question is, is it possible to increase the size of the RX ring > without switching to RSS? > I have a change I've been working on to allow RX/TX ring size to be adjusted through a sysctl. Let me pretty it up a bit and send it to you for test. You should be able to adjust the ring size without enabling RSS. Dave=20