From owner-freebsd-stable@FreeBSD.ORG Mon Nov 28 22:38:35 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BC36F1065675 for ; Mon, 28 Nov 2011 22:38:35 +0000 (UTC) (envelope-from mandrews@bit0.com) Received: from magnum.bit0.com (magnum.bit0.com [IPv6:2604:e700:b0:1::200]) by mx1.freebsd.org (Postfix) with ESMTP id 7F5368FC23 for ; Mon, 28 Nov 2011 22:38:35 +0000 (UTC) Received: from magnum.bit0.com (localhost [127.0.0.1]) by magnum.bit0.com (Postfix) with ESMTP id CC85E57CB; Mon, 28 Nov 2011 17:38:34 -0500 (EST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed; d=bit0.com; h=message-id :date:from:mime-version:to:cc:subject:references:in-reply-to :content-type:content-transfer-encoding; s=boogity; bh=/8kCOfKfi Ac7mvxOmtznYSG8ozzUPYzRDOTs4GnTCXw=; b=TA2NMrA8I9XNpH0VtGdeLRaqg gS/ATMNNS3WGxUVj++yVQa9iM/AfTydc2rft4q36RiIewt9ZLtKjV2n1jCxFOhM2 b7xDeTzEX6lKE0hiJjOw6elx9Q1hTcibu6/0AoH0HTGVgvFGRmfBBRZaoAxY64yi ppVrS99c1OeAck4Tdo= DomainKey-Signature: a=rsa-sha1; c=nofws; d=bit0.com; h=message-id:date :from:mime-version:to:cc:subject:references:in-reply-to :content-type:content-transfer-encoding; q=dns; s=boogity; b=qKG lL5bt6Ie+uFXvWBzRL4bwohxf849W++iaovw+JLtiSF/R11J2enOXcoqLq39V4Pv jpsRU8JrzNlXJqVhLov3qFTdX9lw6ejX8Fby5DLSJSnD5S+Hur8kMQt9jiUoWjaW HzNQS9OlxI7Pulei8WbxwnUF0oFmsyNog9DSc8hs= Received: from spike.int.bit0.com (unknown [IPv6:2001:470:1f11:c3c:d073:ca92:2999:f62f]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by magnum.bit0.com (Postfix) with ESMTPSA id 6C11F57CA; Mon, 28 Nov 2011 17:38:33 -0500 (EST) Message-ID: <4ED40D58.1030107@bit0.com> Date: Mon, 28 Nov 2011 17:38:16 -0500 From: Mike Andrews User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:8.0) Gecko/20111105 Thunderbird/8.0 MIME-Version: 1.0 To: pyunyh@gmail.com References: <4ED154B6.2030304@bit0.com> <20111128013931.GC1830@michelle.cdnetworks.com> In-Reply-To: <20111128013931.GC1830@michelle.cdnetworks.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-stable@freebsd.org Subject: Re: 9.0-RC2 re(4) "no memory for jumbo buffers" issue X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Nov 2011 22:38:35 -0000 On 11/27/11 8:39 PM, YongHyeon PYUN wrote: > On Sat, Nov 26, 2011 at 04:05:58PM -0500, Mike Andrews wrote: >> I have a Supermicro 5015A-H (Intel Atom 330) server with two Realtek >> RTL8111C-GR gigabit NICs on it. As far as I can tell, these support >> jumbo frames up to 7422 bytes. When running them at an MTU of 5000 on > > Actually the maximum size is 6KB for RTL8111C, not 7422. > RTL8111C and newer PCIe based gigabit controllers no longer support > scattering a jumbo frame into multiple RX buffers so a single RX > buffer has to receive an entire jumbo frame. This adds more burden > to system because it has to allocate a jumbo frame even when it > receives a pure TCP ACK. OK, that makes sense. >> FreeBSD 9.0-RC2, after a week or so of update, with fairly light network >> activity, the interfaces die with "no memory for jumbo buffers" errors >> on the console. Unloading and reloading the driver (via serial console) >> doesn't help; only rebooting seems to clear it up. >> > > The jumbo code path is the same as normal MTU sized one so I think > possibility of leaking mbufs in driver is very low. And the > message "no memory for jumbo RX buffers" can only happen either > when you up the interface again or interface restart triggered by > watchdog timeout handler. I don't think you're seeing watchdog > timeouts though. I'm fairly certain the interface isn't changing state when this happens -- it just kinda spontaneously happens after a week or two, with no interface up/down transitions. I don't see any watchdog messages when this happens. > When you see "no memory for jumbo RX buffers" message, did you > check available mbuf pool? Not yet, that's why I asked for debugging tips -- I'll do that the next time this happens. >> What's the best way to go about debugging this... which sysctl's should >> I be looking at first? I have already tried raising kern.ipc.nmbjumbo9 >> to 16384 and it doesn't seem to help things... maybe prolonging it >> slightly, but not by much. The problem is it takes a week or so to >> reproduce the problem each time... >> > > I vaguely guess it could be related with other subsystem which > leaks mbufs such that driver was not able to get more jumbo RX > buffers from system. For instance, r228016 would be worth to try on > your box. I can't clearly explain why em(4) does not suffer from > the issue though. I've just this morning built a kernel with that fix, so we'll see how that goes.