From owner-freebsd-net@FreeBSD.ORG  Thu Aug 28 11:17:12 2014
Return-Path: <owner-freebsd-net@FreeBSD.ORG>
Delivered-To: freebsd-net@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by hub.freebsd.org (Postfix) with ESMTPS id 0743BD5A
 for <freebsd-net@freebsd.org>; Thu, 28 Aug 2014 11:17:12 +0000 (UTC)
Received: from mail.ipfw.ru (mail.ipfw.ru [IPv6:2a01:4f8:120:6141::2])
 (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
 (Client did not present a certificate)
 by mx1.freebsd.org (Postfix) with ESMTPS id BFB5F1470
 for <freebsd-net@freebsd.org>; Thu, 28 Aug 2014 11:17:11 +0000 (UTC)
Received: from [2a02:6b8:0:401:222:4dff:fe50:cd2f] (helo=ptichko.yndx.net)
 by mail.ipfw.ru with esmtpsa (TLSv1:DHE-RSA-AES128-SHA:128)
 (Exim 4.82 (FreeBSD)) (envelope-from <melifaro@FreeBSD.org>)
 id 1XMtjU-000HLb-9V; Thu, 28 Aug 2014 11:02:56 +0400
Message-ID: <53FF0FAE.1010809@FreeBSD.org>
Date: Thu, 28 Aug 2014 15:17:02 +0400
From: "Alexander V. Chernikov" <melifaro@FreeBSD.org>
User-Agent: Mozilla/5.0 (X11; FreeBSD amd64;
 rv:24.0) Gecko/20100101 Thunderbird/24.6.0
MIME-Version: 1.0
To: "Eggert, Lars" <lars@netapp.com>, 
 FreeBSD Net <freebsd-net@freebsd.org>
Subject: Re: igb "requests for mbufs denied"
References: <B170AAF8-E59B-4758-BF68-A0B18430B0AD@netapp.com>
In-Reply-To: <B170AAF8-E59B-4758-BF68-A0B18430B0AD@netapp.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.18-1
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net/>
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Thu, 28 Aug 2014 11:17:12 -0000

On 28.08.2014 12:31, Eggert, Lars wrote:
> Hi,
>
> no matter what value I bump kern.ipc.nmbclusters and kern.ipc.nmbufs to, I still get "requests for mbufs denied" with igb interfaces, and the occasional connection stall, even when dialing down hw.igb.num_queues=1:
>
> [root@laurel: ~] netstat -m
> 3070/1355/4425 mbufs in use (current/cache/total)
> 3069/773/3842/8144896 mbuf clusters in use (current/cache/total/max)
> 3069/772 mbuf+clusters out of packet secondary zone in use (current/cache)
> 0/6/6/1018097 4k (page size) jumbo clusters in use (current/cache/total/max)
> 0/0/0/301658 9k jumbo clusters in use (current/cache/total/max)
> 0/0/0/169682 16k jumbo clusters in use (current/cache/total/max)
> 6905K/1908K/8814K bytes allocated to network (current/cache/total)
> 73/3831/3091 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
> 0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters)
> 0/0/0 requests for jumbo clusters delayed (4k/9k/16k)
> 6/0/0 requests for jumbo clusters denied (4k/9k/16k)
> 0 requests for sfbufs denied
> 0 requests for sfbufs delayed
> 0 requests for I/O initiated by sendfile
>
> The box in question has six igb interfaces, 2x 'I210 Gigabit Network Connection' and 4x '82580 Gigabit Network Connection' and is running:
>
> FreeBSD laurel.muccbc.hq.netapp.com 10.0-RELEASE-p7 FreeBSD 10.0-RELEASE-p7 #0: Tue Jul  8 06:37:44 UTC 2014     root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64
>
> Anyone have any ideas what else to try?
Do you have jumbo frames turned on?
>
> Thanks,
> Lars