From owner-freebsd-stable@FreeBSD.ORG Wed Feb 22 20:51:24 2012 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CA9B0106566B for ; Wed, 22 Feb 2012 20:51:24 +0000 (UTC) (envelope-from luigi@onelab2.iet.unipi.it) Received: from onelab2.iet.unipi.it (onelab2.iet.unipi.it [131.114.59.238]) by mx1.freebsd.org (Postfix) with ESMTP id 896658FC25 for ; Wed, 22 Feb 2012 20:51:24 +0000 (UTC) Received: by onelab2.iet.unipi.it (Postfix, from userid 275) id 058657300A; Wed, 22 Feb 2012 21:52:32 +0100 (CET) Date: Wed, 22 Feb 2012 21:52:32 +0100 From: Luigi Rizzo To: Jack Vogel Message-ID: <20120222205231.GA81949@onelab2.iet.unipi.it> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i Cc: FreeBSD Net , FreeBSD stable , re Subject: Re: nmbclusters: how do we want to fix this for 8.3 ? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Feb 2012 20:51:24 -0000 On Wed, Feb 22, 2012 at 11:56:29AM -0800, Jack Vogel wrote: > Using igb and/or ixgbe on a reasonably powered server requires 1K mbuf > clusters per MSIX vector, > that's how many are in a ring. Either driver will configure 8 queues on a > system with that many or more > cores, so 8K clusters per port... > > My test engineer has a system with 2 igb ports, and 2 10G ixgbe, this is > hardly heavy duty, and yet this > exceeds the default mbuf pool on the installed kernel (1024 + maxusers * > 64). > > Now, this can be immediately fixed by a sysadmin after that first boot, but > it does result in the second > driver that gets started to complain about inadequate buffers. > > I think the default calculation is dated and should be changed, but am not > sure the best way, so are > there suggestions/opinions about this, and might we get it fixed before 8.3 > is baked? I have hit this problem recently, too. Maybe the issue mostly/only exists on 32-bit systems. Here is a possible approach: 1. nmbclusters consume the kernel virtual address space so there must be some upper limit, say VM_LIMIT = 256000 (translates to 512MB of address space) 2. also you don't want the clusters to take up too much of the available memory. This one would only trigger for minimal-memory systems, or virtual machines, but still... MEM_LIMIT = (physical_ram / 2) / 2048 3. one may try to set a suitably large, desirable number of buffers TARGET_CLUSTERS = 128000 4. and finally we could use the current default as the absolute minimum MIN_CLUSTERS = 1024 + maxusers*64 Then at boot the system could say nmbclusters = min(TARGET_CLUSTERS, VM_LIMIT, MEM_LIMIT) nmbclusters = max(nmbclusters, MIN_CLUSTERS) In turn, i believe interfaces should do their part and by default never try to allocate more than a fraction of the total number of buffers, if necessary reducing the number of active queues. what do people think ? cheers luigi