From owner-freebsd-net@FreeBSD.ORG  Fri Mar  8 17:09:57 2013
Return-Path: <owner-freebsd-net@FreeBSD.ORG>
Delivered-To: freebsd-net@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 by hub.freebsd.org (Postfix) with ESMTP id 5D674D4E;
 Fri,  8 Mar 2013 17:09:57 +0000 (UTC)
 (envelope-from wollman@hergotha.csail.mit.edu)
Received: from hergotha.csail.mit.edu
 (wollman-1-pt.tunnel.tserv4.nyc4.ipv6.he.net [IPv6:2001:470:1f06:ccb::2])
 by mx1.freebsd.org (Postfix) with ESMTP id 064B15EA;
 Fri,  8 Mar 2013 17:09:56 +0000 (UTC)
Received: from hergotha.csail.mit.edu (localhost [127.0.0.1])
 by hergotha.csail.mit.edu (8.14.5/8.14.5) with ESMTP id r28H9ugC003460;
 Fri, 8 Mar 2013 12:09:56 -0500 (EST)
 (envelope-from wollman@hergotha.csail.mit.edu)
Received: (from wollman@localhost)
 by hergotha.csail.mit.edu (8.14.5/8.14.4/Submit) id r28H9uOI003457;
 Fri, 8 Mar 2013 12:09:56 -0500 (EST) (envelope-from wollman)
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Message-ID: <20794.7012.265887.99878@hergotha.csail.mit.edu>
Date: Fri, 8 Mar 2013 12:09:56 -0500
From: Garrett Wollman <wollman@bimajority.org>
To: Andre Oppermann <andre@freebsd.org>
Subject: Re: Limits on jumbo mbuf cluster allocation
In-Reply-To: <51399926.6020201@freebsd.org>
References: <20793.36593.774795.720959@hergotha.csail.mit.edu>
 <51399926.6020201@freebsd.org>
X-Mailer: VM 7.17 under 21.4 (patch 22) "Instant Classic" XEmacs Lucid
X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7
 (hergotha.csail.mit.edu [127.0.0.1]); Fri, 08 Mar 2013 12:09:56 -0500 (EST)
X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED
 autolearn=disabled version=3.3.2
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on
 hergotha.csail.mit.edu
Cc: jfv@freebsd.org, freebsd-net@freebsd.org,
 Garrett Wollman <wollman@freebsd.org>
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net>
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Fri, 08 Mar 2013 17:09:57 -0000

<<On Fri, 08 Mar 2013 08:54:14 +0100, Andre Oppermann <andre@freebsd.org> said:

> [stuff I wrote deleted]
> You have an amd64 kernel running HEAD or 9.x?

Yes, these are 9.1 with some patches to reduce mutex contention on the
NFS server's replay "cache".

> Jumbo pages come directly from the kernel_map which on amd64 is 512GB.
> So KVA shouldn't be a problem.  Your problem indeed appears to come
> physical memory fragmentation in pmap.

I hadn't realized that they were physically contiguous, but that makes
perfect sense.

> pages.  Also since you're doing NFS serving almost all memory will be
> in use for file caching.

I actually had the ZFS ARC tuned down to 64 GB (out of 96 GB physmem)
when I experienced this, but there are plenty of data structures in
the kernel that aren't subject to this limit and I could easily
imagine them checkerboarding physical memory to the point where no
contiguous three-page allocations were possible.

-GAWollman