From owner-freebsd-net@FreeBSD.ORG  Wed Feb 12 22:19:44 2014
Return-Path: <owner-freebsd-net@FreeBSD.ORG>
Delivered-To: net@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115])
 (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by hub.freebsd.org (Postfix) with ESMTPS id DBD33C93;
 Wed, 12 Feb 2014 22:19:44 +0000 (UTC)
Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1])
 (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits))
 (No client certificate requested)
 by mx1.freebsd.org (Postfix) with ESMTPS id A8E131518;
 Wed, 12 Feb 2014 22:19:44 +0000 (UTC)
Received: from jhbbsd.localnet (unknown [209.249.190.124])
 by bigwig.baldwin.cx (Postfix) with ESMTPSA id 9C3D7B94B;
 Wed, 12 Feb 2014 17:19:40 -0500 (EST)
From: John Baldwin <jhb@freebsd.org>
To: Adrian Chadd <adrian@freebsd.org>
Subject: Re: Use of contiguous physical memory in cxgbe driver
Date: Wed, 12 Feb 2014 14:46:19 -0500
User-Agent: KMail/1.13.5 (FreeBSD/8.4-CBSD-20130906; KDE/4.5.5; amd64; ; )
References: <21216.22944.314697.179039@hergotha.csail.mit.edu>
 <201402111348.52135.jhb@freebsd.org>
 <CAJ-VmonCdNQPUCQwm0OhqQ3Kt_7x6-g-JwGVZQfzWTgrDYfmqw@mail.gmail.com>
In-Reply-To: <CAJ-VmonCdNQPUCQwm0OhqQ3Kt_7x6-g-JwGVZQfzWTgrDYfmqw@mail.gmail.com>
MIME-Version: 1.0
Content-Type: Text/Plain;
  charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
Message-Id: <201402121446.19278.jhb@freebsd.org>
X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7
 (bigwig.baldwin.cx); Wed, 12 Feb 2014 17:19:40 -0500 (EST)
Cc: FreeBSD Net <freebsd-net@freebsd.org>,
 Garrett Wollman <wollman@freebsd.org>, Navdeep Parhar <np@freebsd.org>,
 "net@freebsd.org" <net@freebsd.org>
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.17
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net/>
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 12 Feb 2014 22:19:44 -0000

On Tuesday, February 11, 2014 6:30:04 pm Adrian Chadd wrote:
> On 11 February 2014 10:48, John Baldwin <jhb@freebsd.org> wrote:
> > On Thursday, January 23, 2014 2:32:51 am Adrian Chadd wrote:
> >> It's about time we taught the physmem allocator to be more conducive
> >> to physically contiguous allocations.
> >>
> >> A server with gigabytes of memory should be able to keep a couple tens
> >> of megabytes of 64k sized allocation chunks around for exactly this.
> >
> > Alan Cox already taught the physmem allocator to do this for superpages.
> > However, this change was part of the superpages changes, so you can't 
count
> > experiences from machines older than about 7.2 when evaluating the
> > effectiveness of the new allocator.
> 
> the problem is that we don't have pressure to _not_ fragment the
> physical memory from the allocator, so all of the memory ends up
> getting very fragmented versy quickly.
> 
> the physmem superpage stuff stops being viable after a short while of
> say, "pound lots of packets around from vm objects" workload, because
> suddenly we end up chewing through all of physical memory with pages
> extremely quickly.

Is this because UMA keeps lots of mbufs cached in your workload?  The physmem 
buddy allocator certainly seeks to minimize fragmentation.  However, it can't 
go yank memory out of UMA caches to do so.

-- 
John Baldwin