From owner-freebsd-stable@FreeBSD.ORG Sun Feb 28 12:11:47 2010 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DF4D2106564A; Sun, 28 Feb 2010 12:11:47 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from kabab.cs.huji.ac.il (kabab.cs.huji.ac.il [132.65.16.84]) by mx1.freebsd.org (Postfix) with ESMTP id 89B018FC0C; Sun, 28 Feb 2010 12:11:47 +0000 (UTC) Received: from pampa.cs.huji.ac.il ([132.65.80.32]) by kabab.cs.huji.ac.il with esmtp id 1Nlhzr-0004ci-3W; Sun, 28 Feb 2010 14:11:43 +0200 X-Mailer: exmh version 2.7.2 01/07/2005 with nmh-1.2 To: Jeremy Chadwick In-reply-to: <20100227220310.GA65110@icarus.home.lan> References: <20100226174021.8feadad9.gerrit@pmp.uni-hannover.de> <20100226224320.8c4259bf.gerrit@pmp.uni-hannover.de> <4B884757.9040001@digiware.nl> <20100227080220.ac6a2e4d.gerrit@pmp.uni-hannover.de> <4B892918.4080701@digiware.nl> <20100227202105.f31cbef7.gerrit@pmp.uni-hannover.de> <20100227193819.GA60576@icarus.home.lan> <4B89943C.70704@digiware.nl> <20100227220310.GA65110@icarus.home.lan> Comments: In-reply-to Jeremy Chadwick message dated "Sat, 27 Feb 2010 14:03:10 -0800." Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Date: Sun, 28 Feb 2010 14:11:42 +0200 From: Daniel Braniss Message-ID: Cc: stable@freebsd.org, freebsd-fs@freebsd.org, Willem Jan Withagen , Gerrit =?iso-8859-1?Q?K=FChn?= , Eirik =?iso-8859-1?Q?=D8verby?= , rwatson@freebsd.org Subject: Re: mbuf leakage with nfs/zfs? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 28 Feb 2010 12:11:48 -0000 > On Sat, Feb 27, 2010 at 10:53:00PM +0100, Willem Jan Withagen wrote: > > On 27-2-2010 21:32, Eirik =D8verby wrote: > > >I've had a discussion with some folks on this for a while. I can eas= il=3Dy > > >reproduce this situation by mounting a FreeBSD ZFS filesystem via > > >NFS-UDP from an OpenBSD machine. Telling the OpenBSD machine to use = TC=3DP > > >instead of UDP makes the problem go away. > > > > > >Other FreeBSD systems mounting the same share, either using UDP or T= CP=3D, > > >does not cause the problem to show up. > > > > > >A patch was suggested by Rick Macklem, but that did not solve the is= su=3De: > > >http://lists.freebsd.org/pipermail/freebsd-current/2009-December/014= 18=3D1.html> >=20 > > I concur. > > Everything in my network is now on TCP, and there is no mbuf leakage.= > > I just don't get over the 5500 mark, no matter what I throw at it. > >=20 > > I do feel that TCP is not as well performing on a local net with Linu= x, > > hence the choice for UDP. But TCP is workable as next best. >=20 > I'm pulling in Robert Watson, who has some familiarity with the UDP > stack/code in FreeBSD. I'm not sure he'll be a sufficient source of > knowledge for this specific issue since it appears (?) to be specific t= o > NFS; Rick Macklem would be a better choice, but as reported, he's MIA. >=20 > Robert, are you aware of any changes or implementation issues which > might cause excessive (read: leaking) mbuf use under UDP-based NFS? Do= > you know of a way folks could determine the source of the leak, either > via DDB or while the system is live? I have been runing some tests in a controlled environment. server and client are both 64bit Xeon/X5550 =40 2.67GHz with 16Gb of mem= ory FreeBSD/SMP: 2 package(s) x 4 core(s) x 2 SMT threads the client is runing latest 8.0 stable the load is created by runing 'make -j32 buildworld' and sleeping 150 sec= . in between runs, this is the straight line you will see in the graphs. Both the src and obj directories are NFS mounted from the server, regular= UFS. when server is running 7.2-stable no leakage is seen. see ftp://ftp.cs.huji.ac.il/users/danny/freebsd/mbufs/=7Btcp,udp=7D-7.2.= ps when server is runing 8.0-stable see ftp://ftp.cs.huji.ac.il/users/danny/freebsd/mbufs/=7Btcp,udp=7D-8.0.= ps you can see that udp is leaking=21 cheers, danny ps: I think the subject should be changed again, removing zfs ...