From owner-freebsd-net@FreeBSD.ORG Tue Mar 19 04:29:59 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id AF551397; Tue, 19 Mar 2013 04:29:59 +0000 (UTC) (envelope-from wollman@hergotha.csail.mit.edu) Received: from hergotha.csail.mit.edu (wollman-1-pt.tunnel.tserv4.nyc4.ipv6.he.net [IPv6:2001:470:1f06:ccb::2]) by mx1.freebsd.org (Postfix) with ESMTP id 281526E; Tue, 19 Mar 2013 04:29:58 +0000 (UTC) Received: from hergotha.csail.mit.edu (localhost [127.0.0.1]) by hergotha.csail.mit.edu (8.14.5/8.14.5) with ESMTP id r2J4Tvgu085622; Tue, 19 Mar 2013 00:29:57 -0400 (EDT) (envelope-from wollman@hergotha.csail.mit.edu) Received: (from wollman@localhost) by hergotha.csail.mit.edu (8.14.5/8.14.4/Submit) id r2J4TvgG085619; Tue, 19 Mar 2013 00:29:57 -0400 (EDT) (envelope-from wollman) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <20807.59845.764047.618551@hergotha.csail.mit.edu> Date: Tue, 19 Mar 2013 00:29:57 -0400 From: Garrett Wollman To: Rick Macklem Subject: Re: Limits on jumbo mbuf cluster allocation In-Reply-To: <75232221.3844453.1363146480616.JavaMail.root@erie.cs.uoguelph.ca> References: <20798.44871.601547.24628@hergotha.csail.mit.edu> <75232221.3844453.1363146480616.JavaMail.root@erie.cs.uoguelph.ca> X-Mailer: VM 7.17 under 21.4 (patch 22) "Instant Classic" XEmacs Lucid X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (hergotha.csail.mit.edu [127.0.0.1]); Tue, 19 Mar 2013 00:29:58 -0400 (EDT) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=disabled version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on hergotha.csail.mit.edu Cc: freebsd-net@freebsd.org, andre@freebsd.org, Ivan Voras X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Mar 2013 04:29:59 -0000 < said: > I've attached a patch that has assorted changes. So I've done some preliminary testing on a slightly modified form of this patch, and it appears to have no major issues. However, I'm still waiting for my user with 500 VMs to have enough free to be able to run some real stress tests for me. I was able to get about 2.5 Gbit/s throughput for a single streaming client over local 10G interfaces with jumbo frames (through a single switch and with LACP on both sides -- how well does lagg(4) interact with TSO and checksum offload?) This is a little bit disappointing (considering that the filesystem can do 14 Gbit/s locally) but still pretty decent for one single-threaded client. This obviously does not implicate the DRC changes at all, but does suggest that there is room for more performance improvement. (In previous tests last year, I was able to get a sustained 8 Gbit/s when using multiple clients.) I also found that one of our 10G switches is reordering TCP segments in a way that causes poor performance. I'll hopefully have some proper testing results later in the week. -GAWollman