From owner-freebsd-net@FreeBSD.ORG Tue Aug 14 00:10:33 2007 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7DD6116A417 for ; Tue, 14 Aug 2007 00:10:33 +0000 (UTC) (envelope-from lstewart@room52.net) Received: from swin.edu.au (gpo3.cc.swin.edu.au [136.186.1.223]) by mx1.freebsd.org (Postfix) with ESMTP id D9CF513C457 for ; Tue, 14 Aug 2007 00:10:32 +0000 (UTC) (envelope-from lstewart@room52.net) Received: from [136.186.229.95] (lstewart.caia.swin.edu.au [136.186.229.95]) by swin.edu.au (8.13.6.20060614/8.13.1) with ESMTP id l7DNqMeH029020; Tue, 14 Aug 2007 09:52:23 +1000 Message-ID: <46C0EEB6.6010702@room52.net> Date: Tue, 14 Aug 2007 09:52:22 +1000 From: Lawrence Stewart User-Agent: Thunderbird 1.5.0.9 (X11/20070123) MIME-Version: 1.0 To: Kevin Oberman References: <20070813183426.C196D45045@ptavv.es.net> In-Reply-To: <20070813183426.C196D45045@ptavv.es.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-1.4 required=5.0 tests=ALL_TRUSTED autolearn=disabled version=3.1.9 X-Spam-Checker-Version: SpamAssassin 3.1.9 (2007-02-13) on gpo3.cc.swin.edu.au Cc: freebsd-net@freebsd.org Subject: Re: Very high wide area TCP tuning X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Aug 2007 00:10:33 -0000 Hi Kevin, Kevin Oberman wrote: > I am attempting to use a FreeBSD box with either a Myricom or Chelsio > 10GE card to generate very large TCP streams over cross-country > links. The RTT for the test path is 94 ms. It is dedicated to my testing > at this time, so I have no contention other than a few KB of routing > updates. > > Clearly, I need a very large window...about 120 MB, but I am unsure how > FreeBSD will handle this. (Unless I do other things, I suspect it will > not be pretty.) I imagine I will need a large kvm space, at the least, > but are there any other sysctls that are likely to need adjusting to > make this all work? IS it likely to work better on a amd64 system than a > i386? > > Any suggestions would be appreciated. > We recently wrote a small technical report that covers some of the issues related to FreeBSD network tuning for some TCP research we've been doing. The report's title is "Tuning and Testing the FreeBSD 6 TCP Stack" and you can grab it from here: http://caia.swin.edu.au/reports/070717B/CAIA-TR-070717B.pdf We didn't tune for 10GB speeds, but I imagine the principles should all still apply. As you correctly point out, you'll also probably want to raise the allowable kernel mem size using the "vm.kmem_size" and "vm.kmem_size_max" sysctls to avoid any random kernel panics. We used a kmem size of 500MB which worked flawlessly with our GigE multiflow tests and left us plenty of room to move. This is something you might have to use a bit of trial and error to figure out though to ensure you get something stable. And of course you're going to want to control the number of flows active at one time based on your kmem setting and hardware constraints... Cheers, Lawrence http://caia.swin.edu.au