Date: Tue, 14 Aug 2007 09:52:22 +1000 From: Lawrence Stewart <lstewart@room52.net> To: Kevin Oberman <oberman@es.net> Cc: freebsd-net@freebsd.org Subject: Re: Very high wide area TCP tuning Message-ID: <46C0EEB6.6010702@room52.net> In-Reply-To: <20070813183426.C196D45045@ptavv.es.net> References: <20070813183426.C196D45045@ptavv.es.net>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Kevin, Kevin Oberman wrote: > I am attempting to use a FreeBSD box with either a Myricom or Chelsio > 10GE card to generate very large TCP streams over cross-country > links. The RTT for the test path is 94 ms. It is dedicated to my testing > at this time, so I have no contention other than a few KB of routing > updates. > > Clearly, I need a very large window...about 120 MB, but I am unsure how > FreeBSD will handle this. (Unless I do other things, I suspect it will > not be pretty.) I imagine I will need a large kvm space, at the least, > but are there any other sysctls that are likely to need adjusting to > make this all work? IS it likely to work better on a amd64 system than a > i386? > > Any suggestions would be appreciated. > We recently wrote a small technical report that covers some of the issues related to FreeBSD network tuning for some TCP research we've been doing. The report's title is "Tuning and Testing the FreeBSD 6 TCP Stack" and you can grab it from here: http://caia.swin.edu.au/reports/070717B/CAIA-TR-070717B.pdf We didn't tune for 10GB speeds, but I imagine the principles should all still apply. As you correctly point out, you'll also probably want to raise the allowable kernel mem size using the "vm.kmem_size" and "vm.kmem_size_max" sysctls to avoid any random kernel panics. We used a kmem size of 500MB which worked flawlessly with our GigE multiflow tests and left us plenty of room to move. This is something you might have to use a bit of trial and error to figure out though to ensure you get something stable. And of course you're going to want to control the number of flows active at one time based on your kmem setting and hardware constraints... Cheers, Lawrence http://caia.swin.edu.au
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?46C0EEB6.6010702>