From owner-freebsd-net@FreeBSD.ORG Thu Mar 10 14:22:09 2005 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8D74416A4CE for ; Thu, 10 Mar 2005 14:22:09 +0000 (GMT) Received: from ints.mail.pike.ru (ints.mail.pike.ru [195.9.45.194]) by mx1.FreeBSD.org (Postfix) with ESMTP id 41BBA43D5A for ; Thu, 10 Mar 2005 14:22:08 +0000 (GMT) (envelope-from babolo@cicuta.babolo.ru) Received: (qmail 26151 invoked from network); 10 Mar 2005 14:22:06 -0000 Received: from cicuta.babolo.ru (194.135.49.133) by ints.mail.pike.ru with SMTP; 10 Mar 2005 14:22:06 -0000 Received: (nullmailer pid 29388 invoked by uid 136); Thu, 10 Mar 2005 14:23:00 -0000 X-ELM-OSV: (Our standard violations) hdr-charset=KOI8-R; no-hdr-encoding=1 In-Reply-To: <3.0.1.32.20050309135120.00a7f5c0@pop.redshift.com> To: ray@redshift.com Date: Thu, 10 Mar 2005 17:23:00 +0300 (MSK) From: "."@babolo.ru X-Mailer: ELM [version 2.4ME+ PL99b (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII Message-Id: <1110464580.382085.29387.nullmailer@cicuta.babolo.ru> cc: freebsd-net@freebsd.org Subject: Re: FreeBSD router question X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Mar 2005 14:22:09 -0000 > Hello (just signed up to this list), > > I am wondering if anyone on the list has any experience using FreeBSD 5.3 as a > router in a high traffic environment? I am building a development cluster here > and have decided to try using FreeBSD as my main network router instead of > something like the Cisco 7200's, Force10, etc. > > I have 10 or 12 Xeon machines in my cluster so far, but may have as many as 50 > to 100 in the future (once our site goes live). Right now I have a 2.40 GHz > Xeon with 2GB of RAM running as the router using FreeBSD 5.3, ipf and ipnat > (this may be upgraded to an AMD64 bit dual core shortly). So far everything > seems to work fine, but it has not been under heavy load yet. The router has > been up for 26 days with no problems and works great. > > I've made the following tweaks (see end of message) to sysctl.conf in an effort > to get things going the right direction. I've also stripped down the kernel > file and recompiled. I read recently that FreeBSD was able to route 1Mpps, > which sounded pretty good, but I don't know if there are any specific tweaks I > need to make in order to obtain this sort of speed, or how fast it works "out of > the box" with just a few modifications? My main concern is that the router > works okay now, but when traffic ramps up, it hits a wall without some large > amount of exotic changes. I'd like to feel comfortable that the machine will > handle at least 50 to 100 megabits of traffic on a fairly sustained basis > without facing any major problems. Is that realistic or are there specific > changes I should make to the OS? > > If anyone on the list has any first hand information/experience that might steer > me the right direction, that would be great. Any feed back would be great, > Thanks very much! :-) We are using a lot of FreeBSD 4 routers. They route up to 35..40 Tbytes/router, 4..70 vlans per router, natd and argus runs for most of vlans, 1 natd and 1 argus per vlan. ipfw config is about 30..100 Kbyte, pipes for about half of traffic. Athlon XP on 760MPX mobo, 1Gbyte of memory. 2000 GHz (real) Athlon XP is 2+ faster router compare to 2.6 GHz Pentium 4. Configurators (route, arp, ipfw utilities) are something buggy under high load (we have up to 500 reconfigures/day), and second CPU is not useful if Athlon MP is used. I have bad impression on my FreeBSD 5 test on our routers and good on DragonFlyBSD test, but have no DragonFlyBSD router under full load yet. ... > net.inet.ip.fastforwarding=0 # not sure about this, but might want to It is hard to build complex ipfw rules with fastforwarding=1, dont know about ipf. > net.inet.tcp.recvspace=65535 # increase TCP window size for better > net.inet.tcp.sendspace=65535 Not used for routing. > kern.ipc.somaxconn=1024 # increase listen queue (defense against > SYN attacks, better performance) [128] Just close router fully, do not accept any connect but from one control interface from fully seperated internal net.