From owner-freebsd-questions@FreeBSD.ORG Fri Sep 3 16:14:08 2004 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7634716A4CE for ; Fri, 3 Sep 2004 16:14:08 +0000 (GMT) Received: from mproxy.gmail.com (rproxy.gmail.com [64.233.170.195]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2A23043D1D for ; Fri, 3 Sep 2004 16:14:08 +0000 (GMT) (envelope-from subhro.kar@gmail.com) Received: by mproxy.gmail.com with SMTP id 80so17823rnl for ; Fri, 03 Sep 2004 09:14:02 -0700 (PDT) Received: by 10.38.206.40 with SMTP id d40mr228863rng; Fri, 03 Sep 2004 09:14:02 -0700 (PDT) Received: by 10.38.206.16 with HTTP; Fri, 3 Sep 2004 09:14:02 -0700 (PDT) Message-ID: Date: Fri, 3 Sep 2004 21:44:02 +0530 From: Subhro To: "freebsd-questions@FreeBSD. ORG" In-Reply-To: <20040903120734.GA28796@pobox.com.> Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit References: <20040903120734.GA28796@pobox.com.> cc: simon@synatech.com.au Subject: Re: 100,000 TCP connections - kernel tuning advice wanted X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: Subhro List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Sep 2004 16:14:08 -0000 netstat -m please Regards S. On Fri, 3 Sep 2004 22:07:35 +1000, Simon Lai wrote: > > Hi all, > > As part of a team, I am working on a TCP multiplexor using FreeBSD. On side A > we have 100,000 TCP connections accepting packets, which are multiplexed > onto a single TCP connection on Side B. Packets going B->A are > demultiplexed in the reverse way. Info - > > - freebsd version is 5.2-RELEASE. Kernel has been recompiled to > use DEVICE_POLLING and unused devices removed. The > HZ parameter has been varied through 1000,2000,4000 but this > does not significantly alter our results. We have also played with > the idle and trap sysctl's for polling. > - our network card is an Intel EtherExpress Pro, running at 100Mbits > - UDP is not an option for us > - Average payload size is 50-100 bytes. The payload is preceeded > by a 32 bit value, which is the size of the payload, so reading > is a matter of grabbing the size, allocating a buffer and then > doing the read. Minimal processing is done on the packet. > - We are using our own specialized memory management. We use writev and > readv whereever possible. > - socket buffers have been increased to 1MB on the B side, but are the > default size on side A. > - we are using kevent/kqueue - this task would be impossible without them > - our current test box has 1.5GB RAM and a 1GHZ Athlon CPU. While we might > go for a faster CPU, we would like to keep within our current RAM constraints. > - Side A is connected to a test client, which has 20% idle time. > - Side B is connected via a switch to another test box, which just echos the > packets back for testing purposes. It has significant idle time. > - Our current rough measurements, using top, show 30% user time, and 60% > kernel time, when this app is running. This multiplexing app is the only > app running on the machine. The machine is CPU bound - the multiplexing > requires no disk I/O. > > Currently we are getting 4000-6000 packets/sec unidirectional throughput, > depending upon the mix of packet types/sizes. This goes up to > 5000-7000 packets/sec for 50,000 connections. > > We are seeking advice on what kernel tunables we can tweak to improve > packet throughput. Constants are TCP, 100,000 connections, 50-100 byte > packet sizes. > > All help appreciated. > > Regs > > Simon > > _______________________________________________ > freebsd-questions@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freebsd.org" > -- Subhro Sankha Kar School of Information Technology Block AQ-13/1 Sector V ZIP 700091 India