From owner-freebsd-performance@FreeBSD.ORG Fri Feb 1 11:49:39 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 64A3E16A420 for ; Fri, 1 Feb 2008 11:49:39 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: from blah.sun-fish.com (blah.sun-fish.com [217.18.249.150]) by mx1.freebsd.org (Postfix) with ESMTP id A492113C45D for ; Fri, 1 Feb 2008 11:49:38 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: by blah.sun-fish.com (Postfix, from userid 1002) id E03531B10EF4; Fri, 1 Feb 2008 12:49:36 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on blah.cmotd.com X-Spam-Level: X-Spam-Status: No, score=-10.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, J_CHICKENPOX_48 autolearn=no version=3.2.3 Received: from [10.1.1.2] (unknown [192.168.25.10]) by blah.sun-fish.com (Postfix) with ESMTP id BBF151B10EF0; Fri, 1 Feb 2008 12:49:33 +0100 (CET) Message-ID: <47A3074A.3040409@moneybookers.com> Date: Fri, 01 Feb 2008 13:49:30 +0200 From: Stefan Lambrev User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) MIME-Version: 1.0 To: gnn@freebsd.org References: <4794E6CC.1050107@moneybookers.com> <47A0B023.5020401@moneybookers.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.91.2/5631/Fri Feb 1 11:41:51 2008 on blah.cmotd.com X-Virus-Status: Clean Cc: freebsd-performance@freebsd.org Subject: Re: network performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Feb 2008 11:49:39 -0000 Greetings, gnn@freebsd.org wrote: > At Wed, 30 Jan 2008 19:13:07 +0200, > Stefan Lambrev wrote: > >> Greetings, >> >> After playing with many settings and testing various configuration, now >> I'm able to to receive on bridge more then 800,000 packets/s >> without errors, which is amazing! >> Unfortunately the server behind bridge can't handle more then 250,000 >> packets/s >> Please advise how I can increase those limits? >> Is is possible? >> >> The servers are with 82573E Gigabit Ethernet Controller (quad port) >> So far I tried with lagg and ng_fec, but with them I see more problems >> then benefits :) >> Tried polling with kern.polling.user_frac from 5 to 95, >> different HZ, but nothing helped. >> > > Increase the size of your socket buffers. > > Increase the amount of mbufs in the system. > > Best, > George > Here is what I put in my sysctl.conf: kern.random.sys.harvest.ethernet=0 kern.ipc.nmbclusters=262144 kern.ipc.maxsockbuf=2097152 kern.ipc.maxsockets=98624 kern.ipc.somaxconn=1024 and in /boot/loader.conf: vm.kmem_size="1024M" kern.hz="500" this is from netstat -m 516/774/1290 mbufs in use (current/cache/total) 513/411/924/262144 mbuf clusters in use (current/cache/total/max) 513/383 mbuf+clusters out of packet secondary zone in use (current/cache) 0/2/2/12800 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) 1155K/1023K/2178K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/0/0 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 0 calls to protocol drain routines But still netstat -w1 -I em0 shows: input (em0) output packets errs bytes packets errs bytes colls 273877 113313 16432620 254270 0 14746500 0 273397 109905 16403820 253946 0 14728810 0 273945 113337 16436700 254285 0 14750560 0 What bothers me is the output of top -S: PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 22 root 1 -68 - 0K 16K CPU1 1 12:11 100.00% em0 taskq 11 root 1 171 ki31 0K 16K RUN 0 21:56 99.17% idle: cpu0 10 root 1 171 ki31 0K 16K RUN 1 9:16 0.00% idle: cpu1 14 root 1 -44 - 0K 16K WAIT 0 0:07 0.00% swi1: net and vmstat: procs memory page disk faults cpu r b w avm fre flt re pi po fr sr ad4 in sy cs us sy id 1 0 0 67088 1939700 0 0 0 0 0 0 0 2759 119 1325 0 50 50 0 0 0 67088 1939700 0 0 0 0 0 0 0 2760 127 1178 0 50 50 0 0 0 67088 1939700 0 0 0 0 0 0 0 2761 120 1269 0 50 50 What I'm missing?