From owner-freebsd-net@FreeBSD.ORG  Fri Jul  4 18:01:23 2008
Return-Path: <owner-freebsd-net@FreeBSD.ORG>
Delivered-To: freebsd-net@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 641EA106564A
	for <freebsd-net@freebsd.org>; Fri,  4 Jul 2008 18:01:23 +0000 (UTC)
	(envelope-from paul@gtcomm.net)
Received: from atlas.gtcomm.net (atlas.gtcomm.net [67.215.15.242])
	by mx1.freebsd.org (Postfix) with ESMTP id 1B74E8FC0A
	for <freebsd-net@freebsd.org>; Fri,  4 Jul 2008 18:01:22 +0000 (UTC)
	(envelope-from paul@gtcomm.net)
Received: from c-76-108-179-28.hsd1.fl.comcast.net ([76.108.179.28]
	helo=[192.168.1.6])
	by atlas.gtcomm.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.67)
	(envelope-from <paul@gtcomm.net>)
	id 1KEpXE-0003H3-JC; Fri, 04 Jul 2008 13:57:28 -0400
Message-ID: <486E65E6.3060301@gtcomm.net>
Date: Fri, 04 Jul 2008 14:03:18 -0400
From: Paul <paul@gtcomm.net>
User-Agent: Thunderbird 2.0.0.14 (Windows/20080421)
MIME-Version: 1.0
To: Ingo Flaschberger <if@xip.at>
References: <4867420D.7090406@gtcomm.net>
	<alpine.LFD.1.10.0807010308320.19444@filebunker.xip.at>
	<486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net>
	<ea7b9c170806302005n2a66f592h2127f87a0ba2c6d2@mail.gmail.com>
	<20080701033117.GH83626@cdnetworks.co.kr>
	<ea7b9c170806302050p2a3a5480t29923a4ac2d7c852@mail.gmail.com>
	<4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net>
	<486A7E45.3030902@gtcomm.net> <486A8F24.5010000@gtcomm.net>
	<486A9A0E.6060308@elischer.org> <486B41D5.3060609@gtcomm.net>
	<alpine.LFD.1.10.0807021052041.557@filebunker.xip.at>
	<486B4F11.6040906@gtcomm.net>
	<alpine.LFD.1.10.0807021155280.557@filebunker.xip.at>
	<486BC7F5.5070604@gtcomm.net>
	<20080703160540.W6369@delplex.bde.org>
	<486C7F93.7010308@gtcomm.net>
	<20080703195521.O6973@delplex.bde.org>
	<486D35A0.4000302@gtcomm.net>
	<alpine.LFD.1.10.0807041106591.19613@filebunker.xip.at>
	<486DF1A3.9000409@gtcomm.net>
	<alpine.LFD.1.10.0807041303490.20760@filebunker.xip.at>
In-Reply-To: <alpine.LFD.1.10.0807041303490.20760@filebunker.xip.at>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Cc: FreeBSD Net <freebsd-net@freebsd.org>
Subject: Re: Freebsd IP Forwarding performance (question,
 and some info) [7-stable, current, em, smp]
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
	<mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net>
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
	<mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Fri, 04 Jul 2008 18:01:23 -0000

I tried all of this :/  still, 256/512 descriptors seem to work the best.
Happy to let you log into the machine and fiddle around if you want :)

Paul

Ingo Flaschberger wrote:
> Dear Paul,
>
>>>> what could cause this?
>>>
>>> *) kern.polling.idle_poll enabled?
>>> *) kern.polling.user_frac ?
>>> *) kern.polling.reg_frac ?
>>> *) kern.polling.burst_max ?
>>> *) kern.polling.each_burst ?
>>
>> I tried tons of different values for these and nothing made any 
>> significant difference.
>> Idle polling makes a difference, allows more pps, but still errors.
>> Without idle polling it seems PPS is limited to HZ * descriptors, or 
>> 1000 * 256 or 512
>> but 1000 * 1024 is the same as 512..  4000 * 256 or 2000 * 512 works 
>> but starts erroring 600kpps (SMP right now but it happens in UP too)
>
> I have patched src/sys/kern/kern_poll.c to support higher burst_max 
> values:
> #define MAX_POLL_BURST_MAX      10000
>
> When setting kern.polling.burst_max to higher values, the server reach 
> a point, where cpu-usage goes up without load, so try to keep below 
> this values. I also have set the network card to 4096 rx-"ram", to 
> have more room for late polls.
>
> Kind regards,
>     Ingo Flaschberger
>
>