From owner-freebsd-net@FreeBSD.ORG  Mon Jul  7 11:32:01 2008
Return-Path: <owner-freebsd-net@FreeBSD.ORG>
Delivered-To: freebsd-net@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 73155106567E;
	Mon,  7 Jul 2008 11:32:01 +0000 (UTC)
	(envelope-from kris@FreeBSD.org)
Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28])
	by mx1.freebsd.org (Postfix) with ESMTP id 8E5198FC18;
	Mon,  7 Jul 2008 11:31:59 +0000 (UTC)
	(envelope-from kris@FreeBSD.org)
Message-ID: <4871FEAF.1060501@FreeBSD.org>
Date: Mon, 07 Jul 2008 13:31:59 +0200
From: Kris Kennaway <kris@FreeBSD.org>
User-Agent: Thunderbird 2.0.0.14 (Macintosh/20080421)
MIME-Version: 1.0
To: Andre Oppermann <andre@freebsd.org>
References: <4867420D.7090406@gtcomm.net>	<4869B025.9080006@gtcomm.net><486A7E45.3030902@gtcomm.net>	<486A8F24.5010000@gtcomm.net><486A9A0E.6060308@elischer.org>	<486B41D5.3060609@gtcomm.net><alpine.LFD.1.10.0807021052041.557@filebunker.xip.at><486B4F11.6040906@gtcomm.net><alpine.LFD.1.10.0807021155280.557@filebunker.xip.at><486BC7F5.5070604@gtcomm.net><20080703160540.W6369@delplex.bde.org><486C7F93.7010308@gtcomm.net><20080703195521.O6973@delplex.bde.org><486D35A0.4000302@gtcomm.net><alpine.LFD.1.10.0807041106591.19613@filebunker.xip.at><486DF1A3.9000409@gtcomm.net><alpine.LFD.1.10.0807041303490.20760@filebunker.xip.at><486E65E6.3060301@gtcomm.net>	<alpine.LFD.1.10.0807052356130.2145@filebunker.xip.at>	<2d3001c8def1$f4309b90$020b000a@bartwrkstxp>	<486FFF70.3090402@gtcomm.net>	<20080706132148.E44832@fledge.watson.org>
	<4871D81B.8070507@freebsd.org>
In-Reply-To: <4871D81B.8070507@freebsd.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Cc: FreeBSD Net <freebsd-net@freebsd.org>, Robert Watson <rwatson@FreeBSD.org>,
	Bart Van Kerckhove <bart@it-ss.be>,
	Ingo Flaschberger <if@xip.at>, Paul <paul@gtcomm.net>
Subject: Re: Freebsd IP Forwarding performance (question,
 and some info) [7-stable, current, em, smp]
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
	<mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net>
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
	<mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Mon, 07 Jul 2008 11:32:01 -0000

Andre Oppermann wrote:
> Robert Watson wrote:
>> Experience suggests that forwarding workloads see significant lock 
>> contention in the routing and transmit queue code.  The former needs 
>> some kernel hacking to address in order to improve parallelism for 
>> routing lookups.  The latter is harder to address given the hardware 
>> you're using: modern 10gbps cards frequently offer multiple transmit 
>> queues that can be used independently (which our cxgb driver 
>> supports), but 1gbps cards generally don't.
> 
> Actually the routing code is not contended.  The workload in router
> is mostly serialized without much opportunity for contention.  With
> many interfaces and any-to-any traffic patterns it may get some
> contention.  The locking overhead per packet is always there and has
> some impact though.
> 

Actually contention from route locking is a major bottleneck even on 
packet generation from multiple CPUs on a single host.  It is becoming 
increasingly necessary that someone look into fixing this.

Kris