From owner-freebsd-net@FreeBSD.ORG  Fri Apr 20 09:00:16 2012
Return-Path: <owner-freebsd-net@FreeBSD.ORG>
Delivered-To: net@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id CDEB2106566C
	for <net@freebsd.org>; Fri, 20 Apr 2012 09:00:16 +0000 (UTC)
	(envelope-from andre@freebsd.org)
Received: from c00l3r.networx.ch (c00l3r.networx.ch [62.48.2.2])
	by mx1.freebsd.org (Postfix) with ESMTP id 9D6338FC21
	for <net@freebsd.org>; Fri, 20 Apr 2012 09:00:15 +0000 (UTC)
Received: (qmail 18337 invoked from network); 20 Apr 2012 08:55:21 -0000
Received: from unknown (HELO [62.48.0.94]) ([62.48.0.94])
	(envelope-sender <andre@freebsd.org>)
	by c00l3r.networx.ch (qmail-ldap-1.03) with SMTP
	for <melifaro@FreeBSD.org>; 20 Apr 2012 08:55:21 -0000
Message-ID: <4F9125CF.8090201@freebsd.org>
Date: Fri, 20 Apr 2012 11:01:03 +0200
From: Andre Oppermann <andre@freebsd.org>
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64;
	rv:11.0) Gecko/20120327 Thunderbird/11.0.1
MIME-Version: 1.0
To: "Alexander V. Chernikov" <melifaro@FreeBSD.org>
References: <20120419133018.GA91364@onelab2.iet.unipi.it>
	<4F907011.9080602@freebsd.org>
	<20120419204622.GA94904@onelab2.iet.unipi.it>
	<CAHM0Q_M4wcEiWGkjWxE1OjLeziQN0vM+4_EYS_WComZ6=j5xhA@mail.gmail.com>
	<4F907FB4.3080400@freebsd.org> <4F911DCD.30001@FreeBSD.org>
In-Reply-To: <4F911DCD.30001@FreeBSD.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Cc: current@freebsd.org, Luigi Rizzo <rizzo@iet.unipi.it>,
	"K. Macy" <kmacy@freebsd.org>, net@freebsd.org
Subject: Re: Some performance measurements on the FreeBSD network stack
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
	<mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net>
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
	<mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Fri, 20 Apr 2012 09:00:16 -0000

On 20.04.2012 10:26, Alexander V. Chernikov wrote:
> On 20.04.2012 01:12, Andre Oppermann wrote:
>> On 19.04.2012 22:34, K. Macy wrote:
>>> If the number of peers is bounded then you can use the flowtable. Max
>>> PPS is much higher bypassing routing lookup. However, it doesn't scale
 >
>  From my experience, turning fastfwd on gives ~20-30% performance
> increase (10G forwarding with firewalling, 1.4MPPS). ip_forward() uses 2
> lookups (ip_rtaddr + ip_output) vs 1 ip_fastfwd().

Another difference is the packet copy the normal forwarding path
does to be able to send a ICMP redirect message if the packet is
forwarded to a different gateway on the same LAN.  fastforward
doesn't do that.

> The worst current problem IMHO is number of locks packet have to
> traverse, not number of lookups.

Agreed.  Actually the locking in itself is not the problem.  It's
the side effects of cache line dirtying/bouncing and contention.
However in the great majority of the cases the data protected by
the lock is only read, not modified making a 'full' lock expensive.

-- 
Andre