From owner-freebsd-net@FreeBSD.ORG Mon Jul 7 10:00:22 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 84F2D106567E for ; Mon, 7 Jul 2008 10:00:22 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail09.syd.optusnet.com.au (mail09.syd.optusnet.com.au [211.29.132.190]) by mx1.freebsd.org (Postfix) with ESMTP id 141748FC13 for ; Mon, 7 Jul 2008 10:00:21 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from besplex.bde.org (c220-239-252-11.carlnfd3.nsw.optusnet.com.au [220.239.252.11]) by mail09.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id m67A0GaM025391 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 7 Jul 2008 20:00:18 +1000 Date: Mon, 7 Jul 2008 20:00:16 +1000 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Andre Oppermann In-Reply-To: <4871DB8E.5070903@freebsd.org> Message-ID: <20080707191918.B4703@besplex.bde.org> References: <4867420D.7090406@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <486A7E45.3030902@gtcomm.net> <486A8F24.5010000@gtcomm.net> <486A9A0E.6060308@elischer.org> <486B41D5.3060609@gtcomm.net> <486B4F11.6040906@gtcomm.net> <486BC7F5.5070604@gtcomm.net> <20080703160540.W6369@delplex.bde.org> <486C7F93.7010308@gtcomm.net> <20080703195521.O6973@delplex.bde.org> <486D35A0.4000302@gtcomm.net> <486DF1A3.9000409@gtcomm.net> <486E65E6.3060301@gtcomm.net> <4871DB8E.5070903@freebsd.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: FreeBSD Net , Ingo Flaschberger , Paul Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Jul 2008 10:00:22 -0000 On Mon, 7 Jul 2008, Andre Oppermann wrote: > Ingo Flaschberger wrote: >> I don't think you will be able to route 64byte packets at 1gbit wirespeed >> (2Mpps) with a current x86 platform. > > You have to take inter-frame gap and other overheads too. That gives > about 1.244Mpps max on a 1GigE interface. What are the other overheads? I calculate 1.644Mpps counting the inter-frame gap, with 64-byte packets and 64-header_size payloads. If the 64 bytes is for the payload, then the max is much lower. >> I hoped to reach 1Mpps with the hardware I mentioned some mails before, but >> 2Mpps is far far away. >> Currently I get 160kpps via pci-32mbit-33mhz-1,2ghz mobile pentium. > > This is more or less expected. PCI32 is not able to sustain high > packet rates. The bus setup times kill the speed. For larger packets > the ratio gets much better and some reasonable throughput can be achieved. I get about 640 kpps without forwarding (sendto: slightly faster; recvfrom: slightly slower) on a 2.2GHz A64. Underclocking the memory from 200MHz to 100MHz only reduces the speed by about 10%, while not overclocking the CPU by 10% reduces the speed by the same 10%, so the system is apparently still mainly CPU-bound. > NetFPGA doesn't have enough TCAM space to be useful for real routing > (as in Internet sized routing table). The trick many embedded networking > CPUs use is cache prefetching that is integrated with the network > controller. The first 64-128bytes of every packet are transferred > automatically into the L2 cache by the hardware. This allows relatively > slow CPUs (700 MHz Broadcom BCM1250 in Cisco NPE-G1 or 1.67-GHz Freescale > 7448 in NPE-G2) to get more than 1Mpps. Until something like this is > possible on Intel or AMD x86 CPUs we have a ceiling limited by RAM speed. Does using fa$ter memory (speed and/or latency) help here? 64 bytes is so small that latency may be more of a problem, especially without a prefetch. Bruce