From owner-freebsd-net@FreeBSD.ORG Mon Jul 7 18:43:00 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4FB95106568A; Mon, 7 Jul 2008 18:42:58 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from atlas.gtcomm.net (atlas.gtcomm.net [67.215.15.242]) by mx1.freebsd.org (Postfix) with ESMTP id 2A3678FC1B; Mon, 7 Jul 2008 18:42:58 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from c-76-108-197-4.hsd1.fl.comcast.net ([76.108.197.4] helo=[192.168.1.6]) by atlas.gtcomm.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1KFvbw-0008H5-6L; Mon, 07 Jul 2008 14:38:52 -0400 Message-ID: <48726422.7050703@gtcomm.net> Date: Mon, 07 Jul 2008 14:44:50 -0400 From: Paul User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: Andre Oppermann References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <486A7E45.3030902@gtcomm.net> <486A8F24.5010000@gtcomm.net> <486A9A0E.6060308@elischer.org> <486B41D5.3060609@gtcomm.net> <4871E85C.8090907@freebsd.org> In-Reply-To: <4871E85C.8090907@freebsd.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Net Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Jul 2008 18:43:00 -0000 > one that will later on handle the taskqueue to process the packets. > That adds overhead. Ideally the interrupt for each network interface > is bound to exactly one pre-determined CPU and the taskqueue is bound > to the same CPU. That way the overhead for interrupt and taskqueue > scheduling can be kept at a minimum. Most of the infrastructure to > do this binding already exists in the kernel but is not yet exposed > to the outside for us to make use of it. I'm also not sure if the > ULE scheduler skips the more global locks when interrupt and the > thread are on the same CPU. > > Distributing the interrupts and taskqueues among the available CPUs > gives concurrent forwarding with bi- or multi-directional traffic. > All incoming traffic from any particular interface is still serialized > though. > I used etherchannel to distribute incoming packets over 3 separate cpus evenly but the output was on one interface.. What I got was less performance than with one cpu and all three cpus were close to 100% utilizied. em0,em1,em2 were all receiving packets and sending them out em3. The machine had 4 cpus in it. em3 taskq was low cpu usage and em0,1,2 were using cpu0,1,2(for example) almost fully used. With all that cpu power being used and I got less performance than with 1 cpu :/ Obviously in SMP there is a big issue somewhere. Also my 82571 NIC supports multiple received queues and multiple transmit queues so why hasn't anyone written the driver to support this? It's not a 10gb card and it still supports it and it's widely available and not too expensive either. The new 82575/6 chips support even more queues and the two port version will be out this month and the 4 port in october (PCI-E cards). Motherboards are already shipping with the 82576.. (82571 supports 2x/2x 575/6 support 4x/4x) Paul