Date: Mon, 07 Jul 2008 14:44:50 -0400 From: Paul <paul@gtcomm.net> To: Andre Oppermann <andre@freebsd.org> Cc: FreeBSD Net <freebsd-net@freebsd.org> Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] Message-ID: <48726422.7050703@gtcomm.net> In-Reply-To: <4871E85C.8090907@freebsd.org> References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <alpine.LFD.1.10.0807010257570.19444@filebunker.xip.at> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <alpine.LFD.1.10.0807010308320.19444@filebunker.xip.at> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <ea7b9c170806302005n2a66f592h2127f87a0ba2c6d2@mail.gmail.com> <20080701033117.GH83626@cdnetworks.co.kr> <ea7b9c170806302050p2a3a5480t29923a4ac2d7c852@mail.gmail.com> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <486A7E45.3030902@gtcomm.net> <486A8F24.5010000@gtcomm.net> <486A9A0E.6060308@elischer.org> <486B41D5.3060609@gtcomm.net> <4871E85C.8090907@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
> one that will later on handle the taskqueue to process the packets. > That adds overhead. Ideally the interrupt for each network interface > is bound to exactly one pre-determined CPU and the taskqueue is bound > to the same CPU. That way the overhead for interrupt and taskqueue > scheduling can be kept at a minimum. Most of the infrastructure to > do this binding already exists in the kernel but is not yet exposed > to the outside for us to make use of it. I'm also not sure if the > ULE scheduler skips the more global locks when interrupt and the > thread are on the same CPU. > > Distributing the interrupts and taskqueues among the available CPUs > gives concurrent forwarding with bi- or multi-directional traffic. > All incoming traffic from any particular interface is still serialized > though. > I used etherchannel to distribute incoming packets over 3 separate cpus evenly but the output was on one interface.. What I got was less performance than with one cpu and all three cpus were close to 100% utilizied. em0,em1,em2 were all receiving packets and sending them out em3. The machine had 4 cpus in it. em3 taskq was low cpu usage and em0,1,2 were using cpu0,1,2(for example) almost fully used. With all that cpu power being used and I got less performance than with 1 cpu :/ Obviously in SMP there is a big issue somewhere. Also my 82571 NIC supports multiple received queues and multiple transmit queues so why hasn't anyone written the driver to support this? It's not a 10gb card and it still supports it and it's widely available and not too expensive either. The new 82575/6 chips support even more queues and the two port version will be out this month and the 4 port in october (PCI-E cards). Motherboards are already shipping with the 82576.. (82571 supports 2x/2x 575/6 support 4x/4x) Paul
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?48726422.7050703>