From owner-freebsd-net@FreeBSD.ORG Fri Jul 4 06:28:56 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EAE44106564A for ; Fri, 4 Jul 2008 06:28:56 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail05.syd.optusnet.com.au (mail05.syd.optusnet.com.au [211.29.132.186]) by mx1.freebsd.org (Postfix) with ESMTP id 7949B8FC1B for ; Fri, 4 Jul 2008 06:28:56 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from c220-239-252-11.carlnfd3.nsw.optusnet.com.au (c220-239-252-11.carlnfd3.nsw.optusnet.com.au [220.239.252.11]) by mail05.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id m646SltI019351 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 4 Jul 2008 16:28:51 +1000 Date: Fri, 4 Jul 2008 16:28:47 +1000 (EST) From: Bruce Evans X-X-Sender: bde@delplex.bde.org To: Paul In-Reply-To: <486DAD0D.8090604@gtcomm.net> Message-ID: <20080704160950.Y9864@delplex.bde.org> References: <4867420D.7090406@gtcomm.net> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <486A7E45.3030902@gtcomm.net> <486A8F24.5010000@gtcomm.net> <486A9A0E.6060308@elischer.org> <486B41D5.3060609@gtcomm.net> <486B4F11.6040906@gtcomm.net> <486BC7F5.5070604@gtcomm.net> <20080703160540.W6369@delplex.bde.org> <486C7F93.7010308@gtcomm.net> <20080703195521.O6973@delplex.bde. org> <486D35A0.4000302@gtcomm.net> <486DAD0D.8090604@gtcomm.net> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: FreeBSD Net , Ingo Flaschberger Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Jul 2008 06:28:57 -0000 On Fri, 4 Jul 2008, Paul wrote: > Numbers are maximum with near 100% cpu usage and some errors occuring, just > for testing. > FreeBSD 7.0-STABLE FreeBSD 7.0-STABLE #6: Thu Jul 3 19:32:38 CDT 2008 > root@foo:/usr/obj/usr/src/sys/ROUTER amd64 > CPU: Dual-Core AMD Opteron(tm) Processor 2222 (3015.47-MHz K8-class CPU) > NON-SMP KERNEL em driver, intel 82571EB NICs > fastforwarding on, isr.direct on, ULE, Preemption (NOTE: Interesting thing, > without preemption gets errors similar to polling) PREEMPTION is certainly needed with UP. Without it, interrupts don't actually work (to work, they need to preempt the running thread, but they often (usually?) don't do that). Then with UP, there is a good chance that the interrupt thread doesn't get scheduled to run for a long time, but with SMP (especially with lots of CPUs) there is a good chance that another CPU gets scheduled to run the interrupt thread. em (unless misconfigured) doesn't have an interrupt thread; it uses a taskq which might take even longer to be scheduled than an interrupt thread. I use PREEMPTION with UP and !PREEMPTION with SMP. With polling, missed polls cause the same packet loss as not preempting. > I tried polling, and I tried the polling patch that was posted to the list > and both work but generate too many errors (missed packets). > Without polling the packet errors ONLY occur when the cpu is near 100% usage Polling should also only cause packet loss when the CPU is near 100% usage, but now transients of near 100% usually cause packet loss, while with interrupts it takes a transient of > 100% on the competing interrupt- driven resources to cause packet loss. Pleas trim quotes. Bruce