From owner-freebsd-net@FreeBSD.ORG Fri Mar 3 11:44:12 2006 Return-Path: X-Original-To: freebsd-net@freebsd.org Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B440C16A422; Fri, 3 Mar 2006 11:44:12 +0000 (GMT) (envelope-from tbyte@otel.net) Received: from mail.otel.net (gw3.OTEL.net [212.36.8.151]) by mx1.FreeBSD.org (Postfix) with ESMTP id A8AEB43D49; Fri, 3 Mar 2006 11:44:11 +0000 (GMT) (envelope-from tbyte@otel.net) Received: from dragon.otel.net ([212.36.8.135]) by mail.otel.net with esmtp (Exim 4.60 (FreeBSD)) (envelope-from ) id 1FF8hd-0004jK-21; Fri, 03 Mar 2006 13:44:09 +0200 From: Iasen Kostov To: Robert Watson In-Reply-To: <20060303012652.A54458@fledge.watson.org> References: <1141306007.70735.16.camel@DraGoN.OTEL.net> <20060303012652.A54458@fledge.watson.org> Content-Type: text/plain Date: Fri, 03 Mar 2006 13:44:08 +0200 Message-Id: <1141386248.71876.39.camel@DraGoN.OTEL.net> Mime-Version: 1.0 X-Mailer: Evolution 2.4.2.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: FreeBSD Net Subject: Re: SMP NAT X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Mar 2006 11:44:12 -0000 On Fri, 2006-03-03 at 01:35 +0000, Robert Watson wrote: > On Thu, 2 Mar 2006, Iasen Kostov wrote: > > > I'm now using a MP system (dual opteron) to do NAT for about 1500 clients at > > once at speed above 200Mbit/sec full-duplex (e.g about 400Mbit/sec) and I'm > > using PF to do the NAT. Bad thing is that the second CPU is idle. As I can > > see from top - about 50% of the cpu is used by irq handler for the ethernet > > adapter (irq27: bge0 bge1 - I'm using only bge0 to route via VLANs) and > > about 30% by the network interrupt handler. I guess that the swi1:net is > > handling the NAT (via PF) and if swi1 and irq27 are in different handlers > > why they don't get executed on different CPUs (second CPU is 98% idle and > > top show that both handlers run on same CPU). Aren't both handlers in > > different kernel threads ? If they are not - is it possible to be in > > different threads on different CPUs ? > > In general, yes -- I frequently look at top -S and see the ithreads running on > different CPUs from each other. As you surmise, the hardware ithread is > handling the hardware interrupts up through link layer processing, and then > the netisr is doing the IP layer processing including NAT. On recent FreeBSD, > generally if a second CPU is idle, we will generally wake up the netisr on > another idle CPU. However, that's a property of the scheduler, and the > details of how that happens vary a bit by FreeBSD version. You don't include > information on which FreeBSD version you're using. It's also worth keeping in > mind that if you have idle CPU time on your first CPU even with both threads > going as fast as the hardware is driving, it's not necessarily "better" to be > running the two tasks on different CPUs, for reasons of memory caching -- > i.e., the second CPU won't have to cache miss and read the packets in from > memory a second time when it begins processing the mbufs previously brought > into memory on the first CPU by the interrupt handler. But if both threads overload 1 of the CPUs and second stays idle ? Performance probablly would be worse because interrput handler won't be able to handle more interrupts or swi1 won't be able to process the packets. > > So a few questions: > > (1) What version of FreeBSD are you running? > 6.0-STABLE #0: Thu Dec 8 21:19:54 UTC 2005 amd64 > (2) Is your network stack running MPSAFE? "sysctl debug.mpsafenet" will > return either 0 or 1. If you're running with certain network features, > such as the KAME IPSEC stack, you may be running with single processor > network stack. > Yep. debug.mpsafenet: 1 (this is the default I think). And no nothing there except PF's NAT. net.inet.ip.fastforwarding: 1 > (3) Are you using SCHED_4BSD (or rather, have you changed to SCHED_ULE)? SCHED_4BSD > > (4) Are you running with PREEMPTION compiled into the kernel? When a thread, > such as the netisr, is preempted by a hardware ithread, it won't > necessarily be bounced to the other CPU immediately. > PREEMPTION is compiled into the kernel. Regards