From owner-freebsd-net@FreeBSD.ORG Wed Oct 29 08:53:31 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6EDF21065676; Wed, 29 Oct 2008 08:53:31 +0000 (UTC) (envelope-from gizmen@blurp.pl) Received: from albion.azs.pwr.wroc.pl (albion.azs.pwr.wroc.pl [156.17.17.145]) by mx1.freebsd.org (Postfix) with ESMTP id 2F7338FC12; Wed, 29 Oct 2008 08:53:30 +0000 (UTC) (envelope-from gizmen@blurp.pl) Received: from [10.8.1.27] (unknown [212.127.90.234]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by albion.azs.pwr.wroc.pl (Postfix) with ESMTPSA id C57C2116F4; Wed, 29 Oct 2008 09:53:29 +0100 (CET) From: Bartosz Giza Organization: BLURP.pl To: freebsd-net@freebsd.org Date: Wed, 29 Oct 2008 09:53:28 +0100 User-Agent: KMail/1.9.7 References: <1225203780.00029971.1225190402@10.7.7.3> <200810281613.13719.gizmen@blurp.pl> <490755A3.4050903@FreeBSD.org> In-Reply-To: <490755A3.4050903@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Message-Id: <200810290953.28237.gizmen@blurp.pl> Cc: Alexander Motin Subject: Re: two NIC on 2 core system (scheduling problem) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Oct 2008 08:53:31 -0000 Tuesday 28 of October 2008 19:10:43 Alexander Motin napisa=C5=82(a): > Bartosz Giza wrote: > >> The CPU time you see there includes much more then just a card > >> handling itself. It also includes CPU time of the most parts of > >> network stack used to process received packet. So if you have NAT, big > >> firewall, netgraph or any other CPU-hungry actions done with packets > >> incoming via em0 you will see such results. > >> Even more interesting is that if bge0 or fxp0 cards will require much > >> CPU time to send packet, this time will also be accounted to em0 > >> process. I have checked this and you are right. When i turned off ipfw; taskq proces= s=20 started to use less cpu. But still what is strange why processing from=20 other cards are counted in em0 taskq ? This is quite strange and in that=20 way em0 taskq process is using more cpu on one of the cores. So what i=20 think the best would be to have only em NICs because processing of the=20 packets would be splitted to those taskq processes is that right ? > > WOW that is weird. Yes i have quite big ipfw firewall and also couple > > of rules from pf. So you are saying that whole overhead from firewall > > is counted to this em taskq process. This is really strange for > > somebody who don't know about this. > > So what in case if i would have two em nic's. How would then overhead > > from firewalls be counted ? Splited to two taskq processes ? > > And another really weird thing is that if i have other card their > > processing are counted to tasq process of different NIC. Why is this > > done in such a way ? > > There is no dedicated processes in system to handle routing, firewall, > netgraph, etc. Many processes would lead to multiple context switches, > bigger latencies and reduced performance. Instead most parts of network > stack implemented in direct call fashion. So NIC receive interrupt > initiates packet handling by doing stack function call, that function > calls another and so on until packet will be completely processed and > transmitted or queued. There is not so many exception from this rule, > for example, dummynet which schedules part of activity to the timer > events. So sometimes you still can see some activity from the swi:net, > dummynet or some other threads. It also tells not so much about who > really consumed the CPU, but more about who initiated that CPU > consumption. Ok, good to know. But how is counted firewall overhead when i would have=20 only bge cards. They don't use taskq so i assume i would see this as system= =20 usage correct ? > In case of two em NICs em0 thread will mostly show load produced by > em0->em1 traffic processing and em1 - load produced by em1->em0 traffic.