From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 18:56:11 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 486811065684 for ; Tue, 1 Jul 2008 18:56:11 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from atlas.gtcomm.net (atlas.gtcomm.net [67.215.15.242]) by mx1.freebsd.org (Postfix) with ESMTP id 06BB18FC1C for ; Tue, 1 Jul 2008 18:56:10 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from c-76-108-179-28.hsd1.fl.comcast.net ([76.108.179.28] helo=[192.168.1.6]) by atlas.gtcomm.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1KDkxz-0004Hl-MJ for freebsd-net@freebsd.org; Tue, 01 Jul 2008 14:52:39 -0400 Message-ID: <486A7E45.3030902@gtcomm.net> Date: Tue, 01 Jul 2008 14:58:13 -0400 From: Paul User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: FreeBSD Net References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> In-Reply-To: <4869B025.9080006@gtcomm.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 18:56:11 -0000 I can't reproduce the 580kpps maximum that I saw when I first compiled for some reason, I don't understand, the max I get even with ULE and preemption is now about 530 and it dips to 480 a lot.. The first time I tried it it was at 580 and dipped to 520...what the?.. (kernel config attached at end) * noticed that SOMETIMES the em0 taskq jumps around cpus and doesn't use 100% of any one cpu * noticed that the netstat packets per second rate varies explicitly with the CPU usage of em0 taskq (top output with ULE/PREEMPTION compiled in): PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 10 root 171 ki31 0K 16K RUN 3 64:12 94.09% idle: cpu3 36 root -68 - 0K 16K CPU1 1 5:43 89.75% em0 taskq 13 root 171 ki31 0K 16K CPU0 0 63:21 87.30% idle: cpu0 12 root 171 ki31 0K 16K RUN 1 62:44 66.75% idle: cpu1 11 root 171 ki31 0K 16K CPU2 2 62:17 56.49% idle: cpu2 39 root -68 - 0K 16K - 0 0:54 10.64% em3 taskq this is about 480-500kpps rate......... now I wait a minute and PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 10 root 171 ki31 0K 16K CPU3 3 64:56 100.00% idle: cpu3 36 root -68 - 0K 16K CPU2 2 6:21 94.14% em0 taskq 13 root 171 ki31 0K 16K RUN 0 63:55 80.18% idle: cpu0 11 root 171 ki31 0K 16K RUN 2 62:48 67.38% idle: cpu2 12 root 171 ki31 0K 16K CPU1 1 63:04 58.40% idle: cpu1 39 root -68 - 0K 16K - 1 1:00 10.21% em3 taskq 530kpps rate....... drops to 85%.. 480kpps rate goes back up to 95% 530kpps it keeps flopping like this........... none of the CPUs are 100% use and none of the cpus add up , like the cpu time of em0 taskq is 94% so one of the cpus should be 6% idle but it's not. This is with ULE/PREEMPTION.. I see different behavior without preemption and with 4bsd.. and I also see different behavior depending on the time of day lol :) Figure that one out I'll post back without preemption and with 4bsd in a min then i'll move on to the 32 bit platform tests