From owner-freebsd-current@FreeBSD.ORG  Mon Sep 27 15:33:52 2010
Return-Path: <owner-freebsd-current@FreeBSD.ORG>
Delivered-To: freebsd-current@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id CE7D9106564A
	for <freebsd-current@freebsd.org>; Mon, 27 Sep 2010 15:33:52 +0000 (UTC)
	(envelope-from ai@kliksys.ru)
Received: from gate.kliksys.ru (gate.kliksys.ru [78.110.241.113])
	by mx1.freebsd.org (Postfix) with ESMTP id 88B798FC0A
	for <freebsd-current@freebsd.org>; Mon, 27 Sep 2010 15:33:52 +0000 (UTC)
Received: from [192.168.0.204] (helo=two.kliksys.ru)
	by gate.kliksys.ru with esmtp (Exim 4.71 (FreeBSD))
	(envelope-from <ai@kliksys.ru>) id 1P0FiB-000H76-9F
	for freebsd-current@freebsd.org; Mon, 27 Sep 2010 19:33:51 +0400
Date: Mon, 27 Sep 2010 19:35:55 +0400
From: Artemiev Igor <ai@kliksys.ru>
To: freebsd-current@freebsd.org
Message-ID: <20100927153555.GA9200@two.kliksys.ru>
References: <20100926235313.GA4848@two.kliksys.ru>
	<alpine.BSF.2.00.1009270947110.82524@fledge.watson.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <alpine.BSF.2.00.1009270947110.82524@fledge.watson.org>
User-Agent: Mutt/1.5.20 (2009-06-14)
X-Spam_score: 0.0
Subject: Re: netisr software flowid
X-BeenThere: freebsd-current@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Discussions about the use of FreeBSD-current
	<freebsd-current.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-current>, 
	<mailto:freebsd-current-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-current>
List-Post: <mailto:freebsd-current@freebsd.org>
List-Help: <mailto:freebsd-current-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-current>,
	<mailto:freebsd-current-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Mon, 27 Sep 2010 15:33:52 -0000

On Mon, Sep 27, 2010 at 09:52:21AM +0100, Robert Watson wrote:
> One reason I haven't merged the earlier patch is that many high-performance 
> 10gbps (and even 1gbps) cards now support multiple input queues in hardware, 
> meaning that they have already done the work distribution by the time the 
> packets get to the OS.  This makes the work distribution choice quite a bit 
> harder: has a packet already been adequately balanced, or is further 
> rebalancing required -- and of so, an equal distribution as selected in that 
> patch might not generate well-balanced CPU load.
> 
> Using just the RSS hash to distribute work, and single-queue input, I am able 
> to get doubled end-host TCP performance with highly concurrent connections at 
> 10gbps, which is a useful result.  I have high on my todo list to get the 
> patch you referenced into the mix as well and see how much the software 
> distrbiution hurts/helps...
Thanks for explanation.

> Since you've done some measurement, what was the throughput on that system 
> without the patch applied, and how many cores?
The server has four cores. Topology:
<groups>
 <group level="1" cache-level="0">
  <cpu count="4" mask="0xf">0, 1, 2, 3</cpu>
  <children>
   <group level="3" cache-level="2">
    <cpu count="4" mask="0xf">0, 1, 2, 3</cpu>
   </group>
  </children>
 </group>
</groups>

Without patch i have only one netisr thread
utilization with 100% cpu load and ~90% packets drop at max 80-90Kpps. The
throughput oscillated from 2MB/s to 30MB/s.

Cores 0,2,3 - netisr with cpu binding
Core 1 - irq256 (bge0) bind via cpuset(1) 

P.S.: bge(4) patched for agressive interrupt moderation. Without this i have
11K+ int/sec and ~99% cpu usage only in the interrupt handling.