From owner-freebsd-current@FreeBSD.ORG Mon Sep 27 08:52:22 2010 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5A319106564A for ; Mon, 27 Sep 2010 08:52:22 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 335B28FC21 for ; Mon, 27 Sep 2010 08:52:22 +0000 (UTC) Received: from fledge.watson.org (fledge.watson.org [65.122.17.41]) by cyrus.watson.org (Postfix) with ESMTPS id CC56E46B1A; Mon, 27 Sep 2010 04:52:21 -0400 (EDT) Date: Mon, 27 Sep 2010 09:52:21 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Artemiev Igor In-Reply-To: <20100926235313.GA4848@two.kliksys.ru> Message-ID: References: <20100926235313.GA4848@two.kliksys.ru> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-current@freebsd.org Subject: Re: netisr software flowid X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Sep 2010 08:52:22 -0000 On Mon, 27 Sep 2010, Artemiev Igor wrote: > What is the status for software flowid calculation? I found the old netisr2 > patch[1] from Robert Watson and took from there code for setting flowid in > tcp_input with some changes[2]. It work for me very well (8.1-stable) - now > the server can handle not transit traffic without drops up to 118Kpps 60MB/s > incoming and up to 107Kpps 50MB/s outgoing, netisr dispatch packets via > three threads by round-robin: Hi Artemiev: I have a large outstanding patch set in Perforce that goes quite a long way further, implementing the RSS model found in many network cards and aligning OS hash tables for connection lookup with RSS. Where the RSS hash is made available by the driver, the patches are also able to implement link-layer dispatch. They largely eliminate the possibility of cache line contention in the TCP/IP input path (as long as the driver also avoids cache line contention) on multi-queue cards. One reason I haven't merged the earlier patch is that many high-performance 10gbps (and even 1gbps) cards now support multiple input queues in hardware, meaning that they have already done the work distribution by the time the packets get to the OS. This makes the work distribution choice quite a bit harder: has a packet already been adequately balanced, or is further rebalancing required -- and of so, an equal distribution as selected in that patch might not generate well-balanced CPU load. Using just the RSS hash to distribute work, and single-queue input, I am able to get doubled end-host TCP performance with highly concurrent connections at 10gbps, which is a useful result. I have high on my todo list to get the patch you referenced into the mix as well and see how much the software distrbiution hurts/helps... Since you've done some measurement, what was the throughput on that system without the patch applied, and how many cores? Robert > > 12 root -44 - 0K 336K CPU2 2 18:43 56.15% {swi1: netisr 2} > 12 root -44 - 0K 336K RUN 3 18:41 54.49% {swi1: netisr 3} > 12 root -44 - 0K 336K CPU0 0 18:39 50.39% {swi1: netisr 0} > 12 root -68 - 0K 336K WAIT 1 8:01 18.07% {irq256: bge0} > > So, what the reason to exclude this code from final version? > > [1] http://www.watson.org/~robert/freebsd/netperf/20090523-netisr2.diff > [2] http://gate.kliksys.ru/~ai/software_flowid.diff > _______________________________________________ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" >