From owner-freebsd-net@FreeBSD.ORG Tue Apr 1 06:20:41 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3AB391065673; Tue, 1 Apr 2008 06:20:41 +0000 (UTC) (envelope-from root@mmu.edu.my) Received: from staff.cyber.mmu.edu.my (staff.cyber.mmu.edu.my [203.106.62.12]) by mx1.freebsd.org (Postfix) with ESMTP id 7A7D98FC13; Tue, 1 Apr 2008 06:20:40 +0000 (UTC) (envelope-from root@mmu.edu.my) Received: by staff.cyber.mmu.edu.my (Postfix, from userid 0) id 672074D5177; Tue, 1 Apr 2008 14:19:22 +0800 (MYT) Received: from mx2.freebsd.org (mx2.freebsd.org [69.147.83.53]) by mmu.edu.my (Postfix) with ESMTP id CC7DD55E4A3 for ; Sun, 30 Mar 2008 21:44:10 +0800 (MYT) Received: from hub.freebsd.org (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx2.freebsd.org (Postfix) with ESMTP id 4CF2F161893; Sun, 30 Mar 2008 13:27:36 +0000 (UTC) (envelope-from owner-freebsd-hackers@freebsd.org) Received: from hub.freebsd.org (localhost [127.0.0.1]) by hub.freebsd.org (Postfix) with ESMTP id 8B33010656D8; Sun, 30 Mar 2008 13:27:34 +0000 (UTC) (envelope-from owner-freebsd-hackers@freebsd.org) Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5CFBA106564A; Sun, 30 Mar 2008 13:27:23 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.freebsd.org (Postfix) with ESMTP id 3D59F8FC14; Sun, 30 Mar 2008 13:27:23 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id D68CE46B3B; Sun, 30 Mar 2008 09:27:22 -0400 (EDT) Date: Sun, 30 Mar 2008 14:27:22 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Alexander Motin In-Reply-To: <47EF77DE.6040200@FreeBSD.org> Message-ID: <20080330142332.Y5921@fledge.watson.org> References: <47EF4F18.502@FreeBSD.org> <20080330112846.Y5921@fledge.watson.org> <47EF77DE.6040200@FreeBSD.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Sender: owner-freebsd-hackers@freebsd.org Errors-To: owner-freebsd-hackers@freebsd.org Cc: freebsd-hackers@freebsd.org, FreeBSD Net Subject: Re: Multiple netgraph threads X-BeenThere: freebsd-net@freebsd.org List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Apr 2008 06:20:41 -0000 On Sun, 30 Mar 2008, Alexander Motin wrote: >> My initial leaning would be that we would like to avoid adding too many >> more threads that will do per-packet work, as that leads to excessive >> context switching. > > Netgraph uses queueing only as last resort, when direct call is not possible > due to locking or stack limitations. For example, while working with kernel > sockets (*upcall)() I have got many issues which make impossible to freely > use received data without queueing as upcall() caller holds some locks > leading to unpredicted LORs in socket/TCP/UDP code. In case of such forced > queueing, node becomes an independent data source which can be pinned to and > processed by whatever specialized thread or netisr, when it will be able to > do it more effectively. I guess my caution is that it does not necessarily follow from a design that allows for explicit parallelism that the implementation will use it well, and that any time context switchs are necessarily introduced, cost goes up. The move to direct dispatch from the ithread, despite reducing opportunities for parallelism, significantly increased performance for many local workloads. If we have a netisr thread, an ithread, and a netgraph thread, the potential context switch overhead is significant, even if we are doing a good job at batching work transfer between them. Often times, the way this behaves in practice is quite dependent on scheduling, and right now we have known defficiencies in this area, so give it a try on an SMP box and see what happens. Since you're a FreeBSD committer, you can sign up to use the netperf cluster, which might not be a bad idea if you don't have local access to a good SMP test setup. Robert N M Watson Computer Laboratory University of Cambridge _______________________________________________ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"