From owner-freebsd-current@FreeBSD.ORG Mon Mar 5 18:52:24 2007 Return-Path: X-Original-To: freebsd-current@freebsd.org Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0558516A404; Mon, 5 Mar 2007 18:52:24 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from server.baldwin.cx (66-23-211-162.clients.speedfactory.net [66.23.211.162]) by mx1.freebsd.org (Postfix) with ESMTP id 7F23F13C4A6; Mon, 5 Mar 2007 18:52:18 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from localhost.corp.yahoo.com (john@localhost [127.0.0.1]) (authenticated bits=0) by server.baldwin.cx (8.13.8/8.13.8) with ESMTP id l25Iq9AE003961; Mon, 5 Mar 2007 13:52:10 -0500 (EST) (envelope-from jhb@freebsd.org) From: John Baldwin To: Robert Watson Date: Mon, 5 Mar 2007 13:52:28 -0500 User-Agent: KMail/1.9.1 References: <45E8276D.60105@freebsd.org> <17900.24574.751134.397740@grasshopper.cs.duke.edu> <20070305182755.S31701@fledge.watson.org> In-Reply-To: <20070305182755.S31701@fledge.watson.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200703051352.29939.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH authentication, not delayed by milter-greylist-2.0.2 (server.baldwin.cx [127.0.0.1]); Mon, 05 Mar 2007 13:52:10 -0500 (EST) X-Virus-Scanned: ClamAV 0.88.3/2735/Mon Mar 5 10:23:59 2007 on server.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-4.4 required=4.2 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.1.3 X-Spam-Checker-Version: SpamAssassin 3.1.3 (2006-06-01) on server.baldwin.cx Cc: freebsd-net@freebsd.org, freebsd-current@freebsd.org, Andre Oppermann , Andrew Gallatin , kmacy@freebsd.org Subject: Re: New optimized soreceive_stream() for TCP sockets, proof of concept X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Mar 2007 18:52:24 -0000 On Monday 05 March 2007 13:29, Robert Watson wrote: > On Mon, 5 Mar 2007, Andrew Gallatin wrote: > > > With the patch, we finally seem to be performance competative on the receive > > side with Linux x86_64 and Solaris/amd64 on this same hardware. Both of > > those OSes do much better (saturate the link with jumbos) when CPU affinity > > is used to bind the interrupt handler and netserver process to different > > cores on the same socket. I imagine FreeBSD may be able to do even better > > if it ever grows CPU affinity support for both interrupt handlers and > > processes. With the patch, it performs at least as well, if not better > > than, Solaris and Linux do without CPU affinity. > > I don't have numbers in front of me, and am currently packing for a trip to > Tokyo so won't find them before traveling, but my experience has been that > binding the ithread to a specific CPU is very helpful in improving receive > performance. You can slap a sched_bind(0) into the interrupt handler the > first time it runs and it should stick appropriately, and add a sysctl to > sched_bind() for a user process as a hack to test it out. > > John has a patch that pins interrupt threads, etc, not sure what the status of > that is. CC'd. Tested and around for over a year. Sent to people several times but no benchmarking has resulted. It lives in p4 in //depot/user/jhb/intr/... I've just regenerated the patch at http://www.FreeBSD.org/~jhb/patches/intr_bind.patch (same URL as the last N times I've posted it). I do now have access again to the sources for a simple intr_bind userland tool that lets you move assign the CPU for a given interrupt. There is currently no way to see which CPU an interrupt is assigned to outside of 'show intr' in ddb or scraping a verbose dmesg, but that can always be added later. What is there now is probably sufficient for testing. -- John Baldwin