From owner-freebsd-net@FreeBSD.ORG Mon Jul 7 12:44:41 2008 Return-Path: Delivered-To: freebsd-net@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A33911065672; Mon, 7 Jul 2008 12:44:41 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.freebsd.org (Postfix) with ESMTP id 3DEB48FC47; Mon, 7 Jul 2008 12:44:40 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 583C546C65; Mon, 7 Jul 2008 08:44:40 -0400 (EDT) Date: Mon, 7 Jul 2008 13:44:40 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Bruce Evans In-Reply-To: <20080707213356.G7572@besplex.bde.org> Message-ID: <20080707134036.S63144@fledge.watson.org> References: <4867420D.7090406@gtcomm.net> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <486A7E45.3030902@gtcomm.net> <486A8F24.5010000@gtcomm.net> <486A9A0E.6060308@elischer.org> <486B41D5.3060609@gtcomm.net> <486B4F11.6040906@gtcomm.net> <486BC7F5.5070604@gtcomm.net> <20080703160540.W6369@delplex.bde.org> <486C7F93.7010308@gtcomm.net> <20080703195521.O6973@delplex.bde.org> <486D35A0.4000302@gtcomm.net> <486DF1A3.9000409@gtcomm.net> <486E65E6.3060301@gtcomm.net> <4871DB8E.5070903@freebsd.org> <20080707191918.B4703@besplex.bde.org> <4871FB66.1060406@freebsd.org> <20080707213356.G7572@besplex.bde.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: FreeBSD Net , Andre Oppermann , Ingo Flaschberger , Paul Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Jul 2008 12:44:41 -0000 On Mon, 7 Jul 2008, Bruce Evans wrote: > I use low-end memory, but on the machine that does 640 kpps it somehow has > latency almost 4 times as low as on new FreeBSD cluster machines (~42 nsec > instead of ~150). perfmon (fixed for AXP and A64) and hwpmc report an > average of 11 k8-dc-misses per sendto() while sending via bge at 640 kpps. > 11 * 42 accounts for 442 nsec out of the 1562 per packet at this rate. 11 * > 150 = 1650 would probably make this rate unachievable despite the system > having 20 times as much CPU and bus. Since you're doing fine-grained performance measurements of a code path that interests me a lot, could you compare the cost per-send on UDP for the following four cases: (1) sendto() to a specific address and port on a socket that has been bound to INADDR_ANY and a specific port. (2) sendto() on a specific address and port on a socket that has been bound to a specific IP address (not INADDR_ANY) and a specific port. (3) send() on a socket that has been connect()'d to a specific IP address and a specific port, and bound to INADDR_ANY and a specific port. (4) send() on a socket that has been connect()'d to a specific IP address and a specific port, and bound to a specific IP address (not INADDR_ANY) and a specific port. The last of these should really be quite a bit faster than the first of these, but I'd be interested in seeing specific measurements for each if that's possible! Thanks, Robert N M Watson Computer Laboratory University of Cambridge