From owner-freebsd-net@FreeBSD.ORG Fri Feb 4 20:48:07 2005 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BCF0E16A4CE for ; Fri, 4 Feb 2005 20:48:07 +0000 (GMT) Received: from tigra.ip.net.ua (tigra.ip.net.ua [82.193.96.10]) by mx1.FreeBSD.org (Postfix) with ESMTP id D9D4543D1F for ; Fri, 4 Feb 2005 20:48:06 +0000 (GMT) (envelope-from ru@ip.net.ua) Received: from localhost (rocky.ip.net.ua [82.193.96.2]) by tigra.ip.net.ua (8.12.11/8.12.11) with ESMTP id j14Km5Mr045283; Fri, 4 Feb 2005 22:48:05 +0200 (EET) (envelope-from ru@ip.net.ua) Received: from tigra.ip.net.ua ([82.193.96.10]) by localhost (rocky.ipnet [82.193.96.2]) (amavisd-new, port 10024) with LMTP id 13080-15; Fri, 4 Feb 2005 22:48:04 +0200 (EET) Received: from heffalump.ip.net.ua (heffalump.ip.net.ua [82.193.96.213]) by tigra.ip.net.ua (8.12.11/8.12.11) with ESMTP id j14Km4vC045280 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 4 Feb 2005 22:48:04 +0200 (EET) (envelope-from ru@ip.net.ua) Received: (from ru@localhost) by heffalump.ip.net.ua (8.13.1/8.13.1) id j14Km4BU033901; Fri, 4 Feb 2005 22:48:04 +0200 (EET) (envelope-from ru) Date: Fri, 4 Feb 2005 22:48:04 +0200 From: Ruslan Ermilov To: Guy Helmer Message-ID: <20050204204804.GC71363@ip.net.ua> References: <4203AAE3.4090906@palisadesys.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="JgQwtEuHJzHdouWu" Content-Disposition: inline In-Reply-To: <4203AAE3.4090906@palisadesys.com> User-Agent: Mutt/1.5.6i X-Virus-Scanned: by amavisd-new at ip.net.ua cc: freebsd-net@FreeBSD.org Subject: Re: Netgraph performance question X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Feb 2005 20:48:07 -0000 --JgQwtEuHJzHdouWu Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi Guy, On Fri, Feb 04, 2005 at 11:03:31AM -0600, Guy Helmer wrote: > A while back, Maxim Konovalov made a commit to usr.sbin/ngctl/main.c to= =20 > increase its socket receive buffer size to help 'ngctl list' deal with a= =20 > big number of nodes, and Ruslan Ermilov responded that setting sysctls=20 > net.graph.recvspace=3D200000 and net.graph.maxdgram=3D200000 was a good i= dea=20 > on a system with a large number of nodes. >=20 > I'm getting what I consider to be sub-par performance under FreeBSD 5.3= =20 > from a userland program using ngsockets connected into ng_tee to play=20 > with packets that are traversing a ng_bridge, and I finally have an=20 > opportunity to look into this. I say "sub-par" because when we've=20 > tested this configuration using three 2.8GHz Xeon machines with Gigabit= =20 > Ethernet interfaces at 1000Mbps full-duplex, we obtained peak=20 > performance of a single TCP stream of about 12MB/sec through the=20 > bridging machine as measured by NetPIPE and netperf. >=20 The bottleneck must be in ng_tee(4) -- the latter uses m_dup(9) when a duplicate is needed, which is very expensive as it has to create a writable copy of the entire mbuf chain (the original chain is DMA'ed into the host memory by the network card). > I'm wondering if bumping the recvspace should help, if changing the=20 > ngsocket hook to queue incoming data should help, if it would be best to= =20 > replace ngsocket with a memory-mapped interface, or if anyone has any=20 > other ideas that would help performance. >=20 If you absolutely need to see *all* GigE traffic in userland, then it's going to be troublesome. If not, filter it with ng_bpf(4). Cheers, --=20 Ruslan Ermilov ru@FreeBSD.org FreeBSD committer --JgQwtEuHJzHdouWu Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.6 (FreeBSD) iD8DBQFCA9+EqRfpzJluFF4RAsT/AJ9iyLMh/ktOtfb8Ko1qt/ApT0oFBQCgiwAZ hoJcv9Tew4LbQtX3zaMdVBo= =q5r3 -----END PGP SIGNATURE----- --JgQwtEuHJzHdouWu--