From owner-freebsd-performance@FreeBSD.ORG Sat Feb 2 23:12:50 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 15D3016A417; Sat, 2 Feb 2008 23:12:50 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from cmail.optima.ua (cmail.optima.ua [195.248.191.121]) by mx1.freebsd.org (Postfix) with ESMTP id CD4B013C461; Sat, 2 Feb 2008 23:12:48 +0000 (UTC) (envelope-from mav@FreeBSD.org) X-Spam-Flag: SKIP X-Spam-Yversion: Spamooborona 1.7.0 Received: from [212.86.226.226] (account mav@alkar.net HELO [192.168.3.2]) by cmail.optima.ua (CommuniGate Pro SMTP 5.1.14) with ESMTPA id 72324289; Sun, 03 Feb 2008 01:12:47 +0200 Message-ID: <47A4F8EE.3070202@FreeBSD.org> Date: Sun, 03 Feb 2008 01:12:46 +0200 From: Alexander Motin User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) MIME-Version: 1.0 To: Robert Watson References: <47A25412.3010301@FreeBSD.org> <47A25A0D.2080508@elischer.org> <47A2C2A2.5040109@FreeBSD.org> <20080201185435.X88034@fledge.watson.org> <47A43873.40801@FreeBSD.org> <20080202095658.R63379@fledge.watson.org> <47A4E934.1050207@FreeBSD.org> <47A4F1AF.9090306@FreeBSD.org> <20080202224923.T66602@fledge.watson.org> In-Reply-To: <20080202224923.T66602@fledge.watson.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Sun, 03 Feb 2008 00:46:01 +0000 Cc: freebsd-hackers@freebsd.org, Kris Kennaway , freebsd-performance@freebsd.org, Julian Elischer Subject: Re: Memory allocation performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Feb 2008 23:12:50 -0000 Robert Watson wrote: > Basically, the goal would be > to make the pcpu cache FIFO as much as possible as that maximizes the > chances that the newly allocated object already has lines in the cache. Why FIFO? I think LIFO (stack) should be better for this goal as the last freed object has more chances to be still present in cache. -- Alexander Motin From owner-freebsd-performance@FreeBSD.ORG Sun Feb 3 02:55:58 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A63E316A419 for ; Sun, 3 Feb 2008 02:55:58 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from fallbackmx08.syd.optusnet.com.au (fallbackmx08.syd.optusnet.com.au [211.29.132.10]) by mx1.freebsd.org (Postfix) with ESMTP id 36FD113C442 for ; Sun, 3 Feb 2008 02:55:57 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from mail15.syd.optusnet.com.au (mail15.syd.optusnet.com.au [211.29.132.196]) by fallbackmx08.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id m12Jb1dF030602 for ; Sun, 3 Feb 2008 06:37:01 +1100 Received: from server.vk2pj.dyndns.org (c220-239-20-82.belrs4.nsw.optusnet.com.au [220.239.20.82]) by mail15.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id m12Javbs006578 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sun, 3 Feb 2008 06:36:58 +1100 Received: from server.vk2pj.dyndns.org (localhost.vk2pj.dyndns.org [127.0.0.1]) by server.vk2pj.dyndns.org (8.14.2/8.14.1) with ESMTP id m12Javsd077015; Sun, 3 Feb 2008 06:36:57 +1100 (EST) (envelope-from peter@server.vk2pj.dyndns.org) Received: (from peter@localhost) by server.vk2pj.dyndns.org (8.14.2/8.14.2/Submit) id m12JauNK077014; Sun, 3 Feb 2008 06:36:56 +1100 (EST) (envelope-from peter) Date: Sun, 3 Feb 2008 06:36:56 +1100 From: Peter Jeremy To: Alexander Motin Message-ID: <20080202193656.GR35170@server.vk2pj.dyndns.org> References: <47A25412.3010301@FreeBSD.org> <47A25A0D.2080508@elischer.org> <47A2C2A2.5040109@FreeBSD.org> <20080201185435.X88034@fledge.watson.org> <47A43873.40801@FreeBSD.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="n/aVsWSeQ4JHkrmm" Content-Disposition: inline In-Reply-To: <47A43873.40801@FreeBSD.org> X-PGP-Key: http://members.optusnet.com.au/peterjeremy/pubkey.asc User-Agent: Mutt/1.5.17 (2007-11-01) X-Mailman-Approved-At: Sun, 03 Feb 2008 04:24:26 +0000 Cc: freebsd-hackers@freebsd.org, freebsd-performance@freebsd.org Subject: Re: Memory allocation performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Feb 2008 02:55:58 -0000 --n/aVsWSeQ4JHkrmm Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Feb 02, 2008 at 11:31:31AM +0200, Alexander Motin wrote: >To check UMA dependency I have made a trivial one-element cache which in m= y=20 >test case allows to avoid two for four allocations per packet. You should be able to implement this lockless using atomic(9). I haven't verified it, but the following should work. >.....alloc..... >- item =3D uma_zalloc(ng_qzone, wait | M_ZERO); >+ mtx_lock_spin(&itemcachemtx); >+ item =3D itemcache; >+ itemcache =3D NULL; >+ mtx_unlock_spin(&itemcachemtx); =3D item =3D atomic_readandclear_ptr(&itemcache); >+ if (item =3D=3D NULL) >+ item =3D uma_zalloc(ng_qzone, wait | M_ZERO); >+ else >+ bzero(item, sizeof(*item)); >.....free..... >- uma_zfree(ng_qzone, item); >+ mtx_lock_spin(&itemcachemtx); >+ if (itemcache =3D=3D NULL) { >+ itemcache =3D item; >+ item =3D NULL; >+ } >+ mtx_unlock_spin(&itemcachemtx); >+ if (item) >+ uma_zfree(ng_qzone, item); =3D if (atomic_cmpset_ptr(&itemcache, NULL, item) =3D=3D 0) =3D uma_zfree(ng_qzone, item); --=20 Peter Jeremy Please excuse any delays as the result of my ISP's inability to implement an MTA that is either RFC2821-compliant or matches their claimed behaviour. --n/aVsWSeQ4JHkrmm Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQFHpMZY/opHv/APuIcRAjX8AJ9Tr3OIDEmNTcd6GFUNOG8/5JOt9wCfTaXz zXwtwl46hYGVRmJI8P2gfXw= =urmE -----END PGP SIGNATURE----- --n/aVsWSeQ4JHkrmm-- From owner-freebsd-performance@FreeBSD.ORG Sun Feb 3 05:33:03 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5727916A52C for ; Sun, 3 Feb 2008 05:33:03 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outE.internet-mail-service.net (outE.internet-mail-service.net [216.240.47.228]) by mx1.freebsd.org (Postfix) with ESMTP id 3D87313C4F3 for ; Sun, 3 Feb 2008 05:32:59 +0000 (UTC) (envelope-from julian@elischer.org) Received: from mx0.idiom.com (HELO idiom.com) (216.240.32.160) by out.internet-mail-service.net (qpsmtpd/0.40) with ESMTP; Sat, 02 Feb 2008 21:32:58 -0800 Received: from julian-mac.elischer.org (localhost [127.0.0.1]) by idiom.com (Postfix) with ESMTP id E13641270B9; Sat, 2 Feb 2008 21:32:57 -0800 (PST) Message-ID: <47A55209.9040106@elischer.org> Date: Sat, 02 Feb 2008 21:32:57 -0800 From: Julian Elischer User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: Robert Watson References: <47A25412.3010301@FreeBSD.org> <47A25A0D.2080508@elischer.org> <47A2C2A2.5040109@FreeBSD.org> <20080201185435.X88034@fledge.watson.org> <47A43873.40801@FreeBSD.org> <20080202095658.R63379@fledge.watson.org> <47A4E934.1050207@FreeBSD.org> <47A4F1AF.9090306@FreeBSD.org> <20080202224923.T66602@fledge.watson.org> In-Reply-To: <20080202224923.T66602@fledge.watson.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, Kris Kennaway , Alexander Motin , freebsd-performance@freebsd.org Subject: Re: Memory allocation performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Feb 2008 05:33:03 -0000 Robert Watson wrote: > be a good time to try to revalidate that. Basically, the goal would be > to make the pcpu cache FIFO as much as possible as that maximizes the you mean FILO or LIFO right? > chances that the newly allocated object already has lines in the cache. > It's a fairly trivial tweak to the UMA allocation code. > > Robert N M Watson > Computer Laboratory > University of Cambridge From owner-freebsd-performance@FreeBSD.ORG Sun Feb 3 12:23:59 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 32EF916A417 for ; Sun, 3 Feb 2008 12:23:59 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id E73F413C468 for ; Sun, 3 Feb 2008 12:23:58 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (localhost [127.0.0.1]) by spam.des.no (Postfix) with ESMTP id 87B21207F; Sun, 3 Feb 2008 13:04:49 +0100 (CET) X-Spam-Tests: AWL X-Spam-Learn: disabled X-Spam-Score: -0.2/3.0 X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on tim.des.no Received: from ds4.des.no (des.no [80.203.243.180]) by smtp.des.no (Postfix) with ESMTP id 65D77207E; Sun, 3 Feb 2008 13:04:49 +0100 (CET) Received: by ds4.des.no (Postfix, from userid 1001) id 405358449D; Sun, 3 Feb 2008 13:04:49 +0100 (CET) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: Julian Elischer References: <47A25412.3010301@FreeBSD.org> <47A25A0D.2080508@elischer.org> <47A2C2A2.5040109@FreeBSD.org> <20080201185435.X88034@fledge.watson.org> <47A43873.40801@FreeBSD.org> <20080202095658.R63379@fledge.watson.org> <47A4E934.1050207@FreeBSD.org> <47A4F1AF.9090306@FreeBSD.org> <20080202224923.T66602@fledge.watson.org> <47A55209.9040106@elischer.org> Date: Sun, 03 Feb 2008 13:04:48 +0100 In-Reply-To: <47A55209.9040106@elischer.org> (Julian Elischer's message of "Sat\, 02 Feb 2008 21\:32\:57 -0800") Message-ID: <86ejbumejj.fsf@ds4.des.no> User-Agent: Gnus/5.110006 (No Gnus v0.6) Emacs/22.1 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-hackers@freebsd.org, Kris Kennaway , Robert Watson , Alexander Motin , freebsd-performance@freebsd.org Subject: Re: Memory allocation performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Feb 2008 12:23:59 -0000 Julian Elischer writes: > Robert Watson writes: > > be a good time to try to revalidate that. Basically, the goal would > > be to make the pcpu cache FIFO as much as possible as that maximizes > > the chances that the newly allocated object already has lines in the > > cache. It's a fairly trivial tweak to the UMA allocation code. > you mean FILO or LIFO right? Uh, no. You want to reuse the last-freed object, as it is most likely to still be in cache. DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-performance@FreeBSD.ORG Sun Feb 3 19:24:36 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7177E16A418 for ; Sun, 3 Feb 2008 19:24:36 +0000 (UTC) (envelope-from bra@fsn.hu) Received: from people.fsn.hu (people.fsn.hu [195.228.252.137]) by mx1.freebsd.org (Postfix) with ESMTP id 3CCEE13C46B for ; Sun, 3 Feb 2008 19:24:35 +0000 (UTC) (envelope-from bra@fsn.hu) Received: from [127.0.0.1] (fw.axelero.hu [195.228.243.120]) by people.fsn.hu (Postfix) with ESMTP id A3BB070712; Sun, 3 Feb 2008 20:24:28 +0100 (CET) Message-ID: <47A614E9.4030501@fsn.hu> Date: Sun, 03 Feb 2008 20:24:25 +0100 From: Attila Nagy User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) MIME-Version: 1.0 To: =?UTF-8?B?SklOTUVJIFRhdHV5YSAvIOelnuaYjumBlOWTiQ==?= References: <475B0F3E.5070100@fsn.hu> <479DFE74.8030004@fsn.hu> <479F02A7.9020607@fsn.hu> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Cc: freebsd-performance@freebsd.org, bind-users@isc.org Subject: Re: max-cache-size doesn't work with 9.5.0b1 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Feb 2008 19:24:36 -0000 On 2008.01.30. 3:28, JINMEI Tatuya / 神明達哉 wrote: > Okay, please use the attached patch (applicable to 9.5.0b1, and also > to 9.5.0b2 when it's published). Build it with: > % STD_CDEFINES='-DLRU_DEBUG2=2' ./configure --enable-threads > (or set STD_CDEFINES using setenv if you use a csh variant) > > The log messages shouldn't be very noisy, but if you find them too > frequent, rebuild it with: > % STD_CDEFINES='-DLRU_DEBUG2=1' ./configure --enable-threads > > Note that, if this is a thread-related bug, it may not always be > reproduceable; please try several times if the problem doesn't seem to > happen. (BTW: did it always occur when you first found the problem?) > Yes, if bind was built with threads, the memory usage always grew behind max-cache-size very quickly. Here is the log: http://people.fsn.hu/~bra/freebsd/bind950-memory-20080203/bind950b1 the memory usage (RSS, reported by top) in megabytes: 19:10:37 466 19:11:20 522 19:11:53 566 19:13:06 666 19:14:17 766 max-cache-size was set to 64M. From owner-freebsd-performance@FreeBSD.ORG Sun Feb 3 23:20:44 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EE81516A46B for ; Sun, 3 Feb 2008 23:20:44 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outO.internet-mail-service.net (outO.internet-mail-service.net [216.240.47.238]) by mx1.freebsd.org (Postfix) with ESMTP id B3A5E13C474 for ; Sun, 3 Feb 2008 23:20:44 +0000 (UTC) (envelope-from julian@elischer.org) Received: from mx0.idiom.com (HELO idiom.com) (216.240.32.160) by out.internet-mail-service.net (qpsmtpd/0.40) with ESMTP; Sun, 03 Feb 2008 15:20:43 -0800 Received: from julian-mac.elischer.org (localhost [127.0.0.1]) by idiom.com (Postfix) with ESMTP id 0F79D1270BD; Sun, 3 Feb 2008 15:20:43 -0800 (PST) Message-ID: <47A64C4A.5020002@elischer.org> Date: Sun, 03 Feb 2008 15:20:42 -0800 From: Julian Elischer User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: =?UTF-8?B?RGFnLUVybGluZyBTbcO4cmdyYXY=?= References: <47A25412.3010301@FreeBSD.org> <47A25A0D.2080508@elischer.org> <47A2C2A2.5040109@FreeBSD.org> <20080201185435.X88034@fledge.watson.org> <47A43873.40801@FreeBSD.org> <20080202095658.R63379@fledge.watson.org> <47A4E934.1050207@FreeBSD.org> <47A4F1AF.9090306@FreeBSD.org> <20080202224923.T66602@fledge.watson.org> <47A55209.9040106@elischer.org> <86ejbumejj.fsf@ds4.des.no> In-Reply-To: <86ejbumejj.fsf@ds4.des.no> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Cc: freebsd-hackers@freebsd.org, Kris Kennaway , Robert Watson , Alexander Motin , freebsd-performance@freebsd.org Subject: Re: Memory allocation performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Feb 2008 23:20:45 -0000 Dag-Erling Smørgrav wrote: > Julian Elischer writes: >> Robert Watson writes: >>> be a good time to try to revalidate that. Basically, the goal would >>> be to make the pcpu cache FIFO as much as possible as that maximizes >>> the chances that the newly allocated object already has lines in the >>> cache. It's a fairly trivial tweak to the UMA allocation code. >> you mean FILO or LIFO right? > > Uh, no. You want to reuse the last-freed object, as it is most likely > to still be in cache. > > DES exactly.. FILO or LIFO (last in First out.) From owner-freebsd-performance@FreeBSD.ORG Sun Feb 3 23:40:15 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6042316A41B; Sun, 3 Feb 2008 23:40:15 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from cmail.optima.ua (cmail.optima.ua [195.248.191.121]) by mx1.freebsd.org (Postfix) with ESMTP id 2DCF013C45A; Sun, 3 Feb 2008 23:40:13 +0000 (UTC) (envelope-from mav@FreeBSD.org) X-Spam-Flag: SKIP X-Spam-Yversion: Spamooborona 1.7.0 Received: from [212.86.226.226] (account mav@alkar.net HELO [192.168.3.2]) by cmail.optima.ua (CommuniGate Pro SMTP 5.1.14) with ESMTPA id 72717087; Mon, 04 Feb 2008 01:40:13 +0200 Message-ID: <47A650DA.8020908@FreeBSD.org> Date: Mon, 04 Feb 2008 01:40:10 +0200 From: Alexander Motin User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) MIME-Version: 1.0 To: Kris Kennaway , Robert Watson References: <47A25412.3010301@FreeBSD.org> <47A25A0D.2080508@elischer.org> <47A2C2A2.5040109@FreeBSD.org> <20080201185435.X88034@fledge.watson.org> <47A43873.40801@FreeBSD.org> <20080202095658.R63379@fledge.watson.org> <47A4E934.1050207@FreeBSD.org> <47A4F1AF.9090306@FreeBSD.org> In-Reply-To: <47A4F1AF.9090306@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Sun, 03 Feb 2008 23:59:34 +0000 Cc: freebsd-hackers@freebsd.org, freebsd-performance@freebsd.org, Julian Elischer Subject: Re: Memory allocation performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Feb 2008 23:40:15 -0000 Kris Kennaway wrote: > You can look at the raw output from pmcstat, which is a collection of > instruction pointers that you can feed to e.g. addr2line to find out > exactly where in those functions the events are occurring. This will > often help to track down the precise causes. Thanks to the hint, it was interesting hunting, but it shown nothing. It hits into very simple lines like: bucket = cache->uc_freebucket; cache->uc_allocs++; if (zone->uz_ctor != NULL) { cache->uc_frees++; and so on. There is no loops, there is no inlines or macroses. Nothing! And the only hint about it is a huge number of "p4-resource-stall"s in those lines. I have no idea what exactly does it means, why does it happens mostly here and how to fight it. I would probably agreed that it might be some profiler fluctuation, but performance benefits I have got from self-made uma calls caching look very real. :( Robert Watson wrote: > There was, FYI, a report a few years ago that there was a measurable > improvement from allocating off the free bucket rather than maintaining > separate alloc and free buckets. It sounded good at the time but I was > never able to reproduce the benefits in my test environment. Now might > be a good time to try to revalidate that. Basically, the goal would be > to make the pcpu cache FIFO as much as possible as that maximizes the > chances that the newly allocated object already has lines in the cache. > It's a fairly trivial tweak to the UMA allocation code. I have tried this, but have not found a difference. May be it gives some benefits, but not in this situation. In this situation profiling shows delays in allocator itself, so as soon as allocator does not touches data objects itself it probably more speaks about management structure's memory caching then about objects caching. I have got one more crazy idea that memory containing zones may have some special hardware or configuration features, like "noncaching" or something alike. That could explain slowdown in accessing it. But as I can't prove it, it just one more crazy theory. :( -- Alexander Motin From owner-freebsd-performance@FreeBSD.ORG Mon Feb 4 00:04:44 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 193EF16A421 for ; Mon, 4 Feb 2008 00:04:44 +0000 (UTC) (envelope-from marka@isc.org) Received: from drugs.dv.isc.org (drugs.dv.isc.org [IPv6:2001:470:1f00:820:214:22ff:fed9:fbdc]) by mx1.freebsd.org (Postfix) with ESMTP id B1C8B13C457 for ; Mon, 4 Feb 2008 00:04:43 +0000 (UTC) (envelope-from marka@isc.org) Received: from drugs.dv.isc.org (localhost [127.0.0.1]) by drugs.dv.isc.org (8.14.2/8.14.1) with ESMTP id m1404eNS038322; Mon, 4 Feb 2008 11:04:42 +1100 (EST) (envelope-from marka@drugs.dv.isc.org) Message-Id: <200802040004.m1404eNS038322@drugs.dv.isc.org> To: Attila Nagy From: Mark Andrews In-reply-to: Your message of "Sun, 03 Feb 2008 20:24:25 BST." <47A614E9.4030501@fsn.hu> Date: Mon, 04 Feb 2008 11:04:40 +1100 Sender: marka@isc.org X-Mailman-Approved-At: Mon, 04 Feb 2008 00:15:29 +0000 Cc: =?UTF-8?B?SklOTUVJIFRhdHV5YSAvIOelnuaYjumBlOWTiQ==?= , freebsd-performance@freebsd.org, bind-users@isc.org Subject: Re: max-cache-size doesn't work with 9.5.0b1 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 00:04:44 -0000 > On 2008.01.30. 3:28, JINMEI Tatuya / 神明達哉 wrote: > > Okay, please use the attached patch (applicable to 9.5.0b1, and also > > to 9.5.0b2 when it's published). Build it with: > > % STD_CDEFINES='-DLRU_DEBUG2=2' ./configure --enable-threads > > (or set STD_CDEFINES using setenv if you use a csh variant) > > > > The log messages shouldn't be very noisy, but if you find them too > > frequent, rebuild it with: > > % STD_CDEFINES='-DLRU_DEBUG2=1' ./configure --enable-threads > > > > Note that, if this is a thread-related bug, it may not always be > > reproduceable; please try several times if the problem doesn't seem to > > happen. (BTW: did it always occur when you first found the problem?) > > > Yes, if bind was built with threads, the memory usage always grew behind > max-cache-size very quickly. > > Here is the log: > http://people.fsn.hu/~bra/freebsd/bind950-memory-20080203/bind950b1 > the memory usage (RSS, reported by top) in megabytes: > 19:10:37 466 > 19:11:20 522 > 19:11:53 566 > 19:13:06 666 > 19:14:17 766 > > max-cache-size was set to 64M. Please try this patch. Mark Index: lib/dns/cache.c =================================================================== RCS file: /proj/cvs/prod/bind9/lib/dns/cache.c,v retrieving revision 1.76 diff -u -r1.76 cache.c --- lib/dns/cache.c 19 Oct 2007 17:15:53 -0000 1.76 +++ lib/dns/cache.c 3 Feb 2008 23:59:46 -0000 @@ -708,8 +708,11 @@ LOCK(&cache->cleaner.lock); - dns_db_overmem(cache->db, overmem); - cache->cleaner.overmem = overmem; + if (overmem != cache->cleaner.overmem) { + dns_db_overmem(cache->db, overmem); + cache->cleaner.overmem = overmem; + isc_mem_water(cache->mctx, mark); + } UNLOCK(&cache->cleaner.lock); } Index: lib/isc/mem.c =================================================================== RCS file: /proj/cvs/prod/bind9/lib/isc/mem.c,v retrieving revision 1.140 diff -u -r1.140 mem.c --- lib/isc/mem.c 18 Jan 2008 23:46:58 -0000 1.140 +++ lib/isc/mem.c 3 Feb 2008 23:59:48 -0000 @@ -1086,7 +1086,6 @@ ADD_TRACE(ctx, ptr, size, file, line); if (ctx->hi_water != 0U && !ctx->hi_called && ctx->inuse > ctx->hi_water) { - ctx->hi_called = ISC_TRUE; call_water = ISC_TRUE; } if (ctx->inuse > ctx->maxinuse) { @@ -1144,8 +1143,6 @@ */ if (ctx->hi_called && (ctx->inuse < ctx->lo_water || ctx->lo_water == 0U)) { - ctx->hi_called = ISC_FALSE; - if (ctx->water != NULL) call_water = ISC_TRUE; } @@ -1155,6 +1152,18 @@ (ctx->water)(ctx->water_arg, ISC_MEM_LOWATER); } +void +isc_mem_water(isc_mem_t *ctx, int flag) { + REQUIRE(VALID_CONTEXT(ctx)); + + MCTXLOCK(ctx, &ctx->lock); + if (flag == ISC_MEM_LOWATER) + ctx->hi_called = ISC_FALSE; + else if (flag == ISC_MEM_HIWATER) + ctx->hi_called = ISC_TRUE; + MCTXUNLOCK(ctx, &ctx->lock); +} + #if ISC_MEM_TRACKLINES static void print_active(isc_mem_t *mctx, FILE *out) { -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: Mark_Andrews@isc.org From owner-freebsd-performance@FreeBSD.ORG Mon Feb 4 00:31:05 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8691D16A418 for ; Mon, 4 Feb 2008 00:31:05 +0000 (UTC) (envelope-from marka@isc.org) Received: from drugs.dv.isc.org (drugs.dv.isc.org [IPv6:2001:470:1f00:820:214:22ff:fed9:fbdc]) by mx1.freebsd.org (Postfix) with ESMTP id E7A3613C44B for ; Mon, 4 Feb 2008 00:31:04 +0000 (UTC) (envelope-from marka@isc.org) Received: from drugs.dv.isc.org (localhost [127.0.0.1]) by drugs.dv.isc.org (8.14.2/8.14.1) with ESMTP id m140V3mt039006; Mon, 4 Feb 2008 11:31:03 +1100 (EST) (envelope-from marka@drugs.dv.isc.org) Message-Id: <200802040031.m140V3mt039006@drugs.dv.isc.org> From: Mark Andrews In-reply-to: Your message of "Mon, 04 Feb 2008 11:04:40 +1100." <200802040004.m1404eNS038322@drugs.dv.isc.org> Date: Mon, 04 Feb 2008 11:31:03 +1100 Sender: marka@isc.org X-Mailman-Approved-At: Mon, 04 Feb 2008 01:13:57 +0000 Cc: Attila Nagy , freebsd-performance@freebsd.org, bind-users@isc.org, =?UTF-8?B?SklOTUVJIFRhdHV5YSAvIOelnuaYjumBlOWTiQ==?= Subject: Re: max-cache-size doesn't work with 9.5.0b1 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 00:31:05 -0000 > > Please try this patch. > > Mark Revised. Index: lib/isc/mem.c =================================================================== RCS file: /proj/cvs/prod/bind9/lib/isc/mem.c,v retrieving revision 1.140 diff -u -r1.140 mem.c --- lib/isc/mem.c 18 Jan 2008 23:46:58 -0000 1.140 +++ lib/isc/mem.c 4 Feb 2008 00:29:44 -0000 @@ -1086,7 +1086,6 @@ ADD_TRACE(ctx, ptr, size, file, line); if (ctx->hi_water != 0U && !ctx->hi_called && ctx->inuse > ctx->hi_water) { - ctx->hi_called = ISC_TRUE; call_water = ISC_TRUE; } if (ctx->inuse > ctx->maxinuse) { @@ -1144,8 +1143,6 @@ */ if (ctx->hi_called && (ctx->inuse < ctx->lo_water || ctx->lo_water == 0U)) { - ctx->hi_called = ISC_FALSE; - if (ctx->water != NULL) call_water = ISC_TRUE; } @@ -1155,6 +1152,18 @@ (ctx->water)(ctx->water_arg, ISC_MEM_LOWATER); } +void +isc_mem_water(isc_mem_t *ctx, int flag) { + REQUIRE(VALID_CONTEXT(ctx)); + + MCTXLOCK(ctx, &ctx->lock); + if (flag == ISC_MEM_LOWATER) + ctx->hi_called = ISC_FALSE; + else if (flag == ISC_MEM_HIWATER) + ctx->hi_called = ISC_TRUE; + MCTXUNLOCK(ctx, &ctx->lock); +} + #if ISC_MEM_TRACKLINES static void print_active(isc_mem_t *mctx, FILE *out) { Index: lib/dns/acache.c =================================================================== RCS file: /proj/cvs/prod/bind9/lib/dns/acache.c,v retrieving revision 1.20 diff -u -r1.20 acache.c --- lib/dns/acache.c 19 Jun 2007 23:47:16 -0000 1.20 +++ lib/dns/acache.c 4 Feb 2008 00:29:46 -0000 @@ -965,10 +965,14 @@ LOCK(&acache->cleaner.lock); - acache->cleaner.overmem = overmem; + if (acache->cleaner.overmem != overmem) { + acache->cleaner.overmem = overmem; - if (acache->cleaner.overmem_event != NULL) - isc_task_send(acache->task, &acache->cleaner.overmem_event); + if (acache->cleaner.overmem_event != NULL) + isc_task_send(acache->task, + &acache->cleaner.overmem_event); + isc_mem_water(acache->mctx, mark); + } UNLOCK(&acache->cleaner.lock); } Index: lib/dns/adb.c =================================================================== RCS file: /proj/cvs/prod/bind9/lib/dns/adb.c,v retrieving revision 1.233 diff -u -r1.233 adb.c --- lib/dns/adb.c 19 Oct 2007 17:15:53 -0000 1.233 +++ lib/dns/adb.c 4 Feb 2008 00:29:50 -0000 @@ -128,6 +128,7 @@ isc_mutex_t lock; isc_mutex_t reflock; /*%< Covers irefcnt, erefcnt */ + isc_mutex_t overmemlock; /*%< Covers overmem */ isc_mem_t *mctx; dns_view_t *view; isc_timermgr_t *timermgr; @@ -2138,6 +2139,7 @@ DESTROYLOCK(&adb->reflock); DESTROYLOCK(&adb->lock); DESTROYLOCK(&adb->mplock); + DESTROYLOCK(&adb->overmemlock); isc_mem_putanddetach(&adb->mctx, adb, sizeof(dns_adb_t)); } @@ -2225,6 +2227,10 @@ if (result != ISC_R_SUCCESS) goto fail0d; + result = isc_mutex_init(&adb->overmemlock); + if (result != ISC_R_SUCCESS) + goto fail0e; + /* * Initialize the bucket locks for names and elements. * May as well initialize the list heads, too. @@ -2343,6 +2349,8 @@ if (adb->afmp != NULL) isc_mempool_destroy(&adb->afmp); + DESTROYLOCK(&adb->overmemlock); + fail0e: DESTROYLOCK(&adb->reflock); fail0d: DESTROYLOCK(&adb->mplock); @@ -3782,16 +3790,21 @@ DP(ISC_LOG_DEBUG(1), "adb reached %s water mark", overmem ? "high" : "low"); - adb->overmem = overmem; + LOCK(&adb->overmemlock); + if (adb->overmem != overmem) { + adb->overmem = overmem; #if 0 /* we don't need this timer for the new cleaning policy. */ - if (overmem) { - isc_interval_t interval; + if (overmem) { + isc_interval_t interval; - isc_interval_set(&interval, 0, 1); - (void)isc_timer_reset(adb->timer, isc_timertype_once, NULL, - &interval, ISC_TRUE); - } -#endif + isc_interval_set(&interval, 0, 1); + (void)isc_timer_reset(adb->timer, isc_timertype_once, + NULL, &interval, ISC_TRUE); + } +#endif + isc_mem_water(adb->mctx, mark); + } + UNLOCK(&adb->overmemlock); } void Index: lib/dns/cache.c =================================================================== RCS file: /proj/cvs/prod/bind9/lib/dns/cache.c,v retrieving revision 1.76 diff -u -r1.76 cache.c --- lib/dns/cache.c 19 Oct 2007 17:15:53 -0000 1.76 +++ lib/dns/cache.c 4 Feb 2008 00:29:51 -0000 @@ -708,8 +708,11 @@ LOCK(&cache->cleaner.lock); - dns_db_overmem(cache->db, overmem); - cache->cleaner.overmem = overmem; + if (overmem != cache->cleaner.overmem) { + dns_db_overmem(cache->db, overmem); + cache->cleaner.overmem = overmem; + isc_mem_water(cache->mctx, mark); + } UNLOCK(&cache->cleaner.lock); } -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: Mark_Andrews@isc.org From owner-freebsd-performance@FreeBSD.ORG Mon Feb 4 02:40:44 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B03EF16A417; Mon, 4 Feb 2008 02:40:44 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail18.syd.optusnet.com.au (mail18.syd.optusnet.com.au [211.29.132.199]) by mx1.freebsd.org (Postfix) with ESMTP id 20C7A13C455; Mon, 4 Feb 2008 02:40:43 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from c211-30-219-213.carlnfd3.nsw.optusnet.com.au (c211-30-219-213.carlnfd3.nsw.optusnet.com.au [211.30.219.213]) by mail18.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id m142eVi6014054 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 4 Feb 2008 13:40:33 +1100 Date: Mon, 4 Feb 2008 13:40:31 +1100 (EST) From: Bruce Evans X-X-Sender: bde@delplex.bde.org To: Alexander Motin In-Reply-To: <47A650DA.8020908@FreeBSD.org> Message-ID: <20080204133437.T63947@delplex.bde.org> References: <47A25412.3010301@FreeBSD.org> <47A25A0D.2080508@elischer.org> <47A2C2A2.5040109@FreeBSD.org> <20080201185435.X88034@fledge.watson.org> <47A43873.40801@FreeBSD.org> <20080202095658.R63379@fledge.watson.org> <47A4E934.1050207@FreeBSD.org> <47A4F1AF.9090306@FreeBSD.org> <47A650DA.8020908@FreeBSD.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-hackers@freebsd.org, Kris Kennaway , Robert Watson , freebsd-performance@freebsd.org, Julian Elischer Subject: Re: Memory allocation performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 02:40:44 -0000 On Mon, 4 Feb 2008, Alexander Motin wrote: > Kris Kennaway wrote: >> You can look at the raw output from pmcstat, which is a collection of >> instruction pointers that you can feed to e.g. addr2line to find out >> exactly where in those functions the events are occurring. This will often >> help to track down the precise causes. > > Thanks to the hint, it was interesting hunting, but it shown nothing. It hits > into very simple lines like: > bucket = cache->uc_freebucket; > cache->uc_allocs++; > if (zone->uz_ctor != NULL) { > cache->uc_frees++; > and so on. > There is no loops, there is no inlines or macroses. Nothing! And the only > hint about it is a huge number of "p4-resource-stall"s in those lines. I have > no idea what exactly does it means, why does it happens mostly here and how > to fight it. Try profiling it one another type of CPU, to get different performance counters but hopefully not very different stalls. If the other CPU doesn't stall at all, put another black mark against P4 and delete your copies of it :-). Bruce From owner-freebsd-performance@FreeBSD.ORG Mon Feb 4 10:35:50 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 34EF616A419; Mon, 4 Feb 2008 10:35:50 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id DB39913C45D; Mon, 4 Feb 2008 10:35:49 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (localhost [127.0.0.1]) by spam.des.no (Postfix) with ESMTP id D88242082; Mon, 4 Feb 2008 11:35:41 +0100 (CET) X-Spam-Tests: AWL X-Spam-Learn: disabled X-Spam-Score: -0.2/3.0 X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on tim.des.no Received: from ds4.des.no (des.no [80.203.243.180]) by smtp.des.no (Postfix) with ESMTP id 5EBB7207F; Mon, 4 Feb 2008 11:35:41 +0100 (CET) Received: by ds4.des.no (Postfix, from userid 1001) id 3A628844A1; Mon, 4 Feb 2008 11:35:41 +0100 (CET) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: Julian Elischer References: <47A25412.3010301@FreeBSD.org> <47A25A0D.2080508@elischer.org> <47A2C2A2.5040109@FreeBSD.org> <20080201185435.X88034@fledge.watson.org> <47A43873.40801@FreeBSD.org> <20080202095658.R63379@fledge.watson.org> <47A4E934.1050207@FreeBSD.org> <47A4F1AF.9090306@FreeBSD.org> <20080202224923.T66602@fledge.watson.org> <47A55209.9040106@elischer.org> <86ejbumejj.fsf@ds4.des.no> <47A64C4A.5020002@elischer.org> Date: Mon, 04 Feb 2008 11:35:41 +0100 In-Reply-To: <47A64C4A.5020002@elischer.org> (Julian Elischer's message of "Sun\, 03 Feb 2008 15\:20\:42 -0800") Message-ID: <86prvdggaq.fsf@ds4.des.no> User-Agent: Gnus/5.110006 (No Gnus v0.6) Emacs/22.1 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-hackers@freebsd.org, Kris Kennaway , Robert Watson , Alexander Motin , freebsd-performance@freebsd.org Subject: Re: Memory allocation performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 10:35:50 -0000 Julian Elischer writes: > Dag-Erling Sm=C3=B8rgrav writes: > > Julian Elischer writes: > > > you mean FILO or LIFO right? > > Uh, no. You want to reuse the last-freed object, as it is most > > likely to still be in cache. > exactly.. FILO or LIFO (last in First out.) Clearly, I must have written the above in an acute state of caffeine deprivation. You are perfectly correct. DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-performance@FreeBSD.ORG Mon Feb 4 15:26:40 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C0B5516A420 for ; Mon, 4 Feb 2008 15:26:40 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: from blah.sun-fish.com (blah.sun-fish.com [217.18.249.150]) by mx1.freebsd.org (Postfix) with ESMTP id 7610F13C46E for ; Mon, 4 Feb 2008 15:26:40 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: by blah.sun-fish.com (Postfix, from userid 1002) id E88C71B10EDC; Mon, 4 Feb 2008 16:26:38 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on blah.cmotd.com X-Spam-Level: X-Spam-Status: No, score=-10.6 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.2.3 Received: from hater.haters.org (hater.cmotd.com [192.168.3.125]) by blah.sun-fish.com (Postfix) with ESMTP id B5DA41B10F2C for ; Mon, 4 Feb 2008 16:26:35 +0100 (CET) Message-ID: <47A72EAB.6070602@moneybookers.com> Date: Mon, 04 Feb 2008 17:26:35 +0200 From: Stefan Lambrev User-Agent: Thunderbird 2.0.0.9 (X11/20071120) MIME-Version: 1.0 To: freebsd-performance@freebsd.org References: <4794E6CC.1050107@moneybookers.com> <47A0B023.5020401@moneybookers.com> <47A3074A.3040409@moneybookers.com> In-Reply-To: <47A3074A.3040409@moneybookers.com> Content-Type: text/plain; charset=windows-1251; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: network performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 15:26:40 -0000 Greetings, In my desire to increase network throughput, and to be able to handle more then ~250-270kpps I started experimenting with lagg and link aggregation control protocol (lacp). To my surprise this doesn't increase the amount of packets my server can handle Here is what netstat reports: netstat -w1 -I lagg0 input (lagg0) output packets errs bytes packets errs bytes colls 267180 0 16030806 254056 0 14735542 0 266875 0 16012506 253829 0 14722260 0 netstat -w1 -I em0 input (em0) output packets errs bytes packets errs bytes colls 124789 72976 7487340 115329 0 6690468 0 126860 67350 7611600 114769 0 6658002 0 netstat -w1 -I em2 input (em2) output packets errs bytes packets errs bytes colls 123695 65533 7421700 113575 0 6584856 0 130277 62646 7816626 113648 0 6592280 0 123545 64171 7412706 113714 0 6596174 0 Using lagg doesn't improve situation at all, and also errors are not reported. Also using lagg increased content switches: procs memory page disk faults cpu r b w avm fre flt re pi po fr sr ad4 in sy cs us sy id 1 0 0 81048 1914640 52 0 0 0 50 0 0 3036 37902 13512 1 20 79 0 0 0 81048 1914640 13 0 0 0 0 0 0 9582 83 22166 0 56 44 0 0 0 81048 1914640 13 0 0 0 0 0 0 9594 80 22028 0 55 45 0 0 0 81048 1914640 13 0 0 0 0 0 0 9593 82 22095 0 56 44 Top showed for CPU states +55% system, which is quite high? I'll use hwpmc and lock_profiling to see where the kernel spends it's time. -- Best Wishes, Stefan Lambrev ICQ# 24134177 From owner-freebsd-performance@FreeBSD.ORG Mon Feb 4 17:01:18 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C0C6716A417 for ; Mon, 4 Feb 2008 17:01:18 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: from blah.sun-fish.com (blah.sun-fish.com [217.18.249.150]) by mx1.freebsd.org (Postfix) with ESMTP id 3C6EB13C45A for ; Mon, 4 Feb 2008 17:01:18 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: by blah.sun-fish.com (Postfix, from userid 1002) id E9ED41B10F39; Mon, 4 Feb 2008 18:01:16 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on blah.cmotd.com X-Spam-Level: X-Spam-Status: No, score=-10.6 required=5.0 tests=ALL_TRUSTED,BAYES_00, NORMAL_HTTP_TO_IP autolearn=ham version=3.2.3 Received: from hater.haters.org (hater.cmotd.com [192.168.3.125]) by blah.sun-fish.com (Postfix) with ESMTP id C59741B10EDC for ; Mon, 4 Feb 2008 18:01:13 +0100 (CET) Message-ID: <47A744D9.5080808@moneybookers.com> Date: Mon, 04 Feb 2008 19:01:13 +0200 From: Stefan Lambrev User-Agent: Thunderbird 2.0.0.9 (X11/20071120) MIME-Version: 1.0 To: freebsd-performance@freebsd.org References: <4794E6CC.1050107@moneybookers.com> <47A0B023.5020401@moneybookers.com> <47A3074A.3040409@moneybookers.com> <47A72EAB.6070602@moneybookers.com> In-Reply-To: <47A72EAB.6070602@moneybookers.com> Content-Type: text/plain; charset=windows-1251; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: network performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 17:01:18 -0000 Greetings, Stefan Lambrev wrote: > Greetings, > > In my desire to increase network throughput, and to be able to handle > more then ~250-270kpps > I started experimenting with lagg and link aggregation control > protocol (lacp). > To my surprise this doesn't increase the amount of packets my server > can handle > > Here is what netstat reports: > > netstat -w1 -I lagg0 > input (lagg0) output > packets errs bytes packets errs bytes colls > 267180 0 16030806 254056 0 14735542 0 > 266875 0 16012506 253829 0 14722260 0 > > netstat -w1 -I em0 > input (em0) output > packets errs bytes packets errs bytes colls > 124789 72976 7487340 115329 0 6690468 0 > 126860 67350 7611600 114769 0 6658002 0 > > netstat -w1 -I em2 > input (em2) output > packets errs bytes packets errs bytes colls > 123695 65533 7421700 113575 0 6584856 0 > 130277 62646 7816626 113648 0 6592280 0 > 123545 64171 7412706 113714 0 6596174 0 > > Using lagg doesn't improve situation at all, and also errors are not > reported. > Also using lagg increased content switches: > > procs memory page disk faults cpu > r b w avm fre flt re pi po fr sr ad4 in sy cs us > sy id > 1 0 0 81048 1914640 52 0 0 0 50 0 0 3036 37902 > 13512 1 20 79 > 0 0 0 81048 1914640 13 0 0 0 0 0 0 9582 83 22166 > 0 56 44 > 0 0 0 81048 1914640 13 0 0 0 0 0 0 9594 80 22028 > 0 55 45 > 0 0 0 81048 1914640 13 0 0 0 0 0 0 9593 82 22095 > 0 56 44 > > Top showed for CPU states +55% system, which is quite high? > > I'll use hwpmc and lock_profiling to see where the kernel spends it's > time. > Greetings, Here is what hwpmc shows (without using lagg): % cumulative self self total time seconds seconds calls ms/call ms/call name 14.7 325801.00 325801.00 0 100.00% MD5Transform [1] 8.4 512008.00 186207.00 0 100.00% _mtx_unlock_flags [2] 6.1 646787.00 134779.00 0 100.00% _mtx_lock_flags [3] 5.6 769909.00 123122.00 0 100.00% uma_zalloc_arg [4] 5.0 879853.00 109944.00 0 100.00% rn_match [5] 3.5 957294.00 77441.00 0 100.00% memcpy [6] 3.1 1025989.00 68695.00 0 100.00% bzero [7] 2.8 1087273.00 61284.00 0 100.00% em_encap [8] 2.6 1145231.00 57958.00 0 100.00% ip_output [9] 2.5 1200105.00 54874.00 0 100.00% bus_dmamap_load_mbuf_sg [10] 2.3 1251626.00 51521.00 0 100.00% syncache_add [11] 2.1 1297826.50 46200.50 0 100.00% syncache_lookup [12] 2.1 1343661.50 45835.00 0 100.00% tcp_input [13] 1.8 1383912.00 40250.50 0 100.00% ip_input [14] 1.5 1417997.00 34085.00 0 100.00% syncache_respond [15] 1.5 1451114.50 33117.50 0 100.00% uma_zfree_internal [16] 1.5 1484046.00 32931.50 0 100.00% critical_exit [17] 1.5 1516899.00 32853.00 0 100.00% MD5Update [18] em0: flags=8843 metric 0 mtu 1500 options=19b ether 00:15:17:58:11:a5 inet 10.3.3.1 netmask 0xffffff00 broadcast 10.3.3.255 media: Ethernet autoselect (1000baseTX ) status: active Is it normal so much time to be spent in MD5Transform with tx/rx enabled? LOCK_PROFILING results here - http://89.186.204.158/lock_profiling2.txt -- Best Wishes, Stefan Lambrev ICQ# 24134177 From owner-freebsd-performance@FreeBSD.ORG Mon Feb 4 18:42:55 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4158616A41A for ; Mon, 4 Feb 2008 18:42:55 +0000 (UTC) (envelope-from thompsa@FreeBSD.org) Received: from heff.fud.org.nz (203-109-251-39.static.bliink.ihug.co.nz [203.109.251.39]) by mx1.freebsd.org (Postfix) with ESMTP id DFD1713C442 for ; Mon, 4 Feb 2008 18:42:54 +0000 (UTC) (envelope-from thompsa@FreeBSD.org) Received: by heff.fud.org.nz (Postfix, from userid 1001) id 8A6227776; Tue, 5 Feb 2008 07:29:45 +1300 (NZDT) Date: Tue, 5 Feb 2008 07:29:45 +1300 From: Andrew Thompson To: Stefan Lambrev Message-ID: <20080204182945.GA49276@heff.fud.org.nz> References: <4794E6CC.1050107@moneybookers.com> <47A0B023.5020401@moneybookers.com> <47A3074A.3040409@moneybookers.com> <47A72EAB.6070602@moneybookers.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <47A72EAB.6070602@moneybookers.com> User-Agent: Mutt/1.5.16 (2007-06-09) X-Mailman-Approved-At: Mon, 04 Feb 2008 19:09:43 +0000 Cc: freebsd-performance@freebsd.org Subject: Re: network performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 18:42:55 -0000 On Mon, Feb 04, 2008 at 05:26:35PM +0200, Stefan Lambrev wrote: > Greetings, > > In my desire to increase network throughput, and to be able to handle more > then ~250-270kpps > I started experimenting with lagg and link aggregation control protocol > (lacp). > To my surprise this doesn't increase the amount of packets my server can > handle > > Using lagg doesn't improve situation at all, and also errors are not > reported. > Also using lagg increased content switches: > > Top showed for CPU states +55% system, which is quite high? > > I'll use hwpmc and lock_profiling to see where the kernel spends it's time. Thanks for investigating this. One thing to note is that ip flows from the same connection always go down the same interface, this is because Ethernet is not allowed to reorder frames. The hash uses src-mac, dst-mac, src-ip and dst-ip (see lagg_hashmbuf), make sure when performance testing that your traffic varies in these values. Adding tcp/udp ports to the hashing may help. I look forward to your profiling results. cheers, Andrew From owner-freebsd-performance@FreeBSD.ORG Mon Feb 4 19:36:57 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 591BD16A418 for ; Mon, 4 Feb 2008 19:36:57 +0000 (UTC) (envelope-from Jinmei_Tatuya@isc.org) Received: from mon.jinmei.org (mon.jinmei.org [IPv6:2001:4f8:3:36::162]) by mx1.freebsd.org (Postfix) with ESMTP id 46FFC13C459 for ; Mon, 4 Feb 2008 19:36:57 +0000 (UTC) (envelope-from Jinmei_Tatuya@isc.org) Received: from dhcp-182.sql1.isc.org (unknown [IPv6:2001:4f8:3:bb:217:f2ff:fee0:a91f]) by mon.jinmei.org (Postfix) with ESMTP id 4163633C59; Mon, 4 Feb 2008 11:36:56 -0800 (PST) Date: Mon, 04 Feb 2008 11:36:56 -0800 Message-ID: From: JINMEI Tatuya / =?ISO-2022-JP?B?GyRCP0BMQEMjOkgbKEI=?= To: Attila Nagy In-Reply-To: <47A614E9.4030501@fsn.hu> References: <475B0F3E.5070100@fsn.hu> <479DFE74.8030004@fsn.hu> <479F02A7.9020607@fsn.hu> <47A614E9.4030501@fsn.hu> User-Agent: Wanderlust/2.14.0 (Africa) Emacs/22.0 Mule/5.0 (SAKAKI) MIME-Version: 1.0 (generated by SEMI 1.14.6 - "Maruoka") Content-Type: text/plain; charset=US-ASCII X-Mailman-Approved-At: Mon, 04 Feb 2008 19:45:29 +0000 Cc: freebsd-performance@freebsd.org, bind-users@isc.org Subject: Re: max-cache-size doesn't work with 9.5.0b1 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 19:36:57 -0000 At Sun, 03 Feb 2008 20:24:25 +0100, Attila Nagy wrote: > Yes, if bind was built with threads, the memory usage always grew behind > max-cache-size very quickly. > > Here is the log: > http://people.fsn.hu/~bra/freebsd/bind950-memory-20080203/bind950b1 > the memory usage (RSS, reported by top) in megabytes: > 19:10:37 466 > 19:11:20 522 > 19:11:53 566 > 19:13:06 666 > 19:14:17 766 > > max-cache-size was set to 64M. Hmm. According to the log message, named seems to control the cache memory pretty well so that it doesn't exceed max-cache-size. So, the memory hog should be somewhere else. One obvious explanation is memory leak, of course. If it occurs within named, you should be able to find it by stopping the daemon (memory leak will trigger assertion failure). Another possible scenario is that you're being hit by known memory leak in the built-in statistics HTTP server (unfortunately, this isn't caught by assertion). If you've enabled the feature and are retrieving statistics via HTTP at a very high rate, your server will possibly eat memory avariciously. I actually suspect that this is NOT the likely cause in this case, from the very rapid growth you showed, but if you enable the built-in HTTP server, could you turn it off and try again? BTW, this leak will be fixed in 9.5.0b2. Finally, at the risk of pointing a finger at someone else who's innocent, is it possible that there's leak in FreeBSD's thread library? For example, busy BIND9 caching servers frequently create and destroy mutex locks; if the thread library fails to cleanly free memory for mutex's, the server memory will grow rapidly. p.s. I'm afraid the patch Mark provided in his response won't solve this particular problem from the information we've got so far. --- JINMEI, Tatuya Internet Systems Consortium, Inc. From owner-freebsd-performance@FreeBSD.ORG Mon Feb 4 20:48:28 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 25CA216A419 for ; Mon, 4 Feb 2008 20:48:28 +0000 (UTC) (envelope-from bra@fsn.hu) Received: from people.fsn.hu (people.fsn.hu [195.228.252.137]) by mx1.freebsd.org (Postfix) with ESMTP id DC8A413C469 for ; Mon, 4 Feb 2008 20:48:27 +0000 (UTC) (envelope-from bra@fsn.hu) Received: from [127.0.0.1] (fw.axelero.hu [195.228.243.120]) by people.fsn.hu (Postfix) with ESMTP id 8E0F2717FF; Mon, 4 Feb 2008 21:48:22 +0100 (CET) Message-ID: <47A77A13.6010802@fsn.hu> Date: Mon, 04 Feb 2008 21:48:19 +0100 From: Attila Nagy User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) MIME-Version: 1.0 To: =?UTF-8?B?SklOTUVJIFRhdHV5YSAvIOelnuaYjumBlOWTiQ==?= References: <475B0F3E.5070100@fsn.hu> <479DFE74.8030004@fsn.hu> <479F02A7.9020607@fsn.hu> <47A614E9.4030501@fsn.hu> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Cc: freebsd-performance@freebsd.org, bind-users@isc.org Subject: Re: max-cache-size doesn't work with 9.5.0b1 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 20:48:28 -0000 On 2008.02.04. 20:36, JINMEI Tatuya / 神明達哉 wrote: > At Sun, 03 Feb 2008 20:24:25 +0100, > Attila Nagy wrote: > > >> Yes, if bind was built with threads, the memory usage always grew behind >> max-cache-size very quickly. >> >> Here is the log: >> http://people.fsn.hu/~bra/freebsd/bind950-memory-20080203/bind950b1 >> the memory usage (RSS, reported by top) in megabytes: >> 19:10:37 466 >> 19:11:20 522 >> 19:11:53 566 >> 19:13:06 666 >> 19:14:17 766 >> >> max-cache-size was set to 64M. >> > > Hmm. According to the log message, named seems to control the cache > memory pretty well so that it doesn't exceed max-cache-size. So, the > memory hog should be somewhere else. > > One obvious explanation is memory leak, of course. If it occurs > within named, you should be able to find it by stopping the daemon > (memory leak will trigger assertion failure). > > Another possible scenario is that you're being hit by known memory > leak in the built-in statistics HTTP server (unfortunately, this isn't > caught by assertion). If you've enabled the feature and are > retrieving statistics via HTTP at a very high rate, your server will > possibly eat memory avariciously. I actually suspect that this is NOT > the likely cause in this case, from the very rapid growth you showed, > but if you enable the built-in HTTP server, could you turn it off and > try again? BTW, this leak will be fixed in 9.5.0b2. > I didn't even look after how could I enable the built-in HTTP server, so if it's not on by default, I haven't had it. > Finally, at the risk of pointing a finger at someone else who's > innocent, is it possible that there's leak in FreeBSD's thread > library? For example, busy BIND9 caching servers frequently create > and destroy mutex locks; if the thread library fails to cleanly free > memory for mutex's, the server memory will grow rapidly. > Bind 9.4.2 works fine on the same machine (threaded), if that counts. > p.s. I'm afraid the patch Mark provided in his response won't solve > this particular problem from the information we've got so far. > I will try it nevertheless. From owner-freebsd-performance@FreeBSD.ORG Mon Feb 4 21:16:54 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9858316A41A; Mon, 4 Feb 2008 21:16:54 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: from blah.sun-fish.com (blah.sun-fish.com [217.18.249.150]) by mx1.freebsd.org (Postfix) with ESMTP id 5200913C45B; Mon, 4 Feb 2008 21:16:54 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: by blah.sun-fish.com (Postfix, from userid 1002) id 1F8741B10F39; Mon, 4 Feb 2008 22:16:53 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on blah.cmotd.com X-Spam-Level: X-Spam-Status: No, score=-10.6 required=5.0 tests=ALL_TRUSTED,BAYES_00, NORMAL_HTTP_TO_IP autolearn=ham version=3.2.3 Received: from [10.1.1.2] (unknown [192.168.25.10]) by blah.sun-fish.com (Postfix) with ESMTP id 925E01B10F1F; Mon, 4 Feb 2008 22:16:50 +0100 (CET) Message-ID: <47A780C0.2060201@moneybookers.com> Date: Mon, 04 Feb 2008 23:16:48 +0200 From: Stefan Lambrev User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) MIME-Version: 1.0 To: Andrew Thompson References: <4794E6CC.1050107@moneybookers.com> <47A0B023.5020401@moneybookers.com> <47A3074A.3040409@moneybookers.com> <47A72EAB.6070602@moneybookers.com> <20080204182945.GA49276@heff.fud.org.nz> In-Reply-To: <20080204182945.GA49276@heff.fud.org.nz> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-performance@freebsd.org Subject: Re: network performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 21:16:54 -0000 Andrew Thompson wrote: > On Mon, Feb 04, 2008 at 05:26:35PM +0200, Stefan Lambrev wrote: > >> Greetings, >> >> In my desire to increase network throughput, and to be able to handle more >> then ~250-270kpps >> I started experimenting with lagg and link aggregation control protocol >> (lacp). >> To my surprise this doesn't increase the amount of packets my server can >> handle >> >> Using lagg doesn't improve situation at all, and also errors are not >> reported. >> Also using lagg increased content switches: >> >> Top showed for CPU states +55% system, which is quite high? >> >> I'll use hwpmc and lock_profiling to see where the kernel spends it's time. >> > > Thanks for investigating this. One thing to note is that ip flows from > the same connection always go down the same interface, this is because > Ethernet is not allowed to reorder frames. The hash uses > src-mac, dst-mac, src-ip and dst-ip (see lagg_hashmbuf), make sure when > performance testing that your traffic varies in these values. Adding > tcp/udp ports to the hashing may help. > The traffic, that I generate is with random/spoofed src part, so it is split between interfaces for sure :) Here you can find results when under load from hwpmc and lock_profiling: http://89.186.204.158/lock_profiling-lagg.txt http://89.186.204.158/lagg-gprof.txt From owner-freebsd-performance@FreeBSD.ORG Mon Feb 4 23:03:24 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A72F116A420; Mon, 4 Feb 2008 23:03:24 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: from blah.sun-fish.com (blah.sun-fish.com [217.18.249.150]) by mx1.freebsd.org (Postfix) with ESMTP id 6117313C459; Mon, 4 Feb 2008 23:03:24 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: by blah.sun-fish.com (Postfix, from userid 1002) id 142A71B10F39; Tue, 5 Feb 2008 00:03:23 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on blah.cmotd.com X-Spam-Level: X-Spam-Status: No, score=-10.6 required=5.0 tests=ALL_TRUSTED,BAYES_00, NORMAL_HTTP_TO_IP autolearn=ham version=3.2.3 Received: from [10.1.1.2] (unknown [192.168.25.10]) by blah.sun-fish.com (Postfix) with ESMTP id 253621B10F22; Tue, 5 Feb 2008 00:03:20 +0100 (CET) Message-ID: <47A799A6.3070502@moneybookers.com> Date: Tue, 05 Feb 2008 01:03:02 +0200 From: Stefan Lambrev User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) MIME-Version: 1.0 To: Andrew Thompson References: <4794E6CC.1050107@moneybookers.com> <47A0B023.5020401@moneybookers.com> <47A3074A.3040409@moneybookers.com> <47A72EAB.6070602@moneybookers.com> <20080204182945.GA49276@heff.fud.org.nz> <47A780C0.2060201@moneybookers.com> In-Reply-To: <47A780C0.2060201@moneybookers.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-performance@freebsd.org Subject: Re: network performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 23:03:24 -0000 Stefan Lambrev wrote: > Andrew Thompson wrote: >> On Mon, Feb 04, 2008 at 05:26:35PM +0200, Stefan Lambrev wrote: >> >>> Greetings, >>> >>> In my desire to increase network throughput, and to be able to >>> handle more then ~250-270kpps >>> I started experimenting with lagg and link aggregation control >>> protocol (lacp). >>> To my surprise this doesn't increase the amount of packets my server >>> can handle >>> >>> Using lagg doesn't improve situation at all, and also errors are not >>> reported. >>> Also using lagg increased content switches: >>> >>> Top showed for CPU states +55% system, which is quite high? >>> >>> I'll use hwpmc and lock_profiling to see where the kernel spends >>> it's time. >>> >> >> Thanks for investigating this. One thing to note is that ip flows from >> the same connection always go down the same interface, this is because >> Ethernet is not allowed to reorder frames. The hash uses >> src-mac, dst-mac, src-ip and dst-ip (see lagg_hashmbuf), make sure when >> performance testing that your traffic varies in these values. Adding >> tcp/udp ports to the hashing may help. >> > The traffic, that I generate is with random/spoofed src part, so it is > split between interfaces for sure :) > > Here you can find results when under load from hwpmc and lock_profiling: > http://89.186.204.158/lock_profiling-lagg.txt > http://89.186.204.158/lagg-gprof.txt > http://89.186.204.158/lagg2-gprof.txt I forget this file :) > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to > "freebsd-performance-unsubscribe@freebsd.org" From owner-freebsd-performance@FreeBSD.ORG Mon Feb 4 23:24:05 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2626416A417 for ; Mon, 4 Feb 2008 23:24:05 +0000 (UTC) (envelope-from todorov@paladin.bulgarpress.com) Received: from paladin.bulgarpress.com (paladin.bulgarpress.com [195.24.42.3]) by mx1.freebsd.org (Postfix) with ESMTP id D69ED13C468 for ; Mon, 4 Feb 2008 23:24:04 +0000 (UTC) (envelope-from todorov@paladin.bulgarpress.com) Received: from localhost (localhost [127.0.0.1]) by paladin.bulgarpress.com (Postfix) with ESMTP id 1BEE16424 for ; Tue, 5 Feb 2008 01:08:13 +0200 (EET) X-Virus-Scanned: amavisd-new at paladin.bulgarpress.com Received: from paladin.bulgarpress.com ([195.24.42.3]) by localhost (paladin.bulgarpress.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id nZCqXeB2pBGx for ; Tue, 5 Feb 2008 01:08:12 +0200 (EET) Received: from [192.168.1.2] (77-85-5-99.btc-net.bg [77.85.5.99]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by paladin.bulgarpress.com (Postfix) with ESMTP id EC987641E for ; Tue, 5 Feb 2008 01:08:11 +0200 (EET) Message-ID: <47A79ADA.9050900@paladin.bulgarpress.com> Date: Tue, 05 Feb 2008 01:08:10 +0200 From: Todorov Organization: Powerforge Net User-Agent: Thunderbird 2.0.0.9 (X11/20080122) MIME-Version: 1.0 To: freebsd-performance@freebsd.org X-Enigmail-Version: 0.95.6 Content-Type: text/plain; charset=windows-1251; format=flowed Content-Transfer-Encoding: 7bit Subject: SMP & HTT on 6.3 (P4) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 23:24:05 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, what do you think for HyperThreading (P4 GHz), which serves FBSD 6.3? Now it's disabled by the BIOS but since today I've upgraded the machine 5.5 to 6.3 though if under 6.XX series it worths or not. I've read for performance penalties under 5.XX series w/ HTT on. I'm not going to change to ULE (it's said to be better for HTT awareness). Please advice. If someone has a machine nearby please paste dmesg & top. Regards, Todor -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHp5raibJkIG65HMcRAkw3AKCSAoL0DATESNTW5+Bm8n6hAlHrAwCgw4Y3 +NEJKcuQwQprYU7IgrK4ztc= =Dd8d -----END PGP SIGNATURE----- From owner-freebsd-performance@FreeBSD.ORG Tue Feb 5 11:24:12 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6F5AE16A418; Tue, 5 Feb 2008 11:24:12 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: from blah.sun-fish.com (blah.sun-fish.com [217.18.249.150]) by mx1.freebsd.org (Postfix) with ESMTP id 210FF13C4DB; Tue, 5 Feb 2008 11:24:11 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: by blah.sun-fish.com (Postfix, from userid 1002) id 548961B10F42; Tue, 5 Feb 2008 12:24:10 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on blah.cmotd.com X-Spam-Level: X-Spam-Status: No, score=-10.6 required=5.0 tests=ALL_TRUSTED,BAYES_00, NORMAL_HTTP_TO_IP autolearn=ham version=3.2.3 Received: from hater.haters.org (hater.cmotd.com [192.168.3.125]) by blah.sun-fish.com (Postfix) with ESMTP id 657431B10F31; Tue, 5 Feb 2008 12:24:06 +0100 (CET) Message-ID: <47A84751.8020109@moneybookers.com> Date: Tue, 05 Feb 2008 13:24:01 +0200 From: Stefan Lambrev User-Agent: Thunderbird 2.0.0.9 (X11/20071120) MIME-Version: 1.0 To: Andrew Thompson References: <4794E6CC.1050107@moneybookers.com> <47A0B023.5020401@moneybookers.com> <47A3074A.3040409@moneybookers.com> <47A72EAB.6070602@moneybookers.com> <20080204182945.GA49276@heff.fud.org.nz> <47A780C0.2060201@moneybookers.com> <47A799A6.3070502@moneybookers.com> In-Reply-To: <47A799A6.3070502@moneybookers.com> Content-Type: text/plain; charset=windows-1251; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.91.2/5692/Tue Feb 5 08:21:55 2008 on blah.cmotd.com X-Virus-Status: Clean Cc: freebsd-performance@freebsd.org Subject: Re: network performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Feb 2008 11:24:12 -0000 Greetings, Stefan Lambrev wrote: > Stefan Lambrev wrote: >> Andrew Thompson wrote: >>> On Mon, Feb 04, 2008 at 05:26:35PM +0200, Stefan Lambrev wrote: >>> >>>> Greetings, >>>> >>>> In my desire to increase network throughput, and to be able to >>>> handle more then ~250-270kpps >>>> I started experimenting with lagg and link aggregation control >>>> protocol (lacp). >>>> To my surprise this doesn't increase the amount of packets my >>>> server can handle >>>> >>>> Using lagg doesn't improve situation at all, and also errors are >>>> not reported. >>>> Also using lagg increased content switches: >>>> >>>> Top showed for CPU states +55% system, which is quite high? >>>> >>>> I'll use hwpmc and lock_profiling to see where the kernel spends >>>> it's time. >>>> >>> >>> Thanks for investigating this. One thing to note is that ip flows from >>> the same connection always go down the same interface, this is because >>> Ethernet is not allowed to reorder frames. The hash uses >>> src-mac, dst-mac, src-ip and dst-ip (see lagg_hashmbuf), make sure when >>> performance testing that your traffic varies in these values. Adding >>> tcp/udp ports to the hashing may help. >>> >> The traffic, that I generate is with random/spoofed src part, so it >> is split between interfaces for sure :) >> >> Here you can find results when under load from hwpmc and lock_profiling: >> http://89.186.204.158/lock_profiling-lagg.txt >> http://89.186.204.158/lagg-gprof.txt >> > http://89.186.204.158/lagg2-gprof.txt I forget this file :) > I found that MD5Transform aways uses ~14% (with rx/txcsum enabled or disabled). And when using without lagg MD5Transform pick up to 20% of the time. Is this normal? -- Best Wishes, Stefan Lambrev ICQ# 24134177 From owner-freebsd-performance@FreeBSD.ORG Tue Feb 5 13:28:36 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E260616A418 for ; Tue, 5 Feb 2008 13:28:36 +0000 (UTC) (envelope-from radek@ceskedomeny.cz) Received: from margaret.starnet.cz (margaret.starnet.cz [62.240.182.134]) by mx1.freebsd.org (Postfix) with SMTP id E50BA13C44B for ; Tue, 5 Feb 2008 13:28:35 +0000 (UTC) (envelope-from radek@ceskedomeny.cz) Received: (qmail 40480 invoked from network); 5 Feb 2008 14:00:32 +0100 Received: from unknown (HELO localhost) (62.240.182.17) by margaret.starnet.cz with SMTP; 5 Feb 2008 14:00:32 +0100 Date: Tue, 5 Feb 2008 14:00:18 +0100 From: "Bc. Radek Krejca" X-Mailer: The Bat! (v3.99.3) Professional Organization: STARNET, s. r. o. X-Priority: 3 (Normal) Message-ID: <107794589.20080205140018@starnet.cz> To: FreeBSD Mailing Lists MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Subject: FBSD 1GBit router? X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: "Bc. Radek Krejca" List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Feb 2008 13:28:37 -0000 Hi, I have FreeBSD box as router FreeBSD pvt-gw.starnet.cz 6.1-RELEASE-p12 FreeBSD 6.1-RELEASE-p12 #2: Wed= Jan 31 21:28:44 CET 2007 root@pvt-gw.starnet.cz:/usr/obj/usr/src/sys/D= L360-G4 i386 But speed is only about 382 Mbit. I have following values in sysctl.conf: net.inet.ip.fastforwarding=3D1 net.inet.tcp.recvspace=3D262144 net.inet.tcp.sendspace=3D262144 kern.ipc.maxsockbuf=3D33554432 I need about 600-700 Mbit. Is any chance on freebsd? Hardware is HP DL360-G4, interrupt takes about 55 % of CPU. I tested it over netperf and result is about 382 Mbit. =20 --=20 Regards, Bc. Radek Krejca ICQ: 65895541=20 From owner-freebsd-performance@FreeBSD.ORG Tue Feb 5 18:46:23 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9989816A418; Tue, 5 Feb 2008 18:46:23 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost2.sentex.ca (smarthost2.sentex.ca [205.211.164.50]) by mx1.freebsd.org (Postfix) with ESMTP id 64BE013C4D1; Tue, 5 Feb 2008 18:46:23 +0000 (UTC) (envelope-from mike@sentex.net) Received: from lava.sentex.ca (pyroxene.sentex.ca [199.212.134.18]) by smarthost2.sentex.ca (8.14.2/8.14.2) with ESMTP id m15IkMTX007836; Tue, 5 Feb 2008 13:46:22 -0500 (EST) (envelope-from mike@sentex.net) Received: from mdt-xp.sentex.net (simeon.sentex.ca [192.168.43.27]) by lava.sentex.ca (8.13.8/8.13.3) with ESMTP id m15IkLnn022153 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 5 Feb 2008 13:46:22 -0500 (EST) (envelope-from mike@sentex.net) Message-Id: <200802051846.m15IkLnn022153@lava.sentex.ca> X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Tue, 05 Feb 2008 13:46:14 -0500 To: Kris Kennaway From: Mike Tancsa In-Reply-To: <47A043FD.9090607@FreeBSD.org> References: <200801281024.11571.darcyb@commandprompt.com> <479E3C5E.1070405@FreeBSD.org> <47A043FD.9090607@FreeBSD.org> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Cc: freebsd-performance@freebsd.org Subject: Re: postgresql-performance using sysbench X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Feb 2008 18:46:23 -0000 At 04:31 AM 1/30/2008, Kris Kennaway wrote: >Claus Guttesen wrote: >>>>I forgot to mention in my first post that I'm using ULE. The p800 >>>>controller has a (factory set) 25/75 read/write cache ratio. >>>There's maybe one additional thing: do you dual-boot Linux and FreeBSD? >>>If so, you'll need to set up a separate additional partition for the >>>database, instead of benchmarking it with the file systems used by the >>>OS, because different areas of the drive(s) have different performance - >>>you can verify this with diskinfo -t. >>I installed FreeBSD onto a boot-partition (p400i-controller) and used >>the external storage (p800) as database-partition (eight 15K rpm >>sas-disks in raid 1+0). Same with Ubuntu. When I re-installed FreeBSD >>and ubuntu I wiped and formatted the previous partitions. Ubuntu used >>ext3 which I guess is default fs. > >Write performance is something that we are working on, expect to >hear about progress over the coming weeks/months. I tried both ronly and read write, and using a very large table on RELENG_7, AMD64, ULE, 8G of RAM, Xeon 3060 dual core sysbench --pgsql-host="" --test=oltp --pgsql-user=pgsql --oltp-table-size=29400000 prepare run=3 clients=40 for a in 2 8 16 32 do for ((b=1; b<=$run; b++)) do echo loop $b of clients $a /usr/local/pgsql/bin/psql -d sbtest -U pgsql -c "vacuum analyse;" sysbench --test=oltp --oltp-read-only --num-threads=${a} --pgsql-user=pgsql --pgsql-host="" --max-time=300 --max-requests=10000 --oltp-table-size=29400000 run >> sysbench-clients-linux-ronly-mike-${a} done done This created ~ an 8G table. The database was on a dedicated Areca RAID10 array and the OS was on a separate disk (IDE). The results are pretty similar. At least for me, they are close enough. For my customer, its a lot of reading from a 18G database and not too many writes. Looking at how his app typically runs against FreeBSD and Fedora, there was very little difference as well. For the ronly test the FreeBSD values were all pretty tight, but for the 2 client one, the initial run for Fedora really was crappy. After that, they had pretty similar distributions. The values below are averages for 3 runs and an average for 5 on the rw tests that had smaller table sizes (--oltp-table-size=1900000) Threads FreeBSD Fedora 8 2 1213 (1219.07,1210.66,1211.44) 674 (128.18,746.81,1266.14) 8 1155 1406 16 1118 1358 32 1069 1192 On the RW tests, Threads FreeBSD Fedora 8 2 404 491 8 222 366 16 198 349 32 162 265 This was also without too much tweaking of the FreeBSD side. kern.ipc.shmall=532768 kern.ipc.shmmax=134217728 kern.ipc.semmap=256 shared_buffers = 24MB max_fsm_pages = 153600 update_process_title = off ---Mike From owner-freebsd-performance@FreeBSD.ORG Tue Feb 5 20:33:29 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C082116A418 for ; Tue, 5 Feb 2008 20:33:29 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: from py-out-1112.google.com (py-out-1112.google.com [64.233.166.177]) by mx1.freebsd.org (Postfix) with ESMTP id 64A2913C4D3 for ; Tue, 5 Feb 2008 20:33:29 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: by py-out-1112.google.com with SMTP id u52so3903217pyb.10 for ; Tue, 05 Feb 2008 12:33:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:references; bh=nN3FhHBYpYSyZEDxfzN1DlBFNcqN6thRvSkCA3LNCN8=; b=QFx3H3DvVEkSoixdL25hbhE4Cn9/dmZxm4pjiwysDkY3qYL/Sld+3P59s5GgthtivP2Qe3SfVWyGMLBmCvuUhQW5+HU2gfs+UOAAZR23I2ognkhv0u3wpdw9IeFefuhGM8ea0G9aS9DADXxDB2Xx0UlGjONoQ41ZrU71qaSKBZs= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:references; b=E4EiT2EfE2l/v6UKPYE9g6KQedNaih1xzDCDgpeeb2kVTs9eWrGUUQryfjnK9BwYJTd/Nrce367IWeMoZX2gbK0rVNX9thin6pn6Z4otMwExOYxi8nwx6VNbqXif0TXVcC/N1IzWj5urw0MguYpajAcTOO4rDqybt9kRKdmMnjI= Received: by 10.65.163.8 with SMTP id q8mr16497692qbo.77.1202241929201; Tue, 05 Feb 2008 12:05:29 -0800 (PST) Received: by 10.64.148.4 with HTTP; Tue, 5 Feb 2008 12:05:29 -0800 (PST) Message-ID: <5f67a8c40802051205t74a38663xd692e2a754d3788b@mail.gmail.com> Date: Tue, 5 Feb 2008 15:05:29 -0500 From: "Zaphod Beeblebrox" To: "Bc. Radek Krejca" In-Reply-To: <107794589.20080205140018@starnet.cz> MIME-Version: 1.0 References: <107794589.20080205140018@starnet.cz> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: FreeBSD Mailing Lists Subject: Re: FBSD 1GBit router? X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Feb 2008 20:33:29 -0000 On Feb 5, 2008 8:00 AM, Bc. Radek Krejca wrote: > I have FreeBSD box as router > FreeBSD pvt-gw.starnet.cz 6.1-RELEASE-p12 FreeBSD 6.1-RELEASE-p12 #2: Wed > Jan 31 21:28:44 CET 2007 root@pvt-gw.starnet.cz:/usr/obj/usr/src/sys/DL360-G4 > i386 > But speed is only about 382 Mbit. I have following values in > sysctl.conf: > > net.inet.ip.fastforwarding=1 > net.inet.tcp.recvspace=262144 > net.inet.tcp.sendspace=262144 > kern.ipc.maxsockbuf=33554432 > The ip.fastforwarding makes a tiny insignificant difference with the caveat that your box won't show up on traceroutes. Fast forwarding is "fast" by virtue of the fact that it doesn't decrement TTL. The other sysctl values effect traffic originating or termating at your router --- they probably have no effect on performance. > I need about 600-700 Mbit. Is any chance on freebsd? Hardware is HP > DL360-G4, interrupt takes about 55 % of CPU. I tested it over > netperf and result is about 382 Mbit. If you need large packet performance of 600-700 mbit (large in this case being packets (on average) of 1000 bytes or more), then this hardware is likely doable. I'd do the following: - upgrade at least to 6.3. upgrading to 7.0 might also be better, depending on hardware choices - ensure your ethernet cards are on fast enough busses. 'em' (Intel Ether Express 1000) flavor ports are my personal favorite - enable polling (this will make a _huge_ difference by itself) - your hardware is (likely) dual core. Make sure every piece of hardware in use doesn't involve any giant locks. Under 6.x consider the mpsafenet sysctl. This is also a point on which 7.0 will shine. From owner-freebsd-performance@FreeBSD.ORG Tue Feb 5 21:16:37 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BE51616A419; Tue, 5 Feb 2008 21:16:37 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 902DC13C4E8; Tue, 5 Feb 2008 21:16:35 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <47A8D233.8020506@FreeBSD.org> Date: Tue, 05 Feb 2008 22:16:35 +0100 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: Stefan Lambrev References: <4794E6CC.1050107@moneybookers.com> <47A0B023.5020401@moneybookers.com> <47A3074A.3040409@moneybookers.com> <47A72EAB.6070602@moneybookers.com> <20080204182945.GA49276@heff.fud.org.nz> <47A780C0.2060201@moneybookers.com> <47A799A6.3070502@moneybookers.com> <47A84751.8020109@moneybookers.com> In-Reply-To: <47A84751.8020109@moneybookers.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-performance@freebsd.org, Andrew Thompson Subject: Re: network performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Feb 2008 21:16:37 -0000 Stefan Lambrev wrote: >>>> Thanks for investigating this. One thing to note is that ip flows from >>>> the same connection always go down the same interface, this is because >>>> Ethernet is not allowed to reorder frames. The hash uses >>>> src-mac, dst-mac, src-ip and dst-ip (see lagg_hashmbuf), make sure when >>>> performance testing that your traffic varies in these values. Adding >>>> tcp/udp ports to the hashing may help. >>>> >>> The traffic, that I generate is with random/spoofed src part, so it >>> is split between interfaces for sure :) >>> >>> Here you can find results when under load from hwpmc and lock_profiling: >>> http://89.186.204.158/lock_profiling-lagg.txt OK, this shows the following major problems: 39 22375065 1500649 5690741 3 0 119007 712359 /usr/src/sys/net/route.c:147 (sleep mutex:radix node head) 21 3012732 1905704 1896914 1 1 14102 496427 /usr/src/sys/netinet/ip_output.c:594 (sleep mutex:rtentry) 22 120 2073128 47 2 44109 0 3 /usr/src/sys/modules/if_lagg/../../net/ieee8023ad_lacp.c:503 (rw:if_lagg rwlock) 39 17857439 4262576 5690740 3 0 95072 1484738 /usr/src/sys/net/route.c:197 (sleep mutex:rtentry) It looks like the if_lagg one has been fixed already in 8.0, it could probably be backported but requires some other infrastructure that might not be in 7.0. The others are to do with concurrent transmission of packets (it is doing silly things with route lookups). kmacy has a WIP that fixes this. If you are interested in testing an 8.0 kernel with the fixes let me know. >>> http://89.186.204.158/lagg-gprof.txt >>> >> http://89.186.204.158/lagg2-gprof.txt I forget this file :) >> > I found that MD5Transform aways uses ~14% (with rx/txcsum enabled or > disabled). Yeah, these don't have anything to do with MD5. > And when using without lagg MD5Transform pick up to 20% of the time. > Is this normal? It is probably from the syncache. You could disable it (net.inet.tcp.syncookies_only) if you don't need strong protection against SYN flooding. Kris From owner-freebsd-performance@FreeBSD.ORG Tue Feb 5 22:02:10 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0A34216A419; Tue, 5 Feb 2008 22:02:10 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: from blah.sun-fish.com (blah.sun-fish.com [217.18.249.150]) by mx1.freebsd.org (Postfix) with ESMTP id 7F51413C43E; Tue, 5 Feb 2008 22:02:09 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: by blah.sun-fish.com (Postfix, from userid 1002) id A84B21B10EF8; Tue, 5 Feb 2008 23:02:07 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on blah.cmotd.com X-Spam-Level: X-Spam-Status: No, score=-10.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, J_CHICKENPOX_55,NORMAL_HTTP_TO_IP autolearn=no version=3.2.3 Received: from [10.1.1.2] (unknown [192.168.25.10]) by blah.sun-fish.com (Postfix) with ESMTP id A1C9F1B10EA4; Tue, 5 Feb 2008 23:02:04 +0100 (CET) Message-ID: <47A8DCD6.3060209@moneybookers.com> Date: Wed, 06 Feb 2008 00:01:58 +0200 From: Stefan Lambrev User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) MIME-Version: 1.0 To: Kris Kennaway References: <4794E6CC.1050107@moneybookers.com> <47A0B023.5020401@moneybookers.com> <47A3074A.3040409@moneybookers.com> <47A72EAB.6070602@moneybookers.com> <20080204182945.GA49276@heff.fud.org.nz> <47A780C0.2060201@moneybookers.com> <47A799A6.3070502@moneybookers.com> <47A84751.8020109@moneybookers.com> <47A8D233.8020506@FreeBSD.org> In-Reply-To: <47A8D233.8020506@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.91.2/5700/Tue Feb 5 20:08:59 2008 on blah.cmotd.com X-Virus-Status: Clean Cc: freebsd-performance@freebsd.org, Andrew Thompson Subject: Re: network performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Feb 2008 22:02:10 -0000 Hello, Kris Kennaway wrote: > Stefan Lambrev wrote: > >>>>> Thanks for investigating this. One thing to note is that ip flows >>>>> from >>>>> the same connection always go down the same interface, this is >>>>> because >>>>> Ethernet is not allowed to reorder frames. The hash uses >>>>> src-mac, dst-mac, src-ip and dst-ip (see lagg_hashmbuf), make sure >>>>> when >>>>> performance testing that your traffic varies in these values. Adding >>>>> tcp/udp ports to the hashing may help. >>>>> >>>> The traffic, that I generate is with random/spoofed src part, so it >>>> is split between interfaces for sure :) >>>> >>>> Here you can find results when under load from hwpmc and >>>> lock_profiling: >>>> http://89.186.204.158/lock_profiling-lagg.txt > > OK, this shows the following major problems: > > 39 22375065 1500649 5690741 3 0 119007 > 712359 /usr/src/sys/net/route.c:147 (sleep mutex:radix node head) > 21 3012732 1905704 1896914 1 1 14102 > 496427 /usr/src/sys/netinet/ip_output.c:594 (sleep mutex:rtentry) > 22 120 2073128 47 2 44109 0 > 3 > /usr/src/sys/modules/if_lagg/../../net/ieee8023ad_lacp.c:503 > (rw:if_lagg rwlock) > 39 17857439 4262576 5690740 3 0 95072 > 1484738 /usr/src/sys/net/route.c:197 (sleep mutex:rtentry) > > It looks like the if_lagg one has been fixed already in 8.0, it could > probably be backported but requires some other infrastructure that > might not be in 7.0. > > The others are to do with concurrent transmission of packets (it is > doing silly things with route lookups). kmacy has a WIP that fixes > this. If you are interested in testing an 8.0 kernel with the fixes > let me know. Well those servers are only for tests so I can test everything, but at some point I'll have to make final decision what to use in production :) > >>>> http://89.186.204.158/lagg-gprof.txt >>>> >>> http://89.186.204.158/lagg2-gprof.txt I forget this file :) >>> >> I found that MD5Transform aways uses ~14% (with rx/txcsum enabled or >> disabled). > > Yeah, these don't have anything to do with MD5. Well I didn't find from where MD5Transform() is called, so I guess it's a some 'magic', that I still do not understand ;) > >> And when using without lagg MD5Transform pick up to 20% of the time. >> Is this normal? > > It is probably from the syncache. You could disable it > (net.inet.tcp.syncookies_only) if you don't need strong protection > against SYN flooding. > > Kris How the server perform during SYN flooding is exactly what I test at the moment :) So I can't disable this. Just for information, if someone is interested - I looked how linux (2.6.22-14-generic ubuntu) perform in the same situation .. by default it doesn't perform at all - it hardly replays to 100-200 packets/s, with syncookies enabled it can handle up to 70-90,000 pps (250-270,000 compared to freebsd), but the server is very loaded and not very responsible. Of course this doesn't mean that FreeBSD can't perform better ;) I plan to test iptables, newer kernel, various options, and may be few others distros. From owner-freebsd-performance@FreeBSD.ORG Tue Feb 5 22:11:26 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2A28E16A417 for ; Tue, 5 Feb 2008 22:11:26 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: from blah.sun-fish.com (blah.sun-fish.com [217.18.249.150]) by mx1.freebsd.org (Postfix) with ESMTP id D878C13C469 for ; Tue, 5 Feb 2008 22:11:25 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: by blah.sun-fish.com (Postfix, from userid 1002) id CCCC61B10EF1; Tue, 5 Feb 2008 23:11:24 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on blah.cmotd.com X-Spam-Level: X-Spam-Status: No, score=-10.6 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.2.3 Received: from [10.1.1.2] (unknown [192.168.25.10]) by blah.sun-fish.com (Postfix) with ESMTP id 2516B1B10EA4; Tue, 5 Feb 2008 23:11:19 +0100 (CET) Message-ID: <47A8DF01.5040008@moneybookers.com> Date: Wed, 06 Feb 2008 00:11:13 +0200 From: Stefan Lambrev User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) MIME-Version: 1.0 To: Zaphod Beeblebrox References: <107794589.20080205140018@starnet.cz> <5f67a8c40802051205t74a38663xd692e2a754d3788b@mail.gmail.com> In-Reply-To: <5f67a8c40802051205t74a38663xd692e2a754d3788b@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.91.2/5700/Tue Feb 5 20:08:59 2008 on blah.cmotd.com X-Virus-Status: Clean Cc: FreeBSD Mailing Lists , "Bc. Radek Krejca" Subject: Re: FBSD 1GBit router? X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Feb 2008 22:11:26 -0000 Zaphod Beeblebrox wrote: > On Feb 5, 2008 8:00 AM, Bc. Radek Krejca wrote: > > > >> I have FreeBSD box as router >> FreeBSD pvt-gw.starnet.cz 6.1-RELEASE-p12 FreeBSD 6.1-RELEASE-p12 #2: Wed >> Jan 31 21:28:44 CET 2007 root@pvt-gw.starnet.cz:/usr/obj/usr/src/sys/DL360-G4 >> i386 >> But speed is only about 382 Mbit. I have following values in >> sysctl.conf: >> >> net.inet.ip.fastforwarding=1 >> net.inet.tcp.recvspace=262144 >> net.inet.tcp.sendspace=262144 >> kern.ipc.maxsockbuf=33554432 >> >> > > The ip.fastforwarding makes a tiny insignificant difference with the caveat > that your box won't show up on traceroutes. Fast forwarding is "fast" by > virtue of the fact that it doesn't decrement TTL. > > The other sysctl values effect traffic originating or termating at your > router --- they probably have no effect on performance. > > > >> I need about 600-700 Mbit. Is any chance on freebsd? Hardware is HP >> DL360-G4, interrupt takes about 55 % of CPU. I tested it over >> netperf and result is about 382 Mbit. >> > > > If you need large packet performance of 600-700 mbit (large in this case > being packets (on average) of 1000 bytes or more), then this hardware is > likely doable. I'd do the following: > > - upgrade at least to 6.3. upgrading to 7.0 might also be better, depending > on hardware choices > - ensure your ethernet cards are on fast enough busses. 'em' (Intel Ether > Express 1000) flavor ports are my personal favorite > - enable polling (this will make a _huge_ difference by itself) > Let me disagree with this - while polling reduce CPU utilization it doesn't perform better. When the network is under pressure polling can lead to lost packets. After all one should test how polling works for him, but with polling enabled my intel network cards does not work better. > - your hardware is (likely) dual core. Make sure every piece of hardware in > use doesn't involve any giant locks. Under 6.x consider the mpsafenet > sysctl. This is also a point on which 7.0 will shine. > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" > From owner-freebsd-performance@FreeBSD.ORG Tue Feb 5 22:23:46 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 603AE16A419; Tue, 5 Feb 2008 22:23:46 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 6377F13C455; Tue, 5 Feb 2008 22:23:45 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <47A8E1F1.4040309@FreeBSD.org> Date: Tue, 05 Feb 2008 23:23:45 +0100 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: Stefan Lambrev References: <4794E6CC.1050107@moneybookers.com> <47A0B023.5020401@moneybookers.com> <47A3074A.3040409@moneybookers.com> <47A72EAB.6070602@moneybookers.com> <20080204182945.GA49276@heff.fud.org.nz> <47A780C0.2060201@moneybookers.com> <47A799A6.3070502@moneybookers.com> <47A84751.8020109@moneybookers.com> <47A8D233.8020506@FreeBSD.org> <47A8DCD6.3060209@moneybookers.com> In-Reply-To: <47A8DCD6.3060209@moneybookers.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-performance@freebsd.org, Andrew Thompson Subject: Re: network performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Feb 2008 22:23:46 -0000 Stefan Lambrev wrote: > Hello, > > Kris Kennaway wrote: >> Stefan Lambrev wrote: >> >>>>>> Thanks for investigating this. One thing to note is that ip flows >>>>>> from >>>>>> the same connection always go down the same interface, this is >>>>>> because >>>>>> Ethernet is not allowed to reorder frames. The hash uses >>>>>> src-mac, dst-mac, src-ip and dst-ip (see lagg_hashmbuf), make sure >>>>>> when >>>>>> performance testing that your traffic varies in these values. Adding >>>>>> tcp/udp ports to the hashing may help. >>>>>> >>>>> The traffic, that I generate is with random/spoofed src part, so it >>>>> is split between interfaces for sure :) >>>>> >>>>> Here you can find results when under load from hwpmc and >>>>> lock_profiling: >>>>> http://89.186.204.158/lock_profiling-lagg.txt >> >> OK, this shows the following major problems: >> >> 39 22375065 1500649 5690741 3 0 119007 >> 712359 /usr/src/sys/net/route.c:147 (sleep mutex:radix node head) >> 21 3012732 1905704 1896914 1 1 14102 >> 496427 /usr/src/sys/netinet/ip_output.c:594 (sleep mutex:rtentry) >> 22 120 2073128 47 2 44109 0 >> 3 >> /usr/src/sys/modules/if_lagg/../../net/ieee8023ad_lacp.c:503 >> (rw:if_lagg rwlock) >> 39 17857439 4262576 5690740 3 0 95072 >> 1484738 /usr/src/sys/net/route.c:197 (sleep mutex:rtentry) >> >> It looks like the if_lagg one has been fixed already in 8.0, it could >> probably be backported but requires some other infrastructure that >> might not be in 7.0. >> >> The others are to do with concurrent transmission of packets (it is >> doing silly things with route lookups). kmacy has a WIP that fixes >> this. If you are interested in testing an 8.0 kernel with the fixes >> let me know. > Well those servers are only for tests so I can test everything, but at > some point I'll have to make final decision what to use in production :) http://www.freebsd.org/~kris/p4-net.tbz is a sys/ tarball from my p4 branch, which includes these and other optimizations. >>>>> http://89.186.204.158/lagg-gprof.txt >>>>> >>>> http://89.186.204.158/lagg2-gprof.txt I forget this file :) >>>> >>> I found that MD5Transform aways uses ~14% (with rx/txcsum enabled or >>> disabled). >> >> Yeah, these don't have anything to do with MD5. > Well I didn't find from where MD5Transform() is called, so I guess it's > a some 'magic', that I still do not understand ;) MD5Transform is an internal function called by other MD5* functions. Check netinet/tcp_syncache.c >> It is probably from the syncache. You could disable it >> (net.inet.tcp.syncookies_only) if you don't need strong protection >> against SYN flooding. >> >> Kris > How the server perform during SYN flooding is exactly what I test at the > moment :) > So I can't disable this. I thought this trace was on the machine you are transmitting the SYNs from, perhaps I misunderstood. > Just for information, if someone is interested - I looked how linux > (2.6.22-14-generic ubuntu) perform in the same situation .. by default > it doesn't perform at all - it hardly replays to 100-200 packets/s, > with syncookies enabled it can handle up to 70-90,000 pps (250-270,000 > compared to freebsd), but the server is very loaded and not very > responsible. > Of course this doesn't mean that FreeBSD can't perform better ;) What do you mean "compared to freebsd"? Kris From owner-freebsd-performance@FreeBSD.ORG Wed Feb 6 09:05:33 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A2DC616A41A; Wed, 6 Feb 2008 09:05:33 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: from blah.sun-fish.com (blah.sun-fish.com [217.18.249.150]) by mx1.freebsd.org (Postfix) with ESMTP id 43C5313C458; Wed, 6 Feb 2008 09:05:33 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: by blah.sun-fish.com (Postfix, from userid 1002) id 96F8F1B10EF7; Wed, 6 Feb 2008 10:05:31 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on blah.cmotd.com X-Spam-Level: X-Spam-Status: No, score=-10.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, J_CHICKENPOX_55,NORMAL_HTTP_TO_IP autolearn=no version=3.2.3 Received: from hater.haters.org (hater.cmotd.com [192.168.3.125]) by blah.sun-fish.com (Postfix) with ESMTP id C95D81B10EE0; Wed, 6 Feb 2008 10:05:27 +0100 (CET) Message-ID: <47A97857.10505@moneybookers.com> Date: Wed, 06 Feb 2008 11:05:27 +0200 From: Stefan Lambrev User-Agent: Thunderbird 2.0.0.9 (X11/20071120) MIME-Version: 1.0 To: Kris Kennaway References: <4794E6CC.1050107@moneybookers.com> <47A0B023.5020401@moneybookers.com> <47A3074A.3040409@moneybookers.com> <47A72EAB.6070602@moneybookers.com> <20080204182945.GA49276@heff.fud.org.nz> <47A780C0.2060201@moneybookers.com> <47A799A6.3070502@moneybookers.com> <47A84751.8020109@moneybookers.com> <47A8D233.8020506@FreeBSD.org> <47A8DCD6.3060209@moneybookers.com> <47A8E1F1.4040309@FreeBSD.org> In-Reply-To: <47A8E1F1.4040309@FreeBSD.org> Content-Type: text/plain; charset=windows-1251; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.91.2/5710/Wed Feb 6 07:45:46 2008 on blah.cmotd.com X-Virus-Status: Clean Cc: freebsd-performance@freebsd.org, Andrew Thompson Subject: Re: network performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Feb 2008 09:05:33 -0000 Greetings, Kris Kennaway wrote: > Stefan Lambrev wrote: >> Hello, >> >> Kris Kennaway wrote: >>> Stefan Lambrev wrote: >>> >>>>>>> Thanks for investigating this. One thing to note is that ip >>>>>>> flows from >>>>>>> the same connection always go down the same interface, this is >>>>>>> because >>>>>>> Ethernet is not allowed to reorder frames. The hash uses >>>>>>> src-mac, dst-mac, src-ip and dst-ip (see lagg_hashmbuf), make >>>>>>> sure when >>>>>>> performance testing that your traffic varies in these values. >>>>>>> Adding >>>>>>> tcp/udp ports to the hashing may help. >>>>>>> >>>>>> The traffic, that I generate is with random/spoofed src part, so >>>>>> it is split between interfaces for sure :) >>>>>> >>>>>> Here you can find results when under load from hwpmc and >>>>>> lock_profiling: >>>>>> http://89.186.204.158/lock_profiling-lagg.txt >>> >>> OK, this shows the following major problems: >>> >>> 39 22375065 1500649 5690741 3 0 >>> 119007 712359 /usr/src/sys/net/route.c:147 (sleep mutex:radix >>> node head) >>> 21 3012732 1905704 1896914 1 1 >>> 14102 496427 /usr/src/sys/netinet/ip_output.c:594 (sleep >>> mutex:rtentry) >>> 22 120 2073128 47 2 44109 >>> 0 3 >>> /usr/src/sys/modules/if_lagg/../../net/ieee8023ad_lacp.c:503 >>> (rw:if_lagg rwlock) >>> 39 17857439 4262576 5690740 3 0 >>> 95072 1484738 /usr/src/sys/net/route.c:197 (sleep mutex:rtentry) >>> >>> It looks like the if_lagg one has been fixed already in 8.0, it >>> could probably be backported but requires some other infrastructure >>> that might not be in 7.0. >>> >>> The others are to do with concurrent transmission of packets (it is >>> doing silly things with route lookups). kmacy has a WIP that fixes >>> this. If you are interested in testing an 8.0 kernel with the fixes >>> let me know. >> Well those servers are only for tests so I can test everything, but >> at some point I'll have to make final decision what to use in >> production :) > > http://www.freebsd.org/~kris/p4-net.tbz is a sys/ tarball from my p4 > branch, which includes these and other optimizations. Just downloaded them - will patch my system and test today. > >>>>>> http://89.186.204.158/lagg-gprof.txt >>>>>> >>>>> http://89.186.204.158/lagg2-gprof.txt I forget this file :) >>>>> >>>> I found that MD5Transform aways uses ~14% (with rx/txcsum enabled >>>> or disabled). >>> >>> Yeah, these don't have anything to do with MD5. >> Well I didn't find from where MD5Transform() is called, so I guess >> it's a some 'magic', that I still do not understand ;) > > MD5Transform is an internal function called by other MD5* functions. > Check netinet/tcp_syncache.c Well now I understand why I see the only on the final delivery host and not on the firewall :) > >>> It is probably from the syncache. You could disable it >>> (net.inet.tcp.syncookies_only) if you don't need strong protection >>> against SYN flooding. >>> >>> Kris >> How the server perform during SYN flooding is exactly what I test at >> the moment :) >> So I can't disable this. > > I thought this trace was on the machine you are transmitting the SYNs > from, perhaps I misunderstood. The first traces when we discussed hping was from the machine that is transmitting the SYNs. Now I'm on the next step where I'm trying to survive the SYN flood. That's why lagg + lacp sounds intriguing for me, because em driver is not really SMPable, but if I the traffic is split between two or more network cards, then I'll be able to utilize two or more CPUs. > >> Just for information, if someone is interested - I looked how linux >> (2.6.22-14-generic ubuntu) perform in the same situation .. by >> default it doesn't perform at all - it hardly replays to 100-200 >> packets/s, >> with syncookies enabled it can handle up to 70-90,000 pps >> (250-270,000 compared to freebsd), but the server is very loaded and >> not very responsible. >> Of course this doesn't mean that FreeBSD can't perform better ;) > > What do you mean "compared to freebsd"? I mean that the same hardware when running Linux is able to survive when bombed with 70-90kpps, and when running FreeBSD it can survive 250-270kpps Of course I'm using some default values for this linux distro, so to make the comparison fair, I'll try to tune and linux too. -- Best Wishes, Stefan Lambrev ICQ# 24134177 From owner-freebsd-performance@FreeBSD.ORG Wed Feb 6 10:33:06 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6157916A41A; Wed, 6 Feb 2008 10:33:06 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: from blah.sun-fish.com (blah.sun-fish.com [217.18.249.150]) by mx1.freebsd.org (Postfix) with ESMTP id 10F7813C467; Wed, 6 Feb 2008 10:33:05 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: by blah.sun-fish.com (Postfix, from userid 1002) id 485B51B10F31; Wed, 6 Feb 2008 11:33:04 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on blah.cmotd.com X-Spam-Level: X-Spam-Status: No, score=-10.6 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.2.3 Received: from hater.haters.org (hater.cmotd.com [192.168.3.125]) by blah.sun-fish.com (Postfix) with ESMTP id 186C91B10F29; Wed, 6 Feb 2008 11:33:01 +0100 (CET) Message-ID: <47A98CDC.2090407@moneybookers.com> Date: Wed, 06 Feb 2008 12:33:00 +0200 From: Stefan Lambrev User-Agent: Thunderbird 2.0.0.9 (X11/20071120) MIME-Version: 1.0 To: Kris Kennaway References: <4794E6CC.1050107@moneybookers.com> <47A0B023.5020401@moneybookers.com> <47A3074A.3040409@moneybookers.com> <47A72EAB.6070602@moneybookers.com> <20080204182945.GA49276@heff.fud.org.nz> <47A780C0.2060201@moneybookers.com> <47A799A6.3070502@moneybookers.com> <47A84751.8020109@moneybookers.com> <47A8D233.8020506@FreeBSD.org> <47A8DCD6.3060209@moneybookers.com> <47A8E1F1.4040309@FreeBSD.org> In-Reply-To: <47A8E1F1.4040309@FreeBSD.org> Content-Type: text/plain; charset=windows-1251; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.91.2/5710/Wed Feb 6 07:45:46 2008 on blah.cmotd.com X-Virus-Status: Clean Cc: freebsd-performance@freebsd.org Subject: Re: network performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Feb 2008 10:33:06 -0000 Greetings, Kris Kennaway wrote: > http://www.freebsd.org/~kris/p4-net.tbz is a sys/ tarball from my p4 > branch, which includes these and other optimizations. I have some problems with compiling new kernel: cc -c -O2 -frename-registers -pipe -fno-strict-aliasing -march=nocona -std=c99 -g -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -fformat-extensions -nostdinc -I. -I/usr/src/sys -I/usr/src/sys/contrib/altq -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -fno-common -finline-limit=8000 --param inline-unit-growth=100 --param large-function-growth=1000 -fno-omit-frame-pointer -mcmodel=kernel -mno-red-zone -mfpmath=387 -mno-sse -mno-sse2 -mno-mmx -mno-3dnow -msoft-float -fno-asynchronous-unwind-tables -ffreestanding -Werror /usr/src/sys/ufs/ufs/ufs_lookup.c /usr/src/sys/ufs/ufs/ufs_lookup.c: In function 'ufs_lookup': /usr/src/sys/ufs/ufs/ufs_lookup.c:171: error: 'td' undeclared (first use in this function) /usr/src/sys/ufs/ufs/ufs_lookup.c:171: error: (Each undeclared identifier is reported only once /usr/src/sys/ufs/ufs/ufs_lookup.c:171: error: for each function it appears in.) /usr/src/sys/ufs/ufs/ufs_lookup.c:173:45: error: macro "VOP_LOCK" passed 3 arguments, but takes just 2 /usr/src/sys/ufs/ufs/ufs_lookup.c:173: error: 'VOP_LOCK' undeclared (first use in this function) *** Error code 1 -- Best Wishes, Stefan Lambrev ICQ# 24134177 From owner-freebsd-performance@FreeBSD.ORG Wed Feb 6 11:02:42 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2A65516A419 for ; Wed, 6 Feb 2008 11:02:42 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 6898E13C457; Wed, 6 Feb 2008 11:02:41 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <47A993D0.1060901@FreeBSD.org> Date: Wed, 06 Feb 2008 12:02:40 +0100 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: Stefan Lambrev References: <4794E6CC.1050107@moneybookers.com> <47A0B023.5020401@moneybookers.com> <47A3074A.3040409@moneybookers.com> <47A72EAB.6070602@moneybookers.com> <20080204182945.GA49276@heff.fud.org.nz> <47A780C0.2060201@moneybookers.com> <47A799A6.3070502@moneybookers.com> <47A84751.8020109@moneybookers.com> <47A8D233.8020506@FreeBSD.org> <47A8DCD6.3060209@moneybookers.com> <47A8E1F1.4040309@FreeBSD.org> <47A98CDC.2090407@moneybookers.com> In-Reply-To: <47A98CDC.2090407@moneybookers.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-performance@freebsd.org Subject: Re: network performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Feb 2008 11:02:42 -0000 Stefan Lambrev wrote: > Greetings, > > Kris Kennaway wrote: >> http://www.freebsd.org/~kris/p4-net.tbz is a sys/ tarball from my p4 >> branch, which includes these and other optimizations. > I have some problems with compiling new kernel: > > cc -c -O2 -frename-registers -pipe -fno-strict-aliasing -march=nocona > -std=c99 -g -Wall -Wredundant-decls -Wnested-externs > -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline > -Wcast-qual -Wundef -Wno-pointer-sign -fformat-extensions -nostdinc > -I. -I/usr/src/sys -I/usr/src/sys/contrib/altq -D_KERNEL > -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -fno-common > -finline-limit=8000 --param inline-unit-growth=100 --param > large-function-growth=1000 -fno-omit-frame-pointer -mcmodel=kernel > -mno-red-zone -mfpmath=387 -mno-sse -mno-sse2 -mno-mmx -mno-3dnow > -msoft-float -fno-asynchronous-unwind-tables -ffreestanding -Werror > /usr/src/sys/ufs/ufs/ufs_lookup.c > /usr/src/sys/ufs/ufs/ufs_lookup.c: In function 'ufs_lookup': > /usr/src/sys/ufs/ufs/ufs_lookup.c:171: error: 'td' undeclared (first use > in this function) > /usr/src/sys/ufs/ufs/ufs_lookup.c:171: error: (Each undeclared > identifier is reported only once > /usr/src/sys/ufs/ufs/ufs_lookup.c:171: error: for each function it > appears in.) > /usr/src/sys/ufs/ufs/ufs_lookup.c:173:45: error: macro "VOP_LOCK" passed > 3 arguments, but takes just 2 > /usr/src/sys/ufs/ufs/ufs_lookup.c:173: error: 'VOP_LOCK' undeclared > (first use in this function) > *** Error code 1 > > Sorry, forgot to check in a fix. Apply this patch: http://www.freebsd.org/~kris/ufs_lookup.c.diff (you'll need to specify the path by hand) Kris From owner-freebsd-performance@FreeBSD.ORG Wed Feb 6 11:17:15 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5706916A419; Wed, 6 Feb 2008 11:17:15 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: from blah.sun-fish.com (blah.sun-fish.com [217.18.249.150]) by mx1.freebsd.org (Postfix) with ESMTP id C15FD13C44B; Wed, 6 Feb 2008 11:17:14 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: by blah.sun-fish.com (Postfix, from userid 1002) id AEC3F1B10F3B; Wed, 6 Feb 2008 12:17:13 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on blah.cmotd.com X-Spam-Level: X-Spam-Status: No, score=-10.6 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.2.3 Received: from hater.haters.org (hater.cmotd.com [192.168.3.125]) by blah.sun-fish.com (Postfix) with ESMTP id 82CDB1B10F29; Wed, 6 Feb 2008 12:17:10 +0100 (CET) Message-ID: <47A99736.8060809@moneybookers.com> Date: Wed, 06 Feb 2008 13:17:10 +0200 From: Stefan Lambrev User-Agent: Thunderbird 2.0.0.9 (X11/20071120) MIME-Version: 1.0 To: Kris Kennaway References: <4794E6CC.1050107@moneybookers.com> <47A0B023.5020401@moneybookers.com> <47A3074A.3040409@moneybookers.com> <47A72EAB.6070602@moneybookers.com> <20080204182945.GA49276@heff.fud.org.nz> <47A780C0.2060201@moneybookers.com> <47A799A6.3070502@moneybookers.com> <47A84751.8020109@moneybookers.com> <47A8D233.8020506@FreeBSD.org> <47A8DCD6.3060209@moneybookers.com> <47A8E1F1.4040309@FreeBSD.org> <47A98CDC.2090407@moneybookers.com> <47A993D0.1060901@FreeBSD.org> In-Reply-To: <47A993D0.1060901@FreeBSD.org> Content-Type: text/plain; charset=windows-1251; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.91.2/5710/Wed Feb 6 07:45:46 2008 on blah.cmotd.com X-Virus-Status: Clean Cc: freebsd-performance@freebsd.org Subject: Re: network performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Feb 2008 11:17:15 -0000 Kris Kennaway wrote: > Stefan Lambrev wrote: >> Greetings, >> >> Kris Kennaway wrote: >>> http://www.freebsd.org/~kris/p4-net.tbz is a sys/ tarball from my p4 >>> branch, which includes these and other optimizations. >> I have some problems with compiling new kernel: >> >> cc -c -O2 -frename-registers -pipe -fno-strict-aliasing -march=nocona >> -std=c99 -g -Wall -Wredundant-decls -Wnested-externs >> -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline >> -Wcast-qual -Wundef -Wno-pointer-sign -fformat-extensions -nostdinc >> -I. -I/usr/src/sys -I/usr/src/sys/contrib/altq -D_KERNEL >> -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -fno-common >> -finline-limit=8000 --param inline-unit-growth=100 --param >> large-function-growth=1000 -fno-omit-frame-pointer -mcmodel=kernel >> -mno-red-zone -mfpmath=387 -mno-sse -mno-sse2 -mno-mmx -mno-3dnow >> -msoft-float -fno-asynchronous-unwind-tables -ffreestanding -Werror >> /usr/src/sys/ufs/ufs/ufs_lookup.c >> /usr/src/sys/ufs/ufs/ufs_lookup.c: In function 'ufs_lookup': >> /usr/src/sys/ufs/ufs/ufs_lookup.c:171: error: 'td' undeclared (first >> use in this function) >> /usr/src/sys/ufs/ufs/ufs_lookup.c:171: error: (Each undeclared >> identifier is reported only once >> /usr/src/sys/ufs/ufs/ufs_lookup.c:171: error: for each function it >> appears in.) >> /usr/src/sys/ufs/ufs/ufs_lookup.c:173:45: error: macro "VOP_LOCK" >> passed 3 arguments, but takes just 2 >> /usr/src/sys/ufs/ufs/ufs_lookup.c:173: error: 'VOP_LOCK' undeclared >> (first use in this function) >> *** Error code 1 >> >> > > Sorry, forgot to check in a fix. Apply this patch: > > http://www.freebsd.org/~kris/ufs_lookup.c.diff > > (you'll need to specify the path by hand) > > Kris Now this compiles, but : cc -O2 -fno-strict-aliasing -pipe -march=nocona -Werror -D_KERNEL -DKLD_MODULE -std=c99 -nostdinc -DHAVE_KERNEL_OPTION_HEADERS -include /usr/obj/usr/src/sys/CORE/opt_global.h -I. -I@ -I@/contrib/altq -finline-limit=8000 --param inline-unit-growth=100 --param large-function-growth=1000 -fno-common -g -fno-omit-frame-pointer -I/usr/obj/usr/src/sys/CORE -mcmodel=kernel -mno-red-zone -mfpmath=387 -mno-sse -mno-sse2 -mno-mmx -mno-3dnow -msoft-float -fno-asynchronous-unwind-tables -ffreestanding -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -fformat-extensions -c /usr/src/sys/modules/mac_lomac/../../security/mac_lomac/mac_lomac.c cc1: warnings being treated as errors /usr/src/sys/modules/mac_lomac/../../security/mac_lomac/mac_lomac.c: In function 'lomac_thread_userret': /usr/src/sys/modules/mac_lomac/../../security/mac_lomac/mac_lomac.c:2172: warning: implicit declaration of function 'PROC_LOCK' /usr/src/sys/modules/mac_lomac/../../security/mac_lomac/mac_lomac.c:2172: warning: nested extern declaration of 'PROC_LOCK' /usr/src/sys/modules/mac_lomac/../../security/mac_lomac/mac_lomac.c:2190: warning: implicit declaration of function 'PROC_UNLOCK' /usr/src/sys/modules/mac_lomac/../../security/mac_lomac/mac_lomac.c:2190: warning: nested extern declaration of 'PROC_UNLOCK' *** Error code 1 Stop in /usr/src/sys/modules/mac_lomac. where -Werror is defined - in the Makefile? :) Is it safe to just remove it for security/ and continue with build? BTW I removed "options ADAPTIVE_GIANT" as it is unknown with your sources. -- Best Wishes, Stefan Lambrev ICQ# 24134177 From owner-freebsd-performance@FreeBSD.ORG Wed Feb 6 11:33:45 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 14A7E16A468 for ; Wed, 6 Feb 2008 11:33:44 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 26F4E13C467; Wed, 6 Feb 2008 11:33:43 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <47A99B16.6030305@FreeBSD.org> Date: Wed, 06 Feb 2008 12:33:42 +0100 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: Stefan Lambrev References: <4794E6CC.1050107@moneybookers.com> <47A0B023.5020401@moneybookers.com> <47A3074A.3040409@moneybookers.com> <47A72EAB.6070602@moneybookers.com> <20080204182945.GA49276@heff.fud.org.nz> <47A780C0.2060201@moneybookers.com> <47A799A6.3070502@moneybookers.com> <47A84751.8020109@moneybookers.com> <47A8D233.8020506@FreeBSD.org> <47A8DCD6.3060209@moneybookers.com> <47A8E1F1.4040309@FreeBSD.org> <47A98CDC.2090407@moneybookers.com> <47A993D0.1060901@FreeBSD.org> <47A99736.8060809@moneybookers.com> In-Reply-To: <47A99736.8060809@moneybookers.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-performance@freebsd.org Subject: Re: network performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Feb 2008 11:33:45 -0000 Stefan Lambrev wrote: > > > Kris Kennaway wrote: >> Stefan Lambrev wrote: >>> Greetings, >>> >>> Kris Kennaway wrote: >>>> http://www.freebsd.org/~kris/p4-net.tbz is a sys/ tarball from my p4 >>>> branch, which includes these and other optimizations. >>> I have some problems with compiling new kernel: >>> >>> cc -c -O2 -frename-registers -pipe -fno-strict-aliasing -march=nocona >>> -std=c99 -g -Wall -Wredundant-decls -Wnested-externs >>> -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline >>> -Wcast-qual -Wundef -Wno-pointer-sign -fformat-extensions -nostdinc >>> -I. -I/usr/src/sys -I/usr/src/sys/contrib/altq -D_KERNEL >>> -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -fno-common >>> -finline-limit=8000 --param inline-unit-growth=100 --param >>> large-function-growth=1000 -fno-omit-frame-pointer -mcmodel=kernel >>> -mno-red-zone -mfpmath=387 -mno-sse -mno-sse2 -mno-mmx -mno-3dnow >>> -msoft-float -fno-asynchronous-unwind-tables -ffreestanding -Werror >>> /usr/src/sys/ufs/ufs/ufs_lookup.c >>> /usr/src/sys/ufs/ufs/ufs_lookup.c: In function 'ufs_lookup': >>> /usr/src/sys/ufs/ufs/ufs_lookup.c:171: error: 'td' undeclared (first >>> use in this function) >>> /usr/src/sys/ufs/ufs/ufs_lookup.c:171: error: (Each undeclared >>> identifier is reported only once >>> /usr/src/sys/ufs/ufs/ufs_lookup.c:171: error: for each function it >>> appears in.) >>> /usr/src/sys/ufs/ufs/ufs_lookup.c:173:45: error: macro "VOP_LOCK" >>> passed 3 arguments, but takes just 2 >>> /usr/src/sys/ufs/ufs/ufs_lookup.c:173: error: 'VOP_LOCK' undeclared >>> (first use in this function) >>> *** Error code 1 >>> >>> >> >> Sorry, forgot to check in a fix. Apply this patch: >> >> http://www.freebsd.org/~kris/ufs_lookup.c.diff >> >> (you'll need to specify the path by hand) >> >> Kris > Now this compiles, but : > > cc -O2 -fno-strict-aliasing -pipe -march=nocona -Werror -D_KERNEL > -DKLD_MODULE -std=c99 -nostdinc -DHAVE_KERNEL_OPTION_HEADERS -include > /usr/obj/usr/src/sys/CORE/opt_global.h -I. -I@ -I@/contrib/altq > -finline-limit=8000 --param inline-unit-growth=100 --param > large-function-growth=1000 -fno-common -g -fno-omit-frame-pointer > -I/usr/obj/usr/src/sys/CORE -mcmodel=kernel -mno-red-zone -mfpmath=387 > -mno-sse -mno-sse2 -mno-mmx -mno-3dnow -msoft-float > -fno-asynchronous-unwind-tables -ffreestanding -Wall -Wredundant-decls > -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes > -Wpointer-arith -Winline -Wcast-qual -Wundef -Wno-pointer-sign > -fformat-extensions -c > /usr/src/sys/modules/mac_lomac/../../security/mac_lomac/mac_lomac.c > cc1: warnings being treated as errors > /usr/src/sys/modules/mac_lomac/../../security/mac_lomac/mac_lomac.c: In > function 'lomac_thread_userret': > /usr/src/sys/modules/mac_lomac/../../security/mac_lomac/mac_lomac.c:2172: > warning: implicit declaration of function 'PROC_LOCK' > /usr/src/sys/modules/mac_lomac/../../security/mac_lomac/mac_lomac.c:2172: > warning: nested extern declaration of 'PROC_LOCK' > /usr/src/sys/modules/mac_lomac/../../security/mac_lomac/mac_lomac.c:2190: > warning: implicit declaration of function 'PROC_UNLOCK' > /usr/src/sys/modules/mac_lomac/../../security/mac_lomac/mac_lomac.c:2190: > warning: nested extern declaration of 'PROC_UNLOCK' > *** Error code 1 > > Stop in /usr/src/sys/modules/mac_lomac. > > where -Werror is defined - in the Makefile? :) > Is it safe to just remove it for security/ and continue with build? > > BTW I removed "options ADAPTIVE_GIANT" as it is unknown with your > sources. Yes, it is gone with 8.0. Disable the module builds because some of them like this one probably need compile fixes. If you need a subset of modules use MODULES_OVERRIDE=list (in /etc/make.conf) Kris From owner-freebsd-performance@FreeBSD.ORG Wed Feb 6 13:29:32 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 82E6716A41B; Wed, 6 Feb 2008 13:29:32 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: from blah.sun-fish.com (blah.sun-fish.com [217.18.249.150]) by mx1.freebsd.org (Postfix) with ESMTP id 0540913C45B; Wed, 6 Feb 2008 13:29:31 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: by blah.sun-fish.com (Postfix, from userid 1002) id 225FE1B10F2B; Wed, 6 Feb 2008 14:29:30 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on blah.cmotd.com X-Spam-Level: X-Spam-Status: No, score=-10.6 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.2.3 Received: from hater.haters.org (hater.cmotd.com [192.168.3.125]) by blah.sun-fish.com (Postfix) with ESMTP id 4477D1B10F1F; Wed, 6 Feb 2008 14:29:27 +0100 (CET) Message-ID: <47A9B636.3040509@moneybookers.com> Date: Wed, 06 Feb 2008 15:29:26 +0200 From: Stefan Lambrev User-Agent: Thunderbird 2.0.0.9 (X11/20071120) MIME-Version: 1.0 To: Kris Kennaway References: <4794E6CC.1050107@moneybookers.com> <47A0B023.5020401@moneybookers.com> <47A3074A.3040409@moneybookers.com> <47A72EAB.6070602@moneybookers.com> <20080204182945.GA49276@heff.fud.org.nz> <47A780C0.2060201@moneybookers.com> <47A799A6.3070502@moneybookers.com> <47A84751.8020109@moneybookers.com> <47A8D233.8020506@FreeBSD.org> <47A8DCD6.3060209@moneybookers.com> <47A8E1F1.4040309@FreeBSD.org> <47A98CDC.2090407@moneybookers.com> <47A993D0.1060901@FreeBSD.org> <47A99736.8060809@moneybookers.com> <47A99B16.6030305@FreeBSD.org> In-Reply-To: <47A99B16.6030305@FreeBSD.org> Content-Type: text/plain; charset=windows-1251; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.91.2/5711/Wed Feb 6 12:22:58 2008 on blah.cmotd.com X-Virus-Status: Clean Cc: freebsd-performance@freebsd.org Subject: Re: network performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Feb 2008 13:29:32 -0000 Greetings, Kris Kennaway wrote: > Yes, it is gone with 8.0. Disable the module builds because some of > them like this one probably need compile fixes. If you need a subset > of modules use MODULES_OVERRIDE=list (in /etc/make.conf) > Yes, kernel builds. I'm still playing with it, but the first results shows that new kernel can handle 800k incoming packets (well may be more but I have not enough power right now to generate more packets). It still answer only to 250K-260K. I guess I'm hitting the limitation of syncache/syncookies ? Anyway this netisr2 looks like huge improvement :) I can't build kernel without option LOCK_PROFILING with your sources: make -V CFILES -V SYSTEM_CFILES -V GEN_CFILES | MKDEP_CPP="cc -E" CC="cc" xargs mkdep -a -f .newdep -O2 -frename-registers -pipe -fno-strict-aliasing -march=nocona -std=c99 -g -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -fformat-extensions -nostdinc -I. -I/usr/src/sys -I/usr/src/sys/contrib/altq -I/usr/src/sys/contrib/ipfilter -I/usr/src/sys/contrib/pf -I/usr/src/sys/dev/ath -I/usr/src/sys/contrib/ngatm -I/usr/src/sys/dev/twa -I/usr/src/sys/gnu/fs/xfs/FreeBSD -I/usr/src/sys/gnu/fs/xfs/FreeBSD/support -I/usr/src/sys/gnu/fs/xfs -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -fno-common -finline-limit=8000 --param inline-unit-growth=100 --param large-function-growth=1000 -mcmodel=kernel -mno-red-zone -mfpmath=387 -mno-sse -mno-sse2 -mno-mmx -mno-3dnow -msoft-float -fno-asynchronous-unwind-tables -ffreestanding In file included from /usr/src/sys/netinet/ip_output.c:47: /usr/src/sys/sys/rwlock.h:153:2: error: #error LOCK_DEBUG not defined, include before mkdep: compile failed *** Error code 1 So I added #include , rebuild kernel and tested again w/o LOCK_PROFILING, but results are the same. I'll use again hwpmc and LOCK_PROFILING to see what's going on. And will try the same benchmark on quad core processor as now numbers of cores/cpus matter :) -- Best Wishes, Stefan Lambrev ICQ# 24134177 From owner-freebsd-performance@FreeBSD.ORG Wed Feb 6 14:29:20 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 516AD16A468; Wed, 6 Feb 2008 14:29:20 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: from blah.sun-fish.com (blah.sun-fish.com [217.18.249.150]) by mx1.freebsd.org (Postfix) with ESMTP id A84AA13C44B; Wed, 6 Feb 2008 14:29:19 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: by blah.sun-fish.com (Postfix, from userid 1002) id 9B8E01B10F2C; Wed, 6 Feb 2008 15:29:18 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on blah.cmotd.com X-Spam-Level: X-Spam-Status: No, score=-10.6 required=5.0 tests=ALL_TRUSTED,BAYES_00, NORMAL_HTTP_TO_IP autolearn=ham version=3.2.3 Received: from hater.haters.org (hater.cmotd.com [192.168.3.125]) by blah.sun-fish.com (Postfix) with ESMTP id D70341B10F22; Wed, 6 Feb 2008 15:29:14 +0100 (CET) Message-ID: <47A9C43A.3030203@moneybookers.com> Date: Wed, 06 Feb 2008 16:29:14 +0200 From: Stefan Lambrev User-Agent: Thunderbird 2.0.0.9 (X11/20071120) MIME-Version: 1.0 To: Kris Kennaway References: <4794E6CC.1050107@moneybookers.com> <47A0B023.5020401@moneybookers.com> <47A3074A.3040409@moneybookers.com> <47A72EAB.6070602@moneybookers.com> <20080204182945.GA49276@heff.fud.org.nz> <47A780C0.2060201@moneybookers.com> <47A799A6.3070502@moneybookers.com> <47A84751.8020109@moneybookers.com> <47A8D233.8020506@FreeBSD.org> <47A8DCD6.3060209@moneybookers.com> <47A8E1F1.4040309@FreeBSD.org> <47A98CDC.2090407@moneybookers.com> <47A993D0.1060901@FreeBSD.org> <47A99736.8060809@moneybookers.com> <47A99B16.6030305@FreeBSD.org> <47A9B636.3040509@moneybookers.com> In-Reply-To: <47A9B636.3040509@moneybookers.com> Content-Type: text/plain; charset=windows-1251; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.91.2/5711/Wed Feb 6 12:22:58 2008 on blah.cmotd.com X-Virus-Status: Clean Cc: freebsd-performance@freebsd.org Subject: Re: network performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Feb 2008 14:29:20 -0000 Greetings, Stefan Lambrev wrote: > Greetings, > > Kris Kennaway wrote: >> Yes, it is gone with 8.0. Disable the module builds because some of >> them like this one probably need compile fixes. If you need a subset >> of modules use MODULES_OVERRIDE=list (in /etc/make.conf) >> > Yes, kernel builds. > I'm still playing with it, but the first results shows that new kernel > can handle 800k incoming packets (well may be more but I have not > enough power right now to generate more packets). > It still answer only to 250K-260K. I guess I'm hitting the limitation > of syncache/syncookies ? > Anyway this netisr2 looks like huge improvement :) > > I can't build kernel without option LOCK_PROFILING with your sources: > > make -V CFILES -V SYSTEM_CFILES -V GEN_CFILES | MKDEP_CPP="cc -E" > CC="cc" xargs mkdep -a -f .newdep -O2 -frename-registers -pipe > -fno-strict-aliasing -march=nocona -std=c99 -g -Wall -Wredundant-decls > -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes > -Wpointer-arith -Winline -Wcast-qual -Wundef -Wno-pointer-sign > -fformat-extensions -nostdinc -I. -I/usr/src/sys > -I/usr/src/sys/contrib/altq -I/usr/src/sys/contrib/ipfilter > -I/usr/src/sys/contrib/pf -I/usr/src/sys/dev/ath > -I/usr/src/sys/contrib/ngatm -I/usr/src/sys/dev/twa > -I/usr/src/sys/gnu/fs/xfs/FreeBSD > -I/usr/src/sys/gnu/fs/xfs/FreeBSD/support -I/usr/src/sys/gnu/fs/xfs > -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h > -fno-common -finline-limit=8000 --param inline-unit-growth=100 --param > large-function-growth=1000 -mcmodel=kernel -mno-red-zone > -mfpmath=387 -mno-sse -mno-sse2 -mno-mmx -mno-3dnow -msoft-float > -fno-asynchronous-unwind-tables -ffreestanding > In file included from /usr/src/sys/netinet/ip_output.c:47: > /usr/src/sys/sys/rwlock.h:153:2: error: #error LOCK_DEBUG not defined, > include before > mkdep: compile failed > *** Error code 1 > > So I added #include , rebuild kernel and tested again w/o > LOCK_PROFILING, but results are the same. > > I'll use again hwpmc and LOCK_PROFILING to see what's going on. > And will try the same benchmark on quad core processor as now numbers > of cores/cpus matter :) > Here are promised results - http://89.186.204.158/lock_profiling-8.txt Btw I got kernel panic first time when I run sysctl debug.lock.prof.stats I'm still trying to get hwpmc working with my cpu's and new kernel. Do you have any patches Kris? Is it supposed to work with your sources on my CPU? I can fetch your latest src/lib/libpmc from from p4 if this will help :) -- Best Wishes, Stefan Lambrev ICQ# 24134177 From owner-freebsd-performance@FreeBSD.ORG Wed Feb 6 20:07:50 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9F00C16A476 for ; Wed, 6 Feb 2008 20:07:50 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id C147213C4FA; Wed, 6 Feb 2008 20:07:49 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <47AA1395.2090501@FreeBSD.org> Date: Wed, 06 Feb 2008 21:07:49 +0100 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: Stefan Lambrev References: <4794E6CC.1050107@moneybookers.com> <47A0B023.5020401@moneybookers.com> <47A3074A.3040409@moneybookers.com> <47A72EAB.6070602@moneybookers.com> <20080204182945.GA49276@heff.fud.org.nz> <47A780C0.2060201@moneybookers.com> <47A799A6.3070502@moneybookers.com> <47A84751.8020109@moneybookers.com> <47A8D233.8020506@FreeBSD.org> <47A8DCD6.3060209@moneybookers.com> <47A8E1F1.4040309@FreeBSD.org> <47A98CDC.2090407@moneybookers.com> <47A993D0.1060901@FreeBSD.org> <47A99736.8060809@moneybookers.com> <47A99B16.6030305@FreeBSD.org> <47A9B636.3040509@moneybookers.com> In-Reply-To: <47A9B636.3040509@moneybookers.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-performance@freebsd.org Subject: Re: network performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Feb 2008 20:07:50 -0000 Stefan Lambrev wrote: > Greetings, > > Kris Kennaway wrote: >> Yes, it is gone with 8.0. Disable the module builds because some of >> them like this one probably need compile fixes. If you need a subset >> of modules use MODULES_OVERRIDE=list (in /etc/make.conf) >> > Yes, kernel builds. > I'm still playing with it, but the first results shows that new kernel > can handle 800k incoming packets (well may be more but I have not enough > power right now to generate more packets). > It still answer only to 250K-260K. I guess I'm hitting the limitation of > syncache/syncookies ? Yes, it could be. You may need to tune the net.inet.tcp.syncache parameters to get better performance. That is good news though. > Anyway this netisr2 looks like huge improvement :) Actually I forgot to mention: you probably want to set net.isr2.direct=1. > I can't build kernel without option LOCK_PROFILING with your sources: > > make -V CFILES -V SYSTEM_CFILES -V GEN_CFILES | MKDEP_CPP="cc -E" > CC="cc" xargs mkdep -a -f .newdep -O2 -frename-registers -pipe > -fno-strict-aliasing -march=nocona -std=c99 -g -Wall -Wredundant-decls > -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes > -Wpointer-arith -Winline -Wcast-qual -Wundef -Wno-pointer-sign > -fformat-extensions -nostdinc -I. -I/usr/src/sys > -I/usr/src/sys/contrib/altq -I/usr/src/sys/contrib/ipfilter > -I/usr/src/sys/contrib/pf -I/usr/src/sys/dev/ath > -I/usr/src/sys/contrib/ngatm -I/usr/src/sys/dev/twa > -I/usr/src/sys/gnu/fs/xfs/FreeBSD > -I/usr/src/sys/gnu/fs/xfs/FreeBSD/support -I/usr/src/sys/gnu/fs/xfs > -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -fno-common > -finline-limit=8000 --param inline-unit-growth=100 --param > large-function-growth=1000 -mcmodel=kernel -mno-red-zone -mfpmath=387 > -mno-sse -mno-sse2 -mno-mmx -mno-3dnow -msoft-float > -fno-asynchronous-unwind-tables -ffreestanding > In file included from /usr/src/sys/netinet/ip_output.c:47: > /usr/src/sys/sys/rwlock.h:153:2: error: #error LOCK_DEBUG not defined, > include before > mkdep: compile failed > *** Error code 1 > > So I added #include , rebuild kernel and tested again w/o > LOCK_PROFILING, but results are the same. Thanks, I think I forgot to check in another fix. As you found though, LOCK_PROFILING does not have a large performance impact when compiled in but not active. > I'll use again hwpmc and LOCK_PROFILING to see what's going on. > And will try the same benchmark on quad core processor as now numbers of > cores/cpus matter :) That will certainly be interesting to test! Kris From owner-freebsd-performance@FreeBSD.ORG Wed Feb 6 20:28:12 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 045A916A41B for ; Wed, 6 Feb 2008 20:28:12 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id D55CC13C43E; Wed, 6 Feb 2008 20:28:09 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <47AA1858.3050307@FreeBSD.org> Date: Wed, 06 Feb 2008 21:28:08 +0100 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: Stefan Lambrev References: <4794E6CC.1050107@moneybookers.com> <47A0B023.5020401@moneybookers.com> <47A3074A.3040409@moneybookers.com> <47A72EAB.6070602@moneybookers.com> <20080204182945.GA49276@heff.fud.org.nz> <47A780C0.2060201@moneybookers.com> <47A799A6.3070502@moneybookers.com> <47A84751.8020109@moneybookers.com> <47A8D233.8020506@FreeBSD.org> <47A8DCD6.3060209@moneybookers.com> <47A8E1F1.4040309@FreeBSD.org> <47A98CDC.2090407@moneybookers.com> <47A993D0.1060901@FreeBSD.org> <47A99736.8060809@moneybookers.com> <47A99B16.6030305@FreeBSD.org> <47A9B636.3040509@moneybookers.com> <47A9C43A.3030203@moneybookers.com> In-Reply-To: <47A9C43A.3030203@moneybookers.com> Content-Type: multipart/mixed; boundary="------------010607010803050401040208" Cc: freebsd-performance@freebsd.org Subject: Re: network performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Feb 2008 20:28:12 -0000 This is a multi-part message in MIME format. --------------010607010803050401040208 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Stefan Lambrev wrote: >> I'll use again hwpmc and LOCK_PROFILING to see what's going on. >> And will try the same benchmark on quad core processor as now numbers >> of cores/cpus matter :) >> > Here are promised results - http://89.186.204.158/lock_profiling-8.txt Thanks. There is further work needed on the route locking, and also you are hitting limitations of the em driver (or possibly hardware; if you only have a single transmit queue then outbound packets from multiple CPUs have to be serialized in the driver no matter what). Hopefully there will be further improvements in the coming months, and these changes will also migrate into CVS. If you want to start hacking things to see how much further progress is feasible, you can apply the attached hack that nulls out all route locking :) This should be OK as long as your routes are not changing, although you might get some spam on the console (if this is excessive, comment out the printfs also ;-). It may not help much though, all the contention will probably just fall through onto the ethernet driver. > Btw I got kernel panic first time when I run sysctl debug.lock.prof.stats Yeah, it is a bit broken in 8.0 even in CVS. Also make sure not to reset it while the CPUs are loaded :) > I'm still trying to get hwpmc working with my cpu's and new kernel. > Do you have any patches Kris? > Is it supposed to work with your sources on my CPU? > I can fetch your latest src/lib/libpmc from from p4 if this will help :) It works on my systems...try with libpmc from my branch, make sure to install the new includes first and then rebuild and reinstall libpmc and pmcstat. I have attached a patch against the CVS libpmc which might be easier than checking it out from p4...it relies on kernel changes also though, which are in the kernel you already have but not in CVS. Kris --------------010607010803050401040208 Content-Type: text/plain; x-mac-type="0"; x-mac-creator="0"; name="pmc.diff" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="pmc.diff" --- //depot/vendor/freebsd/src/lib/libpmc/libpmc.c 2007/12/07 14:42:05 +++ //depot/user/kris/contention/lib/libpmc/libpmc.c 2007/12/28 20:32:24 @@ -46,16 +46,14 @@ #if defined(__i386__) static int k7_allocate_pmc(enum pmc_event _pe, char *_ctrspec, struct pmc_op_pmcallocate *_pmc_config); +static int p5_allocate_pmc(enum pmc_event _pe, char *_ctrspec, + struct pmc_op_pmcallocate *_pmc_config); #endif #if defined(__amd64__) || defined(__i386__) static int k8_allocate_pmc(enum pmc_event _pe, char *_ctrspec, struct pmc_op_pmcallocate *_pmc_config); static int p4_allocate_pmc(enum pmc_event _pe, char *_ctrspec, struct pmc_op_pmcallocate *_pmc_config); -#endif -#if defined(__i386__) -static int p5_allocate_pmc(enum pmc_event _pe, char *_ctrspec, - struct pmc_op_pmcallocate *_pmc_config); static int p6_allocate_pmc(enum pmc_event _pe, char *_ctrspec, struct pmc_op_pmcallocate *_pmc_config); #endif @@ -1282,26 +1280,6 @@ return (0); } -#endif - -#if defined(__i386__) - -/* - * Pentium style PMCs - */ - -static struct pmc_event_alias p5_aliases[] = { - EV_ALIAS("cycles", "tsc"), - EV_ALIAS(NULL, NULL) -}; - -static int -p5_allocate_pmc(enum pmc_event pe, char *ctrspec, - struct pmc_op_pmcallocate *pmc_config) -{ - return (-1 || pe || ctrspec || pmc_config); /* shut up gcc */ -} - /* * Pentium Pro style PMCs. These PMCs are found in Pentium II, Pentium III, * and Pentium M CPUs. @@ -1629,9 +1607,30 @@ return (0); } + #endif +#if defined(__i386__) + /* + * Pentium style PMCs + */ + +static struct pmc_event_alias p5_aliases[] = { + EV_ALIAS("cycles", "tsc"), + EV_ALIAS(NULL, NULL) +}; + +static int +p5_allocate_pmc(enum pmc_event pe, char *ctrspec, + struct pmc_op_pmcallocate *pmc_config) +{ + return -1 || pe || ctrspec || pmc_config; /* shut up gcc */ +} + +#endif + +/* * API entry points */ @@ -1940,6 +1939,8 @@ pmc_mdep_event_aliases = p5_aliases; pmc_mdep_allocate_pmc = p5_allocate_pmc; break; +#endif +#if defined(__amd64__) || defined(__i386__) case PMC_CPU_INTEL_P6: /* P6 ... Pentium M CPUs have */ case PMC_CPU_INTEL_PII: /* similar PMCs. */ case PMC_CPU_INTEL_PIII: @@ -1947,8 +1948,6 @@ pmc_mdep_event_aliases = p6_aliases; pmc_mdep_allocate_pmc = p6_allocate_pmc; break; -#endif -#if defined(__amd64__) || defined(__i386__) case PMC_CPU_INTEL_PIV: pmc_mdep_event_aliases = p4_aliases; pmc_mdep_allocate_pmc = p4_allocate_pmc; --------------010607010803050401040208 Content-Type: text/plain; x-mac-type="0"; x-mac-creator="0"; name="routehack.diff" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="routehack.diff" ==== //depot/user/kris/net/net/route.c#2 - /zoo/kris/net/net/route.c ==== @@ -1153,7 +1153,6 @@ struct radix_node_head *rnh = rt_tables[dst->sa_family]; int dlen = SA_SIZE(dst), glen = SA_SIZE(gate); -again: RT_LOCK_ASSERT(rt); /* @@ -1187,15 +1186,6 @@ return (EADDRINUSE); /* failure */ } /* - * Try to reacquire the lock on rt, and if it fails, - * clean state and restart from scratch. - */ - if (!RT_TRYLOCK(rt)) { - RTFREE_LOCKED(gwrt); - RT_LOCK(rt); - goto again; - } - /* * If there is already a gwroute, then drop it. If we * are asked to replace route with itself, then do * not leak its refcounter. ==== //depot/user/kris/net/net/route.h#2 - /zoo/kris/net/net/route.h ==== @@ -288,6 +288,7 @@ #define RT_LOCK_INIT(_rt) \ rw_init_flags(&(_rt)->rt_lock, "rtentry", RW_DUPOK) +#if 0 #define RT_LOCK(_rt) rw_wlock(&(_rt)->rt_lock) #define RT_TRYLOCK(_rt) rw_try_wlock(&(_rt)->rt_lock) #define RT_UNLOCK(_rt) rw_wunlock(&(_rt)->rt_lock) @@ -297,6 +298,16 @@ #define RT_LOCK_DESTROY(_rt) rw_destroy(&(_rt)->rt_lock) #define RT_LOCK_ASSERT(_rt) rw_assert(&(_rt)->rt_lock, RA_LOCKED) #define RT_UNLOCK_ASSERT(_rt) rw_assert(&(_rt)->rt_lock, RA_UNLOCKED) +#endif +#define RT_LOCK(_rt) +#define RT_TRYLOCK(_rt) +#define RT_UNLOCK(_rt) +#define RT_LOCK_SHARED(_rt) +#define RT_UNLOCK_SHARED(_rt) +#define RT_LOCK_DOWNGRADE(_rt) +#define RT_LOCK_DESTROY(_rt) +#define RT_LOCK_ASSERT(_rt) +#define RT_UNLOCK_ASSERT(_rt) #define RT_ADDREF(_rt) do { \ RT_LOCK_ASSERT(_rt); \ --------------010607010803050401040208-- From owner-freebsd-performance@FreeBSD.ORG Wed Feb 6 22:42:24 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0EF4716A419; Wed, 6 Feb 2008 22:42:24 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: from blah.sun-fish.com (blah.sun-fish.com [217.18.249.150]) by mx1.freebsd.org (Postfix) with ESMTP id 7B09E13C45E; Wed, 6 Feb 2008 22:42:23 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: by blah.sun-fish.com (Postfix, from userid 1002) id 3CAD91B10EF8; Wed, 6 Feb 2008 23:42:22 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on blah.cmotd.com X-Spam-Level: X-Spam-Status: No, score=-10.6 required=5.0 tests=ALL_TRUSTED,BAYES_00, NORMAL_HTTP_TO_IP autolearn=ham version=3.2.3 Received: from [10.1.1.2] (unknown [192.168.25.10]) by blah.sun-fish.com (Postfix) with ESMTP id 6A8091B10EDC; Wed, 6 Feb 2008 23:42:19 +0100 (CET) Message-ID: <47AA37CA.8020208@moneybookers.com> Date: Thu, 07 Feb 2008 00:42:18 +0200 From: Stefan Lambrev User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) MIME-Version: 1.0 To: Kris Kennaway References: <4794E6CC.1050107@moneybookers.com> <47A0B023.5020401@moneybookers.com> <47A3074A.3040409@moneybookers.com> <47A72EAB.6070602@moneybookers.com> <20080204182945.GA49276@heff.fud.org.nz> <47A780C0.2060201@moneybookers.com> <47A799A6.3070502@moneybookers.com> <47A84751.8020109@moneybookers.com> <47A8D233.8020506@FreeBSD.org> <47A8DCD6.3060209@moneybookers.com> <47A8E1F1.4040309@FreeBSD.org> <47A98CDC.2090407@moneybookers.com> <47A993D0.1060901@FreeBSD.org> <47A99736.8060809@moneybookers.com> <47A99B16.6030305@FreeBSD.org> <47A9B636.3040509@moneybookers.com> <47A9C43A.3030203@moneybookers.com> <47AA1858.3050307@FreeBSD.org> In-Reply-To: <47AA1858.3050307@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.91.2/5713/Wed Feb 6 21:05:15 2008 on blah.cmotd.com X-Virus-Status: Clean Cc: freebsd-performance@freebsd.org Subject: Re: network performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Feb 2008 22:42:24 -0000 Greetings, Kris Kennaway wrote: > Stefan Lambrev wrote: > >>> I'll use again hwpmc and LOCK_PROFILING to see what's going on. >>> And will try the same benchmark on quad core processor as now >>> numbers of cores/cpus matter :) >>> >> Here are promised results - http://89.186.204.158/lock_profiling-8.txt Finally I got pmcstat working - http://89.186.204.158/hwpmc-p4.txt The stats are gathered during 600kpp incoming. I think that syncache or what calls MD5Transform is not SMP able, and that's why outgoing 250kpps is the limit that I can't beat. > > Thanks. There is further work needed on the route locking, and also > you are hitting limitations of the em driver (or possibly hardware; if > you only have a single transmit queue then outbound packets from > multiple CPUs have to be serialized in the driver no matter what). > Hopefully there will be further improvements in the coming months, and > these changes will also migrate into CVS. > > If you want to start hacking things to see how much further progress > is feasible, you can apply the attached hack that nulls out all route > locking :) This should be OK as long as your routes are not changing, > although you might get some spam on the console (if this is excessive, > comment out the printfs also ;-). It may not help much though, all > the contention will probably just fall through onto the ethernet driver. I'll do tomorrow. I still did not tested how if_lagg will perform in current situation. I thing that I didn't reached the maximum of the network card today. In this configuration best values was 1,122,674 incoming packets + 210kpps outgoing with both cpu cores 100% busy. systat -ifstat showed: em0 in 63.985 MB/s 63.985 MB/s 47.465 GB out 11.674 MB/s 11.723 MB/s 12.975 GB It is good, that now I hit a bottleneck that can be upgraded, if needed :) > >> Btw I got kernel panic first time when I run sysctl >> debug.lock.prof.stats > > Yeah, it is a bit broken in 8.0 even in CVS. Also make sure not to > reset it while the CPUs are loaded :) > >> I'm still trying to get hwpmc working with my cpu's and new kernel. >> Do you have any patches Kris? >> Is it supposed to work with your sources on my CPU? >> I can fetch your latest src/lib/libpmc from from p4 if this will help :) > > It works on my systems...try with libpmc from my branch, make sure to > install the new includes first and then rebuild and reinstall libpmc > and pmcstat. I have attached a patch against the CVS libpmc which > might be easier than checking it out from p4...it relies on kernel > changes also though, which are in the kernel you already have but not > in CVS. > > Kris Yes there was a problem with the installed include file, I edit it by hand, compiled new libpmc and it works again :) From owner-freebsd-performance@FreeBSD.ORG Wed Feb 6 23:08:43 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DCFC616A41A for ; Wed, 6 Feb 2008 23:08:43 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 2D4EC13C459; Wed, 6 Feb 2008 23:08:42 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <47AA3DF9.5040109@FreeBSD.org> Date: Thu, 07 Feb 2008 00:08:41 +0100 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: Stefan Lambrev References: <4794E6CC.1050107@moneybookers.com> <47A0B023.5020401@moneybookers.com> <47A3074A.3040409@moneybookers.com> <47A72EAB.6070602@moneybookers.com> <20080204182945.GA49276@heff.fud.org.nz> <47A780C0.2060201@moneybookers.com> <47A799A6.3070502@moneybookers.com> <47A84751.8020109@moneybookers.com> <47A8D233.8020506@FreeBSD.org> <47A8DCD6.3060209@moneybookers.com> <47A8E1F1.4040309@FreeBSD.org> <47A98CDC.2090407@moneybookers.com> <47A993D0.1060901@FreeBSD.org> <47A99736.8060809@moneybookers.com> <47A99B16.6030305@FreeBSD.org> <47A9B636.3040509@moneybookers.com> <47A9C43A.3030203@moneybookers.com> <47AA1858.3050307@FreeBSD.org> <47AA37CA.8020208@moneybookers.com> In-Reply-To: <47AA37CA.8020208@moneybookers.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: "freebsd-performance@freebsd.org" Subject: Re: network performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Feb 2008 23:08:44 -0000 Stefan Lambrev wrote: > Greetings, > > Kris Kennaway wrote: >> Stefan Lambrev wrote: >> >>>> I'll use again hwpmc and LOCK_PROFILING to see what's going on. >>>> And will try the same benchmark on quad core processor as now >>>> numbers of cores/cpus matter :) >>>> >>> Here are promised results - http://89.186.204.158/lock_profiling-8.txt > Finally I got pmcstat working - http://89.186.204.158/hwpmc-p4.txt > The stats are gathered during 600kpp incoming. > I think that syncache or what calls MD5Transform is not SMP able, and > that's why outgoing 250kpps is the limit that I can't beat. It looks like the syncache is using most of the CPU time. However you are not hitting problems caused by lack of concurrency there. It does do a *lot* of work with the syncache mutex held (including generation of the cookie, which involves MD5) so it might be an issue in the future, but there are other bottlenecks in the way before that is your main issue. Things may be different with more CPUs. Did you compare to what happens to performance when the syncache is disabled? Kris From owner-freebsd-performance@FreeBSD.ORG Thu Feb 7 09:28:47 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 00E4716A418; Thu, 7 Feb 2008 09:28:47 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: from blah.sun-fish.com (blah.sun-fish.com [217.18.249.150]) by mx1.freebsd.org (Postfix) with ESMTP id 7D8A513C467; Thu, 7 Feb 2008 09:28:46 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: by blah.sun-fish.com (Postfix, from userid 1002) id 6ABBF1B10F39; Thu, 7 Feb 2008 10:28:44 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on blah.cmotd.com X-Spam-Level: X-Spam-Status: No, score=-10.6 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.2.3 Received: from hater.haters.org (hater.cmotd.com [192.168.3.125]) by blah.sun-fish.com (Postfix) with ESMTP id 6F8AF1B10EF7; Thu, 7 Feb 2008 10:28:40 +0100 (CET) Message-ID: <47AACF47.9040101@moneybookers.com> Date: Thu, 07 Feb 2008 11:28:39 +0200 From: Stefan Lambrev User-Agent: Thunderbird 2.0.0.9 (X11/20071120) MIME-Version: 1.0 To: Kris Kennaway References: <4794E6CC.1050107@moneybookers.com> <47A0B023.5020401@moneybookers.com> <47A3074A.3040409@moneybookers.com> <47A72EAB.6070602@moneybookers.com> <20080204182945.GA49276@heff.fud.org.nz> <47A780C0.2060201@moneybookers.com> <47A799A6.3070502@moneybookers.com> <47A84751.8020109@moneybookers.com> <47A8D233.8020506@FreeBSD.org> <47A8DCD6.3060209@moneybookers.com> <47A8E1F1.4040309@FreeBSD.org> <47A98CDC.2090407@moneybookers.com> <47A993D0.1060901@FreeBSD.org> <47A99736.8060809@moneybookers.com> <47A99B16.6030305@FreeBSD.org> <47A9B636.3040509@moneybookers.com> <47AA1395.2090501@FreeBSD.org> In-Reply-To: <47AA1395.2090501@FreeBSD.org> Content-Type: text/plain; charset=windows-1251; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.91.2/5725/Thu Feb 7 08:42:28 2008 on blah.cmotd.com X-Virus-Status: Clean Cc: freebsd-performance@freebsd.org Subject: Re: network performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Feb 2008 09:28:47 -0000 Greetings, Kris Kennaway wrote: > Stefan Lambrev wrote: >> Greetings, >> >> Kris Kennaway wrote: >>> Yes, it is gone with 8.0. Disable the module builds because some of >>> them like this one probably need compile fixes. If you need a >>> subset of modules use MODULES_OVERRIDE=list (in /etc/make.conf) >>> >> Yes, kernel builds. >> I'm still playing with it, but the first results shows that new >> kernel can handle 800k incoming packets (well may be more but I have >> not enough power right now to generate more packets). >> It still answer only to 250K-260K. I guess I'm hitting the limitation >> of syncache/syncookies ? > > Yes, it could be. You may need to tune the net.inet.tcp.syncache > parameters to get better performance. That is good news though. > >> Anyway this netisr2 looks like huge improvement :) > > Actually I forgot to mention: you probably want to set net.isr2.direct=1. net.isr2.direct=1 have very bad impact over performance. When I set it to 1 the server cannot handle more then 230kpps which is worse then vanilla 7.0 :) Do you want to see lock profiling and hwpmc output when enabled? Btw from where I can check if malloc() debugging is enabled, as I want to be sure that no debugging is slowing down the tests? Does `ln -s aj /etc/malloc.conf` disable malloc() dubug? -- Best Wishes, Stefan Lambrev ICQ# 24134177 From owner-freebsd-performance@FreeBSD.ORG Thu Feb 7 09:43:24 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1673216A418; Thu, 7 Feb 2008 09:43:24 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: from blah.sun-fish.com (blah.sun-fish.com [217.18.249.150]) by mx1.freebsd.org (Postfix) with ESMTP id A817013C4D3; Thu, 7 Feb 2008 09:43:23 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: by blah.sun-fish.com (Postfix, from userid 1002) id 20A621B10EF7; Thu, 7 Feb 2008 10:43:22 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on blah.cmotd.com X-Spam-Level: X-Spam-Status: No, score=-10.6 required=5.0 tests=ALL_TRUSTED,BAYES_00, NORMAL_HTTP_TO_IP autolearn=ham version=3.2.3 Received: from hater.haters.org (hater.cmotd.com [192.168.3.125]) by blah.sun-fish.com (Postfix) with ESMTP id 4C6EF1B10EDC; Thu, 7 Feb 2008 10:43:19 +0100 (CET) Message-ID: <47AAD2B6.6070106@moneybookers.com> Date: Thu, 07 Feb 2008 11:43:18 +0200 From: Stefan Lambrev User-Agent: Thunderbird 2.0.0.9 (X11/20071120) MIME-Version: 1.0 To: Kris Kennaway References: <4794E6CC.1050107@moneybookers.com> <47A0B023.5020401@moneybookers.com> <47A3074A.3040409@moneybookers.com> <47A72EAB.6070602@moneybookers.com> <20080204182945.GA49276@heff.fud.org.nz> <47A780C0.2060201@moneybookers.com> <47A799A6.3070502@moneybookers.com> <47A84751.8020109@moneybookers.com> <47A8D233.8020506@FreeBSD.org> <47A8DCD6.3060209@moneybookers.com> <47A8E1F1.4040309@FreeBSD.org> <47A98CDC.2090407@moneybookers.com> <47A993D0.1060901@FreeBSD.org> <47A99736.8060809@moneybookers.com> <47A99B16.6030305@FreeBSD.org> <47A9B636.3040509@moneybookers.com> <47A9C43A.3030203@moneybookers.com> <47AA1858.3050307@FreeBSD.org> <47AA37CA.8020208@moneybookers.com> <47AA3DF9.5040109@FreeBSD.org> In-Reply-To: <47AA3DF9.5040109@FreeBSD.org> Content-Type: text/plain; charset=windows-1251; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.91.2/5725/Thu Feb 7 08:42:28 2008 on blah.cmotd.com X-Virus-Status: Clean Cc: "freebsd-performance@freebsd.org" Subject: Re: network performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Feb 2008 09:43:24 -0000 Greetings, Kris Kennaway wrote: > Stefan Lambrev wrote: >> Greetings, >> >> Kris Kennaway wrote: >>> Stefan Lambrev wrote: >>> >>>>> I'll use again hwpmc and LOCK_PROFILING to see what's going on. >>>>> And will try the same benchmark on quad core processor as now >>>>> numbers of cores/cpus matter :) >>>>> >>>> Here are promised results - http://89.186.204.158/lock_profiling-8.txt >> Finally I got pmcstat working - http://89.186.204.158/hwpmc-p4.txt >> The stats are gathered during 600kpp incoming. >> I think that syncache or what calls MD5Transform is not SMP able, and >> that's why outgoing 250kpps is the limit that I can't beat. > > It looks like the syncache is using most of the CPU time. However you > are not hitting problems caused by lack of concurrency there. It does > do a *lot* of work with the syncache mutex held (including generation > of the cookie, which involves MD5) so it might be an issue in the > future, but there are other bottlenecks in the way before that is your > main issue. Things may be different with more CPUs. > > Did you compare to what happens to performance when the syncache is > disabled? > > Kris > When I disable syncookies the server respond to more packets - from 250kpps enabled to 320kpps when disabled. Can I disable syncache and how? I'll try to increase syncache limits and will test again. Btw is this expected - net.inet.tcp.syncache.count: -387. I think this number should be always > 0. -- Best Wishes, Stefan Lambrev ICQ# 24134177 From owner-freebsd-performance@FreeBSD.ORG Thu Feb 7 10:18:15 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A5CD616A419; Thu, 7 Feb 2008 10:18:15 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: from blah.sun-fish.com (blah.sun-fish.com [217.18.249.150]) by mx1.freebsd.org (Postfix) with ESMTP id D8C1213C46A; Thu, 7 Feb 2008 10:18:14 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: by blah.sun-fish.com (Postfix, from userid 1002) id 1EC491B10F1F; Thu, 7 Feb 2008 11:18:14 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on blah.cmotd.com X-Spam-Level: X-Spam-Status: No, score=-10.6 required=5.0 tests=ALL_TRUSTED,BAYES_00, NORMAL_HTTP_TO_IP autolearn=ham version=3.2.3 Received: from hater.haters.org (hater.cmotd.com [192.168.3.125]) by blah.sun-fish.com (Postfix) with ESMTP id 9B7101B10EE0; Thu, 7 Feb 2008 11:18:10 +0100 (CET) Message-ID: <47AADAE2.4040500@moneybookers.com> Date: Thu, 07 Feb 2008 12:18:10 +0200 From: Stefan Lambrev User-Agent: Thunderbird 2.0.0.9 (X11/20071120) MIME-Version: 1.0 To: Kris Kennaway References: <4794E6CC.1050107@moneybookers.com> <47A0B023.5020401@moneybookers.com> <47A3074A.3040409@moneybookers.com> <47A72EAB.6070602@moneybookers.com> <20080204182945.GA49276@heff.fud.org.nz> <47A780C0.2060201@moneybookers.com> <47A799A6.3070502@moneybookers.com> <47A84751.8020109@moneybookers.com> <47A8D233.8020506@FreeBSD.org> <47A8DCD6.3060209@moneybookers.com> <47A8E1F1.4040309@FreeBSD.org> <47A98CDC.2090407@moneybookers.com> <47A993D0.1060901@FreeBSD.org> <47A99736.8060809@moneybookers.com> <47A99B16.6030305@FreeBSD.org> <47A9B636.3040509@moneybookers.com> <47A9C43A.3030203@moneybookers.com> <47AA1858.3050307@FreeBSD.org> <47AA37CA.8020208@moneybookers.com> <47AA3DF9.5040109@FreeBSD.org> <47AAD2B6.6070106@moneybookers.com> In-Reply-To: <47AAD2B6.6070106@moneybookers.com> Content-Type: text/plain; charset=windows-1251; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.91.2/5727/Thu Feb 7 10:12:33 2008 on blah.cmotd.com X-Virus-Status: Clean Cc: "freebsd-performance@freebsd.org" Subject: Re: network performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Feb 2008 10:18:15 -0000 Greetings, Stefan Lambrev wrote: > Greetings, > > Kris Kennaway wrote: >> Stefan Lambrev wrote: >>> Greetings, >>> >>> Kris Kennaway wrote: >>>> Stefan Lambrev wrote: >>>> >>>>>> I'll use again hwpmc and LOCK_PROFILING to see what's going on. >>>>>> And will try the same benchmark on quad core processor as now >>>>>> numbers of cores/cpus matter :) >>>>>> >>>>> Here are promised results - >>>>> http://89.186.204.158/lock_profiling-8.txt >>> Finally I got pmcstat working - http://89.186.204.158/hwpmc-p4.txt >>> The stats are gathered during 600kpp incoming. >>> I think that syncache or what calls MD5Transform is not SMP able, >>> and that's why outgoing 250kpps is the limit that I can't beat. >> >> It looks like the syncache is using most of the CPU time. However >> you are not hitting problems caused by lack of concurrency there. It >> does do a *lot* of work with the syncache mutex held (including >> generation of the cookie, which involves MD5) so it might be an issue >> in the future, but there are other bottlenecks in the way before that >> is your main issue. Things may be different with more CPUs. >> >> Did you compare to what happens to performance when the syncache is >> disabled? >> >> Kris >> > When I disable syncookies the server respond to more packets - from > 250kpps enabled to 320kpps when disabled. > Can I disable syncache and how? Err you answer this already :) So here are numbers in different syncache/coockies options: net.inet.tcp.syncookies: 1 & net.inet.tcp.syncookies_only: 0 input (em0) output packets errs bytes packets errs bytes colls 519748 0 31184880 252606 0 14651090 0 522321 0 31339260 251459 0 14584490 0 532242 0 31934520 242768 0 14080486 0 533484 0 32009040 249006 0 14442522 0 net.inet.tcp.syncookies: 0 & net.inet.tcp.syncookies_only: 0 input (em0) output packets errs bytes packets errs bytes colls 512100 0 30726000 316873 0 18378866 0 517952 0 31077120 315868 0 18320270 0 531862 0 31911720 318416 0 18468244 0 526125 0 31567440 315283 0 18286340 0 net.inet.tcp.syncookies: 0 & net.inet.tcp.syncookies_only: 1 input (em0) output packets errs bytes packets errs bytes colls 517257 0 31035420 315739 0 18310890 0 518328 0 31099680 316888 0 18381534 0 517168 0 31030080 316693 0 18368426 0 528273 0 31696320 315722 0 18311670 0 533153 0 31989240 314693 0 18252136 0 net.inet.tcp.syncookies: 1 & net.inet.tcp.syncookies_only: 1 input (em0) output packets errs bytes packets errs bytes colls 539069 0 32344140 282434 0 16381272 0 538075 0 32284440 281569 0 16330886 0 538329 0 32299800 281908 0 16350880 0 534111 0 32046660 280906 0 16292258 0 536204 0 32172240 281169 0 16307860 0 Increasing net.inet.tcp.syncache.hashsize to 2048 (512 default) input (em0) output packets errs bytes packets errs bytes colls 532945 0 31976700 186833 0 10835096 0 533507 0 32010420 187053 0 10850292 0 530204 0 31812240 187672 0 10883758 0 531482 0 31888920 187585 0 10879930 0 534651 0 32079060 186977 0 10845884 0 This prove that syncache have negative impact on performance ? > I'll try to increase syncache limits and will test again. > > Btw is this expected - net.inet.tcp.syncache.count: -387. > I think this number should be always > 0. > > -- Best Wishes, Stefan Lambrev ICQ# 24134177 From owner-freebsd-performance@FreeBSD.ORG Thu Feb 7 11:40:48 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1C79216A481; Thu, 7 Feb 2008 11:40:48 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: from blah.sun-fish.com (blah.sun-fish.com [217.18.249.150]) by mx1.freebsd.org (Postfix) with ESMTP id BEE7713C46B; Thu, 7 Feb 2008 11:40:45 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: by blah.sun-fish.com (Postfix, from userid 1002) id 4FDDD1B10EF1; Thu, 7 Feb 2008 12:40:44 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on blah.cmotd.com X-Spam-Level: X-Spam-Status: No, score=-10.6 required=5.0 tests=ALL_TRUSTED,BAYES_00, NORMAL_HTTP_TO_IP autolearn=ham version=3.2.3 Received: from hater.haters.org (hater.cmotd.com [192.168.3.125]) by blah.sun-fish.com (Postfix) with ESMTP id 716A31B10EDC; Thu, 7 Feb 2008 12:40:41 +0100 (CET) Message-ID: <47AAEE39.3020805@moneybookers.com> Date: Thu, 07 Feb 2008 13:40:41 +0200 From: Stefan Lambrev User-Agent: Thunderbird 2.0.0.9 (X11/20071120) MIME-Version: 1.0 To: Kris Kennaway References: <4794E6CC.1050107@moneybookers.com> <47A0B023.5020401@moneybookers.com> <47A3074A.3040409@moneybookers.com> <47A72EAB.6070602@moneybookers.com> <20080204182945.GA49276@heff.fud.org.nz> <47A780C0.2060201@moneybookers.com> <47A799A6.3070502@moneybookers.com> <47A84751.8020109@moneybookers.com> <47A8D233.8020506@FreeBSD.org> <47A8DCD6.3060209@moneybookers.com> <47A8E1F1.4040309@FreeBSD.org> <47A98CDC.2090407@moneybookers.com> <47A993D0.1060901@FreeBSD.org> <47A99736.8060809@moneybookers.com> <47A99B16.6030305@FreeBSD.org> <47A9B636.3040509@moneybookers.com> <47AA1395.2090501@FreeBSD.org> In-Reply-To: <47AA1395.2090501@FreeBSD.org> Content-Type: text/plain; charset=windows-1251; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.91.2/5727/Thu Feb 7 10:12:33 2008 on blah.cmotd.com X-Virus-Status: Clean Cc: freebsd-performance@freebsd.org Subject: Re: network performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Feb 2008 11:40:48 -0000 Greetings, >> >> And will try the same benchmark on quad core processor as now numbers >> of cores/cpus matter :) > > That will certainly be interesting to test! I made some tests with quad core CPU: netstat shows: input (em0) output packets errs bytes packets errs bytes colls 533738 0 32024280 181592 0 10506120 0 526035 0 31562100 181580 0 10508266 0 524260 0 31455600 181683 0 10509310 0 524264 0 31455840 182385 0 10550374 0 pmcstat results - http://89.186.204.158/hwpmc-p4-quadcore.txt not sure if they are correct because the system livelocked during pmcstat run, so I did cold reboot. if_em is compiled as module here, so grpof from if_em is at the end of the file. Quite surprising quad core is slower in this benchmark, and I'm still trying to find what is different if we do not count the cpus :) -- Best Wishes, Stefan Lambrev ICQ# 24134177 From owner-freebsd-performance@FreeBSD.ORG Thu Feb 7 16:42:51 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0A33816A41A; Thu, 7 Feb 2008 16:42:51 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: from blah.sun-fish.com (blah.sun-fish.com [217.18.249.150]) by mx1.freebsd.org (Postfix) with ESMTP id 7571813C447; Thu, 7 Feb 2008 16:42:50 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: by blah.sun-fish.com (Postfix, from userid 1002) id 7E0DE1B10F1F; Thu, 7 Feb 2008 17:42:48 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on blah.cmotd.com X-Spam-Level: X-Spam-Status: No, score=-10.6 required=5.0 tests=ALL_TRUSTED,BAYES_00, NORMAL_HTTP_TO_IP autolearn=ham version=3.2.3 Received: from hater.haters.org (hater.cmotd.com [192.168.3.125]) by blah.sun-fish.com (Postfix) with ESMTP id 5356D1B10EF7; Thu, 7 Feb 2008 17:42:39 +0100 (CET) Message-ID: <47AB34FE.3050309@moneybookers.com> Date: Thu, 07 Feb 2008 18:42:38 +0200 From: Stefan Lambrev User-Agent: Thunderbird 2.0.0.9 (X11/20071120) MIME-Version: 1.0 To: Kris Kennaway References: <4794E6CC.1050107@moneybookers.com> <47A0B023.5020401@moneybookers.com> <47A3074A.3040409@moneybookers.com> <47A72EAB.6070602@moneybookers.com> <20080204182945.GA49276@heff.fud.org.nz> <47A780C0.2060201@moneybookers.com> <47A799A6.3070502@moneybookers.com> <47A84751.8020109@moneybookers.com> <47A8D233.8020506@FreeBSD.org> <47A8DCD6.3060209@moneybookers.com> <47A8E1F1.4040309@FreeBSD.org> <47A98CDC.2090407@moneybookers.com> <47A993D0.1060901@FreeBSD.org> <47A99736.8060809@moneybookers.com> <47A99B16.6030305@FreeBSD.org> <47A9B636.3040509@moneybookers.com> <47AA1395.2090501@FreeBSD.org> <47AACF47.9040101@moneybookers.com> In-Reply-To: <47AACF47.9040101@moneybookers.com> Content-Type: text/plain; charset=windows-1251; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.91.2/5728/Thu Feb 7 13:51:05 2008 on blah.cmotd.com X-Virus-Status: Clean Cc: freebsd-performance@freebsd.org Subject: Re: network performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Feb 2008 16:42:51 -0000 Greetings, Results with if_lagg, p4-kernel, dual core, two gigabit netowrk cards: input (lagg0) output packets errs bytes packets errs bytes colls 512412 0 30744720 218164 0 12653686 0 508765 0 30525900 218720 0 12686514 0 509797 0 30587820 218001 0 12644522 0 529478 0 31768680 217203 0 12598412 0 502720 0 30163200 218808 0 12691154 0 http://89.186.204.158/hwpmc-p4-lagg.txt ================================================ with net.isr2.direct=1 input (lagg0) output packets errs bytes packets errs bytes colls 0 0 0 0 0 0 0 157533 0 9451980 145403 0 8433342 0 208414 0 12504840 193526 0 11224508 0 208731 0 12523860 193808 0 11240864 0 208537 0 12512220 193719 0 11235702 0 209309 0 12558540 194411 0 11275896 0 no pmcstat, but can provide if needed :) ================================================ with routehack.patch: netstat -w1 -I em0 input (em0) output packets errs bytes packets errs bytes colls 473241 110730 28394460 253427 0 14697722 0 446691 88556 26801460 236577 0 13721466 0 440030 86720 26401800 232431 0 13481026 0 433968 108071 26038080 227498 0 13194854 0 pmcstats - http://89.186.204.158/hwpmc-p4-routehack.txt ================================================ with net.isr2.direct=1 + routehack netstat -w1 -I em0 input (em0) output packets errs bytes packets errs bytes colls 250463 274227 15027780 233526 0 13547856 0 250596 271527 15035760 233655 0 13552032 0 251167 286648 15070020 234331 0 13589226 0 250895 286506 15053700 234067 0 13575654 0 250954 284616 15057240 234332 0 13593924 0 no pmcstat ================================================ lagg + routehack netstat -w1 -I lagg0 input (lagg0) output packets errs bytes packets errs bytes colls 503065 0 30183900 258149 0 14973512 0 512911 0 30774660 257825 0 14954546 0 519503 0 31170180 255413 0 14814476 0 pmcstats - http://89.186.204.158/hwpmc-p4-lagg-routehack.txt ================================================ Unfortunately 89.186.204.158 is down right now, but I hope it will be up soon :) P.S. Have in mind that netstat -w1 -I lagg0 do not report errors (they are reported separately for every network card) -- Best Wishes, Stefan Lambrev ICQ# 24134177 From owner-freebsd-performance@FreeBSD.ORG Fri Feb 8 09:00:06 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6F7AA16A41B; Fri, 8 Feb 2008 09:00:06 +0000 (UTC) (envelope-from erik@cederstrand.dk) Received: from mail.itu.dk (pluto.itu.dk [130.226.142.18]) by mx1.freebsd.org (Postfix) with ESMTP id 34F8B13C4DB; Fri, 8 Feb 2008 09:00:06 +0000 (UTC) (envelope-from erik@cederstrand.dk) Received: from [192.168.1.148] (stud1-15.itu.dk [130.226.140.15]) by mail.itu.dk (Postfix) with ESMTP id AD21636E9D6; Fri, 8 Feb 2008 08:41:05 +0000 (UTC) Message-ID: <47AC15A5.5020009@cederstrand.dk> Date: Fri, 08 Feb 2008 09:41:09 +0100 From: Erik Cederstrand User-Agent: Thunderbird 2.0.0.6 (X11/20071022) MIME-Version: 1.0 To: Brooks Davis References: <4796C717.9000507@cederstrand.dk> <20080123193400.N63024@fledge.watson.org> <4797A245.7080202@cederstrand.dk> <20080123202433.E63024@fledge.watson.org> <4797A802.8060509@FreeBSD.org> <47A0BFE7.4070708@cederstrand.dk> <20080130190000.GA18333@lor.one-eyed-alien.net> In-Reply-To: <20080130190000.GA18333@lor.one-eyed-alien.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-performance@freebsd.org, kris@FreeBSD.org Subject: Re: Performance Tracker project update X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Feb 2008 09:00:06 -0000 Brooks Davis skrev: > On Wed, Jan 30, 2008 at 07:20:23PM +0100, Erik Cederstrand wrote: >> >> I'd like a situation where I can very quickly set up a slave with a >> specific version of FreeBSD to run additional tests or provide shell access >> to a developer. This currently involves adding an entry to a queue, >> rebooting and waiting 2 minutes. Quick and easy, but the archiving strategy >> is obviously very inefficient. >> >> I'm thinking of a couple of options: >> 1. Having one full install per month and archiving the rest as diffs >> against that by recursively bsdiff'ing every file in the tree (I >> could bsdiff a whole tarball, but bsdiff is very memory-intensive). >> Quick test: 25 mins. >> 2. Make a hash of all files and only store the binaries where the hash >> is different from the monthly tarball. Faster than 1., but less >> effective. Quick test: 5 mins. >> 3. Use some kind of VCS. My experience with Subversion and binary files >> is that it's very slow. >> 4. Throw hardware at the problem. >> >> I'd say it should not take more than 10 mins to recreate an archived >> version. Any thoughts? > > It seems like you should be able to combine 1 and 2 with checksums to > decide if you need to run diffs. I'd think that would be quite fast. I finally got around to testing this, and with a combination of mtree comparing md5 hashes, bsdiff compacting changed files and hardlinking unchanged files I get a reduction in size from 256MB to 10MB. Pretty good, and the whole operation only takes a few minutes. I have one peculiarity, though. I install python2.5 into the directory containing the build, and even though the python version has not changed, I still get mismatching md5 sums on every .pyo and .pyc file. Any thoughts on this? Erik From owner-freebsd-performance@FreeBSD.ORG Fri Feb 8 15:18:01 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E15B316A41B; Fri, 8 Feb 2008 15:18:01 +0000 (UTC) (envelope-from brooks@lor.one-eyed-alien.net) Received: from lor.one-eyed-alien.net (cl-162.ewr-01.us.sixxs.net [IPv6:2001:4830:1200:a1::2]) by mx1.freebsd.org (Postfix) with ESMTP id 61C2213C442; Fri, 8 Feb 2008 15:18:01 +0000 (UTC) (envelope-from brooks@lor.one-eyed-alien.net) Received: from lor.one-eyed-alien.net (localhost [127.0.0.1]) by lor.one-eyed-alien.net (8.14.1/8.13.8) with ESMTP id m18FHv1p043622; Fri, 8 Feb 2008 09:17:57 -0600 (CST) (envelope-from brooks@lor.one-eyed-alien.net) Received: (from brooks@localhost) by lor.one-eyed-alien.net (8.14.1/8.13.8/Submit) id m18FHvL1043621; Fri, 8 Feb 2008 09:17:57 -0600 (CST) (envelope-from brooks) Date: Fri, 8 Feb 2008 09:17:56 -0600 From: Brooks Davis To: Erik Cederstrand Message-ID: <20080208151756.GA35423@lor.one-eyed-alien.net> References: <4796C717.9000507@cederstrand.dk> <20080123193400.N63024@fledge.watson.org> <4797A245.7080202@cederstrand.dk> <20080123202433.E63024@fledge.watson.org> <4797A802.8060509@FreeBSD.org> <47A0BFE7.4070708@cederstrand.dk> <20080130190000.GA18333@lor.one-eyed-alien.net> <47AC15A5.5020009@cederstrand.dk> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="n8g4imXOkfNTN/H1" Content-Disposition: inline In-Reply-To: <47AC15A5.5020009@cederstrand.dk> User-Agent: Mutt/1.5.16 (2007-06-09) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-3.0 (lor.one-eyed-alien.net [127.0.0.1]); Fri, 08 Feb 2008 09:17:58 -0600 (CST) Cc: freebsd-performance@freebsd.org, Brooks Davis , kris@freebsd.org Subject: Re: Performance Tracker project update X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Feb 2008 15:18:02 -0000 --n8g4imXOkfNTN/H1 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Feb 08, 2008 at 09:41:09AM +0100, Erik Cederstrand wrote: > Brooks Davis skrev: >> On Wed, Jan 30, 2008 at 07:20:23PM +0100, Erik Cederstrand wrote: >>>=20 >>> I'd like a situation where I can very quickly set up a slave with a=20 >>> specific version of FreeBSD to run additional tests or provide shell=20 >>> access to a developer. This currently involves adding an entry to a=20 >>> queue, rebooting and waiting 2 minutes. Quick and easy, but the archivi= ng=20 >>> strategy is obviously very inefficient. >>>=20 >>> I'm thinking of a couple of options: >>> 1. Having one full install per month and archiving the rest as diffs >>> against that by recursively bsdiff'ing every file in the tree (I >>> could bsdiff a whole tarball, but bsdiff is very memory-intensive). >>> Quick test: 25 mins. >>> 2. Make a hash of all files and only store the binaries where the hash >>> is different from the monthly tarball. Faster than 1., but less >>> effective. Quick test: 5 mins. >>> 3. Use some kind of VCS. My experience with Subversion and binary files >>> is that it's very slow. >>> 4. Throw hardware at the problem. >>>=20 >>> I'd say it should not take more than 10 mins to recreate an archived=20 >>> version. Any thoughts? >> It seems like you should be able to combine 1 and 2 with checksums to >> decide if you need to run diffs. I'd think that would be quite fast. >=20 > I finally got around to testing this, and with a combination of mtree=20 > comparing md5 hashes, bsdiff compacting changed files and hardlinking=20 > unchanged files I get a reduction in size from 256MB to 10MB. Pretty good= ,=20 > and the whole operation only takes a few minutes. Cool! > I have one peculiarity, though. I install python2.5 into the directory=20 > containing the build, and even though the python version has not changed,= I=20 > still get mismatching md5 sums on every .pyo and .pyc file. Any thoughts = on=20 > this? I'm not a python guru by any means, but I think .pyc files probably have da= ta about the .py they are generated from because there's some sort of auto-generation available. It may be possible to not store them at all and just generate them before you use them or add some magic build flags to cau= se them to store some sort of cooked values. I'm not sure where the .pyo files come from. -- Brooks --n8g4imXOkfNTN/H1 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (FreeBSD) iD8DBQFHrHKkXY6L6fI4GtQRApHaAJ97Xs/RkROLfXsgnFBV8d6yHmfoCQCgtF9N P5wzW2mvgZCgBv973JH1cMs= =Fzh9 -----END PGP SIGNATURE----- --n8g4imXOkfNTN/H1-- From owner-freebsd-performance@FreeBSD.ORG Fri Feb 8 19:42:16 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2436C16A418 for ; Fri, 8 Feb 2008 19:42:16 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id 919BD13C45B for ; Fri, 8 Feb 2008 19:42:15 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1JNZ6v-0002PZ-GW for freebsd-performance@freebsd.org; Fri, 08 Feb 2008 19:42:09 +0000 Received: from 78-0-94-116.adsl.net.t-com.hr ([78.0.94.116]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 08 Feb 2008 19:42:09 +0000 Received: from ivoras by 78-0-94-116.adsl.net.t-com.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 08 Feb 2008 19:42:09 +0000 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-performance@freebsd.org From: Ivan Voras Date: Fri, 08 Feb 2008 20:42:01 +0100 Lines: 62 Message-ID: References: <4796C717.9000507@cederstrand.dk> <20080123193400.N63024@fledge.watson.org> <4797A245.7080202@cederstrand.dk> <20080123202433.E63024@fledge.watson.org> <4797A802.8060509@FreeBSD.org> <47A0BFE7.4070708@cederstrand.dk> <20080130190000.GA18333@lor.one-eyed-alien.net> <47AC15A5.5020009@cederstrand.dk> <20080208151756.GA35423@lor.one-eyed-alien.net> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig81991AAF532A8F150E7388EA" X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: 78-0-94-116.adsl.net.t-com.hr User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) In-Reply-To: <20080208151756.GA35423@lor.one-eyed-alien.net> X-Enigmail-Version: 0.95.6 Sender: news Subject: Re: Performance Tracker project update X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Feb 2008 19:42:16 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig81991AAF532A8F150E7388EA Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable Brooks Davis wrote: > On Fri, Feb 08, 2008 at 09:41:09AM +0100, Erik Cederstrand wrote: >> I finally got around to testing this, and with a combination of mtree = >> comparing md5 hashes, bsdiff compacting changed files and hardlinking = >> unchanged files I get a reduction in size from 256MB to 10MB. Pretty g= ood,=20 >> and the whole operation only takes a few minutes. >=20 > Cool! >=20 >> I have one peculiarity, though. I install python2.5 into the directory= =20 >> containing the build, and even though the python version has not chang= ed, I=20 >> still get mismatching md5 sums on every .pyo and .pyc file. Any though= ts on=20 >> this? >=20 > I'm not a python guru by any means, but I think .pyc files probably hav= e data > about the .py they are generated from because there's some sort of > auto-generation available. It may be possible to not store them at all= and > just generate them before you use them or add some magic build flags to= cause > them to store some sort of cooked values. I'm not sure where the .pyo = files > come from. =2Epyc/.pyo contain at least the timestamp of the original .py file and=20 maybe a compilation timestamp. You can indeed safely delete all .pyc and = =2Epyo files and forget about them - the only penalty will be slightly=20 slower application startup times as the .py files are compiled every time= =2E =2Epyo are optimized version of .pyc. AFAIK currently the optimizations=20 are not worth it. --------------enig81991AAF532A8F150E7388EA Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHrLCKldnAQVacBcgRApcHAKDUh3GqcnCJvofW7mJHodntV53gFgCZAXU7 MTI1AHbajqvRU4vBjMWRBz0= =OxFf -----END PGP SIGNATURE----- --------------enig81991AAF532A8F150E7388EA-- From owner-freebsd-performance@FreeBSD.ORG Fri Feb 8 20:15:33 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E358116A419; Fri, 8 Feb 2008 20:15:33 +0000 (UTC) (envelope-from cswiger@mac.com) Received: from mail-out3.apple.com (mail-out3.apple.com [17.254.13.22]) by mx1.freebsd.org (Postfix) with ESMTP id C312213C4E3; Fri, 8 Feb 2008 20:15:33 +0000 (UTC) (envelope-from cswiger@mac.com) Received: from relay12.apple.com (relay14.apple.com [17.128.113.52]) by mail-out3.apple.com (Postfix) with ESMTP id DC0DB2096DE2; Fri, 8 Feb 2008 11:56:54 -0800 (PST) Received: from relay14.apple.com (unknown [127.0.0.1]) by relay12.apple.com (Symantec Mail Security) with ESMTP id C239228089; Fri, 8 Feb 2008 11:56:54 -0800 (PST) X-AuditID: 11807134-a7d7cbb0000008b9-fc-47acb4063019 Received: from cswiger1.apple.com (cswiger1.apple.com [17.214.13.96]) by relay12.apple.com (Apple SCV relay) with ESMTP id A0A5D28086; Fri, 8 Feb 2008 11:56:54 -0800 (PST) Message-Id: <9D27D745-2465-4FB2-B7E0-3C5DD411E9B9@mac.com> From: Chuck Swiger To: Ivan Voras In-Reply-To: Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v915) Date: Fri, 8 Feb 2008 11:56:54 -0800 References: <4796C717.9000507@cederstrand.dk> <20080123193400.N63024@fledge.watson.org> <4797A245.7080202@cederstrand.dk> <20080123202433.E63024@fledge.watson.org> <4797A802.8060509@FreeBSD.org> <47A0BFE7.4070708@cederstrand.dk> <20080130190000.GA18333@lor.one-eyed-alien.net> <47AC15A5.5020009@cederstrand.dk> <20080208151756.GA35423@lor.one-eyed-alien.net> X-Mailer: Apple Mail (2.915) X-Brightmail-Tracker: AAAAAA== Cc: freebsd-performance@freebsd.org Subject: Re: Performance Tracker project update X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Feb 2008 20:15:34 -0000 On Feb 8, 2008, at 11:42 AM, Ivan Voras wrote: >> I'm not a python guru by any means, but I think .pyc files probably >> have data >> about the .py they are generated from because there's some sort of >> auto-generation available. It may be possible to not store them at >> all and >> just generate them before you use them or add some magic build >> flags to cause >> them to store some sort of cooked values. I'm not sure where >> the .pyo files >> come from. > > .pyc/.pyo contain at least the timestamp of the original .py file > and maybe a compilation timestamp. You can indeed safely delete > all .pyc and .pyo files and forget about them - the only penalty > will be slightly slower application startup times as the .py files > are compiled every time. Yep, that's right. > .pyo are optimized version of .pyc. AFAIK currently the > optimizations are not worth it. Historically, the Python optimizer wasn't capable of doing much, true, but the more recent versions of the optimizer can actually do some peephole optimizations like algorithmic simplification and constant folding: http://docs.python.org/whatsnew/other-lang.html#SECTION0001320000000000000000 -- -Chuck From owner-freebsd-performance@FreeBSD.ORG Fri Feb 8 20:44:11 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EDEAD16A419 for ; Fri, 8 Feb 2008 20:44:10 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id 63A2713C457 for ; Fri, 8 Feb 2008 20:44:10 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1JNa4s-0005Nq-3m for freebsd-performance@freebsd.org; Fri, 08 Feb 2008 20:44:06 +0000 Received: from 78-0-94-116.adsl.net.t-com.hr ([78.0.94.116]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 08 Feb 2008 20:44:06 +0000 Received: from ivoras by 78-0-94-116.adsl.net.t-com.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 08 Feb 2008 20:44:06 +0000 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-performance@freebsd.org From: Ivan Voras Date: Fri, 08 Feb 2008 21:43:48 +0100 Lines: 53 Message-ID: References: <4796C717.9000507@cederstrand.dk> <20080123193400.N63024@fledge.watson.org> <4797A245.7080202@cederstrand.dk> <20080123202433.E63024@fledge.watson.org> <4797A802.8060509@FreeBSD.org> <47A0BFE7.4070708@cederstrand.dk> <20080130190000.GA18333@lor.one-eyed-alien.net> <47AC15A5.5020009@cederstrand.dk> <20080208151756.GA35423@lor.one-eyed-alien.net> <9D27D745-2465-4FB2-B7E0-3C5DD411E9B9@mac.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig474760FAE99CF3CD10855972" X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: 78-0-94-116.adsl.net.t-com.hr User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) In-Reply-To: <9D27D745-2465-4FB2-B7E0-3C5DD411E9B9@mac.com> X-Enigmail-Version: 0.95.6 Sender: news Subject: Re: Performance Tracker project update X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Feb 2008 20:44:11 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig474760FAE99CF3CD10855972 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable Chuck Swiger wrote: > Historically, the Python optimizer wasn't capable of doing much, true, = > but the more recent versions of the optimizer can actually do some=20 > peephole optimizations like algorithmic simplification and constant=20 > folding: >=20 > http://docs.python.org/whatsnew/other-lang.html#SECTION0001320000000000= 000000=20 A quick test with the built-in pystone mini-benchmark (taken out of the=20 standard library so the optimization can be varied) yields [*]: python without -O : 5802.36 python with -O : 5781.39 In both cases, the commands were: >>> import pystone >>> pystone.main(50000) on Python 2.5.1 (r251:54863, Jan 12 2008, 13:07:38) [GCC 4.2.1 20070719 [FreeBSD]] on freebsd7 I guess they still have to work on the optimizer :) [*] This is on a AMD Geode CPU (500 MHz), so don't compare it with=20 something normal :) --------------enig474760FAE99CF3CD10855972 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHrL8KldnAQVacBcgRAmPbAJ0QJPTNyNM06mSe6R0/bu+tCEQMWQCfbgsC 40aRAyafzCCINCty2CE2xew= =7ImO -----END PGP SIGNATURE----- --------------enig474760FAE99CF3CD10855972-- From owner-freebsd-performance@FreeBSD.ORG Fri Feb 8 22:31:47 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7C1D116A476; Fri, 8 Feb 2008 22:31:47 +0000 (UTC) (envelope-from cswiger@mac.com) Received: from mail-out4.apple.com (mail-out4.apple.com [17.254.13.23]) by mx1.freebsd.org (Postfix) with ESMTP id 1DF9113C4D9; Fri, 8 Feb 2008 22:31:47 +0000 (UTC) (envelope-from cswiger@mac.com) Received: from relay12.apple.com (relay12.apple.com [17.128.113.53]) by mail-out4.apple.com (Postfix) with ESMTP id 1028F218ED76; Fri, 8 Feb 2008 14:31:47 -0800 (PST) Received: from relay12.apple.com (unknown [127.0.0.1]) by relay12.apple.com (Symantec Mail Security) with ESMTP id ECA69464002; Fri, 8 Feb 2008 14:31:46 -0800 (PST) X-AuditID: 11807135-a7830bb00000423e-f7-47acd8522335 Received: from cswiger1.apple.com (cswiger1.apple.com [17.214.13.96]) by relay12.apple.com (Apple SCV relay) with ESMTP id D288C420002; Fri, 8 Feb 2008 14:31:46 -0800 (PST) Message-Id: <967A7CF9-D9FE-454F-92E1-68D21CBDFA5E@mac.com> From: Chuck Swiger To: Ivan Voras In-Reply-To: Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v915) Date: Fri, 8 Feb 2008 14:31:45 -0800 References: <4796C717.9000507@cederstrand.dk> <20080123193400.N63024@fledge.watson.org> <4797A245.7080202@cederstrand.dk> <20080123202433.E63024@fledge.watson.org> <4797A802.8060509@FreeBSD.org> <47A0BFE7.4070708@cederstrand.dk> <20080130190000.GA18333@lor.one-eyed-alien.net> <47AC15A5.5020009@cederstrand.dk> <20080208151756.GA35423@lor.one-eyed-alien.net> <9D27D745-2465-4FB2-B7E0-3C5DD411E9B9@mac.com> X-Mailer: Apple Mail (2.915) X-Brightmail-Tracker: AAAAAA== Cc: freebsd-performance@freebsd.org Subject: Re: Performance Tracker project update X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Feb 2008 22:31:47 -0000 On Feb 8, 2008, at 12:43 PM, Ivan Voras wrote: >> Historically, the Python optimizer wasn't capable of doing much, >> true, but the more recent versions of the optimizer can actually do >> some peephole optimizations like algorithmic simplification and >> constant folding: >> http://docs.python.org/whatsnew/other-lang.html#SECTION0001320000000000000000 > > A quick test with the built-in pystone mini-benchmark (taken out of > the standard library so the optimization can be varied) yields [*]: > > python without -O : 5802.36 > python with -O : 5781.39 That's ~ 0.4% difference, or low enough to be lost in the noise, agreed. I suspect that if the Python optimizer becomes smart enough to do dead- code elimination and code motion of invariants outside of loops that one would see a more significant difference. At the present, it's only smart enough to optimize pretty dumb cases that most humans would already deal with... -- -Chuck From owner-freebsd-performance@FreeBSD.ORG Fri Feb 8 22:38:33 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6883916A46E for ; Fri, 8 Feb 2008 22:38:33 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id D27D513C4EE for ; Fri, 8 Feb 2008 22:38:32 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1JNbra-00030f-5Z for freebsd-performance@freebsd.org; Fri, 08 Feb 2008 22:38:30 +0000 Received: from 78-0-94-116.adsl.net.t-com.hr ([78.0.94.116]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 08 Feb 2008 22:38:30 +0000 Received: from ivoras by 78-0-94-116.adsl.net.t-com.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 08 Feb 2008 22:38:30 +0000 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-performance@freebsd.org From: Ivan Voras Date: Fri, 08 Feb 2008 23:38:12 +0100 Lines: 45 Message-ID: References: <4796C717.9000507@cederstrand.dk> <20080123193400.N63024@fledge.watson.org> <4797A245.7080202@cederstrand.dk> <20080123202433.E63024@fledge.watson.org> <4797A802.8060509@FreeBSD.org> <47A0BFE7.4070708@cederstrand.dk> <20080130190000.GA18333@lor.one-eyed-alien.net> <47AC15A5.5020009@cederstrand.dk> <20080208151756.GA35423@lor.one-eyed-alien.net> <9D27D745-2465-4FB2-B7E0-3C5DD411E9B9@mac.com> <967A7CF9-D9FE-454F-92E1-68D21CBDFA5E@mac.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig3936F310693C58B92DFA45C0" X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: 78-0-94-116.adsl.net.t-com.hr User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) In-Reply-To: <967A7CF9-D9FE-454F-92E1-68D21CBDFA5E@mac.com> X-Enigmail-Version: 0.95.6 Sender: news Subject: Re: Performance Tracker project update X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Feb 2008 22:38:33 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig3936F310693C58B92DFA45C0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable Chuck Swiger wrote: > On Feb 8, 2008, at 12:43 PM, Ivan Voras wrote: >>> Historically, the Python optimizer wasn't capable of doing much,=20 >>> true, but the more recent versions of the optimizer can actually do=20 >>> some peephole optimizations like algorithmic simplification and=20 >>> constant folding: >>> http://docs.python.org/whatsnew/other-lang.html#SECTION00013200000000= 00000000=20 >>> >> >> A quick test with the built-in pystone mini-benchmark (taken out of=20 >> the standard library so the optimization can be varied) yields [*]: >> >> python without -O : 5802.36 >> python with -O : 5781.39 >=20 > That's ~ 0.4% difference, or low enough to be lost in the noise, agreed= =2E Ah sorry, fatal omission: that's loops per second, and it's consistent=20 across several runs of with and without -O. I.e. I'm getting _lower_=20 performance with -O :) --------------enig3936F310693C58B92DFA45C0 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHrNneldnAQVacBcgRAuIQAKDPeP3WnYpCnSbOl1XC6jqssBW9gwCgoohu AOsbWifo85uNFlEf7gRXz+A= =22ja -----END PGP SIGNATURE----- --------------enig3936F310693C58B92DFA45C0-- From owner-freebsd-performance@FreeBSD.ORG Sat Feb 9 17:02:55 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E73C816A418 for ; Sat, 9 Feb 2008 17:02:55 +0000 (UTC) (envelope-from freebsd@levsha.org.ua) Received: from expo.ukrweb.net (expo.ukrweb.net [193.125.78.116]) by mx1.freebsd.org (Postfix) with ESMTP id 9D7A013C45B for ; Sat, 9 Feb 2008 17:02:55 +0000 (UTC) (envelope-from freebsd@levsha.org.ua) Received: from levsha by expo.ukrweb.net with local (Exim 4.68 (FreeBSD)) (envelope-from ) id 1JNsSo-000HeH-Sc; Sat, 09 Feb 2008 18:22:02 +0200 Date: Sat, 9 Feb 2008 18:22:02 +0200 From: Mykola Dzham To: Zaphod Beeblebrox Message-ID: <20080209162202.GB71480@expo.ukrweb.net> References: <107794589.20080205140018@starnet.cz> <5f67a8c40802051205t74a38663xd692e2a754d3788b@mail.gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <5f67a8c40802051205t74a38663xd692e2a754d3788b@mail.gmail.com> X-Operating-System: FreeBSD/5.4-RELEASE-p6 (i386) User-Agent: Mutt/1.5.6i Cc: FreeBSD Mailing Lists , "Bc. Radek Krejca" Subject: Re: FBSD 1GBit router? X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Feb 2008 17:02:56 -0000 Zaphod Beeblebrox wrote: > On Feb 5, 2008 8:00 AM, Bc. Radek Krejca wrote: > > > > I have FreeBSD box as router > > FreeBSD pvt-gw.starnet.cz 6.1-RELEASE-p12 FreeBSD 6.1-RELEASE-p12 #2: Wed > > Jan 31 21:28:44 CET 2007 root@pvt-gw.starnet.cz:/usr/obj/usr/src/sys/DL360-G4 > > i386 > > But speed is only about 382 Mbit. I have following values in > > sysctl.conf: > > > > net.inet.ip.fastforwarding=1 > > net.inet.tcp.recvspace=262144 > > net.inet.tcp.sendspace=262144 > > kern.ipc.maxsockbuf=33554432 > > > > The ip.fastforwarding makes a tiny insignificant difference with the caveat > that your box won't show up on traceroutes. Fast forwarding is "fast" by > virtue of the fact that it doesn't decrement TTL. This effect make other sysctl variable: net.inet.ip.stealth net.inet.ip.fastforwarding described in inet(4). I have appreciable increase in productivity at the Core2Duo processor with several em network cards after enabling net.inet.ip.fastforwarding. Before enabling fastforwarding "swi1: net" use 100% of one cpu, the second core stood idle. After enabling fastforwarding "emX taskq" processes use both cores. -- Mykola Dzham, LEFT-(UANIC|RIPE) JID: levsha@jabber.net.ua From owner-freebsd-performance@FreeBSD.ORG Sat Feb 9 23:10:22 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 00F6D16A418 for ; Sat, 9 Feb 2008 23:10:22 +0000 (UTC) (envelope-from daniel@dgnetwork.com.br) Received: from mail.mastercabo.com.br (mail.mastercabo.com.br [200.179.179.14]) by mx1.freebsd.org (Postfix) with SMTP id A2AC113C44B for ; Sat, 9 Feb 2008 23:10:20 +0000 (UTC) (envelope-from daniel@dgnetwork.com.br) Received: (qmail 16018 invoked by uid 1008); 9 Feb 2008 22:43:38 -0000 X-Spam-Checker-Version: SpamAssassin 3.1.6-unknown (2006-10-03) on srvmail1 X-Spam-Level: X-Spam-Status: No, score=-1.5 required=4.7 tests=AWL,BAYES_00 autolearn=ham version=3.1.6-unknown Received: from unknown (HELO ?10.0.1.10?) (daniel@dgnetwork.com.br@200.243.216.68) by mail.mastercabo.com.br with SMTP; 9 Feb 2008 22:43:32 -0000 Message-ID: <47AE2AEC.5060604@dgnetwork.com.br> Date: Sat, 09 Feb 2008 20:36:28 -0200 From: =?ISO-8859-1?Q?Daniel_Dias_Gon=E7alves?= Organization: DGNET Network Solutions User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Sat, 09 Feb 2008 23:39:21 +0000 Cc: Subject: DEVICE_POLLING IF_EM CPU usage X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: daniel@dgnetwork.com.br List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Feb 2008 23:10:22 -0000 Hi, I activated polling in 8 interfaces em0... em7 (ifconfig em* polling) to carry through performance tests, when activating, it had a significant reduction of CPU usage, load average measured of "0,50, 0,69, 0,52" for "0,43, 0,39, 0,21" and the CPU usage (SNMP Graphic) measured of 35% for 5%. Passed some hours I disactivated polling (ifconfig em* - polling) and CPU usage continues low. Somebody could explain this to me? Some information: FreeBSD 6.3-RELEASE CPU: Intel(R) Xeon(TM) CPU 3.20GHz (3200.13-MHz 686-class CPU) Logical CPUs per core: 2 FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs SMP: AP CPU #1 Launched! Interfaces if_em: vendor = 'Intel Corporation' device = '82546EB Dual Port Gigabit Ethernet Controller' Thanks. Daniel