From owner-freebsd-arch@FreeBSD.ORG Mon Mar 15 19:22:11 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 50FD916A4CE for ; Mon, 15 Mar 2004 19:22:11 -0800 (PST) Received: from smtp.distributel.net (cns2.distributel.NET [66.38.181.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id E859843D39 for ; Mon, 15 Mar 2004 19:22:10 -0800 (PST) (envelope-from bmilekic@technokratis.com) Received: from godel.mtl.distributel.net (nat.MTL.distributel.NET [66.38.181.24]) by smtp.distributel.net (8.12.6/8.12.6) with ESMTP id i2G3MAM2033385 for ; Mon, 15 Mar 2004 22:22:10 -0500 (EST) Received: from godel.mtl.distributel.net (localhost [127.0.0.1]) i2G3KT6p003848 for ; Mon, 15 Mar 2004 22:20:29 -0500 (EST) (envelope-from bmilekic@technokratis.com) Received: (from bmilekic@localhost)i2G3KTwh003847 for freebsd-arch@freebsd.org; Mon, 15 Mar 2004 22:20:29 -0500 (EST) (envelope-from bmilekic@technokratis.com) X-Authentication-Warning: godel.mtl.distributel.net: bmilekic set sender to bmilekic@technokratis.com using -f Date: Mon, 15 Mar 2004 22:20:29 -0500 From: Bosko Milekic To: freebsd-arch@freebsd.org Message-ID: <20040316032029.GA3735@technokratis.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.1i Subject: mbuma: network buffers & UMA X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Mar 2004 03:22:11 -0000 The post below has received exactly zero attention on the freebsd-current mailing list. I'm assuming this is because there are other exciting threads going on there right now, such as style debates and lovely feature requests. :-) I would like to commit this RSN. I need feedback from the Release Engineering people on how close we are to a RELEASE and if it's OK for me to break netstat -m stats for a few weeks or, if not, what the appropriate course of action is. Fellow earthlings, I would like to request some testing on diverse architectures of the following patch (~130K): http://people.freebsd.org/~bmilekic/code/mbuma-v2.diff In short, the above accomplishes several things and is the result of one of my now mature p4 repositories which I have been able to again address following a generous hardware donation. This basically gets rid of the existing mbuf allocator and replaces it with some routines which get hooked on top of UMA, the existing general-purpose SMP-friendly allocator in -CURRENT, after adding some extensions to UMA which hopefully make allocations faster than they would be without them. The current list of implications (whether good or bad is for you to decide): - NMBCLUSTERS, as a compile-time option, is now gone. The kern.ipc.nmbclusters tunable and sysctl still exist and are both read+write. Currently, the sysctl does not effect anything but will soon be made to allow for runtime (dynamic) modification of the maximum number of mbuf clusters allocatable cap. If you set the tunable to zero, though, the number of clusters allocatable is unbounded and the system will scale according to demand. Be careful... leaving these unbounded may not be what you really want. - Memory allocated to network buffers can (and now will) be reclaimed by UMA when required and, for example, after a large network spike. - Current on-par performance with the existing allocator. Soon some modifications to UMA that improve performance drastically in the common case may change that, though - for the better. - For developers: M_WAIT behavior now is the same as for all other UMA allocations, and is no longer special for network buffers. - Currently netstat(1) mbuf stats ('netstat -m') are broken and instead the following is displayed: fermat% netstat -m Mbuf statistics are currently unavailable via netstat(1), see UPDATING. This is only a TEMPORARY measure in -CURRENT. For current mbuf and cluster allocation stats, see sysctl vm.zone I don't plan to fix them prior to doing some other work (which may change the way stats work, anyway) in UMA. Hopefully this won't be too much of a problem for most of you. - So far has been very stable on my dual Athlon and a friend's x86 UP machine (I think it's UP, anyway). Some other architectures, particular 64-bit ones, would be worth testing. - Some things (e.g., dev/nsp) I just noticed might be broken by the above changes due to slight time-delay between the vendor p4 branch and the CVS checked out repo I used to generate the final diff. If something you're using is broken by the diff, it is only broken slightly, and you'll notice it immediately during kernel compile, so you can fix it (it's generally just changing some includes) or Email it to me and I'll send you a custom diff. This only applies to the current diff posted above and will not be committed. That's all I could think of right now. I would like to commit this soon as I have other things in the pipeline that build on top of it. I should also mention that there are various other implications in addition to the above but are more relevant to the developer than the user, so they'll probably appear in a really large commit log. Reviews and constructive criticism is both welcomed and appreciated. Please leave me in the To: line - although I am subscribed to -current, I'd like to see feedback in my main box. Regards, -- Bosko Milekic * bmilekic@technokratis.com * bmilekic@FreeBSD.org TECHNOkRATIS Consulting Services * http://www.technokratis.com/ "It is impossible for anyone to begin to learn what he believes he already knows." -- Epictetus (c.a.d. 55-c135)