Date: Mon, 25 Nov 2002 17:43:15 -0500 From: Bosko Milekic <bmilekic@unixdaemons.com> To: Andrew Gallatin <gallatin@cs.duke.edu> Cc: Julian Elischer <julian@elischer.org>, Robert Watson <rwatson@freebsd.org>, Luigi Rizzo <rizzo@icir.org>, current@freebsd.org Subject: Re: mbuf header bloat ? Message-ID: <20021125174315.C75673@unixdaemons.com> In-Reply-To: <15842.37927.919140.910293@grasshopper.cs.duke.edu>; from gallatin@cs.duke.edu on Mon, Nov 25, 2002 at 04:20:39PM -0500 References: <15840.8629.324788.887872@grasshopper.cs.duke.edu> <Pine.BSF.4.21.0211232306410.28833-100000@InterJet.elischer.org> <15841.17237.826666.653505@grasshopper.cs.duke.edu> <20021125130005.A75177@unixdaemons.com> <15842.27547.385354.151541@grasshopper.cs.duke.edu> <20021125160122.A75673@unixdaemons.com> <15842.37927.919140.910293@grasshopper.cs.duke.edu>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, Nov 25, 2002 at 04:20:39PM -0500, Andrew Gallatin wrote: > I'm not sure what you mean. The problem is that Giant inhibits > scaling on SMPs and I want to get all network drivers out from under > it, not just mine. > > Requiring Giant for mbuf allocations effectivly defeats your elegant, > cleverly designed mbuf allocator with lock-free per-cpu caches, etc. > And it introduces all kinds of windows for races and locking errors if > SMP safe drivers must drop all their mutexes and grab Giant for mbuf > allocations. > > And you cannot grab Giant inside the mbuf allocation code itself > because the mutex rules prohibit acquiring Giant while holding any > other locks. From mutex(9): > > Giant > If Giant must be acquired, it must be acquired prior to acquiring other > mutexes. Put another way: it is impossible to acquire Giant non-recur- > sively while holding another mutex. It is possible to acquire other > mutexes while holding Giant, and it is possible to acquire Giant recur- > sively while holding other mutexes. Well, first of all, I never call kmem_malloc() with any locks held so this argument about grabbing Giant while other locks are held is not applicable in my case. Given that you call the allocator with locks held, then you should only be doing so with M_DONTWAIT as an M_TRYWAIT call may result in a call to m_drain() which - if you are holding any locks - may lead to lock order reversals. So given that you can only call the allocator with M_DONTWAIT for device drivers or any other code paths holding locks, then it would make sense to just make the M_DONTWAIT case never call the VM routines as well. Either that, or make sure that a call to kmem_malloc() with M_DONTWAIT can do its job without requiring Giant (is this possible?) > Emperically, I vaguely remember marking my driver as SMP safe (with > witness and invariants off, of course) provided something like a > 30-40% performance increase on a dual 1GHz PIII system. It was still > not as fast as stable, but the current/stable performance difference > was no longer embarrassing. > > Drew > -- Bosko Milekic * bmilekic@unixdaemons.com * bmilekic@FreeBSD.org To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-current" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20021125174315.C75673>