Date: Mon, 30 Oct 2000 11:52:09 -0800 From: Alfred Perlstein <bright@wintelcom.net> To: David Greenman <dg@root.com> Cc: Bosko Milekic <bmilekic@dsuper.net>, freebsd-net@FreeBSD.ORG Subject: Re: MP: per-CPU mbuf allocation lists Message-ID: <20001030115209.G22110@fw.wintelcom.net> In-Reply-To: <200010301927.LAA01623@implode.root.com>; from dg@root.com on Mon, Oct 30, 2000 at 11:27:50AM -0800 References: <Pine.BSF.4.21.0010301256580.30271-100000@jehovah.technokratis.com> <200010301927.LAA01623@implode.root.com>
next in thread | previous in thread | raw e-mail | index | archive | help
* David Greenman <dg@root.com> [001030 11:30] wrote: > > I recently wrote an initial "scratch pad" design for per-CPU mbuf > > lists (in the MP case). The design consists simply of introducing > > these "fast" lists for each CPU and populating them with mbufs on bootup. > > Allocations from these lists would not need to be protected with a mutex > > as each CPU has its own. The general mmbfree list remains, and remains > > protected with a mutex, in case the per-CPU list is empty. > > I have only one question - is the lock overhead really so high that this > is needed? Yes, see www.horde.org. The intent is that for the most part these allocations only hit the local lock and the only time one has contention is during a producer+consumer situation where the producer and consumer are on different CPUs. Also... Bosko, you should realize the important 'twist' that horde does, which is that it's able to detect which freelist a buffer should go back onto, meaning a buffer allocated on CPU-1 but winds up being free'd is free'd back to CPU-1's freelist. Dillon (cache miester that he is) explained it to me (and several books on SMP agree): Basically when you have a procducer/consumer, the producer is writing to the memory while the consumer is only reading. Now if the consumer free's to his own list rather than the producers list, what happens is that when the consumer does his next allocation he most likely is going to write to a page that has it's writablility "owned" by the producer, this generates bus traffic to inform the producer that he can no longer write/read cache the memory. By freeing to the producer's pool you avoid invalidating the producer's cache. There is a gotcha here though that would need some thourough investigation, thinking about how mbufs work, a lot of the time the mbuf headers are accessed right before they are free'd to either trim or remove them from linked lists, so perhaps the free'ing to the CPU that the mbuf was allocated from should only happen for mbuf clusters, not mbuf headers. -- -Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org] "I have the heart of a child; I keep it in a jar on my desk." To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-net" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20001030115209.G22110>