Date: Mon, 17 Feb 2003 20:33:06 -0500 From: Bosko Milekic <bmilekic@unixdaemons.com> To: Matthew Dillon <dillon@apollo.backplane.com> Cc: freebsd-arch@FreeBSD.ORG Subject: Re: mb_alloc cache balancer / garbage collector Message-ID: <20030217203306.A67720@unixdaemons.com> In-Reply-To: <200302180101.h1I11AWr001132@apollo.backplane.com>; from dillon@apollo.backplane.com on Mon, Feb 17, 2003 at 05:01:10PM -0800 References: <20030216213552.A63109@unixdaemons.com> <15952.62746.260872.18687@grasshopper.cs.duke.edu> <20030217095842.D64558@unixdaemons.com> <200302171742.h1HHgSOq097182@apollo.backplane.com> <20030217154127.A66206@unixdaemons.com> <200302180000.h1I00bvl000432@apollo.backplane.com> <20030217192418.A67144@unixdaemons.com> <20030217192952.A67225@unixdaemons.com> <200302180101.h1I11AWr001132@apollo.backplane.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, Feb 17, 2003 at 05:01:10PM -0800, Matthew Dillon wrote: > > : One more thing I forgot to add that may help clear this up: the less > : the daemon runs, the better. Right now I can see how many times it > : ran by looking at a sysctl-exported counter, mbuf_daemon_ran. I can > : see that it only runs once to populate the caches and then only runs > : if I forcibly spike and return back to 'normal' steady activity and if > : I change the watermarks to force it to run. > : > : We should probably eventually have it track and detect an acceleration > : of running occurances and then, according to that, change the > : watermarks if it starts to increase (i.e., if it starts to run too > : much). As I said, in the normal case, it shouldn't run often. This > : thing doesn't wake up every N ticks. > : > :-- > :Bosko Milekic * bmilekic@unixdaemons.com * bmilekic@FreeBSD.org > > I guess I still don't understand the point of the daemon. The per-cpu > caches are limited (in your patch) to 512 mbufs / 128 clusters. This > represents very little memory even if you multiply by ncpus. We shouldn't > have to 'balance' anything. Who cares if there are 511 mbufs sitting > on cpu 0's cache that aren't being used? These numbers are going to be > tuned for the machine (for example, based on the amount of main memory), > and are far smaller then the total possible. I never said that those (totally arbitrary, by the way) numbers are ideal. In fact, I think they should be changed. > The only case that matters is if a per-cpu cache gets blown up by an > inordinate number of frees being done to it. That is, when the mbuf > or cluster count exceeds mbuf_limit or clust_limit. > > Why is the daemon more preferable for handling this case verses freeing > a bunch (like 8 or 16) mbufs/clusters on the fly at the time of the > free when the per-cpu cache exceeds the limit? I don't see any advantage > to having the daemon at all, and I see several disadvantages. You can't just 'free' a bunch of mbufs back to the VM. You free them wherever you got them from (usually your pcpu cache). If you exceed mbuf_limit on your pcpu cache you'll migrate a bucket over to the global cache, which is what you want. However if your global cache becomes too 'blown up' as you say, then you may want to recover the unused physical pages. Doing that directly from the free has several disadvantages; It can be expensive in more ways than one; for one, the VM call itself is extra overhead. Secondly, sometimes freeing a page means traversing the cache until you hit a page worth of free mbufs to free, so even though you may really need to free a page you'll never actually get to freeing it unless you start traversing the list of buckets in the cache; and that's expensive for a simple free - common case or not. By doing the freeing from the kproc context you're not interfering with parallel allocations but you're also not taking longer than it takes to just cache the data being freed for the free case. That's a big advantage. By having the kproc also fill the pcpu caches according the the configurable watermarks you're ensuring to have a certain number of objects cached and ready for immediate allocations, again without taking longer than it takes to just retrieve the object being allocated from the cache for the allocation case. I think that your argument regarding having to worry about the daemon interfering with the VM system is reasonable but I think that what's been left out is that in the case of mbufd, the behavior is entirely determined by the watermarks, which are variable. The good news is that if mbufd is interfering too much, the watermarks can be modified so that it interferes less. With that said, I don't see how mbufd will eat away from the VM system. It doesn't try to replace the VM system, it just tries to avoid having to go to it for 95% of network buffer allocations. Perhaps I can address your concerns if you give me a specific example where you think the daemon is doing a bad thing, then I can work on fixing that. I think for corner cases it would even make sense to explicitly lower the watermarks (thus forcing the daemon to drain the caches) directly from the VM, if that's really determined to be an issue. > -Matt > Matthew Dillon > <dillon@backplane.com> -- Bosko Milekic * bmilekic@unixdaemons.com * bmilekic@FreeBSD.org "If we open a quarrel between the past and the present, we shall find that we have lost the future." To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20030217203306.A67720>