Date: Sun, 16 Feb 2003 21:35:52 -0500 From: Bosko Milekic <bmilekic@unixdaemons.com> To: freebsd-arch@freebsd.org Subject: mb_alloc cache balancer / garbage collector Message-ID: <20030216213552.A63109@unixdaemons.com>
next in thread | raw e-mail | index | archive | help
I've finally gotten around to implementing the cache balancer/garbage collector for the mbuf allocator. Actually, it's been sitting in a local tree for a while but I finally got one of my -CURRENT machines back up and was able to debug it. Here's a little about what it does right now: - Gets woken up if it is detected that at least one of the caches has a number of mbufs or clusters lower than the tunable low-watermark. - Gets woken up if it is detected (on free) that the global cache has more objects in it than the tunable high-limit watermark (actually, I forgot to add this wakeup in the patch I post below but it's trivial to add). - Checks the per-CPU caches and global cache and if there are less than low-watermark objects replenishes them. - Checks the global cache and if the number of objects is above the tunable limit it frees a chunk of memory back to the system, without interfering with simultaneous network buffer allocations (it doesn't lock up the per-cpu caches while doing this) and without increasing mb_free() latency at all (because the lazy freeing is done from a kproc context). Soon, as we whack this thing around, I hope to implement some auto-tuning algorithms and have the daemon auto-tune its watermarks maximizing performance but also allowing the rest of the system to recover unused physical pages more efficiently. What does this mean for us on the long term? One of the things it means is that we continue to have a high performance scalable network buffer allocations but while also being able to free resources to the rest of the system. The auto-tuning mechanism can be made as complicated as we would like it to be as all the computations would be done from the context of the mbufd kproc, and not in any critical allocation paths. What does this mean for us in the really short term? It means that we can finally make all M_DONTWAIT allocations NOT interface with the VM subsystem at all. Why is this good in the really really short term? For one, you can have network device drivers call the mbuf code without Giant because they'll know for a fact that Giant will never be needed down the line. Since the cache balancer will replenish caches when they're under a low watermark, assuming a well-tuned system, no noticable impact will be felt on mbuf allocations and deallocations. The patch is: http://people.freebsd.org/~bmilekic/code/mbufd.patch I don't think it's quite ready for commit yet. I have to clean up a few minor things (need to make sure that it's totally safe to dynamically change the watermark sysctls in all cases, for one), and test a little longer (so far, so good). Feedback is welcome. In particular, if anyone is familiar with clever cache auto-tuning algorithms, that person's input would be really valuable. Regards, -- Bosko Milekic * bmilekic@unixdaemons.com * bmilekic@FreeBSD.org "If we open a quarrel between the past and the present, we shall find that we have lost the future." To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20030216213552.A63109>