Date: Wed, 10 Aug 2005 09:11:48 -0400 From: John Baldwin <jhb@FreeBSD.org> To: freebsd-arch@freebsd.org Cc: Andre Oppermann <andre@freebsd.org> Subject: Re: Special schedulers, one CPU only kernel, one only userland Message-ID: <200508100911.50004.jhb@FreeBSD.org> In-Reply-To: <42F9ECF2.8080809@freebsd.org> References: <42F9ECF2.8080809@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wednesday 10 August 2005 08:02 am, Andre Oppermann wrote: > When using FreeBSD as a high performance router there are some desirable > changes to the way multiple CPUs are handled. Normally a second CPU > doesn't add much (if any) performance to routing because of locking > overhead and packets randomly being processed by the CPUs wasting cache > efficiency. On the other hand having just one CPU is not optimal in running > the routing daemon in userland. When there are large changes to the table > (eg. BGP full feed flap) userland sucks time away from the packet > forwarding in the kernel. > > The idea is to combine both worlds by designating CPU0 exclusively for > all kernel processing (thus avoiding the expensive mutex synchronization > and bus locking instructions) and CPU1 exclusively for all userland > processing. Whenever a userland program does a syscall the kernel CPU > will take over. When it's done, the process get run by the userland > CPU again. That way we get a very good scalability out of two CPUs for > this particular task. > > Hence my question to the SMP and scheduler gurus: How well does the current > SMP and scheduler architecture lend itself to this kind of special > handling? Is it just a matter of modifying (or plugging in) the schedule or > are there more involved things to consider? First, you might want to go find a copy of "Unix Systems for Modern Architectures: Symmetric Multiprocessing and Caching for Kernel Programmers" by Curt Schimmel as he describes this model as one of the early SMP models for UNIX kernels before fine-grained locking was used. (It's known as a master-slave system IIRC). I think this is the model that BSD/OS employed for SMP in its 4.x series before they did their version of SMPng. If you really want to do this, you can actually go back to spl's and use spl on the Master CPU and the slave CPUs just run user processes. You probably aren't looking for that much work however. In the FreeBSD 5+ world, you might be able to do this using a custom scheduler, but I think you'd need some other hooks as well (need to compile out lock on x86, note that only x86 has this weird case of atomic ops being different on UP and SMP). You'd need a way to know if a thread was about to return to userland or not and thus if it were eligible for running on a slave. You could probably manage that via sched_userret(), but you would need a new sched hook for kernel entry that was called from trap() and syscall(). You would also want to modify the APIC code to force all interrupts to the BSP (not hard), and you would probably need some sort of sched_lock still to manage the shared queues. However, I'd recommend that you go buy the aforementioned book first and read it if you haven't already. -- John Baldwin <jhb@FreeBSD.org> <>< http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve" = http://www.FreeBSD.org
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200508100911.50004.jhb>