Date: Sun, 28 Nov 1999 16:48:20 -0800 (PST) From: Matthew Dillon <dillon@apollo.backplane.com> To: Julian Elischer <julian@whistle.com> Cc: arch@freebsd.org Subject: Re: Threads stuff Message-ID: <199911290048.QAA47070@apollo.backplane.com> References: <Pine.BSF.4.10.9911281602440.544-100000@current1.whistle.com>
next in thread | previous in thread | raw e-mail | index | archive | help
:>
:> * Kernel stack sharing by multiple threads in the non-blocking case,
:> which can lead to better memory and cache resource utilization.
:>
:> * An infinite number of threads can be created as long as they do not
:> all need a kernel stack simultaniously.
:>
:> * Kernel management issues are complex, but not terribly so. For
:> example, you have a thousand threads and they all suddenly decide
:> to block inside the kernel - the kernel needs a mechanism to
:> disallow kernel entry (setup the thread to force a restart of the
:> system call and then switch threads because no KSE resources are
:> available).
:
:The UTS can know its limit on KSEs and simply catch them on the way
:down. Count them at the door like a night club bouncer.. one out, one in..
::-)
This isn't necessarily useful for the UTS to know. The UTS needs a
virtual cpu resource 'environment' and should not have to know anything
beyond that. So, for example, the number of cpu's you tell the UTS
it owns does not have to be the same as the number of actual cpu's in
the system or even the number of cpu's you intend to schedule it across.
:> Problems:
:>
:> * You have to manage dynamic allocation and deallocation of KSE's
:> (verses simply preassigning one per thread).
:>
:
:I don't think this is too hard, and we may endup using the
:same code in things like interrupt handler threads. :-)
In the scheme I am thinking of, an interrupt handler thread can simply
be a KSE associated with a single 'kernel' special 'process'.
:> * You have non-deterministic resource utilization for an application.
:> The worst case KSE useage is going to be one per thread, but the
:> userland may well allow thousands of threads to be allocated first
:> and then realize only later that it does not have the kernel
:...
:
:programmer error?
:In the current world, a programmer may decide to fork 1000000 times too.
In the world of threads the situation is different. Certain programming
problems are best implemented if you can assume a huge number of
available 'threads'. We have to make the concious decision to either
support or not support such environments. I believe it would be possible
to support such environments using the dynamic KSE concept so while I
do not in general like the idea of dynamic KSE's, I'm willing to go along
with it if we can find a sensibile way to implement them.
Also keep in mind that we may, years down the line, wish to expand the
environment to support massively parallel computing - cross platform
computing. In such environments several hundred thousand threads is
not uncommon.
:> * You compound the management issues within the userland scheduler
:> itself because not only can the userland scheduler switch between
:> threads, now so can the kernel when it decides it must block.
:...
:
:No the blocked thread 'returns' to the UTS, notifying it of its blocking.
:There has to be a UTS no matter what, even if it's copntrolling the
:thread contexts by remote control in the kernel (yur suggestion).
:If the context is all in user memory, it's pageable. In the kernel it's wired kernel memory.. which is more valuable?
Well, if you do this you are doubling your context switch overhead. It
may work for system calls but will definitely not work for faults or other
potential blockages without adding a lot of complexity. Considering
the fact that programs use mmap() more and more heavily, having to deal
with faults is a major consideration.
:> such as VM faults, and cannot schedule around such things itself.
:> This can lead to non-deterministic operation.
:
:The UTS can have upcalls for these events. (see the paper on
:Scheduler activations)
Yes, I understand where you are coming from. I believe the scheme to be
too complex, however. Hmm. Let me restate that: I think the kernel
ought to have the support for a UTS, but that it should not be required
for native thread support to function. The kernel should be able to
support threads natively without a UTS simply because most threaded
applications will not need the added sophistication. This isn't to say
that the UTS is entirely absent, only that it need not function in the
heavy-handed complex way you've described in most cases and can simply
be a simpler skeleton that keeps track of the threads.
:> better able to actualy get the thread running deterministically
:> because the userland scheduler might end up 'stuck' in a VM
:> fault or something similar.
:
:that is unlikely to occur.
:I would assume that any thread library would get that memory wired down
:and it would be being touched all the time anyhow.
Maybe, maybe not. It is an added complication and should not be required
for the kernel to operate. Requiring wired down memory for proper
operation is *bad* in many cases. If multi-threaded applications become
ubiquitous, requiring wired down memory eats a valuable resource for idle
processes (unless even more complexity is added to deal with that case
on top of the complexity that already exists)
:> * First, that thread runnability be controlled through a system call.
:> Ther userland can determine when to halt and run a thread and can
:> even assign a cpu, but it must make a system call to actually
:> schedule or deschedule the thread.
:
:We are rforking to make subprocs. a subproc runs at one priority on one CPU.
:it has it's struct proc, (or most of one :-) and the threads are assigned
:to it. Any KSE that is created is automatically hung off that subproc.
:it will ony be run on that cpu unles there is intervention.
:(of which there may be may types, fro example a related subproc may be totally idle at teh time a KSE becomes unblocked.. it would be a waste to not use that other CPU)
This seems like an added unnecessary complication that would be
difficult to scale dynamically to different topologies. Or, put another
way, the burden of knowledge is much greater with this scheme then with
other schemes.
You already have KSE's, that's all the execution environment you really
need. The KSE's will have an associated process, but the kernel is able
to schedule KSE's across available cpu's even if several are
associated with the same process. Every *active* thread needs to have
an active KSE associated with it in order to handle faults, even if it
is not currently in a system call, which means that it is the KSE's
the kernel associates with a cpu, not a process. Runnable (but not
active) threads do not need KSE's associated with them if they are not
in the midst of a system call.
An outline of my scheme would simply be:
* You have a process, the process has threads
* Each thread has a kernel Thread (or Task) structure associated
with it.
* KSE's are assigned to threads as needed, with only two cases that
*require* a KSE assignment.
(1) When the thread is currently running on a cpu, a KSE must be
assigned to it even if the KSE's stack is not being used at
the moment.
Reasoning: So the kernel can preemptively switch away from the
thread, take interrupts, faults, and other things.
(2) When the thread is blocked in a system call, a KSE must be
assigned to it to hold the kernel stack state.
If you do things this way then the implementation within the kernel
becomes trivial. The kernel scheduler only needs to know about KSE's
and MMU contexts (extracted from the associated struct proc), and that's
it. You can still impose scheduling requirements via a UTS by having
the UTS give 'hints' to the kernel, but the UTS would not be
*synchronously* required to wakeup a thread or put it to sleep.
-Matt
Matthew Dillon
<dillon@backplane.com>
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-arch" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199911290048.QAA47070>