Date: Wed, 26 Mar 2003 12:30:52 -0800 (PST) From: Julian Elischer <julian@elischer.org> To: Jeff Roberson <jroberson@chesapeake.net> Cc: kse@elischer.org Subject: Re: 1:1 Threading implementation. Message-ID: <Pine.BSF.4.21.0303261154100.52134-100000@InterJet.elischer.org> In-Reply-To: <20030326031245.O64602-100000@mail.chesapeake.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, 26 Mar 2003, Jeff Roberson wrote:
> I don't understand? There are relatively minor changes to the kernel to
> support this. Since nice is a property of the process, it makes sense
> that there is only one ksegrp per process. I'm starting to think that the
> ksegrp was overkill in general.
>
Instead of making a new KSE for each thread (and thereby blowing out the
code that expects that NKSE <= NCPU per KSEGRP,
allocate each KSE in a new KSEGRP. The overhead is not that much and
you will keep NKSE/KSEGRP <= NCPU. You will also be able to support
system scope threads with differnet priorities, which is a
Posix requirement. It is the equivalent of adding teh "NEWKSEGRP"
flag to each thread creation call, and should have no other
ramifications. It will also mean that you can se this system if people
add a scheduler that doesn't have KSEs (as discussed previously)
(by durectly scheduling threads).
> > Specifically since My plan is to make the "KSE' structure go away..
> > (by which I mean it is only going to be visible within the particular
> > thread_scheduler that uses it and that externally
> > the only structures visible would be :
> > proc, ksegrp(subproc?) thread and upcall.
>
> For M:N I really think this should be proc, thread, and upcall.
> For 1:1 I only need proc and thread.
For your definition, define "1 thread" as:
"A thread with an attached KSE and KSEGRP"
instead of:
"A thread with an attached KSE"
The logic will be very similar but you will get better functionality
by being able to give different threads different priorities etc.
The KSEGRP structure is small. You will not lose much by doing this..
This is how a 1:1 scheme was envisioned and the items in the different
substructures were distributed to work best in this way..
>
> > The KSE would be allocated only by a call into the scheduler and is part
> > of the "scheduler specific private data".
> >
> > i.e. on creation of a new process, shced_newproc() is called
> > and a KSE is added in there is the scheduler in question wants to use
> > KSEs. If it doesn't, no KSE would be added, but it's still possible that
>
> Yes, I think we need more sched hooks here as well. Having only
> sched_fork() makes things sort of gross. We'll have to hook this all up
> later.
I'll try get it hooked up "sooner rather than later".
I think you can make 1:1 threads in the current system by doing:
kse_create(mbox, NEWGROUP); where the mbox points to the function you
want to run and a new stack. The function just runs as normal, not
knowing that it is atually a UTS thread. Since it never yields to
another thread (by KSE terms) it never does any upcalls an voila.. 1:1
threads. (I am sugesting that we don't need a new syscall to do this,
or, at most a new entrypoint which ends up calling much of the same
code.)
Ok but htis breaks things for M:N threads as in M:N threads, teh mask
would be stored "per process" (or at most per group) and the mask is the
"logical OR" of all the masks for the threads in the group/process.
Having a mask per thread and not having one for the bigger unit
means that the masks for the threads must be updated regularly
(maybe at every kernel entry) to be the OR of the masks for ALL THE USER
THREADS, which means that the UTS must do this explicitly.
I'm not thrilled by all the extra work this is going to make for M:N
threads. (Well at least this is my preliminary reading of it.)
>
> > - if (p1->p_flag & P_THREADED) {
> > + if (p1->p_flag & P_THREADED || p1->p_numthreads > 1) {
> >
> > If you are running threads, please set the P_THREADED flag.
> > if you wnat do differentiate between upcalling threads and 1:1
> > threads, please use some auxhilliary flag.
>
> I'd rather not have a flag. The > 1 check is used only in places where we
> have to suspend multiple threads or go to single threading etc. Processes
> in the 1:1 threading model aren't so special as they are with KSE. They
> don't need to be treated specially except when we're trying to funnel them
> down etc.
Ok, well we'll see with time how it works out and if it is ok, that;s
fine.. If it needs work we can do it then.. this will do for now.
>
> > You should be creating a new KSEGRP (subproc) per thread.
> > I think you will find that if you do, things will fall out easier
> > and you won't break the next KSE changes.
>
> I don't understand what I may break?
You are allocating a thread and a KSE.. KSEs may go away (from being
visible to you). If you are referencing them then things will break.
>
> I don't think M:N is the way to go. After looking things over and
> considering where it is theoretically faster I do not think it is a
> worthwhile pursuit.
I agree with yo an may ways, and I think that given teh choice I'd run
KSEs in "P:Q" mode.. (where we don't multiplex any sleeping threads
and have effectively one kernel thread per sleeping thread".
However M:N threads has one advantage. That is where peopel use the
programming model that makes a thread for every object in a program.
This scheme can lead to tens of thousands of small threads in userland.
effectively you do NOT want those to all be kernel threads. Tere are
languages and libraries that promote such programming models.
effectively each object in teh program is an independent intelligent
entity with its own stack and such.. I would like to be able to support
this.
>
> First off, it is many months away from being even beta quality. I think
> the UTS is far more complicated than you may realize. There are all sorts
> of synchronization issues that it was able to avoid before since only one
> thread could run at any time and there essentially was no preemption. It
> now also has to deal with effecient scheduling decisions in a M:N model
> that it didn't have to worry about before.
I'm not sure that teh issues there are as bad as you think.
>
> Aside from that, there are numerous problems with the kernel not being
> able to identify individual threads of execution. Debugging, scheduling,
> profiling, ktrace are all more difficult in a m:n environment. I think it
> is going to contribute to less effecient scheduling decisions over all. I
> have already wrestled with this in ULE.
You are right about some parts of this, but it IS possible to do these
things.
>
> I feel that this is an overwhelming amount of complexity. Because of this
> it will be buggy. Sun claims that they still have open tickets on their
> M:N while their new 1:1 implementation is totally bug free. How long have
> they been doing m:n? I don't think that with our limited resources we're
> going to be able to do better.
I think that the complexity of the KSE M:N model is a lot less that what
sun did.
>
> Furthermore, m:n's basic advantage is less overhead from staying out of
> the kernel. Also, less per thread resources. I think this is bogus for a
> couple of reasons.
It is bogus for a particular class of threaded applications and
true for a particular class of threaded apps.
>
> First, if your application has more threads than cpus it is written
> incorrectly.
Not neccesarily. that's just one way of looking at threads. Active
component threaded programs use threads as a programming model
(see above) and it is a perfectly valid way of writing a program.
Remember.. "Ours is not to specify how a programmer writes, but to
allow the programmer to have a s wide a choice as possible about what
he wants to do."
> For people who are doing thread pools instead of event
> driven IO models they will encounter the same overhead with M:N as 1:1.
True. That is one model of threading..B
In an IO bound app all waiting threads will have kernel contexts so in
effect it approaches 1:1
> I'm not sure what applications are entirely compute and have more threads
> than cpus. These are the only ones which really theoretically benefit. I
> don't think our threading model should be designed to optimize poorly
> thought out applications.
As I said. there are people who like this method of programming.
I don't want to have to say "we only support model A of thread
programming, and if you want model B, it'll really suck."
(This is not saying we shouldn;t have a 1:1 library available..
it's a good idea).
> Furthermore, the amount of work done per slice has been growing with
> processor speeds. Slice time is adjusted for user experience and so it
> remains constant. This means that the constraints are different from when
> this architecture started to come about many (10 or so?) years ago.
> Trying to optimize context switches between threads just doesn't make
> sense when you do so much work per slice.
This is a good argumant, but it still doesn;t make sence to have 10,000
threads all in the kernel.
>
> Then if you look at the number of system calls and shenanigans a UTS must
> do to make proper scheduling decisions it doesn't look like such an
> advantage. I feel that the overhead of all the layers comes close to the
> savings from doing some of it without entering the kernel.
So far it's not doing that much..
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?Pine.BSF.4.21.0303261154100.52134-100000>