From owner-freebsd-arch Mon Nov 20 5:33:40 2000 Delivered-To: freebsd-arch@freebsd.org Received: from mail.interware.hu (mail.interware.hu [195.70.32.130]) by hub.freebsd.org (Postfix) with ESMTP id C36EB37B479; Mon, 20 Nov 2000 05:33:29 -0800 (PST) Received: from victoria-203.budapest.interware.hu ([195.70.63.203] helo=elischer.org) by mail.interware.hu with esmtp (Exim 3.16 #1 (Debian)) id 13xr4X-0004nJ-00; Mon, 20 Nov 2000 14:33:25 +0100 Message-ID: <3A192821.13463950@elischer.org> Date: Mon, 20 Nov 2000 05:33:21 -0800 From: Julian Elischer X-Mailer: Mozilla 4.7 [en] (X11; U; FreeBSD 5.0-CURRENT i386) X-Accept-Language: en, hu MIME-Version: 1.0 To: jasone@freebsd.org, arch@freebsd.org, smp@freebsd.org Subject: Re: Threads (KSE etc) comments References: <3A15A2C1.1F3FB6CD@elischer.org> Content-Type: text/plain; charset=iso-8859-15 Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Netscape made an unholy mess of this. I'm going to try send it again and see if I can unscramble the mess. I also clarified a few points. julian. Julian Elischer wrote: > Hi jason. > I read your Nov 7 doc on the threading model. I have a few comments.. I've been thinking about the scheduling queues, and how to make sure that the process (KSEG actually) acts fairly with respect to other processes. I was confised for a while by your description. I think part of my confusion came from something that we specified in the meeting but has not been written in your document directly. Let me see if we are agreed on what we decided.. A KSEG can only have as a maximum N KSEs associated with it, where N is the number of processors, (unless artificially reduced by a lower concurency declaration). (you said this but only indirectly). In general, KSEs are each assigned to a processor. They do not in general move between processors unless some explicit adjustment is being made(*), and as a general rule, two KSEs will not be assigned to the same processor. (in some transitional moments this may be allowed to briefly happen) This in general if you run a KSEC on the same KSE it was run on last time, you should be on the same processor, (and get any affinity advantages that might exist). (*)I am inclined to make the requirement of binding KSEs to processors HARD,as this allows us to simplify some later decisions. For example, if we hard bind KSEs to procesors then since we assign a different communications mailbox for each KSE we create, we can be sure that different KSEs will never preempt each other when writing out to their mailboxes. this also means that since there can only be one UTS incarnation active per KSE (or one KSE per UTS incarnation), that we can not have a UTS preempted by another incarnation on the same processor. We can therefore make sure that there needs to be no locking on mailboxes, or even any checking. I think this is what we decided.. is this correct? The binding is not really mentioned in your document. When we were talking about it, (at least in my memory) Each KSE had a mailbox. My memory of this was that we called a KSE creation call with a different argument, thus each KSE had a different return stack frame when it made upcalls. In the version you have outlined, there is no KSE creation call only KSEG creation calls. Thus all upcalls have the same frame, and there is the danger of colliding upcalls for different processors. I think it works more naturally with everything just 'falling into place' if we have calls to create KSEs rather than KSEGs. The "make KSEG" call is simply a version of the "make KSE" call that also puts it into the new different group. You are left with teh very first 'original' thread being different in my shceme, but my answer to this would be to simply make the first "make KSE" call reuse the current stack etc. and not return a new one. My memory (where is that photo of the whiteboard that Nicole was supposed to send us) is that each KSE is assigned a differnt mailbox address in userland, which is associated with the frame that it will do upcalls on. One of the fields of the mailbox contains a pointer to a userland context structure which contains apece where the KSE should dump the user context should it need to, and a pointer to other such structures. This structure is defined by the kernel, but included in the UTS's 'per thread info'. Since there is one per thread, there is never a problem in running out of them when the kernel links them together in a linked list of completed operations. This is the structure you were talking about as if it were on the KSE upcall stack. In my memory it was preallocated as part of the thread information and completed (or suspended threads (KSECs) are simply linked together inj a list by the kernel, but it doesn;t have to allocate them. (so you discussion about how many need to be provided for is short circuited. When the thread makes a system call, the KSE looks in the mailbox for the context structure for this thread, and if the thread blocks or resumes, it can save any information it needs to tell the UTS there. The UTS sets the pointer into the mailbox when it schedules the thread, so even involintary blockages (e.g. page faults) have the pointer available. When the UTS is running it's own work, it ZERO's this pointer, which lets the kernel know that it is not really in a safe state for pre-emmpting. I think that we decided that a page fault in the UTS simply blocked until it was satisfied. When an upcall occurs, the stack frame it occurs on, and hence the mailbox pointed to are automatically correct, so the UTS doesn't even have to look it up. (the mailbox is allocated as a local variable in the frame of the KSE creation call and is this in the local frame of the upcall. This is something like I imagined the UTS would do to fire off a new KSE. The reason I was thinking of it this way was so that each KSE has a UTS supplied mailbox and (smallish) stack. /* * use make_new_kse() to do exactly that. * Returns -1 on failure and 1 on success. * * cookie allows the UTS to have it's own way of identifying the * KSE/thread. * This stack is effectively lost to us so we first switch * to a small throw-away stack. It need only have enough space in it for * the upcalls to call the UTS, and whatever the UTS will need. * Some time after creation, there will be an upcall on the new KSE * looking for work. * I could imagine wiring this UTS stack down.. */ void start_new_kse(void * cookie, jmp_buf *jb) /*XXX is jb right? */ { struct kse_mailbox; int return_value; bzero(kse_mailbox, sizeof(kse_mailbox)); return_value = kse_new(&kse_mailbox); switch (return_value) { case -1:  perror("make_new_kse() failed"); _longjmp(jb, -1); case 0: printf ("successfully created kse %d\n", kse_mailbox.kse_id); _longjmp(jb, 1); exit (1); /* not reached */ default: printf(" An upcall of type %d occured\n", return_value); uts_scheduler(cookie, &kse_mailbox, return_value); /* must never return */ printf ("it returned!\n"); exit (1); } } make_new_kse(void * cookie) { int retval; jmp_buf env; if ((retval = _setjmp(env)) == 0) { load_new_stack() /* load a new smaller stack, but copy the top 100 bytes or so from the old stack so that our local variables appear to be the same. */ start_new_kse(cookie, env); /* not reached */ } return (retval) } When we have per-processor scheduling queues, there is only at most ONE KSE from any given KSEG in the scheduling queues for any given processor. With the single scheduling queue we have now do we allow N to be in the queues at once? (or do we put the KSEG in instead?) The terms KSE etc. have probably served their useful life. It's time to think of or find names that really describe them better KSE -- a per process processor.. slot? openning? (a-la CAM/SCSI) KSEC ---- stack plus context... KSC..trying to do something (task?) KSEG ---- a class of schedulable entities.. A slot cluster? :-) PROC ---- probably needs to stay the same. -- __--_|\ Julian Elischer / \ julian@elischer.org ( OZ ) World tour 2000 ---> X_.---._/ presently in: Budapest v To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message