Date: Thu, 15 Feb 2001 23:36:10 +0000 (GMT) From: Terry Lambert <tlambert@primenet.com> To: seth@pengar.com (Seth Leigh) Cc: nate@yogotech.com (Nate Williams), freebsd-smp@FreeBSD.ORG Subject: Re: possible problem with SMP? Message-ID: <200102152336.QAA09980@usr08.primenet.com> In-Reply-To: <5.0.2.1.0.20010215020051.01b9fc70@hobbiton.shire.net> from "Seth Leigh" at Feb 15, 2001 02:21:14 AM
next in thread | previous in thread | raw e-mail | index | archive | help
> There are problems of cpu starvation with a Many to Many model in that > usually (unless you manually step up the concurrency) the Many to Many ends > up being Many to Number of CPUs. If Number of CPUs is equal to 1, then > usually you will only get one LWP beyond the ones that are sleeping. Using > the Solaris libpthread, a user-level thread will not give up the LWP it's > using until it voluntarily makes a call into libthread which gives > libthread an opportunity to deschedule that thread and schedule a different > one. To illustrate this, I talked to a customer the other day whose > application had around 200 threads. He had around 40 LWPs created, but 39 > of them were blocked in the kernel, and 1 appeared to be running when > looking at the threads in dbx. The application appeared to be hung, > though. Finally he figured out that the one thread that was actually > running on the one free LWP was in fact running a hugely inefficient code > chunk that was simply taking forever to complete, and since it wasn't > calling into libthread, it was hogging that cpu and the other 160 or so > threads in the program simply were not getting *any* time whatsoever. This application is poorly written; here's why. The application you are describing is using threads in order to implement multiple concurrent blocking calls. It's an attempt to increase concurrency of requests, not concurrency of code execution. The current user space threads code in BSD is very, very similar in effect: it makes non-blocking calls, and for those calls which will block, assumes that the OS will schedule the requested work to have completed by the nex time that a non-blocking call is made, so that the call will complete (e.g. an attempt to read data into a user buffer that can't be satisfied results in a fault and a return, and the next read attempt will find the fault satisfied). Writes are satisfied if there is buffer space available. In effect, both these models are a call-conversion model, where you exchange a blocking call for a non-blocking call (or a block on an LWP, and the creation of a new LWP to continue running), plus a thread context switch. Neither of these models improves concurrency on a UP system. On an SMP system, the Solaris model _may_ improce concurrency, _if_ the LWP is not a transient object. If it sticks around, then it would permit another process to be scheduled into user space. > I am starting to believe, based on the number of customers I have talked to > writing threaded code, that most real-world applications that use threads > use threads that sometimes block in the kernel. Threaded applications will block on resource unavailability; it's really irrelevent whether this happens in the kernel or in user space. If the blocking operation is due to a pthreads mutex, you are much better served by blocking in user space in a user space portion of the threads scheduler, so as to both avoid protection domain crossing (by way of a kernel call), and heavy weight kernel locking semantics to implement the mutex (indeed, if you have a program bug, you do not want your program deadlock to escalate to a kernel deadlock simply because you are using the same locking code). > That being the case, I honestly see no real reason not to make them > bound. If this were the only way they were being used, and if using them this way were correct threads programming, I'd agree. But the point of threads is to improve concurrency (virtual, on UP, real on SMP) of user space code, and to provide linear programmers a way to be able to think of state machine operations as if they were linear procedural operations (it turns out that anything you can do with threads can be described as a state machine, and can probably be significantly simplified using Carnot algebra, the same tool used in simplification of digital logic circuits). If the only reason it to permit concurrent blocking, well, then wok is not being done, for the most part, and only in the I/O case is concurrency occurring, and then only if it can make it all the way down to the driver, and your physical hadware support concurrent outstanding operations (e.g. SCSI tagged commands). > But basically, the question of Many to One, One to One, Many to Many, etc. > really comes down to a question of how complex the library will be, and how > threads will typically be used. I maintain, without proof, that the case > of pure user-level threads that don't block at all, that really benefit > from the fast user-level context switching, is not all that frequent in the > real world. I disagree. What if I want CPU affinity in an SMP system, to avoid cache-busting, and to let me use more than a tiny number (say 4) of processors? > Something in my mind likes the idea of only having one set of scheduler > code in a system. Not multiple sets, such as a UTS plus a kernel > scheduler. Simplicity can be a virtue. CPU affinity is most easily (and, I think, elegantly) implemented by having per CPU run queues. This is about the highest level of scheduler complexity that is acceptable in the kernel. Remeber the UNIX philosophy: adding kernel complexity to avoid user space complexity, unnecessarily, is a bad thing. If you could demonstrate that adding the code to the kernel would save a lot of user space processing, on average you might have a point. I maintain, however, that the majority of code written today is not threaded code, and that for all but a few classes of problem, threads are really the wrong tool for the job, and the programmers using them are copping out on having to do hard work. Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200102152336.QAA09980>