Date: Thu, 15 Feb 2001 02:21:14 -0500 From: Seth Leigh <seth@pengar.com> To: nate@yogotech.com (Nate Williams) Cc: freebsd-smp@FreeBSD.ORG Subject: Re: possible problem with SMP? Message-ID: <5.0.2.1.0.20010215020051.01b9fc70@hobbiton.shire.net> In-Reply-To: <14987.27399.662691.291906@nomad.yogotech.com> References: <5.0.2.1.0.20010214234050.026959f8@hobbiton.shire.net> <5.0.2.1.0.20010214222040.00aac5c0@hobbiton.shire.net> <Pine.GSO.3.96.1010214142231.6504A-100000@p1> <20010214191110.D78224@mollari.cthul.hu> <5.0.2.1.0.20010214234050.026959f8@hobbiton.shire.net>
next in thread | previous in thread | raw e-mail | index | archive | help
There are problems of cpu starvation with a Many to Many model in that usually (unless you manually step up the concurrency) the Many to Many ends up being Many to Number of CPUs. If Number of CPUs is equal to 1, then usually you will only get one LWP beyond the ones that are sleeping. Using the Solaris libpthread, a user-level thread will not give up the LWP it's using until it voluntarily makes a call into libthread which gives libthread an opportunity to deschedule that thread and schedule a different one. To illustrate this, I talked to a customer the other day whose application had around 200 threads. He had around 40 LWPs created, but 39 of them were blocked in the kernel, and 1 appeared to be running when looking at the threads in dbx. The application appeared to be hung, though. Finally he figured out that the one thread that was actually running on the one free LWP was in fact running a hugely inefficient code chunk that was simply taking forever to complete, and since it wasn't calling into libthread, it was hogging that cpu and the other 160 or so threads in the program simply were not getting *any* time whatsoever. The solution to this was either to attempt to set the concurrency manually to something like "a few" LWPs higher than the number he expected to block, or throw in some sched_yield()s into all of his threads to make sure others got some time, or else simply make them bound. I advised him to just make them bound, run some tests to see how this affected his performance, and then decide from there. In all likelihood the majority of his threads can and do block at some point, and I see very little reason if that is the case why he shouldn't just make his threads bound. Daniel Berg and Bil Lewis, in their threads book, seem to agree. I am starting to believe, based on the number of customers I have talked to writing threaded code, that most real-world applications that use threads use threads that sometimes block in the kernel. That being the case, I honestly see no real reason not to make them bound. Fact is, once they block, they cause a new LWP to be created anyways, and then basically if all your threads block at least once in a relatively short amount of time you end up with most of your threads having their "own" LWP anyhow, plus you have the additional overhead of the more complicated thread library. In short, for many if not most real-world applications, user-level threads don't really buy you anything. As for information going from the kernel to the threads library, I read about the callups in that paper at the FreeBSD web site. I really like the idea of using Solaris Doors for that sort of thing. Not only do you get the full benefit of direct callup into the application with the door server, but you have a generalized callup interface that isn't hard-wired to work just a few different ways in the kernel, as the threads-specific callups seem to be planned to do. This could pay off in a big way, as the FreeBSD team found other places where Doors could really improve things. Are the FreeBSD developers who are planning and designing this whole thing aware of what Solaris Doors are, and how the idea of doors might be used for FreeBSD? But basically, the question of Many to One, One to One, Many to Many, etc. really comes down to a question of how complex the library will be, and how threads will typically be used. I maintain, without proof, that the case of pure user-level threads that don't block at all, that really benefit from the fast user-level context switching, is not all that frequent in the real world. Something in my mind likes the idea of only having one set of scheduler code in a system. Not multiple sets, such as a UTS plus a kernel scheduler. Simplicity can be a virtue. I will have to write some more code at work and test it against both /usr/lib/libthread.so.1 and /usr/lib/lwp/libthread.so.1 and see what I can measure in terms of timing. Seth At 10:37 PM 2/14/2001 -0700, Nate Williams wrote: > > I don't see the logic in the argument "letting the kernel scheduler handle > > the scheduling of all the threads (via lwps) is bad, because it overloads > > the kernel scheduler, so we are going to do all of this thread scheduling > > using a user-level thread scheduler". > >That's not the only thing. IMO, the single-biggest gain from a >many-many mapping is that 'userland threads' are much lighter weight >than any kernel thread doing context switching. The kernel has alot >more context it must keep track of, and the fact of the matter is that >in many cases, there is no need for multiple 'kernel' threads of >execution. > >For a good example of this, I'll use Solaris itself. JDK1.1.8 using >Green Threads and the stock JIT can often *blow the doors off* the same >setup using kernel threads in many applications. From talks given at >JavaOne, this is primarily because of the 'context switching' overhead >of using kernel threads. (Sun's JDK uses a one-one mapping, because >Java has no way of intelligently determining which threads should stay >in the userland, vs. which need to own a 'kernel' context.) > >However, it turns out that it's *really easy* to blow yourself up and >cause massive slowdowns if your Java application calls certain kind of >I/O processes (in particular, read directly from stdin, or writing to >files). Because there is no way to determine if a thread is going to do >this kind of thing a-priori, Sun changed their default threading model >in subsuquent versions of the JDK to use kernel threads. In most cases, >the people won't see a measurable difference, but in certain cases they >no longer see big 'pauses' due to syscalls that previously blocked all >running threads. > >However, if performance is a goal, a properly written Java application >can run faster using the 'many-one' model, where 'many threads' are >mapped to a single kernel thread. Unfortunately, because Java doesn't >support the 'many-many' model, you can't really choose the appropriate >context for your thread to run in, and make Java run even better on your >hardware. > >Kernel context switches are simply expensive, and even if they aren't as >expensive, simply moving back-forth into the kernel is expensive because >of the safety precautions that must be taken if you want the system to >be robust. > > > >Nate > > >To Unsubscribe: send mail to majordomo@FreeBSD.org >with "unsubscribe freebsd-smp" in the body of the message To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5.0.2.1.0.20010215020051.01b9fc70>