Date: Thu, 20 Jun 2002 23:28:04 -0700 (PDT) From: Julian Elischer <julian@elischer.org> To: Terry Lambert <tlambert2@mindspring.com> Cc: Gary Thorpe <gat7634@hotmail.com>, freebsd-arch@freebsd.org Subject: Re: multiple threads for interrupts Message-ID: <Pine.BSF.4.21.0206202313570.34612-100000@InterJet.elischer.org> In-Reply-To: <3D1293AE.FDEC441D@mindspring.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, 20 Jun 2002, Terry Lambert wrote: > Gary Thorpe wrote: > > >Seigo Tanimura wrote: > > THat's a general scheduling problem. The solution is well known, > and implemented in Dynix, then IRIX, now Linux. Alfred Perlstein > has some patches that take it most of the way there, but leave an > interlock, by maintaining a global queue. > > The solution I'm talking about is per CPU scheduling queues, where > threads are only migrated between scheduling queues under extraordinary > conditiona, so most of the scheduling never requires the locking the > FreeBSD-current has today. > Per CPU scheduling queues are I think over-rated.. it turns out that you need to have quite a lot of leakage between the queues to get a real priority based system, otherwise you have processor A running a sequence of threads that are lower in priority than threads in Processor B's run queue. To get around this you need to let processor A steal threads from Processor B's queue. Once you have this then it rapidly escalates into a mess. It may be worth while if you have 64 or 32 processors but for less than that it complicates things more than you win.. > This solves the affinity problem. yes, but it introduces synchronisation problems that were not there before.. You still have to lock the other queues while you check that you are not running low prio threads to teh disadvantage of Hi prioones elsewhere, and then since any processor can want to look at any other processor's queue, you get lock-order problems. These can be worked around but it's not a trivial problem, and I'm not implementing them for the forst version of KSEs. > > I'm not sure the affinity fix solves the NETISR problem, because I > think that the issue there is that the affinity you want in that > case is mbuf<->cpu affinity. Basically, if you take the network > interrupt on CPU 3, then you want to run the NETISR code that is > associated with the protocol processing on CPU 3, as well, to avoid > cache busting. > You have both data and instrauction caches to consider.. 2 processors running the net stack means that you are flushing stuff out of the instruction caches of 2 processors. This would degrade other stuff. > The way I would suggest doing this is to run the protocol processing > up to user space at interrupt time (LRP). This gets rid of NETISR. maybe.. > > > What this all boils down to is that you should only permit receive > data interrupts to occur at the rate that you can move the data from > the wire, all the way through the system, to completion. if it were a netgraph problem, I would push the data as far as I could without hitting a blocking lock, at which point it would be handled asynchronsly thereafter.. > > I'm convinced that CPU affinity needs to happen. Somewhat though I'm coming to believe that per cpu queues is not the way to do it.. > > I'm also convinced that, for the most part, running NETISR in > kernel threads, rather than to completion at interrupt, is the > wrong way to go. Mach did it that way. (threads). It's not necessarily right to push all the way up in one go. My personal opinion is that it shoudl go 'as far as it can' and that that may not be all the way up all the time.... > To me, it looks like a lot of people are believing something is > better because they are being told that it is better, not because > they have personally measured it, gotten better numbers, and have > proven to themselves that those better numbers were a result of > what they thought they were measuring, rather than an artifact > that could be exploited in the old code, as well. one could say that about PCPU queues..... To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?Pine.BSF.4.21.0206202313570.34612-100000>