Date: Wed, 26 Dec 2001 02:18:00 -0500 From: Bosko Milekic <bmilekic@technokratis.com> To: freebsd-arch@freebsd.org, freebsd-smp@freebsd.org Subject: SMPng: Interrupt Latency Issues Message-ID: <20011226021800.A14608@technokratis.com>
next in thread | raw e-mail | index | archive | help
Hi, It has become obvious, recently in particular, that some important improvements are required in the way we take interrupts in -CURRENT SMPng. As previously mentionned, we are experiencing lousy interrupt latency in -CURRENT. This comes as no surprise. The purpose of this Email is to get the ball rolling on a discussion pertaining not only to the approach that we're going to take to properly remedy the latency issues, but also on the overall ways that we will be handeling interrupts in SMPng. I know that there are already several ideas floating around, and briefly talking to Jake and some others over IRC, I see that there are a couple that are more common than others. Now I know that this is a sensitive topic but in discussing it, I would appreciate it if all points are strictly technical and deal with the implementation, that we not over-generalize (in other words, that we stay `on topic' [*]), and that after the discussion is done, that we set up some milestones and get the work done _soon_ (I'm prepared to take up some of the work, yes - however, I do not feel that *I* am the right person to oversee _all_ of the technical aspects of this heavily burdened task alone). [*] This means please don't start bashing anything and everything in our system and stating useless things like "well, yeah, but our VM system does not do X, Y, and Z" if it is of no _direct_ relevance to the way we take interrupts. With that out of the way, here's our situation: 1. We presently take an interrupt, and in the general case, proceed to schedule an interrupt thread to run. While placing the thread on the run queue and because we need to check for the "SWAIT" process flag we must acquire the sched_lock. This is where the jist of the problem lies. We bottleneck here because it means that only one CPU can be scheduling the interrupt thread at a time, and the rest have to spin waiting for sched_lock. To make matters worse, interrupts are disabled on the local CPU and we cannot take any other interrupts either. This is where we stand. 2. BSD/OS does things differently. An interrupt comes in and in the general case, the interrupted thread's VM address space is borrowed and the handler is immediately executed after interrupts are re-enabled. Then the handler(s) run and only if it happens to hit a mutex lock for which it must wait, a clean-up is done to provide a thread that can block on the mutex for the interrupt, and to allow the interrupted thread to continue doing its thing. The tradeoff is that the interrupted thread _cannot_ run anywhere else, not even on another CPU, while the interrupt that pre-empted it is not finished executing or has not hit a mutex and could not aquire it quickly. This may result in some kernel thread priority rules not being 100% obeyed but I guess that this is part of the tradeoff. 3. I believe that some others have alternative suggestions. I encourage them to present them here clearly, assuming that they are realistic and implementable approaches, so that we are sure to make the right decision before we setup milestones. In particular, I know that, after briefly speaking to Jake, there is the idea of per-CPU "interrupt-only run queues" floating around. The jist of this method would be to keep per-CPU run queues to which each CPU can schedule their own interrupt threads without having to acquire any locks (i.e., only have interrupts disabled). The un-balancing of the queues as well as special priority cases - should they arise - could be handeled by issuing IPIs. I also know that some others (notably Rik van Riel - I hope I spelled that correctly :-) ) have mentionned realistic ideas that are quite logical in light of what we presently have and the points above. To me, following a very brief overview, some of them shared some of the qualities of the BSD/OS way of doing it so I'd like to invite him (and others) to lay those methods out for us now, so that we have a greater picture of various alternatives. That's all for now. I hope that we can agree on something worthwhile soon so that we can establish clear milestones and get this thing done. It's been long overdue. Best regards and Seasons Greetings to you all, -- Bosko Milekic bmilekic@FreeBSD.org To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20011226021800.A14608>