Date: Wed, 2 Jan 2002 19:02:53 +0000 From: Josef Karthauser <joe@tao.org.uk> To: Bosko Milekic <bmilekic@technokratis.com> Cc: freebsd-arch@freebsd.org, freebsd-smp@freebsd.org Subject: Re: SMPng: Interrupt Latency Issues Message-ID: <20020102190253.B47550@tao.org.uk> In-Reply-To: <20011226021800.A14608@technokratis.com>; from bmilekic@technokratis.com on Wed, Dec 26, 2001 at 02:18:00AM -0500 References: <20011226021800.A14608@technokratis.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--tsOsTdHNUZQcU9Ye Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Dec 26, 2001 at 02:18:00AM -0500, Bosko Milekic wrote: >=20 > Hi, >=20 > It has become obvious, recently in particular, that some important > improvements are required in the way we take interrupts in -CURRENT > SMPng. As previously mentionned, we are experiencing lousy interrupt > latency in -CURRENT. This comes as no surprise. I'm interested in helping with with, although my experience in this area is zero ;(. I was waiting with bated breath for answers to this email and am a bit disappointed that no-one's replied. I hope that doesn't mean that it's not important to anyone. (I was hoping to learn something :) My problem is that my -current box regularly dies, usually when resuming from ACPI, which I now avoid because of it. I'm sure that it's interrupt related. A noticeable characteristic is that a console bell (beep) once started never ends, suggesting that a timer has stopped working, also things like top hang and never refresh. Joe > The purpose of this Email is to get the ball rolling on a discussion > pertaining not only to the approach that we're going to take to properly > remedy the latency issues, but also on the overall ways that we will be > handeling interrupts in SMPng. I know that there are already several > ideas floating around, and briefly talking to Jake and some others over > IRC, I see that there are a couple that are more common than others. Now > I know that this is a sensitive topic but in discussing it, I would > appreciate it if all points are strictly technical and deal with the > implementation, that we not over-generalize (in other words, that we > stay `on topic' [*]), and that after the discussion is done, that we set = up > some milestones and get the work done _soon_ (I'm prepared to take up > some of the work, yes - however, I do not feel that *I* am the right > person to oversee _all_ of the technical aspects of this heavily > burdened task alone). >=20 > [*] This means please don't start bashing anything and everything in our > system and stating useless things like "well, yeah, but our VM system=20 > does not do X, Y, and Z" if it is of no _direct_ relevance to > the way we take interrupts. >=20 > With that out of the way, here's our situation: >=20 > 1. We presently take an interrupt, and in the general case, proceed to > schedule an interrupt thread to run. While placing the thread on the > run queue and because we need to check for the "SWAIT" process flag > we must acquire the sched_lock. This is where the jist of the problem > lies. We bottleneck here because it means that only one CPU can be > scheduling the interrupt thread at a time, and the rest have to spin > waiting for sched_lock. To make matters worse, interrupts are > disabled on the local CPU and we cannot take any other interrupts > either. This is where we stand.=20 >=20 > 2. BSD/OS does things differently. An interrupt comes in and in the > general case, the interrupted thread's VM address space is borrowed > and the handler is immediately executed after interrupts are > re-enabled. Then the handler(s) run and only if it happens to hit a > mutex lock for which it must wait, a clean-up is done to provide a > thread that can block on the mutex for the interrupt, and to allow > the interrupted thread to continue doing its thing. The tradeoff is > that the interrupted thread _cannot_ run anywhere else, not even on > another CPU, while the interrupt that pre-empted it is not finished > executing or has not hit a mutex and could not aquire it quickly. > This may result in some kernel thread priority rules not being 100% > obeyed but I guess that this is part of the tradeoff. >=20 > 3. I believe that some others have alternative suggestions. I encourage > them to present them here clearly, assuming that they are realistic > and implementable approaches, so that we are sure to make the right > decision before we setup milestones. >=20 > In particular, I know that, after briefly speaking to Jake, there is > the idea of per-CPU "interrupt-only run queues" floating around. The > jist of this method would be to keep per-CPU run queues to which each > CPU can schedule their own interrupt threads without having to acquire > any locks (i.e., only have interrupts disabled). The un-balancing of the > queues as well as special priority cases - should they arise - could be > handeled by issuing IPIs. >=20 > I also know that some others (notably Rik van Riel - I hope I spelled > that correctly :-) ) have mentionned realistic ideas that are quite > logical in light of what we presently have and the points above. To me, > following a very brief overview, some of them shared some of the > qualities of the BSD/OS way of doing it so I'd like to invite him (and > others) to lay those methods out for us now, so that we have a greater > picture of various alternatives. >=20 > That's all for now. I hope that we can agree on something worthwhile > soon so that we can establish clear milestones and get this thing done. > It's been long overdue. >=20 > Best regards and Seasons Greetings to you all, > --=20 > Bosko Milekic > bmilekic@FreeBSD.org >=20 >=20 > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-arch" in the body of the message --tsOsTdHNUZQcU9Ye Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (FreeBSD) Comment: For info see http://www.gnupg.org iEYEARECAAYFAjwzWVwACgkQXVIcjOaxUBbqfwCfUiOchBvdRxYpj2+r6jD8bJ5D w4oAoKSs7R/XPJxsxYml+bUV7F1Kh/fJ =MOw6 -----END PGP SIGNATURE----- --tsOsTdHNUZQcU9Ye-- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20020102190253.B47550>