Date: Tue, 31 Oct 2017 11:05:13 +0100 From: Mateusz Guzik <mjguzik@gmail.com> To: Matt Joras <mjoras@freebsd.org> Cc: Ian Lepore <ian@freebsd.org>, Hans Petter Selasky <hps@selasky.org>, "freebsd-arch@FreeBSD.org" <freebsd-arch@freebsd.org> Subject: Re: Allow faster eventhandler dispatching by keeping pointers to handler lists. Message-ID: <CAGudoHHrTheYbHxZ_rwvFHGFBC%2B_jsXH4ZGC-iX7%2B9rQdXV9_A@mail.gmail.com> In-Reply-To: <20171031095920.5jn7ib63ofdm7wwf@mguzik> References: <1509243567.56824.103.camel@freebsd.org> <3a71dd31-99cb-c891-9d52-a7f2e7010011@FreeBSD.org> <1509293552.21609.5.camel@freebsd.org> <1509294247.21609.12.camel@freebsd.org> <7b59ff3d-3458-0bca-e6b4-13454b13efb0@FreeBSD.org> <1509296624.21609.24.camel@freebsd.org> <0fad1391-726d-8215-075d-9411abdf6edb@selasky.org> <1509379503.21609.103.camel@freebsd.org> <54876119-9262-c15b-d3fe-1220cf7f17f6@FreeBSD.org> <20171031095920.5jn7ib63ofdm7wwf@mguzik>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, Oct 31, 2017 at 10:59 AM, Mateusz Guzik <mjguzik@gmail.com> wrote: > On Mon, Oct 30, 2017 at 09:38:49AM -0700, Matt Joras wrote: > > On 10/30/2017 09:05, Ian Lepore wrote: > > > On Mon, 2017-10-30 at 09:17 +0100, Hans Petter Selasky wrote: > > >> On 10/29/17 18:03, Ian Lepore wrote: > > >>> Oh. Right. Hmmm, I think malloc() is required to support the case > > >>> where a handler registers before the static list init is invoked, > and I > > >>> do think that's a useful feature to preserve. That means the lists > > >>> aren't really static, though, which then makes STATIC a bit out of > > >>> place in the new function/macro names. > > >> Why not use RCU here, and then update sys/queue.h to be RCU safe? > > >> > > >> --HPS > > > I'm not sure how that suggestion relates to that paragraph, but the > > > code already does use a form of RCU to handle deletion of registered > > > handlers from individual event lists. It could probably benefit from a > > > rewrite to more fully embrace RCU and use it for insertions as well, > > > but that's beyond the scope of these changes, which are trying to > > > eliminate the worst uses of a global lock at a higher level than > > > maintaining the list for a single event. > > > > > > -- Ian > > > > Agreed. There are all sorts of clever things we can do with a more > > comprehensive rewrite. It would be a nice place to try out RCU-ish > > methods of handling object removal without locks. > > The current EH code delightet us long enough. It has to be dropped. > > delighted even > I don't think RCU (which we don't have anyway, apart from a rather poor > substitute) is suitable for use here. > > First off, vast majority of all lists are never going away and most > callbacks are not going away either. > > The gist is that in the worst case we can use rmlocks, which boil down > to per-cpu locks. Employing rcu would require refing and unrefing which > slows things down due to cacheline bounces. > > So here is is roughly how I see it: > 1. always-existing list + never-removed callbacks > > Note that here callbacks can be *added*, but never removed. If > implemented as a list it is really trivial with what hps referred to as > "rcu safe" macros - you set everything up and update the ->next pointer > last. Then the list is always safe to traverse with the worst case being > you finished before the addition of the next element completed. But you > never see half-constructed anything. > > 2. always-existing list + removable callbacks > > They can be stored *separately* to never-removed callbacks. Here is > where rmlocks come to play. An important detail is that the current eh > mechanism allows deregeistration to be performed while *within* a > callback. I think that's a rather weird-ass feature which needs to be > dropped from the replacement. I don't see any added value and I see > *loss* in that things have to get refed/unrefed and stuff relocked. > > 3. lists which can disappear > > Once more rmlocks. > I can't stress enough that writing to shared areas is a major performance problem on multicore systems. Code which does it for no good reason is an excellent candidate for a revamp. I don't see how said writing can be avoided while providing the current EH guarantee that things can be deregistered from the callback. One more reason to create a *new* mechanism from scratch and let the old code die off. -- Mateusz Guzik <mjguzik gmail.com>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAGudoHHrTheYbHxZ_rwvFHGFBC%2B_jsXH4ZGC-iX7%2B9rQdXV9_A>