Date: Wed, 12 Nov 2014 00:49:09 -0800 From: John-Mark Gurney <jmg@funkthat.com> To: J David <j.david.lists@gmail.com> Cc: "freebsd-hackers@freebsd.org" <freebsd-hackers@freebsd.org>, "freebsd-questions@freebsd.org" <freebsd-questions@freebsd.org> Subject: Re: How thread-friendly is kevent? Message-ID: <20141112084909.GV24601@funkthat.com> In-Reply-To: <CABXB=RStLz6J9L3--KsM308-0h0N5ZeZZvw1GbDi%2BZvKO4U64g@mail.gmail.com> References: <CABXB=RQWxu-d30raZ%2BFcrnrGsr5gG2Za_=cx8-jCnLSgJDSF=Q@mail.gmail.com> <20141110071353.GO24601@funkthat.com> <CABXB=RStLz6J9L3--KsM308-0h0N5ZeZZvw1GbDi%2BZvKO4U64g@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
J David wrote this message on Mon, Nov 10, 2014 at 22:00 -0500: > On Mon, Nov 10, 2014 at 2:13 AM, John-Mark Gurney <jmg@funkthat.com> wrote: > > you > > really need to use one of _DISPATCH or _ONESHOT to ensure that the > > event only gets delivered to a single thread.... > > That's what one would expect, which is why the observed behavior was > so surprising. After increasing the testing load considerably, it did > behave as expected (waking more than one thread for one event). But > even so, the occurrences were very rare. It would wake up at most one > "extra" thread in slightly less than 1 out of 100,000 events. This is odd... I would expect that the event w/o _ONESHOT and _DISPATCH to be delivered many times... Is it possible you have locks in your userland side of things that make this less likely? > > Though if you mean how many threads will be woken up in the kernel > > and find that there are no events remaining as one of the other kernel > > threads has delivered the event, then yes, I have looked at the code, > > and there will be a thundering herd problem... > > Thanks for that, that's exactly the kind of information I was hoping to find. > > Is that something that can happen without any usermode-visible > effects? I.e. all the threads wake up, but they almost all go back to > sleep without leaving the kevent() syscall since they can see there's > nothing to do anymore. If so, that would match the observed behavior, > but could add up to a lot of hidden overhead. I have an idea that should only be a few lines of changes that would prevent all the threads waking up... As we lock the kq before doing the wakeup, we can change KQ_SLEEP from a flag to a count for how many threads are sleeping for an event, and if non-zero, do a wakeup_one... Then when kqueue_scan is about to exit, check to see if there are still events and threads waiting, and then do another wakeup_one... Currently, KQ_SLEEP is only a flag, so we have to do wakeup to make sure everyone wakes up... Well, if you don't have _ONESHOT and _DISPATCH, any changes I make should make it more reliable that all threads get the events dispatched to them... :) > > And if you do, it would make more sense to > > use the recent RSS work that Adrian has been working on, and have one > > kq per CPU w/ the proper cpu binding for that set of sockets... > > The most recent information I was able to find: > > http://adrianchadd.blogspot.com/2014/10/more-rss-udp-tests-this-time-on-dell.html > > suggests that this work, while admirable and important, is quite some > ways away from being production-stable for usermode code: > > "hopefully I can get my network / rss library up and running enough to > prototype an RSS-aware memcached and see if it'll handle this > particular workload." > > It's definitely something to keep an eye on, but probably not a viable > approach for us right now. True... But some of this is making sure you only run enough threads as necessary... As the kq lock is a single lock, having extra threads that don't really do much work only increasing contention and other issues... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not."
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20141112084909.GV24601>