Skip site navigation (1)Skip section navigation (2)
Date:      Wed, 12 Nov 2014 14:14:05 -0800
From:      "K. Macy" <>
To:        J David <>,  "" <>,  "" <>
Subject:   Re: How thread-friendly is kevent?
Message-ID:  <>
In-Reply-To: <>
References:  <> <> <> <>

Next in thread | Previous in thread | Raw E-Mail | Index | Archive | Help
This may not be related, but it's interesting: a client I worked for
many years ago switched from kqueue to poll for their web server
because kqueue did not distribute the workload evenly. I imagine this
is unchanged.


On Wed, Nov 12, 2014 at 12:49 AM, John-Mark Gurney <> wrote:
> J David wrote this message on Mon, Nov 10, 2014 at 22:00 -0500:
>> On Mon, Nov 10, 2014 at 2:13 AM, John-Mark Gurney <> wrote:
>> > you
>> > really need to use one of _DISPATCH or _ONESHOT to ensure that the
>> > event only gets delivered to a single thread....
>> That's what one would expect, which is why the observed behavior was
>> so surprising.  After increasing the testing load considerably, it did
>> behave as expected (waking more than one thread for one event).  But
>> even so, the occurrences were very rare.  It would wake up at most one
>> "extra" thread in slightly less than 1 out of 100,000 events.
> This is odd...  I would expect that the event w/o _ONESHOT and _DISPATCH
> to be delivered many times...  Is it possible you have locks in your
> userland side of things that make this less likely?
>> > Though if you mean how many threads will be woken up in the kernel
>> > and find that there are no events remaining as one of the other kernel
>> > threads has delivered the event, then yes, I have looked at the code,
>> > and there will be a thundering herd problem...
>> Thanks for that, that's exactly the kind of information I was hoping to find.
>> Is that something that can happen without any usermode-visible
>> effects?  I.e. all the threads wake up, but they almost all go back to
>> sleep without leaving the kevent() syscall since they can see there's
>> nothing to do anymore.  If so, that would match the observed behavior,
>> but could add up to a lot of hidden overhead.
> I have an idea that should only be a few lines of changes that would
> prevent all the threads waking up...  As we lock the kq before doing
> the wakeup, we can change KQ_SLEEP from a flag to a count for how many
> threads are sleeping for an event, and if non-zero, do a wakeup_one...
> Then when kqueue_scan is about to exit, check to see if there are
> still events and threads waiting, and then do another wakeup_one...
> Currently, KQ_SLEEP is only a flag, so we have to do wakeup to make
> sure everyone wakes up...
> Well, if you don't have _ONESHOT and _DISPATCH, any changes I make
> should make it more reliable that all threads get the events dispatched
> to them... :)
>> > And if you do, it would make more sense to
>> > use the recent RSS work that Adrian has been working on, and have one
>> > kq per CPU w/ the proper cpu binding for that set of sockets...
>> The most recent information I was able to find:
>> suggests that this work, while admirable and important, is quite some
>> ways away from being production-stable for usermode code:
>> "hopefully I can get my network / rss library up and running enough to
>> prototype an RSS-aware memcached and see if it'll handle this
>> particular workload."
>> It's definitely something to keep an eye on, but probably not a viable
>> approach for us right now.
> True... But some of this is making sure you only run enough threads as
> necessary...  As the kq lock is a single lock, having extra threads that
> don't really do much work only increasing contention and other issues...
> --
>   John-Mark Gurney                              Voice: +1 415 225 5579
>      "All that I will do, has been done, All that I have, has not."
> _______________________________________________
> mailing list
> To unsubscribe, send any mail to ""

Want to link to this message? Use this URL: <>