Date: Wed, 12 Nov 2014 14:14:05 -0800 From: "K. Macy" <kmacy@freebsd.org> To: J David <j.david.lists@gmail.com>, "freebsd-hackers@freebsd.org" <freebsd-hackers@freebsd.org>, "freebsd-questions@freebsd.org" <freebsd-questions@freebsd.org> Subject: Re: How thread-friendly is kevent? Message-ID: <CAHM0Q_MJjH=Cq0AKTn64RkzgwPBMoKnDVu-GvcX6qBcFYDMpOg@mail.gmail.com> In-Reply-To: <20141112084909.GV24601@funkthat.com> References: <CABXB=RQWxu-d30raZ%2BFcrnrGsr5gG2Za_=cx8-jCnLSgJDSF=Q@mail.gmail.com> <20141110071353.GO24601@funkthat.com> <CABXB=RStLz6J9L3--KsM308-0h0N5ZeZZvw1GbDi%2BZvKO4U64g@mail.gmail.com> <20141112084909.GV24601@funkthat.com>
next in thread | previous in thread | raw e-mail | index | archive | help
This may not be related, but it's interesting: a client I worked for many years ago switched from kqueue to poll for their web server because kqueue did not distribute the workload evenly. I imagine this is unchanged. -K On Wed, Nov 12, 2014 at 12:49 AM, John-Mark Gurney <jmg@funkthat.com> wrote: > J David wrote this message on Mon, Nov 10, 2014 at 22:00 -0500: >> On Mon, Nov 10, 2014 at 2:13 AM, John-Mark Gurney <jmg@funkthat.com> wrote: >> > you >> > really need to use one of _DISPATCH or _ONESHOT to ensure that the >> > event only gets delivered to a single thread.... >> >> That's what one would expect, which is why the observed behavior was >> so surprising. After increasing the testing load considerably, it did >> behave as expected (waking more than one thread for one event). But >> even so, the occurrences were very rare. It would wake up at most one >> "extra" thread in slightly less than 1 out of 100,000 events. > > This is odd... I would expect that the event w/o _ONESHOT and _DISPATCH > to be delivered many times... Is it possible you have locks in your > userland side of things that make this less likely? > >> > Though if you mean how many threads will be woken up in the kernel >> > and find that there are no events remaining as one of the other kernel >> > threads has delivered the event, then yes, I have looked at the code, >> > and there will be a thundering herd problem... >> >> Thanks for that, that's exactly the kind of information I was hoping to find. >> >> Is that something that can happen without any usermode-visible >> effects? I.e. all the threads wake up, but they almost all go back to >> sleep without leaving the kevent() syscall since they can see there's >> nothing to do anymore. If so, that would match the observed behavior, >> but could add up to a lot of hidden overhead. > > I have an idea that should only be a few lines of changes that would > prevent all the threads waking up... As we lock the kq before doing > the wakeup, we can change KQ_SLEEP from a flag to a count for how many > threads are sleeping for an event, and if non-zero, do a wakeup_one... > Then when kqueue_scan is about to exit, check to see if there are > still events and threads waiting, and then do another wakeup_one... > > Currently, KQ_SLEEP is only a flag, so we have to do wakeup to make > sure everyone wakes up... > > Well, if you don't have _ONESHOT and _DISPATCH, any changes I make > should make it more reliable that all threads get the events dispatched > to them... :) > >> > And if you do, it would make more sense to >> > use the recent RSS work that Adrian has been working on, and have one >> > kq per CPU w/ the proper cpu binding for that set of sockets... >> >> The most recent information I was able to find: >> >> http://adrianchadd.blogspot.com/2014/10/more-rss-udp-tests-this-time-on-dell.html >> >> suggests that this work, while admirable and important, is quite some >> ways away from being production-stable for usermode code: >> >> "hopefully I can get my network / rss library up and running enough to >> prototype an RSS-aware memcached and see if it'll handle this >> particular workload." >> >> It's definitely something to keep an eye on, but probably not a viable >> approach for us right now. > > True... But some of this is making sure you only run enough threads as > necessary... As the kq lock is a single lock, having extra threads that > don't really do much work only increasing contention and other issues... > > -- > John-Mark Gurney Voice: +1 415 225 5579 > > "All that I will do, has been done, All that I have, has not." > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAHM0Q_MJjH=Cq0AKTn64RkzgwPBMoKnDVu-GvcX6qBcFYDMpOg>