From owner-freebsd-hackers@FreeBSD.ORG Mon Nov 10 07:13:55 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9C3F3479; Mon, 10 Nov 2014 07:13:55 +0000 (UTC) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "funkthat.com", Issuer "funkthat.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 77DA9793; Mon, 10 Nov 2014 07:13:55 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id sAA7DrvJ094454 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sun, 9 Nov 2014 23:13:54 -0800 (PST) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id sAA7DrUd094453; Sun, 9 Nov 2014 23:13:53 -0800 (PST) (envelope-from jmg) Date: Sun, 9 Nov 2014 23:13:53 -0800 From: John-Mark Gurney To: J David Subject: Re: How thread-friendly is kevent? Message-ID: <20141110071353.GO24601@funkthat.com> Mail-Followup-To: J David , freebsd-hackers@freebsd.org, "freebsd-questions@freebsd.org" References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Sun, 09 Nov 2014 23:13:54 -0800 (PST) Cc: freebsd-hackers@freebsd.org, "freebsd-questions@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Nov 2014 07:13:55 -0000 J David wrote this message on Sun, Nov 09, 2014 at 14:57 -0500: > How thread-friendly is kevent()? > > In most cases, a dedicated thread does the kevent()-ing and > dispatching work to the thread pool, but at extremely high rates that > thread becomes a significant bottleneck. > > As an example, consider a pool of, say, 17 threads on a 16 core > machine all in kevent() waiting for one of 32000 open TCP connections > to be read-ready. One connection becomes read-ready. How many > threads will have kevent() return with that event in eventlist? Is > there potential for a thundering herd problem? It depends... kevent is a level triggered system... By default, if you don't use _ONESHOT or _DISPATCH, all threads will be woken up.. So, yes, there is potential for a thundering herd problem... > Limited small-scale experimentation suggests that only one thread > returns per event, but it's not documented that way, so it's not clear > if that behavior is intended, an implementation detail, or a > coincidence that won't hold up at scale. > > Is this behavior at all guaranteed / by design / intentional? > > Although it would be ideal if so, it would also make sense to have to > rely on EV_DISPATCH in multithreaded applications to prevent events > from being delivered more than once, or to use EV_ONESHOT and re-add > the event entirely, depending on which approach better suits the > internal data structure the kernel uses for kqueue. If you do have a single kq w/ multiple threads (you cannot have a shared kq between processes, as the fork clears out all events), you really need to use one of _DISPATCH or _ONESHOT to ensure that the event only gets delivered to a single thread.... Though if you mean how many threads will be woken up in the kernel and find that there are no events remaining as one of the other kernel threads has delivered the event, then yes, I have looked at the code, and there will be a thundering herd problem... This could be mitigated with out too much work, though the question I have is why do you have so many threads listening on the same kqueue? The most common use of this is for socket IO (there isn't much else except maybe vnodes) that you can wait on that you'd have such a highly multithreaded program... And if you do, it would make more sense to use the recent RSS work that Adrian has been working on, and have one kq per CPU w/ the proper cpu binding for that set of sockets... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not."