From owner-freebsd-questions@FreeBSD.ORG  Sun Nov  9 19:57:06 2014
Return-Path: <owner-freebsd-questions@FreeBSD.ORG>
Delivered-To: freebsd-questions@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by hub.freebsd.org (Postfix) with ESMTPS id A7E233D1;
 Sun,  9 Nov 2014 19:57:06 +0000 (UTC)
Received: from mail-ig0-x230.google.com (mail-ig0-x230.google.com
 [IPv6:2607:f8b0:4001:c05::230])
 (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits))
 (Client CN "smtp.gmail.com",
 Issuer "Google Internet Authority G2" (verified OK))
 by mx1.freebsd.org (Postfix) with ESMTPS id 716ED3F34;
 Sun,  9 Nov 2014 19:57:06 +0000 (UTC)
Received: by mail-ig0-f176.google.com with SMTP id l13so16805303iga.3
 for <multiple recipients>; Sun, 09 Nov 2014 11:57:06 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
 h=mime-version:sender:date:message-id:subject:from:to:content-type;
 bh=T5VAjA8s/fq4M7y2Uv3hHsT/Hqzdx17P+V2SC7QYM9E=;
 b=A1W4DK5Lclzih0gDmDF18dbUgHVt6GhJSnTgN5rv3bt2z+EPxEtKweqA7lHET/ouFv
 p4fDPOHd9GRhhu9ynbR/8dfTa+dOsVWAg9IdksHNK38kBpp6GvAfNNFigBDy8j2Ez+HL
 9CeYnLuMjL/cxgUExSCYyFly5hE+OSIU/Qm0Ij7QXIKVg+R/mZRSKd7q2zWM1HRjTEDn
 sNogOeW7ruZCfiREUaaJwQpFmXuuFaT0IIkFV3/dP01NkLorxB4XjhNOVfR+//c3L4oG
 Z8XXBNqgivDXNkolljkP6TRcZ6e9SkDl3skYNIXpas4M81Fpk/ATWkmQ2z/ZZBWgftG+
 nZuA==
MIME-Version: 1.0
X-Received: by 10.50.79.135 with SMTP id j7mr20085270igx.14.1415563025937;
 Sun, 09 Nov 2014 11:57:05 -0800 (PST)
Sender: jdavidlists@gmail.com
Received: by 10.43.96.202 with HTTP; Sun, 9 Nov 2014 11:57:05 -0800 (PST)
Date: Sun, 9 Nov 2014 14:57:05 -0500
X-Google-Sender-Auth: jI5Ynmsqk4-ZcORMCfJAqM33LuM
Message-ID: <CABXB=RQWxu-d30raZ+FcrnrGsr5gG2Za_=cx8-jCnLSgJDSF=Q@mail.gmail.com>
Subject: How thread-friendly is kevent?
From: J David <j.david.lists@gmail.com>
To: freebsd-hackers@freebsd.org, 
 "freebsd-questions@freebsd.org" <freebsd-questions@freebsd.org>
Content-Type: text/plain; charset=UTF-8
X-BeenThere: freebsd-questions@freebsd.org
X-Mailman-Version: 2.1.18-1
Precedence: list
List-Id: User questions <freebsd-questions.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-questions>, 
 <mailto:freebsd-questions-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-questions/>
List-Post: <mailto:freebsd-questions@freebsd.org>
List-Help: <mailto:freebsd-questions-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-questions>, 
 <mailto:freebsd-questions-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Sun, 09 Nov 2014 19:57:06 -0000

How thread-friendly is kevent()?

In most cases, a dedicated thread does the kevent()-ing and
dispatching work to the thread pool, but at extremely high rates that
thread becomes a significant bottleneck.

As an example, consider a pool of, say, 17 threads on a 16 core
machine all in kevent() waiting for one of 32000 open TCP connections
to be read-ready.  One connection becomes read-ready.  How many
threads will have kevent() return with that event in eventlist?  Is
there potential for a thundering herd problem?

Limited small-scale experimentation suggests that only one thread
returns per event, but it's not documented that way, so it's not clear
if that behavior is intended, an implementation detail, or a
coincidence that won't hold up at scale.

Is this behavior at all guaranteed / by design / intentional?

Although it would be ideal if so, it would also make sense to have to
rely on EV_DISPATCH in multithreaded applications to prevent events
from being delivered more than once, or to use EV_ONESHOT and re-add
the event entirely, depending on which approach better suits the
internal data structure the kernel uses for kqueue.

Thanks for any advice!