From owner-freebsd-questions@FreeBSD.ORG Wed Nov 12 22:24:03 2014 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 33971576; Wed, 12 Nov 2014 22:24:03 +0000 (UTC) Received: from mail-ie0-x22e.google.com (mail-ie0-x22e.google.com [IPv6:2607:f8b0:4001:c03::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EC2D01B3; Wed, 12 Nov 2014 22:24:02 +0000 (UTC) Received: by mail-ie0-f174.google.com with SMTP id x19so14650967ier.19 for ; Wed, 12 Nov 2014 14:24:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:content-type; bh=1nSLXNyihbjkx2Vx5ikZIVjfnDI2BVH3kybRAktw2Ec=; b=u1YJXiR83k9/5ST8rBm29vDOkWpI1wngL5qmW76zdE4JlPxZ/SwsrGe8czeGBuhjrP 2GRV67AfwOnaR03wmulCffhTqQjwBkN4sNlLvqzo5NQvgjxHN97FgxKuSgngQSHJC2Py G3ZGXg9dSgy8FtV/SZnBd3R+29bKA77gCnLTrK5AELT7MGCXR5vYQHAhdgg/QwcHJb7j iZqB/wEn66/IC3kgiz2JyjCrahuMmrVVQs21GFfoDz2s7e3nEDHxiLBOHpEVonvHR+rC K+ZPNAqW0jTlznZ69GFJZ+Htwf31LgHlngEM7fNgPvMTx3PSJUUhgovnIqEk2cNeHiG3 RizQ== MIME-Version: 1.0 X-Received: by 10.50.66.227 with SMTP id i3mr43418093igt.25.1415831042507; Wed, 12 Nov 2014 14:24:02 -0800 (PST) Sender: jdavidlists@gmail.com Received: by 10.43.96.202 with HTTP; Wed, 12 Nov 2014 14:24:02 -0800 (PST) In-Reply-To: <20141112084909.GV24601@funkthat.com> References: <20141110071353.GO24601@funkthat.com> <20141112084909.GV24601@funkthat.com> Date: Wed, 12 Nov 2014 17:24:02 -0500 X-Google-Sender-Auth: 2khuzCpSBX4KFNUMXIjW6tdAguU Message-ID: Subject: Re: How thread-friendly is kevent? From: J David To: J David , "freebsd-hackers@freebsd.org" , "freebsd-questions@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Nov 2014 22:24:03 -0000 On Wed, Nov 12, 2014 at 3:49 AM, John-Mark Gurney wrote: > This is odd... I would expect that the event w/o _ONESHOT and _DISPATCH > to be delivered many times... Is it possible you have locks in your > userland side of things that make this less likely? Nope, the test code is (intentionally) entirely lock-free in userland. > I have an idea that should only be a few lines of changes that would > prevent all the threads waking up... As we lock the kq before doing > the wakeup, we can change KQ_SLEEP from a flag to a count for how many > threads are sleeping for an event, and if non-zero, do a wakeup_one... > Then when kqueue_scan is about to exit, check to see if there are > still events and threads waiting, and then do another wakeup_one... This sounds like it could optimize some workloads at substantial penalties for others. If pursued, maybe it needs its own flag. > Currently, KQ_SLEEP is only a flag, so we have to do wakeup to make > sure everyone wakes up... > > Well, if you don't have _ONESHOT and _DISPATCH, any changes I make > should make it more reliable that all threads get the events dispatched > to them... :) Using _DISPATCH is no problem, although a solution that didn't require two kevent()-calls per event would obviously be better when every syscall matters. Albeit that is largely an issue on VM's where the syscall penalty is artificially large. In production, this will of course run on bare metal. The other option is do wrap kevent() with a mutex on the user side. That's what Apache does with accept(), IIRC. > But some of this is making sure you only run enough threads as > necessary... That's almost always true. But, almost always, determining the correct value of "enough" requires a blood sacrifice. :) Thanks!