From owner-freebsd-questions@FreeBSD.ORG Tue Nov 11 05:12:08 2014 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6EE3A57E; Tue, 11 Nov 2014 05:12:08 +0000 (UTC) Received: from mail-wi0-x235.google.com (mail-wi0-x235.google.com [IPv6:2a00:1450:400c:c05::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id F224B1AC; Tue, 11 Nov 2014 05:12:07 +0000 (UTC) Received: by mail-wi0-f181.google.com with SMTP id n3so486002wiv.8 for ; Mon, 10 Nov 2014 21:12:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=LU+XrniIRDh6BvRFj+Mi3A90wLvOXb/UWKc/QB1HsV4=; b=0wsPjSnxNyEeQK39XPJVovqI9XMiVo3edPK8SQo0O+seUZMoVpjOVyz153M0o7izkj ujUlakueAVcZ3ewxQ/ggj3wPf22x9xSEMos4OcpkaV53GVwdK/iZ56ipkKA6+gWDxjlw 4aDNhTLbgPQPcyNzWCdsgRHh6Q+Gc8uzKA3jzpAQ9GJpahtWiWoOMgRA1FoshVbvOBLK k+jFYTYl1hZ/zGvQ/ZxjXdPY5hloOFkrMYcECMrb7Lr63wWAplnTdrVB1+JYSd/SGmxZ Hu7AfDX2RLMUVSWGVLZJoy1Oovd2nfRmETH0YMiezh6SvI4mjhy4UCKxGWjOrDREVhuG SECA== MIME-Version: 1.0 X-Received: by 10.180.99.105 with SMTP id ep9mr37244112wib.26.1415682725850; Mon, 10 Nov 2014 21:12:05 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.216.106.136 with HTTP; Mon, 10 Nov 2014 21:12:05 -0800 (PST) In-Reply-To: References: <20141110071353.GO24601@funkthat.com> Date: Mon, 10 Nov 2014 21:12:05 -0800 X-Google-Sender-Auth: v8tp0WI_6zoUkiY9Z-wXU6icZEI Message-ID: Subject: Re: How thread-friendly is kevent? From: Adrian Chadd To: J David Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-hackers@freebsd.org" , "freebsd-questions@freebsd.org" X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Nov 2014 05:12:08 -0000 On 10 November 2014 19:00, J David wrote: > On Mon, Nov 10, 2014 at 2:13 AM, John-Mark Gurney wrote: >> you >> really need to use one of _DISPATCH or _ONESHOT to ensure that the >> event only gets delivered to a single thread.... > > That's what one would expect, which is why the observed behavior was > so surprising. After increasing the testing load considerably, it did > behave as expected (waking more than one thread for one event). But > even so, the occurrences were very rare. It would wake up at most one > "extra" thread in slightly less than 1 out of 100,000 events. > >> Though if you mean how many threads will be woken up in the kernel >> and find that there are no events remaining as one of the other kernel >> threads has delivered the event, then yes, I have looked at the code, >> and there will be a thundering herd problem... > > Thanks for that, that's exactly the kind of information I was hoping to find. > > Is that something that can happen without any usermode-visible > effects? I.e. all the threads wake up, but they almost all go back to > sleep without leaving the kevent() syscall since they can see there's > nothing to do anymore. If so, that would match the observed behavior, > but could add up to a lot of hidden overhead. > >> And if you do, it would make more sense to >> use the recent RSS work that Adrian has been working on, and have one >> kq per CPU w/ the proper cpu binding for that set of sockets... > > The most recent information I was able to find: > > http://adrianchadd.blogspot.com/2014/10/more-rss-udp-tests-this-time-on-dell.html > > suggests that this work, while admirable and important, is quite some > ways away from being production-stable for usermode code: > > "hopefully I can get my network / rss library up and running enough to > prototype an RSS-aware memcached and see if it'll handle this > particular workload." > > It's definitely something to keep an eye on, but probably not a viable > approach for us right now. Why? It's likely going to be heading in the direction that you'd want it to for scaling networking workloads and I could do with more users / feedback on it. (Ie, you should totally use it, because this is the direction things are likely going to head in the future, and you right now get to help shape its direction. :) -adrian