From owner-freebsd-arch@FreeBSD.ORG Sat Sep 23 18:57:30 2006 Return-Path: X-Original-To: freebsd-arch@FreeBSD.org Delivered-To: freebsd-arch@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6672D16A403; Sat, 23 Sep 2006 18:57:30 +0000 (UTC) (envelope-from jmg@hydrogen.funkthat.com) Received: from hydrogen.funkthat.com (gate.funkthat.com [69.17.45.168]) by mx1.FreeBSD.org (Postfix) with ESMTP id E75C643D53; Sat, 23 Sep 2006 18:57:29 +0000 (GMT) (envelope-from jmg@hydrogen.funkthat.com) Received: from hydrogen.funkthat.com (8racgo61w6f7sbmh@localhost.funkthat.com [127.0.0.1]) by hydrogen.funkthat.com (8.13.6/8.13.3) with ESMTP id k8NIvTBU045693; Sat, 23 Sep 2006 11:57:29 -0700 (PDT) (envelope-from jmg@hydrogen.funkthat.com) Received: (from jmg@localhost) by hydrogen.funkthat.com (8.13.6/8.13.3/Submit) id k8NIvRlf045692; Sat, 23 Sep 2006 11:57:27 -0700 (PDT) (envelope-from jmg) Date: Sat, 23 Sep 2006 11:57:27 -0700 From: John-Mark Gurney To: Igor Sysoev Message-ID: <20060923185727.GW23915@funkthat.com> Mail-Followup-To: Igor Sysoev , freebsd-current@FreeBSD.org, freebsd-arch@FreeBSD.org References: <20060917210426.GI9421@funkthat.com> <20060922171542.G17859@is.park.rambler.ru> <20060922165848.GS23915@funkthat.com> <20060923105426.B20782@is.park.rambler.ru> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20060923105426.B20782@is.park.rambler.ru> User-Agent: Mutt/1.4.2.1i X-Operating-System: FreeBSD 5.4-RELEASE-p6 i386 X-PGP-Fingerprint: B7 EC EF F8 AE ED A7 31 96 7A 22 B3 D8 56 36 F4 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html Cc: freebsd-current@FreeBSD.org, freebsd-arch@FreeBSD.org Subject: Re: kqueue disable on delivery... X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: John-Mark Gurney List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Sep 2006 18:57:30 -0000 Igor Sysoev wrote this message on Sat, Sep 23, 2006 at 11:40 +0400: > On Fri, 22 Sep 2006, John-Mark Gurney wrote: > > >Igor Sysoev wrote this message on Fri, Sep 22, 2006 at 17:25 +0400: > >>On Sun, 17 Sep 2006, John-Mark Gurney wrote: > >> > >>>I have implemented a couple additional features to kqueue. These allow > >>>kqueue to be a multithreaded event delivery system that can guarantee > >>>that the event will only be active in one thread at any time. > >>> > >>>The first is EV_DOD, aka disable on delivery. When the event will be > >>>delivered to userland, the knote is marked disabled so we don't > >>>have to go through the expense of reallocing the knote each time. > >>>(Reallocation of the knote is also lock intensive, and disabling is > >>>cheap.) > >> > >>In my opinion, it's too implementation specific flag. > > > >How else are you doing to solve having multiple threads servicing > >the same queue at the same time? Also, Apple is planing on having > >a similar flag to EV_DOD, but I don't know what they are naming it.. > >I've tried for a while to find out, but haven't been able to... > > As I understand EV_DOD or EV_CLEAR|EV_DOD are like simple EV_ONESHOT, > except the filter is not deleted on delivery, but is disabled skipping > some in-kernel lock overhead. That's I'd named it too implementation > specific. > > Yes, the EV_CLEAR|EV_DOD guarantees that the event will be active > in one thread only at any time. But in my practice I saw there is > necessity to guarantee that the socket (both events - EVFILT_READ > and EVFILT_WRITE) will be active in one thread only at any time. > It seems that is the reason why heavy threaded Solaris 10 event ports > use the oneshot only model where a socket is deleted from port on delivery. Only if you need to both read and write active on the socket at once... In some/many servers, you only need one or the other, such as file transfer servers like http and ftp... > >>>Even though this means that the event will only ever be active in a > >>>thread at a time, (when you're done handling the event, you reenable > >>>it), removing the event from the queue outside the event handler (say > >>>a timeout handler for the connection) poses to be a problem. If you > >>>simply close the socket, the event disappears, but then there is a > >>>race between another event being created with the same socket, and > >>>notification of the handler that you want the event to stop. > >>> > >>>In order to handle that situation, I have come up w/ EV_FORCEOS, aka > >>>FORCE ONE_SHOT. EV_ONESHOT events have the advantage that once queued, > >>>they don't care if they have been activated or not, they will be returned > >>>the next round. This means that the timeout handler can safely set > >>>EV_FORCEOS on the handler, and either if it's _DISABLED (handler running > >>>and will reenable it), or it's _ENABLED, it will get dispatched, allowing > >>>the handler to detect the EV_FORCEOS flag and teardown the connection. > >> > >>I think it should be EVFILT_USER event, allowing to > >>EV_SET(&kev, fd, EVFILT_USER, 0, 0, 0, udata); > >>and the event should automatically sets the EV_ONESHOT flag internally. > > > >I'll agree EV_FORCEOS is open for discussion, but you did see how much > >code it adds right? I was surprised at how small the patch was for the > >additional functionality.. > > Yes, EV_FORCEOS is small patch. However, EVFILT_USER is more generic > (by the way, Solaris 10 event ports allow to send user-specific > PORT_SOURCE_USER notification). I agree EVFILT_USER would be a useful thing, but it is still different from EV_FORCEOS... Would you like to contribute some the to EVFILT_USER? I'll look at integrating it... > Two years ago I was implementing threads for my server nginx > on FreeBSD 4.x, using rfork(). In the absence of EVFILT_USER I made > the condition variables using kill() and EV_SIGNAL and this user-level > code may panic kernel. Does it still? > >What happens if you are in the process of tearing down udata when > >this happens, but you haven't gotten far enough to drop it? Then > >you'd have to deal w/ possible lock inversions between the timeout > >list and your object lock, deal w/ flags on the object and ref counts.. > > > >With _DOD and _FORCEOS, you are able to continue to not require special > >state flags, locks nor reference counting on your objects serviced by > >kqueue... > > > >I wrote this code in anticipation of supporting sun4v boxes where it'd > >be useful to have 32 threads (or more) servicing a single kqueue... > > You still need user locks to guarantee that the socket will be active > in one thread only at any time. In proxy mode you still need locks > to guarantee that two sockets will be active in one thread only. > If you assemble your response from several proxied servers, then > you need locks to guarantee that all these sockets will be active > in one thread only. As we have just found out, our target servers have different designs... Since mine is a simple http server, I will only ever read or write at a time.. I use accept filters so that I don't have to do an EVFILT_READ on the socket, and if I don't get a complete HTTP/1.x request, I reject it.. So my server only ever sets EVFILT_WRITE for the sockets... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not."