From owner-freebsd-arch@FreeBSD.ORG Sun Sep 17 21:04:35 2006 Return-Path: X-Original-To: freebsd-arch@FreeBSD.org Delivered-To: freebsd-arch@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1FE4616A415; Sun, 17 Sep 2006 21:04:35 +0000 (UTC) (envelope-from jmg@hydrogen.funkthat.com) Received: from hydrogen.funkthat.com (gate.funkthat.com [69.17.45.168]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6252643D70; Sun, 17 Sep 2006 21:04:27 +0000 (GMT) (envelope-from jmg@hydrogen.funkthat.com) Received: from hydrogen.funkthat.com (45tjhvivp3m82nrj@localhost.funkthat.com [127.0.0.1]) by hydrogen.funkthat.com (8.13.6/8.13.3) with ESMTP id k8HL4QBu090411; Sun, 17 Sep 2006 14:04:27 -0700 (PDT) (envelope-from jmg@hydrogen.funkthat.com) Received: (from jmg@localhost) by hydrogen.funkthat.com (8.13.6/8.13.3/Submit) id k8HL4QG3090410; Sun, 17 Sep 2006 14:04:26 -0700 (PDT) (envelope-from jmg) Date: Sun, 17 Sep 2006 14:04:26 -0700 From: John-Mark Gurney To: freebsd-arch@FreeBSD.org Message-ID: <20060917210426.GI9421@funkthat.com> Mail-Followup-To: freebsd-arch@FreeBSD.org, freebsd-current@FreeBSD.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.2.1i X-Operating-System: FreeBSD 5.4-RELEASE-p6 i386 X-PGP-Fingerprint: B7 EC EF F8 AE ED A7 31 96 7A 22 B3 D8 56 36 F4 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html Cc: freebsd-current@FreeBSD.org Subject: kqueue disable on delivery... X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: John-Mark Gurney List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 17 Sep 2006 21:04:35 -0000 I have implemented a couple additional features to kqueue. These allow kqueue to be a multithreaded event delivery system that can guarantee that the event will only be active in one thread at any time. The first is EV_DOD, aka disable on delivery. When the event will be delivered to userland, the knote is marked disabled so we don't have to go through the expense of reallocing the knote each time. (Reallocation of the knote is also lock intensive, and disabling is cheap.) Even though this means that the event will only ever be active in a thread at a time, (when you're done handling the event, you reenable it), removing the event from the queue outside the event handler (say a timeout handler for the connection) poses to be a problem. If you simply close the socket, the event disappears, but then there is a race between another event being created with the same socket, and notification of the handler that you want the event to stop. In order to handle that situation, I have come up w/ EV_FORCEOS, aka FORCE ONE_SHOT. EV_ONESHOT events have the advantage that once queued, they don't care if they have been activated or not, they will be returned the next round. This means that the timeout handler can safely set EV_FORCEOS on the handler, and either if it's _DISABLED (handler running and will reenable it), or it's _ENABLED, it will get dispatched, allowing the handler to detect the EV_FORCEOS flag and teardown the connection. I have expanded a custom web server to make use of these features and so it's been stable. I believe most of the bugs are concurency bugs going from a single threaded kqueue design, to allowing multiple threads access to some of the data structures. The patch is available at: http://people.FreeBSD.org/~jmg/kqueue.dod.patch As part of the patch, I added more strict checking of the flags, so that unknown flags will return an EINVAL. Comments? Suggestions? -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not."