From owner-freebsd-hackers Wed Feb 13 3:31:33 2002 Delivered-To: freebsd-hackers@freebsd.org Received: from mail.nsu.ru (mx.nsu.ru [193.124.215.71]) by hub.freebsd.org (Postfix) with ESMTP id 4723F37B402 for ; Wed, 13 Feb 2002 03:31:24 -0800 (PST) Received: from regency.nsu.ru ([193.124.210.26] helo=cytherea.weblab.nsu.ru) by mail.nsu.ru with esmtp (Exim 3.20 #1) id 16axcI-0001rb-00; Wed, 13 Feb 2002 17:30:26 +0600 Received: (from danfe@localhost) by cytherea.weblab.nsu.ru (8.11.6/8.11.6) id g1DBUBG26563; Wed, 13 Feb 2002 17:30:11 +0600 (NOVT) (envelope-from danfe) Date: Wed, 13 Feb 2002 17:30:10 +0600 From: Alexey Dokuchaev To: "Eugene L. Vorokov" Cc: freebsd-hackers@freebsd.org, thttpd@acme.com Subject: Re: THTTPD web server: problems with KQUEUE on FreeBSD 4.5-STABLE Message-ID: <20020213173010.E14414@cytherea.weblab.nsu.ru> References: <20020213160014.A97359@cytherea.weblab.nsu.ru> <200202131028.g1DASTY88632@bugz.infotecs.ru> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5.1i In-Reply-To: <200202131028.g1DASTY88632@bugz.infotecs.ru>; from vel@bugz.infotecs.ru on Wed, Feb 13, 2002 at 01:28:29PM +0300 Sender: owner-freebsd-hackers@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Wed, Feb 13, 2002 at 01:28:29PM +0300, Eugene L. Vorokov wrote: > > I still wonder, whether this problem occurs because of how thttpd does > > things, or how FreeBSD implements kqueue stuff, however, I am not sure > > that I will have enough time to dig deep into this. Right now I'm pretty > > happy with the fact that I got thttpd working again, and those "200 0" > > messages are no longer in my logs. However, it is still an issue to > > worry about, I believe, and I will be a lot more happy if my experience > > helps either folks to find and fix some probable bugs (if any) in their > > excellent software. > > Hm, there is no such thing as kqread() I think. There are kqueue() and Oh, I merely meant that thttpd process was in kqread state in top(1) output. Sorry for confusing you. > kevent(). You didn't show the piece of code that uses it, so it's hard > to say what the problem is. I recommend you to read jlemon's paper OK, this is how kqueue mechanism is used in thttpd. This is from fdwatch.c source file, starting at line 300 (I apologize for it's length): #ifdef HAVE_KQUEUE static struct kevent* kqevents; static int nkqevents; static int* kqrfdidx; static int kq; static int kqueue_init( int nfiles ) { kq = kqueue(); if ( kq == -1 ) return -1; kqevents = (struct kevent*) malloc( sizeof(struct kevent) * nfiles ); kqrfdidx = (int*) malloc( sizeof(int) * nfiles ); if ( kqevents == (struct kevent*) 0 || kqrfdidx == (int*) 0 ) return -1; (void) memset( kqevents, 0, sizeof(struct kevent) * nfiles ); (void) memset( kqrfdidx, 0, sizeof(int) * nfiles ); return 0; } static void kqueue_add_fd( int fd, int rw ) { kqevents[nkqevents].ident = fd; kqevents[nkqevents].flags = EV_ADD; switch ( rw ) { case FDW_READ: kqevents[nkqevents].filter = EVFILT_READ; break; case FDW_WRITE: kqevents[nkqevents].filter = EVFILT_WRITE; break; default: break; } ++nkqevents; } static void kqueue_del_fd( int fd ) { kqevents[nkqevents].ident = fd; kqevents[nkqevents].flags = EV_DELETE; switch ( fd_rw[fd] ) { case FDW_READ: kqevents[nkqevents].filter = EVFILT_READ; break; case FDW_WRITE: kqevents[nkqevents].filter = EVFILT_WRITE; break; } ++nkqevents; } static int kqueue_watch( long timeout_msecs ) { int i, r; if ( timeout_msecs == INFTIM ) r = kevent( kq, kqevents, nkqevents, kqevents, nfiles, (struct timespec*) 0 ); else { struct timespec ts; ts.tv_sec = timeout_msecs / 1000L; ts.tv_nsec = ( timeout_msecs % 1000L ) * 1000000L; r = kevent( kq, kqevents, nkqevents, kqevents, nfiles, &ts ); } nkqevents = 0; if ( r == -1 ) return -1; for ( i = 0; i < r; ++i ) kqrfdidx[kqevents[i].ident] = i; return r; } static int kqueue_check_fd( int fd ) { int ridx = kqrfdidx[fd]; if ( kqevents[ridx].ident != fd ) return 0; if ( kqevents[ridx].flags & EV_ERROR ) return 0; switch ( fd_rw[fd] ) { case FDW_READ: return kqevents[ridx].filter == EVFILT_READ; case FDW_WRITE: return kqevents[ridx].filter == EVFILT_WRITE; default: return 0; } } static int kqueue_get_fd( int ridx ) { return kqevents[ridx].ident; } #else /* HAVE_KQUEUE */ In my previous mail I cited the code where error is often returned, and that confuses the server. It was this one: if ( ! fdwatch_check_fd( hc->conn_fd ) ) { /* Something went wrong. */ printf("!!!!!!!!!!!!!!!!!!\n"); clear_connection( c, &tv ); } The fdwatch_check_fd() function is merely this one: /* Check if a descriptor was ready. */ int fdwatch_check_fd( int fd ) { #ifdef HAVE_KQUEUE return kqueue_check_fd( fd ); #else # ifdef HAVE_POLL return poll_check_fd( fd ); # else /* HAVE_POLL */ # ifdef HAVE_SELECT return select_check_fd( fd ); # else /* HAVE_SELECT */ return 0; # endif /* HAVE_SELECT */ # endif /* HAVE_POLL */ #endif /* HAVE_KQUEUE */ } I really do hope that it will help someone to spot any possible problems with this kqueue code. > at http://www.freebsd.org/~jlemon/kqueue.pdf to see how to you should > work with kqueue() and kevent(). > As for performance, I can confirm that kqueue() functions is better > than select()/poll() at least in situations where we have to do non-blocking > i/o on some large number of fd's (say, several thousands), and actually > each time we see some activity only on small number of them (say, > several hundred). That's how ircd usually works, and system load goes > down significantly with kqueue() comparing to select(). WBR, Alexey To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message