From owner-freebsd-hackers  Mon Jan 24 10:12:17 2000
Delivered-To: freebsd-hackers@freebsd.org
Received: from bomber.avantgo.com (ws1.avantgo.com [207.214.200.194])
	by hub.freebsd.org (Postfix) with ESMTP id C4961155A5
	for <freebsd-hackers@freebsd.org>; Mon, 24 Jan 2000 10:12:12 -0800 (PST)
	(envelope-from scott@avantgo.com)
Received: from river ([10.0.128.30]) by bomber.avantgo.com
          (Netscape Messaging Server 3.5)  with SMTP id 278
          for <freebsd-hackers@freebsd.org>;
          Mon, 24 Jan 2000 10:08:08 -0800
Message-ID: <01b601bf6696$60701930$1e80000a@avantgo.com>
From: "Scott Hess" <scott@avantgo.com>
To: <freebsd-hackers@freebsd.org>
Subject: Performance issue with rfork() and single socketpairs versus multiple socketpairs.
Date: Mon, 24 Jan 2000 10:11:03 -0800
MIME-Version: 1.0
Content-Type: multipart/mixed;
	boundary="----=_NextPart_000_01B3_01BF6653.523EF680"
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 5.00.2919.6600
X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6600
Sender: owner-freebsd-hackers@FreeBSD.ORG
Precedence: bulk
X-Loop: FreeBSD.ORG

This is a multi-part message in MIME format.

------=_NextPart_000_01B3_01BF6653.523EF680
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: 7bit

I've found an odd performance issue that I cannot explain.  I'm using
socketpairs to communicate with multiple rfork(RFPROC) processes.
Initially, I used a seperate socketpair to communicate requests to each
process, with locking in the parent to synchronize access to each client.
I determined that by using a single socketpair, I could save on all the
socketpairs, and also perhaps improve performance by allowing more requests
to be dispatched than there were processes to handle them.  Whenever a
worker process finished one request, it would immediately be able to start
the next, without having to wait for the parent to receive the response and
reprocess the request structures.

Unfortunately, I've found that having a group of processes reading from a
group of socketpairs has better performance than having them all read from
a single socketpair.  I've been unable to determine why.  I've reduced the
problem down to a simple program, included as an attachment (sorry about
that).  The results of two runs of the program:

ganja% time ./commtest --single
./commtest --single  0.00s user 0.66s system 15% cpu 4.132 total
ganja% time ./commtest --multi
./commtest --multi  0.00s user 0.46s system 68% cpu 0.675 total

Note that in the --single case, the system time rises a bit - but the
wallclock time rises a _lot_.  At first I thought this was a variant on the
"thundering herd" problem, but the CPU times taken don't seem to bear this
out.

Any ideas?  Running under 3.2-RELEASE on an SMP machine, though I saw the
same results on 3.4-RELEASE.

Thanks,
scott


------=_NextPart_000_01B3_01BF6653.523EF680
Content-Type: application/octet-stream;
	name="commtest.c"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: attachment;
	filename="commtest.c"

// commtest.c=0A=
// gcc -Wall -g -o commtest commtest.c=0A=
//=0A=
// Test performance differences for multiple socketpairs versus a=0A=
// single shared socketpair.=0A=
#include <unistd.h>=0A=
#include <stdio.h>=0A=
#include <stdlib.h>=0A=
#include <fcntl.h>=0A=
#include <sys/types.h>=0A=
#include <sys/socket.h>=0A=
#include <string.h>=0A=
#include <errno.h>=0A=
=0A=
typedef unsigned char request_t;=0A=
=0A=
#define CLIENT_EXIT ((request_t)(~0))=0A=
#define CLIENT_COUNT 32=0A=
#define REQUEST_TARGET 10000=0A=
=0A=
int client_fd_count=3D0;=0A=
int client_fds[ CLIENT_COUNT];=0A=
int server_fds[ CLIENT_COUNT];=0A=
=0A=
/* Reflect requests. */=0A=
void client( int fd)=0A=
{=0A=
    request_t request;=0A=
    int rc;=0A=
    =0A=
    while( 1) {=0A=
        if( (rc=3Dread( fd, &request, sizeof( request)))=3D=3D-1) {=0A=
            perror( "client read");=0A=
            _exit( 1);=0A=
        } else if( rc<sizeof( request)) {=0A=
            _exit( 0);=0A=
        } else if( (rc=3Dwrite( fd, &request, sizeof( =
request)))=3D=3D-1) {=0A=
            perror( "client write");=0A=
            _exit( 1);=0A=
        } else if( rc<sizeof( request)) {=0A=
            _exit( 0);=0A=
        }=0A=
    }=0A=
}=0A=
=0A=
void main( int argc, char **argv)=0A=
{=0A=
    int ii, flags, maxfd, rc;=0A=
    int pair[ 2];=0A=
    fd_set default_fdset, fdset;=0A=
    pid_t cpid;=0A=
    request_t request;=0A=
    int request_count=3D0;=0A=
    =0A=
    client_fd_count=3D1;=0A=
    if( argc=3D=3D2 && !strcmp( argv[ 1], "--multi")) {=0A=
        client_fd_count=3DCLIENT_COUNT;=0A=
    } else if( argc=3D=3D2 && !strcmp( argv[ 1], "--single")) {=0A=
        client_fd_count=3D1;=0A=
    } else if( argc=3D=3D2 && !strcmp( argv[ 1], "--handful")) {=0A=
        client_fd_count=3D4;=0A=
    }=0A=
    =0A=
    maxfd=3D0;=0A=
    FD_ZERO( &default_fdset);=0A=
    for( ii=3D0; ii<client_fd_count; ii++) {=0A=
        if( socketpair(AF_UNIX,SOCK_STREAM,0,pair)=3D=3D-1) {=0A=
            perror( "creating socketpair");=0A=
            goto err_exit;=0A=
        } else if( (flags=3Dfcntl( pair[ 0], F_GETFL, NULL))=3D=3D-1) {=0A=
            perror( "getting flags");=0A=
            goto err_exit;=0A=
        } else if( fcntl( pair[ 0], F_SETFL, flags|O_NONBLOCK)=3D=3D-1) {=0A=
            perror( "setting flags");=0A=
            goto err_exit;=0A=
        }=0A=
        =0A=
        client_fds[ ii]=3Dpair[ 0];=0A=
        server_fds[ ii]=3Dpair[ 1];=0A=
        =0A=
        FD_SET( client_fds[ ii], &default_fdset);=0A=
        if( client_fds[ ii]>maxfd) {=0A=
            maxfd=3Dclient_fds[ ii];=0A=
        }=0A=
    }=0A=
    =0A=
    /* Spin off children to process requests. */=0A=
    for( ii=3D0; ii<CLIENT_COUNT; ii++) {=0A=
        if( (cpid=3Drfork(RFPROC))=3D=3D-1) {=0A=
            perror( "rfork");=0A=
            goto err_exit;=0A=
        } else if( cpid=3D=3D0) {=0A=
            client( server_fds[ ii%client_fd_count]);=0A=
        } else {=0A=
            request=3Dii;=0A=
            if( write( client_fds[ ii%client_fd_count], &request, =
sizeof( request))=3D=3D-1) {=0A=
                perror( "writing initial request");=0A=
                goto err_exit;=0A=
            }=0A=
        }=0A=
    }=0A=
=0A=
    while( request_count<REQUEST_TARGET) {=0A=
        fdset=3Ddefault_fdset;=0A=
        select( maxfd+1, &fdset, NULL, &fdset, NULL);=0A=
        =0A=
        for( ii=3D0; ii<client_fd_count && request_count<REQUEST_TARGET; =
ii++) {=0A=
            while( (rc=3Dread( client_fds[ ii], &request, sizeof( =
request)))=3D=3Dsizeof( request) && request_count<REQUEST_TARGET) {=0A=
                if( write( client_fds[ ii], &request, sizeof( =
request))=3D=3D-1) {=0A=
                    perror( "writing request");=0A=
                    goto err_exit;=0A=
                }=0A=
                request_count++;=0A=
            }=0A=
            if( rc=3D=3D-1 && errno!=3DEAGAIN && errno!=3DEWOULDBLOCK) {=0A=
                perror( "reading response");=0A=
                goto err_exit;=0A=
            }=0A=
        }=0A=
    }=0A=
    =0A=
    for( ii=3D0; ii<client_fd_count; ii++) {=0A=
        close( client_fds[ ii]);=0A=
    }=0A=
    exit( 0);=0A=
=0A=
err_exit:=0A=
    for( ii=3D0; ii<client_fd_count; ii++) {=0A=
        close( client_fds[ ii]);=0A=
    }=0A=
    exit( 1);=0A=
}=0A=

------=_NextPart_000_01B3_01BF6653.523EF680--



To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-hackers" in the body of the message