From owner-freebsd-performance@FreeBSD.ORG Wed Mar 9 00:56:31 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 66ABE16A4CE for ; Wed, 9 Mar 2005 00:56:31 +0000 (GMT) Received: from coverity.dreamhost.com (coverity.dreamhost.com [66.33.192.105]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2444843D53 for ; Wed, 9 Mar 2005 00:56:31 +0000 (GMT) (envelope-from phessler@coverity.com) Received: from foo.coverity.int (64-173-147-27.ded.pacbell.net [64.173.147.27]) by coverity.dreamhost.com (Postfix) with ESMTP id 5C6D1908B8 for ; Tue, 8 Mar 2005 16:56:28 -0800 (PST) Date: Tue, 8 Mar 2005 16:56:28 -0800 From: Peter Hessler To: freebsd-performance@freebsd.org Organization: Coverity X-Mailer: Sylpheed version 0.9.99 (GTK+ 1.2.10; i386-unknown-openbsd3.6) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Message-Id: <20050309005628.5C6D1908B8@coverity.dreamhost.com> Subject: NFS small files X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Mar 2005 00:56:31 -0000 I'm setting up an NFS server to be used for compiling, and it seems that while speeds are acceptable for large files, small files take much longer than expected. Copying 10000 16K files (in a directory) takes 54seconds, while copying a 170M single file takes 5s. Are there any tricks for speeding up small file performance? I'm willing to give up large file performance. /etc/sysctl.conf: vfs.vmiodirenable=1 kern.ipc.somaxconn=2048 kern.maxfiles=65536 net.isr.enable=1 /boot/loader.conf: kern.ipc.nmbclusters=32768 Kernel Conf: include GENERIC ident NFS options DEVICE_POLLING options HZ=1000 From owner-freebsd-performance@FreeBSD.ORG Wed Mar 9 01:40:43 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 654F116A4CE for ; Wed, 9 Mar 2005 01:40:43 +0000 (GMT) Received: from priv-edtnes40.telusplanet.net (outbound05.telus.net [199.185.220.224]) by mx1.FreeBSD.org (Postfix) with ESMTP id DD26643D1F for ; Wed, 9 Mar 2005 01:40:42 +0000 (GMT) (envelope-from pfak@telus.net) Received: from [192.168.1.253] (really [64.180.103.26]) by priv-edtnes40.telusplanet.net (InterMail vM.6.01.04.00 201-2131-118-20041027) with ESMTP <20050309014042.NFGH28142.priv-edtnes40.telusplanet.net@[192.168.1.253]>; Tue, 8 Mar 2005 18:40:42 -0700 Message-ID: <422E541C.8040108@telus.net> Date: Tue, 08 Mar 2005 17:40:44 -0800 From: Peter Kieser User-Agent: Mozilla Thunderbird 0.6+ (Windows/20050216) MIME-Version: 1.0 To: Peter Hessler References: <20050309005628.5C6D1908B8@coverity.dreamhost.com> In-Reply-To: <20050309005628.5C6D1908B8@coverity.dreamhost.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit cc: freebsd-performance@freebsd.org Subject: Re: NFS small files X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Mar 2005 01:40:43 -0000 Copying single files are always going to take longer, because the drive has to seek more thus latency is introduced. --Peter Peter Hessler wrote: >I'm setting up an NFS server to be used for compiling, and it seems that >while speeds are acceptable for large files, small files take much >longer than expected. > >Copying 10000 16K files (in a directory) takes 54seconds, while copying >a 170M single file takes 5s. > >Are there any tricks for speeding up small file performance? I'm >willing to give up large file performance. > >/etc/sysctl.conf: > vfs.vmiodirenable=1 > kern.ipc.somaxconn=2048 > kern.maxfiles=65536 > net.isr.enable=1 > >/boot/loader.conf: > kern.ipc.nmbclusters=32768 > >Kernel Conf: > include GENERIC > ident NFS > options DEVICE_POLLING > options HZ=1000 >_______________________________________________ >freebsd-performance@freebsd.org mailing list >http://lists.freebsd.org/mailman/listinfo/freebsd-performance >To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" > > > From owner-freebsd-performance@FreeBSD.ORG Wed Mar 9 01:46:33 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1D48416A4CE for ; Wed, 9 Mar 2005 01:46:33 +0000 (GMT) Received: from coverity.dreamhost.com (coverity.dreamhost.com [66.33.192.105]) by mx1.FreeBSD.org (Postfix) with ESMTP id D7D5543D49 for ; Wed, 9 Mar 2005 01:46:32 +0000 (GMT) (envelope-from phessler@coverity.com) Received: from foo.coverity.int (64-173-147-27.ded.pacbell.net [64.173.147.27]) by coverity.dreamhost.com (Postfix) with ESMTP id 49895908B7 for ; Tue, 8 Mar 2005 17:46:32 -0800 (PST) Date: Tue, 8 Mar 2005 17:46:31 -0800 From: Peter Hessler To: freebsd-performance@freebsd.org In-Reply-To: <422E541C.8040108@telus.net> References: <20050309005628.5C6D1908B8@coverity.dreamhost.com> <422E541C.8040108@telus.net> Organization: Coverity X-Mailer: Sylpheed version 0.9.99 (GTK+ 1.2.10; i386-unknown-openbsd3.6) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Message-Id: <20050309014632.49895908B7@coverity.dreamhost.com> Subject: Re: NFS small files X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Mar 2005 01:46:33 -0000 These files are sitting right next to each other on disk, I created them at the same time: """for i in `jot 10000`; do dd if=/dev/random of=$i bs=128 count=128; done""". Then I timed the copy of that directory. Does seek really cause an 11x performance penalty? On Tue, 08 Mar 2005 17:40:44 -0800 Peter Kieser wrote: : Copying single files are always going to take longer, because the drive : has to seek more thus latency is introduced. : : --Peter : : Peter Hessler wrote: : : >I'm setting up an NFS server to be used for compiling, and it seems that : >while speeds are acceptable for large files, small files take much : >longer than expected. : > : >Copying 10000 16K files (in a directory) takes 54seconds, while copying : >a 170M single file takes 5s. : > : >Are there any tricks for speeding up small file performance? I'm : >willing to give up large file performance. : > : >/etc/sysctl.conf: : > vfs.vmiodirenable=1 : > kern.ipc.somaxconn=2048 : > kern.maxfiles=65536 : > net.isr.enable=1 : > : >/boot/loader.conf: : > kern.ipc.nmbclusters=32768 : > : >Kernel Conf: : > include GENERIC : > ident NFS : > options DEVICE_POLLING : > options HZ=1000 : >_______________________________________________ : >freebsd-performance@freebsd.org mailing list : >http://lists.freebsd.org/mailman/listinfo/freebsd-performance : >To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" : > : > : > : : _______________________________________________ : freebsd-performance@freebsd.org mailing list : http://lists.freebsd.org/mailman/listinfo/freebsd-performance : To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" From owner-freebsd-performance@FreeBSD.ORG Wed Mar 9 01:51:20 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5513116A4CE for ; Wed, 9 Mar 2005 01:51:20 +0000 (GMT) Received: from coverity.dreamhost.com (coverity.dreamhost.com [66.33.192.105]) by mx1.FreeBSD.org (Postfix) with ESMTP id E755043D4C for ; Wed, 9 Mar 2005 01:51:19 +0000 (GMT) (envelope-from phessler@coverity.com) Received: from foo.coverity.int (64-173-147-27.ded.pacbell.net [64.173.147.27]) by coverity.dreamhost.com (Postfix) with ESMTP id 44566908B7 for ; Tue, 8 Mar 2005 17:51:19 -0800 (PST) Date: Tue, 8 Mar 2005 17:51:19 -0800 From: Peter Hessler To: freebsd-performance@freebsd.org In-Reply-To: <20050309014632.49895908B7@coverity.dreamhost.com> References: <20050309005628.5C6D1908B8@coverity.dreamhost.com> <422E541C.8040108@telus.net> <20050309014632.49895908B7@coverity.dreamhost.com> Organization: Coverity X-Mailer: Sylpheed version 0.9.99 (GTK+ 1.2.10; i386-unknown-openbsd3.6) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Message-Id: <20050309015119.44566908B7@coverity.dreamhost.com> Subject: Re: NFS small files X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Mar 2005 01:51:20 -0000 I should mention that copying the same directory on the nfs server takes 15s, rather than the 54s over nfs. On Tue, 8 Mar 2005 17:46:31 -0800 Peter Hessler wrote: : These files are sitting right next to each other on disk, I created them : at the same time: """for i in `jot 10000`; do dd if=/dev/random of=$i : bs=128 count=128; done""". Then I timed the copy of that directory. : : Does seek really cause an 11x performance penalty? : : On Tue, 08 Mar 2005 17:40:44 -0800 : Peter Kieser wrote: : : : Copying single files are always going to take longer, because the : drive : : has to seek more thus latency is introduced. : : : : --Peter : : : : Peter Hessler wrote: : : : : >I'm setting up an NFS server to be used for compiling, and it seems : that : : >while speeds are acceptable for large files, small files take much : : >longer than expected. : : > : : >Copying 10000 16K files (in a directory) takes 54seconds, while : copying : : >a 170M single file takes 5s. : : > : : >Are there any tricks for speeding up small file performance? I'm : : >willing to give up large file performance. : : > : : >/etc/sysctl.conf: : : > vfs.vmiodirenable=1 : : > kern.ipc.somaxconn=2048 : : > kern.maxfiles=65536 : : > net.isr.enable=1 : : > : : >/boot/loader.conf: : : > kern.ipc.nmbclusters=32768 : : > : : >Kernel Conf: : : > include GENERIC : : > ident NFS : : > options DEVICE_POLLING : : > options HZ=1000 : : >_______________________________________________ : : >freebsd-performance@freebsd.org mailing list : : >http://lists.freebsd.org/mailman/listinfo/freebsd-performance : : >To unsubscribe, send any mail to : "freebsd-performance-unsubscribe@freebsd.org" : : > : : > : : > : : : : _______________________________________________ : : freebsd-performance@freebsd.org mailing list : : http://lists.freebsd.org/mailman/listinfo/freebsd-performance : : To unsubscribe, send any mail to : "freebsd-performance-unsubscribe@freebsd.org" : _______________________________________________ : freebsd-performance@freebsd.org mailing list : http://lists.freebsd.org/mailman/listinfo/freebsd-performance : To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" From owner-freebsd-performance@FreeBSD.ORG Wed Mar 9 02:22:43 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7DABF16A4CE for ; Wed, 9 Mar 2005 02:22:43 +0000 (GMT) Received: from mh1.centtech.com (moat3.centtech.com [207.200.51.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 48D5643D3F for ; Wed, 9 Mar 2005 02:22:42 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from [192.168.42.22] (andersonbox2.centtech.com [192.168.42.22]) by mh1.centtech.com (8.13.1/8.13.1) with ESMTP id j292MZ1B023495; Tue, 8 Mar 2005 20:22:35 -0600 (CST) (envelope-from anderson@centtech.com) Message-ID: <422E5DE1.9040901@centtech.com> Date: Tue, 08 Mar 2005 20:22:25 -0600 From: Eric Anderson User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.5) Gecko/20050210 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Peter Hessler References: <20050309005628.5C6D1908B8@coverity.dreamhost.com> <422E541C.8040108@telus.net> <20050309014632.49895908B7@coverity.dreamhost.com> <20050309015119.44566908B7@coverity.dreamhost.com> In-Reply-To: <20050309015119.44566908B7@coverity.dreamhost.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.82/757/Tue Mar 8 17:14:36 2005 on mh1.centtech.com X-Virus-Status: Clean cc: freebsd-performance@freebsd.org Subject: Re: NFS small files X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Mar 2005 02:22:43 -0000 Peter Hessler wrote: > I should mention that copying the same directory on the nfs server takes > 15s, rather than the 54s over nfs. Are you using TCP or UDP NFS connection, version 2 or 3? > On Tue, 8 Mar 2005 17:46:31 -0800 > Peter Hessler wrote: > > : These files are sitting right next to each other on disk, I created > them > : at the same time: """for i in `jot 10000`; do dd if=/dev/random of=$i > : bs=128 count=128; done""". Then I timed the copy of that directory. > : > : Does seek really cause an 11x performance penalty? > : > : On Tue, 08 Mar 2005 17:40:44 -0800 > : Peter Kieser wrote: > : > : : Copying single files are always going to take longer, because the > : drive > : : has to seek more thus latency is introduced. > : : > : : --Peter > : : > : : Peter Hessler wrote: > : : > : : >I'm setting up an NFS server to be used for compiling, and it seems > : that > : : >while speeds are acceptable for large files, small files take much > : : >longer than expected. > : : > > : : >Copying 10000 16K files (in a directory) takes 54seconds, while > : copying > : : >a 170M single file takes 5s. > : : > > : : >Are there any tricks for speeding up small file performance? I'm > : : >willing to give up large file performance. > : : > > : : >/etc/sysctl.conf: > : : > vfs.vmiodirenable=1 > : : > kern.ipc.somaxconn=2048 > : : > kern.maxfiles=65536 > : : > net.isr.enable=1 > : : > > : : >/boot/loader.conf: > : : > kern.ipc.nmbclusters=32768 > : : > > : : >Kernel Conf: > : : > include GENERIC > : : > ident NFS > : : > options DEVICE_POLLING > : : > options HZ=1000 > : : >_______________________________________________ > : : >freebsd-performance@freebsd.org mailing list > : : >http://lists.freebsd.org/mailman/listinfo/freebsd-performance > : : >To unsubscribe, send any mail to > : "freebsd-performance-unsubscribe@freebsd.org" > : : > > : : > > : : > > : : > : : _______________________________________________ > : : freebsd-performance@freebsd.org mailing list > : : http://lists.freebsd.org/mailman/listinfo/freebsd-performance > : : To unsubscribe, send any mail to > : "freebsd-performance-unsubscribe@freebsd.org" > : _______________________________________________ > : freebsd-performance@freebsd.org mailing list > : http://lists.freebsd.org/mailman/listinfo/freebsd-performance > : To unsubscribe, send any mail to > "freebsd-performance-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" -- ------------------------------------------------------------------------ Eric Anderson Sr. Systems Administrator Centaur Technology I have seen the future and it is just like the present, only longer. ------------------------------------------------------------------------ From owner-freebsd-performance@FreeBSD.ORG Wed Mar 9 02:25:04 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A4C7B16A4CE for ; Wed, 9 Mar 2005 02:25:04 +0000 (GMT) Received: from mh1.centtech.com (moat3.centtech.com [207.200.51.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id E3B5543D31 for ; Wed, 9 Mar 2005 02:25:03 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from [192.168.42.22] (andersonbox2.centtech.com [192.168.42.22]) by mh1.centtech.com (8.13.1/8.13.1) with ESMTP id j292P1Yq023511; Tue, 8 Mar 2005 20:25:01 -0600 (CST) (envelope-from anderson@centtech.com) Message-ID: <422E5E78.3000703@centtech.com> Date: Tue, 08 Mar 2005 20:24:56 -0600 From: Eric Anderson User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.5) Gecko/20050210 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Peter Hessler References: <20050309005628.5C6D1908B8@coverity.dreamhost.com> In-Reply-To: <20050309005628.5C6D1908B8@coverity.dreamhost.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.82/757/Tue Mar 8 17:14:36 2005 on mh1.centtech.com X-Virus-Status: Clean cc: freebsd-performance@freebsd.org Subject: Re: NFS small files X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Mar 2005 02:25:04 -0000 Peter Hessler wrote: > I'm setting up an NFS server to be used for compiling, and it seems that > while speeds are acceptable for large files, small files take much > longer than expected. > > Copying 10000 16K files (in a directory) takes 54seconds, while copying > a 170M single file takes 5s. > > Are there any tricks for speeding up small file performance? I'm > willing to give up large file performance. What mount options are you using on the client? -- ------------------------------------------------------------------------ Eric Anderson Sr. Systems Administrator Centaur Technology I have seen the future and it is just like the present, only longer. ------------------------------------------------------------------------ From owner-freebsd-performance@FreeBSD.ORG Wed Mar 9 04:15:52 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4BF8116A4CE for ; Wed, 9 Mar 2005 04:15:52 +0000 (GMT) Received: from rproxy.gmail.com (rproxy.gmail.com [64.233.170.199]) by mx1.FreeBSD.org (Postfix) with ESMTP id D057E43D1D for ; Wed, 9 Mar 2005 04:15:51 +0000 (GMT) (envelope-from joseph.koshy@gmail.com) Received: by rproxy.gmail.com with SMTP id j1so73199rnf for ; Tue, 08 Mar 2005 20:15:51 -0800 (PST) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:reply-to:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:references; b=GoXCZAl8jhFaFiYTwy9qbpLhPYkjrDpgMRj3X6GPXOSdkMKdN2ZeTNY9I9NJBDOKahEajJuNMWrnTX0lfHMESKsrJ2UJ/TYVa56qdSv9/N0XHjgebOaEG4GGI72gavUm4BKziwdPwQtDyr6dpDTs24/04tXCPXkAVKFvUW/a2Gw= Received: by 10.38.181.32 with SMTP id d32mr441137rnf; Tue, 08 Mar 2005 20:15:51 -0800 (PST) Received: by 10.38.209.22 with HTTP; Tue, 8 Mar 2005 20:15:51 -0800 (PST) Message-ID: <84dead720503082015c11bb53@mail.gmail.com> Date: Wed, 9 Mar 2005 04:15:51 +0000 From: Joseph Koshy To: Peter Hessler In-Reply-To: <20050309015119.44566908B7@coverity.dreamhost.com> Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit References: <20050309005628.5C6D1908B8@coverity.dreamhost.com> <422E541C.8040108@telus.net> <20050309014632.49895908B7@coverity.dreamhost.com> <20050309015119.44566908B7@coverity.dreamhost.com> cc: freebsd-performance@freebsd.org Subject: Re: NFS small files X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: Joseph Koshy List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Mar 2005 04:15:52 -0000 > I should mention that copying the same directory on the nfs server takes Filename resolution over NFS involves multiple round-trip requests, one for each component of the file's pathname. For example, assuming that '/mnt' is an NFS mount-point, accessing a file named '/mnt/FreeBSD/src/sys/sys/file.h,v' results in the following NFS requests: [Edited output from ethereal] 1 0.000000 ::1 ::1 NFS V3 LOOKUP Call (Reply In 2), DH:0xa9a212b8/FreeBSD V3 Procedure: LOOKUP (3) what: dir Name: FreeBSD 2 0.000116 ::1 ::1 NFS V3 LOOKUP Reply (Call In 1), FH:0xcd6610c4 Status: OK (0) 3 0.000162 ::1 ::1 NFS V3 LOOKUP Call (Reply In 4), DH:0xcd6610c4/src V3 Procedure: LOOKUP (3) what: dir Name: src 4 0.000194 ::1 ::1 NFS V3 LOOKUP Reply (Call In 3), FH:0x5c9e0ec6 Status: OK (0) 5 0.000224 ::1 ::1 NFS V3 LOOKUP Call (Reply In 6), DH:0x5c9e0ec6/sys V3 Procedure: LOOKUP (3) what: dir Name: sys 6 0.000253 ::1 ::1 NFS V3 LOOKUP Reply (Call In 5), FH:0xb4d2113a Status: OK (0) 7 0.000281 ::1 ::1 NFS V3 LOOKUP Call (Reply In 8), DH:0xb4d2113a/sys V3 Procedure: LOOKUP (3) what: dir Name: sys 8 0.000312 ::1 ::1 NFS V3 LOOKUP Reply (Call In 7), FH:0xb0f9f738 Status: OK (0) 9 0.000338 ::1 ::1 NFS V3 LOOKUP Call (Reply In 10), DH:0xb0f9f738/file.h,v V3 Procedure: LOOKUP (3) what: dir Name: file.h,v 10 0.000366 ::1 ::1 NFS V3 LOOKUP Reply (Call In 9), FH:0x58ea0f3a Status: OK (0) 11 0.002220 ::1 ::1 NFS V3 ACCESS Call, FH:0xa9a212b8 V3 Procedure: ACCESS (4) object access: 0x3f I can imagine that the extra latency induced by these round-trips would be adding to elapsed time for your test. -- FreeBSD Volunteer, http://people.freebsd.org/~jkoshy From owner-freebsd-performance@FreeBSD.ORG Wed Mar 9 10:09:35 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 991C116A4CF for ; Wed, 9 Mar 2005 10:09:35 +0000 (GMT) Received: from elaine.ispinfo.fr (elaine.ispinfo.fr [81.255.64.45]) by mx1.FreeBSD.org (Postfix) with ESMTP id 82D9D43D69 for ; Wed, 9 Mar 2005 10:09:34 +0000 (GMT) (envelope-from mlh@ispinfo.fr) Received: from uriens.ispinfo.fr (smtp0.ispinfo.fr [81.255.64.47]) by elaine.ispinfo.fr (8.9.3/8.9.3) with ESMTP id LAA43653 for ; Wed, 9 Mar 2005 11:09:33 +0100 (CET) (envelope-from mlh@ispinfo.fr) Received: from smtpz.awape.fr (smtpz.ispinfo.fr [81.255.64.49]) by uriens.ispinfo.fr (8.11.1/8.11.1) with ESMTP id j29A9WJ15144 for ; Wed, 9 Mar 2005 11:09:32 +0100 (CET) (envelope-from mlh@ispinfo.fr) Received: from smtpe.ispinfo.fr (smtpe.ispinfo.fr [81.255.64.52]) by smtpz.awape.fr (8.12.11/8.12.11) with ESMTP id j29A9USO004324 for ; Wed, 9 Mar 2005 11:09:30 +0100 (CET) (envelope-from mlh@ispinfo.fr) Received: from [192.168.1.82] ([192.168.1.82]) by smtpe.ispinfo.fr (8.12.8p1/8.12.3) with ESMTP id j29A9U0m010316 for ; Wed, 9 Mar 2005 11:09:30 +0100 (CET) (envelope-from mlh@ispinfo.fr) Message-ID: <422ECAF7.4050407@ispinfo.fr> Date: Wed, 09 Mar 2005 11:07:51 +0100 From: ISP Informatique Organization: ISP Informatique User-Agent: Mozilla Thunderbird 1.0 (X11/20050124) X-Accept-Language: en-us, en MIME-Version: 1.0 To: freebsd-performance@freebsd.org X-Enigmail-Version: 0.89.5.0 X-Enigmail-Supports: pgp-inline, pgp-mime Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit Subject: FreeBSD 5.3 et multiprocessor X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Mar 2005 10:09:35 -0000 Hello, I recently have just passed a server of FreeBSD 4.2 to FreeBSD 5.3. The nasty surprise being which I note a notable reduction in the performances. The machine is a HP LT6000r, quadri Xeon 700 with 2Go of RAM. The results are appreciably the same ones with a kernel including or not "options SMP". Did I miss some thing? or perhaps did this already arrive at others? In particular, I has just crossed this in http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html > > options SCHED_4BSD # 4BSD scheduler > > > The traditional scheduler for FreeBSD. Depending on your system's > workload, you may gain performance by using the new ULE scheduler for > FreeBSD that has been designed specially for SMP, but works just fine > on UP systems too. If you wish to try it out, replace SCHED_4BSD with > SCHED_ULE in your configuration file. Did you already test? By advance, thank you, -- Hubert Adgié ISP Informatique www.ispinfo.fr 0890 710 147 From owner-freebsd-performance@FreeBSD.ORG Wed Mar 9 10:30:23 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6B9BB16A4CE for ; Wed, 9 Mar 2005 10:30:23 +0000 (GMT) Received: from web26802.mail.ukl.yahoo.com (web26802.mail.ukl.yahoo.com [217.146.176.78]) by mx1.FreeBSD.org (Postfix) with SMTP id D411743D5E for ; Wed, 9 Mar 2005 10:30:22 +0000 (GMT) (envelope-from cguttesen@yahoo.dk) Received: (qmail 38903 invoked by uid 60001); 9 Mar 2005 10:30:21 -0000 Message-ID: <20050309103021.38901.qmail@web26802.mail.ukl.yahoo.com> Received: from [194.248.174.58] by web26802.mail.ukl.yahoo.com via HTTP; Wed, 09 Mar 2005 11:30:21 CET Date: Wed, 9 Mar 2005 11:30:21 +0100 (CET) From: Claus Guttesen To: ISP Informatique , freebsd-performance@freebsd.org In-Reply-To: 6667 MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Subject: Re: FreeBSD 5.3 et multiprocessor X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Mar 2005 10:30:23 -0000 > The nasty surprise being which I note a notable > reduction in the performances. > The machine is a HP LT6000r, quadri Xeon 700 with > 2Go of RAM. > > The results are appreciably the same ones with a > kernel including or not > "options SMP". Did you copy GENERIC to a new file and performed the required changes to this new file, or did you change and recompiled GENERIC? What does dmesg give you? regards Claus From owner-freebsd-performance@FreeBSD.ORG Wed Mar 9 11:34:15 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8F50416A4CE for ; Wed, 9 Mar 2005 11:34:15 +0000 (GMT) Received: from elaine.ispinfo.fr (elaine.ispinfo.fr [81.255.64.45]) by mx1.FreeBSD.org (Postfix) with ESMTP id 81F5D43D5F for ; Wed, 9 Mar 2005 11:34:14 +0000 (GMT) (envelope-from mlh@ispinfo.fr) Received: from uriens.ispinfo.fr (smtp0.ispinfo.fr [81.255.64.47]) by elaine.ispinfo.fr (8.9.3/8.9.3) with ESMTP id MAA48837 for ; Wed, 9 Mar 2005 12:34:13 +0100 (CET) (envelope-from mlh@ispinfo.fr) Received: from smtpz.awape.fr (smtpz.ispinfo.fr [81.255.64.49]) by uriens.ispinfo.fr (8.11.1/8.11.1) with ESMTP id j29BYCJ83742 for ; Wed, 9 Mar 2005 12:34:13 +0100 (CET) (envelope-from mlh@ispinfo.fr) Received: from smtpe.ispinfo.fr (smtpe.ispinfo.fr [81.255.64.52]) by smtpz.awape.fr (8.12.11/8.12.11) with ESMTP id j29BY5ae007735 for ; Wed, 9 Mar 2005 12:34:09 +0100 (CET) (envelope-from mlh@ispinfo.fr) Received: from [192.168.1.82] ([192.168.1.82]) by smtpe.ispinfo.fr (8.12.8p1/8.12.3) with ESMTP id j29BY10m011274 for ; Wed, 9 Mar 2005 12:34:02 +0100 (CET) (envelope-from mlh@ispinfo.fr) Message-ID: <422EDF29.2090401@ispinfo.fr> Date: Wed, 09 Mar 2005 12:34:01 +0100 From: ISP Informatique Organization: ISP Informatique User-Agent: Mozilla Thunderbird 1.0 (X11/20050124) X-Accept-Language: en-us, en MIME-Version: 1.0 To: freebsd-performance@freebsd.org References: <20050309103021.38901.qmail@web26802.mail.ukl.yahoo.com> In-Reply-To: <20050309103021.38901.qmail@web26802.mail.ukl.yahoo.com> X-Enigmail-Version: 0.89.5.0 X-Enigmail-Supports: pgp-inline, pgp-mime Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit Subject: Re: FreeBSD 5.3 et multiprocessor X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Mar 2005 11:34:15 -0000 Claus Guttesen wrote: > Did you copy GENERIC to a new file and performed the > required changes to this new file, or did you change > and recompiled GENERIC? Copied, modified and recompiled. > What does dmesg give you? here it is : > Copyright (c) 1992-2004 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > The Regents of the University of California. All rights reserved. > FreeBSD 5.3-RELEASE-p5 #0: Sun Mar 6 00:57:32 CET 2005 > root@lancelot.acces-industrie.com:/usr/obj/usr/src/sys/GENERIC > MPTable: > Timecounter "i8254" frequency 1193182 Hz quality 0 > CPU: Intel Pentium III Xeon (700.02-MHz 686-class CPU) > Origin = "GenuineIntel" Id = 0x6a1 Stepping = 1 > Features=0x387fbff > real memory = 2147418112 (2047 MB) > avail memory = 2095939584 (1998 MB) > ioapic0: Assuming intbase of 0 > ioapic1: Assuming intbase of 16 > ioapic0 irqs 0-15 on motherboard > ioapic1 irqs 16-31 on motherboard > npx0: [FAST] > npx0: on motherboard > npx0: INT 16 interface > pcib1: pcibus 1 on motherboard > pci1: on pcib1 > pcib0: pcibus 0 on motherboard > pci0: on pcib0 > fxp0: port 0x1800-0x183f mem 0xec900000-0xec9fffff,0xec801000-0xec801fff irq 18 at device 6.0 on pci0 > miibus0: on fxp0 > inphy0: on miibus0 > inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto > fxp0: Ethernet address: 00:30:6e:02:f8:f0 > pci0: at device 7.0 (no driver attached) > isab0: port 0x1040-0x104f at device 15.0 on pci0 > isa0: on isab0 > atapci0: port 0x1840-0x184f,0x376,0x170-0x177,0x3f6,0x1f0-0x1f7 at device 15.1 on pci0 > ata0: channel #0 on atapci0 > ata1: channel #1 on atapci0 > pcib4: pcibus 4 on motherboard > pci4: on pcib4 > pci4: at device 2.0 (no driver attached) > pci4: at device 2.1 (no driver attached) > pcib2: at device 3.0 on pci4 > pci5: on pcib2 > pci5: at device 8.0 (no driver attached) > amr0: mem 0xf4000000-0xf7ffffff irq 20 at device 3.1 on pci4 > amr0: [GIANT-LOCKED] > amr0: Firmware E.01.08, BIOS B.02.03, 64MB RAM > cpu0 on motherboard > orm0: at iomem 0xc8000-0xc8fff,0xc0000-0xc7fff on isa0 > pmtimer0 on isa0 > atkbdc0: at port 0x64,0x60 on isa0 > atkbd0: irq 1 on atkbdc0 > kbd0 at atkbd0 > atkbd0: [GIANT-LOCKED] > psm0: irq 12 on atkbdc0 > psm0: [GIANT-LOCKED] > psm0: model IntelliMouse, device ID 3 > fdc0: at port 0x3f0-0x3f5 irq 6 drq 2 on isa0 > fdc0: [FAST] > fd0: <1440-KB 3.5" drive> on fdc0 drive 0 > ppc0: at port 0x378-0x37f irq 7 on isa0 > ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode > ppbus0: on ppc0 > plip0: on ppbus0 > lpt0: on ppbus0 > lpt0: Interrupt-driven port > ppi0: on ppbus0 > sc0: at flags 0x100 on isa0 > sc0: VGA <16 virtual consoles, flags=0x300> > sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0 > sio0: type 16550A > sio1 at port 0x2f8-0x2ff irq 3 on isa0 > sio1: type 16550A > vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 > unknown: can't assign resources (port) > unknown: can't assign resources (port) > unknown: can't assign resources (port) > unknown: can't assign resources (irq) > unknown: can't assign resources (port) > unknown: can't assign resources (port) > unknown: can't assign resources (port) > Timecounter "TSC" frequency 700015817 Hz quality 800 > Timecounters tick every 10.000 msec > acd0: CDROM at ata0-master PIO4 > amrd0: on amr0 > amrd0: 17365MB (35563520 sectors) RAID 1 (optimal) > ses0 at amr0 bus 0 target 5 lun 0 > ses0: Fixed Processor SCSI-2 device ses0: SAF-TE Compliant Device > Mounting root from ufs:/dev/amrd0s1a looking at the administrative mail, i saw that I got this in the first dmesg after install (and so before kernel recompiling) MPTable: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 4 then cpu1 on motherboard cpu2 on motherboard cpu3 on motherboard and at the end : SMP: AP CPU #1 Launched! SMP: AP CPU #2 Launched! SMP: AP CPU #3 Launched! guess that something is missing in the kernel :( -- Hubert Adgié ISP Informatique www.ispinfo.fr 0890 710 147 From owner-freebsd-performance@FreeBSD.ORG Wed Mar 9 11:40:15 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 073B816A4CE for ; Wed, 9 Mar 2005 11:40:15 +0000 (GMT) Received: from web26806.mail.ukl.yahoo.com (web26806.mail.ukl.yahoo.com [217.146.176.82]) by mx1.FreeBSD.org (Postfix) with SMTP id 74CA243D54 for ; Wed, 9 Mar 2005 11:40:14 +0000 (GMT) (envelope-from cguttesen@yahoo.dk) Received: (qmail 76622 invoked by uid 60001); 9 Mar 2005 11:40:13 -0000 Message-ID: <20050309114013.76620.qmail@web26806.mail.ukl.yahoo.com> Received: from [194.248.174.58] by web26806.mail.ukl.yahoo.com via HTTP; Wed, 09 Mar 2005 12:40:13 CET Date: Wed, 9 Mar 2005 12:40:13 +0100 (CET) From: Claus Guttesen To: ISP Informatique , freebsd-performance@freebsd.org In-Reply-To: <422EDF29.2090401@ispinfo.fr> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Subject: Re: FreeBSD 5.3 et multiprocessor X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Mar 2005 11:40:15 -0000 > > Did you copy GENERIC to a new file and performed > > the required changes to this new file, or did you > > change and recompiled GENERIC? > > Copied, modified and recompiled. > > > What does dmesg give you? >root@lancelot.acces-industrie.com:/usr/obj/usr/src/sys/GENERIC Looks like you recompiled GENERIC. Did you do a 'make buildkernel KERNCONF=mykernel'? regards Claus From owner-freebsd-performance@FreeBSD.ORG Wed Mar 9 11:43:10 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 374EA16A4CE for ; Wed, 9 Mar 2005 11:43:10 +0000 (GMT) Received: from indigo.declera.com (indigo.declera.com [212.116.131.21]) by mx1.FreeBSD.org (Postfix) with SMTP id 05CC443D54 for ; Wed, 9 Mar 2005 11:43:09 +0000 (GMT) (envelope-from dachev@nove.bg) Received: (qmail 3662 invoked by uid 1474); 9 Mar 2005 11:40:14 -0000 Received: from dachev@nove.bg by indigo.declera.com by uid 401 with qmail-scanner-1.22 Clear:RC:1(192.168.6.2):. Processed in 0.012917 secs); 09 Mar 2005 11:40:14 -0000 Received: from klater.hutnet.declera.com (HELO ?192.168.6.2?) (192.168.6.2) by indigo.declera.com with SMTP; 9 Mar 2005 11:40:14 -0000 Message-ID: <422EE14A.9050606@nove.bg> Date: Wed, 09 Mar 2005 13:43:06 +0200 From: Velko Ivanov User-Agent: Mozilla Thunderbird 1.0 (X11/20050128) X-Accept-Language: en-us, en MIME-Version: 1.0 To: ISP Informatique References: <20050309103021.38901.qmail@web26802.mail.ukl.yahoo.com> <422EDF29.2090401@ispinfo.fr> In-Reply-To: <422EDF29.2090401@ispinfo.fr> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit cc: freebsd-performance@freebsd.org Subject: Re: FreeBSD 5.3 et multiprocessor X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Mar 2005 11:43:10 -0000 > guess that something is missing in the kernel :( > Always read /usr/src/UPDATING first: 20041023: GENERIC kernel config files for amd64 and i386 no longer have "options SMP". That has been moved to a new config file named SMP for users who want an easy way to build an SMP kernel. From owner-freebsd-performance@FreeBSD.ORG Wed Mar 9 22:51:51 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 451EA16A4CE for ; Wed, 9 Mar 2005 22:51:51 +0000 (GMT) Received: from coverity.dreamhost.com (coverity.dreamhost.com [66.33.192.105]) by mx1.FreeBSD.org (Postfix) with ESMTP id 029DE43D41 for ; Wed, 9 Mar 2005 22:51:51 +0000 (GMT) (envelope-from phessler@coverity.com) Received: from leela.theapt.org (piro.theapt.org [208.201.244.164]) by coverity.dreamhost.com (Postfix) with ESMTP id 721EA9089D for ; Wed, 9 Mar 2005 14:51:50 -0800 (PST) Date: Wed, 9 Mar 2005 14:48:26 -0800 From: Peter Hessler To: freebsd-performance@freebsd.org Message-Id: <20050309144826.2e44ebeb@leela.theapt.org> In-Reply-To: <422E5DE1.9040901@centtech.com> References: <20050309005628.5C6D1908B8@coverity.dreamhost.com> <422E541C.8040108@telus.net> <20050309014632.49895908B7@coverity.dreamhost.com> <20050309015119.44566908B7@coverity.dreamhost.com> <422E5DE1.9040901@centtech.com> X-Mailer: Sylpheed version 0.9.9claws (GTK+ 1.2.10; powerpc-unknown-openbsd3.6) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Subject: Re: NFS small files X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Mar 2005 22:51:51 -0000 On Tue, 08 Mar 2005 20:22:25 -0600 Eric Anderson wrote: :Peter Hessler wrote: :> I should mention that copying the same directory on the nfs server takes :> 15s, rather than the 54s over nfs. : : :Are you using TCP or UDP NFS connection, version 2 or 3? : TCP and v3. I didn't see a difference between TCP and UDP. : : :> On Tue, 8 Mar 2005 17:46:31 -0800 :> Peter Hessler wrote: :> :> : These files are sitting right next to each other on disk, I created :> them :> : at the same time: """for i in `jot 10000`; do dd if=/dev/random of=$i :> : bs=128 count=128; done""". Then I timed the copy of that directory. :> : :> : Does seek really cause an 11x performance penalty? :> : :> : On Tue, 08 Mar 2005 17:40:44 -0800 :> : Peter Kieser wrote: :> : :> : : Copying single files are always going to take longer, because the :> : drive :> : : has to seek more thus latency is introduced. :> : : :> : : --Peter :> : : :> : : Peter Hessler wrote: :> : : :> : : >I'm setting up an NFS server to be used for compiling, and it seems :> : that :> : : >while speeds are acceptable for large files, small files take much :> : : >longer than expected. :> : : > :> : : >Copying 10000 16K files (in a directory) takes 54seconds, while :> : copying :> : : >a 170M single file takes 5s. :> : : > :> : : >Are there any tricks for speeding up small file performance? I'm :> : : >willing to give up large file performance. :> : : > :> : : >/etc/sysctl.conf: :> : : > vfs.vmiodirenable=1 :> : : > kern.ipc.somaxconn=2048 :> : : > kern.maxfiles=65536 :> : : > net.isr.enable=1 :> : : > :> : : >/boot/loader.conf: :> : : > kern.ipc.nmbclusters=32768 :> : : > :> : : >Kernel Conf: :> : : > include GENERIC :> : : > ident NFS :> : : > options DEVICE_POLLING :> : : > options HZ=1000 :> : : >_______________________________________________ :> : : >freebsd-performance@freebsd.org mailing list :> : : >http://lists.freebsd.org/mailman/listinfo/freebsd-performance :> : : >To unsubscribe, send any mail to :> : "freebsd-performance-unsubscribe@freebsd.org" :> : : > :> : : > :> : : > :> : : :> : : _______________________________________________ :> : : freebsd-performance@freebsd.org mailing list :> : : http://lists.freebsd.org/mailman/listinfo/freebsd-performance :> : : To unsubscribe, send any mail to :> : "freebsd-performance-unsubscribe@freebsd.org" :> : _______________________________________________ :> : freebsd-performance@freebsd.org mailing list :> : http://lists.freebsd.org/mailman/listinfo/freebsd-performance :> : To unsubscribe, send any mail to :> "freebsd-performance-unsubscribe@freebsd.org" :> _______________________________________________ :> freebsd-performance@freebsd.org mailing list :> http://lists.freebsd.org/mailman/listinfo/freebsd-performance :> To unsubscribe, send any mail to :"freebsd-performance-unsubscribe@freebsd.org" : : :-- :------------------------------------------------------------------------ :Eric Anderson Sr. Systems Administrator Centaur Technology :I have seen the future and it is just like the present, only longer. :------------------------------------------------------------------------ :_______________________________________________ :freebsd-performance@freebsd.org mailing list :http://lists.freebsd.org/mailman/listinfo/freebsd-performance :To unsubscribe, send any mail to :"freebsd-performance-unsubscribe@freebsd.org" From owner-freebsd-performance@FreeBSD.ORG Wed Mar 9 22:51:55 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 592D616A4DF for ; Wed, 9 Mar 2005 22:51:55 +0000 (GMT) Received: from coverity.dreamhost.com (coverity.dreamhost.com [66.33.192.105]) by mx1.FreeBSD.org (Postfix) with ESMTP id 05D6943D2F for ; Wed, 9 Mar 2005 22:51:55 +0000 (GMT) (envelope-from phessler@coverity.com) Received: from leela.theapt.org (piro.theapt.org [208.201.244.164]) by coverity.dreamhost.com (Postfix) with ESMTP id 8D9339089D for ; Wed, 9 Mar 2005 14:51:54 -0800 (PST) Date: Wed, 9 Mar 2005 14:51:49 -0800 From: Peter Hessler To: freebsd-performance@freebsd.org Message-Id: <20050309145149.602e9ed2@leela.theapt.org> In-Reply-To: <422E5E78.3000703@centtech.com> References: <20050309005628.5C6D1908B8@coverity.dreamhost.com> <422E5E78.3000703@centtech.com> X-Mailer: Sylpheed version 0.9.9claws (GTK+ 1.2.10; powerpc-unknown-openbsd3.6) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Subject: Re: NFS small files X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Mar 2005 22:51:55 -0000 On Tue, 08 Mar 2005 20:24:56 -0600 Eric Anderson wrote: :Peter Hessler wrote: :> I'm setting up an NFS server to be used for compiling, and it seems that :> while speeds are acceptable for large files, small files take much :> longer than expected. :> :> Copying 10000 16K files (in a directory) takes 54seconds, while copying :> a 170M single file takes 5s. :> :> Are there any tricks for speeding up small file performance? I'm :> willing to give up large file performance. : :What mount options are you using on the client? : : nfsvers=3,tcp,noac,intr,lock,rsize=65536,wsize=65536 on linux and freebsd clients. No speed difference between udp and tcp. : :-- :------------------------------------------------------------------------ :Eric Anderson Sr. Systems Administrator Centaur Technology :I have seen the future and it is just like the present, only longer. :------------------------------------------------------------------------ :_______________________________________________ :freebsd-performance@freebsd.org mailing list :http://lists.freebsd.org/mailman/listinfo/freebsd-performance :To unsubscribe, send any mail to :"freebsd-performance-unsubscribe@freebsd.org" From owner-freebsd-performance@FreeBSD.ORG Wed Mar 9 23:31:49 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D375716A4CE for ; Wed, 9 Mar 2005 23:31:49 +0000 (GMT) Received: from smtp801.mail.sc5.yahoo.com (smtp801.mail.sc5.yahoo.com [66.163.168.180]) by mx1.FreeBSD.org (Postfix) with SMTP id AF78243D58 for ; Wed, 9 Mar 2005 23:31:49 +0000 (GMT) (envelope-from noackjr@alumni.rice.edu) Received: from unknown (HELO optimator.noacks.org) (noacks@swbell.net@70.240.225.210 with login) by smtp801.mail.sc5.yahoo.com with SMTP; 9 Mar 2005 23:31:48 -0000 Received: from localhost (localhost [127.0.0.1]) by optimator.noacks.org (Postfix) with ESMTP id C0C8C61AD; Wed, 9 Mar 2005 17:31:47 -0600 (CST) Received: from optimator.noacks.org ([127.0.0.1]) by localhost (optimator.noacks.org [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 07368-04; Wed, 9 Mar 2005 17:31:45 -0600 (CST) Received: from [127.0.0.1] (optimator [192.168.1.11]) by optimator.noacks.org (Postfix) with ESMTP id 3301A60CF; Wed, 9 Mar 2005 17:31:45 -0600 (CST) Message-ID: <422F8753.50200@alumni.rice.edu> Date: Wed, 09 Mar 2005 17:31:31 -0600 From: Jon Noack User-Agent: Mozilla Thunderbird 1.0 (Windows/20041206) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Peter Hessler References: <20050309005628.5C6D1908B8@coverity.dreamhost.com> <422E5E78.3000703@centtech.com> <20050309145149.602e9ed2@leela.theapt.org> In-Reply-To: <20050309145149.602e9ed2@leela.theapt.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: amavisd-new at noacks.org cc: freebsd-performance@freebsd.org Subject: Re: NFS small files X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: noackjr@alumni.rice.edu List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Mar 2005 23:31:49 -0000 Peter Hessler wrote: > On Tue, 08 Mar 2005 20:24:56 -0600 > Eric Anderson wrote: > :Peter Hessler wrote: > :> I'm setting up an NFS server to be used for compiling, and it seems that > :> while speeds are acceptable for large files, small files take much > :> longer than expected. > :> > :> Copying 10000 16K files (in a directory) takes 54seconds, while copying > :> a 170M single file takes 5s. > :> > :> Are there any tricks for speeding up small file performance? I'm > :> willing to give up large file performance. > : > :What mount options are you using on the client? > > nfsvers=3,tcp,noac,intr,lock,rsize=65536,wsize=65536 on linux and freebsd > clients. No speed difference between udp and tcp. I don't see some of those in mount_nfs(8): nfsvers=3 -> nfsv3 noac -> ? (does this option exist on FreeBSD?) lock -> lockd (which is the default) rsize=65536 -> -r=65536 wsize=65536 -> -w=65536 It is possible the documentation is not complete and those options are perfectly valid. If so, I apologize. Jon From owner-freebsd-performance@FreeBSD.ORG Thu Mar 10 00:16:19 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 51F5416A4CE for ; Thu, 10 Mar 2005 00:16:19 +0000 (GMT) Received: from coverity.dreamhost.com (coverity.dreamhost.com [66.33.192.105]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2F9EC43D49 for ; Thu, 10 Mar 2005 00:16:19 +0000 (GMT) (envelope-from phessler@coverity.com) Received: from leela.theapt.org (piro.theapt.org [208.201.244.164]) by coverity.dreamhost.com (Postfix) with ESMTP id CCFD89089D for ; Wed, 9 Mar 2005 16:16:18 -0800 (PST) Date: Wed, 9 Mar 2005 16:14:31 -0800 From: Peter Hessler To: freebsd-performance@freebsd.org Message-Id: <20050309161431.3a97c0dd@leela.theapt.org> In-Reply-To: <422F8753.50200@alumni.rice.edu> References: <20050309005628.5C6D1908B8@coverity.dreamhost.com> <422E5E78.3000703@centtech.com> <20050309145149.602e9ed2@leela.theapt.org> <422F8753.50200@alumni.rice.edu> X-Mailer: Sylpheed version 0.9.9claws (GTK+ 1.2.10; powerpc-unknown-openbsd3.6) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Subject: Re: NFS small files X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Mar 2005 00:16:19 -0000 On Wed, 09 Mar 2005 17:31:31 -0600 Jon Noack wrote: :Peter Hessler wrote: :> On Tue, 08 Mar 2005 20:24:56 -0600 :> Eric Anderson wrote: :> :Peter Hessler wrote: :> :> I'm setting up an NFS server to be used for compiling, and it seems :that> :> while speeds are acceptable for large files, small files take much :> :> longer than expected. :> :> :> :> Copying 10000 16K files (in a directory) takes 54seconds, while :copying> :> a 170M single file takes 5s. :> :> :> :> Are there any tricks for speeding up small file performance? I'm :> :> willing to give up large file performance. :> : :> :What mount options are you using on the client? :> :> nfsvers=3,tcp,noac,intr,lock,rsize=65536,wsize=65536 on linux and freebsd :> clients. No speed difference between udp and tcp. : :I don't see some of those in mount_nfs(8): :nfsvers=3 -> nfsv3 :noac -> ? (does this option exist on FreeBSD?) :lock -> lockd (which is the default) :rsize=65536 -> -r=65536 :wsize=65536 -> -w=65536 : :It is possible the documentation is not complete and those options are :perfectly valid. If so, I apologize. : Yea, sorry about that. I did a cut-n-paste from the machine I was activly working on (Linux), but the FreeBSD (and other clients) had the appropriate syntax for their systems. :Jon From owner-freebsd-performance@FreeBSD.ORG Thu Mar 10 21:42:26 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id CDCE216A4CE for ; Thu, 10 Mar 2005 21:42:26 +0000 (GMT) Received: from cyrus.watson.org (cyrus.watson.org [204.156.12.53]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7A5AF43D54 for ; Thu, 10 Mar 2005 21:42:26 +0000 (GMT) (envelope-from robert@fledge.watson.org) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by cyrus.watson.org (Postfix) with SMTP id CB87746B8C; Thu, 10 Mar 2005 16:42:25 -0500 (EST) Date: Thu, 10 Mar 2005 21:40:04 +0000 (GMT) From: Robert Watson X-Sender: robert@fledge.watson.org To: ISP Informatique In-Reply-To: <422ECAF7.4050407@ispinfo.fr> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=ISO-8859-1 Content-Transfer-Encoding: QUOTED-PRINTABLE cc: freebsd-performance@freebsd.org Subject: Re: FreeBSD 5.3 et multiprocessor X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Mar 2005 21:42:26 -0000 On Wed, 9 Mar 2005, ISP Informatique wrote: > Hello, I recently have just passed a server of FreeBSD 4.2 to FreeBSD > 5.3. The nasty surprise being which I note a notable reduction in the > performances. The machine is a HP LT6000r, quadri Xeon 700 with 2Go of > RAM. >=20 > The results are appreciably the same ones with a kernel including or not > "options SMP". Could you tell us a bit more about what kind of performance difference you're seeing? Specifically, what sort of workload? There are a lot of changes between 4.x and 5.x -- SMP model changes, threading changes, file system changes, device driver changes, etc. Trying to figure out what's going on will require narrowing down a bit what the variables are. Thanks, Robert N M Watson >=20 > Did I miss some thing? or perhaps did this already arrive at others?=20 > In particular, I has just crossed this in > http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-co= nfig.html=20 >=20 >=20 > > > > options SCHED_4BSD # 4BSD scheduler > > =20 > > > > The traditional scheduler for FreeBSD. Depending on your system's=20 > > workload, you may gain performance by using the new ULE scheduler for= =20 > > FreeBSD that has been designed specially for SMP, but works just fine= =20 > > on UP systems too. If you wish to try it out, replace SCHED_4BSD with= =20 > > SCHED_ULE in your configuration file. >=20 > Did you already test?=20 >=20 > By advance, thank you, >=20 > --=20 > Hubert Adgi=E9 >=20 > ISP Informatique > www.ispinfo.fr > 0890 710 147 >=20 >=20 > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd= =2Eorg" >=20 From owner-freebsd-performance@FreeBSD.ORG Fri Mar 11 02:00:50 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id F15E316A4CE for ; Fri, 11 Mar 2005 02:00:49 +0000 (GMT) Received: from outgoing.redshift.com (outgoing.redshift.com [207.177.231.8]) by mx1.FreeBSD.org (Postfix) with ESMTP id B081443D39 for ; Fri, 11 Mar 2005 02:00:49 +0000 (GMT) (envelope-from ray@redshift.com) Received: from workstation (216-228-19-21.dsl.redshift.com [216.228.19.21]) by outgoing.redshift.com (Postfix) with SMTP id 4AB599708F for ; Thu, 10 Mar 2005 18:00:49 -0800 (PST) Message-Id: <3.0.1.32.20050310180051.00a7e908@pop.redshift.com> X-Mailer: na X-Sender: redshift.com Date: Thu, 10 Mar 2005 18:00:51 -0800 To: freebsd-performance@freebsd.org From: ray@redshift.com Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Subject: performance modifications X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Mar 2005 02:00:50 -0000 Hi, I'm wondering if anyone on the list has a good source for the major sysctl settings and/or kernel settings that can be modified in order to bring up the performance level on a FreeBSD 5.3 machine that is used with apache under heavy load. I've done all the common stuff: recompiled the kernel and stripped out unused drivers strip -s kernel file maxusers -> 256 net.inet.tcp.recvspace=65535 net.inet.tcp.sendspace=65535 kern.ipc.somaxconn=1024 But I'm wondering if there are other areas (such as hard drive r/w buffers in the kernel and/or memory allocation) that can be readily increased that will have a positive impact which I am over looking. Any other areas I should be looking at and/or any ideas? Thanks! Ray From owner-freebsd-performance@FreeBSD.ORG Fri Mar 11 02:58:57 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BF4EB16A4CE for ; Fri, 11 Mar 2005 02:58:57 +0000 (GMT) Received: from rproxy.gmail.com (rproxy.gmail.com [64.233.170.196]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2463E43D60 for ; Fri, 11 Mar 2005 02:58:57 +0000 (GMT) (envelope-from joseph.koshy@gmail.com) Received: by rproxy.gmail.com with SMTP id j1so646830rnf for ; Thu, 10 Mar 2005 18:58:56 -0800 (PST) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:reply-to:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:references; b=YaLGAxxFPR/VNkXmKYc4WcODaa3apD2M9Crhzbs+ThIVw7vbvFu44WlCkBvhINyKtCOpUMeukg1YMnmWf4N9YzArDlFxj+RVuRvliyYcWaOks6QJAfnmyK/GPBUbelmhu2XVRgm8Rpw798+hIzUBAXq6flDTC+ykmVCbDXcrH2I= Received: by 10.38.78.51 with SMTP id a51mr749898rnb; Thu, 10 Mar 2005 18:58:56 -0800 (PST) Received: by 10.38.209.22 with HTTP; Thu, 10 Mar 2005 18:58:56 -0800 (PST) Message-ID: <84dead720503101858890b444@mail.gmail.com> Date: Fri, 11 Mar 2005 02:58:56 +0000 From: Joseph Koshy To: ISP Informatique In-Reply-To: <422ECAF7.4050407@ispinfo.fr> Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit References: <422ECAF7.4050407@ispinfo.fr> cc: freebsd-performance@freebsd.org Subject: Re: FreeBSD 5.3 et multiprocessor X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: Joseph Koshy List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Mar 2005 02:58:58 -0000 > The nasty surprise being which I note a notable reduction in > The machine is a HP LT6000r, quadri Xeon 700 with 2Go of RAM. Please post 'dmesg' output from 4.2 and 5.3. What workload do you see the slowdown on? How are you measuring performance? We cannot help you unless we have more information about the problem. -- FreeBSD Volunteer, http://people.freebsd.org/~jkoshy From owner-freebsd-performance@FreeBSD.ORG Fri Mar 11 03:23:02 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5001C16A4CE for ; Fri, 11 Mar 2005 03:23:02 +0000 (GMT) Received: from rproxy.gmail.com (rproxy.gmail.com [64.233.170.204]) by mx1.FreeBSD.org (Postfix) with ESMTP id DA17E43D58 for ; Fri, 11 Mar 2005 03:23:01 +0000 (GMT) (envelope-from joseph.koshy@gmail.com) Received: by rproxy.gmail.com with SMTP id j1so650787rnf for ; Thu, 10 Mar 2005 19:23:01 -0800 (PST) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:reply-to:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:references; b=r49VT+vRhM1iBUEWdSFU6rZR7uS5plwHCX/+PR1Gs0zgLj/H7o8gqtQriNAD5oHv1pK11vu5/LvfC65KrMndHSWGQYOQgmpE39+Tf02y/x4ebN4TS36+qoZWuYxN6sHRDWHIKG7oA6FoUT8XrvsnbAZjMkrRFuJlfwXx+evtvwA= Received: by 10.38.89.36 with SMTP id m36mr2514583rnb; Thu, 10 Mar 2005 19:23:01 -0800 (PST) Received: by 10.38.209.22 with HTTP; Thu, 10 Mar 2005 19:23:01 -0800 (PST) Message-ID: <84dead7205031019235f37f5cc@mail.gmail.com> Date: Fri, 11 Mar 2005 03:23:01 +0000 From: Joseph Koshy To: "ray@redshift.com" In-Reply-To: <3.0.1.32.20050310180051.00a7e908@pop.redshift.com> Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit References: <3.0.1.32.20050310180051.00a7e908@pop.redshift.com> cc: freebsd-performance@freebsd.org Subject: Re: performance modifications X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: Joseph Koshy List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Mar 2005 03:23:02 -0000 > I'm wondering if anyone on the list has a good source for > the major sysctl settings and/or kernel settings that can be > modified in order to bring up the performance level on a > FreeBSD 5.3 machine that is used with apache under heavy > load. There is the tuning(7) manual page. There are better web servers than Apache for demanding loads; ones that used a non-forking, event-driven I/O model. Aolserver and thttpd come to mind. Yaoping Ruan and Vivek Pai from Princeton have reported excellent results with their "Flash" [1] web server. While many of their recommended changes to FreeBSD have been folded into the base source, I'm not sure how many popular web-servers are using these speedups. [1] Making the "Box" Transparent: System Call Performance as a First-class Result Yaoping Ruan, Vivek Pai http://www.cs.princeton.edu/~yruan/debox/debox.pdf -- FreeBSD Volunteer, http://people.freebsd.org/~jkoshy From owner-freebsd-performance@FreeBSD.ORG Fri Mar 11 03:30:40 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0849B16A4E5 for ; Fri, 11 Mar 2005 03:30:40 +0000 (GMT) Received: from outgoing.redshift.com (outgoing.redshift.com [207.177.231.8]) by mx1.FreeBSD.org (Postfix) with ESMTP id C8D7143D31 for ; Fri, 11 Mar 2005 03:30:39 +0000 (GMT) (envelope-from ray@redshift.com) Received: from workstation (216-228-19-21.dsl.redshift.com [216.228.19.21]) by outgoing.redshift.com (Postfix) with SMTP id C5A8196FB1; Thu, 10 Mar 2005 19:30:15 -0800 (PST) Message-Id: <3.0.1.32.20050310193015.00a7e908@pop.redshift.com> X-Mailer: na X-Sender: redshift.com Date: Thu, 10 Mar 2005 19:30:15 -0800 To: Joseph Koshy From: ray@redshift.com In-Reply-To: <84dead7205031019235f37f5cc@mail.gmail.com> References: <3.0.1.32.20050310180051.00a7e908@pop.redshift.com> <3.0.1.32.20050310180051.00a7e908@pop.redshift.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" cc: freebsd-performance@freebsd.org Subject: Re: performance modifications X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Mar 2005 03:30:40 -0000 | > I'm wondering if anyone on the list has a good source for | > the major sysctl settings and/or kernel settings that can be | > modified in order to bring up the performance level on a | > FreeBSD 5.3 machine that is used with apache under heavy | > load. | | There is the tuning(7) manual page. Yes, have read those from start to finish. Just wondering if there are other areas to look into that may not be covered there. | There are better web servers than Apache for demanding loads; | ones that used a non-forking, event-driven I/O model. | Aolserver and thttpd come to mind. I tried thttpd a number of years ago, but due to its limited cgi support (at least at the time) I didn't do too much with it. It did seem fast, however. | Yaoping Ruan and Vivek Pai from Princeton have reported | excellent results with their "Flash" [1] web server. While | many of their recommended changes to FreeBSD have been folded | into the base source, I'm not sure how many popular web-servers | are using these speedups. | | [1] Making the "Box" Transparent: System Call Performance as a | First-class Result | Yaoping Ruan, Vivek Pai | http://www.cs.princeton.edu/~yruan/debox/debox.pdf Okay, great, thanks. I'll check into that area. My biggest problem right now is that PHP brings down the speed of everything. I may have to go back to Perl and use mod_perl or look into some other alternatives. The main thing I wanted to do was get the OS running at it's best, so the benchmarks were not being affected by some area of FreeBSD. Anyway, thanks very much for the info! Ray From owner-freebsd-performance@FreeBSD.ORG Fri Mar 11 08:10:52 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C7DEA16A4CE for ; Fri, 11 Mar 2005 08:10:52 +0000 (GMT) Received: from web26809.mail.ukl.yahoo.com (web26809.mail.ukl.yahoo.com [217.146.176.85]) by mx1.FreeBSD.org (Postfix) with SMTP id D42AB43D39 for ; Fri, 11 Mar 2005 08:10:51 +0000 (GMT) (envelope-from cguttesen@yahoo.dk) Received: (qmail 68376 invoked by uid 60001); 11 Mar 2005 08:10:51 -0000 Message-ID: <20050311081051.68374.qmail@web26809.mail.ukl.yahoo.com> Received: from [194.248.174.58] by web26809.mail.ukl.yahoo.com via HTTP; Fri, 11 Mar 2005 09:10:50 CET Date: Fri, 11 Mar 2005 09:10:50 +0100 (CET) From: Claus Guttesen To: ray@redshift.com, freebsd-performance@freebsd.org In-Reply-To: 6667 MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Subject: Re: performance modifications X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Mar 2005 08:10:52 -0000 > I'm wondering if anyone on the list has a good > source for the major sysctl > settings and/or kernel settings that can be modified > in order to bring up the > performance level on a FreeBSD 5.3 machine that is > used with apache under heavy > load. > > I've done all the common stuff: > > recompiled the kernel and stripped out unused > drivers Did you try options ACCEPT_FILTER_DATA options ACCEPT_FILTER_HTTP in your kernel? These options makes FreeBSD do the 3way-handshake rather than apache. The usual KeepAlive Off in /usr/local/etc/apache/httpd.conf and MaxClients xyz in same file. Max-clients is computed by 'avail. ram / size of one process', where you need to leave some room for kernel, buffers etc. in avail. ram. regards Claus From owner-freebsd-performance@FreeBSD.ORG Fri Mar 11 08:30:28 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1F8B816A4CE for ; Fri, 11 Mar 2005 08:30:28 +0000 (GMT) Received: from silver.he.iki.fi (helenius.fi [193.64.42.241]) by mx1.FreeBSD.org (Postfix) with ESMTP id 87F5743D2F for ; Fri, 11 Mar 2005 08:30:26 +0000 (GMT) (envelope-from pete@he.iki.fi) Received: from [193.64.42.134] (h86.vuokselantie10.fi [193.64.42.134]) by silver.he.iki.fi (8.13.1/8.11.4) with ESMTP id j2B8UOdK021574; Fri, 11 Mar 2005 10:30:24 +0200 (EET) (envelope-from pete@he.iki.fi) Message-ID: <4231572C.2030407@he.iki.fi> Date: Fri, 11 Mar 2005 10:30:36 +0200 From: Petri Helenius User-Agent: Mozilla Thunderbird 1.0 (Windows/20041206) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Claus Guttesen References: <20050311081051.68374.qmail@web26809.mail.ukl.yahoo.com> In-Reply-To: <20050311081051.68374.qmail@web26809.mail.ukl.yahoo.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit cc: ray@redshift.com cc: freebsd-performance@freebsd.org Subject: Re: performance modifications X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Mar 2005 08:30:28 -0000 Claus Guttesen wrote: >> I'm wondering if anyone on the list has a good >>source for the major sysctl >>settings and/or kernel settings that can be modified >>in order to bring up the >>performance level on a FreeBSD 5.3 machine that is >>used with apache under heavy >>load. >> >>I've done all the common stuff: >> >>recompiled the kernel and stripped out unused >>drivers >> >> > >Did you try > >options ACCEPT_FILTER_DATA >options ACCEPT_FILTER_HTTP > >in your kernel? These options makes FreeBSD do the >3way-handshake rather than apache. > > > Does this need any modifications to apache? (installed from the port) Pete >The usual > >KeepAlive Off > >in /usr/local/etc/apache/httpd.conf and > >MaxClients xyz > >in same file. Max-clients is computed by 'avail. ram / >size of one process', where you need to leave some >room for kernel, buffers etc. in avail. ram. > >regards >Claus > >_______________________________________________ >freebsd-performance@freebsd.org mailing list >http://lists.freebsd.org/mailman/listinfo/freebsd-performance >To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" > > From owner-freebsd-performance@FreeBSD.ORG Fri Mar 11 08:34:15 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2732A16A4CE for ; Fri, 11 Mar 2005 08:34:15 +0000 (GMT) Received: from web26810.mail.ukl.yahoo.com (web26810.mail.ukl.yahoo.com [217.146.176.86]) by mx1.FreeBSD.org (Postfix) with SMTP id 9DBD443D48 for ; Fri, 11 Mar 2005 08:34:14 +0000 (GMT) (envelope-from cguttesen@yahoo.dk) Received: (qmail 62186 invoked by uid 60001); 11 Mar 2005 08:34:13 -0000 Message-ID: <20050311083413.62184.qmail@web26810.mail.ukl.yahoo.com> Received: from [194.248.174.58] by web26810.mail.ukl.yahoo.com via HTTP; Fri, 11 Mar 2005 09:34:13 CET Date: Fri, 11 Mar 2005 09:34:13 +0100 (CET) From: Claus Guttesen To: Petri Helenius In-Reply-To: 6667 MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit cc: ray@redshift.com cc: freebsd-performance@freebsd.org Subject: Re: performance modifications X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Mar 2005 08:34:15 -0000 > >Did you try > > > >options ACCEPT_FILTER_DATA > >options ACCEPT_FILTER_HTTP > > > Does this need any modifications to apache? > (installed from the port) > > Pete No, this kernel-change is (fortunately) completely transparent in relation to apache. regards Claus From owner-freebsd-performance@FreeBSD.ORG Fri Mar 11 08:50:55 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4CE1C16A4CE for ; Fri, 11 Mar 2005 08:50:55 +0000 (GMT) Received: from silver.he.iki.fi (helenius.fi [193.64.42.241]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7819143D2F for ; Fri, 11 Mar 2005 08:50:54 +0000 (GMT) (envelope-from pete@he.iki.fi) Received: from [193.64.42.134] (h86.vuokselantie10.fi [193.64.42.134]) by silver.he.iki.fi (8.13.1/8.11.4) with ESMTP id j2B8oraA021640; Fri, 11 Mar 2005 10:50:53 +0200 (EET) (envelope-from pete@he.iki.fi) Message-ID: <42315BF9.4020005@he.iki.fi> Date: Fri, 11 Mar 2005 10:51:05 +0200 From: Petri Helenius User-Agent: Mozilla Thunderbird 1.0 (Windows/20041206) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Claus Guttesen References: <20050311083413.62184.qmail@web26810.mail.ukl.yahoo.com> In-Reply-To: <20050311083413.62184.qmail@web26810.mail.ukl.yahoo.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit cc: ray@redshift.com cc: freebsd-performance@freebsd.org Subject: Re: performance modifications X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Mar 2005 08:50:55 -0000 Claus Guttesen wrote: > >No, this kernel-change is (fortunately) completely >transparent in relation to apache. > > Ok, that was not the impression I got from accf_http man page. Pete From owner-freebsd-performance@FreeBSD.ORG Fri Mar 11 09:21:52 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0D54D16A4CE for ; Fri, 11 Mar 2005 09:21:52 +0000 (GMT) Received: from web26808.mail.ukl.yahoo.com (web26808.mail.ukl.yahoo.com [217.146.176.84]) by mx1.FreeBSD.org (Postfix) with SMTP id 4300143D5C for ; Fri, 11 Mar 2005 09:21:51 +0000 (GMT) (envelope-from cguttesen@yahoo.dk) Received: (qmail 88047 invoked by uid 60001); 11 Mar 2005 09:21:50 -0000 Message-ID: <20050311092150.88045.qmail@web26808.mail.ukl.yahoo.com> Received: from [194.248.174.58] by web26808.mail.ukl.yahoo.com via HTTP; Fri, 11 Mar 2005 10:21:49 CET Date: Fri, 11 Mar 2005 10:21:49 +0100 (CET) From: Claus Guttesen To: Petri Helenius In-Reply-To: 6667 MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit cc: ray@redshift.com cc: freebsd-performance@freebsd.org Subject: Re: performance modifications X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Mar 2005 09:21:52 -0000 > >No, this kernel-change is (fortunately) completely > >transparent in relation to apache. > > > Ok, that was not the impression I got from accf_http > man page. I checked apache.org just to be shure :-) http://unix.derkeiler.com/Mailing-Lists/FreeBSD/performance/2003-07/0071.html gives some hints, and http://httpd.apache.org/docs/mod/core.html#acceptfilter says that SO_ACCEPTFILTER is enabled by default. So when options ACCEPT_FILTER_DATA options ACCEPT_FILTER_HTTP is compiled into the kernel or accf_http is loaded, apache "should" utilize this option. regards Claus From owner-freebsd-performance@FreeBSD.ORG Fri Mar 11 11:10:18 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5478816A4D0 for ; Fri, 11 Mar 2005 11:10:18 +0000 (GMT) Received: from quentin.asdf.dk (quentin.asdf.dk [217.116.240.101]) by mx1.FreeBSD.org (Postfix) with ESMTP id C091443D53 for ; Fri, 11 Mar 2005 11:10:17 +0000 (GMT) (envelope-from hroi@ngdc.net) Received: from localhost (localhost [127.0.0.1]) by quentin.asdf.dk (Postfix) with ESMTP id 4E1FDB843 for ; Fri, 11 Mar 2005 11:11:05 +0000 (UTC) Received: from quentin.asdf.dk ([127.0.0.1]) by localhost (quentin.asdf.dk [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 86169-02 for ; Fri, 11 Mar 2005 11:10:58 +0000 (UTC) Received: from [10.0.0.45] (dc0.fw.netgroup.dk [217.116.225.1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by quentin.asdf.dk (Postfix) with ESMTP id 44663B83F for ; Fri, 11 Mar 2005 11:10:58 +0000 (UTC) Message-ID: <42317C90.3080005@ngdc.net> Date: Fri, 11 Mar 2005 12:10:08 +0100 From: Hroi Sigurdsson Organization: NetGroup Data Center A/S User-Agent: Mozilla Thunderbird 1.0+ (Windows/20050221) MIME-Version: 1.0 To: freebsd-performance@freebsd.org References: <20050311083413.62184.qmail@web26810.mail.ukl.yahoo.com> <42315BF9.4020005@he.iki.fi> In-Reply-To: <42315BF9.4020005@he.iki.fi> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit X-Virus-Scanned: amavisd-new at quentin.asdf.dk Subject: Re: performance modifications X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Mar 2005 11:10:18 -0000 Petri Helenius wrote: >> No, this kernel-change is (fortunately) completely >> transparent in relation to apache. >> >> > Ok, that was not the impression I got from accf_http man page. A setsockopt call is needed to use the filters. Apache2 support for the "httpready" filter was added in 2.0.51. -- Hroi Sigurdsson · NetGroup DataCenter A/S St. Kongensgade 40H · DK-1264 Copenhagen K, Denmark Phone: +45 3370 1544 · Fax: +45 7025 2687 From owner-freebsd-performance@FreeBSD.ORG Sat Mar 12 15:44:07 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 40F5E16A4CE for ; Sat, 12 Mar 2005 15:44:07 +0000 (GMT) Received: from cyrus.watson.org (cyrus.watson.org [204.156.12.53]) by mx1.FreeBSD.org (Postfix) with ESMTP id C54AA43D31 for ; Sat, 12 Mar 2005 15:44:06 +0000 (GMT) (envelope-from robert@fledge.watson.org) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by cyrus.watson.org (Postfix) with SMTP id 6499D46B9C; Sat, 12 Mar 2005 10:44:06 -0500 (EST) Date: Sat, 12 Mar 2005 15:41:40 +0000 (GMT) From: Robert Watson X-Sender: robert@fledge.watson.org To: ray@redshift.com In-Reply-To: <3.0.1.32.20050310180051.00a7e908@pop.redshift.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-performance@freebsd.org Subject: Re: performance modifications X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 Mar 2005 15:44:07 -0000 On Thu, 10 Mar 2005 ray@redshift.com wrote: > I'm wondering if anyone on the list has a good source for the major > sysctl settings and/or kernel settings that can be modified in order to > bring up the performance level on a FreeBSD 5.3 machine that is used > with apache under heavy load. The first thing you want to do is to try to identify what factors are bounding the performance of your box. A good way to start is by using the "systat -vmstat 1" command, which will show you statistics snapshots on various kernel performance characteristics. Among other things, this will tell you where the majority of CPU time is being spent (system, interrupt, user process, idle, etc), as well as interrupt loads, I/O loads, whether the system is paging/swapping, and so on. This might help point at whether you're hitting limits on processor or memory resources, disk resources, and so on. A few snapshots of that would be helpful. Robert N M Watson > > I've done all the common stuff: > > recompiled the kernel and stripped out unused drivers > strip -s kernel file > maxusers -> 256 > net.inet.tcp.recvspace=65535 > net.inet.tcp.sendspace=65535 > kern.ipc.somaxconn=1024 > > But I'm wondering if there are other areas (such as hard drive r/w buffers in > the kernel and/or memory allocation) that can be readily increased that will > have a positive impact which I am over looking. > > Any other areas I should be looking at and/or any ideas? > > Thanks! > > Ray > > > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" >