From owner-freebsd-performance@FreeBSD.ORG Mon Jun 9 12:34:51 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B442737B401 for ; Mon, 9 Jun 2003 12:34:51 -0700 (PDT) Received: from otter3.centtech.com (moat3.centtech.com [207.200.51.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id ED3D743FAF for ; Mon, 9 Jun 2003 12:34:50 -0700 (PDT) (envelope-from anderson@centtech.com) Received: from centtech.com (electron.centtech.com [204.177.173.173]) by otter3.centtech.com (8.12.3/8.12.3) with ESMTP id h59JYo56037292 for ; Mon, 9 Jun 2003 14:34:50 -0500 (CDT) (envelope-from anderson@centtech.com) Message-ID: <3EE4E156.6030603@centtech.com> Date: Mon, 09 Jun 2003 14:34:46 -0500 From: Eric Anderson User-Agent: Mozilla/5.0 (X11; U; Linux i386; en-US; rv:1.0.1) Gecko/20020823 Netscape/7.0 X-Accept-Language: en-us, en MIME-Version: 1.0 To: freebsd-performance@freebsd.org Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Subject: Slow disk write speeds over network X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Jun 2003 19:34:52 -0000 Ok, I have a file server (NFS) running FreeBSD 4.8-RC1, which is having incredibly slow disk write speeds. Locally, doing something like: dd if=/dev/zero of=/partition/testfile Shows 14MB/s - which is what I expect.. However, ftping a file or cp'ing over NFS shows speeds less than 1MB/s - which is not what I would expect - it's got gigabit ethernet, and has very fast read times (via ftp or nfs) over the network. After looking around, I noticed (when doing a systat -vmstat), that the vnode information looked a little strange.. like so: 54120 desiredvnodes 49163 numvnodes 35903 freevnodes Shouldn't I have a higher "numvnodes" than "desiredvnodes"? What else am I missing here? I have built many FreeBSD NFS file servers, and this is the first time I've had a slow one. I suspect maybe a sysctl tweak I bumped upon building the machine, or possibly something I didn't tweak. Any hints? Eric -- ------------------------------------------------------------------ Eric Anderson Systems Administrator Centaur Technology Attitudes are contagious, is yours worth catching? ------------------------------------------------------------------ From owner-freebsd-performance@FreeBSD.ORG Mon Jun 9 12:47:27 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 042F437B401 for ; Mon, 9 Jun 2003 12:47:27 -0700 (PDT) Received: from perrin.int.nxad.com (internal.ext.nxad.com [69.1.70.251]) by mx1.FreeBSD.org (Postfix) with ESMTP id 672E343FFB for ; Mon, 9 Jun 2003 12:47:06 -0700 (PDT) (envelope-from sean@nxad.com) Received: by perrin.int.nxad.com (Postfix, from userid 1001) id D33D220F01; Mon, 9 Jun 2003 12:47:05 -0700 (PDT) Date: Mon, 9 Jun 2003 12:47:05 -0700 From: Sean Chittenden To: Eric Anderson Message-ID: <20030609194705.GJ65470@perrin.int.nxad.com> References: <3EE4E156.6030603@centtech.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <3EE4E156.6030603@centtech.com> X-PGP-Key: finger seanc@FreeBSD.org X-PGP-Fingerprint: 3849 3760 1AFE 7B17 11A0 83A6 DD99 E31F BC84 B341 X-Web-Homepage: http://sean.chittenden.org/ User-Agent: Mutt/1.5.4i cc: freebsd-performance@freebsd.org Subject: Re: Slow disk write speeds over network X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Jun 2003 19:47:27 -0000 > Ok, I have a file server (NFS) running FreeBSD 4.8-RC1, which is having > incredibly slow disk write speeds. Locally, doing something like: > dd if=/dev/zero of=/partition/testfile > Shows 14MB/s - which is what I expect.. Check your cabling... you may have a bad pair in your cable on your send wires, or at least a crummy pair with high resistance. See if netstat -i has any errors listed. A change in performance that dramatic and only in one direction is disturbing and doesn't sound like a kernel or software issue, though I could be quite wrong. -sc -- Sean Chittenden From owner-freebsd-performance@FreeBSD.ORG Mon Jun 9 12:49:37 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6AF6C37B401 for ; Mon, 9 Jun 2003 12:49:37 -0700 (PDT) Received: from otter3.centtech.com (moat3.centtech.com [207.200.51.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id B5AC343FBD for ; Mon, 9 Jun 2003 12:49:36 -0700 (PDT) (envelope-from anderson@centtech.com) Received: from centtech.com (electron.centtech.com [204.177.173.173]) by otter3.centtech.com (8.12.3/8.12.3) with ESMTP id h59JnY56038409; Mon, 9 Jun 2003 14:49:34 -0500 (CDT) (envelope-from anderson@centtech.com) Message-ID: <3EE4E4CB.4060108@centtech.com> Date: Mon, 09 Jun 2003 14:49:31 -0500 From: Eric Anderson User-Agent: Mozilla/5.0 (X11; U; Linux i386; en-US; rv:1.0.1) Gecko/20020823 Netscape/7.0 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Sean Chittenden References: <3EE4E156.6030603@centtech.com> <20030609194705.GJ65470@perrin.int.nxad.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit cc: freebsd-performance@freebsd.org Subject: Re: Slow disk write speeds over network X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Jun 2003 19:49:37 -0000 Sean Chittenden wrote: >>Ok, I have a file server (NFS) running FreeBSD 4.8-RC1, which is having >>incredibly slow disk write speeds. Locally, doing something like: >>dd if=/dev/zero of=/partition/testfile >>Shows 14MB/s - which is what I expect.. > > > Check your cabling... you may have a bad pair in your cable on your > send wires, or at least a crummy pair with high resistance. See if > netstat -i has any errors listed. A change in performance that > dramatic and only in one direction is disturbing and doesn't sound > like a kernel or software issue, though I could be quite wrong. Well, I don't believe that it is a problem - I have 4 gigabit network cards in the machine (connected to 4 different networks), and all act this same way - all use different types of cables too. I've also tried several different client machines. Eric -- ------------------------------------------------------------------ Eric Anderson Systems Administrator Centaur Technology Attitudes are contagious, is yours worth catching? ------------------------------------------------------------------ From owner-freebsd-performance@FreeBSD.ORG Mon Jun 9 13:37:42 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E285A37B405 for ; Mon, 9 Jun 2003 13:37:42 -0700 (PDT) Received: from web14911.mail.yahoo.com (web14911.mail.yahoo.com [216.136.225.249]) by mx1.FreeBSD.org (Postfix) with SMTP id 90A7F43FD7 for ; Mon, 9 Jun 2003 13:37:42 -0700 (PDT) (envelope-from nirv199@yahoo.com) Message-ID: <20030609203742.30333.qmail@web14911.mail.yahoo.com> Received: from [200.163.192.181] by web14911.mail.yahoo.com via HTTP; Mon, 09 Jun 2003 13:37:42 PDT Date: Mon, 9 Jun 2003 13:37:42 -0700 (PDT) From: Paulo Roberto To: freebsd-performance@freebsd.org In-Reply-To: <3EE4E4CB.4060108@centtech.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Subject: Re: Slow disk write speeds over network X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Jun 2003 20:37:43 -0000 > Sean Chittenden wrote: > >>Ok, I have a file server (NFS) running FreeBSD 4.8-RC1, which is > having > >>incredibly slow disk write speeds. Locally, doing something like: Just an idea (I might be saying something pretty stupid...): Have you tried increasing the network buffer size?? Paulo Roberto __________________________________ Do you Yahoo!? Yahoo! Calendar - Free online calendar with sync to Outlook(TM). http://calendar.yahoo.com From owner-freebsd-performance@FreeBSD.ORG Mon Jun 9 13:42:07 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9973D37B401 for ; Mon, 9 Jun 2003 13:42:07 -0700 (PDT) Received: from otter3.centtech.com (moat3.centtech.com [207.200.51.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id CC4C543F75 for ; Mon, 9 Jun 2003 13:42:06 -0700 (PDT) (envelope-from anderson@centtech.com) Received: from centtech.com (electron.centtech.com [204.177.173.173]) by otter3.centtech.com (8.12.3/8.12.3) with ESMTP id h59Kg656043339 for ; Mon, 9 Jun 2003 15:42:06 -0500 (CDT) (envelope-from anderson@centtech.com) Message-ID: <3EE4F11A.4070809@centtech.com> Date: Mon, 09 Jun 2003 15:42:02 -0500 From: Eric Anderson User-Agent: Mozilla/5.0 (X11; U; Linux i386; en-US; rv:1.0.1) Gecko/20020823 Netscape/7.0 X-Accept-Language: en-us, en MIME-Version: 1.0 To: freebsd-performance@freebsd.org References: <20030609203742.30333.qmail@web14911.mail.yahoo.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: Slow disk write speeds over network X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Jun 2003 20:42:07 -0000 Paulo Roberto wrote: >>Sean Chittenden wrote: >> >>>>Ok, I have a file server (NFS) running FreeBSD 4.8-RC1, which is >>> >>having >> >>>>incredibly slow disk write speeds. Locally, doing something like: >>> > > Just an idea (I might be saying something pretty stupid...): Have you > tried increasing the network buffer size?? Here's what netstat -m currently shows: 642/2560/26432 mbufs in use (current/peak/max): 642 mbufs allocated to data 640/2352/6608 mbuf clusters in use (current/peak/max) 5344 Kbytes allocated to network (26% of mb_map in use) 0 requests for memory denied 0 requests for memory delayed 0 calls to protocol drain routines Now, I usually jack this way up, but I didn't happen to on this machine, however it doesn't look like I'm needing it - unless I'm missing something.. Eric -- ------------------------------------------------------------------ Eric Anderson Systems Administrator Centaur Technology Attitudes are contagious, is yours worth catching? ------------------------------------------------------------------ From owner-freebsd-performance@FreeBSD.ORG Mon Jun 9 14:08:00 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0EA1A37B401 for ; Mon, 9 Jun 2003 14:08:00 -0700 (PDT) Received: from web14901.mail.yahoo.com (web14901.mail.yahoo.com [216.136.225.53]) by mx1.FreeBSD.org (Postfix) with SMTP id B094A43F85 for ; Mon, 9 Jun 2003 14:07:59 -0700 (PDT) (envelope-from nirv199@yahoo.com) Message-ID: <20030609210759.68351.qmail@web14901.mail.yahoo.com> Received: from [200.163.192.181] by web14901.mail.yahoo.com via HTTP; Mon, 09 Jun 2003 14:07:59 PDT Date: Mon, 9 Jun 2003 14:07:59 -0700 (PDT) From: Paulo Roberto To: freebsd-performance@freebsd.org In-Reply-To: <3EE4F11A.4070809@centtech.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Subject: Re: Slow disk write speeds over network X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Jun 2003 21:08:00 -0000 --- Eric Anderson wrote: > Now, I usually jack this way up, but I didn't happen to on this > machine, > however it doesn't look like I'm needing it - unless I'm missing > something.. Did you increase it on the *client* machine?? Paulo Roberto __________________________________ Do you Yahoo!? Yahoo! Calendar - Free online calendar with sync to Outlook(TM). http://calendar.yahoo.com From owner-freebsd-performance@FreeBSD.ORG Mon Jun 9 14:10:12 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 62E9937B401 for ; Mon, 9 Jun 2003 14:10:12 -0700 (PDT) Received: from otter3.centtech.com (moat3.centtech.com [207.200.51.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id C942643FBF for ; Mon, 9 Jun 2003 14:10:11 -0700 (PDT) (envelope-from anderson@centtech.com) Received: from centtech.com (electron.centtech.com [204.177.173.173]) by otter3.centtech.com (8.12.3/8.12.3) with ESMTP id h59LAB56046620 for ; Mon, 9 Jun 2003 16:10:11 -0500 (CDT) (envelope-from anderson@centtech.com) Message-ID: <3EE4F7B0.4020004@centtech.com> Date: Mon, 09 Jun 2003 16:10:08 -0500 From: Eric Anderson User-Agent: Mozilla/5.0 (X11; U; Linux i386; en-US; rv:1.0.1) Gecko/20020823 Netscape/7.0 X-Accept-Language: en-us, en MIME-Version: 1.0 To: freebsd-performance@freebsd.org References: <20030609210759.68351.qmail@web14901.mail.yahoo.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: Slow disk write speeds over network X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Jun 2003 21:10:12 -0000 Paulo Roberto wrote: > --- Eric Anderson wrote: > >>Now, I usually jack this way up, but I didn't happen to on this >>machine, >>however it doesn't look like I'm needing it - unless I'm missing >>something.. > > > Did you increase it on the *client* machine?? No, considering I would have to update about 600 hosts. Plus, those same hosts are running just fine with my other file servers at the moment. Eric -- ------------------------------------------------------------------ Eric Anderson Systems Administrator Centaur Technology Attitudes are contagious, is yours worth catching? ------------------------------------------------------------------ From owner-freebsd-performance@FreeBSD.ORG Mon Jun 9 14:15:27 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 65CA837B401 for ; Mon, 9 Jun 2003 14:15:27 -0700 (PDT) Received: from web14908.mail.yahoo.com (web14908.mail.yahoo.com [216.136.225.60]) by mx1.FreeBSD.org (Postfix) with SMTP id F1D7E43FCB for ; Mon, 9 Jun 2003 14:15:26 -0700 (PDT) (envelope-from nirv199@yahoo.com) Message-ID: <20030609211526.58641.qmail@web14908.mail.yahoo.com> Received: from [200.163.192.181] by web14908.mail.yahoo.com via HTTP; Mon, 09 Jun 2003 14:15:26 PDT Date: Mon, 9 Jun 2003 14:15:26 -0700 (PDT) From: Paulo Roberto To: freebsd-performance@freebsd.org In-Reply-To: <3EE4F7B0.4020004@centtech.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Subject: Re: Slow disk write speeds over network X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Jun 2003 21:15:27 -0000 --- Eric Anderson wrote: > No, considering I would have to update about 600 hosts. Plus, those > same hosts are running just fine with my other file servers at the > moment. Eric, If your server's network buffers are ok, I would guess that the "bottleneck" is on your clients machines. just my 2 cents... best regards, Paulo ROberto __________________________________ Do you Yahoo!? Yahoo! Calendar - Free online calendar with sync to Outlook(TM). http://calendar.yahoo.com From owner-freebsd-performance@FreeBSD.ORG Mon Jun 9 14:23:02 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id EB8DE37B407 for ; Mon, 9 Jun 2003 14:23:02 -0700 (PDT) Received: from out003.verizon.net (out003pub.verizon.net [206.46.170.103]) by mx1.FreeBSD.org (Postfix) with ESMTP id ADC8943FDD for ; Mon, 9 Jun 2003 14:23:01 -0700 (PDT) (envelope-from cswiger@mac.com) Received: from mac.com ([141.149.47.46]) by out003.verizon.net (InterMail vM.5.01.05.33 201-253-122-126-133-20030313) with ESMTP id <20030609212300.XSJJ4805.out003.verizon.net@mac.com>; Mon, 9 Jun 2003 16:23:00 -0500 Message-ID: <3EE4FAAF.7060302@mac.com> Date: Mon, 09 Jun 2003 17:22:55 -0400 From: Chuck Swiger Organization: The Courts of Chaos User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.4) Gecko/20030529 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Eric Anderson References: <3EE4E156.6030603@centtech.com> In-Reply-To: <3EE4E156.6030603@centtech.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Authentication-Info: Submitted using SMTP AUTH at out003.verizon.net from [141.149.47.46] at Mon, 9 Jun 2003 16:23:00 -0500 cc: freebsd-performance@freebsd.org Subject: Re: Slow disk write speeds over network X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Jun 2003 21:23:03 -0000 Eric Anderson wrote: > Ok, I have a file server (NFS) running FreeBSD 4.8-RC1, which is having > incredibly slow disk write speeds. Locally, doing something like: > dd if=/dev/zero of=/partition/testfile > Shows 14MB/s - which is what I expect.. > > However, ftping a file or cp'ing over NFS shows speeds less than 1MB/s - > which is not what I would expect - it's got gigabit ethernet, and has > very fast read times (via ftp or nfs) over the network. If the slowdown was only with regard to NFS, I'd ask you about what your NFS clients looked like. However, if you're seeing a slowdown for FTP and other network protocols, it's more likely to be a physical problem with cabling, a NIC getting flaky, or something along those lines. Do you have this connected to a managed switch so that you can take a look at the interface statistics for problems? Maybe try cranking the NIC down from gigabit to 100Mbs ethernet speed and see whether that makes a difference? -Chuck From owner-freebsd-performance@FreeBSD.ORG Mon Jun 9 14:24:02 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5429D37B401 for ; Mon, 9 Jun 2003 14:24:01 -0700 (PDT) Received: from otter3.centtech.com (moat3.centtech.com [207.200.51.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0DD1543FD7 for ; Mon, 9 Jun 2003 14:24:01 -0700 (PDT) (envelope-from anderson@centtech.com) Received: from centtech.com (electron.centtech.com [204.177.173.173]) by otter3.centtech.com (8.12.3/8.12.3) with ESMTP id h59LO056047842 for ; Mon, 9 Jun 2003 16:24:00 -0500 (CDT) (envelope-from anderson@centtech.com) Message-ID: <3EE4FAED.6090603@centtech.com> Date: Mon, 09 Jun 2003 16:23:57 -0500 From: Eric Anderson User-Agent: Mozilla/5.0 (X11; U; Linux i386; en-US; rv:1.0.1) Gecko/20020823 Netscape/7.0 X-Accept-Language: en-us, en MIME-Version: 1.0 To: freebsd-performance@freebsd.org References: <20030609211526.58641.qmail@web14908.mail.yahoo.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: Slow disk write speeds over network X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Jun 2003 21:24:02 -0000 Paulo Roberto wrote: > --- Eric Anderson wrote: > >>No, considering I would have to update about 600 hosts. Plus, those >>same hosts are running just fine with my other file servers at the >>moment. > > > Eric, > > If your server's network buffers are ok, I would guess that the > "bottleneck" is on your clients machines. > > just my 2 cents... Paulo, thanks for the thoughts, however, that doesn't make a whole lot of sense (although I see why you are saying that), because in this case I have several fileservers on the same gigabit switches as this fileserver, and clients can transfer data quickly between all, in both directions, all the time, except data being sent TO this one host. I've even tried machines that are idle, on the same switch as this server, and I still receive slow transfers. Eric -- ------------------------------------------------------------------ Eric Anderson Systems Administrator Centaur Technology Attitudes are contagious, is yours worth catching? ------------------------------------------------------------------ From owner-freebsd-performance@FreeBSD.ORG Mon Jun 9 15:16:39 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 47B4137B401 for ; Mon, 9 Jun 2003 15:16:39 -0700 (PDT) Received: from otter3.centtech.com (moat3.centtech.com [207.200.51.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 75FBA43FBF for ; Mon, 9 Jun 2003 15:16:38 -0700 (PDT) (envelope-from anderson@centtech.com) Received: from centtech.com (electron.centtech.com [204.177.173.173]) by otter3.centtech.com (8.12.3/8.12.3) with ESMTP id h59MGb56051497 for ; Mon, 9 Jun 2003 17:16:37 -0500 (CDT) (envelope-from anderson@centtech.com) Message-ID: <3EE50743.1090704@centtech.com> Date: Mon, 09 Jun 2003 17:16:35 -0500 From: Eric Anderson User-Agent: Mozilla/5.0 (X11; U; Linux i386; en-US; rv:1.0.1) Gecko/20020823 Netscape/7.0 X-Accept-Language: en-us, en MIME-Version: 1.0 To: freebsd-performance@freebsd.org References: <3EE4E156.6030603@centtech.com> <3EE4FAAF.7060302@mac.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: Slow disk write speeds over network X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Jun 2003 22:16:39 -0000 Chuck Swiger wrote: > Eric Anderson wrote: > >> Ok, I have a file server (NFS) running FreeBSD 4.8-RC1, which is >> having incredibly slow disk write speeds. Locally, doing something like: >> dd if=/dev/zero of=/partition/testfile >> Shows 14MB/s - which is what I expect.. >> >> However, ftping a file or cp'ing over NFS shows speeds less than 1MB/s >> - which is not what I would expect - it's got gigabit ethernet, and >> has very fast read times (via ftp or nfs) over the network. > > > If the slowdown was only with regard to NFS, I'd ask you about what your > NFS clients looked like. However, if you're seeing a slowdown for FTP > and other network protocols, it's more likely to be a physical problem > with cabling, a NIC getting flaky, or something along those lines. I see it in both NFS and ftp transfers, but like I said in one of my other notes, I have 4 network cards in this machine, all act the same way. > Do you have this connected to a managed switch so that you can take a > look at the interface statistics for problems? Maybe try cranking the > NIC down from gigabit to 100Mbs ethernet speed and see whether that > makes a difference? Yes - and I just tried that suggestion - same thing. No help. Interfaces look ok. Eric -- ------------------------------------------------------------------ Eric Anderson Systems Administrator Centaur Technology Attitudes are contagious, is yours worth catching? ------------------------------------------------------------------ From owner-freebsd-performance@FreeBSD.ORG Mon Jun 9 15:30:20 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D8F5F37B401 for ; Mon, 9 Jun 2003 15:30:20 -0700 (PDT) Received: from smtp-relay1.barrysworld.com (ns1.barrysworld.com [213.221.172.238]) by mx1.FreeBSD.org (Postfix) with ESMTP id DB10F43FAF for ; Mon, 9 Jun 2003 15:30:19 -0700 (PDT) (envelope-from killing@barrysworld.com) Received: from [213.221.181.50] (helo=barrysworld.com) by smtp-relay1.barrysworld.com with esmtp (Exim 4.12) id 19PV9a-0006oB-00; Mon, 09 Jun 2003 23:30:14 +0100 Received: from vader [212.135.219.179] by barrysworld.com with ESMTP (SMTPD32-7.15) id AB348D0E011C; Mon, 09 Jun 2003 23:33:24 +0100 Message-ID: <008601c32ed6$aff69e00$b3db87d4@vader> From: "Steven Hartland" To: "Eric Anderson" , References: <3EE4E156.6030603@centtech.com> Date: Mon, 9 Jun 2003 23:30:10 +0100 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1158 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1165 Subject: Re: Slow disk write speeds over network X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: Steven Hartland List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Jun 2003 22:30:21 -0000 I've seen an issue like this before turned out to switch interoperability. Im my case Extreme and Planet. Worth checking. If you've got another machine on the local switch try from there. Symptoms where very slow 1Mb's or lower even stalling on ftp download. Solution was to either only use one type of switch or to eliminated the use of all Gig ports ( use only 100MB ports ) N.B. coding Gig ports to 100Mb didn't work. Steve / K From owner-freebsd-performance@FreeBSD.ORG Mon Jun 9 15:51:55 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1C6E637B401 for ; Mon, 9 Jun 2003 15:51:55 -0700 (PDT) Received: from mailbox.cachenetworks.com (iits0193.inlink.com [209.135.140.93]) by mx1.FreeBSD.org (Postfix) with SMTP id 0BAF143FA3 for ; Mon, 9 Jun 2003 15:51:53 -0700 (PDT) (envelope-from matt@deliver3.com) Received: (qmail 26878 invoked by uid 104); 9 Jun 2003 22:51:43 -0000 Received: from matt@deliver3.com by thor.deliver3.com by uid 101 with qmail-scanner-1.16 (. spamassassin: 2.53. Clear:. Processed in 0.090034 secs); 09 Jun 2003 22:51:43 -0000 Received: from local (HELO ?192.168.0.2?) (8.8.8.8) by mailbox.cachenetworks.com with SMTP; 9 Jun 2003 22:51:43 -0000 User-Agent: Microsoft-Entourage/10.1.1.2418 Date: Mon, 09 Jun 2003 18:50:53 -0400 From: Matt Levine To: Eric Anderson Message-ID: In-Reply-To: <3EE50743.1090704@centtech.com> Mime-version: 1.0 Content-type: text/plain; charset="US-ASCII" Content-transfer-encoding: 7bit X-Mailman-Approved-At: Mon, 09 Jun 2003 16:04:29 -0700 cc: freebsd-performance@freebsd.org Subject: Re: Slow disk write speeds over network X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Jun 2003 22:51:55 -0000 On 6/9/03 6:16 PM, "Eric Anderson" wrote: > Chuck Swiger wrote: >> Eric Anderson wrote: >> >>> Ok, I have a file server (NFS) running FreeBSD 4.8-RC1, which is >>> having incredibly slow disk write speeds. Locally, doing something like: >>> dd if=/dev/zero of=/partition/testfile >>> Shows 14MB/s - which is what I expect.. >>> >>> However, ftping a file or cp'ing over NFS shows speeds less than 1MB/s >>> - which is not what I would expect - it's got gigabit ethernet, and >>> has very fast read times (via ftp or nfs) over the network. >> >> >> If the slowdown was only with regard to NFS, I'd ask you about what your >> NFS clients looked like. However, if you're seeing a slowdown for FTP >> and other network protocols, it's more likely to be a physical problem >> with cabling, a NIC getting flaky, or something along those lines. > > I see it in both NFS and ftp transfers, but like I said in one of my > other notes, I have 4 network cards in this machine, all act the same way. > >> Do you have this connected to a managed switch so that you can take a >> look at the interface statistics for problems? Maybe try cranking the >> NIC down from gigabit to 100Mbs ethernet speed and see whether that >> makes a difference? > > Yes - and I just tried that suggestion - same thing. No help. > Interfaces look ok. Not to beat a dead horse, but are these the same nics as in the other machines? Is the switch manageable? Does it agree with the nic(s) on speed/duplex? Tried turning off autonegotiate and forcing on both sides? This has all the symptoms of a duplex mismatch. > > Eric -- Matt Levine "The Trouble with doing anything right the first time is that nobody appreciates how difficult it was." -BIX From owner-freebsd-performance@FreeBSD.ORG Mon Jun 9 16:40:37 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 077CB37B434 for ; Mon, 9 Jun 2003 16:40:37 -0700 (PDT) Received: from web14905.mail.yahoo.com (web14905.mail.yahoo.com [216.136.225.57]) by mx1.FreeBSD.org (Postfix) with SMTP id AE33543F3F for ; Mon, 9 Jun 2003 16:40:36 -0700 (PDT) (envelope-from nirv199@yahoo.com) Message-ID: <20030609234036.46302.qmail@web14905.mail.yahoo.com> Received: from [200.163.192.181] by web14905.mail.yahoo.com via HTTP; Mon, 09 Jun 2003 16:40:36 PDT Date: Mon, 9 Jun 2003 16:40:36 -0700 (PDT) From: Paulo Roberto To: freebsd-performance@freebsd.org In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Subject: Re: Slow disk write speeds over network X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Jun 2003 23:40:37 -0000 Eric, Just one more wild guess: net.inet.tcp.inflight_enable cheers Paulo __________________________________ Do you Yahoo!? Yahoo! Calendar - Free online calendar with sync to Outlook(TM). http://calendar.yahoo.com From owner-freebsd-performance@FreeBSD.ORG Tue Jun 10 01:26:40 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BFBEE37B404 for ; Tue, 10 Jun 2003 01:26:40 -0700 (PDT) Received: from heron.mail.pas.earthlink.net (heron.mail.pas.earthlink.net [207.217.120.189]) by mx1.FreeBSD.org (Postfix) with ESMTP id 33B3343F3F for ; Tue, 10 Jun 2003 01:26:40 -0700 (PDT) (envelope-from tlambert2@mindspring.com) Received: from user-38lc1ah.dialup.mindspring.com ([209.86.5.81] helo=mindspring.com) by heron.mail.pas.earthlink.net with asmtp (SSLv3:RC4-MD5:128) (Exim 3.33 #1) id 19PeSj-0000vk-00; Tue, 10 Jun 2003 01:26:38 -0700 Message-ID: <3EE595D2.B223CA19@mindspring.com> Date: Tue, 10 Jun 2003 01:24:50 -0700 From: Terry Lambert X-Mailer: Mozilla 4.79 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Eric Anderson References: <20030609211526.58641.qmail@web14908.mail.yahoo.com> <3EE4FAED.6090603@centtech.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-ELNK-Trace: b1a02af9316fbb217a47c185c03b154d40683398e744b8a4658d19a7afa5b5fe8205ece409146b5d350badd9bab72f9c350badd9bab72f9c350badd9bab72f9c cc: freebsd-performance@freebsd.org Subject: Re: Slow disk write speeds over network X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Jun 2003 08:26:41 -0000 Eric Anderson wrote: > >>No, considering I would have to update about 600 hosts. Plus, those > >>same hosts are running just fine with my other file servers at the > >>moment. > > > > If your server's network buffers are ok, I would guess that the > > "bottleneck" is on your clients machines. > > > > just my 2 cents... > > Paulo, thanks for the thoughts, however, that doesn't make a whole lot > of sense (although I see why you are saying that), because in this case > I have several fileservers on the same gigabit switches as this > fileserver, and clients can transfer data quickly between all, in both > directions, all the time, except data being sent TO this one host. I've > even tried machines that are idle, on the same switch as this server, > and I still receive slow transfers. Swap cables with another box. BTW: 4 Gigabit cards in one box, with you going to local disk... you've got about 8 cards worth of traffic over your PCI bus. Unless this is a PCI-X based box, you are most likely livelocked; even if it's a PCI-X based box, you could still be livelocked. You haven't said if you were using UDP or TCP for the mounts; you should definitely use TCP with FreeBSD NFS servers; it's also just generally a good idea, since UDP frags act as a fixed non-sliding window: NFS over UDP sucks. Also, you haven't said whether you are using aliases on your network cards; aliases and NFS tend to interact badly. Finally, you probably want to tweak some sysctl's, e.g. net.inet.ip.check_interface=0 net.inet.tcp.inflight_enable=1 net.inet.tcp.inflight_debug=0 net.inet.tcp.msl=3000 net.inet.tcp.inflight_min=6100 net.isr.enable=1 Given your overloading of your bus, that last one is probably the most important one: it enables direct dispatch. You'll also want to enable DEVICE_POLLING in your kernel config file (assuming you have a good ethernet card whose driver supports it): options DEVICE_POLLING options HZ=2000 ...and yet more sysctl's for this: kern.polling.enable=1 kern.polling.user_frac=50 # 0..100; whatever works best If you've got a really terrible Gigabit Ethernet card, then you may be copying all your packets over again (e.g. m_pullup()), and that could be eating your bus, too. -- Terry From owner-freebsd-performance@FreeBSD.ORG Tue Jun 10 08:27:33 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 928A837B401 for ; Tue, 10 Jun 2003 08:27:33 -0700 (PDT) Received: from otter3.centtech.com (moat3.centtech.com [207.200.51.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9BFB143FDF for ; Tue, 10 Jun 2003 08:27:32 -0700 (PDT) (envelope-from anderson@centtech.com) Received: from centtech.com (electron.centtech.com [204.177.173.173]) by otter3.centtech.com (8.12.3/8.12.3) with ESMTP id h5AFRV56010718; Tue, 10 Jun 2003 10:27:31 -0500 (CDT) (envelope-from anderson@centtech.com) Message-ID: <3EE5F8DE.30001@centtech.com> Date: Tue, 10 Jun 2003 10:27:26 -0500 From: Eric Anderson User-Agent: Mozilla/5.0 (X11; U; Linux i386; en-US; rv:1.0.1) Gecko/20020823 Netscape/7.0 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Terry Lambert References: <20030609211526.58641.qmail@web14908.mail.yahoo.com> <3EE4FAED.6090603@centtech.com> <3EE595D2.B223CA19@mindspring.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit cc: freebsd-performance@freebsd.org Subject: Re: Slow disk write speeds over network X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Jun 2003 15:27:33 -0000 Good news, but not done yet.. Keep reading: Terry Lambert wrote: [..snippity snip..] > > Swap cables with another box. > > BTW: 4 Gigabit cards in one box, with you going to local disk... > you've got about 8 cards worth of traffic over your PCI bus. I'm going to a RAID50 (hardware), and I know there's the PCI bus limits - I'm not planning on filling 4 Gig E's at once continually.. > Unless this is a PCI-X based box, you are most likely livelocked; > even if it's a PCI-X based box, you could still be livelocked. > > You haven't said if you were using UDP or TCP for the mounts; > you should definitely use TCP with FreeBSD NFS servers; it's > also just generally a good idea, since UDP frags act as a fixed > non-sliding window: NFS over UDP sucks. Most clients are TCP, but some are still UDP (due to bugs in unmentioned linux distros nfs clients). > Also, you haven't said whether you are using aliases on your > network cards; aliases and NFS tend to interact badly. Nope, no aliases.. I have one card on each network, with one IP per card. I have full subnets (/24) full of P4's trying to slam the NFS server for data all the time.. > Finally, you probably want to tweak some sysctl's, e.g. > > net.inet.ip.check_interface=0 > net.inet.tcp.inflight_enable=1 > net.inet.tcp.inflight_debug=0 > net.inet.tcp.msl=3000 > net.inet.tcp.inflight_min=6100 > net.isr.enable=1 Ok - done.. some where defaults, and I couldn't find net.isr.enable.. Did I need to config something on my kernel for it to show up? Also, can you explain any of those tweaks? > Given your overloading of your bus, that last one is probably > the most important one: it enables direct dispatch. > > You'll also want to enable DEVICE_POLLING in your kernel > config file (assuming you have a good ethernet card whose > driver supports it): > > options DEVICE_POLLING > options HZ=2000 Well, the LINT file says only a few cards support it - not sure if I should trust that or not, but I have Intel PRO/1000T Server Adapters - which should be good enough cards to support it.. I've also put 100Mbit cards in place of the gige's for now to make sure I wasn't hitting a GigE problem or negotiation problem.. > ...and yet more sysctl's for this: > > kern.polling.enable=1 > kern.polling.user_frac=50 # 0..100; whatever works best > > If you've got a really terrible Gigabit Ethernet card, then > you may be copying all your packets over again (e.g. m_pullup()), > and that could be eating your bus, too. Ok, so the end result is that after playing around with sysctl's, I've found that the tcp transfers are doing 20MB/s over FTP, but my NFS is around 1-2MB/s - still slow.. So we've cleared up some tcp issues, but yet still NFS is stinky.. Any more ideas? Eric -- ------------------------------------------------------------------ Eric Anderson Systems Administrator Centaur Technology Attitudes are contagious, is yours worth catching? ------------------------------------------------------------------ From owner-freebsd-performance@FreeBSD.ORG Tue Jun 10 12:56:35 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id AAC2237B421 for ; Tue, 10 Jun 2003 12:56:34 -0700 (PDT) Received: from perrin.int.nxad.com (internal.ext.nxad.com [69.1.70.251]) by mx1.FreeBSD.org (Postfix) with ESMTP id AD88643FBD for ; Tue, 10 Jun 2003 12:56:33 -0700 (PDT) (envelope-from sean@nxad.com) Received: by perrin.int.nxad.com (Postfix, from userid 1001) id 69E6420F01; Tue, 10 Jun 2003 12:56:32 -0700 (PDT) Date: Tue, 10 Jun 2003 12:56:32 -0700 From: Sean Chittenden To: Eric Anderson Message-ID: <20030610195632.GQ65470@perrin.int.nxad.com> References: <20030609211526.58641.qmail@web14908.mail.yahoo.com> <3EE4FAED.6090603@centtech.com> <3EE595D2.B223CA19@mindspring.com> <3EE5F8DE.30001@centtech.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <3EE5F8DE.30001@centtech.com> X-PGP-Key: finger seanc@FreeBSD.org X-PGP-Fingerprint: 3849 3760 1AFE 7B17 11A0 83A6 DD99 E31F BC84 B341 X-Web-Homepage: http://sean.chittenden.org/ User-Agent: Mutt/1.5.4i cc: freebsd-performance@freebsd.org cc: Terry Lambert Subject: Re: Slow disk write speeds over network X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Jun 2003 19:56:35 -0000 > >...and yet more sysctl's for this: > > > > kern.polling.enable=1 > > kern.polling.user_frac=50 # 0..100; whatever works best > > > >If you've got a really terrible Gigabit Ethernet card, then > >you may be copying all your packets over again (e.g. m_pullup()), > >and that could be eating your bus, too. > > Ok, so the end result is that after playing around with sysctl's, > I've found that the tcp transfers are doing 20MB/s over FTP, but my > NFS is around 1-2MB/s - still slow.. So we've cleared up some tcp > issues, but 2yet still NFS is stinky.. > > Any more ideas? I'm using UDP NFS over a 100Mbit/FD link with the following settings and get about 12-14Mbps: net.inet.tcp.recvspace=65536 net.inet.tcp.sendspace=65536 kern.maxfiles=65536 kern.ipc.maxsockbuf=2097152 kern.ipc.somaxconn=8192 net.inet.tcp.delayed_ack=0 net.inet.udp.recvspace=65536 net.inet.udp.maxdgram=57344 net.local.stream.sendspace=65536 net.local.stream.recvspace=65536 vfs.nfs.async=1 net.inet.udp.log_in_vain=1 net.inet.icmp.icmplim=20000 I'm not taking into account jumbo frames or anything like that, so you may want to increase the size of some of these values where appropriate, but some of these may be a start. -sc -- Sean Chittenden From owner-freebsd-performance@FreeBSD.ORG Tue Jun 10 15:23:29 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B445837B401 for ; Tue, 10 Jun 2003 15:23:29 -0700 (PDT) Received: from dfmm.org (walter.dfmm.org [209.151.233.240]) by mx1.FreeBSD.org (Postfix) with ESMTP id 362C343F93 for ; Tue, 10 Jun 2003 15:23:29 -0700 (PDT) (envelope-from jason@shalott.net) Received: (qmail 78996 invoked by uid 1000); 10 Jun 2003 22:23:28 -0000 Received: from localhost (sendmail-bs@127.0.0.1) by localhost with SMTP; 10 Jun 2003 22:23:28 -0000 Date: Tue, 10 Jun 2003 15:23:28 -0700 (PDT) From: Jason Stone X-X-Sender: To: In-Reply-To: <3EE595D2.B223CA19@mindspring.com> Message-ID: <20030610150502.H14379-100000@walter> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Subject: Re: Slow disk write speeds over network X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Jun 2003 22:23:30 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 > You haven't said if you were using UDP or TCP for the mounts; you > should definitely use TCP with FreeBSD NFS servers; it's also just > generally a good idea, since UDP frags act as a fixed non-sliding > window: NFS over UDP sucks. Huh. I thought that the conventional wisdom was that on a local network with no packet loss (and therefore no re-transmission penalties), udp was way faster because the overhead was so much less. Sorry if this seems like a pretty basic question, but can you explain this? -Jason -------------------------------------------------------------------------- Freud himself was a bit of a cold fish, and one cannot avoid the suspicion that he was insufficiently fondled when he was an infant. -- Ashley Montagu -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.1 (FreeBSD) Comment: See https://private.idealab.com/public/jason/jason.gpg iD8DBQE+5lpgswXMWWtptckRAqIUAKCQxIoFOoYuqhOQ5vF9JNwbTe0kbQCghJVe HblPYP54wIyhZVGFwqrDe30= =kbba -----END PGP SIGNATURE----- From owner-freebsd-performance@FreeBSD.ORG Tue Jun 10 15:27:38 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 94D2D37B401 for ; Tue, 10 Jun 2003 15:27:38 -0700 (PDT) Received: from mta1.adelphia.net (mta1.adelphia.net [64.8.50.175]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9392943F75 for ; Tue, 10 Jun 2003 15:27:37 -0700 (PDT) (envelope-from Support@Netflag.Net) Received: from nfn2.Netflag.Net ([68.69.240.78]) by mta5.adelphia.net (InterMail vM.5.01.05.32 201-253-122-126-132-20030307) with ESMTP id <20030610222610.ETPS1551.mta5.adelphia.net@nfn2.Netflag.Net>; Tue, 10 Jun 2003 18:26:10 -0400 Message-Id: <5.2.0.9.2.20030610152538.00b48be0@pop.dc3.adelphia.net> X-Sender: pedramn@pop.dc3.adelphia.net X-Mailer: QUALCOMM Windows Eudora Version 5.2.0.9 Date: Tue, 10 Jun 2003 15:25:49 -0700 To: Jason Stone , From: Pedram Nimreezi In-Reply-To: <20030610150502.H14379-100000@walter> References: <3EE595D2.B223CA19@mindspring.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Mailman-Approved-At: Tue, 10 Jun 2003 15:28:38 -0700 Subject: Re: Slow disk write speeds over network X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Jun 2003 22:27:38 -0000 I'd venture to say NFS generally sucks At 03:23 PM 6/10/2003 -0700, Jason Stone wrote: >-----BEGIN PGP SIGNED MESSAGE----- >Hash: SHA1 > > > > You haven't said if you were using UDP or TCP for the mounts; you > > should definitely use TCP with FreeBSD NFS servers; it's also just > > generally a good idea, since UDP frags act as a fixed non-sliding > > window: NFS over UDP sucks. > >Huh. I thought that the conventional wisdom was that on a local network >with no packet loss (and therefore no re-transmission penalties), udp was >way faster because the overhead was so much less. > >Sorry if this seems like a pretty basic question, but can you explain >this? > > > -Jason > > -------------------------------------------------------------------------- > Freud himself was a bit of a cold fish, and one cannot avoid the suspicion > that he was insufficiently fondled when he was an infant. > -- Ashley Montagu >-----BEGIN PGP SIGNATURE----- >Version: GnuPG v1.2.1 (FreeBSD) >Comment: See https://private.idealab.com/public/jason/jason.gpg > >iD8DBQE+5lpgswXMWWtptckRAqIUAKCQxIoFOoYuqhOQ5vF9JNwbTe0kbQCghJVe >HblPYP54wIyhZVGFwqrDe30= >=kbba >-----END PGP SIGNATURE----- > >_______________________________________________ >freebsd-performance@freebsd.org mailing list >http://lists.freebsd.org/mailman/listinfo/freebsd-performance >To unsubscribe, send any mail to >"freebsd-performance-unsubscribe@freebsd.org" From owner-freebsd-performance@FreeBSD.ORG Tue Jun 10 15:30:19 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D64FB37B404 for ; Tue, 10 Jun 2003 15:30:19 -0700 (PDT) Received: from perrin.int.nxad.com (internal.ext.nxad.com [69.1.70.251]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4D9D443FBD for ; Tue, 10 Jun 2003 15:30:19 -0700 (PDT) (envelope-from sean@nxad.com) Received: by perrin.int.nxad.com (Postfix, from userid 1001) id B7D7220F01; Tue, 10 Jun 2003 15:30:18 -0700 (PDT) Date: Tue, 10 Jun 2003 15:30:18 -0700 From: Sean Chittenden To: Pedram Nimreezi Message-ID: <20030610223018.GV65470@perrin.int.nxad.com> References: <3EE595D2.B223CA19@mindspring.com> <5.2.0.9.2.20030610152538.00b48be0@pop.dc3.adelphia.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <5.2.0.9.2.20030610152538.00b48be0@pop.dc3.adelphia.net> X-PGP-Key: finger seanc@FreeBSD.org X-PGP-Fingerprint: 3849 3760 1AFE 7B17 11A0 83A6 DD99 E31F BC84 B341 X-Web-Homepage: http://sean.chittenden.org/ User-Agent: Mutt/1.5.4i cc: freebsd-performance@freebsd.org cc: Jason Stone Subject: Re: Slow disk write speeds over network X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Jun 2003 22:30:20 -0000 > I'd venture to say NFS generally sucks [ Please stop here, do not pass go, do not collect your $200 ] I don't think anyone would disagree, but in terms of performance, is there a different solution that is faster? If not, these kinds of comments don't help this discussion. Open to suggestions, Sean -- Sean Chittenden From owner-freebsd-performance@FreeBSD.ORG Tue Jun 10 15:29:11 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B47A137B401 for ; Tue, 10 Jun 2003 15:29:11 -0700 (PDT) Received: from perrin.int.nxad.com (internal.ext.nxad.com [69.1.70.251]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4D58643FA3 for ; Tue, 10 Jun 2003 15:29:09 -0700 (PDT) (envelope-from hmp@nxad.com) Received: by perrin.int.nxad.com (Postfix, from userid 1072) id A312B20F01; Tue, 10 Jun 2003 15:29:08 -0700 (PDT) Date: Tue, 10 Jun 2003 15:29:08 -0700 From: Hiten Pandya To: Eric Anderson Message-ID: <20030610222908.GC81267@perrin.int.nxad.com> References: <20030609211526.58641.qmail@web14908.mail.yahoo.com> <3EE4FAED.6090603@centtech.com> <3EE595D2.B223CA19@mindspring.com> <3EE5F8DE.30001@centtech.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <3EE5F8DE.30001@centtech.com> X-Operating-System: FreeBSD FreeBSD 4.7-STABLE User-Agent: Mutt/1.5.4i X-Mailman-Approved-At: Tue, 10 Jun 2003 15:30:44 -0700 cc: freebsd-performance@freebsd.org cc: Terry Lambert Subject: Re: Slow disk write speeds over network X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Jun 2003 22:29:12 -0000 On Tue, Jun 10, 2003 at 10:27:26AM -0500, Eric Anderson wrote: > Good news, but not done yet.. Keep reading: > > Any more ideas? You could also try increasing your MTU to 4K or 8K if the driver supports it (which it should since its GigE...) Cheers. -- Hiten (hmp@FreeBSD.ORG) From owner-freebsd-performance@FreeBSD.ORG Tue Jun 10 15:53:55 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6D43037B401 for ; Tue, 10 Jun 2003 15:53:55 -0700 (PDT) Received: from mta1.adelphia.net (mta1.adelphia.net [64.8.50.175]) by mx1.FreeBSD.org (Postfix) with ESMTP id AC5C143FBF for ; Tue, 10 Jun 2003 15:53:54 -0700 (PDT) (envelope-from Support@Netflag.Net) Received: from nfn2.Netflag.Net ([68.69.240.78]) by mta9.adelphia.net (InterMail vM.5.01.05.32 201-253-122-126-132-20030307) with ESMTP id <20030610223624.QIAO1549.mta9.adelphia.net@nfn2.Netflag.Net>; Tue, 10 Jun 2003 18:36:24 -0400 Message-Id: <5.2.0.9.2.20030610153334.00bc2c98@pop.dc3.adelphia.net> X-Sender: pedramn@pop.dc3.adelphia.net X-Mailer: QUALCOMM Windows Eudora Version 5.2.0.9 Date: Tue, 10 Jun 2003 15:36:03 -0700 To: Sean Chittenden From: Pedram Nimreezi In-Reply-To: <20030610223018.GV65470@perrin.int.nxad.com> References: <5.2.0.9.2.20030610152538.00b48be0@pop.dc3.adelphia.net> <3EE595D2.B223CA19@mindspring.com> <5.2.0.9.2.20030610152538.00b48be0@pop.dc3.adelphia.net> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed cc: freebsd-performance@freebsd.org Subject: Re: Slow disk write speeds over network X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Jun 2003 22:53:55 -0000 I'm sorry I don't mean to be vague but I just get a bit tired of reading 40 posts on NFS itself... I have never once used NFS and don't see the need for it, but if one feels there is a need. Identify the causality... I need to integrate a shared medium with 100+ computers because ... 1. .... At 03:30 PM 6/10/2003 -0700, Sean Chittenden wrote: > > I'd venture to say NFS generally sucks > >[ Please stop here, do not pass go, do not collect your $200 ] > >I don't think anyone would disagree, but in terms of performance, is >there a different solution that is faster? If not, these kinds of >comments don't help this discussion. > >Open to suggestions, > Sean > >-- >Sean Chittenden >_______________________________________________ >freebsd-performance@freebsd.org mailing list >http://lists.freebsd.org/mailman/listinfo/freebsd-performance >To unsubscribe, send any mail to >"freebsd-performance-unsubscribe@freebsd.org" From owner-freebsd-performance@FreeBSD.ORG Tue Jun 10 18:08:19 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5AD4537B401 for ; Tue, 10 Jun 2003 18:08:19 -0700 (PDT) Received: from out005.verizon.net (out005pub.verizon.net [206.46.170.143]) by mx1.FreeBSD.org (Postfix) with ESMTP id 744FD43F75 for ; Tue, 10 Jun 2003 18:08:18 -0700 (PDT) (envelope-from cswiger@mac.com) Received: from mac.com ([141.149.47.46]) by out005.verizon.net (InterMail vM.5.01.05.33 201-253-122-126-133-20030313) with ESMTP id <20030611010817.RIOS20032.out005.verizon.net@mac.com> for ; Tue, 10 Jun 2003 20:08:17 -0500 Message-ID: <3EE680FD.1060800@mac.com> Date: Tue, 10 Jun 2003 21:08:13 -0400 From: Chuck Swiger Organization: The Courts of Chaos User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.4) Gecko/20030529 X-Accept-Language: en-us, en MIME-Version: 1.0 To: freebsd-performance@freebsd.org References: <5.2.0.9.2.20030610152538.00b48be0@pop.dc3.adelphia.net> <3EE595D2.B223CA19@mindspring.com> <5.2.0.9.2.20030610152538.00b48be0@pop.dc3.adelphia.net> <5.2.0.9.2.20030610153334.00bc2c98@pop.dc3.adelphia.net> In-Reply-To: <5.2.0.9.2.20030610153334.00bc2c98@pop.dc3.adelphia.net> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Authentication-Info: Submitted using SMTP AUTH at out005.verizon.net from [141.149.47.46] at Tue, 10 Jun 2003 20:08:17 -0500 Subject: Re: Slow disk write speeds over network X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Jun 2003 01:08:19 -0000 Pedram Nimreezi wrote: > I'm sorry I don't mean to be vague but I just get a bit tired of reading > 40 posts on NFS itself... I haven't seen a lot of list traffic about NFS. > I have never once used NFS and don't see the need for it, but if one > feels there is a need. Identify the causality... I need to integrate > a shared medium with 100+ computers because ... 1. You want to share filesystems like user home directories, project work areas, etc among all of those client machines without having to copy all of the files (can resyncronize them, make room on each local drive, etc)...? -Chuck From owner-freebsd-performance@FreeBSD.ORG Tue Jun 10 23:41:16 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3D1CE37B401 for ; Tue, 10 Jun 2003 23:41:16 -0700 (PDT) Received: from katmai.eltopia.com (katmai.eltopia.com [64.146.93.25]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2944343F85 for ; Tue, 10 Jun 2003 23:41:15 -0700 (PDT) (envelope-from aseelye-lists@eltopia.com) Received: from metallus (unverified [207.254.34.18]) by katmai.eltopia.com for ; Tue, 10 Jun 2003 23:41:13 -0700 Message-ID: <003b01c32fe4$731e4b40$1222fecf@metallus> From: "Aaron Seelye" To: References: <5.2.0.9.2.20030610152538.00b48be0@pop.dc3.adelphia.net><3EE595D2.B223CA19@mindspring.com><5.2.0.9.2.20030610152538.00b48be0@pop.dc3.adelphia.net><5.2.0.9.2.20030610153334.00bc2c98@pop.dc3.adelphia.net> <3EE680FD.1060800@mac.com> Date: Tue, 10 Jun 2003 23:41:11 -0700 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2720.3000 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 Subject: Re: Slow disk write speeds over network X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Jun 2003 06:41:16 -0000 2. Because "make buildworld" on 100+ computers is insane? -Aaron ----- Original Message ----- From: "Chuck Swiger" To: Sent: Tuesday, June 10, 2003 6:08 PM Subject: Re: Slow disk write speeds over network > Pedram Nimreezi wrote: > > I'm sorry I don't mean to be vague but I just get a bit tired of reading > > 40 posts on NFS itself... > > I haven't seen a lot of list traffic about NFS. > > > I have never once used NFS and don't see the need for it, but if one > > feels there is a need. Identify the causality... I need to integrate > > a shared medium with 100+ computers because ... > > 1. You want to share filesystems like user home directories, project work areas, > etc among all of those client machines without having to copy all of the files > (can resyncronize them, make room on each local drive, etc)...? > > -Chuck > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" From owner-freebsd-performance@FreeBSD.ORG Wed Jun 11 00:47:53 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id EA86D37B401 for ; Wed, 11 Jun 2003 00:47:53 -0700 (PDT) Received: from heron.mail.pas.earthlink.net (heron.mail.pas.earthlink.net [207.217.120.189]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5C61643FE5 for ; Wed, 11 Jun 2003 00:47:53 -0700 (PDT) (envelope-from tlambert2@mindspring.com) Received: from user-38lc0s4.dialup.mindspring.com ([209.86.3.132] helo=mindspring.com) by heron.mail.pas.earthlink.net with asmtp (SSLv3:RC4-MD5:128) (Exim 3.33 #1) id 19Q0Kl-0002wb-00; Wed, 11 Jun 2003 00:47:52 -0700 Message-ID: <3EE6DE3B.8B8EE11B@mindspring.com> Date: Wed, 11 Jun 2003 00:46:03 -0700 From: Terry Lambert X-Mailer: Mozilla 4.79 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Jason Stone References: <20030610150502.H14379-100000@walter> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-ELNK-Trace: b1a02af9316fbb217a47c185c03b154d40683398e744b8a41efc0b0e78180801e5cbf26f5e597f2c387f7b89c61deb1d350badd9bab72f9c350badd9bab72f9c cc: freebsd-performance@freebsd.org Subject: Re: Slow disk write speeds over network X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Jun 2003 07:47:54 -0000 Jason Stone wrote: > > You haven't said if you were using UDP or TCP for the mounts; you > > should definitely use TCP with FreeBSD NFS servers; it's also just > > generally a good idea, since UDP frags act as a fixed non-sliding > > window: NFS over UDP sucks. > > Huh. I thought that the conventional wisdom was that on a local network > with no packet loss (and therefore no re-transmission penalties), udp was > way faster because the overhead was so much less. > > Sorry if this seems like a pretty basic question, but can you explain > this? Sure: 1) There is no such thing as no packet loss. 2) The UDP packets are reassembled in a reassembly queue on the receiver. While this is happening, you can only have one datagram outstanding at a time. With TCP, you get a sliding window; with UDP, you stall waiting for the reassembly, effectively giving you a non-sliding window (request/response, with round trip latencies per packet, instead of two of them amortized across a 100M file transfer). 3) When a packet is lost, the UDP retransmit code is rather crufty. It resends the whole series of packets, and you eat the overhead for that. TCP, on the other hand, can do selective acknowledgement, or, if it's not supported by both ends, it can at least acknowledge the packets that did get through, saving you a retransmit. 4) FreeBSD's UDP fragment reassembly buffer code is well known to pretty much suck. This is true of most UDP fragment reassembly code in the universe, however, and is not that specific to FreeBSD. So sending UDP packets that get fragged because they're larger than your MTU is not a very clever way of achieving a fixed window size larger than the MTU (see also #2, above, for why you do not want to used an effectively fixed window protocol anyway). Even if there were no packet loss at all with UDP, unless all your data is around the size of one rsize/wsize/packet, the combined RTT overhead for even a moderately large number of packets in a single run is enough to trigger the amortized cost of the additional TCP overhead being lower than the UDP overhead from the latency. Depending on your hardware (switch latency, half duplex, etc.), you could also be talking about a significant combined bandwidth delay product. Now add to all this that you have to send explicit ACKs with UDP, while you can use piggy-back ACKs on the return payloads for TCP. I think the idea that UDP was OK for nearly-lossless short-haul came about from people who couldn't code a working TCP NFS client. 8-). -- Terry From owner-freebsd-performance@FreeBSD.ORG Wed Jun 11 00:49:50 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6987A37B405 for ; Wed, 11 Jun 2003 00:49:50 -0700 (PDT) Received: from heron.mail.pas.earthlink.net (heron.mail.pas.earthlink.net [207.217.120.189]) by mx1.FreeBSD.org (Postfix) with ESMTP id 181ED43FE0 for ; Wed, 11 Jun 2003 00:49:49 -0700 (PDT) (envelope-from tlambert2@mindspring.com) Received: from user-38lc0s4.dialup.mindspring.com ([209.86.3.132] helo=mindspring.com) by heron.mail.pas.earthlink.net with asmtp (SSLv3:RC4-MD5:128) (Exim 3.33 #1) id 19Q0Md-000376-00; Wed, 11 Jun 2003 00:49:48 -0700 Message-ID: <3EE6DEAE.D9158761@mindspring.com> Date: Wed, 11 Jun 2003 00:47:58 -0700 From: Terry Lambert X-Mailer: Mozilla 4.79 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Pedram Nimreezi References: <3EE595D2.B223CA19@mindspring.com> <5.2.0.9.2.20030610152538.00b48be0@pop.dc3.adelphia.net> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-ELNK-Trace: b1a02af9316fbb217a47c185c03b154d40683398e744b8a41efc0b0e781808015ec3c318c6a002932601a10902912494350badd9bab72f9c350badd9bab72f9c cc: freebsd-performance@freebsd.org cc: Jason Stone Subject: Re: Slow disk write speeds over network X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Jun 2003 07:49:50 -0000 Pedram Nimreezi wrote: > I'd venture to say NFS generally sucks 8-). It's better in v3 and v4, where you are allowed to do things like cache the results of VOP_ACCESS. Hardly anyone does, but still, in theiry, you can save half your wire transactions. -- Terry From owner-freebsd-performance@FreeBSD.ORG Wed Jun 11 02:44:08 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 954A637B401 for ; Wed, 11 Jun 2003 02:44:08 -0700 (PDT) Received: from heron.mail.pas.earthlink.net (heron.mail.pas.earthlink.net [207.217.120.189]) by mx1.FreeBSD.org (Postfix) with ESMTP id EA2E343FE9 for ; Wed, 11 Jun 2003 02:44:07 -0700 (PDT) (envelope-from tlambert2@mindspring.com) Received: from user-38lc0s4.dialup.mindspring.com ([209.86.3.132] helo=mindspring.com) by heron.mail.pas.earthlink.net with asmtp (SSLv3:RC4-MD5:128) (Exim 3.33 #1) id 19Q29E-0005nv-00; Wed, 11 Jun 2003 02:44:05 -0700 Message-ID: <3EE6F918.1C1FF28E@mindspring.com> Date: Wed, 11 Jun 2003 02:40:40 -0700 From: Terry Lambert X-Mailer: Mozilla 4.79 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Sean Chittenden References: <20030609211526.58641.qmail@web14908.mail.yahoo.com> <3EE4FAED.6090603@centtech.com> <3EE595D2.B223CA19@mindspring.com> <3EE5F8DE.30001@centtech.com> <20030610195632.GQ65470@perrin.int.nxad.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-ELNK-Trace: b1a02af9316fbb217a47c185c03b154d40683398e744b8a46e7e10ac48e93404979dab5c78387674a8438e0f32a48e08350badd9bab72f9c350badd9bab72f9c cc: freebsd-performance@freebsd.org cc: Eric Anderson Subject: Re: Slow disk write speeds over network X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Jun 2003 09:44:08 -0000 Sean Chittenden wrote: > > >...and yet more sysctl's for this: > > > > > > kern.polling.enable=1 > > > kern.polling.user_frac=50 # 0..100; whatever works best > > > > > >If you've got a really terrible Gigabit Ethernet card, then > > >you may be copying all your packets over again (e.g. m_pullup()), > > >and that could be eating your bus, too. > > > > Ok, so the end result is that after playing around with sysctl's, > > I've found that the tcp transfers are doing 20MB/s over FTP, but my > > NFS is around 1-2MB/s - still slow.. So we've cleared up some tcp > > issues, but 2yet still NFS is stinky.. > > > > Any more ideas? > > I'm using UDP NFS over a 100Mbit/FD link with the following settings > and get about 12-14Mbps: Numbers taken in context of original poster... YMMV: > net.inet.tcp.recvspace=65536 This is most important for writes. The sendspace is pretty well not going to help you out, unless you are starvation deadlocked; it didn't look like you were from your previous posting. BTW: I believe this is the default. > net.inet.tcp.sendspace=65536 Double the default. Might not be a good idea, unless you have a ton of memory. You will potentially use 64K send + 64K receive times number of sockets. Assuming 4G and near-perfect tuning, you will be limited to 16384 simultaneous connections fully packed before memory pressure causes your machine to crash. I tend to like smaller buffers and more connections. If you only have 512M, drop this number to 2048 simultaneous connections if all buffers are full. > kern.maxfiles=65536 Seems kind of overkill for the number of connections you can support without overcommit, and the number of client machines you say you have. > kern.ipc.maxsockbuf=2097152 > kern.ipc.somaxconn=8192 IPC numbers; not relevent. > net.inet.tcp.delayed_ack=0 This will make it more responsive, at some cost in overhead. > net.inet.udp.recvspace=65536 > net.inet.udp.maxdgram=57344 These are important for UDP NFS. I do not reccomend it. > net.local.stream.sendspace=65536 > net.local.stream.recvspace=65536 IPC numbers; not relelvent. > vfs.nfs.async=1 This is very dangerous, if you care about your data. It permits NFS to ACK writes before they have been committed to stable storage. With a large enough window size, this should not be necessary. > net.inet.udp.log_in_vain=1 This is just overhead; I reccomend turning it off. > net.inet.icmp.icmplim=20000 This is only useful for TCP; but it can be very useful. Basically, this is "connection rate limiting". If you have a ton of clients, or trying to "netbench" the system, then set this number up. For 100 NFS clients, it likely does not matter. > I'm not taking into account jumbo frames or anything like that, so you > may want to increase the size of some of these values where > appropriate, but some of these may be a start. -sc In my experience, Intel GigE cards do not play nice with others when it comes to jumbo frames or negotiation. I much prefer the Tigon/Alteon/Broadcom/whoever-they-are-this-week-still-no-firmware, though I would obviously like the same firmware access to the Tigon III's as they used to give us to the Tigon II's. -- Terry From owner-freebsd-performance@FreeBSD.ORG Wed Jun 11 03:23:11 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id F3AC437B401 for ; Wed, 11 Jun 2003 03:23:10 -0700 (PDT) Received: from heron.mail.pas.earthlink.net (heron.mail.pas.earthlink.net [207.217.120.189]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3A05A43FAF for ; Wed, 11 Jun 2003 03:23:10 -0700 (PDT) (envelope-from tlambert2@mindspring.com) Received: from user-38lc0s4.dialup.mindspring.com ([209.86.3.132] helo=mindspring.com) by heron.mail.pas.earthlink.net with asmtp (SSLv3:RC4-MD5:128) (Exim 3.33 #1) id 19Q2ky-0001yo-00; Wed, 11 Jun 2003 03:23:06 -0700 Message-ID: <3EE7021E.F2928B7@mindspring.com> Date: Wed, 11 Jun 2003 03:19:10 -0700 From: Terry Lambert X-Mailer: Mozilla 4.79 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Eric Anderson References: <20030609211526.58641.qmail@web14908.mail.yahoo.com> <3EE4FAED.6090603@centtech.com> <3EE595D2.B223CA19@mindspring.com> <3EE5F8DE.30001@centtech.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-ELNK-Trace: b1a02af9316fbb217a47c185c03b154d40683398e744b8a4793ab7015ff2800459f5ec9db8306898350badd9bab72f9c350badd9bab72f9c350badd9bab72f9c cc: freebsd-performance@freebsd.org Subject: Re: Slow disk write speeds over network X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Jun 2003 10:23:11 -0000 Eric Anderson wrote: > Good news, but not done yet.. Keep reading: Sean Chittenden also had a couple of good pieces of advice; read his posting too. > > You haven't said if you were using UDP or TCP for the mounts; > > you should definitely use TCP with FreeBSD NFS servers; it's > > also just generally a good idea, since UDP frags act as a fixed > > non-sliding window: NFS over UDP sucks. > > Most clients are TCP, but some are still UDP (due to bugs in unmentioned > linux distros nfs clients). These will be able to starve each other out. There is a nifty DOS against the UDP reassembly code that operates by sending all the frags in an overly large datagram, but one. > > Also, you haven't said whether you are using aliases on your > > network cards; aliases and NFS tend to interact badly. > > Nope, no aliases.. I have one card on each network, with one IP per > card. I have full subnets (/24) full of P4's trying to slam the NFS > server for data all the time.. Good that you have no aliases; the aliasing code is not efficient for a large number of aliases. Also, the in_pcbhash code could use a rewrite to handle INADDR_ANY sockets better. Not a problem for your load level or configuration. > > Finally, you probably want to tweak some sysctl's, e.g. > > > > net.inet.ip.check_interface=0 > > net.inet.tcp.inflight_enable=1 > > net.inet.tcp.inflight_debug=0 > > net.inet.tcp.msl=3000 > > net.inet.tcp.inflight_min=6100 > > net.isr.enable=1 > > Ok - done.. some where defaults, and I couldn't find net.isr.enable.. > Did I need to config something on my kernel for it to show up? You have to set a compile option; look in /usr/src/sys/net; grep for "netisr_dispatch" or just "dispatch". > Also, can you explain any of those tweaks? The check_interface makes FreeBSD not care if the interface a response comes in on is the same as the one a request did. I told you to set that one in case your network topology was at fault. The inflight_enable allows inflight processing. This will cause it to use an expedited processing path. The debug is on by default )oir was) when inflight was used, and adds overhead, so it should be turned off. Both of these implement about 1/3 of a receiver livelock solution. Setting the MSL down decreases your relative bandwidth delay product; since you are using GigE, this should be relatively low. If you had non-local users on a VPN over a slow link, this would probably be a bad thing. Local GigE, though, and it's desirable. The net.isr.enable=1 will save you about 100ms per packet, minimum, and more if you have a high interrupt overhead that livelocks you from running NETISR. What it does is turns on direct processing by IP and TCP of packets as they come in the interface and you get the interrupt. Combined with soft interrupt coelescing and polling, they should give you another 1/3 of the receiver livelock fixup. The final third isn't available, unless you are willing to hack network stack code and scheduler code, since FreeBSD doesn't include LRP or Weighted Fair Share Queuing. > > Given your overloading of your bus, that last one is probably > > the most important one: it enables direct dispatch. > > > > You'll also want to enable DEVICE_POLLING in your kernel > > config file (assuming you have a good ethernet card whose > > driver supports it): > > > > options DEVICE_POLLING > > options HZ=2000 > > Well, the LINT file says only a few cards support it - not sure if I > should trust that or not, but I have Intel PRO/1000T Server Adapters - > which should be good enough cards to support it.. I've also put 100Mbit > cards in place of the gige's for now to make sure I wasn't hitting a > GigE problem or negotiation problem.. You should grep for DEVICE_POLLING in the network device drivers you are interested in using to see if they have the support. Also you can get up to 15% by adding soft interrupt coelescing code, if the driver doesn't already support it (I added it for a couple of drivers, and it was committed after the benchmarks showed it was good, but it's not everywhere); the basic idea is you take the interrupt, run rx_eof(), and call ether_input(). Then repeat the process until you hit some count limit, or until there's no more data. The direct dispatch (net.isr.enable) combined with that will process most packet trains to completion at interrupt, saving you 10ms up and 10ms back down per packet exchange (NETISR only runs on exit from spl or at the HZ time, which is default every 10ms). > > ...and yet more sysctl's for this: > > > > kern.polling.enable=1 > > kern.polling.user_frac=50 # 0..100; whatever works best > > > > If you've got a really terrible Gigabit Ethernet card, then > > you may be copying all your packets over again (e.g. m_pullup()), > > and that could be eating your bus, too. > > Ok, so the end result is that after playing around with sysctl's, I've > found that the tcp transfers are doing 20MB/s over FTP, but my NFS is > around 1-2MB/s - still slow.. So we've cleared up some tcp issues, but > yet still NFS is stinky.. > > Any more ideas? If you have a choice on the disks, go SCSI; you probably won't have a choice, though, if you haven't bought them already. The tagged command queuing in ATAPI can't disconnect during a write, only during a read, so writes serialize and reads don't. On SCSI, neither writes nor reads serialize (at least until you hit your tag queue depth). Standard advice about MBUFS/NMBCLUSTERS; see the NOTES files about these config options. Also, I would make sure maxusers was non-zero: disable the auto-tuning, it's generally not going to give you an optimal mix for a dedicated server, no matter what it's dedicated to doing. There's Sean's suggestions... I don't reccomend some of them, for data integrity reasons (see my comments in response to his post), but others are very good. If you can get your Intel cards to play nice with your switch, going to 8K packets (jumbograms) will help. In my experience, Intel doesn't play nice with other card vendors, and there's no real standards for MTU negotiation, so you have to futs with a lot of equipment to get it setup (manually locking the MTU). Also, many switchs (e.g. Alpine) don't really have enough memory in them to deal with this. Some GigE cards also have too little memory to do this and offload the TCP checksum processing. Reminds me: make sure your checksums are being done by your cards, if you can: checksum calculations in software are brutal on your performance. It is(/was) an ifconfig option. Just for grins (not for production!) you may want to mount your FS async, and set the NFS async option Sean wrote about. I would probably disable SYN-caching and SYN-cookie. I *always* disable SYN-cookie on any exposed machine (computational DOS attack is possible); the SYN-cache is a good defense against a DOS attack, but if this is an interior machine (and it should be), then your firewall already protects it; SYN-cache adds some overhead (read: latency) you probably don't want, and the cookie code will be harmless, but isn't terribly useful unless you are getting a huge connection attempt per second rate. You may also want to disable slowstart and the Nagle algorithm, but you will have to look those up (doing it makes you a bad network citizen, and I would be aiding and abetting ;^)). It shouldn't be *too* bad if you're switched rather than bridged or hub'ed all the way through (L4, not L2, so no Alpine GigE). If you are willing to hack code, PSC at CMU had a nice rate halving implementation for a slightly older version of the BSD stack, and both Rice University and Duke University have an LRP implementation (Duke's is more modern), but you'll have to know what you're doing in the stack to port any of these. You probably don't need to worry about load-shedding until your machine is spending all its time in interrupt, so there's no use going into RED-queueing or other programming work. -- Terry From owner-freebsd-performance@FreeBSD.ORG Wed Jun 11 12:35:33 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 827FD37B401 for ; Wed, 11 Jun 2003 12:35:33 -0700 (PDT) Received: from mta1.adelphia.net (mta1.adelphia.net [64.8.50.175]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9BCDE43F85 for ; Wed, 11 Jun 2003 12:35:32 -0700 (PDT) (envelope-from Support@Netflag.Net) Received: from nfn2.Netflag.Net ([68.69.240.78]) by mta3.adelphia.net (InterMail vM.5.01.05.32 201-253-122-126-132-20030307) with ESMTP id <20030611165123.ROXR1347.mta3.adelphia.net@nfn2.Netflag.Net>; Wed, 11 Jun 2003 12:51:23 -0400 Message-Id: <5.2.0.9.2.20030611095023.01893d00@pop.dc3.adelphia.net> X-Sender: pedramn@pop.dc3.adelphia.net X-Mailer: QUALCOMM Windows Eudora Version 5.2.0.9 Date: Wed, 11 Jun 2003 09:51:07 -0700 To: Terry Lambert From: Pedram Nimreezi In-Reply-To: <3EE6DEAE.D9158761@mindspring.com> References: <3EE595D2.B223CA19@mindspring.com> <5.2.0.9.2.20030610152538.00b48be0@pop.dc3.adelphia.net> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed cc: freebsd-performance@freebsd.org Subject: Re: Slow disk write speeds over network X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Jun 2003 19:35:33 -0000 sounds promising At 12:47 AM 6/11/2003 -0700, Terry Lambert wrote: >Pedram Nimreezi wrote: > > I'd venture to say NFS generally sucks > >8-). > >It's better in v3 and v4, where you are allowed to do things >like cache the results of VOP_ACCESS. Hardly anyone does, >but still, in theiry, you can save half your wire transactions. > >-- Terry >_______________________________________________ >freebsd-performance@freebsd.org mailing list >http://lists.freebsd.org/mailman/listinfo/freebsd-performance >To unsubscribe, send any mail to >"freebsd-performance-unsubscribe@freebsd.org" From owner-freebsd-performance@FreeBSD.ORG Wed Jun 11 13:55:50 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D76D737B401 for ; Wed, 11 Jun 2003 13:55:50 -0700 (PDT) Received: from perrin.int.nxad.com (internal.ext.nxad.com [69.1.70.251]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5606343F75 for ; Wed, 11 Jun 2003 13:55:50 -0700 (PDT) (envelope-from sean@nxad.com) Received: by perrin.int.nxad.com (Postfix, from userid 1001) id 928F421058; Wed, 11 Jun 2003 13:55:49 -0700 (PDT) Date: Wed, 11 Jun 2003 13:55:49 -0700 From: Sean Chittenden To: Terry Lambert Message-ID: <20030611205549.GY65470@perrin.int.nxad.com> References: <20030609211526.58641.qmail@web14908.mail.yahoo.com> <3EE4FAED.6090603@centtech.com> <3EE595D2.B223CA19@mindspring.com> <3EE5F8DE.30001@centtech.com> <20030610195632.GQ65470@perrin.int.nxad.com> <3EE6F918.1C1FF28E@mindspring.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <3EE6F918.1C1FF28E@mindspring.com> X-PGP-Key: finger seanc@FreeBSD.org X-PGP-Fingerprint: 3849 3760 1AFE 7B17 11A0 83A6 DD99 E31F BC84 B341 X-Web-Homepage: http://sean.chittenden.org/ User-Agent: Mutt/1.5.4i cc: freebsd-performance@freebsd.org cc: Eric Anderson Subject: Re: Slow disk write speeds over network X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Jun 2003 20:55:51 -0000 > > net.inet.tcp.sendspace=65536 > > Double the default. Might not be a good idea, unless you have a ton > of memory. You will potentially use 64K send + 64K receive times > number of sockets. Assuming 4G and near-perfect tuning, you will be > limited to 16384 simultaneous connections fully packed before memory > pressure causes your machine to crash. I tend to like smaller > buffers and more connections. If you only have 512M, drop this > number to 2048 simultaneous connections if all buffers are full. 4GB of memory in this box and it's not directly available to the Internet: a very important point to take into note when tuning. > > kern.maxfiles=65536 > > Seems kind of overkill for the number of connections you can support > without overcommit, and the number of client machines you say you > have. This machine has a busy database on it too. :) > > net.inet.udp.recvspace=65536 > > net.inet.udp.maxdgram=57344 > > These are important for UDP NFS. I do not reccomend it. Other than I do use UDP NFS, so I do recommend these values. :) The relevance of the UDP vs TCP NFS should be kept to the archives, we don't need to re-open that bikeshed. :) > > vfs.nfs.async=1 > > This is very dangerous, if you care about your data. It permits > NFS to ACK writes before they have been committed to stable > storage. With a large enough window size, this should not be > necessary. It's just as dangerous as write caching, same argument for/against write caching, this is no worse. > > net.inet.udp.log_in_vain=1 > > This is just overhead; I reccomend turning it off. Ehh.... there's something floating around someplace where I've been getting errant NFS UDP packets and I haven't figured out what's going on... this is useful for debugging in the absence of a firewall. > > net.inet.icmp.icmplim=20000 > > This is only useful for TCP; but it can be very useful. Basically, > this is "connection rate limiting". If you have a ton of clients, > or trying to "netbench" the system, then set this number up. For > 100 NFS clients, it likely does not matter. I beg to differ... any ideas on how you throttle UDP connections without ICMP? A busy UDP NFS server will require this to be set higher. -sc -- Sean Chittenden From owner-freebsd-performance@FreeBSD.ORG Wed Jun 11 16:37:58 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4349737B401 for ; Wed, 11 Jun 2003 16:37:58 -0700 (PDT) Received: from sherryl.salk.edu (sherryl.snl.salk.edu [198.202.70.151]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9CFC843FBF for ; Wed, 11 Jun 2003 16:37:55 -0700 (PDT) (envelope-from cadams@salk.edu) Received: from malacarne.salk.edu (malacarne.snl.salk.edu [198.202.70.215]) by sherryl.salk.edu (8.12.9/8.12.9) with ESMTP id h5BNbrMw041728 for ; Wed, 11 Jun 2003 16:37:53 -0700 (PDT) Received: from cadams by malacarne.salk.edu with local (Exim 3.35 #1 (Debian)) id 19QFAB-0005tJ-00 for ; Wed, 11 Jun 2003 16:37:55 -0700 Date: Wed, 11 Jun 2003 16:37:55 -0700 To: freebsd-performance@freebsd.org Message-ID: <20030611233755.GA22453@salk.edu> References: <3EE4E156.6030603@centtech.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <3EE4E156.6030603@centtech.com> User-Agent: Mutt/1.5.4i From: Chris Adams Subject: Re: Slow disk write speeds over network X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Jun 2003 23:37:58 -0000 On Mon, Jun 09, 2003 at 02:34:46PM -0500, Eric Anderson wrote: > However, ftping a file or cp'ing over NFS shows speeds less than 1MB/s - > which is not what I would expect - it's got gigabit ethernet, and has > very fast read times (via ftp or nfs) over the network. Silly question - how are the switch ports configured? I'd try toggling it from whatever it it is now. We've seen some similar sounding problems with Intel NICs and Cisco switches where either auto or forced port settings will cause extremely slow transfers without causing any errors on either end. It's card/port specific, too, as we've seen cases where allegedly identical hardware must be configured as the opposite of its "twin". Chris From owner-freebsd-performance@FreeBSD.ORG Wed Jun 11 18:03:50 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 91B7F37B401 for ; Wed, 11 Jun 2003 18:03:50 -0700 (PDT) Received: from perrin.int.nxad.com (internal.ext.nxad.com [69.1.70.251]) by mx1.FreeBSD.org (Postfix) with ESMTP id 024A543FD7 for ; Wed, 11 Jun 2003 18:03:50 -0700 (PDT) (envelope-from hmp@nxad.com) Received: by perrin.int.nxad.com (Postfix, from userid 1072) id 6EFA520F01; Wed, 11 Jun 2003 18:03:49 -0700 (PDT) Date: Wed, 11 Jun 2003 18:03:49 -0700 From: Hiten Pandya To: Terry Lambert Message-ID: <20030612010349.GA23018@perrin.int.nxad.com> References: <20030609211526.58641.qmail@web14908.mail.yahoo.com> <3EE4FAED.6090603@centtech.com> <3EE595D2.B223CA19@mindspring.com> <3EE5F8DE.30001@centtech.com> <3EE7021E.F2928B7@mindspring.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <3EE7021E.F2928B7@mindspring.com> X-Operating-System: FreeBSD FreeBSD 4.7-STABLE User-Agent: Mutt/1.5.4i cc: freebsd-performance@freebsd.org cc: Eric Anderson Subject: Re: Slow disk write speeds over network X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jun 2003 01:03:50 -0000 On Wed, Jun 11, 2003 at 03:19:10AM -0700, Terry Lambert wrote: > > Ok - done.. some where defaults, and I couldn't find net.isr.enable.. > > Did I need to config something on my kernel for it to show up? > > You have to set a compile option; look in /usr/src/sys/net; grep > for "netisr_dispatch" or just "dispatch". Terry, ``NetISR Dispatch'' and friends are not available in FreeBSD 4.x versions. I am sure that he is using 4.x and that's why he can't find it in his source. > The net.isr.enable=1 will save you about 100ms per packet, > minimum, and more if you have a high interrupt overhead that > livelocks you from running NETISR. What it does is turns on > direct processing by IP and TCP of packets as they come in > the interface and you get the interrupt. Combined with soft > interrupt coelescing and polling, they should give you > another 1/3 of the receiver livelock fixup. The final third > isn't available, unless you are willing to hack network stack > code and scheduler code, since FreeBSD doesn't include LRP or > Weighted Fair Share Queuing. Does it help if queueing discipline is changed? > > which should be good enough cards to support it.. I've also put 100Mbit > > cards in place of the gige's for now to make sure I wasn't hitting a > > GigE problem or negotiation problem.. > > You should grep for DEVICE_POLLING in the network device > drivers you are interested in using to see if they have the > support. Also you can get up to 15% by adding soft interrupt > coelescing code, if the driver doesn't already support it (I > added it for a couple of drivers, and it was committed after > the benchmarks showed it was good, but it's not everywhere); > the basic idea is you take the interrupt, run rx_eof(), and > call ether_input(). Then repeat the process until you hit > some count limit, or until there's no more data. The direct > dispatch (net.isr.enable) combined with that will process most > packet trains to completion at interrupt, saving you 10ms up > and 10ms back down per packet exchange (NETISR only runs on > exit from spl or at the HZ time, which is default every 10ms). Hmm, can you point these drivers to me? Cheers. -- Hiten (hmp@FreeBSD.ORG) From owner-freebsd-performance@FreeBSD.ORG Thu Jun 12 01:47:14 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7971637B401 for ; Thu, 12 Jun 2003 01:47:14 -0700 (PDT) Received: from bluejay.mail.pas.earthlink.net (bluejay.mail.pas.earthlink.net [207.217.120.218]) by mx1.FreeBSD.org (Postfix) with ESMTP id BDDA643FA3 for ; Thu, 12 Jun 2003 01:47:13 -0700 (PDT) (envelope-from tlambert2@mindspring.com) Received: from user-38lc12n.dialup.mindspring.com ([209.86.4.87] helo=mindspring.com) by bluejay.mail.pas.earthlink.net with asmtp (SSLv3:RC4-MD5:128) (Exim 3.33 #1) id 19QNjc-0002Cj-00; Thu, 12 Jun 2003 01:47:05 -0700 Message-ID: <3EE82E16.99672F49@mindspring.com> Date: Thu, 12 Jun 2003 00:39:02 -0700 From: Terry Lambert X-Mailer: Mozilla 4.79 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Hiten Pandya References: <20030609211526.58641.qmail@web14908.mail.yahoo.com> <3EE4FAED.6090603@centtech.com> <3EE595D2.B223CA19@mindspring.com> <3EE5F8DE.30001@centtech.com> <3EE7021E.F2928B7@mindspring.com> <20030612010349.GA23018@perrin.int.nxad.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-ELNK-Trace: b1a02af9316fbb217a47c185c03b154d40683398e744b8a4eb820347f7d4f842928140926f4fb6cc350badd9bab72f9c350badd9bab72f9c350badd9bab72f9c cc: freebsd-performance@freebsd.org cc: Eric Anderson Subject: Re: Slow disk write speeds over network X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jun 2003 08:47:14 -0000 Hiten Pandya wrote: > ``NetISR Dispatch'' and friends are not available in > FreeBSD 4.x versions. I am sure that he is using 4.x and > that's why he can't find it in his source. Actually, that hadn't occurred to me. 8-). > > The net.isr.enable=1 will save you about 100ms per packet, > > minimum, and more if you have a high interrupt overhead that > > livelocks you from running NETISR. What it does is turns on > > direct processing by IP and TCP of packets as they come in > > the interface and you get the interrupt. Combined with soft > > interrupt coelescing and polling, they should give you > > another 1/3 of the receiver livelock fixup. The final third > > isn't available, unless you are willing to hack network stack > > code and scheduler code, since FreeBSD doesn't include LRP or > > Weighted Fair Share Queuing. > > Does it help if queueing discipline is changed? No. The WFQ code is not for packet queueing, it's for the scheduling of user processes that are servicing the data that comes in over the network. Looking at the "QLinux" web site would probably be informative. Note that I've been advocating these approaches before the QLinux project had a web site up. 8-). The problem here is that if you spend all your time handling interrupts, you never run NETISR; if you spend all your time handling interrupts AND running NETISR, you never run the user space program that's supposed to take the data out of the socket buffer. > > You should grep for DEVICE_POLLING in the network device > > drivers you are interested in using to see if they have the > > support. Also you can get up to 15% by adding soft interrupt > > coelescing code, if the driver doesn't already support it (I > > added it for a couple of drivers, and it was committed after > > the benchmarks showed it was good, but it's not everywhere); > > the basic idea is you take the interrupt, run rx_eof(), and > > call ether_input(). Then repeat the process until you hit > > some count limit, or until there's no more data. The direct > > dispatch (net.isr.enable) combined with that will process most > > packet trains to completion at interrupt, saving you 10ms up > > and 10ms back down per packet exchange (NETISR only runs on > > exit from spl or at the HZ time, which is default every 10ms). > > Hmm, can you point these drivers to me? Look in if_dc.c, around line 3067 in dc_intr(), for one: if (curpkts == ifp->if_ipackets) { while(dc_rx_resync(sc)) dc_rxeof(sc); } Also if_pcn.c, if_rl.c, if_sf.c, if_sis.c, if_sk.c, etc. ...just look for a status-check loop that calls the *_rxeof() function in the *_intr() function; if it's there, that's a software coelescing driver. -- Terry From owner-freebsd-performance@FreeBSD.ORG Thu Jun 12 01:48:09 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0F61637B401 for ; Thu, 12 Jun 2003 01:48:09 -0700 (PDT) Received: from bluejay.mail.pas.earthlink.net (bluejay.mail.pas.earthlink.net [207.217.120.218]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5076043FAF for ; Thu, 12 Jun 2003 01:48:08 -0700 (PDT) (envelope-from tlambert2@mindspring.com) Received: from user-38lc12n.dialup.mindspring.com ([209.86.4.87] helo=mindspring.com) by bluejay.mail.pas.earthlink.net with asmtp (SSLv3:RC4-MD5:128) (Exim 3.33 #1) id 19QNkb-0002Lo-00; Thu, 12 Jun 2003 01:48:06 -0700 Message-ID: <3EE82E16.99672F49@mindspring.com> Date: Thu, 12 Jun 2003 00:39:02 -0700 From: Terry Lambert X-Mailer: Mozilla 4.79 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Hiten Pandya References: <20030609211526.58641.qmail@web14908.mail.yahoo.com> <3EE4FAED.6090603@centtech.com> <3EE595D2.B223CA19@mindspring.com> <3EE5F8DE.30001@centtech.com> <3EE7021E.F2928B7@mindspring.com> <20030612010349.GA23018@perrin.int.nxad.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-ELNK-Trace: b1a02af9316fbb217a47c185c03b154d40683398e744b8a4eb820347f7d4f842e817cb048700a791350badd9bab72f9c350badd9bab72f9c350badd9bab72f9c cc: freebsd-performance@freebsd.org cc: Eric Anderson Subject: Re: Slow disk write speeds over network X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jun 2003 08:48:09 -0000 Hiten Pandya wrote: > ``NetISR Dispatch'' and friends are not available in > FreeBSD 4.x versions. I am sure that he is using 4.x and > that's why he can't find it in his source. Actually, that hadn't occurred to me. 8-). > > The net.isr.enable=1 will save you about 100ms per packet, > > minimum, and more if you have a high interrupt overhead that > > livelocks you from running NETISR. What it does is turns on > > direct processing by IP and TCP of packets as they come in > > the interface and you get the interrupt. Combined with soft > > interrupt coelescing and polling, they should give you > > another 1/3 of the receiver livelock fixup. The final third > > isn't available, unless you are willing to hack network stack > > code and scheduler code, since FreeBSD doesn't include LRP or > > Weighted Fair Share Queuing. > > Does it help if queueing discipline is changed? No. The WFQ code is not for packet queueing, it's for the scheduling of user processes that are servicing the data that comes in over the network. Looking at the "QLinux" web site would probably be informative. Note that I've been advocating these approaches before the QLinux project had a web site up. 8-). The problem here is that if you spend all your time handling interrupts, you never run NETISR; if you spend all your time handling interrupts AND running NETISR, you never run the user space program that's supposed to take the data out of the socket buffer. > > You should grep for DEVICE_POLLING in the network device > > drivers you are interested in using to see if they have the > > support. Also you can get up to 15% by adding soft interrupt > > coelescing code, if the driver doesn't already support it (I > > added it for a couple of drivers, and it was committed after > > the benchmarks showed it was good, but it's not everywhere); > > the basic idea is you take the interrupt, run rx_eof(), and > > call ether_input(). Then repeat the process until you hit > > some count limit, or until there's no more data. The direct > > dispatch (net.isr.enable) combined with that will process most > > packet trains to completion at interrupt, saving you 10ms up > > and 10ms back down per packet exchange (NETISR only runs on > > exit from spl or at the HZ time, which is default every 10ms). > > Hmm, can you point these drivers to me? Look in if_dc.c, around line 3067 in dc_intr(), for one: if (curpkts == ifp->if_ipackets) { while(dc_rx_resync(sc)) dc_rxeof(sc);X-Mozilla-Status: 0009 } Also if_pcn.c, if_rl.c, if_sf.c, if_sis.c, if_sk.c, etc. ...just look for a status-check loop that calls the *_rxeof() function in the *_intr() function; if it's there, that's a software coelescing driver. -- Terry From owner-freebsd-performance@FreeBSD.ORG Thu Jun 12 01:48:34 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 39BA037B401 for ; Thu, 12 Jun 2003 01:48:34 -0700 (PDT) Received: from bluejay.mail.pas.earthlink.net (bluejay.mail.pas.earthlink.net [207.217.120.218]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9B36E43FA3 for ; Thu, 12 Jun 2003 01:48:33 -0700 (PDT) (envelope-from tlambert2@mindspring.com) Received: from user-38lc12n.dialup.mindspring.com ([209.86.4.87] helo=mindspring.com) by bluejay.mail.pas.earthlink.net with asmtp (SSLv3:RC4-MD5:128) (Exim 3.33 #1) id 19QNl1-0002NO-00; Thu, 12 Jun 2003 01:48:31 -0700 Message-ID: <3EE83E1E.BB3E9865@mindspring.com> Date: Thu, 12 Jun 2003 01:47:26 -0700 From: Terry Lambert X-Mailer: Mozilla 4.79 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Hiten Pandya References: <20030609211526.58641.qmail@web14908.mail.yahoo.com> <3EE4FAED.6090603@centtech.com> <3EE595D2.B223CA19@mindspring.com> <3EE5F8DE.30001@centtech.com> <3EE7021E.F2928B7@mindspring.com> <20030612010349.GA23018@perrin.int.nxad.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-ELNK-Trace: b1a02af9316fbb217a47c185c03b154d40683398e744b8a4eb820347f7d4f8421b78ccc62ee4b684350badd9bab72f9c350badd9bab72f9c350badd9bab72f9c cc: freebsd-performance@freebsd.org cc: Eric Anderson Subject: Re: Slow disk write speeds over network X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jun 2003 08:48:34 -0000 Hiten Pandya wrote: > ``NetISR Dispatch'' and friends are not available in > FreeBSD 4.x versions. I am sure that he is using 4.x and > that's why he can't find it in his source. Actually, that hadn't occurred to me. 8-). > > The net.isr.enable=1 will save you about 100ms per packet, > > minimum, and more if you have a high interrupt overhead that > > livelocks you from running NETISR. What it does is turns on > > direct processing by IP and TCP of packets as they come in > > the interface and you get the interrupt. Combined with soft > > interrupt coelescing and polling, they should give you > > another 1/3 of the receiver livelock fixup. The final third > > isn't available, unless you are willing to hack network stack > > code and scheduler code, since FreeBSD doesn't include LRP or > > Weighted Fair Share Queuing. > > Does it help if queueing discipline is changed? No. The WFQ code is not for packet queueing, it's for the scheduling of user processes that are servicing the data that comes in over the network. Looking at the "QLinux" web site would probably be informative. Note that I've been advocating these approaches before the QLinux project had a web site up. 8-). The problem here is that if you spend all your time handling interrupts, you never run NETISR; if you spend all your time handling interrupts AND running NETISR, you never run the user space program that's supposed to take the data out of the socket buffer. > > You should grep for DEVICE_POLLING in the network device > > drivers you are interested in using to see if they have the > > support. Also you can get up to 15% by adding soft interrupt > > coelescing code, if the driver doesn't already support it (I > > added it for a couple of drivers, and it was committed after > > the benchmarks showed it was good, but it's not everywhere); > > the basic idea is you take the interrupt, run rx_eof(), and > > call ether_input(). Then repeat the process until you hit > > some count limit, or until there's no more data. The direct > > dispatch (net.isr.enable) combined with that will process most > > packet trains to completion at interrupt, saving you 10ms up > > and 10ms back down per packet exchange (NETISR only runs on > > exit from spl or at the HZ time, which is default every 10ms). > > Hmm, can you point these drivers to me? Look in if_dc.c, around line 3067 in dc_intr(), for one: if (curpkts == ifp->if_ipackets) { while(dc_rx_resync(sc)) dc_rxeof(sc);X-MX-Mozilla-Status: 0009 } Also if_pcn.c, if_rl.c, if_sf.c, if_sis.c, if_sk.c, etc. ...just look for a status-check loop that calls the *_rxeof() function in the *_intr() function; if it's there, that's a software coelescing driver. -- Terry From owner-freebsd-performance@FreeBSD.ORG Thu Jun 12 21:55:12 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C444F37B401 for ; Thu, 12 Jun 2003 21:55:12 -0700 (PDT) Received: from freebsd-bg.org (FreeBSD-BG.org [212.36.9.164]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0C6C843F85 for ; Thu, 12 Jun 2003 21:55:12 -0700 (PDT) (envelope-from petko@freebsd-bg.org) Received: by freebsd-bg.org (Postfix, from userid 1006) id 9CE5C59F4; Fri, 13 Jun 2003 07:55:36 +0300 (EEST) Date: Fri, 13 Jun 2003 07:55:37 +0300 From: Petko Popadiyski To: performance@freebsd.org Message-ID: <20030613045537.GA4781@freebsd-bg.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.4i Subject: JBoss with hotspot and jdk 1.3.1 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Jun 2003 04:55:13 -0000 Hello, I need a jboss and SwiftMQ(swiftmq.com - JMS enterprise messaging platform) but i found that the hotspot VM with jdk 1.3.1 is not very stable under freebsd. can i tweak it enough for a production server ( 4.8 ) and how? -- Best wishes, Petko Popadiyski ICQ: 59468934 From owner-freebsd-performance@FreeBSD.ORG Fri Jun 13 15:02:57 2003 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 33C2A37B401 for ; Fri, 13 Jun 2003 15:02:57 -0700 (PDT) Received: from ns1.xcllnt.net (209-128-86-226.BAYAREA.NET [209.128.86.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id 57A7243FE0 for ; Fri, 13 Jun 2003 15:02:56 -0700 (PDT) (envelope-from marcel@xcllnt.net) Received: from athlon.pn.xcllnt.net (athlon.pn.xcllnt.net [192.168.4.3]) by ns1.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5DM2uhS035720 for ; Fri, 13 Jun 2003 15:02:56 -0700 (PDT) (envelope-from marcel@piii.pn.xcllnt.net) Received: from athlon.pn.xcllnt.net (localhost [127.0.0.1]) by athlon.pn.xcllnt.net (8.12.9/8.12.9) with ESMTP id h5DM2uLg001185 for ; Fri, 13 Jun 2003 15:02:56 -0700 (PDT) (envelope-from marcel@athlon.pn.xcllnt.net) Received: (from marcel@localhost) by athlon.pn.xcllnt.net (8.12.9/8.12.9/Submit) id h5DM2tBQ001184 for performance@FreeBSD.org; Fri, 13 Jun 2003 15:02:55 -0700 (PDT) Date: Fri, 13 Jun 2003 15:02:55 -0700 From: Marcel Moolenaar To: performance@FreeBSD.org Message-ID: <20030613220255.GA1100@athlon.pn.xcllnt.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.4i Subject: lmbench results of pluto1.freebsd.org X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Jun 2003 22:02:57 -0000 Gang, I ran lmbench on pluto1.freebsd.org and the results are: L M B E N C H 1 . 0 S U M M A R Y ------------------------------------ Processor, Processes - times in microseconds -------------------------------------------- Host OS Mhz Null Null Simple /bin/sh Mmap 2-proc 8-proc Syscall Process Process Process lat ctxsw ctxsw --------- ------------- ---- ------- ------- ------- ------- ---- ------ ------ pluto1 FreeBSD 5.1-C 128846 3 1K 3K 6K 13 3 3 *Local* Communication latencies in microseconds ----------------------------------------------- Host OS Pipe UDP RPC/ TCP RPC/ UDP TCP --------- ------------- ------- ------- ------- ------- ------- pluto1 FreeBSD 5.1-C 13 39 66 45 84 *Local* Communication bandwidths in megabytes/second ---------------------------------------------------- Host OS Pipe TCP File Mmap Bcopy Bcopy Mem Mem reread reread (libc) (hand) read write --------- ------------- ---- ---- ------ ------ ------ ------ ---- ----- pluto1 FreeBSD 5.1-C 732 51 447 425 496 383 542 595 Memory latencies in nanoseconds (WARNING - may not be correct, check graphs) -------------------------------------------- Host OS Mhz L1 $ L2 $ Main mem Guesses --------- ------------- --- ---- ---- -------- ------- pluto1 FreeBSD 5.1-C 128846 2 6 124 The raw results file can be found here: http://people.freebsd.org/~marcel/lmbench/pluto1 Kernel did not have INVARIANT* nor WITNESS* May we should collect cross-architecture statistics per FreeBSD release? FYI, -- Marcel Moolenaar USPA: A-39004 marcel@xcllnt.net