From owner-freebsd-performance@FreeBSD.ORG Fri Feb 18 15:03:50 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7F0DB16A4D1 for ; Fri, 18 Feb 2005 15:03:50 +0000 (GMT) Received: from mh2.centtech.com (moat3.centtech.com [207.200.51.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5338C43D5A for ; Fri, 18 Feb 2005 15:03:49 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from [10.177.171.220] (neutrino.centtech.com [10.177.171.220]) by mh2.centtech.com (8.13.1/8.13.1) with ESMTP id j1IF3m5c081260; Fri, 18 Feb 2005 09:03:48 -0600 (CST) (envelope-from anderson@centtech.com) Message-ID: <421603D0.3000403@centtech.com> Date: Fri, 18 Feb 2005 09:03:44 -0600 From: Eric Anderson User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.5) Gecko/20050210 X-Accept-Language: en-us, en MIME-Version: 1.0 To: David Rice References: <200502171636.10361.drice@globat.com> In-Reply-To: <200502171636.10361.drice@globat.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.82/708/Thu Feb 17 16:37:03 2005 on mh2.centtech.com X-Virus-Status: Clean cc: freebsd-performance@freebsd.org Subject: Re: High traffic NFS performance and availability problems X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Feb 2005 15:03:50 -0000 David Rice wrote: > We are a web hosting company that runs exclusivly on FreeBSD. We are having > storage availability and performance problems. All of our storage is exported > via NFS to the client machines Any suggestions or advice will be greatly > appreciated. We are willing to pay someone on a consulting basis to help us > solve these problems. Please email me off list if you are a consultant. I have lots of heavily abused NFS servers running FreeBSD (I usually have 400-500 3.XGHz P4's hammering each NFS server), so maybe I can help a bit. > On the file server side we have: > > Dell PowerEdge 1750's > Dell Perc4 RAID controller > Dell Power Vault 220 storage shelf. > (12) 146GB SCSI drives on one SCSI Bus with a hot spare (~1.3TB file system) > GigaBit ethernet to the NFS client machines > 1GB RAM > (2) 2.4 GHZ Xenon Processors > FreeBSD 5.2.1 or FreBSD 5.3 If you are on the 5.x branch, make sure you are running -STABLE (cvsup and buildworld if you can). Minimum 5.3-RELEASE. > On the client side we have: > > Dell PowerEdge 1750's > 1GB RAM > (2) 2.4 GHZ Xenon Processors > GigaBit ethernet > 36GB SCSI root disk > FreeBSD 4.9 to FreeBSD 4.11 > > On the network side we have: > Gigabit ethernet > Foundry BigIron and NetIron swithes > Cisco 6509 with Gigabit switch blades > > > Typicly we have 7 client boxes mounting storage from a single file server. > Each client box servers 1000 web sites and associate email. We have > done the basic NFS tuning (ie: Read write size optimization and kernel tuning) > > > The problems we are having is as follows. > > 1. Slow perfomance during peek traffic periods > 2. Client boxes have high load averages and sometimes crashes due to slow NFS > performance. > 3. File servers that randomly crash with "Fatal trap 12: page fault while in > kernel mode" > 4. With soft updates enabled during FSCK the fileserver will freeze with all > NFS processs in the "snaplck" state. We disabled soft updates because of > this. > > I can provide and other details about our configuration if needed. During peak performance, from the clients, how do you 'feel' the slowness? Does an 'ls' in the directory over NFS stall? How about an ls -al? What kind of authentication mechanism are you using? How many nfsd processes do you have running on the server? During peak performance, how many of those nfsd processes are in a state other than 'nfsd' (use top, ps, etc)? Eric -- ------------------------------------------------------------------------ Eric Anderson Sr. Systems Administrator Centaur Technology I have seen the future and it is just like the present, only longer. ------------------------------------------------------------------------