From owner-freebsd-stable@FreeBSD.ORG Sat Aug 14 03:18:09 2010 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B227110656A5 for ; Sat, 14 Aug 2010 03:18:09 +0000 (UTC) (envelope-from oberman@es.net) Received: from mailgw.es.net (mail1.es.net [IPv6:2001:400:201:1::2]) by mx1.freebsd.org (Postfix) with ESMTP id 971498FC1B for ; Sat, 14 Aug 2010 03:18:09 +0000 (UTC) Received: from ptavv.es.net (ptavv.es.net [IPv6:2001:400:910::29]) by mailgw.es.net (8.14.3/8.14.3) with ESMTP id o7E3I81e005349 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Fri, 13 Aug 2010 20:18:08 -0700 Received: from ptavv.es.net (localhost [127.0.0.1]) by ptavv.es.net (Tachyon Server) with ESMTP id 042A21CC3A; Fri, 13 Aug 2010 20:18:08 -0700 (PDT) To: Jeremy Chadwick In-reply-to: Your message of "Fri, 13 Aug 2010 16:29:54 PDT." <20100813232954.GA53935@icarus.home.lan> Date: Fri, 13 Aug 2010 20:18:08 -0700 From: "Kevin Oberman" Message-Id: <20100814031808.042A21CC3A@ptavv.es.net> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.0.10011, 1.0.148, 0.0.0000 definitions=2010-08-13_06:2010-08-14, 2010-08-13, 1970-01-01 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 ipscore=0 phishscore=0 bulkscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx engine=5.0.0-1005130000 definitions=main-1008130238 Cc: stable@freebsd.org Subject: Re: Inconsistent IO performance X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 14 Aug 2010 03:18:09 -0000 > Date: Fri, 13 Aug 2010 16:29:54 -0700 > From: Jeremy Chadwick > > On Fri, Aug 13, 2010 at 09:01:09AM -0700, Kevin Oberman wrote: > > For some time I have seen very odd issues with IO performance on > > 8-Stable. Going back to November of last year when 8.0 was released, I > > see variations of up to 22% in identical operations. This is not a > > degradation as the performance moves up and down. > > > > This is a very simplistic case. I have two identical disks (Fujitsu 80G) > > on a ThinkPad T43 with a 2 GHz CPU and 2G RAM. I run the command: > > dd bs=516096 if=/dev/ad0 of=/dev/ad2 > > > > I do this in single user mode immediately after a boot with no disks > > mounted for write. Just a 'boot -s', ,Enter> to start the shell, and the > > dd. I would expect very consistent performance from run to run, but I > > don't get it. Here are the results since 8.0 was released: > > Date Xfer rate Kernel date > > 12/4/09 19,242,573 Nov. 26 kernel (8.0-stable) > > 12/9/09 18,304,565 Dec. 6 kernel > > 12/17/0923,676,086 > > 1/5/10 18,648,609 > > 1/14/10 23,488,540 Jan. 6 kernel > > 1/21/10 19,551,680 Jan. 15 kernel > > 1/27/10 21,176,385 Jan. 21 kernel > > 2/5/10 22,387,745 > > 2/11/10 23,387,894 > > 2/17/10 20,412,172 Feb. 16 kernel > > 2/25/10 22,049,128 > > 3/4/10 22,099,624 Mar. 3 kernel > > 3/17/10 20,334,896 Mar. 3 kernel > > 3/31/10 21,655,213 Mar. 25 kernel > > 4/8/10 19,673,170 > > 4/14/10 22,235,518 > > 4/30/10 21,262,223 Apr. 14 kernel > > 6/3/10 22,838,125 May 24 kernel > > 6/17/10 18,481,270 > > 6/28/10 20,958,356 > > 7/8/10 19,698,282 June 28 kernel > > 7/21/10 23,330,556 > > 7/28/10 20,544,392 July 24 kernel (8.1-stable) > > 8/13/10 22,093,259 Aug. 9 kernel > > > > Note the dramatic differences even on the same kernel. For the December > > 6 kernel, for example, I see a maximum of 23,676,086 and a minimum of > > just 18,304,565. ???? > > > > Can anyone explain what might be causing such a dramatic difference? > > > > I should also note that the system was consistent back in V6 and V7 > > days. Consistently slow, but consistent. 17.5M was the norm in V6 and > > 18.0M in V7. The performance jumped to about 19M in March of 09 and jumped > > to its current speeds with 8.0. So performance has greatly improved to > > where the slowest times are better than the fastest prior to March of > > 09. Just very inconsistent. > > > > I don't know that anything is wrong, but I'd love to understand why this > > is happening. > > The system in question is a Thinkpad T43 laptop[1], which is from circa > 2005 and uses an ICH6-M southbridge (note the -M). We don't know the > exact model of Fujitsu hard disk used, but since it's a laptop my guess > is that it's 5400rpm, and PATA. They are Fujitsu MHV2080AH drives. 80G PATA. > > System drastically differs on laptops if being powered off the battery > vs. AC. Were the tests performed consistently with the exact same setup > (which: battery or AC?) every time? Given that the system role is a > laptop, I imagine not. I usually do my backups on battery, but there does not seem to be any correlation between power source and performance. In single user mode powerd. I have had fast times and slow times with both power sources. > > Can you also provide output from these commands? > > - atacontrol list ATA channel 0: Master: ad0 ATA/ATAPI revision 6 Slave: no device present ATA channel 1: Master: ad2 ATA/ATAPI revision 6 Slave: no device present > - atacontrol info ataX (where X is the channel number the ad0 drive > is connected to) Master: ad0 ATA/ATAPI revision 6 Slave: no device present Master: ad2 ATA/ATAPI revision 6 Slave: no device present > - atacontrol cap ad0 Protocol ATA/ATAPI revision 6 device model FUJITSU MHV2080AH serial number NT02T53258D4 firmware revision 00000096 cylinders 16383 heads 16 sectors/track 63 lba supported 156301488 sectors lba48 not supported dma supported overlap not supported Feature Support Enable Value Vendor write cache yes yes read ahead yes yes Tagged Command Queuing (TCQ) no no 0/0x00 SMART yes yes microcode download yes yes security yes no power management yes yes advanced power management yes yes 16512/0x4080 automatic acoustic management yes yes 254/0xFE 254/0xFE Oh, crap! I could have sworn I had turned off acoustic management. I'll do this ASAP and see if it helps. (I can't try tonight.) > Anyway, I would expect the system to be seeing 50-60MB/sec, but I'm > pulling those numbers out of thin air. An ICH6-M may be "old", but keep > reading for a comparison system that's even older... > > The deviation in your disk I/O isn't a major surprise (to me anyway), > given the system specs. What *does* surprise me is your abysmal I/O > speeds in general. 18MB/sec min, 24MB/sec max?! ICH6-M can do a lot > more than that. Something isn't right. > > It sounds to me like the disk itself has some kind of internal problem > (cache that's gone bad, something mechanical that isn't audible, etc.); > even for a 5400rpm drive those numbers are very low. Other > possibilities include a southbridge that's going bad, or some kind of > power-related problem that's causing the drive to spin at a lower speed > than 5400rpm (though SMART sometimes can notice this). I'm grasping at > straws with this one, but excessive dust can slow a system down (from > what I'm told by EE folks; something about more electricity being > required to push voltage across a trace...) > > Now the comparison -- here's a system that's way older than yours. > > - FreeBSD 6.4-STABLE, i386 > - Supermicro SuperServer 5010E [2] > - Intel Pentium 3 (not sure of speed) > - 1GB RAM > - Intel ICH2 southbridge > - ad0: Maxtor STM3160815A disk (160GB, 8MB cache, 7200rpm, ATA133) > - ad1: Maxtor STM3160815A disk (160GB, 8MB cache, 7200rpm, ATA133) > - Disk connected via ICH2 > - System in multi-user with some light load > - Command #1: dd if=/dev/ad0 of=/dev/null bs=64k count=100000 > - Command #2: dd if=/dev/ad1 of=/dev/null bs=64k count=100000 > - Commands run 4 times in succession > > Result for command #1: > > 6553600000 bytes transferred in 84.110282 secs (77916752 bytes/sec) > 6553600000 bytes transferred in 84.197506 secs (77836035 bytes/sec) > 6553600000 bytes transferred in 84.205020 secs (77829089 bytes/sec) > 6553600000 bytes transferred in 84.662426 secs (77408602 bytes/sec) > > Result for command #2: > > 6553600000 bytes transferred in 85.052923 secs (77053201 bytes/sec) > 6553600000 bytes transferred in 85.040286 secs (77064652 bytes/sec) > 6553600000 bytes transferred in 85.040805 secs (77064181 bytes/sec) > 6553600000 bytes transferred in 85.040560 secs (77064403 bytes/sec) > > My recommendation: start looking at replacing hardware. > > Start by replacing the hard disk with something newer and see what > becomes of that. If you want recommendations: if power draw, noise, and > thermals are a concern point for you, get a 5400rpm drive. Try to find > something with 16MB cache or more. The WD Scorpio Blue drives are > supposedly quite nice, but I don't know if they come in PATA. I'm really more interested in the inconsistency. I'll have this system for another 10 months, at most. I really don't see getting new drives. (It's getting pretty hard to fine PATA drives, and my next system will certainly be SATA, in any case.) > I'm not particularly fond of Fujitsu drives (my day job consists of > watching their SCSI disks freak out in random ways), so I'd choose to > stay away from those. Same goes for Toshiba (just recently replaced one > of those on a laptop; cache went bad, disk I/O was absurd). I've been using them (PATA only) for about 5 years with good results, though I would prefer Seagates or Hitachi's. I did have one die after about 10 months, but they replaced it very quickly with no hassles. > If you want something totally crazy to try, how about booting a FreeBSD > 7.2 or 7.3 LiveFS CD and doing your dd's? I can try this, but I doubt it will make any real difference. dd in single user should not need to the system for anything, once it loads the dd image. It is certainly not swapping or paging. And, why 7.2 or 7.3 and not 8.1? Thanks! I am amazed at the time you manage to spend helping with other people's problems with FreeBSD. -- R. Kevin Oberman, Network Engineer Energy Sciences Network (ESnet) Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab) E-mail: oberman@es.net Phone: +1 510 486-8634 Key fingerprint:059B 2DDF 031C 9BA3 14A4 EADA 927D EBB3 987B 3751