From owner-freebsd-stable@FreeBSD.ORG Sat Aug 14 02:12:37 2010 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 436691065672 for ; Sat, 14 Aug 2010 02:12:37 +0000 (UTC) (envelope-from oberman@es.net) Received: from mailgw.es.net (mail1.es.net [IPv6:2001:400:201:1::2]) by mx1.freebsd.org (Postfix) with ESMTP id 298FA8FC13 for ; Sat, 14 Aug 2010 02:12:37 +0000 (UTC) Received: from ptavv.es.net (ptavv.es.net [IPv6:2001:400:910::29]) by mailgw.es.net (8.14.3/8.14.3) with ESMTP id o7E2CZEv026920 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Fri, 13 Aug 2010 19:12:36 -0700 Received: from ptavv.es.net (localhost [127.0.0.1]) by ptavv.es.net (Tachyon Server) with ESMTP id A12771CC3A; Fri, 13 Aug 2010 19:12:35 -0700 (PDT) To: Roland Smith In-reply-to: Your message of "Fri, 13 Aug 2010 23:32:05 +0200." <20100813213205.GB29150@slackbox.erewhon.net> Date: Fri, 13 Aug 2010 19:12:35 -0700 From: "Kevin Oberman" Message-Id: <20100814021235.A12771CC3A@ptavv.es.net> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.0.10011, 1.0.148, 0.0.0000 definitions=2010-08-13_06:2010-08-14, 2010-08-13, 1970-01-01 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 ipscore=0 phishscore=0 bulkscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx engine=5.0.0-1005130000 definitions=main-1008130227 Cc: stable@freebsd.org Subject: Re: Inconsistent IO performance X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 14 Aug 2010 02:12:37 -0000 > Date: Fri, 13 Aug 2010 23:32:05 +0200 > From: Roland Smith > > On Fri, Aug 13, 2010 at 09:01:09AM -0700, Kevin Oberman wrote: > > For some time I have seen very odd issues with IO performance on > > 8-Stable. Going back to November of last year when 8.0 was released, I > > see variations of up to 22% in identical operations. This is not a > > degradation as the performance moves up and down. > > > > This is a very simplistic case. I have two identical disks (Fujitsu 80G) > > on a ThinkPad T43 with a 2 GHz CPU and 2G RAM. I run the command: > > dd bs=516096 if=/dev/ad0 of=/dev/ad2 > > Why are you using this peculiar block size? It was suggested by a co-worker with a similar system, but I have tried larger values with no clear difference. It is the size of one cylinder based on disk geometry. I was (and remain) sceptical since CHS has nothing to do with real geometry in modern disks. > > Note the dramatic differences even on the same kernel. For the December > > 6 kernel, for example, I see a maximum of 23,676,086 and a minimum of > > just 18,304,565. ???? > > Both figures seem quite low to me? I cannot exactly reproduce your test, > because I don't have an empty second disk handy, but doing > > dd if=/dev/zero bs=1m count=100 of=/tmp/foo > > yields the following writing speed on 8.1-RELEASE amd64, > WDC WD5001ABYS SATA harddisk @ 7200 rpm.: > > 1) 87263174 bytes/sec > 2) 87878728 bytes/sec > 3) 86397125 bytes/sec > 4) 86550094 bytes/sec > 5) 86524741 bytes/sec > > Th maximum variation in write speed is (87878728-86397125)/86397125*100% = > 1.7%, which doesn't seem that much to me. > > This is in multi-user, with X11 running but on an otherwise idling machine, > and with filesystem overhead to boot. Still the numbers are a lot higher than > yours, which puzzles me. > > Trying only reading does yield very inconsistent results because of caching, I > think; > > dd if=/tmp/foo bs=1m count=100 of=/dev/null > > 1) 1454216957 bytes/sec > 2) 1003691691 bytes/sec > 3) 1429956761 bytes/sec > 4) 2324794646 bytes/sec > 5) 1804563681 bytes/sec > > This is a (2324794646-1003691691)/1003691691*100% = 132% difference. OTOH, > your data set should be big enough to negate caching effects, I guess. :-) > > What this does show is that writing seems to be the bottleneck. > > If I both read from and write to a file (on the same disk & partition); > > dd if=/tmp/foo bs=1m count=100 of=/tmp/bar > > gives > > 1) 85161534 bytes/sec > 2) 84978770 bytes/sec > 3) 87966613 bytes/sec > 4) 83036312 bytes/sec > 5) 86536879 bytes/sec > > This is a (87966613-83036312)/83036312*100% = 5.9% difference between largest > and smallest. The speed seems to be dictated by the writing. > > > Can anyone explain what might be causing such a dramatic difference? > > Maybe there is a hardware component here? Are both disks on the same > controller? Or if not are both controllers using the same interrupt line? No. Each is on its one controller and is the only disk on that controller. > You should have a look at 'systat -vmstat' with dd running in the > background. That might give a clue as to where the bottleneck is. I guess I can give this a try. Thanks! -- R. Kevin Oberman, Network Engineer Energy Sciences Network (ESnet) Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab) E-mail: oberman@es.net Phone: +1 510 486-8634 Key fingerprint:059B 2DDF 031C 9BA3 14A4 EADA 927D EBB3 987B 3751