From owner-freebsd-questions@FreeBSD.ORG Wed Aug 7 19:15:50 2013 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id A3452698 for ; Wed, 7 Aug 2013 19:15:50 +0000 (UTC) (envelope-from jamesgosnell@gmail.com) Received: from mail-ea0-x22c.google.com (mail-ea0-x22c.google.com [IPv6:2a00:1450:4013:c01::22c]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 22E5C221B for ; Wed, 7 Aug 2013 19:15:49 +0000 (UTC) Received: by mail-ea0-f172.google.com with SMTP id r16so1017182ead.17 for ; Wed, 07 Aug 2013 12:15:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=iza1jNlM62LrrDp53FakvHoew6Y2pM4/KopalNq881o=; b=vCAJQ+CH0NvW4Jd/701VqgzhXt+rqC5+XemHiE8FKPs0EtoW9+F3gf/me+0iLkC+8k K4Vfhyrvt7kYo4AJB3uwuqwRwyaLnarbpDId/lBD/hrMILEyoK0my4iNbYZ+/DrVgPwe I5wUybtZEsd6It6BHvRUZb1hzXWO3tErM6O5Flu62mqBY5K9/Xbda4m3AGa/aPRlWXku kxogvvB+x42EFl3ION6vELqZTtg1e9nLVPuMe97Wyd46ks++lf+U6pdT+e4G93rhO8KW 1CJFGfvuelVUmSKZuOrWsZotljYpTvyAfMPcdpktsyJOeDNNUpZGzjnq/bnP+LZR+zxS w7qw== MIME-Version: 1.0 X-Received: by 10.14.9.129 with SMTP id 1mr4568198eet.59.1375902947297; Wed, 07 Aug 2013 12:15:47 -0700 (PDT) Received: by 10.223.195.196 with HTTP; Wed, 7 Aug 2013 12:15:47 -0700 (PDT) In-Reply-To: References: Date: Wed, 7 Aug 2013 14:15:47 -0500 Message-ID: Subject: Re: Terrible disk performance with LSI / FreeBSD 9.2-RC1 From: James Gosnell To: freebsd-questions@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Aug 2013 19:15:50 -0000 Maybe one of your drives is bad, so it's constantly doing error correction? On Tue, Aug 6, 2013 at 9:48 PM, J David wrote: > We have a machine running 9.2-RC1 that's getting terrible disk I/O > performance. Its performance has always been pretty bad, but it > didn't really become clear how bad until we did a zpool replace on one > of the drives and realized it was going to take 3 weeks to rebuild a > <1TB drive. > > The hardware specs are: > - 2 x Xeon L5420 > - 32 GiB RAM > - LSI Logic SAS 1068E > - 2 x 32GB SSD's > - 6 x 1TB Western Digital RE3 7200RPM SATA > > The LSI controller has the most recent firmware I'm aware of > (6.36.00.00 / 1.33.00.00 dated 2011.08.24), is in IT mode, and appears > to be working fine: > > mpt0 Adapter: > Board Name: USASLP-L8i > Board Assembly: USASLP-L8i > Chip Name: C1068E > Chip Revision: B3 > RAID Levels: none > > mpt0 Configuration: 0 volumes, 8 drives > drive da0 (30G) ONLINE SATA > drive da1 (29G) ONLINE SATA > drive da2 (931G) ONLINE SATA > drive da3 (931G) ONLINE SATA > drive da4 (931G) ONLINE SATA > drive da5 (931G) ONLINE SATA > drive da6 (931G) ONLINE SATA > drive da7 (931G) ONLINE SATA > > The eight drives are configured as ZIL, L2ARC on SSD and a six drive > raidz2 on the spinning disks. > > We did a ZFS replace on the last drive in the line, and the resilver > is proceeding at less than 800k/sec. > > extended device statistics > device r/s w/s kr/s kw/s qlen svc_t %b > da0 0.0 0.0 0.0 0.1 0 0.9 0 > da1 0.0 8.2 0.0 19.9 0 0.1 0 > da2 125.6 23.0 768.2 40.5 4 33.0 88 > da3 126.6 23.1 769.0 41.3 4 32.3 89 > da4 126.0 24.0 768.5 42.7 4 32.1 88 > da5 125.9 22.0 768.2 40.1 4 31.6 87 > da6 124.0 22.0 766.6 39.9 5 31.4 84 > da7 0.0 136.9 0.0 801.3 0 0.6 4 > > The system has plenty of free RAM, is 99.7% idle, has nothing else > going on, and runs like a one-legged dog. > > There are no error messages or any sign of a problem anywhere, other > than the really terrible performance. (When not rebuilding, it does > light NFS duty. That performance is similarly bad, but has never > really mattered.) > > Similar systems running Solaris put out 10x these numbers claiming 30% > busy instead of 90% busy. > > Does anyone have any suggestions for how I could troubleshoot this > further? At this point, I'm kind of at a loss as to where to go from > here. My goal is to try to phase out the Solaris machines, but this > is kind of a roadblock. > > Thanks for any advice! > _______________________________________________ > freebsd-questions@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to " > freebsd-questions-unsubscribe@freebsd.org" > -- James Gosnell, ACP