From owner-freebsd-questions@FreeBSD.ORG Wed Aug 7 02:48:19 2013 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id CF624CA5 for ; Wed, 7 Aug 2013 02:48:19 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Received: from mail-ob0-x22f.google.com (mail-ob0-x22f.google.com [IPv6:2607:f8b0:4003:c01::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 9E27A2062 for ; Wed, 7 Aug 2013 02:48:19 +0000 (UTC) Received: by mail-ob0-f175.google.com with SMTP id xn12so2609244obc.6 for ; Tue, 06 Aug 2013 19:48:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:date:message-id:subject:from:to:content-type; bh=2KiSXYejwUg+pZwqPqgBnB1CKFzcJBDxkBYxtWSpbaY=; b=XL/wmm8xo8p0x/ZwEN2POsehYFovM75vOa26qYjlma672PfpoE/qA8pUxD3/6kO2T0 Jg9H1jBEeU0sqtTmLk76yR107aa+HI+aCYp+7oDw0xIUkMlz6T/u7vH7erus88l8VKiX eQhiOfB96tZ7cMiBi8zieiLf2fYzgte2or8Ea+OHP0QBlvzVWff1mN22lK/1Nm5DQ89f aWVwfYJwnhsET3FnQfRND4N8WyTa3XdpylPIqn2xUcBX0Al3BcL5rpRPfd7ny86FcX4r fv2iKAXtEwdVNY1baxwACa/GE5SeoSOFIC7wC3R/jBNl4W0WYgo2eLHTgjDfvW4M3giq F1ew== MIME-Version: 1.0 X-Received: by 10.42.199.5 with SMTP id eq5mr87318icb.1.1375843698862; Tue, 06 Aug 2013 19:48:18 -0700 (PDT) Sender: jdavidlists@gmail.com Received: by 10.42.150.196 with HTTP; Tue, 6 Aug 2013 19:48:18 -0700 (PDT) Date: Tue, 6 Aug 2013 22:48:18 -0400 X-Google-Sender-Auth: C2ZEVivH-OvOK3f2o484eQaZ_wo Message-ID: Subject: Terrible disk performance with LSI / FreeBSD 9.2-RC1 From: J David To: freebsd-questions@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Aug 2013 02:48:20 -0000 We have a machine running 9.2-RC1 that's getting terrible disk I/O performance. Its performance has always been pretty bad, but it didn't really become clear how bad until we did a zpool replace on one of the drives and realized it was going to take 3 weeks to rebuild a <1TB drive. The hardware specs are: - 2 x Xeon L5420 - 32 GiB RAM - LSI Logic SAS 1068E - 2 x 32GB SSD's - 6 x 1TB Western Digital RE3 7200RPM SATA The LSI controller has the most recent firmware I'm aware of (6.36.00.00 / 1.33.00.00 dated 2011.08.24), is in IT mode, and appears to be working fine: mpt0 Adapter: Board Name: USASLP-L8i Board Assembly: USASLP-L8i Chip Name: C1068E Chip Revision: B3 RAID Levels: none mpt0 Configuration: 0 volumes, 8 drives drive da0 (30G) ONLINE SATA drive da1 (29G) ONLINE SATA drive da2 (931G) ONLINE SATA drive da3 (931G) ONLINE SATA drive da4 (931G) ONLINE SATA drive da5 (931G) ONLINE SATA drive da6 (931G) ONLINE SATA drive da7 (931G) ONLINE SATA The eight drives are configured as ZIL, L2ARC on SSD and a six drive raidz2 on the spinning disks. We did a ZFS replace on the last drive in the line, and the resilver is proceeding at less than 800k/sec. extended device statistics device r/s w/s kr/s kw/s qlen svc_t %b da0 0.0 0.0 0.0 0.1 0 0.9 0 da1 0.0 8.2 0.0 19.9 0 0.1 0 da2 125.6 23.0 768.2 40.5 4 33.0 88 da3 126.6 23.1 769.0 41.3 4 32.3 89 da4 126.0 24.0 768.5 42.7 4 32.1 88 da5 125.9 22.0 768.2 40.1 4 31.6 87 da6 124.0 22.0 766.6 39.9 5 31.4 84 da7 0.0 136.9 0.0 801.3 0 0.6 4 The system has plenty of free RAM, is 99.7% idle, has nothing else going on, and runs like a one-legged dog. There are no error messages or any sign of a problem anywhere, other than the really terrible performance. (When not rebuilding, it does light NFS duty. That performance is similarly bad, but has never really mattered.) Similar systems running Solaris put out 10x these numbers claiming 30% busy instead of 90% busy. Does anyone have any suggestions for how I could troubleshoot this further? At this point, I'm kind of at a loss as to where to go from here. My goal is to try to phase out the Solaris machines, but this is kind of a roadblock. Thanks for any advice!