From owner-freebsd-fs@freebsd.org Thu Jan 19 07:56:12 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A4357CB76FD for ; Thu, 19 Jan 2017 07:56:12 +0000 (UTC) (envelope-from tobias.oberstein@gmail.com) Received: from mail-lf0-x22f.google.com (mail-lf0-x22f.google.com [IPv6:2a00:1450:4010:c07::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2B6131443 for ; Thu, 19 Jan 2017 07:56:12 +0000 (UTC) (envelope-from tobias.oberstein@gmail.com) Received: by mail-lf0-x22f.google.com with SMTP id k86so29502245lfi.0 for ; Wed, 18 Jan 2017 23:56:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=to:from:subject:message-id:date:user-agent:mime-version :content-transfer-encoding; bh=uLj4aHt8zIOP8C/k0CuSg0OzlhXbaJftZx85jOVpkO0=; b=F6w69LZy2GMVJ8ilj/Ih+edFYxNlVi3dmdhOCIkIInJQC66a5sRv52nN/ETr+oP7+Y /pwxMGzmZBmwAeq+W+e9OC18dNivp2afHhxyCh/PuPusKlTT384b9yz+Uw7pRmUXkAva dNLi2/JqU30T4rCXhVJW7Tl/uanelLm66p+GuJ0K3GvjXPjecJMTpkdABC9avEijcJKM tV4eD7IEKZuiV+/ZZV3rdtzCFFBEAXFnxfwXWfKSDLWsGo9HRg7j5QB/yPHW8I2giWeU zfTp31kThja4vpQcjMd6p7/4NuHWt4LTaduFOucpnTHnKW5Slbt4ZOnxBjrHdHzkntu0 AyCQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:to:from:subject:message-id:date:user-agent :mime-version:content-transfer-encoding; bh=uLj4aHt8zIOP8C/k0CuSg0OzlhXbaJftZx85jOVpkO0=; b=AWoUL2ia3sgKa2qk68tzD/VjG09HXY1C2k5miKRmrQkYA0hRDSS31l+1Z0IvUPkAGf tYGalILaVaJ3DFwN+6GdfOYaxZzXpcC8+pfThgXwbgjRgHLC9KkmTsch5hjaJ8B75dMz URDvtNz2tAxkfgPcgaVp4F++Uhs73J7oSFfYOIffbSZ3KjAsh27PmCKNgvXw9A9XCJTr EbyehS3lBIaccrKhNbrgRYTTTAX5u/Fzxx4WtFn6G351d7BUptEHFn3OH97a5OgLH41Q Q9W/m9FLcsE7PF3L/QqKu1fUGcYfB6j6LW4y1U0gDqxpc/7qXkFwGy34GDAGVksCqvO8 U9uA== X-Gm-Message-State: AIkVDXJEJezRTnVYllNoEyy4T6QhasBF1TE8526T/+sQyhDYNxQauCubffg0IGWa9xsXRA== X-Received: by 10.25.67.19 with SMTP id q19mr2536878lfa.9.1484812569207; Wed, 18 Jan 2017 23:56:09 -0800 (PST) Received: from [10.200.202.23] ([195.14.242.85]) by smtp.googlemail.com with ESMTPSA id 66sm1245561lfy.42.2017.01.18.23.56.08 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 18 Jan 2017 23:56:08 -0800 (PST) To: freebsd-fs@freebsd.org From: Tobias Oberstein Subject: stuck on low scrub performance Message-ID: Date: Thu, 19 Jan 2017 08:56:05 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.5.1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Jan 2017 07:56:12 -0000 Hi, I've got a 146TB RAID-Z2 magnetic platter pool which I am scrubbing, but performance is very poor: pool: tank state: ONLINE scan: scrub in progress since Tue Jan 17 11:13:29 2017 5.02T scanned out of 89.4T at 32.2M/s, (scan is slow, no estimated time) 0 repaired, 5.61% done I've already tried modifying various knobs, with no success (I changed oids while the scrub is running, without restarting - does that take effect?). I've tried to collect as much info as possible using this script https://gist.github.com/oberstet/c66baf777ea77ae843b64271df7420a3 with this output https://gist.github.com/oberstet/a8e12227486ebeb1e85f6f56b880ce7d --- My expectation from the general performance of this array would be scrubbing with 500MB/s (10x more) - this is what we regularily see on this array when it goes to the magnetic platters on sequential read loads. Any helps would be greatly appreciated, as at that speed, the scrubbing takes weeks =( Cheers, /Tobias