From owner-freebsd-questions@FreeBSD.ORG Sun Aug 5 15:45:18 2012 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D64A8106566B for ; Sun, 5 Aug 2012 15:45:18 +0000 (UTC) (envelope-from steve@sohara.org) Received: from uk1rly2283.eechost.net (relay01a.mail.uk1.eechost.net [217.69.40.75]) by mx1.freebsd.org (Postfix) with ESMTP id 693C18FC15 for ; Sun, 5 Aug 2012 15:45:18 +0000 (UTC) Received: from [31.186.37.179] (helo=rpi-1.marelmo.com) by uk1rly2283.eechost.net with esmtpa (Exim 4.72) (envelope-from ) id 1Sy2wU-0005fR-LG; Sun, 05 Aug 2012 16:40:34 +0100 Received: from [192.168.63.1] (helo=steve.marelmo.com) by rpi-1.marelmo.com with smtp (Exim 4.72) (envelope-from ) id 1Sy30p-0005Uk-Ju; Sun, 05 Aug 2012 16:45:03 +0100 Date: Sun, 5 Aug 2012 16:44:58 +0100 From: Steve O'Hara-Smith To: Wojciech Puchar Message-Id: <20120805164458.1c012ac4.steve@sohara.org> In-Reply-To: References: <20120804142758.0126fded.steve@sohara.org> X-Mailer: Sylpheed 3.1.3 (GTK+ 2.24.6; amd64-portbld-freebsd9.0) X-Face: %]+HVL}K`P8>+8ZcY-WGHP6j@&mxMo9JH6_WdgIgUGH)JX/usO0%jy7T~IVgqjumD^OBqX, Kv^-GM6mlw(fI^$"QRKyZ$?xx/ Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Auth-Info: 15567@permanet.ie (plain) Cc: freebsd-questions@freebsd.org Subject: Re: ZFS bonnie puzzlement X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Aug 2012 15:45:18 -0000 On Sun, 5 Aug 2012 13:29:51 +0200 (CEST) Wojciech Puchar wrote: > > are showing me. Read performance OTOH is strange, zpool and systat both > > reporting consistently an aggregated read speed of around 120MB/s during > > the block read tests (which seems a bit slow for the drives - and indeed > > systat reports the drives at less than 50% utilisation) but bonnie is > > only reporting 35MB/s, I see similar discrepancies with simple dd block > > reads to /dev/null, in which case my stopwatch agrees with dd. > no it is not wrong. > > Do more tests (possibly your own doing heavy mixed workload) to > understand well why you should not use this "last word in filesystems". First surprise, with only 4GB I had set primarycache=metadata, changing that to primarycache=all caused the systat, zpool iostat and bonnie figures all to agree - and made them all a bit better too. Lesson from this - don't bother setting primarycache=metadata. With that puzzle gone testing and tuning becomes more useful: Enabling prefetch made a huge difference to the per char sequential read, but didn't really change anything else. Indeed this test is now CPU limited in bonnie - that'll do. Rebooting with zfs.cache_flush_disable=1 made everything faster. Block writes and reads maxed out the discs at around 110MB/s and 200MB/s respectively - pretty close to the raw disc speed. Rewrite nearly doubled in speed too. Next stop NFS tuning. -- Steve O'Hara-Smith | Directable Mirror Arrays C:>WIN | A better way to focus the sun The computer obeys and wins. | licences available see You lose and Bill collects. | http://www.sohara.org/