From owner-freebsd-fs@FreeBSD.ORG Wed Sep 15 20:00:42 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DF74A1065696; Wed, 15 Sep 2010 20:00:42 +0000 (UTC) (envelope-from prvs=1874f602db=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id F2B7E8FC0A; Wed, 15 Sep 2010 20:00:41 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Wed, 15 Sep 2010 21:00:36 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Wed, 15 Sep 2010 21:00:36 +0100 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 by mail1.multiplay.co.uk (MDaemon PRO v10.0.4) with ESMTP id md50011249987.msg; Wed, 15 Sep 2010 21:00:36 +0100 X-Authenticated-Sender: Killing@multiplay.co.uk X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=1874f602db=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk Message-ID: <3F29E8CED7B24805B2D93F62A4EC9559@multiplay.co.uk> From: "Steven Hartland" To: "Andriy Gapon" References: <5DB6E7C798E44D33A05673F4B773405E@multiplay.co.uk><4C85E91E.1010602@icyb.net.ua><4C873914.40404@freebsd.org><20100908084855.GF2465@deviant.kiev.zoral.com.ua><4C874F00.3050605@freebsd.org><4C8D087B.5040404@freebsd.org><03537796FAB54E02959E2D64FC83004F@multiplay.co.uk><4C8D280F.3040803@freebsd.org><3FBF66BF11AA4CBBA6124CA435A4A31B@multiplay.co.uk><4C8E4212.30000@freebsd.org> <4C90B4C8.90203@freebsd.org> <6DFACB27CA8A4A22898BC81E55C4FD36@multiplay.co.uk> <4C90D3A1.7030008@freebsd.org> <0B1A90A08DFE4ADA9540F9F3846FDF38@multiplay.co.uk> <4C90EDB8.3040709@freebsd.org> Date: Wed, 15 Sep 2010 21:00:38 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="utf-8"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.5931 Cc: freebsd-fs@freebsd.org, jhell , Pawel Jakub Dawidek Subject: Re: zfs very poor performance compared to ufs due to lack of cache? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Sep 2010 20:00:43 -0000 ----- Original Message ----- From: "Andriy Gapon" > Or even try the opposite, if your version of ZFS permits it. > You can set primarycache=metadata on the filesystem where you have the data that > you serve via sendfile. With that setting it shouldn't get cached in ARC, but it > should be still cached in VM cache, so you should get UFS-like behavior. > > Will you test it? :) Ok given this a whirl, don't have the full results just yet but does seem that buf cache is not used at all? Mem: 32M Active, 1378M Inact, 159M Wired, 120K Cache, 21M Buf, 5348M Free Swap: 4096M Total, 4096M Free It also appears to have totally destroyed overall disk IO performance, particularly for small reads e.g. those used in cat. I just tried to cat a 1.8GB file from disk to /dev/null which I would usually expect to ~170MB/s on initial read but I only to got 1.8MB/s even though the disk subsystem was doing 200MB/s to sustain this request. Massive over read which is then just thrown away because there is no data cache? This seems backed up by using dd with a block size over 128K (zfs block size I believe?) results in normal performance of 180MB/s but repeat runs still only get the same, where on a similar box with ufs I see 1.3GB/s on repeat runs. Going to pull the test there as the machines struggling to keep up with even 10 clients. Next test I think should be sendfile on but with no special zfs loader.conf options to see what really happens to arc without any limits. I've got a suspicion that I may end up with close to or zero arc due to inact memory use. Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk.