From owner-freebsd-stable@FreeBSD.ORG Sun Jan 17 23:21:21 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2C999106566B for ; Sun, 17 Jan 2010 23:21:21 +0000 (UTC) (envelope-from garrettmoore@gmail.com) Received: from mail-iw0-f177.google.com (mail-iw0-f177.google.com [209.85.223.177]) by mx1.freebsd.org (Postfix) with ESMTP id E1E188FC13 for ; Sun, 17 Jan 2010 23:21:20 +0000 (UTC) Received: by iwn7 with SMTP id 7so1721861iwn.7 for ; Sun, 17 Jan 2010 15:21:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type; bh=yewUqBfp/mAhdJIIQFiVWzhD84OAYIsNJBWFPJPmcrQ=; b=USjssp0CF3Y39THQqpqFhvuxkzXaz1DBr0+uikOL0ql2JaQKs1fjZ8NmOobB32VgRJ i7d9pmnrFzVaZKiM4pyu2pST+Rf6ET8X3M3Te8qxvOhSy/8BeLWuh89cf9AaOT5e7XV0 dvelhv3Ip6y8hYqmzXedKKSfWdsLCOWXM7BFw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=KMzFzUVwJszibTOoDmvnDX2AbBOQgIWuwloeMa3u/2bofuwbIfw845AXsazp7J+tBD v22MtV7xR4GZM9rNgv5ry5NmmCDjB10fyukHv3POH7y5ldWiqvIQ9p4/zGfn9+QTZHB7 wnHDpX0pkq+Aq1IzCHyQadVVLwB7X+FtJKD90= MIME-Version: 1.0 Received: by 10.231.59.7 with SMTP id j7mr3661296ibh.12.1263770480097; Sun, 17 Jan 2010 15:21:20 -0800 (PST) In-Reply-To: <7346c5c61001091706m45a3a2a5k3ca8bb0c4bec5ea8@mail.gmail.com> References: <7346c5c61001030842r7dc76199y51e4c1c90a3eea6e@mail.gmail.com> <7346c5c61001080831w375d158fu5b1996ee58cb0f8d@mail.gmail.com> <2ae8edf31001091051l156cb57alf549cfe06f1c7197@mail.gmail.com> <7346c5c61001091640w1a0ffb5y3ffe97cb88c76436@mail.gmail.com> <20100110005621.GA45015@icarus.home.lan> <7346c5c61001091706m45a3a2a5k3ca8bb0c4bec5ea8@mail.gmail.com> Date: Sun, 17 Jan 2010 18:21:20 -0500 Message-ID: <7346c5c61001171521w1ca4738w98e8fcca24643cda@mail.gmail.com> From: Garrett Moore To: freebsd-stable@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Re: ZFS performance degradation over time X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 17 Jan 2010 23:21:21 -0000 I upgraded my system to 8GB of ram to see if that would help. It hasn't made much of a difference. After having rTorrent running for a while, my performance again tanked. Around 6.5GB of memory was showing as 'Active' according to top. Copying a file from the zpool to another location on the zpool had the following performance (from zpool iostat 1): tank 2.57T 8.31T 11 215 273K 745K tank 2.57T 8.31T 22 97 816K 540K tank 2.57T 8.31T 6 67 13.5K 887K tank 2.57T 8.31T 5 101 639K 190K tank 2.57T 8.31T 14 81 1.14M 154K tank 2.57T 8.31T 6 152 12.5K 897K tank 2.57T 8.31T 11 153 143K 790K tank 2.57T 8.31T 3 172 134K 566K tank 2.57T 8.31T 3 105 7.48K 699K tank 2.57T 8.31T 5 136 138K 383K Combined read/write of <2MB/s -- pretty pathetic. I then tried Artem's trick, running ` perl -e '$x="x"x3000000000'` to force a swapout. This command completed in about 10 seconds, and I then had >5GB of memory showing as 'Free' according to top. Checking zpool iostat 1 again showed: tank 2.57T 8.31T 375 477 46.1M 57.9M tank 2.57T 8.31T 18 472 1.75M 44.8M tank 2.57T 8.31T 129 0 16.1M 0 tank 2.57T 8.31T 428 0 53.2M 0 tank 2.57T 8.31T 262 947 31.8M 103M tank 2.57T 8.30T 80 105 9.61M 196K tank 2.57T 8.30T 612 0 75.8M 0 tank 2.57T 8.30T 155 951 18.4M 103M tank 2.57T 8.30T 662 0 82.1M 0 tank 2.57T 8.30T 176 945 21.1M 103M Which is obviously much better, and a respectable rate of performance. This was all done during the same single `copy` command - I did not stop/restart the copy when running the perl oneliner. So it seems based on this that ZFS is keeping too much data cached and not being smart about when to release 'old' cache entries in favour of new ones. Any suggestions at this point? The only tuning I have done so far is to disable prefetch, since my primary usage is streaming HD media, and prefetch has been known to cause problems in that situation.