From owner-freebsd-fs@FreeBSD.ORG Fri Feb 1 19:24:38 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 5C6C27E6 for ; Fri, 1 Feb 2013 19:24:38 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from vps.rulingia.com (host-122-100-2-194.octopus.com.au [122.100.2.194]) by mx1.freebsd.org (Postfix) with ESMTP id E7B77A73 for ; Fri, 1 Feb 2013 19:24:37 +0000 (UTC) Received: from server.rulingia.com (c220-239-236-213.belrs5.nsw.optusnet.com.au [220.239.236.213]) by vps.rulingia.com (8.14.5/8.14.5) with ESMTP id r11JONrl039865 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Sat, 2 Feb 2013 06:24:24 +1100 (EST) (envelope-from peter@rulingia.com) X-Bogosity: Ham, spamicity=0.000000 Received: from server.rulingia.com (localhost.rulingia.com [127.0.0.1]) by server.rulingia.com (8.14.5/8.14.5) with ESMTP id r11JOIN8027080 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 2 Feb 2013 06:24:18 +1100 (EST) (envelope-from peter@server.rulingia.com) Received: (from peter@localhost) by server.rulingia.com (8.14.5/8.14.5/Submit) id r11JOGLh027079; Sat, 2 Feb 2013 06:24:16 +1100 (EST) (envelope-from peter) Date: Sat, 2 Feb 2013 06:24:16 +1100 From: Peter Jeremy To: Kevin Day Subject: Re: Improving ZFS performance for large directories Message-ID: <20130201192416.GA76461@server.rulingia.com> References: <19DB8F4A-6788-44F6-9A2C-E01DEA01BED9@dragondata.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="envbJBWh7q8WU6mo" Content-Disposition: inline In-Reply-To: X-PGP-Key: http://www.rulingia.com/keys/peter.pgp User-Agent: Mutt/1.5.21 (2010-09-15) Cc: FreeBSD Filesystems X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Feb 2013 19:24:38 -0000 --envbJBWh7q8WU6mo Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2013-Jan-29 18:06:01 -0600, Kevin Day wrote: >On Jan 29, 2013, at 5:42 PM, Matthew Ahrens wrote: >> On Tue, Jan 29, 2013 at 3:20 PM, Kevin Day wrote: >> I'm prepared to try an L2arc cache device (with secondarycache=3Dmetadat= a), >>=20 >> You might first see how long it takes when everything is cached. E.g. b= y doing this in the same directory several times. This will give you a low= er bound on the time it will take (or put another way, an upper bound on th= e improvement available from a cache device). >> =20 > >Doing it twice back-to-back makes a bit of difference but it's still slow = either way. ZFS can very conservative about caching data and twice might not be enough. I suggest you try 8-10 times, or until the time stops reducing. >I think some of the issue is that nothing is being allowed to stay cached = long. Well ZFS doesn't do any time-based eviction so if things aren't staying in the cache, it's because they are being evicted by things that ZFS considers more deserving. Looking at the zfs-stats you posted, it looks like your workload has very low locality of reference (the data hitrate is very) low. If this is not what you expect then you need more RAM. OTOH, your vfs.zfs.arc_meta_used being above vfs.zfs.arc_meta_limit suggests that ZFS really wants to cache more metadata (by default ZFS has a 25% metadata, 75% data split in ARC to prevent metadata caching starving data caching). I would go even further than the 50:50 split suggested later and try 75:25 (ie, triple the current vfs.zfs.arc_meta_limit). Note that if there is basically no locality of reference in your workload (as I suspect), you can even turn off data caching for specific filesystems with zfs set primarycache=3Dmetadata tank/foo (note that you still need to increase vfs.zfs.arc_meta_limit to allow ZFS to use the the ARC to cache metadata). --=20 Peter Jeremy --envbJBWh7q8WU6mo Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iEYEARECAAYFAlEMFmAACgkQ/opHv/APuIecWACgn5H+MWNyBmOSD6dCkZOrkIF7 mUgAn0tVC7elSQq2Z22FqQ5/wNi+0Fvn =u4yZ -----END PGP SIGNATURE----- --envbJBWh7q8WU6mo--