Date: Sat, 2 Feb 2013 06:24:16 +1100 From: Peter Jeremy <peter@rulingia.com> To: Kevin Day <toasty@dragondata.com> Cc: FreeBSD Filesystems <freebsd-fs@freebsd.org> Subject: Re: Improving ZFS performance for large directories Message-ID: <20130201192416.GA76461@server.rulingia.com> In-Reply-To: <F4420A8C-FB92-4771-B261-6C47A736CF7F@dragondata.com> References: <19DB8F4A-6788-44F6-9A2C-E01DEA01BED9@dragondata.com> <CAJjvXiE%2B8OMu_yvdRAsWugH7W=fhFW7bicOLLyjEn8YrgvCwiw@mail.gmail.com> <F4420A8C-FB92-4771-B261-6C47A736CF7F@dragondata.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--envbJBWh7q8WU6mo Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2013-Jan-29 18:06:01 -0600, Kevin Day <toasty@dragondata.com> wrote: >On Jan 29, 2013, at 5:42 PM, Matthew Ahrens <mahrens@delphix.com> wrote: >> On Tue, Jan 29, 2013 at 3:20 PM, Kevin Day <toasty@dragondata.com> wrote: >> I'm prepared to try an L2arc cache device (with secondarycache=3Dmetadat= a), >>=20 >> You might first see how long it takes when everything is cached. E.g. b= y doing this in the same directory several times. This will give you a low= er bound on the time it will take (or put another way, an upper bound on th= e improvement available from a cache device). >> =20 > >Doing it twice back-to-back makes a bit of difference but it's still slow = either way. ZFS can very conservative about caching data and twice might not be enough. I suggest you try 8-10 times, or until the time stops reducing. >I think some of the issue is that nothing is being allowed to stay cached = long. Well ZFS doesn't do any time-based eviction so if things aren't staying in the cache, it's because they are being evicted by things that ZFS considers more deserving. Looking at the zfs-stats you posted, it looks like your workload has very low locality of reference (the data hitrate is very) low. If this is not what you expect then you need more RAM. OTOH, your vfs.zfs.arc_meta_used being above vfs.zfs.arc_meta_limit suggests that ZFS really wants to cache more metadata (by default ZFS has a 25% metadata, 75% data split in ARC to prevent metadata caching starving data caching). I would go even further than the 50:50 split suggested later and try 75:25 (ie, triple the current vfs.zfs.arc_meta_limit). Note that if there is basically no locality of reference in your workload (as I suspect), you can even turn off data caching for specific filesystems with zfs set primarycache=3Dmetadata tank/foo (note that you still need to increase vfs.zfs.arc_meta_limit to allow ZFS to use the the ARC to cache metadata). --=20 Peter Jeremy --envbJBWh7q8WU6mo Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iEYEARECAAYFAlEMFmAACgkQ/opHv/APuIecWACgn5H+MWNyBmOSD6dCkZOrkIF7 mUgAn0tVC7elSQq2Z22FqQ5/wNi+0Fvn =u4yZ -----END PGP SIGNATURE----- --envbJBWh7q8WU6mo--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20130201192416.GA76461>