Date: Wed, 5 Sep 2018 10:54:36 -0400 From: Allan Jude <allanjude@freebsd.org> To: freebsd-current@freebsd.org Subject: Re: ZFS perfomance regression in FreeBSD 12 APLHA3->ALPHA4 Message-ID: <e26c530e-88e4-0481-9012-f814dd8ed569@freebsd.org> In-Reply-To: <CAFt_eMrhFiV0OJQ2w05-pFj4EkYr9dbYf4APLuJdBykwkOjRsQ@mail.gmail.com> References: <CAFt_eMrhFiV0OJQ2w05-pFj4EkYr9dbYf4APLuJdBykwkOjRsQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --7zBeATgR61BbcOzwawR91adYyBSQekAnN Content-Type: multipart/mixed; boundary="hMUDOWnqlhCxhScvf4riWXfBPMk5mtVUC"; protected-headers="v1" From: Allan Jude <allanjude@freebsd.org> To: freebsd-current@freebsd.org Message-ID: <e26c530e-88e4-0481-9012-f814dd8ed569@freebsd.org> Subject: Re: ZFS perfomance regression in FreeBSD 12 APLHA3->ALPHA4 References: <CAFt_eMrhFiV0OJQ2w05-pFj4EkYr9dbYf4APLuJdBykwkOjRsQ@mail.gmail.com> In-Reply-To: <CAFt_eMrhFiV0OJQ2w05-pFj4EkYr9dbYf4APLuJdBykwkOjRsQ@mail.gmail.com> --hMUDOWnqlhCxhScvf4riWXfBPMk5mtVUC Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 2018-09-05 10:04, Subbsd wrote: > Hi, >=20 > I'm seeing a huge loss in performance ZFS after upgrading FreeBSD 12 > to latest revision (r338466 the moment) and related to ARC. >=20 > I can not say which revision was before except that the newver.sh > pointed to ALPHA3. >=20 > Problems are observed if you try to limit ARC. In my case: >=20 > vfs.zfs.arc_max=3D"128M" >=20 > I know that this is very small. However, for two years with this there > were no problems. >=20 > When i send SIGINFO to process which is currently working with ZFS, i > see "arc_reclaim_waiters_cv": >=20 > e.g when i type: >=20 > /bin/csh >=20 > I have time (~5 seconds) to press several times 'ctrl+t' before csh is = executed: >=20 > load: 0.70 cmd: csh 5935 [arc_reclaim_waiters_cv] 1.41r 0.00u 0.00s 0%= 3512k > load: 0.70 cmd: csh 5935 [zio->io_cv] 1.69r 0.00u 0.00s 0% 3512k > load: 0.70 cmd: csh 5935 [arc_reclaim_waiters_cv] 1.98r 0.00u 0.01s 0%= 3512k > load: 0.73 cmd: csh 5935 [arc_reclaim_waiters_cv] 2.19r 0.00u 0.01s 0%= 4156k >=20 > same story with find or any other commans: >=20 > load: 0.34 cmd: find 5993 [zio->io_cv] 0.99r 0.00u 0.00s 0% 2676k > load: 0.34 cmd: find 5993 [arc_reclaim_waiters_cv] 1.13r 0.00u 0.00s 0= % 2676k > load: 0.34 cmd: find 5993 [arc_reclaim_waiters_cv] 1.25r 0.00u 0.00s 0= % 2680k > load: 0.34 cmd: find 5993 [arc_reclaim_waiters_cv] 1.38r 0.00u 0.00s 0= % 2684k > load: 0.34 cmd: find 5993 [arc_reclaim_waiters_cv] 1.51r 0.00u 0.00s 0= % 2704k > load: 0.34 cmd: find 5993 [arc_reclaim_waiters_cv] 1.64r 0.00u 0.00s 0= % 2716k > load: 0.34 cmd: find 5993 [arc_reclaim_waiters_cv] 1.78r 0.00u 0.00s 0= % 2760k >=20 > this problem goes away after increasing vfs.zfs.arc_max > _______________________________________________ > freebsd-current@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.o= rg" >=20 Previously, ZFS was not actually able to evict enough dnodes to keep your arc_max under 128MB, it would have been much higher based on the number of open files you had. A recent improvement from upstream ZFS (r337653 and r337660) was pulled in that fixed this, so setting an arc_max of 128MB is much more effective now, and that is causing the side effect of "actually doing what you asked it to do", in this case, what you are asking is a bit silly. If you have a working set that is greater than 128MB, and you ask ZFS to use less than that, it'll have to constantly try to reclaim memory to keep under that very low bar. --=20 Allan Jude --hMUDOWnqlhCxhScvf4riWXfBPMk5mtVUC-- --7zBeATgR61BbcOzwawR91adYyBSQekAnN Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) iQIcBAEBAgAGBQJbj+4vAAoJEBmVNT4SmAt+sOwQALRaxWcOi75oPRj6k29S5/ob 0QKww9YwFaFwESKuceMl39ZIioqyzQuOQISsklKqKYekc5LY0qF8opZlWA9XHnI/ t35uklgGIyeBf2sXbNv4/Q20l2iQhZy8v29QKVQ2XiHGWVH499YwLU9fKVAlp1LJ lndB+1TjusMoy46AvCWWKzQFJT6Udm4hIdNSoLHoe1yk6gSiVgn7E9x+WHNMbTFu PUOD+hs0u6Qnrf6Hw/wB1iZ0SostXtpiRKWseDZFTp++ZblWcazfe438Gp1gCeeq Wa+O9FNJ+xE58jYaeRuoWTAGMnjlKG9lRyTAK60Ps2rB7Oht/D+ggBGQYWAUuXjr nRHvadpApnUSANUqomdq3RDRrsAc4a/LWJJSiwRr1QFZf7fQxQy3ScfJast/9wVy OAdBQSgW6Qz95LhyxWMtuGpAvOvVPoAIYjp+eKCqJrv9KGzEUvEBPSWXSJ5+jj4I 1k+E5/MWce7axAb+GXkIobVAWtNzKrEVJKx9/Pzb8Gzf6MAwuBCzIdjJOGzPQmjU Aq9o8kw8AbLDNVhcC/4/TsRSUX1NOP1PMUgyLHzkGW4pkIpDMdExkZtAs7FDAm6L M8+ZewBpeoKM8nIXM+jF8vKNfvX6WGnWXleBtb7THVRqb1Ob6FBJAdh/cFKjD/5f dFl36nOmKUnsZJXzdhWg =eqrU -----END PGP SIGNATURE----- --7zBeATgR61BbcOzwawR91adYyBSQekAnN--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?e26c530e-88e4-0481-9012-f814dd8ed569>