Date: Tue, 28 Aug 2018 10:50:48 +0200 From: Yamagi Burmeister <lists@yamagi.org> To: lev@FreeBSD.org Cc: freebsd-fs@freebsd.org, MacCrest@gmail.com Subject: Re: ZFS (ARC) performance regression in r321610 Message-ID: <20180828105048.d5a4133c8a932eff900bd758@yamagi.org> In-Reply-To: <20180827182723.cd8ec6bac6448793b25ef91c@yamagi.org> References: <20180827154727.80f92fff9bbc931b37928d43@yamagi.org> <3c6f8c96-6ac9-7257-c8c0-8be2063a7c19@FreeBSD.org> <20180827182723.cd8ec6bac6448793b25ef91c@yamagi.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--Signature=_Tue__28_Aug_2018_10_50_48_+0200_Pg40drlT/n+flvi. Content-Type: text/plain; charset=US-ASCII Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, 27 Aug 2018 18:27:23 +0200 Yamagi Burmeister <lists@yamagi.org> wrote: > On Mon, 27 Aug 2018 17:19:10 +0300 > Lev Serebryakov <lev@FreeBSD.org> wrote: >=20 > > On 27.08.2018 16:47, Yamagi Burmeister wrote: > >=20 > > > With this program I was able to bisect the source and identify commit > > > r321610 (MFV r318946: 8021 ARC buf data scatter-ization) as the culpr= it: > > > https://svnweb.freebsd.org/base?view=3Drevision&revision=3D321610 > > You could try to set "zfs_abd_scatter_enabled =3D 0" via kgdb and repe= at > > tests to be sure, that this code is a problem. >=20 > With my testcase this looks very, very good. I'll give it a try in > production and report back. That'll take at least 12 hours to be sure > that it really works. Yes, it works. The database is being able to keep up with the workload. The CPU load even seems to be lower than under FreeBSD 11.1, but it's too early to be sure about that. ZFS on Linux exposes zfs_abd_scatter_enabled through a module parameter. What about turning it into a sysctl? Something like this: diff --git a/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/abd.c b/sys/cdd= l/contrib/opensolaris/uts/common/fs/zfs/abd.c index 2cd87c2b87b..88b76d1ffbc 100644 --- a/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/abd.c +++ b/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/abd.c @@ -140,6 +140,13 @@ static abd_stats_t abd_stats =3D { */ boolean_t zfs_abd_scatter_enabled =3D B_TRUE; =20 +#if defined(__FreeBSD__) && defined(_KERNEL) +SYSCTL_DECL(_vfs_zfs); + +SYSCTL_UINT(_vfs_zfs, OID_AUTO, abd_scatter_enabled, CTLFLAG_RWTUN, + &zfs_abd_scatter_enabled, 1, "Make all future ABDs linear"); +#endif + /* * The size of the chunks ABD allocates. Because the sizes allocated from = the * kmem_cache can't change, this tunable can only be modified at boot. Cha= nging @@ -151,9 +158,7 @@ boolean_t zfs_abd_scatter_enabled =3D B_TRUE; size_t zfs_abd_chunk_size =3D 4096; =20 #if defined(__FreeBSD__) && defined(_KERNEL) -SYSCTL_DECL(_vfs_zfs); - -SYSCTL_ULONG(_vfs_zfs, OID_AUTO, abd_chunk_size, CTLFLAG_RDTUN, +SYSCTL_ULONG(_vfs_zfs, OID_AUTO, abd_chunk_size, CTLFLAG_RWTUN, &zfs_abd_chunk_size, 0, "The size of the chunks ABD allocates"); #endif --=20 Homepage: https://www.yamagi.org Github: https://github.com/yamagi GPG: 0x1D502515 --Signature=_Tue__28_Aug_2018_10_50_48_+0200_Pg40drlT/n+flvi. Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEOXu/lxyufwz0gC5x6xRy5x1QJRUFAluFDOkACgkQ6xRy5x1Q JRW3/g//bpkp/B5mAWB6qlCQzgmK7BD92+8p3QDeIw1ke8dqkAZEZ3boX8vyc7+2 dgeCDDgBqabxa8clPlTjDBlK3TK4KcFKlZzAZOLxGsTAKT9seIfHMkUdDNByp5ZF xLtDlLM77JaQfAgqDoaFVMmkfTu9Rl0nOVDgCWimrZI0HbJiu+MpZRJX9T0YdtsG zv67sdzC5UG3FmT7k/xIH3Q4SPUDus6flYOzVajyVUz6f1S7JgsBHrEYASenl+SI Gle4lro/UQp/jk5iRDQ3Txp3YyVUqRTuhvpiQUEo5NZM026aSoBewzCQnv/Ygl2A ESSFIL+Xlv/gvB7sxTKDDnsUA71/LevrjpGd1UE3uVG9u++OnvlRe9iVhOzVLi06 bWYkHiN+jm8GXAF0oOJom9Gyqc1OW0s5VzgXqZv/e6Be8DwWR+V+4woOOo0nRjfp z44Lw1XFchWGBs5I8YHNO/UpyhxLtB9m3GK/kteFA1yyLQQ8BM5RI60Ix02FjW1I Rj05ucrSxU8Phh+anx8LdLdwlszDf0MrMZJSF5quR3JOrZhWhz3uN4+rcofWN1tH BWYm8bEvfgflfVDlbGjtoeBps46XmiplxIw0HB7ClGlfD0J+UP+2NHYYf3KSZRxs zgt134qWMBK/Qvq+iSeap/fOmi12+93PTnc4AkTR5mYrKiRRXYg= =dN3I -----END PGP SIGNATURE----- --Signature=_Tue__28_Aug_2018_10_50_48_+0200_Pg40drlT/n+flvi.--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20180828105048.d5a4133c8a932eff900bd758>