Date: Wed, 29 Aug 2018 17:22:07 -0400 From: Mark Johnston <markj@freebsd.org> To: Paul <devgs@ukr.net> Cc: freebsd-fs@freebsd.org, developer@open-zfs.org Subject: Re: Potential bug recently introduced in arc_adjust() that leads to unintended pressure on MFU eventually leading to dramatic reduction in its size Message-ID: <20180829212207.GF2709@raichu> In-Reply-To: <1535534257.46692673.obgqw5dr@frv33.fwdcdn.com> References: <1535534257.46692673.obgqw5dr@frv33.fwdcdn.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Aug 29, 2018 at 12:42:33PM +0300, Paul wrote: > Hello team, > > > It seems like a commit on Mar 23 introduced a bug: if during execution of arc_adjust() > target is reached after MRU is evicted current code continues evicting MFU. Before said > commit, on the step prior to MFU eviction, target value was recalculated as: > > target = arc_size - arc_c; > > arc_size here is a global variable that was being updated accordingly, during MRU eviction, > hence this expression, resulted in zero or negative target if MRU eviction was enough > to reach the original goal. > > Modern version uses cached value of arc_size, called asize: > > target = asize - arc_c; > > Because asize stays constant during execution of whole body of arc_adjust() it means that > above expression will always be evaluated to value > 0, causing MFU to be evicted every > time, even if MRU eviction has reached the goal already. Because of the difference in > nature of MFU and MRU, globally it leads to eventual reduction of amount of MFU in ARC > to dramatic numbers. Hi Paul, Your analysis does seem right to me. I cc'ed the openzfs mailing list so that an actual ZFS expert can chime in; it looks like this behaviour is consistent between FreeBSD, illumos and ZoL. Have you already tried the obvious "fix" of subtracting total_evicted from the MFU target? > Servers that run the version of FreeBSD prior to the issue have this picture of ARC: > > ARC: 369G Total, 245G MFU, 97G MRU, 36M Anon, 3599M Header, 24G Other > > As you can see, MFU dominates. This is a nature of our workload: we have a considerably > small dataset that we use constantly and repeatedly; and a large dataset that we use > rarely. > > But on the modern version of FreeBSD picture is dramatically different: > > ARC: 360G Total, 50G MFU, 272G MRU, 211M Anon, 7108M Header, 30G Other > > This leads to a much heavier burden on the disk sub-system. > > > Commit that introduced a bug: > https://github.com/freebsd/freebsd/commit/555f9563c9dc217341d4bb5129f5d233cf1f92b8
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20180829212207.GF2709>