Skip site navigation (1)Skip section navigation (2)
Date:      Tue, 3 Apr 2018 21:42:48 -0700 (PDT)
From:      Don Lewis <truckman@FreeBSD.org>
To:        Andriy Gapon <avg@FreeBSD.org>
Cc:        Bryan Drewery <bdrewery@FreeBSD.org>, Peter Jeremy <peter@rulingia.com>, Jeff Roberson <jroberson@jroberson.net>, FreeBSD current <freebsd-current@FreeBSD.org>
Subject:   Re: Strange ARC/Swap/CPU on yesterday's -CURRENT
Message-ID:  <tkrat.7e60fe1978ea51c0@FreeBSD.org>
In-Reply-To: <tkrat.11e402b8455bd0fa@FreeBSD.org>
References:  <20180306173455.oacyqlbib4sbafqd@ler-imac.lerctr.org> <201803061816.w26IGaW5050053@pdx.rh.CN85.dnsmgr.net> <20180306193645.vv3ogqrhauivf2tr@ler-imac.lerctr.org> <20180306221554.uyshbzbboai62rdf@dx240.localdomain> <20180307103911.GA72239@kloomba> <20180311004737.3441dbf9@thor.intern.walstatt.dynvpn.de> <alpine.BSF.2.21.1803111038460.1232@desktop> <20180320070745.GA12880@server.rulingia.com> <2b3db2af-03c7-65ff-25e7-425cfd8815b6@FreeBSD.org> <1fd2b47b-b559-69f8-7e39-665f0f599c8f@FreeBSD.org> <tkrat.9bab32187c0e8d01@FreeBSD.org> <tkrat.11e402b8455bd0fa@FreeBSD.org>

next in thread | previous in thread | raw e-mail | index | archive | help
On  3 Apr, Don Lewis wrote:
> On  1 Apr, Don Lewis wrote:
>> On 27 Mar, Andriy Gapon wrote:
>>> On 24/03/2018 01:21, Bryan Drewery wrote:
>>>> On 3/20/2018 12:07 AM, Peter Jeremy wrote:
>>>>>
>>>>> On 2018-Mar-11 10:43:58 -1000, Jeff Roberson <jroberson@jroberson.net> wrote:
>>>>>> Also, if you could try going back to r328953 or r326346 and let me know if 
>>>>>> the problem exists in either.  That would be very helpful.  If anyone is 
>>>>>> willing to debug this with me contact me directly and I will send some 
>>>>>> test patches or debugging info after you have done the above steps.
>>>>>
>>>>> I ran into this on 11-stable and tracked it to r326619 (MFC of r325851).
>>>>> I initially got around the problem by reverting that commit but either
>>>>> it or something very similar is still present in 11-stable r331053.
>>>>>
>>>>> I've seen it in my main server (32GB RAM) but haven't managed to reproduce
>>>>> it in smaller VBox guests - one difficulty I faced was artificially filling
>>>>> ARC.
>>> 
>>> First, it looks like maybe several different issues are being discussed and
>>> possibly conflated in this thread.  I see reports related to ZFS, I see reports
>>> where ZFS is not used at all.  Some people report problems that appeared very
>>> recently while others chime in with "yes, yes, I've always had this problem".
>>> This does not help to differentiate between problems and to analyze them.
>>> 
>>>> Looking at the ARC change you referred to from r325851
>>>> https://reviews.freebsd.org/D12163, I am convinced that ARC backpressure
>>>> is completely broken.
>>> 
>>> Does your being convinced come from the code review or from experiments?
>>> If the former, could you please share your analysis?
>>> 
>>>> On my 78GB RAM system with ARC limited to 40GB and
>>>> doing a poudriere build of all LLVM and GCC packages at once in tmpfs I
>>>> can get swap up near 50GB and yet the ARC remains at 40GB through it
>>>> all.  It's always been slow to give up memory for package builds but it
>>>> really seems broken right now.
>>> 
>>> Well, there are multiple angles.  Maybe it's ARC that does not react properly,
>>> or maybe it's not being asked properly.
>>> 
>>> It would be useful to monitor the system during its transition to the state that
>>> you reported.  There are some interesting DTrace probes in ARC, specifically
>>> arc-available_memory and arc-needfree are first that come to mind.  Their
>>> parameters and how frequently they are called are of interest.  VM parameters
>>> could be of interest as well.
>>> 
>>> A rant.
>>> 
>>> Basically, posting some numbers and jumping to conclusions does not help at all.
>>> Monitoring, graphing, etc does help.
>>> ARC is a complex dynamic system.
>>> VM (pagedaemon, UMA caches) is a complex dynamic system.
>>> They interact in a complex dynamic ways.
>>> Sometimes a change in ARC is incorrect and requires a fix.
>>> Sometimes a change in VM is incorrect and requires a fix.
>>> Sometimes a change in VM requires a change in ARC.
>>> These three kinds of problems can all appear as a "problem with ARC".
>>> 
>>> For instance, when vm.lowmem_period was introduced you wouldn't find any mention
>>> of ZFS/ARC.  But it does affect period between arc_lowmem() calls.
>>> 
>>> Also, pin-pointing a specific commit requires proper bisecting and proper
>>> testing to correctly attribute systemic behavior changes to code changes.
>> 
>> I just upgraded my main package build box (12.0-CURRENT, 8 cores, 32 GB
>> RAM) from r327616 to r331716.  I was seeing higher swap usage and larger
>> ARC sizes before the upgrade than I remember from the distant past, but
>> ARC was still at least somewhat responsive to memory pressure and I
>> didn't notice any performance issues.
>> 
>> After the upgrade, ARC size seems to be pretty unresponsive to memory
>> demand.  Currently the machine is near the end of a poudriere run to
>> build my usual set of ~1800 ports.  The only currently running build is
>> chromium and the machine is paging heavily.  Settings of interest are:
>>   USE_TMPFS="wrkdir data localbase"
>>   ALLOW_MAKE_JOBS=yes
>> 
>> last pid: 96239;  load averages:  1.86,  1.76,  1.83    up 3+14:47:00  12:38:11
>> 108 processes: 3 running, 105 sleeping
>> CPU: 18.6% user,  0.0% nice,  2.4% system,  0.0% interrupt, 79.0% idle
>> Mem: 129M Active, 865M Inact, 61M Laundry, 29G Wired, 1553K Buf, 888M Free
>> ARC: 23G Total, 8466M MFU, 10G MRU, 5728K Anon, 611M Header, 3886M Other
>>      17G Compressed, 32G Uncompressed, 1.88:1 Ratio
>> Swap: 40G Total, 17G Used, 23G Free, 42% Inuse, 4756K In
>> 
>>   PID USERNAME       THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAN
>> 96239 nobody           1  76    0   140M 93636K CPU5    5   0:01  82.90% clang-
>> 96238 nobody           1  75    0   140M 92608K CPU7    7   0:01  80.81% clang-
>>  5148 nobody           1  20    0   590M   113M swread  0   0:31   0.29% clang-
>> 57290 root             1  20    0 12128K  2608K zio->i  7   8:11   0.28% find
>> 78958 nobody           1  20    0   838M   299M swread  0   0:23   0.19% clang-
>> 97840 nobody           1  20    0   698M   140M swread  4   0:27   0.13% clang-
>> 96066 nobody           1  20    0   463M   104M swread  1   0:32   0.12% clang-
>> 11050 nobody           1  20    0   892M   154M swread  2   0:39   0.12% clang-
>> 
>> Pre-upgrade I was running r327616, which is newer than either of the
>> commits that Jeff mentioned above.  It seems like there has been a
>> regression since then.
>> 
>> I also don't recall seeing this problem on my Ryzen box, though it has
>> 2x the core count and 2x the RAM.  The last testing that I did on it was
>> with r329844.
> 
> I reconfigured my Ryzen box to be more similar to my default package
> builder by disabling SMT and half of the RAM, to limit it to 8 cores
> and 32 GB and then started bisecting to try to track down the problem.
> For each test, I first filled ARC by tarring /usr/ports/distfiles to
> /dev/null.  The commit range that I was searching was r329844 to
> r331716.  I narrowed the range to r329844 to r329904.  With r329904
> and newer, ARC is totally unresponsive to memory pressure and the
> machine pages heavily.  I see ARC sizes of 28-29GB and 30GB of wired
> RAM, so there is not much leftover for getting useful work done.  Active
> memory and free memory both hover under 1GB each.  Looking at the
> commit logs over this range, the most likely culprit is:
> 
> r329882 | jeff | 2018-02-23 14:51:51 -0800 (Fri, 23 Feb 2018) | 13 lines
> 
> Add a generic Proportional Integral Derivative (PID) controller algorithm and
> use it to regulate page daemon output.
> 
> This provides much smoother and more responsive page daemon output, anticipating
> demand and avoiding pageout stalls by increasing the number of pages to match
> the workload.  This is a reimplementation of work done by myself and mlaier at
> Isilon.
> 
> 
> It is quite possible that the recent fixes to the PID controller will
> fix the problem.  Not that r329844 was trouble free ... I left tar
> running over lunchtime to fill ARC and the OOM killer nuked top, tar,
> ntpd, both of my ssh sessions into the machine, and multiple instances
> of getty while I was away.  I was able to log in again and successfully
> run poudriere, and ARC did respond to the memory pressure and cranked
> itself down to about 5 GB by the end of the run.  I did not see the same
> problem with tar when I did the same with r329904.

I just tried r331966 and see no improvement.  No OOM process kills
during the tar run to fill ARC, but with ARC filled, the machine is
thrashing itself at the start of the poudriere run while trying to build
ports-mgmt/pkg (39 minutes so far).  ARC appears to be unresponsive to
memory demand.  I've seen no decrease in ARC size or wired memory since
starting poudriere.

last pid: 75652;  load averages:  0.46,  0.27,  0.25    up 0+02:03:48  21:33:00
48 processes:  1 running, 47 sleeping
CPU:  0.7% user,  0.0% nice,  0.8% system,  0.1% interrupt, 98.4% idle
Mem: 4196K Active, 328K Inact, 148K Laundry, 30G Wired, 506M Free
ARC: 29G Total, 309M MFU, 28G MRU, 3767K Anon, 103M Header, 79M Other
     27G Compressed, 29G Uncompressed, 1.05:1 Ratio
Swap: 80G Total, 234M Used, 80G Free, 2348K In, 420K Out

  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
 1340 root          1  52    0 13048K  1024K nanslp  6   0:10   0.72% sh
41806 nobody        1  20    0   133M  1440K swread  6   0:17   0.31% cc
42554 nobody        1  20    0   129M  1048K swread  6   0:12   0.22% cc
  934 dl            1  20    0 19124K   592K select  6   0:03   0.14% sshd
  938 dl            1  20    0 13596K   740K CPU7    7   0:07   0.12% top
40784 nobody        1  20    0  5024K   104K select  5   0:00   0.03% make
41142 nobody        1  20    0  5024K   124K select  6   0:00   0.01% make
  638 root          1  20    0 11344K   252K swread  7   0:00   0.01% syslogd
41428 nobody        1  20    0  5024K   124K select  4   0:00   0.01% make
40782 nobody        1  20    0  5024K   124K select  4   0:00   0.01% make
41164 nobody        1  20    0  5024K   124K select  7   0:00   0.01% make
 1050 root          1  23    0 13032K   468K select  0   0:06   0.00% sh
  848 root          1  20    0 11400K   492K nanslp  5   0:00   0.00% cron
36198 root          1  20    0 13048K   396K select  7   0:00   0.00% sh




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?tkrat.7e60fe1978ea51c0>