Date: Fri, 16 Oct 2020 17:50:19 +0000 (UTC) From: doug <doug@fledge.watson.org> To: Jon Schneider <freebsd@jschneider.net> Cc: freebsd-questions@freebsd.org, Twingly Customer Support <team@twingly.com> Subject: Re: FreeBSD using swap even though there's a lot of free memory Message-ID: <alpine.BSF.2.21.9999.2010161722590.72530@fledge.watson.org> In-Reply-To: <b221d5c4-e247-6fea-e613-b03e94cda280@jschneider.net> References: <5f885b772d622_95aa2adab2b9c5b41576495c3@sirportly-app-02.mail> <b221d5c4-e247-6fea-e613-b03e94cda280@jschneider.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, 15 Oct 2020, Jon Schneider wrote: > top -w -oswap > > seems to report the right thing. Would be interesting if it's _not_ MySQL. > > Jon > > On 15/10/2020 15:23, Twingly Customer Support wrote: >> Hi, >> >> We have a server running FreeBSD 12.1-RELEASE-p10. We currently have a >> problem where FreeBSD starting to swap when running ZFS scrub, even though >> we have ~70G of free memory. >> This did not happen before when running FreeBSD 11.3 for example. It >> started happening at approximately the time we upgraded from >> 12.1-RELEASE-p5 to 12.1-RELEASE-p6, but if the upgrade is the cause of the >> problem is unclear, though FreeBSD never swapped for us before that. >> "Laundry" memory was not something we saw before either, it started to >> appear at the same time as FreeBSD started swapping. >> >> Eventually, after scrubbing a few times, the swap becomes full and we start >> seeing "swap_pager_getswapspace(24): failed" etc. in dmesg. >> This is the memory usage a while after scrubbing, note the values for >> Mem/Free and Swap: >> >> ``` >> % top | head -n 7 >> last pid: 8112; load averages: 1.82, 1.77, 1.73 up 6+01:37:42 >> 10:53:48 >> 35 processes: 1 running, 34 sleeping >> CPU: 4.9% user, 0.0% nice, 4.2% system, 0.2% interrupt, 90.7% idle >> Mem: 110G Active, 27G Inact, 5413M Laundry, 39G Wired, 68G Free >> ARC: 34G Total, 28G MFU, 4101M MRU, 53M Anon, 1317M Header, 225M Other >> 30G Compressed, 53G Uncompressed, 1.77:1 Ratio >> Swap: 8192M Total, 6434M Used, 1757M Free, 78% Inuse >> ``` >> >> We are running MySQL, which has been configured to use ~50% of the total >> amount memory (using innodb_buffer_pool_size=127748M) >> ZFS ARC has been configured to use 25% of the total memory (using >> vfs.zfs.arc_max="63874M") >> >> We have tried raising both vfs.zfs.arc_max and innodb_buffer_pool_size, but >> this did not make any change to the total memory usage, the free memory >> stays at around 70G and FreeBSD still started swapping. >> It's as if the memory is capped at around 180G for some reason. >> >> Are there any configuration values that could cause FreeBSD to swap even >> though there's free memory? Are there any config values one could try to >> change in order to get FreeBSD to use the remaining ~70G of free memory >> instead of swapping? >> >> Let me know if there's any more details you want me to provide and I'll >> attach those. >> >> Thanks! >> >> // Mattias I see similar things. The Jails in question are 11.1. The systems updated to 12.1 do not display this behavior. This 11.1 system runs 5 jails. Swapinfo is shown below: Device 1K-blocks Used Avail Capacity /dev/aacd0p3 4194304 1776000 2418304 42% These numbers are developed from top on the base system [ 0 50861 ] root [ 20 281903 ] camden squirellmail/roundcube/postfix/mysql [ 21 322759 ] bassharbor wordpress/php56 [ 19 343522 ] monhegan wordpress/php56 [ 18 369139 ] newharbor apache24/sendmail [ 17 587332 ] pemaquid wordpress/php74 Jails: 1904655 total: 1955516 I read somewhere that the virtual memory system pre-pages modified pages as a just-in-case measure. If this is correct, 12.1 does not do this on a non-paging system. The system shown above uses about 10% swapspace after a reboot and works its way to the 42% shown above in a day or so.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?alpine.BSF.2.21.9999.2010161722590.72530>