From owner-freebsd-questions@freebsd.org Fri Oct 16 17:50:22 2020 Return-Path: Delivered-To: freebsd-questions@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 422DE43BD5B for ; Fri, 16 Oct 2020 17:50:22 +0000 (UTC) (envelope-from doug@fledge.watson.org) Received: from cyrus.watson.org (cyrus.watson.org [204.107.128.30]) by mx1.freebsd.org (Postfix) with ESMTP id 4CCYb13YD2z40gM for ; Fri, 16 Oct 2020 17:50:21 +0000 (UTC) (envelope-from doug@fledge.watson.org) Received: from fledge.watson.org (fledge.watson.org [198.74.231.63]) by cyrus.watson.org (Postfix) with ESMTPS id 321379021C; Fri, 16 Oct 2020 17:50:20 +0000 (UTC) Received: from fledge.watson.org (doug@localhost [127.0.0.1]) by fledge.watson.org (8.16.1/8.16.1) with ESMTPS id 09GHoKbV011688 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Fri, 16 Oct 2020 17:50:20 GMT (envelope-from doug@fledge.watson.org) Received: from localhost (doug@localhost) by fledge.watson.org (8.16.1/8.16.1/Submit) with ESMTP id 09GHoJIR011685; Fri, 16 Oct 2020 17:50:19 GMT (envelope-from doug@fledge.watson.org) Date: Fri, 16 Oct 2020 17:50:19 +0000 (UTC) From: doug Reply-To: doug@safeport.com To: Jon Schneider cc: freebsd-questions@freebsd.org, Twingly Customer Support Subject: Re: FreeBSD using swap even though there's a lot of free memory In-Reply-To: Message-ID: References: <5f885b772d622_95aa2adab2b9c5b41576495c3@sirportly-app-02.mail> User-Agent: Alpine 2.21.9999 (BSF 287 2018-06-16) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Rspamd-Queue-Id: 4CCYb13YD2z40gM X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of doug@fledge.watson.org has no SPF policy when checking 204.107.128.30) smtp.mailfrom=doug@fledge.watson.org X-Spamd-Result: default: False [-0.48 / 15.00]; HAS_REPLYTO(0.00)[doug@safeport.com]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.18)[-0.180]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-0.77)[-0.767]; MIME_GOOD(-0.10)[text/plain]; REPLYTO_DOM_NEQ_FROM_DOM(0.00)[]; DMARC_NA(0.00)[watson.org]; AUTH_NA(1.00)[]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.53)[-0.529]; R_SPF_NA(0.00)[no SPF record]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:11288, ipnet:204.107.128.0/24, country:US]; MID_RHS_MATCH_FROM(0.00)[]; MAILMAN_DEST(0.00)[freebsd-questions] X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Oct 2020 17:50:22 -0000 On Thu, 15 Oct 2020, Jon Schneider wrote: > top -w -oswap > > seems to report the right thing. Would be interesting if it's _not_ MySQL. > > Jon > > On 15/10/2020 15:23, Twingly Customer Support wrote: >> Hi, >> >> We have a server running FreeBSD 12.1-RELEASE-p10. We currently have a >> problem where FreeBSD starting to swap when running ZFS scrub, even though >> we have ~70G of free memory. >> This did not happen before when running FreeBSD 11.3 for example. It >> started happening at approximately the time we upgraded from >> 12.1-RELEASE-p5 to 12.1-RELEASE-p6, but if the upgrade is the cause of the >> problem is unclear, though FreeBSD never swapped for us before that. >> "Laundry" memory was not something we saw before either, it started to >> appear at the same time as FreeBSD started swapping. >> >> Eventually, after scrubbing a few times, the swap becomes full and we start >> seeing "swap_pager_getswapspace(24): failed" etc. in dmesg. >> This is the memory usage a while after scrubbing, note the values for >> Mem/Free and Swap: >> >> ``` >> % top | head -n 7 >> last pid: 8112; load averages: 1.82, 1.77, 1.73 up 6+01:37:42 >> 10:53:48 >> 35 processes: 1 running, 34 sleeping >> CPU: 4.9% user, 0.0% nice, 4.2% system, 0.2% interrupt, 90.7% idle >> Mem: 110G Active, 27G Inact, 5413M Laundry, 39G Wired, 68G Free >> ARC: 34G Total, 28G MFU, 4101M MRU, 53M Anon, 1317M Header, 225M Other >> 30G Compressed, 53G Uncompressed, 1.77:1 Ratio >> Swap: 8192M Total, 6434M Used, 1757M Free, 78% Inuse >> ``` >> >> We are running MySQL, which has been configured to use ~50% of the total >> amount memory (using innodb_buffer_pool_size=127748M) >> ZFS ARC has been configured to use 25% of the total memory (using >> vfs.zfs.arc_max="63874M") >> >> We have tried raising both vfs.zfs.arc_max and innodb_buffer_pool_size, but >> this did not make any change to the total memory usage, the free memory >> stays at around 70G and FreeBSD still started swapping. >> It's as if the memory is capped at around 180G for some reason. >> >> Are there any configuration values that could cause FreeBSD to swap even >> though there's free memory? Are there any config values one could try to >> change in order to get FreeBSD to use the remaining ~70G of free memory >> instead of swapping? >> >> Let me know if there's any more details you want me to provide and I'll >> attach those. >> >> Thanks! >> >> // Mattias I see similar things. The Jails in question are 11.1. The systems updated to 12.1 do not display this behavior. This 11.1 system runs 5 jails. Swapinfo is shown below: Device 1K-blocks Used Avail Capacity /dev/aacd0p3 4194304 1776000 2418304 42% These numbers are developed from top on the base system [ 0 50861 ] root [ 20 281903 ] camden squirellmail/roundcube/postfix/mysql [ 21 322759 ] bassharbor wordpress/php56 [ 19 343522 ] monhegan wordpress/php56 [ 18 369139 ] newharbor apache24/sendmail [ 17 587332 ] pemaquid wordpress/php74 Jails: 1904655 total: 1955516 I read somewhere that the virtual memory system pre-pages modified pages as a just-in-case measure. If this is correct, 12.1 does not do this on a non-paging system. The system shown above uses about 10% swapspace after a reboot and works its way to the 42% shown above in a day or so.