Date: Mon, 19 Oct 2020 15:06:22 +0100 From: Twingly Customer Support <team@twingly.com> To: team@twingly.com Cc: freebsd-questions@freebsd.org, freebsd@jschneider.net Subject: Re: FreeBSD using swap even though there's a lot of free memory Message-ID: <5f8d9d5e14b4c_dad82b12ba6585a44346f@sirportly-app-01.mail> In-Reply-To: <5f885b772d622_95aa2adab2b9c5b41576495c3@sirportly-app-02.mail> References: <20201016195748.ffc15759b311a5feefc91ef5@sohara.org> <alpine.BSF.2.21.9999.2010161722590.72530@fledge.watson.org> <b221d5c4-e247-6fea-e613-b03e94cda280@jschneider.net> <5f885b772d622_95aa2adab2b9c5b41576495c3@sirportly-app-02.mail>
next in thread | previous in thread | raw e-mail | index | archive | help
> Are you using a swap backed tmpfs ?
No, we don't use tmpfs. This is how we define our swap partition:
% cat /etc/fstab
# Device Mountpoint FStype Options Dump Pass#
/dev/mirror/swap.eli none swap sw 0 0
Fortunately the problem does no longer occur since this Friday, Oct 16, which is when we ran "pkg upgrade -y" the last time. There has been no swap usage (or laundry either for that matter) since then, also the memory (active + inactive + wired) has now been over 200G for the first time since the problems started. I have started ZFS scrub, and even after that the swap is empty.
We update packages regularly, and we have done that multiple times since our problems started, but this time the problem went away. We have not made any other changes to the system recently (other than updating packages) that could have caused these problems. I have not taken a closer look at what exactly has changed in the packages that got updated yet, but these are all the packages that was changed (output from "pkg upgrade -y"):
Installed packages to be UPGRADED:
gnutls: 3.6.13 -> 3.6.14
libnghttp2: 1.40.0 -> 1.41.0
perl5: 5.30.2 -> 5.30.3
python37: 3.7.7 -> 3.7.7_1
I'll continue and see if I manage to figure out which (if any) of the above packages could have caused this.
Also, thanks to Doug for the following:
> I encountered this issue a year or so ago. In my case it turned out to =
> be a process that was allocating anonymous segments using mmap. It =
> tended to forget to free them after the usage was complete. As a =
> result, the system would run out of swap even though there was plenty of =
> free memory. Anonymous segments are mapped to swap space. I traced the =
> problem using:
>
> procstat -va | grep df
>
> That generated a lot of data. I looked for processes that had an =
> increasing number of df files. Eventually I found the right one and =
> fixed it.
I'll be sure to remember this in case I run into things like these in the future.
// Mattias
--------------------------------------------------------------------------------
On October 16, 2020 20:57 Steve O'Hara-Smith wrote:
On Thu, 15 Oct 2020 15:23:51 +0100
Twingly Customer Support <team@twingly.com> wrote:
> Eventually, after scrubbing a few times, the swap becomes full and we
> start seeing "swap_pager_getswapspace(24): failed" etc. in dmesg.
Are you using a swap backed tmpfs ?
--
Steve O'Hara-Smith <steve@sohara.org>
--------------------------------------------------------------------------------
On October 16, 2020 19:50 doug wrote:
On Thu, 15 Oct 2020, Jon Schneider wrote:
>top -w -oswap
>
>seems to report the right thing. Would be interesting if it's _not_ MySQL.
>
>Jon
>
>On 15/10/2020 15:23, Twingly Customer Support wrote:
>>Hi,
>>
>>We have a server running FreeBSD 12.1-RELEASE-p10. We currently have a problem where FreeBSD starting to swap when running ZFS scrub, even though we have ~70G of free memory.
>>This did not happen before when running FreeBSD 11.3 for example. It started happening at approximately the time we upgraded from 12.1-RELEASE-p5 to 12.1-RELEASE-p6, but if the upgrade is the cause of the problem is unclear, though FreeBSD never swapped for us before that. "Laundry" memory was not something we saw before either, it started to appear at the same time as FreeBSD started swapping.
>>
>>Eventually, after scrubbing a few times, the swap becomes full and we start seeing "swap_pager_getswapspace(24): failed" etc. in dmesg.
>>This is the memory usage a while after scrubbing, note the values for Mem/Free and Swap:
>>
>>```
>>% top | head -n 7
>>last pid: 8112; load averages: 1.82, 1.77, 1.73 up 6+01:37:42 10:53:48
>>35 processes: 1 running, 34 sleeping
>>CPU: 4.9% user, 0.0% nice, 4.2% system, 0.2% interrupt, 90.7% idle
>>Mem: 110G Active, 27G Inact, 5413M Laundry, 39G Wired, 68G Free
>>ARC: 34G Total, 28G MFU, 4101M MRU, 53M Anon, 1317M Header, 225M Other
>> 30G Compressed, 53G Uncompressed, 1.77:1 Ratio
>>Swap: 8192M Total, 6434M Used, 1757M Free, 78% Inuse
>>```
>>
>>We are running MySQL, which has been configured to use ~50% of the total amount memory (using innodb_buffer_pool_size=127748M)
>>ZFS ARC has been configured to use 25% of the total memory (using vfs.zfs.arc_max="63874M")
>>
>>We have tried raising both vfs.zfs.arc_max and innodb_buffer_pool_size, but this did not make any change to the total memory usage, the free memory stays at around 70G and FreeBSD still started swapping.
>>It's as if the memory is capped at around 180G for some reason.
>>
>>Are there any configuration values that could cause FreeBSD to swap even though there's free memory? Are there any config values one could try to change in order to get FreeBSD to use the remaining ~70G of free memory instead of swapping?
>>
>>Let me know if there's any more details you want me to provide and I'll attach those.
>>
>>Thanks!
>>
>>// Mattias
I see similar things. The Jails in question are 11.1. The systems updated to 12.1 do not display this behavior. This 11.1 system runs 5 jails. Swapinfo is shown below:
Device 1K-blocks Used Avail Capacity
/dev/aacd0p3 4194304 1776000 2418304 42%
These numbers are developed from top on the base system
[ 0 50861 ] root
[ 20 281903 ] camden squirellmail/roundcube/postfix/mysql
[ 21 322759 ] bassharbor wordpress/php56
[ 19 343522 ] monhegan wordpress/php56
[ 18 369139 ] newharbor apache24/sendmail
[ 17 587332 ] pemaquid wordpress/php74
Jails: 1904655
total: 1955516
I read somewhere that the virtual memory system pre-pages modified pages as a just-in-case measure. If this is correct, 12.1 does not do this on a non-paging system. The system shown above uses about 10% swapspace after a reboot and works its way to the 42% shown above in a day or so.
--------------------------------------------------------------------------------
On October 15, 2020 18:21 Jon Schneider wrote:
top -w -oswap
seems to report the right thing. Would be interesting if it's _not_ MySQL.
Jon
--------------------------------------------------------------------------------
On October 15, 2020 16:24 Team Twingly wrote:
Hi,
We have a server running FreeBSD 12.1-RELEASE-p10. We currently have a problem where FreeBSD starting to swap when running ZFS scrub, even though we have ~70G of free memory.
This did not happen before when running FreeBSD 11.3 for example. It started happening at approximately the time we upgraded from 12.1-RELEASE-p5 to 12.1-RELEASE-p6, but if the upgrade is the cause of the problem is unclear, though FreeBSD never swapped for us before that. "Laundry" memory was not something we saw before either, it started to appear at the same time as FreeBSD started swapping.
Eventually, after scrubbing a few times, the swap becomes full and we start seeing "swap_pager_getswapspace(24): failed" etc. in dmesg.
This is the memory usage a while after scrubbing, note the values for Mem/Free and Swap:
```
% top | head -n 7
last pid: 8112; load averages: 1.82, 1.77, 1.73 up 6+01:37:42 10:53:48
35 processes: 1 running, 34 sleeping
CPU: 4.9% user, 0.0% nice, 4.2% system, 0.2% interrupt, 90.7% idle
Mem: 110G Active, 27G Inact, 5413M Laundry, 39G Wired, 68G Free
ARC: 34G Total, 28G MFU, 4101M MRU, 53M Anon, 1317M Header, 225M Other
30G Compressed, 53G Uncompressed, 1.77:1 Ratio
Swap: 8192M Total, 6434M Used, 1757M Free, 78% Inuse
```
We are running MySQL, which has been configured to use ~50% of the total amount memory (using innodb_buffer_pool_size=127748M)
ZFS ARC has been configured to use 25% of the total memory (using vfs.zfs.arc_max="63874M")
We have tried raising both vfs.zfs.arc_max and innodb_buffer_pool_size, but this did not make any change to the total memory usage, the free memory stays at around 70G and FreeBSD still started swapping.
It's as if the memory is capped at around 180G for some reason.
Are there any configuration values that could cause FreeBSD to swap even though there's free memory? Are there any config values one could try to change in order to get FreeBSD to use the remaining ~70G of free memory instead of swapping?
Let me know if there's any more details you want me to provide and I'll attach those.
Thanks!
// Mattias
From owner-freebsd-questions@freebsd.org Mon Oct 19 15:14:57 2020
Return-Path: <owner-freebsd-questions@freebsd.org>
Delivered-To: freebsd-questions@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
by mailman.nyi.freebsd.org (Postfix) with ESMTP id 16EAA42B7AF
for <freebsd-questions@mailman.nyi.freebsd.org>;
Mon, 19 Oct 2020 15:14:57 +0000 (UTC)
(envelope-from sl8ixw@rp.postal.labs.k.io)
Received: from mx95.labs.k.io (mx95.labs.k.io [IPv6:2a03:2800:300::95])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256)
(Client did not present a certificate)
by mx1.freebsd.org (Postfix) with ESMTPS id 4CFL0J1120z3V18
for <freebsd-questions@freebsd.org>; Mon, 19 Oct 2020 15:14:55 +0000 (UTC)
(envelope-from sl8ixw@rp.postal.labs.k.io)
Resent-Sender: sl8ixw@rp.postal.labs.k.io
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=email.sirportly.com;
s=postal-QEE0Jp; t=1603120492; bh=S9Et1GKk6Ppui3roXnmtlZJ16/vAp0/+01wpIA5BqBg=;
h=date:from:to:cc:message-id:in-reply-to:references:subject:mime-version:content-type:content-transfer-encoding;
b=XnYlHHVvG39ymlL7OLY+9iJjmpg/2JPKzTgLdm+rHUSvzgyD0laOmiHSFn7rJqqrXVHW98SaGeWxsvZvHEt5JugOwkOB8dQgHzOrzjVBwnquLOmkGbVeKlWJf363h8gQhKVfqbeH8LvpOkrf+P6MYI2KMmia/69BrqV8+wJSxSM=
X-Postal-MsgID: ZVqIHLKwwuGv
Received: from app.sirportly.com (::ffff:185.22.211.50 [::ffff:185.22.211.50])
by postal.labs.k.io with SMTP; Mon, 19 Oct 2020 15:14:52 -0000
Date: Mon, 19 Oct 2020 16:14:52 +0100
From: Twingly Customer Support <team@twingly.com>
To: team@twingly.com
Cc: freebsd-questions@freebsd.org,
freebsd@jschneider.net
Message-ID: <5f8dad6c5225e_11f782aaeb993e5bc705b@sirportly-app-01.mail>
In-Reply-To: <5f885b772d622_95aa2adab2b9c5b41576495c3@sirportly-app-02.mail>
References: <5f8d9d5e14b4c_dad82b12ba6585a44346f@sirportly-app-01.mail>
<20201016195748.ffc15759b311a5feefc91ef5@sohara.org>
<alpine.BSF.2.21.9999.2010161722590.72530@fledge.watson.org>
<b221d5c4-e247-6fea-e613-b03e94cda280@jschneider.net>
<5f885b772d622_95aa2adab2b9c5b41576495c3@sirportly-app-02.mail>
Subject: Re: FreeBSD using swap even though there's a lot of free memory
Mime-Version: 1.0
X-Mailer: Sirportly/5.16.2
X-Rspamd-Queue-Id: 4CFL0J1120z3V18
X-Spamd-Bar: --
Authentication-Results: mx1.freebsd.org;
dkim=pass header.d=email.sirportly.com header.s=postal-QEE0Jp
header.b=XnYlHHVv; dmarc=none;
spf=pass (mx1.freebsd.org: domain of sl8ixw@rp.postal.labs.k.io designates
2a03:2800:300::95 as permitted sender)
smtp.mailfrom=sl8ixw@rp.postal.labs.k.io
X-Spamd-Result: default: False [-2.47 / 15.00]; ARC_NA(0.00)[];
NEURAL_HAM_MEDIUM(-1.03)[-1.028];
R_DKIM_ALLOW(-0.20)[email.sirportly.com:s=postal-QEE0Jp];
FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3];
MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ip6:2a03:2800:300::/64];
MIME_GOOD(-0.10)[multipart/alternative,text/plain];
TO_DN_NONE(0.00)[]; DMARC_NA(0.00)[twingly.com];
NEURAL_HAM_LONG(-0.95)[-0.946]; TO_MATCH_ENVRCPT_SOME(0.00)[];
DKIM_TRACE(0.00)[email.sirportly.com:+];
NEURAL_HAM_SHORT(-0.79)[-0.791];
FORGED_SENDER(0.30)[team@twingly.com,sl8ixw@rp.postal.labs.k.io];
MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_TLS_LAST(0.00)[];
RCVD_COUNT_TWO(0.00)[2];
ASN(0.00)[asn:12488, ipnet:2a03:2800::/29, country:GB];
FROM_NEQ_ENVFROM(0.00)[team@twingly.com,sl8ixw@rp.postal.labs.k.io];
MAILMAN_DEST(0.00)[freebsd-questions]
Content-Type: text/plain;
charset=UTF-8
Content-Transfer-Encoding: 7bit
X-Content-Filtered-By: Mailman/MimeDel 2.1.33
X-BeenThere: freebsd-questions@freebsd.org
X-Mailman-Version: 2.1.33
Precedence: list
List-Id: User questions <freebsd-questions.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-questions>,
<mailto:freebsd-questions-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-questions/>;
List-Post: <mailto:freebsd-questions@freebsd.org>
List-Help: <mailto:freebsd-questions-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-questions>,
<mailto:freebsd-questions-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Mon, 19 Oct 2020 15:14:57 -0000
A correction:
The output from "pkg upgrade -y" in my previous email was the version of the packages that was installed right as the problem started.
The following are the updates that was installed last Friday, which seems to have solved the problem:
New packages to be INSTALLED:
bash-completion: 2.11,2
glib: 2.66.0_1,1
postgresql12-client: 12.4
Installed packages to be UPGRADED:
bash: 5.0.17 -> 5.0.18_3
ca_root_nss: 3.56 -> 3.57
fio: 3.20 -> 3.23
gettext-runtime: 0.20.2 -> 0.21
git: 2.27.0 -> 2.28.0
iozone: 3.487 -> 3.487_1
libffi: 3.2.1_3 -> 3.3_1
libgpg-error: 1.38 -> 1.39
memcached: 1.6.5 -> 1.6.7
mosh: 1.3.2_13 -> 1.3.2_14
munin-common: 2.0.63 -> 2.0.64
munin-node: 2.0.63 -> 2.0.64
net-snmp: 5.7.3_20,1 -> 5.9,1
p11-kit: 0.23.20 -> 0.23.21
p5-DBD-Pg: 3.13.0 -> 3.14.2
p5-DateTime-HiRes: 0.01_1 -> 0.04
p5-DateTime-Locale: 1.25 -> 1.28
p5-HTTP-Message: 6.24 -> 6.26
p5-Log-Log4perl: 1.49 -> 1.53
p5-Mozilla-CA: 20180117 -> 20200520
p5-Net-DNS: 1.25,1 -> 1.27,1
p5-XML-LibXML: 2.0204,1 -> 2.0206,1
p5-libwww: 6.46 -> 6.49
perl5: 5.30.3 -> 5.32.0
postgresql11-client: 11.8 -> 11.9
protobuf: 3.12.3,1 -> 3.13.0,1
py37-pip: 19.1.1 -> 20.2.3
ruby: 2.6.6,1 -> 2.6.6_1,1
sqlite3: 3.32.2,1 -> 3.33.0,1
sudo: 1.9.2 -> 1.9.3p1
vim-console: 8.2.1110 -> 8.2.1558
xxhash: 0.7.4 -> 0.8.0
zstd: 1.4.5 -> 1.4.5_1
Sorry about the confusion! :)
// Mattias
--------------------------------------------------------------------------------
On October 19, 2020 16:06 Twingly Customer Support wrote:
> Are you using a swap backed tmpfs ?
No, we don't use tmpfs. This is how we define our swap partition:
% cat /etc/fstab
# Device Mountpoint FStype Options Dump Pass#
/dev/mirror/swap.eli none swap sw 0 0
Fortunately the problem does no longer occur since this Friday, Oct 16, which is when we ran "pkg upgrade -y" the last time. There has been no swap usage (or laundry either for that matter) since then, also the memory (active + inactive + wired) has now been over 200G for the first time since the problems started. I have started ZFS scrub, and even after that the swap is empty.
We update packages regularly, and we have done that multiple times since our problems started, but this time the problem went away. We have not made any other changes to the system recently (other than updating packages) that could have caused these problems. I have not taken a closer look at what exactly has changed in the packages that got updated yet, but these are all the packages that was changed (output from "pkg upgrade -y"):
Installed packages to be UPGRADED:
gnutls: 3.6.13 -> 3.6.14
libnghttp2: 1.40.0 -> 1.41.0
perl5: 5.30.2 -> 5.30.3
python37: 3.7.7 -> 3.7.7_1
I'll continue and see if I manage to figure out which (if any) of the above packages could have caused this.
Also, thanks to Doug for the following:
I'll be sure to remember this in case I run into things like these in the future.
// Mattias
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
On October 19, 2020 16:05 Mattias Roback wrote:
> Are you using a swap backed tmpfs ?
No, we don't use tmpfs. This is how we define our swap partition:
% cat /etc/fstab
# Device Mountpoint FStype Options Dump Pass#
/dev/mirror/swap.eli none swap sw 0 0
Fortunately the problem does no longer occur since this Friday, Oct 16, which is when we ran "pkg upgrade -y" the last time. There has been no swap usage (or laundry either for that matter) since then, also the memory (active + inactive + wired) has now been over 200G for the first time since the problems started. I have started ZFS scrub, and even after that the swap is empty.
We update packages regularly, and we have done that multiple times since our problems started, but this time the problem went away. We have not made any other changes to the system recently (other than updating packages) that could have caused these problems. I have not taken a closer look at what exactly has changed in the packages that got updated yet, but these are all the packages that was changed (output from "pkg upgrade -y"):
Installed packages to be UPGRADED:
gnutls: 3.6.13 -> 3.6.14
libnghttp2: 1.40.0 -> 1.41.0
perl5: 5.30.2 -> 5.30.3
python37: 3.7.7 -> 3.7.7_1
I'll continue and see if I manage to figure out which (if any) of the above packages could have caused this.
Also, thanks to Doug for the following:
> I encountered this issue a year or so ago. In my case it turned out to =
> be a process that was allocating anonymous segments using mmap. It =
> tended to forget to free them after the usage was complete. As a =
> result, the system would run out of swap even though there was plenty of =
> free memory. Anonymous segments are mapped to swap space. I traced the =
> problem using:
>
> procstat -va | grep df
>
> That generated a lot of data. I looked for processes that had an =
> increasing number of df files. Eventually I found the right one and =
> fixed it.
I'll be sure to remember this in case I run into things like these in the future.
// Mattias
--------------------------------------------------------------------------------
On October 16, 2020 20:57 Steve O'Hara-Smith wrote:
On Thu, 15 Oct 2020 15:23:51 +0100
Twingly Customer Support <team@twingly.com> wrote:
> Eventually, after scrubbing a few times, the swap becomes full and we
> start seeing "swap_pager_getswapspace(24): failed" etc. in dmesg.
Are you using a swap backed tmpfs ?
--
Steve O'Hara-Smith <steve@sohara.org>
--------------------------------------------------------------------------------
On October 16, 2020 19:50 doug wrote:
On Thu, 15 Oct 2020, Jon Schneider wrote:
>top -w -oswap
>
>seems to report the right thing. Would be interesting if it's _not_ MySQL.
>
>Jon
>
>On 15/10/2020 15:23, Twingly Customer Support wrote:
>>Hi,
>>
>>We have a server running FreeBSD 12.1-RELEASE-p10. We currently have a problem where FreeBSD starting to swap when running ZFS scrub, even though we have ~70G of free memory.
>>This did not happen before when running FreeBSD 11.3 for example. It started happening at approximately the time we upgraded from 12.1-RELEASE-p5 to 12.1-RELEASE-p6, but if the upgrade is the cause of the problem is unclear, though FreeBSD never swapped for us before that. "Laundry" memory was not something we saw before either, it started to appear at the same time as FreeBSD started swapping.
>>
>>Eventually, after scrubbing a few times, the swap becomes full and we start seeing "swap_pager_getswapspace(24): failed" etc. in dmesg.
>>This is the memory usage a while after scrubbing, note the values for Mem/Free and Swap:
>>
>>```
>>% top | head -n 7
>>last pid: 8112; load averages: 1.82, 1.77, 1.73 up 6+01:37:42 10:53:48
>>35 processes: 1 running, 34 sleeping
>>CPU: 4.9% user, 0.0% nice, 4.2% system, 0.2% interrupt, 90.7% idle
>>Mem: 110G Active, 27G Inact, 5413M Laundry, 39G Wired, 68G Free
>>ARC: 34G Total, 28G MFU, 4101M MRU, 53M Anon, 1317M Header, 225M Other
>> 30G Compressed, 53G Uncompressed, 1.77:1 Ratio
>>Swap: 8192M Total, 6434M Used, 1757M Free, 78% Inuse
>>```
>>
>>We are running MySQL, which has been configured to use ~50% of the total amount memory (using innodb_buffer_pool_size=127748M)
>>ZFS ARC has been configured to use 25% of the total memory (using vfs.zfs.arc_max="63874M")
>>
>>We have tried raising both vfs.zfs.arc_max and innodb_buffer_pool_size, but this did not make any change to the total memory usage, the free memory stays at around 70G and FreeBSD still started swapping.
>>It's as if the memory is capped at around 180G for some reason.
>>
>>Are there any configuration values that could cause FreeBSD to swap even though there's free memory? Are there any config values one could try to change in order to get FreeBSD to use the remaining ~70G of free memory instead of swapping?
>>
>>Let me know if there's any more details you want me to provide and I'll attach those.
>>
>>Thanks!
>>
>>// Mattias
I see similar things. The Jails in question are 11.1. The systems updated to 12.1 do not display this behavior. This 11.1 system runs 5 jails. Swapinfo is shown below:
Device 1K-blocks Used Avail Capacity
/dev/aacd0p3 4194304 1776000 2418304 42%
These numbers are developed from top on the base system
[ 0 50861 ] root
[ 20 281903 ] camden squirellmail/roundcube/postfix/mysql
[ 21 322759 ] bassharbor wordpress/php56
[ 19 343522 ] monhegan wordpress/php56
[ 18 369139 ] newharbor apache24/sendmail
[ 17 587332 ] pemaquid wordpress/php74
Jails: 1904655
total: 1955516
I read somewhere that the virtual memory system pre-pages modified pages as a just-in-case measure. If this is correct, 12.1 does not do this on a non-paging system. The system shown above uses about 10% swapspace after a reboot and works its way to the 42% shown above in a day or so.
--------------------------------------------------------------------------------
On October 15, 2020 18:21 Jon Schneider wrote:
top -w -oswap
seems to report the right thing. Would be interesting if it's _not_ MySQL.
Jon
--------------------------------------------------------------------------------
On October 15, 2020 16:24 Team Twingly wrote:
Hi,
We have a server running FreeBSD 12.1-RELEASE-p10. We currently have a problem where FreeBSD starting to swap when running ZFS scrub, even though we have ~70G of free memory.
This did not happen before when running FreeBSD 11.3 for example. It started happening at approximately the time we upgraded from 12.1-RELEASE-p5 to 12.1-RELEASE-p6, but if the upgrade is the cause of the problem is unclear, though FreeBSD never swapped for us before that. "Laundry" memory was not something we saw before either, it started to appear at the same time as FreeBSD started swapping.
Eventually, after scrubbing a few times, the swap becomes full and we start seeing "swap_pager_getswapspace(24): failed" etc. in dmesg.
This is the memory usage a while after scrubbing, note the values for Mem/Free and Swap:
```
% top | head -n 7
last pid: 8112; load averages: 1.82, 1.77, 1.73 up 6+01:37:42 10:53:48
35 processes: 1 running, 34 sleeping
CPU: 4.9% user, 0.0% nice, 4.2% system, 0.2% interrupt, 90.7% idle
Mem: 110G Active, 27G Inact, 5413M Laundry, 39G Wired, 68G Free
ARC: 34G Total, 28G MFU, 4101M MRU, 53M Anon, 1317M Header, 225M Other
30G Compressed, 53G Uncompressed, 1.77:1 Ratio
Swap: 8192M Total, 6434M Used, 1757M Free, 78% Inuse
```
We are running MySQL, which has been configured to use ~50% of the total amount memory (using innodb_buffer_pool_size=127748M)
ZFS ARC has been configured to use 25% of the total memory (using vfs.zfs.arc_max="63874M")
We have tried raising both vfs.zfs.arc_max and innodb_buffer_pool_size, but this did not make any change to the total memory usage, the free memory stays at around 70G and FreeBSD still started swapping.
It's as if the memory is capped at around 180G for some reason.
Are there any configuration values that could cause FreeBSD to swap even though there's free memory? Are there any config values one could try to change in order to get FreeBSD to use the remaining ~70G of free memory instead of swapping?
Let me know if there's any more details you want me to provide and I'll attach those.
Thanks!
// Mattias
From owner-freebsd-questions@freebsd.org Mon Oct 19 15:33:09 2020
Return-Path: <owner-freebsd-questions@freebsd.org>
Delivered-To: freebsd-questions@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4964942BFB3
for <freebsd-questions@mailman.nyi.freebsd.org>;
Mon, 19 Oct 2020 15:33:09 +0000 (UTC)
(envelope-from freebsd@edvax.de)
Received: from mout.kundenserver.de (mout.kundenserver.de [212.227.126.130])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest
SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256)
(Client CN "mout.kundenserver.de",
Issuer "TeleSec ServerPass Class 2 CA" (verified OK))
by mx1.freebsd.org (Postfix) with ESMTPS id 4CFLPJ0zHvz3WDS
for <freebsd-questions@freebsd.org>; Mon, 19 Oct 2020 15:33:07 +0000 (UTC)
(envelope-from freebsd@edvax.de)
Received: from r56.edvax.de ([94.222.3.6]) by mrelayeu.kundenserver.de
(mreue012 [212.227.15.167]) with ESMTPA (Nemesis) id
1MLAZe-1kmknM2A8w-00IGlA; Mon, 19 Oct 2020 17:33:02 +0200
Date: Mon, 19 Oct 2020 17:33:01 +0200
From: Polytropon <freebsd@edvax.de>
To: "John R. Levine" <johnl@iecc.com>
Cc: "Steve O'Hara-Smith" <steve@sohara.org>, freebsd-questions@freebsd.org,
naddy@mips.inka.de
Subject: Re: printf(1) and UTF-8 multi-byte chars
Message-Id: <20201019173301.0a8fe5b4.freebsd@edvax.de>
In-Reply-To: <3c62a326-887f-4f4e-dbb2-56666f7571a0@iecc.com>
References: <slrnroo8n9.1iu4.naddy@lorvorc.mips.inka.de>
<20201018154838.49CBC239CEDF@ary.qy>
<20201018182309.490ff752536eae2092533c5a@sohara.org>
<3c62a326-887f-4f4e-dbb2-56666f7571a0@iecc.com>
Reply-To: Polytropon <freebsd@edvax.de>
Organization: EDVAX
X-Mailer: Sylpheed 3.1.1 (GTK+ 2.24.5; i386-portbld-freebsd8.2)
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
X-Provags-ID: V03:K1:vYFrt/8kxHGC2uvXvn8Q3q6TqJLVm1MxGSagCtR6ZlWRwl5POdg
BRq4uPQzyV3AP21afxE5e/iEhw0iFE0p5FL2LEQeJY+dxpj6DlADvTK9SEtlpQDkUwc41ok
2b0qDLeUIF+3u2Bbso+uJy+g4ybmCIN0tSGCFnbiFdpAYohwJur7NMFrFQ1HR+29nEc9faO
Af72u0zlNUPANotIoEQfw==
X-Spam-Flag: NO
X-UI-Out-Filterresults: notjunk:1;V03:K0:t+DicC1sugs=:L++zT83+VkAaDQXKxu5miP
pPTWo6rJOtjM6C2K1cnY5RylHiNXsJ5Kqe8L3fudGHw9zhuvUpTrJGSrw01pSRVwRIp/D6nUb
12ZIX1xT411ggwIRshEqxu9qAWo6RDeZGh3N7WgeNzmkdIUMwsIkV/tPE9ogx1liGG/z1p+UI
z1IiE6IsmM6Y785TEz7Mg9V37L1auUHtlVdejcKPF5zIzU3tfwsMmv/gNpQsuXQzrQZudVVu+
HTVArY5DtOrvIB/T/ZE1zeegS+rYz4OoCGd5cqwutZpIFLTK5RQe812PzmCOVLJfNfyKckVh2
tIVzF1lWjiAUWk207WxqxfSa/4DmqYmSIEJsIufSet9VDfQzlWQbFdTRxN6Qc8Bb4ZeZ5Y+r9
P6v9N+4WW4aIRBfVNvOvBB2bsSVSgHcFAjmnShFNudwzMPztX/hTzAJmdGJKW
X-Rspamd-Queue-Id: 4CFLPJ0zHvz3WDS
X-Spamd-Bar: /
Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none;
spf=none (mx1.freebsd.org: domain of freebsd@edvax.de has no SPF policy when
checking 212.227.126.130) smtp.mailfrom=freebsd@edvax.de
X-Spamd-Result: default: False [0.94 / 15.00];
HAS_REPLYTO(0.00)[freebsd@edvax.de];
RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[];
MV_CASE(0.50)[]; HAS_ORG_HEADER(0.00)[];
NEURAL_HAM_SHORT(-0.69)[-0.693];
RECEIVED_SPAMHAUS_PBL(0.00)[94.222.3.6:received];
RCVD_TLS_LAST(0.00)[]; R_DKIM_NA(0.00)[];
FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+];
ASN(0.00)[asn:8560, ipnet:212.227.0.0/16, country:DE];
ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.64)[-0.636];
REPLYTO_EQ_FROM(0.00)[]; FROM_HAS_DN(0.00)[];
RCPT_COUNT_THREE(0.00)[4]; NEURAL_HAM_LONG(-0.14)[-0.136];
MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[edvax.de];
AUTH_NA(1.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[];
MID_CONTAINS_FROM(1.00)[];
RCVD_IN_DNSWL_NONE(0.00)[212.227.126.130:from];
R_SPF_NA(0.00)[no SPF record];
RWL_MAILSPIKE_POSSIBLE(0.00)[212.227.126.130:from];
RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-questions]
X-BeenThere: freebsd-questions@freebsd.org
X-Mailman-Version: 2.1.33
Precedence: list
List-Id: User questions <freebsd-questions.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-questions>,
<mailto:freebsd-questions-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-questions/>;
List-Post: <mailto:freebsd-questions@freebsd.org>
List-Help: <mailto:freebsd-questions-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-questions>,
<mailto:freebsd-questions-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Mon, 19 Oct 2020 15:33:09 -0000
On 18 Oct 2020 14:05:46 -0400, John R. Levine wrote:
> > There are good reasons for using all three levels, here are some:
> >
> > Bytes: Content length headers, malloc calls - storage related
>
> Sure.
>
> > Glyphs: Truncation, apparent length, sorting - appearance related
>
> Not so much. I suppose it's preferable to truncate at a glyph boundary,
> [...]
Depends. Some gylphs depicting ligatures decay to different
single characters upon truncation / hyphenation.
> [...]
> but sorting UTF-8 bytes gives you the same order as sorting the glyphs,
> and for useful sorting you need to deal with issues like normalized forms
> and case folding. Not sure what use apparent length would be since the
> number of glyphs tells you neither the number of visible characters nor
> how wide they are.
Exactly that is the main problem with "byte length != string
length" as a general problem with non-ASCII text. Processing
and even displaying can be quite tricky, and printf() is not
trivial anymore... ;-)
--
Polytropon
Magdeburg, Germany
Happy FreeBSD user since 4.0
Andra moi ennepe, Mousa, ...
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5f8d9d5e14b4c_dad82b12ba6585a44346f>