From owner-freebsd-stable@freebsd.org Mon Jun 29 20:20:39 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id E6A5D35403E for ; Mon, 29 Jun 2020 20:20:39 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic312-24.consmr.mail.gq1.yahoo.com (sonic312-24.consmr.mail.gq1.yahoo.com [98.137.69.205]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 49wf4k4VyXz4LW0 for ; Mon, 29 Jun 2020 20:20:38 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: rOXCY2gVM1neCWmegTh4QfZnBMW90XKwzBmIJAfTFhNV3KMm9VcinWM3MJttVR5 qrszsefdDGDCfw858MOEb6ZbNbtgdQ9txv38QsxuAtgARK5AWUZ2HEVYllNud8AIEFXwPi0RSd2A vUE5dX9qH4fy.JgePTheGNUB745O8.iWskNBbiAvY5_4httFkP3KjS6XzUguDnw_.JR_C.9cUiof ybm.WEw1BknPdrr1M44A5qu.cjjYfzrR3oir2RK0XAsE3eQaXEC7qhR90WsZ502vigIx_Bm5nld7 EP.vIzOjD2rM4QSmO9Wdf65S7PtP4bOUn8RpeFem.Lq33_NAOxzwSXxrPcSvXq9QAcxSAzg_EuC2 bLzQJferF3Z93fCAYrl6YINSkXLEsJu6aGTW7yPE5vEKOuSwmsOxqejF5OaIE7fO0_IBqrF5T7ug xaqLkdJXp3Z3LpVGuOYfPsQH17GL4c22D51xHcj5i51sMIQ.ird5i_vMFZsiIu8cTwvzg5jy7d9J LRMjWMSVGcWaMVwpeGsxuHtprx1VSebWJKQFAdKj6Z3ebMcDrsrRVU56.sP33wI1uQVwWXt.k2fG 4yNsyTPMvjO9vWfTxEqCPNXPxmTqmTF3XOolgTdc2KiOTLxALiTpL44a49BBIhETDGKJriPKxla3 D9Az6IZldrcCN7dPbcHcY7hPB2tlEsd04in1nxKG0zVj_HccUYVipcnf6GQm..01VjJ0H3sm9mov 31sD.FbsNgaWq6RUbKc8z8KcS28q.D3kGlRgMXKLWIdRUGDJdeAdjqT02PPJX27DvkOtDroWsaZW 7bi5SfYDFDdAtgBJ9bIx8c1qHetijASEUS8XJpTZyn1mIHDYtvLvG6eXFljnk4FwktvpUdmghSjC Hs9WJLiNHdawa8qMv2eh1AShKLokogY.zmZR.zuX9nf2TczxxyO3ga4c5iwMOg7eIoBsa2gz8Njo QYwkig65v_XobChh5n6kNxdyHIp1.taiUbU1GXd.Caq67OBYxXDIPXBg.VA43lVoInGiNlkfbOFN wpHxdSIqqQBX6AF7CRIScSHf157TC3oaxrAGk.WsnxECjgMiEy90nE65kXqmvBVg4lJiBsvVrbm5 L0Yj7_SoTS1W8nOs1BX50NX15j6H.hYnvSo6dfebDSa7wzM6IX31cpT7zJwSq6MWaOnp0SvBDHdw I8C2Eg8C9MP1wXC2UAi14Xi_36FF_58f9rZQYtG9ERNycGIBgcnNXKx5CQcRfroWJMVCTKY.BGSx 4FnFL5yIPDB7ofrmSzVHo9OqMWV1oHNrVuy2BQvfCNdk_rcRaY.FdxF49._BpnbA5Ah1Bz4ls1M3 qyZpW1aOfN4ALKWZ3jZTSHJUW.krk.B2t6F4HxLuUkyOFdaxk43yM.prSoOlboPIcli7O7F051ak nrptWBkkyTYC.oqq4rzTAj0.1qQ-- Received: from sonic.gate.mail.ne1.yahoo.com by sonic312.consmr.mail.gq1.yahoo.com with HTTP; Mon, 29 Jun 2020 20:20:36 +0000 Received: by smtp429.mail.bf1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID b164c7835c0c9d196e2850e92a893960; Mon, 29 Jun 2020 20:20:35 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\)) Subject: Re: swap space issues From: Mark Millard In-Reply-To: Date: Mon, 29 Jun 2020 13:20:33 -0700 Cc: freebsd-stable@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <2D0E1E39-2607-4D62-A232-F39C6BE1CC0D@yahoo.com> References: To: dwilde1@gmail.com X-Mailer: Apple Mail (2.3608.80.23.2.2) X-Rspamd-Queue-Id: 49wf4k4VyXz4LW0 X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.75 / 15.00]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; TO_DN_NONE(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_SHORT(-1.20)[-1.204]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.05)[-1.047]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.002]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.69.205:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.69.205:from]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Jun 2020 20:20:40 -0000 [I'm now subscribed so my messages should go through to the list.] On 2020-Jun-29, at 06:17, Donald Wilde wrote: > [adding maintainers of synth and ccache] >=20 > On 6/29/20, Mark Millard wrote: >> Based on "small arm system" context experiments >> mostly . . . >>=20 >> If your console messasges do not include >> messages about "swap_pager_getswapspace(...): failed", >> then it is unlikely that being out of swap space >> is the actual issue even when it reports: "was killed: >> out of swap space" messages. For such contexts, making >> the swap area bigger does not help. >>=20 >=20 > It did not show those getswapspace messages. Any other potentially of interest console messages? >> In other words, "was killed: out of swap space" >> is frequently a misnomer and not to be believed >> for "why" the kill happened or what should be >> done about it --without other evidence also being >> present anyway. >>=20 >> Other causes include: >>=20 >> Sustained low free RAM (via stays-runnable processes). >> A sufficiently delayed pageout. >> The swap blk uma zone was exhausted. >> The swap pctrie uma zone was exhausted. >>=20 >> (stays-runnable processes are not swapped out >> [kernel stacks are not swapped out] but do actively >> compete for RAM via paging activity. In such a >> context, free RAM can stay low.) >>=20 >> The below material does not deal with the >> the "exhausted" causes but does deal with >> the other 2. >>=20 >> Presuming that you are getting "was killed: out >> of swap space" notices but are not getting >> "swap_pager_getswapspace failed" notices and >> that kern.maxswzone vs. system load has not >> been adjusted in a way that leads to bad >> memory tradeoffs . . . >>=20 >> I recommend attempting use of, say, (from >> my /etc/sysctl.conf ): >>=20 > Attached is what I tried, but when I ran synth again, I got a > corrupted HDD that fsck refuses to fix, whether in 1U mode or with fs > mounted. It just will not SALVAGE even when I add the -y flag. That is a horrible result. I assume that you rebooted after editing sysctl.conf or manually applied the values separately instead. What sort of console messages were generated? Was the corruption the only issue? Did the system crash? In what way? Your notes on what you set have a incorrect comment about a case that you did not use: # For plunty of swap/paging space (will not # run out), avoid pageout delays leading to # Out Of Memory killing of processes: #vm.pfault_oom_attempts=3D-1 # infinite vm.pfault_oom_attempts being -1 is a special value that disables the the logic for the vm.pfault_oom_attempts and vm.pfault_oom_wait pair: Willing to wait indefinitely relative to how long the pageout takes, no retries. (Other OOM criteria may still be active.) You report using: # For possibly insufficient swap/paging space # (might run out), increase the pageout delay # that leads to Out Of Memory killing of # processes: vm.pfault_oom_attempts=3D 10 vm.pfault_oom_wait=3D 1 # (The multiplication is the total but there # are other potential tradoffs in the factors # multiplied for the same total.) Note: kib might be interested in what happens for, say, 10 and 1, 5 and 2, and 1 and 10. He has asked for such before from someone having OOM problems but, to my knowledge, no one has taken him up on such testing. (He might be only after 10/1 and 1/10 or other specific figures. Best to ask him if you want to try such things for him.) I've always set up to use vm.pfault_oom_attempts=3D-1 (avoiding running out of swap space by how I configure things and what I choose to run). I avoid things like tempfs that compete for RAM, especially in low memory contexts. For 64-bit environments I've never had to have enough swapspace that the boot reported an issue for kern.maxswapzone : more swap is allowed for the same amount of RAM as is allowed for a 32-bit environment. In the 64-bit type of context with 1 GiByte+ of RAM I do -j4 build world buildkernel, 3072 MiBytes of swap. For 2 GiByte+ of RAM I use 4 poudriere builders (one per core), each allowed 4 processes (ALLOW_MAKE_JOBS=3Dyes), so the load average can at times reach around 16 over significant periods. I also use USB SSDs instead of spinning rust. The port builds include a couple of llvm's and other toolchains. But there could be other stuff around that would not fit. (So synth for you vs. poudriere for me is a difference in our contexts. ALso, I stick to default kern.maxswapzone use without boot messages about exceeding the maximum recommended amount. Increasing kern.maxswapzone trades off KVM available for other purposes and I avoid the tradeoffs that I do not understand.) For 32-bit environments with environments with 2 GiByte+ of RAM I have to be more careful to be sure of avoiding running out of swap for what I do. PARALLEL_JOBS=3D2 and ALLOW_MAKE_JOBS=3Dyes for poudruere (so load average around 8 over some periods). -j4 for buildworld buildkernel . For 32-bit 1 GiByte I used -j2 for buildworld buildkernel , 1800 MiBytes of swap. As I remember, I used MAKE_JOBS_NUMBER_LIMIT=3D2 and PARALLEL_JOBS=3D2 for port building with poudriere. (It has been a while.) (My context is head, not stable.) For reference: # sysctl -d vm.pfault_oom_wait vm.pfault_oom_wait: Number of seconds to wait for free pages before = retrying the page fault handler # sysctl -d vm.pfault_oom_attempts vm.pfault_oom_attempts: Number of page allocation attempts in page fault = handler before it triggers OOM handling # sysctl -d vm.pageout_oom_seq vm.pageout_oom_seq: back-to-back calls to oom detector to start OOM (You reported using vm.pageout_oom_seq=3D120 like I use.) > What got corrupted was one of the /usr/.ccache directories, but > 'ccache -C' doesn't clear it. I've not used ccache. So that is another variation in our contexts. I use UFS, not ZFS. I avoid tmpfs and such that complete for memory. > I restored the original /etc/sysctl.conf, but I can't add packages or > ports any more, so I'm afraid I'm going to have to dd if=3D/dev/zero = the > disk and reload from 12.1R and start over again. The corruption itself might be of interest to some system folks if it is reproducible, depending on how things got that way (crash/panic?). > I can't even 'rm -Rf /usr/.ccache'. It says 'Directory not empty'. >=20 > I don't need this system up and running, so I'm not going to make any > more changes until I see if any of you have suggestions to try first. Note: Below I include the part of my original full message that did not make it to the list: >> . . . >>=20 >> # >> # Delay when persistent low free RAM leads to >> # Out Of Memory killing of processes. The >> # delay is a count of kernel-attempts to gain >> # free RAM (so not time units). >> vm.pageout_oom_seq=3D120 >> # >> # For plunty of swap/paging space (will not >> # run out), avoid pageout delays leading to >> # Out Of Memory killing of processes: >> vm.pfault_oom_attempts=3D-1 >> # >> # For possibly insufficient swap/paging space >> # (might run out), increase the pageout delay >> # that leads to Out Of Memory killing of >> # processes: >> #vm.pfault_oom_attempts=3D 10 >> #vm.pfault_oom_wait=3D ??? >> # (The multiplication is the total but there >> # are other potential tradoffs in the factors >> # multiplied for the same total.) >>=20 >> Note: As of stable/12 -r351776 , stable got >> support for vm.pfault_oom_attempts and >> vm.vm.pfault_oom_wait via an MFC. Stable has >> had support for vm.pageout_oom_seq for >> much longer than that. >>=20 >> Note: Larger values of vm.pageout_oom_seq >> will wait even longer. The default value is >> 12 (last I checked). No figure turns off >> the specific mechanism as far as I know. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)