Date: Tue, 29 Dec 2020 17:40:09 +0000 (UTC) From: doug@safeport.com To: Michael Schuster <michaelsprivate@gmail.com> Cc: Pete Wright <pete@nomadlogic.org>, freeBSD Mailing List <freebsd-questions@freebsd.org> Subject: Re: Observations on virtual memory operations Message-ID: <af8ceb5-8ac-57f5-24f8-72b114fa8a@safeport.com> In-Reply-To: <CADqw_gKWdMaMpCq=%2By=_pvd6wp2bJvDsWkv5GYTwrznhCSpfeQ@mail.gmail.com> References: <167603f-a82a-7031-6850-2d08f17a36@fledge.watson.org> <8f3a278a-56cd-c732-68a0-cf6fa5d50a3f@nomadlogic.org> <CADqw_gKWdMaMpCq=%2By=_pvd6wp2bJvDsWkv5GYTwrznhCSpfeQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, 29 Dec 2020, Michael Schuster wrote: > On Tue, Dec 29, 2020, 00:37 Pete Wright <pete@nomadlogic.org> wrote: > >> >> >> On 12/28/20 3:25 PM, doug wrote: >>> I have two servers running jails that "routinely" run out of swapspace >>> with >>> no demand paging activity. To try and get a handle on VM/swapspace >>> management I have been tracking swapinfo vs memory use as measured by >>> top. >>> The numbers do not exactly add up but I assume that is not involved in my >>> issue. >>> >> <snip> >>> >>> The other day I caught the system at 73% swapspace used. At this level >>> the >>> system was in a near thrashing state in that typing a key got it >>> echoed in >>> 10 <--> 30 seconds. There was about 600MB of swapspace at this point. I >>> would think there is no way to debug this except as a thought experiment. >> >> The first thing that comes to mind is do you have the ability to hook >> any metrics/monitoring onto this system. For example, I use collectd on >> my systems to report overall CPU/memory metrics as well as per-process >> memory metrics. >> >> Alternatively you could write a simple shell script that run's "ps" and >> parses the output of memory utilization on a per-process basis. >> >> either of the above approaches should give you some insight into where >> the memory leak is coming from (assuming you already do not know). >> >> one trick i use is to invoke a process with "limits" to ensure it does >> not exceed a certain amount of memory that I allocate to it. for example >> with firefox i do this: >> $ limits -m 6g -v 6g /usr/local/bin/firefox >> >> that should at least buy you enough time to investigate why the process >> needs so much memory and see what you can do about it. > Thank you all for the information and thoughts. If vmstat produces correct infomation there is no demand paging. The limiting condition on these systems is swapfile space rather than real memory. There are 69 sysctl elements dealing with paging and swapfile. If there is documentation (other than in C) on these that would be helpful perhaps. Most are totals, demand paging rates may be in this set, but not so as I can tell. The one time I caught the system dying the limiting resource was swapspace. There was no paging (last vmstat) and about 670MB left in the swapfile. In this state I could recover by restarting apache.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?af8ceb5-8ac-57f5-24f8-72b114fa8a>