Date: Sat, 27 Oct 2018 07:38:19 +0300 From: Konstantin Belousov <kostikbel@gmail.com> To: Robert <robert.ayrapetyan@gmail.com> Cc: Mark Millard <marklmi@yahoo.com>, FreeBSD <freebsd-hackers@freebsd.org>, Mark Johnston <markj@freebsd.org>, Rozhuk Ivan <rozhuk.im@gmail.com> Subject: Re: Sudden grow of memory in "Laundry" state Message-ID: <20181027043819.GX5335@kib.kiev.ua> In-Reply-To: <42f6544f-830c-18c5-e1a8-0acc4c3f09cc@gmail.com> References: <ce38cbfa-e1c5-776e-ef2e-2b867c9a520f@gmail.com> <20180911150849.GD92634@raichu> <104be96a-c16b-7e7c-7d0d-00338ab5a106@gmail.com> <20180928152550.GA3609@raichu> <e705099c-ea42-4985-1012-50e9fa11addd@gmail.com> <20181024211237.302b72d9@gmail.com> <981C887D-78EB-46D2-AEE5-877E269AF066@yahoo.com> <c25e19a4-d3ef-e419-06f8-8a86082dbf31@gmail.com> <E4B508E7-04CC-41BD-934B-19EE69E85800@yahoo.com> <42f6544f-830c-18c5-e1a8-0acc4c3f09cc@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, Oct 26, 2018 at 03:07:07PM -0700, Robert wrote: > Sorry, let me be more specific. > > Please look into: > https://docs.google.com/spreadsheets/d/152qBBNokl4mJUc6T6wVTcxaWOskT4KhcvdpOL68gEUM/edit?usp=sharing > (wait until charts fully loaded). > > These are all memory states and mmap\munmap stats collected. Y axis is > in MBs, X is a timeline. > > It's not a problem to understand which process produces allocations and > is being swapped. I know this for sure. > > The issue is: I strongly believe that by some reason FreeBSD kernel > fails to reuse deallocated memory properly. > > Looking into graphs we can see following: > > 1. When process allocates memory (mmap), "Active" memory increases, > "Free" memory decreases (that's expected). No, mmap(2) call only causes some small amount of the kernel memory allocation, to track the mmaped region. Pages are allocated on the first touch, lets ignore the prefaulting mechanism which is not too relevant for the discussion. Mapped page can be either active or inactive, this is determined by the usage history. > > 2. When process deallocates memory (munmap), "Inactive" memory > increases, "Active" memory decreases. Again no. Unmapped pages does not get active references from userspace which touch them, so eventually pagedaemon moves active unmapped memory to inactive. > > Memory never returns into "Free" state. That's kind of expected as well. When the last mapping of the anonymous swap-backed page is destroyed, there is no point in retaining the page content, so it is freed. For the pages backed by the file, there is no point to drop content which we already paid for by read. > > 3. At some point, when sum of "Active" and "Inactive" memory exceeds > some upper memory limits, > > OS starts to push "Inactive" memory into "Laundry" and "Swap". This > happens very quick and unexpectedly. So this is the pattern of behaviour of your applications. > > Now why OS doesn't reuse huge amounts of "Inactive" memory when calling > mmap? Because typically active and inactive pages carry some useful content, which cannot be lost. Before reusing the page, pagedaemon must ensure that the page is written to file for file-backed mappings, or to swap for anonymous mappings, so its data can be restored when needed again. If the page content is synced with the permanent storage, it is called clean, otherwise dirty. Reuse of dirty page requires changing its state to clean by write-out. Laundry is the queue where the dirty pages sit waiting for write, this is done to not clog inactive queue since the page was already processed by the page daemon and decided for laundry. > > Or my assumption about availability of "Inactive" memory is wrong? Which > one is free for allocations then? Yes, you assumptions are wrong. Active/inactive represent the usage/reference history, not the reusability (clean/dirty) state. > > Thanks. > > > On 10/24/18 11:58 AM, Mark Millard wrote: > > On 2018-Oct-24, at 1:25 PM, Robert <robert.ayrapetyan at gmail.com> wrote: > > > >> Sorry, that wasn't my output, mine (related to the screenshot I've sent earlier) is: > > No screen shot made it through the list back out to those that > > get messages from the freebsd-hackers at freebsd.org reference > > in the CC. The list limits itself to text as I understand. > > > >> Mem: 1701M Active, 20G Inact, 6225M Laundry, 2625M Wired, 280M Free > >> ARC: 116M Total, 6907K MFU, 53M MRU, 544K Anon, 711K Header, 55M Other > >> 6207K Compressed, 54M Uncompressed, 8.96:1 Ratio > >> Swap: 32G Total, 15G Used, 17G Free, 46% Inuse > > Relative to my limited point: I do not know the status of > > mutually-exclusive categorizations vs. not for ZFS ARC and > > Mem. > > > > Unfortunately, as I understand things, it is questionable if > > adding -S to the top command gives you swap information that > > can point to what makes up the 15G swapped out by totaling > > the sizes listed. But you might at least be able to infer > > what processes became swapped out even if you can not get > > a good size for the swap space use for each. > > > > Using -ores does seem like it puts the top users of resident > > memory at the top of top's process list. > > > > Sufficient Active RAM use by processes that stay active will > > tend to cause inactive processes to be swapped out. FreeBSD > > does not swap out processes that stay active: it pages those > > as needed instead of swapping. > > > > So using top -Sores might allow watching what active > > process(es) grow and stay active and what inactive processes > > are swapped out at the time of the activity. > > > > I do infer that the 15G Used for Swap is tied to processes > > that were not active when swapped out. > > > >> I'm OK with a low "Free" memory if OS can effectively allocate from "Inactive", > >> > >> but I'm worrying about a sudden move of a huge piece of memory into "Swap" without any relevant mmap calls. > >> > >> > >> My question is: what else (except mmap) may reduce "Free" memory and increase "Laundry"\"Swap" in the system? > >> > >> Thanks. > >> > >> > >> On 10/24/18 9:34 AM, Mark Millard wrote: > >>> On 2018-Oct-24, at 11:12 AM, Rozhuk Ivan <rozhuk.im at gmail.com> wrote: > >>> > >>>> On Wed, 24 Oct 2018 10:19:20 -0700 > >>>> Robert <robert.ayrapetyan@gmail.com> wrote: > >>>> > >>>>> So the issue is still happening. Please check attached screenshot. > >>>>> The green area is "inactive + cached + free". > >>>>> > >>>>> . . . > >>>> +1 > >>>> Mem: 845M Active, 19G Inact, 4322M Laundry, 6996M Wired, 1569M Buf, 617M Free > >>>> Swap: 112G Total, 19M Used, 112G Free > >>> Just a limited point based on my understanding of "Buf" in > >>> top's display . . . > >>> > >>> If "cached" means "Buf" in top's output, my understanding of Buf > >>> is that it is not a distinct memory area. Instead it totals the > >>> buffer space that is spread across multiple states: Active, > >>> Inactive, Laundry, and possibly Wired(?). > >>> > >>> In other words: TotalMemory = Active+Inact+Laundry+Wired+Free. > >>> If Buf is added to that then there is double counting of > >>> everything included in Buf and the total will be larger > >>> than the TotalMemory. > >>> > >>> Also Inact+Buf+Free may double count some of the Inact space, > >>> the space that happens to be inactive buffer space. > >>> > >>> I may be wrong, but that is my understanding. > >>> > > > > === > > Mark Millard > > marklmi at yahoo.com > > ( dsl-only.net went > > away in early 2018-Mar) > > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20181027043819.GX5335>