Skip site navigation (1)Skip section navigation (2) 
Date:      Mon, 11 Sep 2017 03:56:45 +0000
From:      Zhixin Wan <Zhixin.Wan@watchguard.com>
To:        "freebsd-hackers@freebsd.org" <freebsd-hackers@freebsd.org>
Subject:   OOM-killer can't work on FreeBSD 11.0
Message-ID:  <DM5PR10MB1754328DAF0A05B39D4B43FBEB680@DM5PR10MB1754.namprd10.prod.outlook.com>
next in thread | raw e-mail | index | archive | help 
Hi,

I have a mail system running FreeBSD 9.3 which is put on VMWare ESXi, it's =
assigned a low memory (1G or 2G) and a reasonable swap disk size (2 x Memor=
y size).
The mail system was running for several years, and didn't see any freeze ev=
en a lot of mail traffic through it.

Recently I upgraded this mail system from FreeBSD 9.3 to FreeBSD 11.0, and =
after running a few days, the mail system got freeze. I can't get any respo=
nse from the console,
and can't login to the mail system with SSH either, except ping to the syst=
em got response. I look into the message log and found a lot of messages:

swap_pager_getswapspace(3): failed
swap_pager_getswapspace(3): failed
swap_pager_getswapspace(4): failed
swap_pager_getswapspace(3): failed
swap_pager_getswapspace(3): failed
swap_pager_getswapspace(4): failed
swap_pager_getswapspace(3): failed
swap_pager_getswapspace(3): failed
swap_pager_getswapspace(3): failed
swap_pager_getswapspace(3): failed
swap_pager_getswapspace(3): failed
swap_pager_getswapspace(3): failed
swap_pager_getswapspace(3): failed
swap_pager_getswapspace(5): failed
swap_pager_getswapspace(3): failed
swap_pager_getswapspace(3): failed
swap_pager_getswapspace(4): failed
swap_pager: out of swap space
swap_pager_getswapspace(1): failed
swap_pager_getswapspace(16): failed
swap_pager_getswapspace(12): failed
swap_pager_getswapspace(9): failed
swap_pager_getswapspace(16): failed
...

It seems that the out of swap cause the system freeze.

To figure out this problem, restore the mail system to previous backup snap=
shot which is running on FreeBSD 9.3.
Put mail traffic pressure on the mail system, and observe the memory and sw=
ap space usage with a simple shell:

#!/bin/sh
while [ 1 ]; do
vmstat
pstat -s
sleep 60
done

>From the console, I saw the memory and swap space usage increased quickly. =
Once the swap space was eat out,
out of swap messages will be shown in message log:

swap_pager_getswapspace(4): failed
swap_pager_getswapspace(3): failed
swap_pager_getswapspace(3): failed
swap_pager_getswapspace(3): failed
swap_pager_getswapspace(4): failed
swap_pager_getswapspace(6): failed
swap_pager_getswapspace(2): failed
swap_pager_getswapspace(2): failed
swap_pager_getswapspace(2): failed
swap_pager_getswapspace(5): failed
swap_pager_getswapspace(8): failed
swap_pager_getswapspace(2): failed
swap_pager_getswapspace(4): failed
Sep 6 08:30:58 mail-system kernel: pid 92324 (bm_scanner), uid 5500, was ki=
lled: out of swap space

Compared to FreeBSD 11.0, there are still a lot of "swap_pager_getswapspace=
 failed" messages, except FreeBSD 9.3 will kill a process to free memory.
This behavior cause the mail system can go on running, but FreeBSD 11.0 can=
't. Observe the system memory and swap space usage continuously,
the OOM-killer works accurately: once the swap space usage is 100%, the OOM=
-killer will be called to kill a process to free memory.

Dig into the source code of FreeBSD 9.3, file vm_pageout.c, function vm_pag=
eout_scan():
                /*
                * If we are critically low on one of RAM or swap and low on
                * the other, kill the largest process.  However, we avoid
                * doing this on the first pass in order to give ourselves a
                * chance to flush out dirty vnode-backed pages and to allow
                * active pages to be moved to the inactive queue and reclai=
med.
                */
                if (pass !=3D 0 &&
                    ((swap_pager_avail < 64 && vm_page_count_min()) ||
                     (swap_pager_full && vm_paging_target() > 0)))
                                vm_pageout_oom(VM_OOM_MEM);

the corresponding source code in FreeBSD 11.0, file vm_pageout.c, function =
vm_pageout_scan():
        /*
         * If the inactive queue scan fails repeatedly to meet its
         * target, kill the largest process.
         */
        vm_pageout_mightbe_oom(vmd, page_shortage, starting_page_shortage);

The OOM-killer function vm_pageout_oom() is wrapped with function vm_pageou=
t_mightbe_oom().

To know from which commit this behavior was changed, I search the FreeBSD S=
VN page and find a clue.
https://svnweb.freebsd.org/base?view=3Drevision&revision=3D290920
In SVN commit r290920, a new sysctl node called vm.pageout_oom_seq was adde=
d to control the sensitivity of OOM-killer.
The default value of pageout_oom_seq is 12, the commit log said:
The number of passes to trigger OOM was selected empirically and
tested both on small (32M-64M i386 VM) and large (32G amd64)
configurations.

However, in my case, even vm.pageout_oom_seq is 12 by default, it didn't wo=
rk as expected.
I doubt it's a bug, but I'm not pretty sure since I can't fully understand =
these codes.
I just want OOM-killer behaving on FreeBSD 11.0 like FreeBSD 9.3 does.
Is there anyone know how to solve it?

Thanks!

Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?DM5PR10MB1754328DAF0A05B39D4B43FBEB680>