Date: Mon, 18 Mar 2019 15:31:40 +0100 From: peter.blok@bsd4all.org To: FreeBSD Stable <freebsd-stable@freebsd.org> Subject: Re: Observations from a ZFS reorganization on 12-STABLE Message-ID: <F2685A09-DC42-4EBA-B727-1D5C28CB37BA@bsd4all.org> In-Reply-To: <76f222db-8b75-80ef-ce48-a43217f10e60@denninger.net> References: <58eb1994-41bd-cd22-be66-0024bcbc36e6@denninger.net> <e5c648b4-ba73-5d3a-2924-be4d52e4c267@grosbein.net> <2baf16fd-3767-1dda-d519-995f7ebaf0cb@ingresso.co.uk> <20190318091431.W52549@mulder.mintsol.com> <76f222db-8b75-80ef-ce48-a43217f10e60@denninger.net>
next in thread | previous in thread | raw e-mail | index | archive | help
Same here using mfsbsd from 11-RELEASE. First attempt I forgot to add = swap - it killed the ssh I was using to issue a zfs send on the remote = system. Next attempt I added swap, but ssh got killed too. Third attempt I used mfsbsd from 12-RELEASE. It succeeded. Now I am using mfsbsd 11-RELEASE with added swap and vis.zfs.arc_min and = arc_max to 128Mb (it is a 4GB system) and it succeeds > On 18 Mar 2019, at 15:14, Karl Denninger <karl@denninger.net> wrote: >=20 > On 3/18/2019 08:37, Walter Cramer wrote: >> I suggest caution in raising vm.v_free_min, at least on 11.2-RELEASE >> systems with less RAM. I tried "65536" (256MB) on a 4GB mini-server, >> with vfs.zfs.arc_max of 2.5GB. Bad things happened when the cron >> daemon merely tried to run `periodic daily`. >>=20 >> A few more details - ARC was mostly full, and "bad things" was 1: >> `pagedaemon` seemed to be thrashing memory - using 100% of CPU, with >> little disk activity, and 2: many normal processes seemed unable to >> run. The latter is probably explained by `man 3 sysctl` (see entry = for >> "VM_V_FREE_MIN"). >>=20 >>=20 >> On Mon, 18 Mar 2019, Pete French wrote: >>=20 >>> On 17/03/2019 21:57, Eugene Grosbein wrote: >>>> I agree. Recently I've found kind-of-workaround for this problem: >>>> increase vm.v_free_min so when "FREE" memory goes low, >>>> page daemon wakes earlier and shrinks UMA (and ZFS ARC too) moving >>>> some memory >>>> from WIRED to FREE quick enough so it can be re-used before bad >>>> things happen. >>>>=20 >>>> But avoid increasing vm.v_free_min too much (e.g. over 1/4 of total >>>> RAM) >>>> because kernel may start behaving strange. For 16Gb system it = should >>>> be enough >>>> to raise vm.v_free_min upto 262144 (1GB) or 131072 (512M). >>>>=20 >>>> This is not permanent solution in any way but it really helps. >>>=20 >>> Ah, thats very interesting, thankyou for that! I;ve been bitten by >>> this issue too in the past, and it is (as mentioned) much improved = on >>> 12, but the act it could still cause issues worries me. >>>=20 > Raising free_target should *not* result in that sort of thrashing.=20 > However, that's not really a fix standing alone either since the > underlying problem is not being addressed by either change. It is > especially dangerous to raise the pager wakeup thresholds if you still > run into UMA allocated-but-not-in-use not being cleared out issues as > there's a risk of severe pathological behavior arising that's worse = than > the original problem. >=20 > 11.1 and before (I didn't have enough operational experience with 11.2 > to know, as I went to 12.x from mostly-11.1 installs around here) were > essentially unusable in my workload without either my patch set or the > Phabricator one. >=20 > This is *very* workload-specific however, or nobody would use ZFS on > earlier releases, and many do without significant problems. >=20 > --=20 > Karl Denninger > karl@denninger.net <mailto:karl@denninger.net> = <mailto:karl@denninger.net <mailto:karl@denninger.net>> > /The Market Ticker/ > /[S/MIME encrypted email preferred]/
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?F2685A09-DC42-4EBA-B727-1D5C28CB37BA>