Date: Wed, 8 Aug 2018 17:15:18 -0600 From: Warner Losh <imp@bsdimp.com> To: bob prohaska <fbsd@www.zefox.net> Cc: Mark Johnston <markj@freebsd.org>, "freebsd-arm@freebsd.org" <freebsd-arm@freebsd.org> Subject: Re: RPI3 swap experiments ["was killed: out of swap space" with: "v_free_count: 5439, v_inactive_count: 1"] Message-ID: <CANCZdfqXEsR7WD=SRB%2BkoCUTyDiXoytYwNWiaAQYnxM20zE=Bw@mail.gmail.com> In-Reply-To: <20180808224227.GB29312@www.zefox.net> References: <2222ABBD-E689-4C3B-A7D3-50AECCC5E7B2@yahoo.com> <20180801034511.GA96616@www.zefox.net> <201808010405.w7145RS6086730@donotpassgo.dyslexicfish.net> <6BFE7B77-A0E2-4FAF-9C68-81951D2F6627@yahoo.com> <20180802002841.GB99523@www.zefox.net> <20180802015135.GC99523@www.zefox.net> <EC74A5A6-0DF4-48EB-88DA-543FD70FEA07@yahoo.com> <20180806155837.GA6277@raichu> <20180808214224.GA29312@www.zefox.net> <CANCZdfpsTapeVYmwPNy1vKCueMh_D4osB8ohJJEV=GPabuxLYw@mail.gmail.com> <20180808224227.GB29312@www.zefox.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Aug 8, 2018 at 4:42 PM, bob prohaska <fbsd@www.zefox.net> wrote: > On Wed, Aug 08, 2018 at 03:55:52PM -0600, Warner Losh wrote: > > On Wed, Aug 8, 2018 at 3:42 PM, bob prohaska <fbsd@www.zefox.net> wrote: > > > > > On Mon, Aug 06, 2018 at 11:58:37AM -0400, Mark Johnston wrote: > > > > > > > > If the above suggestion doesn't help, the next thing to try would be > to > > > > revert the oom_seq value to the default, apply this patch, and see if > > > > the problem continues to occur. If this doesn't help, please try > > > > applying both measures, i.e., set oom_seq to 120 _and_ apply the > patch. > > > > > > > > > > Applying both measures resulted in a panic, not entirely unlike the > effect > > > of deliberately running the machine out of swap on microSD. The first > > > obvious > > > sign of trouble was errors attributed to da0: > > > > > > (da0:umass-sim0:0:0:0): WRITE(10). CDB: 2a 00 00 25 bc b8 00 00 80 00 > > > (da0:umass-sim0:0:0:0): CAM status: CCB request completed with an error > > > (da0:umass-sim0:0:0:0): Retrying command, 3 more tries remain > > > > > > > So we're pushing so hard that the writes are actively failing... > > > > With the da driver there's some hope. Add options IOSCHED to the kernel > > config file and reboot. This will give you some detailed statistics, as > > well as power-of-two bucketized latency histograms. It may even be a > vector > > forward to slow the writes / trims down, though there's some issues when > > you slow writes down TOO much, it helps *A*LOT* keep the system > responsive. > > We do that at work to make our consumer SSDs not suck for serving content > > (reading) while we're doing some writes to them... The thumb drives are > > like the consumer SSDs we buy, only crappier... > > > > I've added some sorted versions of the swapscript output keyed on ms/read > and > ms/write to the directory at > http://www.zefox.net/~fbsd/rpi3/swaptests/r337226M/1gbsdflash_1gbusbflash/ > batchqueue/pageout120/ > and randomly through the older testcases. > > Some of the delays approach a minute, and that's been the case for as long > as I've > been keeping track. However, the big delays tend to be scattered through > the gstat > log and aren't necessarily associated with trouble, nor always with high > levels of > swap usage. > > Mark Johnston's slow_swap.diff patch is compiling now, if that succeeds > I'll > try it first. If it fails (I may have damaged /usr/obj in the last test > run) > I'll add > options IOSCHED > to the GENERIC config file before trying again. Yea, my I/O sched only tracks latencies out to 8.192s, but if we see any writes in that bucket, we know we're 100x slower than normal... Warner >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CANCZdfqXEsR7WD=SRB%2BkoCUTyDiXoytYwNWiaAQYnxM20zE=Bw>