Date: Wed, 8 Aug 2018 15:42:27 -0700 From: bob prohaska <fbsd@www.zefox.net> To: Warner Losh <imp@bsdimp.com> Cc: Mark Johnston <markj@freebsd.org>, "freebsd-arm@freebsd.org" <freebsd-arm@freebsd.org>, bob prohaska <fbsd@www.zefox.net> Subject: Re: RPI3 swap experiments ["was killed: out of swap space" with: "v_free_count: 5439, v_inactive_count: 1"] Message-ID: <20180808224227.GB29312@www.zefox.net> In-Reply-To: <CANCZdfpsTapeVYmwPNy1vKCueMh_D4osB8ohJJEV=GPabuxLYw@mail.gmail.com> References: <2222ABBD-E689-4C3B-A7D3-50AECCC5E7B2@yahoo.com> <20180801034511.GA96616@www.zefox.net> <201808010405.w7145RS6086730@donotpassgo.dyslexicfish.net> <6BFE7B77-A0E2-4FAF-9C68-81951D2F6627@yahoo.com> <20180802002841.GB99523@www.zefox.net> <20180802015135.GC99523@www.zefox.net> <EC74A5A6-0DF4-48EB-88DA-543FD70FEA07@yahoo.com> <20180806155837.GA6277@raichu> <20180808214224.GA29312@www.zefox.net> <CANCZdfpsTapeVYmwPNy1vKCueMh_D4osB8ohJJEV=GPabuxLYw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Aug 08, 2018 at 03:55:52PM -0600, Warner Losh wrote: > On Wed, Aug 8, 2018 at 3:42 PM, bob prohaska <fbsd@www.zefox.net> wrote: > > > On Mon, Aug 06, 2018 at 11:58:37AM -0400, Mark Johnston wrote: > > > > > > If the above suggestion doesn't help, the next thing to try would be to > > > revert the oom_seq value to the default, apply this patch, and see if > > > the problem continues to occur. If this doesn't help, please try > > > applying both measures, i.e., set oom_seq to 120 _and_ apply the patch. > > > > > > > Applying both measures resulted in a panic, not entirely unlike the effect > > of deliberately running the machine out of swap on microSD. The first > > obvious > > sign of trouble was errors attributed to da0: > > > > (da0:umass-sim0:0:0:0): WRITE(10). CDB: 2a 00 00 25 bc b8 00 00 80 00 > > (da0:umass-sim0:0:0:0): CAM status: CCB request completed with an error > > (da0:umass-sim0:0:0:0): Retrying command, 3 more tries remain > > > > So we're pushing so hard that the writes are actively failing... > > With the da driver there's some hope. Add options IOSCHED to the kernel > config file and reboot. This will give you some detailed statistics, as > well as power-of-two bucketized latency histograms. It may even be a vector > forward to slow the writes / trims down, though there's some issues when > you slow writes down TOO much, it helps *A*LOT* keep the system responsive. > We do that at work to make our consumer SSDs not suck for serving content > (reading) while we're doing some writes to them... The thumb drives are > like the consumer SSDs we buy, only crappier... > I've added some sorted versions of the swapscript output keyed on ms/read and ms/write to the directory at http://www.zefox.net/~fbsd/rpi3/swaptests/r337226M/1gbsdflash_1gbusbflash/batchqueue/pageout120/ and randomly through the older testcases. Some of the delays approach a minute, and that's been the case for as long as I've been keeping track. However, the big delays tend to be scattered through the gstat log and aren't necessarily associated with trouble, nor always with high levels of swap usage. Mark Johnston's slow_swap.diff patch is compiling now, if that succeeds I'll try it first. If it fails (I may have damaged /usr/obj in the last test run) I'll add options IOSCHED to the GENERIC config file before trying again. Thanks for reading! bob prohaska
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20180808224227.GB29312>