Date: Fri, 02 Mar 2018 10:15:57 -0700 From: Ian Lepore <ian@freebsd.org> To: Warner Losh <imp@bsdimp.com>, Mike <the.lists@mgm51.com> Cc: "freebsd-arm@freebsd.org" <freebsd-arm@freebsd.org> Subject: Re: Is maximum swap usage tunable? Message-ID: <1520010957.23690.10.camel@freebsd.org> In-Reply-To: <CANCZdfq_MpxjUyVd-=%2BMiAQAER4TeDh9irhx_evdXwa3yt3h0g@mail.gmail.com> References: <20180228170311.GA26187@www.zefox.net> <a759ecea-83f4-b0b2-7113-c39633f68637@mgm51.com> <20180228185517.GB26187@www.zefox.net> <8f422161-885e-aa91-eacd-018540222d65@mgm51.com> <20180228214301.GA29481@www.zefox.net> <b82801b8-bc29-414c-1170-621bb4a5d937@mgm51.com> <CANCZdfq_MpxjUyVd-=%2BMiAQAER4TeDh9irhx_evdXwa3yt3h0g@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, 2018-03-02 at 10:04 -0700, Warner Losh wrote: > On Fri, Mar 2, 2018 at 9:19 AM, Mike <the.lists@mgm51.com> wrote: > > > > > On 2/28/2018 4:43 PM, bob prohaska wrote: > > > > > > On Wed, Feb 28, 2018 at 02:37:36PM -0500, Mike wrote: > > > > > > > > On 2/28/2018 1:55 PM, bob prohaska wrote: > > > > > > > > > > On Wed, Feb 28, 2018 at 12:20:56PM -0500, Mike wrote: > > > > > > > > > > > > On 2/28/2018 12:03 PM, bob prohaska wrote: > > > > > > > > > > > > > > In watching system compilations on an RPi3 it looks as if the > > > > > > > system starts killing processes with "out of swap" warnings > > > > > > > well below 50% of full utilization (in this case, 2 GB). One > > > > > > > recent instance of make -j4 kernel-toolchain killed llvm-tblgen > > > > > > > with only 34% of the swap in use. > > > > > > > > > > > > > > Is the maximum swap usage limit adjustable in any way? I didn't > > > > > > > recognize anything useful in the page at > > > > > > > https://www.freebsd.org/cgi/man.cgi?query=sysctl(8)&; > > sektion=&manpath=freebsd-release-ports > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > It's possible the problem is really swap speed, rather than size, so > > I'd > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > like to try changing size limits if possible. The swap media claims > > 2-3 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > MB/sec random write speed and observations with gstat seem to > > support the > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > claim, but transient stalls are hard to observe. An RPi2 with similar > > > > > > > hardware seems to have no problems. > > > > > > > > > > > > > > > > > > > > > > > > > > > It's possible the problem is really swap speed > > > > > > I was running into swap speed / timeout issues. There were messages > > on > > > > > > > > > > > > > > > > > > > > > > > > the console to that effect. > > > > > > > > > > > > Once I put the swap space on rotating rust, that part of the compile > > > > > > problem disappeared. I use 1GB swap space. > > > > > > > > > > > The latest kernel versions seem to have largely done away with the > > > > > "indefinite wait buffobj" warnings. They're few and far between, > > > > > the compile proceeds unless they're abundant. The fact that armv7 > > > > > has no problem, and the system is reporting "out of swap" with 34% > > > > > in use strikes me as suspicious. Kernel and userland are in sync, > > > > > so the figure of 34% swap usage is probably accurate. > > > > > > > > > > Perhaps my question is better phrased thus: How does FreeBSD-arm > > > > > determine when it's out of swap? > > > > > > > > > > > > > Thanks for the follow-up. > > > > > > > > I was planning to download and try the > > > > > > > > RPI3-20180226-r330034 > > > > > > > > image file this evening. Per your comment, I'll not use the rusty swap > > > > space, and see what happens. > > > > > > > > I'll report back, probably on the morrow... > > > > > > > FWIW, I've been using Sandisk Extreme USB 3 flash drives for > > > /usr /var/ and swap on the Pi3. It seems that flash write > > > speed is a tighter bottleneck than USB 2.0 ports on the Pi. > > > Attempts to use older, slower USB 2 flash drives on a Pi2 > > > didn't work, though the symptoms were never "out of swap". > > > > > buildworld is still running. I started it around 3PM (EST) on March 1. > > > > I'm seeing a lot of swap messages on the console. The text is: > > > > swap_pager: indefinite wait buffer: bufobj: 0, > > blkno: [varies], size: [varies] > > This message comes from the swap pager when the I/O doesn't complete > quickly enough. It has nothing to do with sizes of the swap space. > > I've seen three causes for this: (1) The driver going crazy and not > completing I/Os it gets notified as being done; (2) The driver not having a > backstop timer on I/Os to fail them with timeout; (3) timeouts not firing > for the driver to fail the I/O up the stack. (4) when a drive suddenly slow > way way down (~100x or more slower), the I/O queues balloons and you can > get these. > > Warner You forgot a cause: (5) swap is on an sdcard where taking 30-90 seconds to complete an IO is "normal". Making it even more fun, there's a sort of (5.5) bullet: an sdcard can "lend" its horrible performance to every other storage device in the system. If there is a ton of IO queued up to the sd device (such as when .o files from a a make -j are ending up there) then all the buffers in the system get stacked up in the sd device's bio queue and IO to other devices suffers. I've always thought the new(ish) IO scheduler stuff should be able to help with that in some way, but I never get around to looking at it. -- Ian
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1520010957.23690.10.camel>