From owner-freebsd-arm@freebsd.org Wed Aug 8 23:15:20 2018 Return-Path: Delivered-To: freebsd-arm@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 764AA106EB89 for ; Wed, 8 Aug 2018 23:15:20 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-it0-x230.google.com (mail-it0-x230.google.com [IPv6:2607:f8b0:4001:c0b::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 05DB87169A for ; Wed, 8 Aug 2018 23:15:20 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-it0-x230.google.com with SMTP id j81-v6so154075ite.0 for ; Wed, 08 Aug 2018 16:15:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=AaPjKjDCrKLsA2ONly+7HNzZSKy1qRpC9M+2ujQDyP0=; b=xsnkjhCGTP/tbvy4EEmTyNjBaxTU20yJ1vMIH2mK1H+EJy13hLzQ0m+pbN6KIoxd7f SMU0xCC/eQfu/8Ky7P5k6updd2ittyaKSZ0indOfCwYPtaolHgxFLdwF5l1tcBMTAEos J4nQxndufzTRVSSFaJ0cJs3R8zyBgZ24hK0IeFdhsla3uQ9dH+hliWUZDaVs6DX0og8Q ajyIc0dBY0DDvEyBRSkODIO0V5CEVIjyGZPxp9kSBZ/zTpJ0R60Ip8R8WIO3dggRoWNA oVj/3oDQZVcHtdGC78rhL5fGsexrGaivzb7mXxEqji3X+jMdrI/pPqhlr8oY9M4uzxnE 6VmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=AaPjKjDCrKLsA2ONly+7HNzZSKy1qRpC9M+2ujQDyP0=; b=Emo0S7A/WaN96Y91jZ0KRNmYtXVOaRskuG2M2nm1dt1aq3xSFGxwDn4qJm81bTg800 PStZNlgbKOvqf4j8OGknjxY6OsZc5EUWg316LTrvxhLeeCWO81A7cRwJ7kNuxIVJUWI7 RNu9UlOYKUF/YcBSW6EOS/cUitchtPJf3KHMhWkGI63oIJ4ZxxqNiCElh84iB/NsvEyQ ufvkQwoI4MiivBtLDLc2JZb36NGq2Og8j2cc5mAumyjAFS3+Sj3sL8E6LS+VCcPcd2mh L45KuR6pUuapmTJxQBGkgqWc46PXwFEZSCw+zEO9IIyx4UrLD8dTiEr0nMIlFWKaNV8e AZKA== X-Gm-Message-State: AOUpUlFurY5ddRiqTp65gqk2GRnyZS2FIrvGkIdT9zpRoUC/UaLso/EK yuWRoos6rIwLgsz7Is7sjimcDHAJVUXS8yllKkIFxLlVduDYQA== X-Google-Smtp-Source: AA+uWPwFyWIsFpaTmMXwsfk+bQzCRrQikDkXOZ0CY0ZEdYYJPPcqKlARic2Y/4sOLJM/Zoylx/RLV3NsRHazT4p6imA= X-Received: by 2002:a24:b211:: with SMTP id u17-v6mr40748ite.1.1533770119286; Wed, 08 Aug 2018 16:15:19 -0700 (PDT) MIME-Version: 1.0 Sender: wlosh@bsdimp.com Received: by 2002:a4f:381a:0:0:0:0:0 with HTTP; Wed, 8 Aug 2018 16:15:18 -0700 (PDT) X-Originating-IP: [2603:300b:6:5100:1052:acc7:f9de:2b6d] In-Reply-To: <20180808224227.GB29312@www.zefox.net> References: <2222ABBD-E689-4C3B-A7D3-50AECCC5E7B2@yahoo.com> <20180801034511.GA96616@www.zefox.net> <201808010405.w7145RS6086730@donotpassgo.dyslexicfish.net> <6BFE7B77-A0E2-4FAF-9C68-81951D2F6627@yahoo.com> <20180802002841.GB99523@www.zefox.net> <20180802015135.GC99523@www.zefox.net> <20180806155837.GA6277@raichu> <20180808214224.GA29312@www.zefox.net> <20180808224227.GB29312@www.zefox.net> From: Warner Losh Date: Wed, 8 Aug 2018 17:15:18 -0600 X-Google-Sender-Auth: FSMv6HTPEt_AU5hs9Px44EPq0Jw Message-ID: Subject: Re: RPI3 swap experiments ["was killed: out of swap space" with: "v_free_count: 5439, v_inactive_count: 1"] To: bob prohaska Cc: Mark Johnston , "freebsd-arm@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Aug 2018 23:15:20 -0000 On Wed, Aug 8, 2018 at 4:42 PM, bob prohaska wrote: > On Wed, Aug 08, 2018 at 03:55:52PM -0600, Warner Losh wrote: > > On Wed, Aug 8, 2018 at 3:42 PM, bob prohaska wrote: > > > > > On Mon, Aug 06, 2018 at 11:58:37AM -0400, Mark Johnston wrote: > > > > > > > > If the above suggestion doesn't help, the next thing to try would be > to > > > > revert the oom_seq value to the default, apply this patch, and see if > > > > the problem continues to occur. If this doesn't help, please try > > > > applying both measures, i.e., set oom_seq to 120 _and_ apply the > patch. > > > > > > > > > > Applying both measures resulted in a panic, not entirely unlike the > effect > > > of deliberately running the machine out of swap on microSD. The first > > > obvious > > > sign of trouble was errors attributed to da0: > > > > > > (da0:umass-sim0:0:0:0): WRITE(10). CDB: 2a 00 00 25 bc b8 00 00 80 00 > > > (da0:umass-sim0:0:0:0): CAM status: CCB request completed with an error > > > (da0:umass-sim0:0:0:0): Retrying command, 3 more tries remain > > > > > > > So we're pushing so hard that the writes are actively failing... > > > > With the da driver there's some hope. Add options IOSCHED to the kernel > > config file and reboot. This will give you some detailed statistics, as > > well as power-of-two bucketized latency histograms. It may even be a > vector > > forward to slow the writes / trims down, though there's some issues when > > you slow writes down TOO much, it helps *A*LOT* keep the system > responsive. > > We do that at work to make our consumer SSDs not suck for serving content > > (reading) while we're doing some writes to them... The thumb drives are > > like the consumer SSDs we buy, only crappier... > > > > I've added some sorted versions of the swapscript output keyed on ms/read > and > ms/write to the directory at > http://www.zefox.net/~fbsd/rpi3/swaptests/r337226M/1gbsdflash_1gbusbflash/ > batchqueue/pageout120/ > and randomly through the older testcases. > > Some of the delays approach a minute, and that's been the case for as long > as I've > been keeping track. However, the big delays tend to be scattered through > the gstat > log and aren't necessarily associated with trouble, nor always with high > levels of > swap usage. > > Mark Johnston's slow_swap.diff patch is compiling now, if that succeeds > I'll > try it first. If it fails (I may have damaged /usr/obj in the last test > run) > I'll add > options IOSCHED > to the GENERIC config file before trying again. Yea, my I/O sched only tracks latencies out to 8.192s, but if we see any writes in that bucket, we know we're 100x slower than normal... Warner >