From owner-freebsd-hackers@freebsd.org Sun Nov 18 00:00:00 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E47D1110B52F for ; Sat, 17 Nov 2018 23:59:59 +0000 (UTC) (envelope-from ian@freebsd.org) Received: from outbound3d.ore.mailhop.org (outbound3d.ore.mailhop.org [54.186.57.195]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2D67D86B27 for ; Sat, 17 Nov 2018 23:59:59 +0000 (UTC) (envelope-from ian@freebsd.org) ARC-Seal: i=1; a=rsa-sha256; t=1542499184; cv=none; d=outbound.mailhop.org; s=arc-outbound20181012; b=u6G4ddPhcLAW2JkCpIq3nT0C/hRCyxXTG+pXZdHSdkOWPk8OB3MjJOox9hp3qXcOETrU8CegMlQJE LnujYaz+2svgLg2q2chOtlNWyuayU/XoJfve/NPrx4pT/xldSmbPwbn9nHmABq3AduWLtA+QnvxWt/ 4Ymk9AtzYHTg53d/s9T0KKZ8vaWjRwnOd1FxS23ld62nw3C7r4KtrsUxklrKPcc+4ACNYQUubiLcj4 ODftH72gMln5eaHbwrcf9Jj93Rw9GXTuwZrJejlSABjYtuapkcuF9w1IZV/4OXWNOs6GXcBnF5IlAt bqQulou7ewD04K6sowSIBaD5x6S7ffw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=outbound.mailhop.org; s=arc-outbound20181012; h=content-transfer-encoding:mime-version:content-type:references:in-reply-to: date:cc:to:from:subject:message-id:dkim-signature:from; bh=veg/9AszBpH1+s+kmoTl6RFsFGteZl8Flcs4ZI1AfOc=; b=FkgzUH5glJrjjQtSrpDkt+w3zMX0ZLtmlYg9GnI8pwfLt6KbR8g8GHhFR283vX9ett6foeyi95cvr FMKur1f6bA4Q7uFmzIjfXhsmg2DGRJZ4VNG2nVTIMZKjhaiNPwqG6dx0WWJPtEnLwdbuqbUn6ft9V/ +rbwn+sSiLaHq59lSOToIr3nn0AdTpbMoP21vcipDLyR2ZmDhf99q71w7S2ap9at6aNw/bNJtm8hFW 6XnMJ2FetVMLW8wVFvpU8UGgY/G4SgzAOeZaNX8Pkk1/T+ZIZcvhdpgLykIWkpRx8Pv34OC3qK7o0k vejR2JXvCgsoW39UaLtgXmDalt1fsEg== ARC-Authentication-Results: i=1; outbound3.ore.mailhop.org; spf=softfail smtp.mailfrom=freebsd.org smtp.remote-ip=67.177.211.60; dmarc=none header.from=freebsd.org; arc=none header.oldest-pass=0; DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=outbound.mailhop.org; s=dkim-high; h=content-transfer-encoding:mime-version:content-type:references:in-reply-to: date:cc:to:from:subject:message-id:from; bh=veg/9AszBpH1+s+kmoTl6RFsFGteZl8Flcs4ZI1AfOc=; b=gQ0HyV/M4Iqt6gpcVWAYaWFeAkUYn9GCjPXjc+9pDhA/rX1nd6lMBfT8recN1pDmNT6wXhXZAsoYI LHOZew7f1NJARHNI2yM3Um032i6tk6x3GNhGxw22R7+cfgxZVOhiKWw+P93xqPD+Q+CtqKoQ96350t 0bx0XZWo+zxt2jLTF5WLI+2DQZtyE7NwXZG+Js6qsE2EvmNVo2NZhVm1/FxVSuCX5UpAGumk7syU5X ThutT7VxUtXuBfyt8R7zEt9ag6/KoAIBsOhvXFj4IJx9F+ArJJTnkrbUR/frwAJnTVf1l/hT6RxGs8 VIk9lcBcm9/UKEmS/yR5TsAJKI3uarQ== X-MHO-RoutePath: aGlwcGll X-MHO-User: d94d7790-eac4-11e8-a59a-7b143e15dabc X-Report-Abuse-To: https://support.duocircle.com/support/solutions/articles/5000540958-duocircle-standard-smtp-abuse-information X-Originating-IP: 67.177.211.60 X-Mail-Handler: DuoCircle Outbound SMTP Received: from ilsoft.org (unknown [67.177.211.60]) by outbound3.ore.mailhop.org (Halon) with ESMTPSA id d94d7790-eac4-11e8-a59a-7b143e15dabc; Sat, 17 Nov 2018 23:59:43 +0000 (UTC) Received: from rev (rev [172.22.42.240]) by ilsoft.org (8.15.2/8.15.2) with ESMTP id wAHNxmJi008501; Sat, 17 Nov 2018 16:59:48 -0700 (MST) (envelope-from ian@freebsd.org) Message-ID: <1542499188.56571.59.camel@freebsd.org> Subject: Re: 13-CURRENT: several GB swap being used despite plenty of free RAM From: Ian Lepore To: Wojciech Puchar , Rebecca Cran Cc: freebsd-hackers Hackers Date: Sat, 17 Nov 2018 16:59:48 -0700 In-Reply-To: References: <1748688.u6MfGjpqfb@photon.int.bluestop.org> Content-Type: text/plain; charset="ISO-8859-1" X-Mailer: Evolution 3.18.5.1 FreeBSD GNOME Team Port Mime-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 2D67D86B27 X-Spamd-Result: default: False [-0.03 / 15.00]; local_wl_from(0.00)[freebsd.org]; NEURAL_HAM_SHORT(-0.03)[-0.030,0]; ASN(0.00)[asn:16509, ipnet:54.186.0.0/15, country:US] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Nov 2018 00:00:00 -0000 On Sat, 2018-11-17 at 22:52 +0100, Wojciech Puchar wrote: > freebsd will not swap with that lots of free ram. > but it's 90GB free NOW, how about before? > Your information is outdated. For at least a couple years now (since approximately the 10.1 - 10.2 timeframe is my vague estimate), freebsd will page out application memory that hasn't been referenced for some time, even when the system has no shortage of free memory at all. The advice I was recently given to revert to the old behavior is:   sysctl vm.pageout_update_period=0 I've been using it on a couple systems here for a few days now, and so far results are promising, I am no longer seeing gratuitous swapfile usage on systems that have so much free physical ram that they should never need to page anything out. I haven't yet pushed one of those systems hard enough to check what happens when they do need to start proactively paging out inactive memory due to shortages -- it could be that turning off the new behavior has downsides for some workloads. -- Ian >  > If something got swapped it isn't read back just because there is > free  > memory but when it will be needed. > On Sat, 17 Nov 2018, Rebecca Cran via freebsd-hackers wrote: > > > > > I'm running 13-CURRENT from a few days ago. I noticed the system > > using several > > GB of swap despite having 90GB RAM still free. I know FreeBSD will > > use *some*, > > but 3450MB seems excessive when there's still 90GB RAM free. > > > > CPU:  0.1% user,  0.0% nice,  0.0% system,  0.0% interrupt, 99.8% > > idle > > Mem: 3392M Active, 5489M Inact, 390M Laundry, 25G Wired, 90G Free > > ARC: 19G Total, 8080M MFU, 9242M MRU, 64K Anon, 197M Header, 1570M > > Other > >     15G Compressed, 26G Uncompressed, 1.73:1 Ratio > > Swap: 8192M Total, 3450M Used, 4742M Free, 42% Inuse, 36K In > > > > Quitting firefox caused the swap usage to drop to just 460MB. I > > don't > > understand why it would decide to swap out so much, especially > > since I have > > vm.swap_idle_enabled set to 0. > > From owner-freebsd-hackers@freebsd.org Sun Nov 18 00:13:23 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1645C110C447 for ; Sun, 18 Nov 2018 00:13:23 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-io1-xd41.google.com (mail-io1-xd41.google.com [IPv6:2607:f8b0:4864:20::d41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F57F8834F; Sun, 18 Nov 2018 00:13:22 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-io1-xd41.google.com with SMTP id f12-v6so19823278iog.0; Sat, 17 Nov 2018 16:13:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:content-transfer-encoding:in-reply-to :user-agent; bh=Jym2OMjb0O+7D4t4j5vD2uyzkE/fGqxs4ZhFuqxOTbI=; b=EjhafZRpLZ1ULayOBeXe81XhlAuoEP3ELO0lIhPIaTQENrEBzbyC0oeHnDzwX/TfHe C4v1nUs38I37d0T4BXwSoezJCk3VhRve2pJCa8LoAbOOmIpUBXwgTWiRfGgjsr65Aekp OC/nv0qy+jOk8x5Q9Go6+KTbuVNzI/v4ldI35ZqZScBF92dbtc//2J5OZPqDMCxWxT0h f60ZZHmKs8/yh4HH0e5RL4f7fLX1LE2lrg3LxV3BPQWtOHU+wl/qqUQI2RbOxO8e49wl kZ7LqZ8ruPAfjAoYBtrkxoAHmUZNU6EXP2yY5EwoO971qkguN2sraNCi8sM/hwjC4BDy bj4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition :content-transfer-encoding:in-reply-to:user-agent; bh=Jym2OMjb0O+7D4t4j5vD2uyzkE/fGqxs4ZhFuqxOTbI=; b=aiuvXDl+81q0SKad1swJPTXXEruS0Ytt0FQ9m6oEmoBacdeIpnhvda1tXCYpkSgeL1 YGhcFavgPT/yANxg0FDqfbsblNoeD08B2mj91BssMO+MyIzZtFKRvN0k3pECO8bm6SRV NNOMDQegRBjxXhmr1NAv0mp2w7ryrG/5/9XzEeC2Ya2OfVDsOfnnjriLrAa64b5Lvs5k wCO1X+BxhZFPoSX+e1oE+jowvRRiCX0GyjOVPW5v394Ngh+OiD4VusfUrFpX/vZeMJc/ QSMfdg7sQd1bUBLqMQ9z/1xvzj1bCVfRyUJBYQo9jM5SuFyPCxl5EAkJY3fchhE7pgit j8Hg== X-Gm-Message-State: AGRZ1gKNWbSbblmp611yZGCFEgn+dFJHItffpW4dnuni1cl8YBomNJBI wqKQYzul8LBDHA6lJUWZcvVVIqV7bQM= X-Google-Smtp-Source: AJdET5fIya1ejygqD7TIJXGQ2JPDlO9bA64y+4AaOqgFhkSpVHEMolnus+ei1R3BrQYqVT+2R0Us7g== X-Received: by 2002:a6b:c082:: with SMTP id q124-v6mr13546618iof.25.1542500001378; Sat, 17 Nov 2018 16:13:21 -0800 (PST) Received: from raichu (toroon0560w-lp130-07-64-229-95-98.dsl.bell.ca. [64.229.95.98]) by smtp.gmail.com with ESMTPSA id t6sm6341558ioc.87.2018.11.17.16.13.20 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 17 Nov 2018 16:13:20 -0800 (PST) Sender: Mark Johnston Date: Sat, 17 Nov 2018 19:13:18 -0500 From: Mark Johnston To: Ian Lepore Cc: Wojciech Puchar , Rebecca Cran , freebsd-hackers Hackers Subject: Re: 13-CURRENT: several GB swap being used despite plenty of free RAM Message-ID: <20181118001318.GB2799@raichu> References: <1748688.u6MfGjpqfb@photon.int.bluestop.org> <1542499188.56571.59.camel@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <1542499188.56571.59.camel@freebsd.org> User-Agent: Mutt/1.10.1 (2018-07-13) X-Rspamd-Queue-Id: 4F57F8834F X-Spamd-Result: default: False [-1.29 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[freebsd.org]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; RCVD_IN_DNSWL_NONE(0.00)[1.4.d.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; NEURAL_HAM_SHORT(-0.94)[-0.939,0]; IP_SCORE(-0.64)[ip: (1.26), ipnet: 2607:f8b0::/32(-2.61), asn: 15169(-1.77), country: US(-0.10)]; FORGED_SENDER(0.30)[markj@freebsd.org,markjdb@gmail.com]; MID_RHS_NOT_FQDN(0.50)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[markj@freebsd.org,markjdb@gmail.com] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Nov 2018 00:13:23 -0000 On Sat, Nov 17, 2018 at 04:59:48PM -0700, Ian Lepore wrote: > On Sat, 2018-11-17 at 22:52 +0100, Wojciech Puchar wrote: > > freebsd will not swap with that lots of free ram. > > but it's 90GB free NOW, how about before? > > > > Your information is outdated. For at least a couple years now (since > approximately the 10.1 - 10.2 timeframe is my vague estimate), freebsd > will page out application memory that hasn't been referenced for some > time, even when the system has no shortage of free memory at all. No, FreeBSD will only ever swap when there is a free page shortage. The difference is that we now slowly age unreferenced pages into the inactive queue, which makes them candidates for pageout and subsequent eviction. With pageout_update_period=0, anonymous memory won't get paged out unless there's a shortage of inactive pages, or an application calls madvise(MADV_DONTNEED) on a range of memory (which moves any backing pages to the inactive queue). > The advice I was recently given to revert to the old behavior is: > >   sysctl vm.pageout_update_period=0 > > I've been using it on a couple systems here for a few days now, and so > far results are promising, I am no longer seeing gratuitous swapfile > usage on systems that have so much free physical ram that they should > never need to page anything out. I haven't yet pushed one of those > systems hard enough to check what happens when they do need to start > proactively paging out inactive memory due to shortages -- it could be > that turning off the new behavior has downsides for some workloads. From owner-freebsd-hackers@freebsd.org Sun Nov 18 00:42:00 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9FD8B110DAB0 for ; Sun, 18 Nov 2018 00:42:00 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic315-21.consmr.mail.ne1.yahoo.com (sonic315-21.consmr.mail.ne1.yahoo.com [66.163.190.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DDA0E89C42 for ; Sun, 18 Nov 2018 00:41:59 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: fHYIUTAVM1llHEvusd010khDBTsT2Hqd8ArVO2um0qkqY.K_M_o0w0Q4oECgP6b TgtQRn_IjcJZhpVnoBDQjVDpWYdcKisnGPwKgttq0JiAWSy6JpfrPhFWea1RyZTdXD8uOuqklHtD q_6NtCS0haITxRqmZ8sQ.7TL3vLpnp81MUuwjbyHwROK78M.yzje4XDTPdrMj6PctKo3SH6HvRIe 8sNrP1KpMAHoeDypzQLxIFyJ8Zp9KkHKLtHKkF4TarAGZWVqNsJcZbm5kSquBcPAPBah2t_bhDtr nwVEwylY5zouuKzWKV5VRqIxIEtXeMry9KuCdcBlKN6dl6Hp253zUIzj2CkJbBTYghq4dF2b00YS 9WJvavzCHZ.6LxQI9eKD557isuTHmjqJyg.lvifhp3roQgix8Zo0u5Xv4Bo_FQeIXCVL5g.kaPkc Yao24ypmSyqN.LW4L2t4O06jWJReXudo3R9jUyaRUKpU5FzUs4BWXoAELtBsSkRt8MxvEHGoSJKG psXbl352v860yqTg3saMQ.rVJhj9VUb7WqsbjdsYITSmheyvaJ73LIvEKhN1rp2nTL.pnsFMv5rN hvrTFrqfQki2XKy9jMHCi80aDAR_F7XU0_VVf7uk_9Y5AciLi5L6PfRl49e2uJTzNdCVIVGtiPgR CZpeTPTSSkiBih9Yy95IM1sALgKcRfHJP1gJUjqOSxtcxTc3I716uJE9FLluXGFlu_S7pWL4IukE Uo8eOWBuBrFzCq_JxvcQyrG8fMYSgCK0ImLWj35o2OfrYmN5ea.b7iZtKJLn5uiD.6Gel3tr9iyj t6h.PUlYlB2lDwKSr7I24YA44AsYv9PB.aQvPgdCIth0c4qpY0nextzB36Hc2OF5tUVnwQlyNfGL b60t0I2nZAUYuXKpC3bWEmQTVBI1sf9ECFy0VEjG4QdeHQBNgus8z_fzJFdDO.aGSo3.ApAI39_. hf_ZSmd_32wZlGCbl7zXkrEEr7pLb0kFSmpDNL_DQXz2AzazYtKcIQySgqOOgkfYUd3TLAr7j_NC zzptuKikDuFAHKY2sImv.1PtKrNOiJTkTjH4PNWIGPpKTkq9.07dTZI0- Received: from sonic.gate.mail.ne1.yahoo.com by sonic315.consmr.mail.ne1.yahoo.com with HTTP; Sun, 18 Nov 2018 00:41:58 +0000 Received: from c-76-115-7-162.hsd1.or.comcast.net (EHLO [192.168.1.25]) ([76.115.7.162]) by smtp406.mail.ne1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 8ec4251bca550b848f230a10789d5767; Sun, 18 Nov 2018 00:41:56 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 12.1 \(3445.101.1\)) Subject: Re: 13-CURRENT: several GB swap being used despite plenty of free RAM From: Mark Millard In-Reply-To: <20181118001318.GB2799@raichu> Date: Sat, 17 Nov 2018 16:41:55 -0800 Cc: Ian Lepore , Rebecca Cran , freebsd-hackers Hackers , Wojciech Puchar Content-Transfer-Encoding: 7bit Message-Id: References: <1748688.u6MfGjpqfb@photon.int.bluestop.org> <1542499188.56571.59.camel@freebsd.org> <20181118001318.GB2799@raichu> To: Mark Johnston X-Mailer: Apple Mail (2.3445.101.1) X-Rspamd-Queue-Id: DDA0E89C42 X-Spamd-Result: default: False [1.88 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; FREEMAIL_FROM(0.00)[yahoo.com]; NEURAL_SPAM_SHORT(0.87)[0.868,0]; RCPT_COUNT_FIVE(0.00)[5]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; MX_GOOD(-0.01)[cached: mta6.am0.yahoodns.net]; RCVD_IN_DNSWL_NONE(0.00)[147.190.163.66.list.dnswl.org : 127.0.5.0]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; IP_SCORE(1.53)[ip: (4.28), ipnet: 66.163.184.0/21(1.92), asn: 36646(1.53), country: US(-0.10)]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36646, ipnet:66.163.184.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Nov 2018 00:42:00 -0000 On 2018-Nov-17, at 16:13, Mark Johnston wrote: > > On Sat, Nov 17, 2018 at 04:59:48PM -0700, Ian Lepore wrote: >> On Sat, 2018-11-17 at 22:52 +0100, Wojciech Puchar wrote: >>> freebsd will not swap with that lots of free ram. >>> but it's 90GB free NOW, how about before? >>> >> >> Your information is outdated. For at least a couple years now (since >> approximately the 10.1 - 10.2 timeframe is my vague estimate), freebsd >> will page out application memory that hasn't been referenced for some >> time, even when the system has no shortage of free memory at all. > > No, FreeBSD will only ever swap when there is a free page shortage. The > difference is that we now slowly age unreferenced pages into the > inactive queue, which makes them candidates for pageout and subsequent > eviction. With pageout_update_period=0, anonymous memory won't get > paged out unless there's a shortage of inactive pages, or an application > calls madvise(MADV_DONTNEED) on a range of memory (which moves any > backing pages to the inactive queue). Swapping is built on top of paging as I understand. The system can page without swapping but can not swap without (effectively) paging to implement the swapping, if I understand right. If I understand right, swapped-out means that kernel stacks have been written out and have to be loaded back in RAM before the process/threads can even run. (I might not understand.) If I've got that right, are there distinctions here for paging that is not part of swapping vs. actual swapping (and its use of paging)? Saying that something does not swap does not necessarily imply that it does not page: it still could have paging activity that does not include moving the kernel stacks for the process to backing media? At times I have trouble interpreting when wording goes back and forth between swapping and paging, both for the intended meaning and for the technical implications. >> The advice I was recently given to revert to the old behavior is: >> >> sysctl vm.pageout_update_period=0 >> >> I've been using it on a couple systems here for a few days now, and so >> far results are promising, I am no longer seeing gratuitous swapfile >> usage on systems that have so much free physical ram that they should >> never need to page anything out. I haven't yet pushed one of those >> systems hard enough to check what happens when they do need to start >> proactively paging out inactive memory due to shortages -- it could be >> that turning off the new behavior has downsides for some workloads. === Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-hackers@freebsd.org Sun Nov 18 01:51:47 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5D271112135F for ; Sun, 18 Nov 2018 01:51:47 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-it1-x142.google.com (mail-it1-x142.google.com [IPv6:2607:f8b0:4864:20::142]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AAA4F8C1D4 for ; Sun, 18 Nov 2018 01:51:46 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-it1-x142.google.com with SMTP id k141-v6so3003103itk.3 for ; Sat, 17 Nov 2018 17:51:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=SPe8QtCvYnSiXpqfQG/M8ZW5Qi4n9NsYbau32vRTNIM=; b=HMMnRhbpzkLspbzZo2uacJO5Uze5EcbRDON4PYrvTJCxq3LLOmSZhD7zJD9b0miN5o eUL4UVAJ/N63Xidxe2F/xNdB0Ep3p8uezoH6MJzj9EuD+VErUakvsiKqibAUYLK9A7jx YH3CewZu5+8kIqk7w4sBpbRHeJbSk8VbRc8PS0wqFwSIbZ2AECiWmBHYpzozGjyEti2B FcxjgNkjYd1i+cmoA2Ruijk4OC3qI1RFsWzaCswvenEkhj9oNsCzTb0Jrk+Mt/tlNajr 9WEwZjuZKho+6jvewNmOF73K0BwzikP9bdJgUApnCEakEuddmF2R3jUpA+On2ZG4y+z3 awDA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=SPe8QtCvYnSiXpqfQG/M8ZW5Qi4n9NsYbau32vRTNIM=; b=mp8dAaylgxJuLLjMl5gE/i2u8ASl3EYc7yPGqIUyMEVomabf9YczB9pCFUJJY4ocF/ TBWghvfUL2sTsmK82OWy3cUOeR15ITw3TfHWdBlDi/OPhDYJ44UY1sEaZeGYFcioearf Aa9NelxOzfnt/EEX0D759nFg27PR7NeM5eYrqq3pM+wiZnBX1KHbtZzCx3SFi1rHoFka wfYTnLzvsz3GYvUK7NO8pLlP1xgUF8avVdkJpYxb2usnvUl4iIoYA/VjdY+komYu+7s4 EsQwhQBt+xGor98H7SDGsQjr3al73GVGPle1o8WjOnWHrdCnsho983R+VcWhdIkyBHbU Dexw== X-Gm-Message-State: AGRZ1gLtzUHt/oy46Qb4WCxkkeqORiNMQlNwCMnTkNgBk6wTLcAi6KnZ nwLaeJSYHIpNMJRQkAIaEEk= X-Google-Smtp-Source: AJdET5fuhZX8k9+UD0kEqQ4bfNZQNRS9JC5go39Mx3kZkzitGvmThgKfJfX1Vqm/ROet3gCPK8TWQw== X-Received: by 2002:a02:4007:: with SMTP id n7-v6mr14142120jaa.89.1542505905892; Sat, 17 Nov 2018 17:51:45 -0800 (PST) Received: from raichu (toroon0560w-lp130-07-64-229-95-98.dsl.bell.ca. [64.229.95.98]) by smtp.gmail.com with ESMTPSA id g186-v6sm13268413ita.22.2018.11.17.17.51.44 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 17 Nov 2018 17:51:45 -0800 (PST) Sender: Mark Johnston Date: Sat, 17 Nov 2018 20:51:43 -0500 From: Mark Johnston To: Mark Millard Cc: freebsd-hackers Hackers Subject: Re: 13-CURRENT: several GB swap being used despite plenty of free RAM Message-ID: <20181118015143.GD2799@raichu> References: <1748688.u6MfGjpqfb@photon.int.bluestop.org> <1542499188.56571.59.camel@freebsd.org> <20181118001318.GB2799@raichu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) X-Rspamd-Queue-Id: AAA4F8C1D4 X-Spamd-Result: default: False [-0.74 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; NEURAL_HAM_SHORT(-0.84)[-0.840,0]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[2.4.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_TO(0.00)[yahoo.com]; FORGED_SENDER(0.30)[markj@freebsd.org,markjdb@gmail.com]; MID_RHS_NOT_FQDN(0.50)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[markj@freebsd.org,markjdb@gmail.com]; IP_SCORE(-0.19)[ip: (3.49), ipnet: 2607:f8b0::/32(-2.61), asn: 15169(-1.76), country: US(-0.10)] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Nov 2018 01:51:47 -0000 On Sat, Nov 17, 2018 at 04:41:55PM -0800, Mark Millard wrote: > On 2018-Nov-17, at 16:13, Mark Johnston wrote: > > > > On Sat, Nov 17, 2018 at 04:59:48PM -0700, Ian Lepore wrote: > >> On Sat, 2018-11-17 at 22:52 +0100, Wojciech Puchar wrote: > >>> freebsd will not swap with that lots of free ram. > >>> but it's 90GB free NOW, how about before? > >>> > >> > >> Your information is outdated. For at least a couple years now (since > >> approximately the 10.1 - 10.2 timeframe is my vague estimate), freebsd > >> will page out application memory that hasn't been referenced for some > >> time, even when the system has no shortage of free memory at all. > > > > No, FreeBSD will only ever swap when there is a free page shortage. The > > difference is that we now slowly age unreferenced pages into the > > inactive queue, which makes them candidates for pageout and subsequent > > eviction. With pageout_update_period=0, anonymous memory won't get > > paged out unless there's a shortage of inactive pages, or an application > > calls madvise(MADV_DONTNEED) on a range of memory (which moves any > > backing pages to the inactive queue). > > Swapping is built on top of paging as I understand. The system > can page without swapping but can not swap without (effectively) > paging to implement the swapping, if I understand right. Right. > If I > understand right, swapped-out means that kernel stacks have > been written out and have to be loaded back in RAM before the > process/threads can even run. (I might not understand.) When free pages are scarce, one measure that the kernel may take to address the shortage is to swap out the kernel stacks of the threads in a process, thus allowing the pages backing the stacks to be reused for some other purpose, but preventing that process from running on a CPU until the stacks are swapped back in and "locked" (wired) into memory. Most of the pages consumed by an application like firefox are not used for kernel stacks. Most of them are used for the application's heap memory, and are thus private to that process. In general, pieces of such memory are subject to being paged out to the swap device, particularly when they are not frequently referenced (read or written to) by the application, in order to replenish the pool of free pages. Such memory is often said to be swapped out. As a side note, there are some system calls that modify this behaviour. mlock(2) effectively prevents the kernel from swapping out pages backing the specified virtual addresses; this guarantees that an access of the virtual memory range will never incur the cost of an expensive page-in. madvise(MADV_FREE) tells the kernel that the specified pages may be freed without first being written to the swap device. Thus, a subsequent read of an affected page may return the page's previous contents (if the page had not yet been reclaimed to make up for a shortage), or all zeroes (if the page had been freed without saving its contents to swap). > If I've got that right, are there distinctions here for > paging that is not part of swapping vs. actual swapping > (and its use of paging)? Saying that something does not > swap does not necessarily imply that it does not page: > it still could have paging activity that does not include > moving the kernel stacks for the process to backing media? Indeed, as I tried to describe above, kernel stack swapouts usually represent only a small portion of many applications' total swap usage, to the point where I at least usually don't think much about them. > At times I have trouble interpreting when wording goes back > and forth between swapping and paging, both for the intended > meaning and for the technical implications. When discussing paging activity related to the swap pager, I typically use those terms interchangeably. Sorry for the confusion. From owner-freebsd-hackers@freebsd.org Sun Nov 18 01:54:59 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2628211215AA for ; Sun, 18 Nov 2018 01:54:59 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Received: from smtp-out-no.shaw.ca (smtp-out-no.shaw.ca [64.59.134.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "Client", Issuer "CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 0FC618C2B4; Sun, 18 Nov 2018 01:54:57 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Received: from spqr.komquats.com ([70.67.125.17]) by shaw.ca with ESMTPA id OCIUg1o6qP7x2OCIWgs8Zn; Sat, 17 Nov 2018 18:54:55 -0700 X-Authority-Analysis: v=2.3 cv=X9GD11be c=1 sm=1 tr=0 a=VFtTW3WuZNDh6VkGe7fA3g==:117 a=VFtTW3WuZNDh6VkGe7fA3g==:17 a=kj9zAlcOel0A:10 a=JHtHm7312UAA:10 a=CjxXgO3LAAAA:8 a=6I5d2MoRAAAA:8 a=YVOhz5M6AAAA:8 a=YxBL1-UpAAAA:8 a=pRaooDUxaH_drk4u2DsA:9 a=Jamy6s-ylp-W1LSt:21 a=A-3pYsgeXlPRBYIO:21 a=CjuIK1q_8ugA:10 a=IjZwj45LgO3ly-622nXo:22 a=sbbTL3E6IKcx-RquDtO-:22 a=Ia-lj3WSrqcvXOmTRaiG:22 Received: from slippy.cwsent.com (slippy [10.1.1.91]) by spqr.komquats.com (Postfix) with ESMTPS id 66B8CA6B; Sat, 17 Nov 2018 17:54:50 -0800 (PST) Received: from slippy.cwsent.com (localhost [127.0.0.1]) by slippy.cwsent.com (8.15.2/8.15.2) with ESMTP id wAI1sn0X049217; Sat, 17 Nov 2018 17:54:49 -0800 (PST) (envelope-from Cy.Schubert@cschubert.com) Received: from slippy (cy@localhost) by slippy.cwsent.com (8.15.2/8.15.2/Submit) with ESMTP id wAI1smhg049214; Sat, 17 Nov 2018 17:54:48 -0800 (PST) (envelope-from Cy.Schubert@cschubert.com) Message-Id: <201811180154.wAI1smhg049214@slippy.cwsent.com> X-Authentication-Warning: slippy.cwsent.com: cy owned process doing -bs X-Mailer: exmh version 2.8.0 04/21/2012 with nmh-1.7.1 Reply-to: Cy Schubert From: Cy Schubert X-os: FreeBSD X-Sender: cy@cwsent.com X-URL: http://www.cschubert.com/ To: Mark Millard cc: Mark Johnston , Rebecca Cran , Wojciech Puchar , Ian Lepore , freebsd-hackers Hackers Subject: Re: 13-CURRENT: several GB swap being used despite plenty of free RAM In-Reply-To: Message from Mark Millard via freebsd-hackers of "Sat, 17 Nov 2018 16:41:55 -0800." Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Sat, 17 Nov 2018 17:54:48 -0800 X-CMAE-Envelope: MS4wfHl8bG2kLuIQGMshfd/g6QO4wY9xy3JzJaSIyYHgg/WqApQnQQ7WmQnFVRQkcUqvNObfjF6HV0gRWFxRZFwwz0+PBBYYEiFnCk7pnK2D/VIsBZVnj/1L lrd6ckbrarucXblmUrQGs4sVkMtAxCvz1SAHVs0xqe1473esa1AYzgp+yeWzYKBfqpsWt/0OC2VX/PhnzUxuIbiTo/r9Wy1de0SXx67WdEUFOSBc5OxfetJU T8+fAFUPtHnc3Lq981wCVL9xkS1+t5+TymXdB2cdp2i+NmcRwlS8+dywr6cTseBtfcv5UZQ5nZ7dRwyOG+9aGjkYQOP29+zBBlITdhm+gDM= X-Rspamd-Queue-Id: 0FC618C2B4 X-Spamd-Result: default: False [-1.51 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_FIVE(0.00)[5]; HAS_REPLYTO(0.00)[Cy.Schubert@cschubert.com]; FROM_HAS_DN(0.00)[]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; HAS_XAW(0.00)[]; RCPT_COUNT_FIVE(0.00)[6]; REPLYTO_EQ_FROM(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[cached: spqr.komquats.com]; NEURAL_HAM_SHORT(-0.93)[-0.926,0]; R_SPF_NA(0.00)[]; FREEMAIL_TO(0.00)[yahoo.com]; RECEIVED_SPAMHAUS_PBL(0.00)[17.125.67.70.zen.spamhaus.org : 127.0.0.11]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:6327, ipnet:64.59.128.0/20, country:CA]; RCVD_TLS_LAST(0.00)[]; IP_SCORE(-0.87)[ipnet: 64.59.128.0/20(-2.38), asn: 6327(-1.90), country: CA(-0.10)]; RCVD_IN_DNSWL_LOW(-0.10)[13.134.59.64.list.dnswl.org : 127.0.5.1] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Nov 2018 01:54:59 -0000 In message , Mark Millard via f reebsd-hackers writes: > On 2018-Nov-17, at 16:13, Mark Johnston wrote: > > > > On Sat, Nov 17, 2018 at 04:59:48PM -0700, Ian Lepore wrote: > >> On Sat, 2018-11-17 at 22:52 +0100, Wojciech Puchar wrote: > >>> freebsd will not swap with that lots of free ram. > >>> but it's 90GB free NOW, how about before? > >>> > >> > >> Your information is outdated. For at least a couple years now (since > >> approximately the 10.1 - 10.2 timeframe is my vague estimate), freebsd > >> will page out application memory that hasn't been referenced for some > >> time, even when the system has no shortage of free memory at all. > > > > No, FreeBSD will only ever swap when there is a free page shortage. The > > difference is that we now slowly age unreferenced pages into the > > inactive queue, which makes them candidates for pageout and subsequent > > eviction. With pageout_update_period=0, anonymous memory won't get > > paged out unless there's a shortage of inactive pages, or an application > > calls madvise(MADV_DONTNEED) on a range of memory (which moves any > > backing pages to the inactive queue). > > Swapping is built on top of paging as I understand. The system > can page without swapping but can not swap without (effectively) > paging to implement the swapping, if I understand right. If I > understand right, swapped-out means that kernel stacks have > been written out and have to be loaded back in RAM before the > process/threads can even run. (I might not understand.) > > If I've got that right, are there distinctions here for > paging that is not part of swapping vs. actual swapping > (and its use of paging)? Saying that something does not > swap does not necessarily imply that it does not page: > it still could have paging activity that does not include > moving the kernel stacks for the process to backing media? This is generally the old-school definition, IIRC third year comp sci, original BSD definition, which was also the way IBM defined it for MVS. (IBM also virtually swapped out address spaces, not written to DASD, as a means to deny CPU cycles through the scheduler not given the chance to consider the tasks (threads) in the address space). You can disable swapping by setting vm.swap_enabled=0. > > At times I have trouble interpreting when wording goes back > and forth between swapping and paging, both for the intended > meaning and for the technical implications. > > >> The advice I was recently given to revert to the old behavior is: > >> > >> sysctl vm.pageout_update_period=0 > >> > >> I've been using it on a couple systems here for a few days now, and so > >> far results are promising, I am no longer seeing gratuitous swapfile > >> usage on systems that have so much free physical ram that they should > >> never need to page anything out. I haven't yet pushed one of those > >> systems hard enough to check what happens when they do need to start > >> proactively paging out inactive memory due to shortages -- it could be > >> that turning off the new behavior has downsides for some workloads. > > > > > === > Mark Millard > marklmi at yahoo.com > ( dsl-only.net went > away in early 2018-Mar) > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > -- Cheers, Cy Schubert FreeBSD UNIX: Web: http://www.FreeBSD.org The need of the many outweighs the greed of the few. From owner-freebsd-hackers@freebsd.org Sun Nov 18 12:11:52 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5AD101137E60 for ; Sun, 18 Nov 2018 12:11:52 +0000 (UTC) (envelope-from sblachmann@gmail.com) Received: from mail-oi1-x22f.google.com (mail-oi1-x22f.google.com [IPv6:2607:f8b0:4864:20::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 415047A8B2; Sun, 18 Nov 2018 12:11:51 +0000 (UTC) (envelope-from sblachmann@gmail.com) Received: by mail-oi1-x22f.google.com with SMTP id u18so4945017oie.10; Sun, 18 Nov 2018 04:11:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=T/DIoXoeLEpuWrVeCQ+RorkrmTrE7/L8UxLH0t7vL2I=; b=mThDViwboKMeZVjk0S17wtSLHwqOirRs3f3FUd0wjjWjruANPGa8hghC0LZW1U/rcW uACNQg+4ZFsI+u2Y5WIaXrRRwCwfJwYGD4jLbtqupIYuN/tje+CO0v3uORfSmsHf/pnB WK57IRdwGCqkt2qRAaxyqNTHe0vd50x5sFnCE9Pp3O2wiT8vklxqRKD5JsFefMAlrDjL PjeFFCiMp6UtKr9mz/bRRebqeRWdjXZjjewDVszZLpX1sPxyF2sB6hVuNzKd8opdaVHn GTZVg/tWgUjoXGDobItjPPy8Z5d/rpNr4f4HQGpbZ48aHXO61UJJ7Q2wgqzyouVoVnEv ZlRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=T/DIoXoeLEpuWrVeCQ+RorkrmTrE7/L8UxLH0t7vL2I=; b=ckc8Jw1IoZBjSyVrSHaQVr3UUQTG7Oahw58effPEWB24N3tCV0mjI1U5wFIqbWm0aj e+cw/AH3oc07edQmi+Xqi3cnUf0gh1rfYtcsAss+MIJx60HK/EZH3AsAR3cLP17pd/oZ KgdaGYXYFlspsSh2kphx4VDGKqjPUnnPKkkwL3vI4/3L2b3oh/vIAo5hgens8308rqpC +sCCY3zYVtUjJsmFb1kDkJ8bSoBzUeJ7JZ+Ke7qpWKpGaVbsAlvNiiCCYhHqawMjZLFt h6zCXumY4udhwkjTcUFu0Zt4SqZMgngcB0SW52MgU4a/hN7DXrOFDz8IeeOtW5cl7+4k 5wDw== X-Gm-Message-State: AGRZ1gJLTTTqlrpfKTzgPy/M63dcDNaYX3HpU7qHdIxkjyr+Q4/RdR2Y 50uRjqaGFjtNHetVEdWuzropycIKWO4vHuTPCGQ= X-Google-Smtp-Source: AJdET5e+gHe165376dWnf6tQ5rk6y0/UJwbX9+lYC5dgcTUFsOWYlAa5KO0Xjl2F/4bN86CrJBpcKst7BjBot5VHv1Q= X-Received: by 2002:aca:3b89:: with SMTP id i131-v6mr4722570oia.242.1542543110302; Sun, 18 Nov 2018 04:11:50 -0800 (PST) MIME-Version: 1.0 Received: by 2002:ac9:3052:0:0:0:0:0 with HTTP; Sun, 18 Nov 2018 04:11:49 -0800 (PST) In-Reply-To: <201811180154.wAI1smhg049214@slippy.cwsent.com> References: <201811180154.wAI1smhg049214@slippy.cwsent.com> From: Stefan Blachmann Date: Sun, 18 Nov 2018 13:11:49 +0100 Message-ID: Subject: Re: 13-CURRENT: several GB swap being used despite plenty of free RAM To: Cy Schubert Cc: Mark Millard , Rebecca Cran , freebsd-hackers Hackers , Mark Johnston , Ian Lepore , Wojciech Puchar Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 415047A8B2 X-Spamd-Result: default: False [-2.75 / 15.00]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com]; FROM_HAS_DN(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(-0.88)[ipnet: 2607:f8b0::/32(-2.58), asn: 15169(-1.75), country: US(-0.10)]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_SEVEN(0.00)[7]; RCVD_IN_DNSWL_NONE(0.00)[f.2.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.86)[-0.858,0]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FREEMAIL_CC(0.00)[yahoo.com] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Nov 2018 12:11:52 -0000 The inconveniences that the new swapping strategy causes are a regular topic in the FreeBSD forums. Desktop users complain about lagginess, server users complain of long delays because server processes intended to be kept in memory for quick response times got swapped out and need to be swapped in again, resulting in outrageously poor server performance in spite of plenty of unused memory. Turning off swap completely, as Cy Schubert suggests, is strongly discouraged in the forums, as it can lead to kernel panicking because of being unable to swap out in critical kernel memory shortage situations, leading to the risk of very serious filesystem corruption. However, Cy Schubert is probably right when stating that the new swapping strategy resembles the 1960s-1980s industry's main swapping strategy. The bad thing is now, that nowadays memory is no longer scarce and people can dimension their memory such that under normal circumstances there will never be any need to swap. So I guess the unwillingness of the developer team to add an option like "NoPreemptiveSwapping", which disables swapping out as long as there is free physical memory available, is of psychological nature. Lacking such an option, there is still the possibility to use rctl to disable swapping for particular users, processes, jails etc to mitigate the problems caused by the new swapping strategy to some degree. On 11/18/18, Cy Schubert wrote: > In message , Mark > Millard via f > reebsd-hackers writes: >> On 2018-Nov-17, at 16:13, Mark Johnston wrote: >> > >> > On Sat, Nov 17, 2018 at 04:59:48PM -0700, Ian Lepore wrote: >> >> On Sat, 2018-11-17 at 22:52 +0100, Wojciech Puchar wrote: >> >>> freebsd will not swap with that lots of free ram. >> >>> but it's 90GB free NOW, how about before? >> >>> >> >> >> >> Your information is outdated. For at least a couple years now (since >> >> approximately the 10.1 - 10.2 timeframe is my vague estimate), freebsd >> >> will page out application memory that hasn't been referenced for some >> >> time, even when the system has no shortage of free memory at all. >> > >> > No, FreeBSD will only ever swap when there is a free page shortage. >> > The >> > difference is that we now slowly age unreferenced pages into the >> > inactive queue, which makes them candidates for pageout and subsequent >> > eviction. With pageout_update_period=0, anonymous memory won't get >> > paged out unless there's a shortage of inactive pages, or an >> > application >> > calls madvise(MADV_DONTNEED) on a range of memory (which moves any >> > backing pages to the inactive queue). >> >> Swapping is built on top of paging as I understand. The system >> can page without swapping but can not swap without (effectively) >> paging to implement the swapping, if I understand right. If I >> understand right, swapped-out means that kernel stacks have >> been written out and have to be loaded back in RAM before the >> process/threads can even run. (I might not understand.) >> >> If I've got that right, are there distinctions here for >> paging that is not part of swapping vs. actual swapping >> (and its use of paging)? Saying that something does not >> swap does not necessarily imply that it does not page: >> it still could have paging activity that does not include >> moving the kernel stacks for the process to backing media? > > This is generally the old-school definition, IIRC third year comp sci, > original BSD definition, which was also the way IBM defined it for MVS. > (IBM also virtually swapped out address spaces, not written to DASD, as > a means to deny CPU cycles through the scheduler not given the chance > to consider the tasks (threads) in the address space). > > You can disable swapping by setting vm.swap_enabled=0. > >> >> At times I have trouble interpreting when wording goes back >> and forth between swapping and paging, both for the intended >> meaning and for the technical implications. >> >> >> The advice I was recently given to revert to the old behavior is: >> >> >> >> sysctl vm.pageout_update_period=0 >> >> >> >> I've been using it on a couple systems here for a few days now, and so >> >> far results are promising, I am no longer seeing gratuitous swapfile >> >> usage on systems that have so much free physical ram that they should >> >> never need to page anything out. I haven't yet pushed one of those >> >> systems hard enough to check what happens when they do need to start >> >> proactively paging out inactive memory due to shortages -- it could be >> >> that turning off the new behavior has downsides for some workloads. >> >> >> >> >> === >> Mark Millard >> marklmi at yahoo.com >> ( dsl-only.net went >> away in early 2018-Mar) >> >> _______________________________________________ >> freebsd-hackers@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers >> To unsubscribe, send any mail to >> "freebsd-hackers-unsubscribe@freebsd.org" >> > > -- > Cheers, > Cy Schubert > FreeBSD UNIX: Web: http://www.FreeBSD.org > > The need of the many outweighs the greed of the few. > > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > From owner-freebsd-hackers@freebsd.org Sun Nov 18 13:23:47 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4E82E1101F22 for ; Sun, 18 Nov 2018 13:23:47 +0000 (UTC) (envelope-from george+freebsd@m5p.com) Received: from mailhost.m5p.com (mailhost.m5p.com [74.104.188.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "m5p.com", Issuer "Let's Encrypt Authority X3" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 785B97DA30 for ; Sun, 18 Nov 2018 13:23:41 +0000 (UTC) (envelope-from george+freebsd@m5p.com) Received: from [10.100.0.31] (haymarket.m5p.com [10.100.0.31]) (authenticated bits=0) by mailhost.m5p.com (8.15.2/8.15.2) with ESMTPSA id wAID07Le001653 (version=TLSv1.2 cipher=AES128-SHA bits=128 verify=NO) for ; Sun, 18 Nov 2018 08:00:21 -0500 (EST) (envelope-from george+freebsd@m5p.com) Subject: Re: 13-CURRENT: several GB swap being used despite plenty of free RAM To: freebsd-hackers@freebsd.org References: <201811180154.wAI1smhg049214@slippy.cwsent.com> From: George Mitchell Openpgp: preference=signencrypt Autocrypt: addr=george+freebsd@m5p.com; prefer-encrypt=mutual; keydata= xsFNBFgnLnwBEADAJDiBKQX77LFRz9wZW8mz3KvaQol2nIremcws0F1mz/zgFlk6uhQVtwnL wb4XL5LdFwcNE1+QZzPLcbYWoWQlz0lBw1bMuKAgr0S6V2e0+I0DqhKeslVFctcTwtvT6pnK VLZXO/7ZGAaLzG4K5vSPzgoevU+YI/pxNsVCH2UO/c3jQW63uEt25mIZbCF1Pu4jgp4RhIgF ujn877r/j6OwBwjzRUu3E6ADp+U825d+5YCuQMEH0wIPnn9GTpXvfdKdbwOIl2akqXqs4cnk iATWfK3r6D4mvDEj1OPHlTvJYcfic7aOIiAwmx1C1v78GjXOdOOA0SGffNix3C2/8oZUO1+V Aet4MKpUKkduWSvULhIkHNZ5Nu8SIJOqge8pmtHxuNXAMfMrAjMdjPwwBFLsYg3Xa2E2oJwg ehTauwd/EDJFcVCyDCyCAYOi/BH/+XQyxzgDlY9N9qj9tHqhVPI6XK7t8UVffGiZUq4rHp5J RdOToqiTNC6eCJBczhMIW+DuFvWU9e6W708T1dz0Accn6Lrgk4eRIn3GFPBG+TxnpjAqHsbW 607dcnD3YKAqY4e+khczL4EObhe7dC1v2fmZiAC6Ds3WHR11IfqoUgCkIwJ590Ej+ElygJFF XxI82wtEz9hkeLLvItpyEJNVjppViRW+Dgl/U7ypHB3qDgYjgwARAQABzSBHZW9yZ2UgTWl0 Y2hlbGwgPGdlb3JnZUBtNXAuY29tPsLBlwQTAQgAQQIbIwUJCWYBgAULCQgHAgYVCAkKCwIE FgIDAQIeAQIXgBYhBDXTOGR5LbCVuZCmV8EREt5vqeH5BQJYWXFRAhkBAAoJEMEREt5vqeH5 SRsQAIb/crcxXyAyeokAuTjN+YXkEFdVv/JrOgNXKdCukXt/UGd3nZTcAzpllytDIvIlPTNI 2/nZ5sm6ymeyVwmvkrM/r3sRUib5ftakJpbv0wn2j+eCGAca8IA2frBUg9bEXKcHRJRQCztG cpousHzOziTCRQ/1NfPcIFBNbQMPUVoQ96cJPvM/XfFGOISKKsSI78skHm6Oazh1hLCQTKvy hNNgVpNP/PHCHMbla1+SNgyn35CUelTh153lnkOhw1XyX6IxY4o5Bhcf3YrxAVcoeHq3FEH0 T3ygAdI14VrZGXXitBAMI78QLB0FiWMoPQ3Oddnoh6tlT4djnsc9IW0Tzk6ozvQL7sKdYgO8 ZlIpkBqQQfpSHzwPu9EkOFggPWB9WtP3IajQ0lt1LovqMug4C8APRC2/1cvi+GUIRwjVsop0 ukhlLTTJd3/S4Muh3s87M4Rdek1xpOiMKjYOVaxmhQQ91oz971zJuJJWpX7uUQiXx9oEwCDW TvI5yEuqYLsMUwx3d2iFTr+HbtlBJmF+Vguyrn/a3vFK8P/TH9fMvNeTdln9SuOOa1SAMMcy czOpBYg9RpzNLshUJVrhKzugT59Rl2wsNQsQCUkzgF3f8cZHJyl+8x6t0nuM9LTkMv13YIXS Mde3UOD2EaBhmeIqvC/adQHxpNudvxM1viFJDnTnzsFNBFgnLnwBEAC7kzsZqjBRPonnr/63 C98FSa3LikvqQWygmPSCC9DsFX/fB0CSXIHTrHQ4a7lXdfZyTZcGdxXN+MC8O8thjvVq6WYm CpyOJ/bq4SxOa9cnQSJ5SP9VCmVoDN/3T4ybXFzLAt756kfQ5jsVuP0m6iQ4z918zhZXk+Mo qdwGjYTxsBD02a7m1aeYafyaI2mZ+vdEy0cDhV4PDXI6ThLNAavTPji6ZrBdH5a4LMg30u8v kkNe0eCKvU+0cWb2VQIeddMhhiGSBE2Dv+A4eNe1VZvoGpLDlYdnoHraVFL2GHNFGymj/uRf 1hja5kW9Rekisqby5SpGABwrFFEs7ABpfYG0IRBKbjjG1Cqdfe/R6ETJFvvNOOpCKPAWeqYf Isxv4OHWTmhKihhIanYWCAe1MDLRRbj7UrOTZeia2WJ4v58xbU5rVQoHI/Tzaq86rXzRWITc 5w+kresMad0zpvQ900BdHc8ATNY2aW/Fr5it5OqMvIW6Y4gc8Z95MkQPVc8vj3WzfxuWtZNK 6Wbv4r6Qbiwg1RpY1JoEIoF4OsZJOVMQaB6ezD7xvaa2eY6nRGtq9SoJxo2qvlSbLlKq8NdH VYHhtTbpQ1NfrEJfv5sLX5W5IpoYww48bFYH67+7r1f/W3voBptSgE7qnYAm6Em9bEAKOQEX 8BXwoa54fn03z6TyhwARAQABwsFkBBgBCAAPBQJYJy58AhsMBQkJZgGAAAoJEMEREt5vqeH5 6PcP+JvrMM7ZM8UlnbrY4Er9psPj3ayllRhQFA9h6GNUKYuSzSxOrPaT96s8KUGMCr4jrn1S WFmeeNLLOgSJmQRicMh6LmnKq6WY6UaOfn7Y9O62NUjXfEI3Bw4ID36YCdQ+CJd14r0YOf4M 5F50bvHV3lbzD9TXZPxecHKC2ZUMBbT37tsckWCLL3lzKMsqQLwBUmgBl1NIUc7gyXxiNyxb 6SPVF1NguDDM438mcg9jSRAyjgAk6POUEM8YIXkw0Gg6HF1tNMJJ1xTMBCLYl6fHTtsxJpf9 yo+Hnw346hqYzXn4ytHJ49Ngcre8uhqM1l8iMpa17tEjfalkc1FWR9/qvoowOKtxpvblsy3a YzeEFgIomhLzISz6IafQ3S7Mt5AFlqwN/qQHx0k2V66GzDG0ngZBPROP1sXSpdJzO0zbJQFn MZE3f+y8vXMcE/MBXR7kAdYYApiEMQzVxy9TdQDU3lGLptcPZ1IOntTNFFrvp5NwsKi+6C9i mXtd5kJ1PwhcJYW3/ov/490l60C5SFUL/RZ/NOW8SHFaPcqlGcqIlexFKbzrMQwmYXo95jWB eZ0Qn+raxCUFZNGiwtusyQGBMcpHVJUanOCNd1z4ZbfmhUjDJKC/7YWDunvaDRSukGiRCl6J s8caqXHiVZjx+s76iWzm6AHRP5jg9D6EtTOrGiE= Message-ID: <0fd769af-9a7d-95cf-7cba-5e41e3bc3c30@m5p.com> Date: Sun, 18 Nov 2018 08:02:53 -0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.3.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="XVc9df5vdVSPHE3rcRFs2LyhdBuvqkizr" X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.6.2 (mailhost.m5p.com [10.100.0.247]); Sun, 18 Nov 2018 08:00:23 -0500 (EST) X-Rspamd-Queue-Id: 785B97DA30 X-Spamd-Result: default: False [-3.03 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+a]; TO_MATCH_ENVRCPT_ALL(0.00)[]; HAS_ATTACHMENT(0.00)[]; MIME_GOOD(-0.20)[multipart/signed,multipart/mixed,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; TO_DN_NONE(0.00)[]; MX_GOOD(-0.01)[mailhost.m5p.com]; NEURAL_HAM_SHORT(-0.45)[-0.450,0]; DMARC_NA(0.00)[m5p.com]; SIGNED_PGP(-2.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; IP_SCORE(-0.17)[asn: 701(-0.77), country: US(-0.09)]; ASN(0.00)[asn:701, ipnet:74.104.0.0/16, country:US]; TAGGED_FROM(0.00)[freebsd]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Nov 2018 13:23:47 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --XVc9df5vdVSPHE3rcRFs2LyhdBuvqkizr Content-Type: multipart/mixed; boundary="rJOsc9QsAnfUOkqf21F8wVTeJwhF6WhiX"; protected-headers="v1" From: George Mitchell To: freebsd-hackers@freebsd.org Message-ID: <0fd769af-9a7d-95cf-7cba-5e41e3bc3c30@m5p.com> Subject: Re: 13-CURRENT: several GB swap being used despite plenty of free RAM References: <201811180154.wAI1smhg049214@slippy.cwsent.com> In-Reply-To: --rJOsc9QsAnfUOkqf21F8wVTeJwhF6WhiX Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 11/18/18 7:11 AM, Stefan Blachmann wrote: > The inconveniences that the new swapping strategy causes are a regular > topic in the FreeBSD forums. >=20 > Desktop users complain about lagginess, server users complain of long > delays because server processes intended to be kept in memory for > quick response times got swapped out and need to be swapped in again, > resulting in outrageously poor server performance in spite of plenty > of unused memory. [...] Before necessarily blaming swapping, have you tried substituting SCHED_4BSD for SCHED_ULE in the kernel? SCHED_ULE is not the world champion scheduler for interactive tasks. (I'm ducking back into my rabbit hole now and no one needs to throw any more brickbats at me. Sorry for the digression.) -- George --rJOsc9QsAnfUOkqf21F8wVTeJwhF6WhiX-- --XVc9df5vdVSPHE3rcRFs2LyhdBuvqkizr Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEENdM4ZHktsJW5kKZXwRES3m+p4fkFAlvxYwQACgkQwRES3m+p 4fmjRBAAihC7l1I+In9fPAphzcafqWDN6pm2ota5pSP96qs+zZqXBFZJSD6SOOYS 436n0ie2aYWEBhO199gqHTm+w//eQgyM+AuAH/RSonC/PlUf/TNLDYhxJJXdroWd UIapoR/PkYuMScGrqVv5Kqp2TyQa2eUJIFz9PDFlo28yiIczyU0D4uq+vUP0QgHu kuAJGIFWVMDHH7Lsh0IuuyPNGSYpmopaduQJY7TUVxmhfEcdrLf3sgnbjpnP0hpa bXk/d0SwlKG0KqDvyKhNdE17UT/eEUuUTvs/1A1ib2s98vyUV9hxwfdau0Az2E2+ Zx28yu3UdM7R5DhYF5dHmIu8Spl7LYiw7/XN0mm+6zHiScBrZoHa84w0s5MVg47d +dVpag0eGoXO9aeroR14ioQP1jCY/HyB9CBMAfxKDvUdHV7JvjXR/8MU6IjcM9U3 tqatLylitR+0foMApgV3hXhMgQV5to5GJI5K/mqzrZUKlmyGXTH3s7NFxRO3/K6q /+p5o4/aV9cMee+WWDXjnyLbGeFj2uIKS1/M6w304vDWwk+G/uPurYgBkmkngF+h HEgVK/J/Tk7cC++Ert4TX+mvSwOwIa4Mz64djWNVzUMQrpz0rED/VM0kxn0r5TkT QP97ZMr2C+rgvaWgIXxM7aVRBaLAvzCWMGKt4JuEPuvIvxQQv0Q= =/ywx -----END PGP SIGNATURE----- --XVc9df5vdVSPHE3rcRFs2LyhdBuvqkizr-- From owner-freebsd-hackers@freebsd.org Sun Nov 18 14:31:51 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 06BB71103B16 for ; Sun, 18 Nov 2018 14:31:51 +0000 (UTC) (envelope-from sblachmann@gmail.com) Received: from mail-ot1-x329.google.com (mail-ot1-x329.google.com [IPv6:2607:f8b0:4864:20::329]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 440F47FC0F for ; Sun, 18 Nov 2018 14:31:50 +0000 (UTC) (envelope-from sblachmann@gmail.com) Received: by mail-ot1-x329.google.com with SMTP id t5so25488806otk.1 for ; Sun, 18 Nov 2018 06:31:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=5B+tA+NfjGl7L29YLmiIhx0fB6cYcZhluJqA0lYx1Gg=; b=H/7ITaTVYeHBT9OaCOwnJUVy+Rt053DOazQ9vFdC5bfm12HHI5VoNu8kBXHUwXlk34 mdXDkZUMUWOf1NPzP6ZLr7SsBKxThme0e9++zSdcR2S7DU/9tEmyJZ2j1xXQ37gNLjnO TVbkpQTZN1YeQo6lC8VbOZ/4F6Piwa5cWD5JVtUbN0SshS/Gb3SJqdlggcYZXF6QBccU 8/G7DERm+7aa/BqiO/unKM9/mCZ30lJyOu2v0Xbc4ZSWvlyX11VoOBH/K1HNiuqUss5v hSHy/NQztVRuoI53OuwywVhRvlaiUSiSweI/xQU8f9FdNwFyzpi7oZmE2MsIqfQ9HQoM qiAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=5B+tA+NfjGl7L29YLmiIhx0fB6cYcZhluJqA0lYx1Gg=; b=KJtbKeYNa0SS9qwidBMzNL4dsvZ1oqrIqW6hVcT0jKHtuP2wKQNTJNN42cef2RGgXt XIn12OA3M/kF8wRDJCt7sSxBMQAVOE6Z0oWdoSyCXfq4G+lPHOW5hcbqOD3P3/fiOetx 08684YD2dKIspTsks8OQrPXfInTdJKYeHzvGGtEjiCjNfjT2/iUQWiwyNKLIcgq4QtSc 43R0p1iTfFwD20To6ix+rNF9oCL52GzPuq7EdAuibgy5zECkm/kU1qz67Amedemr09yJ l7lQs2pkluXTroSMKeom+x0QNf3IU48RaRIUyk5rBzUmqSlC0Mp7crigsjVNtgWc9Vq8 Yqdw== X-Gm-Message-State: AGRZ1gLk4jzf8nNYeEFSoaqzAl7580PLF0CctGW/SA6muSLSOa7CKZaR hMeVyrYoQQIazSJ+Ldy4KMEny+5KUBYYb3r5pnQ= X-Google-Smtp-Source: AJdET5cMoW+O1sPxN0zgT3TBzHsm2mT7i4JKQnrEXfwlOONbU/EhjOCQmw6Kx5gcyOeaSXBwK00GcLxYdAnQwTtP444= X-Received: by 2002:a9d:2e0a:: with SMTP id q10mr11610153otb.176.1542551509546; Sun, 18 Nov 2018 06:31:49 -0800 (PST) MIME-Version: 1.0 Received: by 2002:ac9:3052:0:0:0:0:0 with HTTP; Sun, 18 Nov 2018 06:31:48 -0800 (PST) In-Reply-To: <0fd769af-9a7d-95cf-7cba-5e41e3bc3c30@m5p.com> References: <201811180154.wAI1smhg049214@slippy.cwsent.com> <0fd769af-9a7d-95cf-7cba-5e41e3bc3c30@m5p.com> From: Stefan Blachmann Date: Sun, 18 Nov 2018 15:31:48 +0100 Message-ID: Subject: Re: 13-CURRENT: several GB swap being used despite plenty of free RAM To: George Mitchell Cc: freebsd-hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 440F47FC0F X-Spamd-Result: default: False [-2.80 / 15.00]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; TAGGED_RCPT(0.00)[freebsd]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; MIME_GOOD(-0.10)[text/plain]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[9.2.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.90)[-0.902,0]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; IP_SCORE(-0.88)[ipnet: 2607:f8b0::/32(-2.58), asn: 15169(-1.74), country: US(-0.09)] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Nov 2018 14:31:51 -0000 Probably a good suggestion to try the alternative scheduler, thank you. However, as people a) always see disk activity LED flashing up when PC is "lagging", which can be clearly identified as swap I/O using the various utilities, and b) these "lag" issues do not appear with swap deactivated, I think I have reason to believe the performance issues are not scheduler-related. On 11/18/18, George Mitchell wrote: > On 11/18/18 7:11 AM, Stefan Blachmann wrote: >> The inconveniences that the new swapping strategy causes are a regular >> topic in the FreeBSD forums. >> >> Desktop users complain about lagginess, server users complain of long >> delays because server processes intended to be kept in memory for >> quick response times got swapped out and need to be swapped in again, >> resulting in outrageously poor server performance in spite of plenty >> of unused memory. [...] > Before necessarily blaming swapping, have you tried substituting > SCHED_4BSD for SCHED_ULE in the kernel? SCHED_ULE is not the world > champion scheduler for interactive tasks. > > (I'm ducking back into my rabbit hole now and no one needs to throw > any more brickbats at me. Sorry for the digression.) -- George > > From owner-freebsd-hackers@freebsd.org Sun Nov 18 16:54:10 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AC6D9110726A for ; Sun, 18 Nov 2018 16:54:10 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: from puchar.net (puchar.net [194.1.144.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "puchar.net", Issuer "puchar.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 0EED2850A2; Sun, 18 Nov 2018 16:54:09 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.15.2/8.15.2) with ESMTPS id wAIGrvpm024621 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sun, 18 Nov 2018 17:53:57 +0100 (CET) (envelope-from puchar-wojtek@puchar.net) Received: from localhost (puchar-wojtek@localhost) by puchar.net (8.15.2/8.15.2/Submit) with ESMTP id wAIGrq6J024614; Sun, 18 Nov 2018 17:53:52 +0100 (CET) (envelope-from puchar-wojtek@puchar.net) Date: Sun, 18 Nov 2018 17:53:52 +0100 (CET) From: Wojciech Puchar To: Stefan Blachmann cc: Cy Schubert , Ian Lepore , Wojciech Puchar , freebsd-hackers Hackers , Rebecca Cran , Mark Johnston Subject: Re: 13-CURRENT: several GB swap being used despite plenty of free RAM In-Reply-To: Message-ID: References: <201811180154.wAI1smhg049214@slippy.cwsent.com> User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Rspamd-Queue-Id: 0EED2850A2 X-Spamd-Result: default: False [0.08 / 15.00]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_SPAM_SHORT(0.38)[0.381,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[puchar.net]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[cached: puchar.net]; RCPT_COUNT_SEVEN(0.00)[7]; RCVD_IN_DNSWL_NONE(0.00)[90.144.1.194.list.dnswl.org : 127.0.10.0]; IP_SCORE(0.00)[country: PL(0.02)]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:43476, ipnet:194.1.144.0/24, country:PL]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Nov 2018 16:54:10 -0000 why someone changes WELL WORKING swapping algorithm? On Sun, 18 Nov 2018, Stefan Blachmann wrote: > So I guess the unwillingness of the developer team to add an option > like "NoPreemptiveSwapping", which disables swapping out as long as > there is free physical memory available, is of psychological nature. in FReeBSD 12 there is vm.swap_idle_threshold2: 10 vm.swap_idle_threshold1: 2 vm.swap_idle_enabled: 0 which works as expected and can be turned on or off as required. what's wrong with it From owner-freebsd-hackers@freebsd.org Sun Nov 18 17:51:32 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D5FF11108863 for ; Sun, 18 Nov 2018 17:51:31 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic307-3.consmr.mail.bf2.yahoo.com (sonic307-3.consmr.mail.bf2.yahoo.com [74.6.134.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5DB098694E for ; Sun, 18 Nov 2018 17:51:31 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: bqApfoYVM1lNU9DQXysJjeiYPCvRvPlP5NhocyIIMoBpCXCp3EgtyQKAg15Eq3e duX9jZIw60tWMQdR9CAvS6sk43bD9pPX5I05KcjV6O21wRK6RxGjYg_4R8blCxXbKcunoF5L9ZYR HR9mpe0pyYGmT6RwQf4ojsthaFjUyMEwwDDArtAc.zvlibsCBMZC7OaWr_v.ZBlE9jYf9k30b_hj EHa0HiWdSN_qQTB1_uNqGh5GS6TeA3JfAy0rmy4fzGVWjFPu0W.I5kxqvbPLakIbYDfnQEXF6MQw emOMZI6y99_T8TvWeJK0ICoeJ3VOXV0UWwGoyhyo1gAKbm4VaZjeAG0cAq9.qH91X5E9rujZ7H59 UmtINxSK_A8eTG6Fr3fk4zGcKCwozYtAPtLoDdlAlAVOgipeKzuED1x.9ZvrXcWUw4.ZGFJ3B_Q_ 4RPLSgoZVi7zBRrn._fcB_HtcpdYqos3DpVBUpHXg2OomzztmRF61b.Kpa1lk77Eruy7GtOdNjdu p8y2.yMMexOMLp0Avu5Q7zSWC5qBEc1W_e60r44JUCu5C7k7wnkpiDkQ9hEHS8d.ibt.2cuxhXZl 10GFNNWDDizJMuJS6zT1Y.pNVWoud_HBku1bS3N5A20_NSozKzKGkK_7ZJqSv_ZpmnAQKDJcvmLU Hodlz8LFkPz2hR9jzA.3xu6Uzo1Ke9XisFPDdw_vS4a7e.TEk632YIWC83YPsWFDYwLqDy60Hv5l CxMglIeIzYCOfAi6Wn8XksCQ7w4LZJ_DwhFWbw9lQuRT3X60PHi9xtrSqf9r2thvLsgUUS0YuFx8 u6ThPefzMOi.MqF6M7fjJLNaOyPY6znjQcBkPEyTZiCZg0ujrqc2mKmyQ1NCdftfulJS4b0XAK_. D2Tu_qRhKdzBKmyIoAdB.Uu_x226_rbeM0dEZ8iPxPIXDCBfpw1vH1.fzlSE7zzczUGQRKVZ3rzb w2UdE2PkKcdiJVprHoEvxPnrxSSqIps6ifDcYZ_MpZ0NObuXpriG_NV_CZnA_q371RI4gBUs5x9Y 9NGmgEyEVqgVZ4oAGJQL2iuhw1loLWWYc3zj2zp2BMt9CE_DmPwEhYN7jpNqX Received: from sonic.gate.mail.ne1.yahoo.com by sonic307.consmr.mail.bf2.yahoo.com with HTTP; Sun, 18 Nov 2018 17:51:30 +0000 Received: from c-76-115-7-162.hsd1.or.comcast.net (EHLO [192.168.1.25]) ([76.115.7.162]) by smtp429.mail.bf1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 11f13a1d842a98d891153e8e8bd0a5e0; Sun, 18 Nov 2018 17:51:28 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 12.1 \(3445.101.1\)) Subject: Re: 13-CURRENT: several GB swap being used despite plenty of free RAM From: Mark Millard In-Reply-To: Date: Sun, 18 Nov 2018 09:51:25 -0800 Cc: Cy Schubert , Rebecca Cran , freebsd-hackers Hackers , Mark Johnston , Ian Lepore , Wojciech Puchar Content-Transfer-Encoding: 7bit Message-Id: References: <201811180154.wAI1smhg049214@slippy.cwsent.com> To: Stefan Blachmann X-Mailer: Apple Mail (2.3445.101.1) X-Rspamd-Queue-Id: 5DB098694E X-Spamd-Result: default: False [2.37 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; FREEMAIL_FROM(0.00)[yahoo.com]; NEURAL_SPAM_MEDIUM(0.65)[0.645,0]; NEURAL_SPAM_SHORT(0.57)[0.569,0]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; MX_GOOD(-0.01)[cached: mta6.am0.yahoodns.net]; RCVD_IN_DNSWL_NONE(0.00)[42.134.6.74.list.dnswl.org : 127.0.5.0]; RCPT_COUNT_SEVEN(0.00)[7]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; IP_SCORE(1.66)[ip: (4.59), ipnet: 74.6.128.0/21(2.12), asn: 26101(1.70), country: US(-0.09)]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:26101, ipnet:74.6.128.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Nov 2018 17:51:32 -0000 On 2018-Nov-18, at 04:11, Stefan Blachmann wrote: > The inconveniences that the new swapping strategy causes are a regular > topic in the FreeBSD forums. > > Desktop users complain about lagginess, server users complain of long > delays because server processes intended to be kept in memory for > quick response times got swapped out and need to be swapped in again, > resulting in outrageously poor server performance in spite of plenty > of unused memory. I've no clue of the variations in workloads involved for various folks, but I only see swapping/paging once free memory is low in my UFS based contexts. (I've not used ZFS in a long time.) I do not see free RAM decreasing without known, expected use of the RAM. Mostly I've had to learn how late it may be before pages for cached information explicitly moves back to free, no matter if that involves paging/swapping out or not. As I understand what Mark Johnston reported, he said that there is no preemptive paging/swapping: "FreeBSD will only ever swap when there is a free page shortage" Moving between Active and Inactive does not of itself involve paging/swapping. So that part of his description is a different issue. So the issue may trace back to why there ends up being a prior free RAM shortage and/or when pages are moved back to RAM after such a shortage. I believe my report above is consistent with what Mark Johnston reported about the modern algorithm(s) involved. > Turning off swap completely, as Cy Schubert suggests, is strongly > discouraged in the forums, as it can lead to kernel panicking because > of being unable to swap out in critical kernel memory shortage > situations, leading to the risk of very serious filesystem > corruption. > > However, Cy Schubert is probably right when stating that the new > swapping strategy resembles the 1960s-1980s industry's main swapping > strategy. Important parts of "new" way seems to have been in place since after 4.4BSD (so BSD 5.0 and later) . . . The book "The Design and Implementation of the FreeBSD Operating System" (2nd edition) states (page labeled 296): QUOTE: The FreeBSD swap-out daemon will not select a runnable processes to swap out. So, if the set of runnable processes do not fit in memory, the machine will effectively deadlock. Current machines have enough memory that this condition usually does not arise. If it does, FreeBSD avoids deadlock by killing the largest process. If the condition begins to arise in normal operation, the 4.4BSD algorithm will need to be restored. END QUOTE. (I've not found vm.pageout_oom_seq in the book. It is for controlling how much effort is put into freeing RAM before kills start, and so, indirectly, the elapsed time that happens before those kills start.) I'm not sure what specific changes that are newer might be under discussion. > The bad thing is now, that nowadays memory is no longer scarce and > people can dimension their memory such that under normal circumstances > there will never be any need to swap. My typical use is apparently not normal. I'd agree with that. > So I guess the unwillingness of the developer team to add an option > like "NoPreemptiveSwapping", which disables swapping out as long as > there is free physical memory available, is of psychological nature. Unsure how to interpret this given: "FreeBSD will only ever swap when there is a free page shortage" > Lacking such an option, there is still the possibility to use rctl to > disable swapping for particular users, processes, jails etc to > mitigate the problems caused by the new swapping strategy to some > degree. Whatever is going on for this, it seems a more technical identification of what it is would be needed in order to get to the right context. (It may be non-trivial to identify.) === Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-hackers@freebsd.org Sun Nov 18 19:30:21 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 79A44110C8A2 for ; Sun, 18 Nov 2018 19:30:21 +0000 (UTC) (envelope-from rwmaillists@googlemail.com) Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4EDCD6C35F for ; Sun, 18 Nov 2018 19:30:20 +0000 (UTC) (envelope-from rwmaillists@googlemail.com) Received: by mail-wr1-x434.google.com with SMTP id l9so16892756wrt.13 for ; Sun, 18 Nov 2018 11:30:20 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=M/nWWUXlQHa/Vl9v9Rf0QEZ1wFwjaYuNL/pSllXm5oI=; b=P3/FoHBIuyx61onKdTiEsK8jm0hMCIXGvhPr2cat7kpukoQ3HQY5DJsgfhNMGAc30U OTiQESur6Hd5fMqTiNOz+wO41D3veCTei4C6ZUSycoLzE3Zne3jt1msrPlXHUG71/b3B HydbPAKTVceVvUuf+snGMavooaXjMWwdr65JeVDLEue6tQHC6zfswC3cbap3E1y6qOJ7 qIVMGYWYtVmoseCx8v2yzw0uOLYjd+tTsO7lDu5NUwpZVOGOzqtLmL0JDw3jPUGV1EOt Ykioruj0T9BHSPF1p8awRUsmCW76uU/VbEdmS330iqYPUE0M+05Qf/4RL0F63TJM2xka z2OQ== X-Gm-Message-State: AA+aEWbqTv0/phr9y2P6Wz2U+aesu+tgiLGLc89cyoqWJ3YLHZ55sgBT yKfkLlp2+sa9POc++tFFu7205PNP X-Google-Smtp-Source: AFSGD/Udqn2sVYo+64wnDgqmo5elje6EHNbtQL4d3z440q6OTl3c2e/MR4vGt8xabPMR2k8sXNFQUQ== X-Received: by 2002:a5d:4b01:: with SMTP id v1mr9846684wrq.5.1542569418751; Sun, 18 Nov 2018 11:30:18 -0800 (PST) Received: from gumby.homeunix.com ([90.211.70.103]) by smtp.gmail.com with ESMTPSA id y83-v6sm26218869wmb.20.2018.11.18.11.30.17 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sun, 18 Nov 2018 11:30:18 -0800 (PST) Date: Sun, 18 Nov 2018 19:30:16 +0000 From: RW To: freebsd-hackers@freebsd.org Subject: Re: 13-CURRENT: several GB swap being used despite plenty of free RAM Message-ID: <20181118193016.094cca9d@gumby.homeunix.com> In-Reply-To: References: <201811180154.wAI1smhg049214@slippy.cwsent.com> X-Mailer: Claws Mail 3.16.0 (GTK+ 2.24.32; amd64-portbld-freebsd11.1) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4EDCD6C35F X-Spamd-Result: default: False [-3.63 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[googlemail.com]; NEURAL_HAM_MEDIUM(-1.00)[-0.998,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; FREEMAIL_FROM(0.00)[googlemail.com]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_THREE(0.00)[3]; IP_SCORE(-0.72)[ipnet: 2a00:1450::/32(-1.78), asn: 15169(-1.73), country: US(-0.09)]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_TRACE(0.00)[googlemail.com:+]; DMARC_POLICY_ALLOW(-0.50)[googlemail.com,quarantine]; RCVD_IN_DNSWL_NONE(0.00)[4.3.4.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; MX_GOOD(-0.01)[alt3.gmail-smtp-in.l.google.com,alt4.gmail-smtp-in.l.google.com,gmail-smtp-in.l.google.com,alt2.gmail-smtp-in.l.google.com,alt1.gmail-smtp-in.l.google.com]; NEURAL_HAM_SHORT(-0.90)[-0.900,0]; RECEIVED_SPAMHAUS_PBL(0.00)[103.70.211.90.zen.spamhaus.org : 127.0.0.10]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[googlemail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_TLS_LAST(0.00)[] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Nov 2018 19:30:21 -0000 On Sun, 18 Nov 2018 17:53:52 +0100 (CET) Wojciech Puchar wrote: > why someone changes WELL WORKING swapping algorithm? > > On Sun, 18 Nov 2018, Stefan Blachmann wrote: > > > So I guess the unwillingness of the developer team to add an option > > like "NoPreemptiveSwapping", which disables swapping out as long as > > there is free physical memory available, is of psychological > > nature. > > in FReeBSD 12 there is > > vm.swap_idle_threshold2: 10 > vm.swap_idle_threshold1: 2 > vm.swap_idle_enabled: 0 > > > which works as expected and can be turned on or off as required. The above settings are about process level swapping, i.e. the deactivation of whole processes, they don't control swapping in the sense of paging out to the swap device. By default process swapping is only used under extreme memory shortage, setting vm.swap_idle_enabled=1 allows idle processes to be deactivated. It's only really intended for some special cases like login servers, which tend to have a lot of inactive shell and editor processes. Changing those setting wont reduce swap usage, they may increase it. From owner-freebsd-hackers@freebsd.org Sun Nov 18 22:05:20 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AB99D1125720 for ; Sun, 18 Nov 2018 22:05:20 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Received: from smtp-out-no.shaw.ca (smtp-out-no.shaw.ca [64.59.134.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "Client", Issuer "CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 92CE3731E3; Sun, 18 Nov 2018 22:05:19 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Received: from spqr.komquats.com ([70.67.125.17]) by shaw.ca with ESMTPA id OVBkg5bbMP7x2OVBlgu86r; Sun, 18 Nov 2018 15:05:11 -0700 X-Authority-Analysis: v=2.3 cv=X9GD11be c=1 sm=1 tr=0 a=VFtTW3WuZNDh6VkGe7fA3g==:117 a=VFtTW3WuZNDh6VkGe7fA3g==:17 a=kj9zAlcOel0A:10 a=JHtHm7312UAA:10 a=xfDLHkLGAAAA:8 a=YxBL1-UpAAAA:8 a=6I5d2MoRAAAA:8 a=J_VOeTAUeCDFlHUBMgUA:9 a=CjuIK1q_8ugA:10 a=IfaqVvZgccqrtc8gcwf2:22 a=Ia-lj3WSrqcvXOmTRaiG:22 a=IjZwj45LgO3ly-622nXo:22 Received: from slippy.cwsent.com (slippy [10.1.1.91]) by spqr.komquats.com (Postfix) with ESMTPS id 7D8FF35A9; Sun, 18 Nov 2018 14:05:07 -0800 (PST) Received: from slippy.cwsent.com (localhost [127.0.0.1]) by slippy.cwsent.com (8.15.2/8.15.2) with ESMTP id wAIM56q7036244; Sun, 18 Nov 2018 14:05:06 -0800 (PST) (envelope-from Cy.Schubert@cschubert.com) Received: from slippy (cy@localhost) by slippy.cwsent.com (8.15.2/8.15.2/Submit) with ESMTP id wAIM543W036241; Sun, 18 Nov 2018 14:05:04 -0800 (PST) (envelope-from Cy.Schubert@cschubert.com) Message-Id: <201811182205.wAIM543W036241@slippy.cwsent.com> X-Authentication-Warning: slippy.cwsent.com: cy owned process doing -bs X-Mailer: exmh version 2.8.0 04/21/2012 with nmh-1.7.1 Reply-to: Cy Schubert From: Cy Schubert X-os: FreeBSD X-Sender: cy@cwsent.com X-URL: http://www.cschubert.com/ To: Stefan Blachmann cc: Cy Schubert , Mark Millard , Rebecca Cran , freebsd-hackers Hackers , Mark Johnston , Ian Lepore , Wojciech Puchar Subject: Re: 13-CURRENT: several GB swap being used despite plenty of free RAM In-Reply-To: Message from Stefan Blachmann of "Sun, 18 Nov 2018 13:11:49 +0100." Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Sun, 18 Nov 2018 14:05:04 -0800 X-CMAE-Envelope: MS4wfESIzqdUbe6vDrEggYLVoj/H9qpzCTuG+2TzbC09RZG9OBTt+m8KwwWc1Ku8JdhJmVq3tGB67zvOChDKqV1LyB+/99oo7nSvqUbAJJurvQ9GypKhh+dJ OeW4zVIWCnZaN3OCR2a8t/EjLkqF9+GCswA72L3ou1+3e2Ub0myrYM1oWNLEDhMDIekbnetinGP+cN90OknsvCXuxTIYShf5v8cxA9fTan90a9WodasDYzG6 41feLkKeVjwgkkKOKZaIyk3U6RTfp6IsRpsQ6KpaztfpzNwZBF4myNDbnip/0+ej69PbjzBG5PiC5RnTYik4zfEfqYsNlzRYs+FZmar8R5QyRYSY9Q7M0F4j XvIdfX0x X-Rspamd-Queue-Id: 92CE3731E3 X-Spamd-Result: default: False [-2.54 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_FIVE(0.00)[5]; RECEIVED_SPAMHAUS_PBL(0.00)[17.125.67.70.zen.spamhaus.org : 127.0.0.11]; FROM_HAS_DN(0.00)[]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; HAS_XAW(0.00)[]; HAS_REPLYTO(0.00)[Cy.Schubert@cschubert.com]; REPLYTO_EQ_FROM(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[cached: spqr.komquats.com]; NEURAL_HAM_SHORT(-0.94)[-0.945,0]; RCPT_COUNT_SEVEN(0.00)[8]; NEURAL_HAM_MEDIUM(-0.99)[-0.994,0]; R_SPF_NA(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_IN_DNSWL_LOW(-0.10)[13.134.59.64.list.dnswl.org : 127.0.5.1]; R_DKIM_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:6327, ipnet:64.59.128.0/20, country:CA]; IP_SCORE(-0.89)[ipnet: 64.59.128.0/20(-2.41), asn: 6327(-1.93), country: CA(-0.10)]; FROM_EQ_ENVFROM(0.00)[] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Nov 2018 22:05:20 -0000 In message , Stefan Blachmann writes: > The inconveniences that the new swapping strategy causes are a regular > topic in the FreeBSD forums. > > Desktop users complain about lagginess, server users complain of long > delays because server processes intended to be kept in memory for > quick response times got swapped out and need to be swapped in again, > resulting in outrageously poor server performance in spite of plenty > of unused memory. There are many factors for this. One of the biggest mistakes people make is use UFS and ZFS on the same system. Limiting ZFS ARC max would be a good place to start. Unfortunately like many operating systems these days, FreeBSD's tuning parameters are generally baked in. And, fencing (an IBM term to limit address space memory use or to establish a minimum) isn't available. So, tuning for a site specific workload is more difficult. Making sure there is plenty of RAM installed and tuning down ZFS ARC can go a long way. Having said that, it's been a while since I've had to do this. The updates made to ZFS ARC and NUMA have allowed me to rely on the algorithms baked into FreeBSD. My 8 GB systems have performed rather well. > > Turning off swap completely, as Cy Schubert suggests, is strongly > discouraged in the forums, as it can lead to kernel panicking because > of being unable to swap out in critical kernel memory shortage > situations, leading to the risk of very serious filesystem > corruption. Actually, I didn't suggest it. I simply pointed out the option. Furthermore, it was a mistake for Linux to remove swapping entirely from their kernel. For those who want to try it, however, the option is there. > > However, Cy Schubert is probably right when stating that the new > swapping strategy resembles the 1960s-1980s industry's main swapping > strategy. > The bad thing is now, that nowadays memory is no longer scarce and > people can dimension their memory such that under normal circumstances > there will never be any need to swap. Adding more RAM has been the general thinking over the last 15-20 years. > > So I guess the unwillingness of the developer team to add an option > like "NoPreemptiveSwapping", which disables swapping out as long as > there is free physical memory available, is of psychological nature. How do you know processes are indeed swapped out rather than (many) pages paged out. You can't tell and using top's w mode, sorted by swap doesn't tell us anything because IMO it's bogus, incorrect numbers. > > Lacking such an option, there is still the possibility to use rctl to > disable swapping for particular users, processes, jails etc to > mitigate the problems caused by the new swapping strategy to some > degree. Our options are limited. I'd prefer something similar to what we had on MVS, when I worked on it, however even IBM has dumbed down SRM (System Resource Manager) such that knobs that were once there no longer are. Tuning based upon analysis of site specific stats are a thing of the past. It's 2018 and sadly one size fits all. -- Cheers, Cy Schubert FreeBSD UNIX: Web: http://www.FreeBSD.org The need of the many outweighs the greed of the few. From owner-freebsd-hackers@freebsd.org Mon Nov 19 04:44:11 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4FA451133C54 for ; Mon, 19 Nov 2018 04:44:11 +0000 (UTC) (envelope-from rebecca@bluestop.org) Received: from muon.bluestop.org (muon.bluestop.org [96.73.9.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 55E7381129; Mon, 19 Nov 2018 04:44:10 +0000 (UTC) (envelope-from rebecca@bluestop.org) Received: from muon.bluestop.org (localhost [127.0.0.1]) by muon.bluestop.org (Postfix) with ESMTP id 45999D32FB; Sun, 18 Nov 2018 21:44:47 -0700 (MST) Received: from muon.bluestop.org ([127.0.0.1]) by muon.bluestop.org (muon.bluestop.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id hJ2pjCJWyXUT; Sun, 18 Nov 2018 21:44:47 -0700 (MST) Received: from photon.int.bluestop.org (gw.bluestop.org [96.73.9.3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by muon.bluestop.org (Postfix) with ESMTPSA; Sun, 18 Nov 2018 21:44:47 -0700 (MST) From: Rebecca Cran To: freebsd-hackers@freebsd.org, Cy Schubert Cc: Stefan Blachmann , Ian Lepore , Wojciech Puchar , Mark Johnston Subject: Re: 13-CURRENT: several GB swap being used despite plenty of free RAM Date: Sun, 18 Nov 2018 21:44:02 -0700 Message-ID: <3912010.ex36aFohjY@photon.int.bluestop.org> In-Reply-To: <201811182205.wAIM543W036241@slippy.cwsent.com> References: <201811182205.wAIM543W036241@slippy.cwsent.com> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Rspamd-Queue-Id: 55E7381129 X-Spamd-Result: default: False [-5.71 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[bluestop.org]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(-3.21)[ip: (-9.81), ipnet: 96.64.0.0/11(-4.89), asn: 7922(-1.25), country: US(-0.09)]; RCPT_COUNT_FIVE(0.00)[6]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[bluestop.org:+]; DMARC_POLICY_ALLOW(-0.50)[bluestop.org,quarantine]; MX_GOOD(-0.01)[mail.bluestop.org]; NEURAL_HAM_SHORT(-0.99)[-0.993,0]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; CTE_CASE(0.50)[]; ASN(0.00)[asn:7922, ipnet:96.64.0.0/11, country:US]; FREEMAIL_CC(0.00)[gmail.com] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Nov 2018 04:44:11 -0000 On Sunday, 18 November 2018 15:05:04 MST Cy Schubert wrote: > Unfortunately like many operating systems these days, FreeBSD's tuning > parameters are generally baked in. And, fencing (an IBM term to limit > address space memory use or to establish a minimum) isn't available. > So, tuning for a site specific workload is more difficult. Talking of tuning, this seems like a good resource for tuning for desktop use: https://cooltrainer.org/a-freebsd-desktop-howto/ . -- Rebecca From owner-freebsd-hackers@freebsd.org Mon Nov 19 10:24:11 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A80FF113B63A for ; Mon, 19 Nov 2018 10:24:11 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: from puchar.net (puchar.net [194.1.144.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "puchar.net", Issuer "puchar.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id BC8456F588 for ; Mon, 19 Nov 2018 10:24:10 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.15.2/8.15.2) with ESMTPS id wAJAO7Bn053630 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 19 Nov 2018 11:24:07 +0100 (CET) (envelope-from puchar-wojtek@puchar.net) Received: from localhost (puchar-wojtek@localhost) by puchar.net (8.15.2/8.15.2/Submit) with ESMTP id wAJAO1Yq053627; Mon, 19 Nov 2018 11:24:01 +0100 (CET) (envelope-from puchar-wojtek@puchar.net) Date: Mon, 19 Nov 2018 11:24:01 +0100 (CET) From: Wojciech Puchar To: RW cc: freebsd-hackers@freebsd.org Subject: Re: 13-CURRENT: several GB swap being used despite plenty of free RAM In-Reply-To: <20181118193016.094cca9d@gumby.homeunix.com> Message-ID: References: <201811180154.wAI1smhg049214@slippy.cwsent.com> <20181118193016.094cca9d@gumby.homeunix.com> User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Rspamd-Queue-Id: BC8456F588 X-Spamd-Result: default: False [-0.35 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.06)[-0.060,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_SPAM_SHORT(0.01)[0.011,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[puchar.net]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[puchar.net]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[90.144.1.194.list.dnswl.org : 127.0.10.0]; IP_SCORE(0.00)[country: PL(0.02)]; FREEMAIL_TO(0.00)[googlemail.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:43476, ipnet:194.1.144.0/24, country:PL]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Nov 2018 10:24:11 -0000 > > The above settings are about process level swapping, i.e. the > deactivation of whole processes, they don't control swapping in the > sense of paging out to the swap device. > > By default process swapping is only used under extreme memory shortage, > setting vm.swap_idle_enabled=1 allows idle processes to be deactivated. > It's only really intended for some special cases like login servers, > which tend to have a lot of inactive shell and editor processes. > and my servers that have LOTS of httpd servers each for one webpage which are usually rarely visited. > Changing those setting wont reduce swap usage, they may increase > it. > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > > From owner-freebsd-hackers@freebsd.org Mon Nov 19 10:25:11 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 066CE113B677 for ; Mon, 19 Nov 2018 10:25:11 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: from puchar.net (puchar.net [194.1.144.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "puchar.net", Issuer "puchar.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 690426F63A; Mon, 19 Nov 2018 10:25:10 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.15.2/8.15.2) with ESMTPS id wAJAP06Z054115 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 19 Nov 2018 11:25:00 +0100 (CET) (envelope-from puchar-wojtek@puchar.net) Received: from localhost (puchar-wojtek@localhost) by puchar.net (8.15.2/8.15.2/Submit) with ESMTP id wAJAOtYr054112; Mon, 19 Nov 2018 11:24:55 +0100 (CET) (envelope-from puchar-wojtek@puchar.net) Date: Mon, 19 Nov 2018 11:24:55 +0100 (CET) From: Wojciech Puchar To: Cy Schubert cc: Stefan Blachmann , Mark Millard , Rebecca Cran , freebsd-hackers Hackers , Mark Johnston , Ian Lepore , Wojciech Puchar Subject: Re: 13-CURRENT: several GB swap being used despite plenty of free RAM In-Reply-To: <201811182205.wAIM543W036241@slippy.cwsent.com> Message-ID: References: <201811182205.wAIM543W036241@slippy.cwsent.com> User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Rspamd-Queue-Id: 690426F63A X-Spamd-Result: default: False [-0.84 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.03)[-0.026,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[puchar.net]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[cached: puchar.net]; NEURAL_HAM_SHORT(-0.50)[-0.505,0]; RCPT_COUNT_SEVEN(0.00)[8]; RCVD_IN_DNSWL_NONE(0.00)[90.144.1.194.list.dnswl.org : 127.0.10.0]; IP_SCORE(0.00)[country: PL(0.02)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:43476, ipnet:194.1.144.0/24, country:PL]; FREEMAIL_CC(0.00)[gmail.com]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Nov 2018 10:25:11 -0000 > can go a long way. Having said that, it's been a while since I've had > to do this. The updates made to ZFS ARC and NUMA have allowed me to > rely on the algorithms baked into FreeBSD. My 8 GB systems have > performed rather well. 8GB is LOT LOT LOT of memory. From owner-freebsd-hackers@freebsd.org Mon Nov 19 10:32:53 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DE3CA113BA78 for ; Mon, 19 Nov 2018 10:32:52 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-ed1-x52f.google.com (mail-ed1-x52f.google.com [IPv6:2a00:1450:4864:20::52f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 137A06FC63 for ; Mon, 19 Nov 2018 10:32:51 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-ed1-x52f.google.com with SMTP id o10so2279459edt.13 for ; Mon, 19 Nov 2018 02:32:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-transfer-encoding:content-language; bh=bW/lo4HEIymRt5PKfJumr95ucob0cRUYz+wo9J8+CtE=; b=Av+hwz8vE7n4TjYx9OiHveusNCvhoQDi2ux5P0cxPvhJKVbsMA/7A6TJKrr1qzB+zv ViecncpKNaPCbEjwF7uNVJYGzpuz0/8TNQ0RKWVKNtf1dvioxhyKt+Ul1wI/LayJmHMt gNDeOGnuhSj0D1V+P9Lh5dVb3EUbDFyKG1Wcb1V/pgkC0ARvfg8CvwCVYCc2uORDhRWk jx+22rIA8XjgwZiGIlojoQZTTvWblajH2mFiIQJe/kk3rF9cV5ruavpDzIPpzvGwaNto gCUGnpbSziqczWO+lAF3unfEvJ0xkBoaWrfoFUM14WU2lW2nnPxQeBtQiHV8AHsJAtCJ B+xQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=bW/lo4HEIymRt5PKfJumr95ucob0cRUYz+wo9J8+CtE=; b=L6JEzQE4x4KugSfP6mv9+KaQODrxQExqX6JvkcRxd/EzEA8HMKAxPT2bWVEX2dHIBR 6RkbGxqH+qT7jlHe0JxNxXZbXwCZ63axaRt+WpduUylRIIHsrsnzGMG1mC9wIfj8KH12 zKTgiv06pVaE65tavoDDiljeT6gi0ti11dwKKRisHreqW3Smkmspio2Km55QjtT5BZNV FxHUmQMtF46aPKNKYhnE+LLLoVOZpmDh/cVenAWASlFQl6Av9ld+hFjiyyLLxRsWPXCZ E3o9mcMt+csallbCiNLVmsMJJTVJ7sHaiwT+6HDB1QON5zmSqQzX1U1DR7q00NnwBtFf sjLw== X-Gm-Message-State: AA+aEWYoECzLEQ0Fwpne2hjF9Pk0JA2Jahnvk4AoLGiDliS3/S3im7DH QNEfOC1hqjyvtC81ht+hTqg0ObMqLw8= X-Google-Smtp-Source: AFSGD/XOQFhvMhs40I6uRpXRWCbLHdYdu93Fk8+8XEbtdys9jVc2ZgipmEuhYE3lJNa5C+KJlMvLTQ== X-Received: by 2002:a50:8262:: with SMTP id 89mr2011493edf.125.1542623570389; Mon, 19 Nov 2018 02:32:50 -0800 (PST) Received: from [10.44.128.75] ([161.12.40.153]) by smtp.gmail.com with ESMTPSA id g40sm9429250edg.39.2018.11.19.02.32.49 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Nov 2018 02:32:49 -0800 (PST) Subject: Re: 13-CURRENT: several GB swap being used despite plenty of free RAM To: freebsd-hackers@freebsd.org References: <201811182205.wAIM543W036241@slippy.cwsent.com> From: Steven Hartland Message-ID: <201f06e9-54b8-32f5-e4ef-be694d511d93@multiplay.co.uk> Date: Mon, 19 Nov 2018 10:32:50 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.3.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-Rspamd-Queue-Id: 137A06FC63 X-Spamd-Result: default: False [-4.80 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[multiplay-co-uk.20150623.gappssmtp.com]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_THREE(0.00)[3]; DMARC_NA(0.00)[multiplay.co.uk]; MX_GOOD(-0.01)[cached: ASPMX.L.GOOGLE.COM]; DKIM_TRACE(0.00)[multiplay-co-uk.20150623.gappssmtp.com:+]; RCVD_IN_DNSWL_NONE(0.00)[f.2.5.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; NEURAL_HAM_SHORT(-0.96)[-0.958,0]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; IP_SCORE(-2.33)[ip: (-8.13), ipnet: 2a00:1450::/32(-1.73), asn: 15169(-1.70), country: US(-0.09)]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Nov 2018 10:32:53 -0000 It really isn't On 19/11/2018 10:24, Wojciech Puchar wrote: >> can go a long way. Having said that, it's been a while since I've had >> to do this. The updates made to ZFS ARC and NUMA have allowed me to >> rely on the algorithms baked into FreeBSD. My 8 GB systems have >> performed rather well. > > 8GB is LOT LOT LOT of memory. > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to > "freebsd-hackers-unsubscribe@freebsd.org" From owner-freebsd-hackers@freebsd.org Mon Nov 19 12:47:19 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 531A711029DA for ; Mon, 19 Nov 2018 12:47:19 +0000 (UTC) (envelope-from steevanxperia@gmail.com) Received: from mail-ot1-x342.google.com (mail-ot1-x342.google.com [IPv6:2607:f8b0:4864:20::342]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 41BC0740B6; Mon, 19 Nov 2018 12:47:18 +0000 (UTC) (envelope-from steevanxperia@gmail.com) Received: by mail-ot1-x342.google.com with SMTP id a11so23842567otr.10; Mon, 19 Nov 2018 04:47:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=s88MjzisMAdlXVyBoiQJJI73aj+mBbz9PchaDenqvKk=; b=YSw9AB0+rz/+RodAmCexR4YS1gF71nAvRrOUZPr2UlDgWCnCpCDvyI3A/G6drLD9Ez d77tCHJ6pMzeBAql2SfWKv/xLoqbOgOkWb1iOgO8rjuYDT3jv/Bzoq1Yo7MuSeCPbSQl zLZe0omp8Ie3pvlepPRtlbpAlMeTi11gWgxLfH6onPFhGEsSZp7u2xFSaeos4hnXjPj0 4XlkI9Zn/5Pq+q9O657kShzr9bwAOE7GWT/VdCBKRz72gyAVbzz8ApJ8DdOXDkpHPz2O 2Pp46hzSvwkoVJR33pOzOc8exikSlKLApIFr0OqoelPfd7S/K00ntMYn7F39/I55gal9 KFFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=s88MjzisMAdlXVyBoiQJJI73aj+mBbz9PchaDenqvKk=; b=WF2UkHNfIaCrsoXcuVUvmDT8Efw8BlmHhLtxGF3Pa9RKeOYaTf3rQ+kppbpcFQvScc zq4hIidKvghzLshkBn5Lzac/BgylIV6NnQBRbwOtMmvsu/0ASXr4LLzA9lwy9/JmbXRt DCI5XYSqAwjy1JBkqOXnjDgC8xE+QRwDc6NgxJLznYLm896mE9l2ws+glkBOeNgBpiPf CitdOKsyV5n5mSideJHXHcFNCaqLzG6FgQG1g3yrbcIyyTGgF5o8nALnuoLO1CCvd+28 ArrLM+HfuS2yDB+vx2GjlczLy0DYgDEgTmgo13sQaMc/zH9PeaQwDYaOkYw/udKmOwUb Neog== X-Gm-Message-State: AA+aEWZAxYOJqcpxYePXZhMkT/EyjqsBsslqBmRoL/LPIN3mScHR/H7i +6CsfFQCvz0887M5gMHvbncK+ROue099YYa5zsw+kNMxfwI= X-Google-Smtp-Source: AFSGD/VpaWHgW+GhriZWbF7NF5TJ+/W23cmDLGrjFvBWgkwiF0hduAt7ZiK85DqvQ6YpVlJ7lON2Ouht3xgw6TDV2q0= X-Received: by 2002:a9d:1541:: with SMTP id z1mr181249otz.305.1542631637387; Mon, 19 Nov 2018 04:47:17 -0800 (PST) MIME-Version: 1.0 References: <7bb8463e-aa5d-bade-7ae7-6c612bae65f9@gmail.com> <0f7e583f-9ad9-f210-4bdb-1ace13ab8c5f@freebsd.org> In-Reply-To: <0f7e583f-9ad9-f210-4bdb-1ace13ab8c5f@freebsd.org> From: Steevan Rodrigues Date: Mon, 19 Nov 2018 18:17:05 +0530 Message-ID: Subject: Re: FreeBSD 11.x thread creation time is 9000+ microseconds on Intel Xeon Gold series CPU To: se@freebsd.org Cc: mad@madpilot.net, freebsd-hackers@freebsd.org, =?UTF-8?Q?V=C3=A1clav_Haisman?= X-Rspamd-Queue-Id: 41BC0740B6 X-Spamd-Result: default: False [-3.07 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.95)[-0.954,0]; R_DKIM_ALLOW(-0.20)[gmail.com]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; IP_SCORE(-0.31)[ip: (2.78), ipnet: 2607:f8b0::/32(-2.52), asn: 15169(-1.70), country: US(-0.09)]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[2.4.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; NEURAL_HAM_SHORT(-0.80)[-0.804,0]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Server: mx1.freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Nov 2018 12:47:19 -0000 I tried FreeBSD 12 Beta 3 version on this server with Xeon Gold 5115 CPU. All these problems have disappeared. Thread creation time has improved greatly . It is now below 100 usec ( in FreeBSD 11.x 9000+ usec) Also I had another issue with FreeBSD 11.x which related to contigmalloc and contigfree . Actually contigfree was taking too much time on FreeBSD 11.x on this same server with Xeon Gold 5115 CPU. In FreeBSD 12 Beta3 also contigfree takes much more time compared to contigmalloc. However when I compare the values to FreeBSD 11.x number I can see huge improvement in FreeBSD 12 Beta 3 . Because of this contigfree issue my driver unload used to take 5 to 20 minutes in FreeBSD 11.x. Now my driver takes only a few seconds to load and a few seconds to unload in FreeBSD 12. BEta 3. Hence it looks like the problem is with FreeBSD 11.x. I am still waiting for final release version of FreeBSD 12.0 Regards, Steevan On Tue, Nov 6, 2018 at 6:18 PM Stefan Esser wrote: > Am 06.11.18 um 12:05 schrieb Steevan Rodrigues: > > Thanks to all of you for the quick responses. > > > > I see this issue only on a particular server. > > I do not see this issue on another server with slightly older processor > > Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz (dual socket 6 core per > socket) > > In this system the same program takes only about 800 usec > > > > I wonder whether it is problem with 11.x FreeBSD release on this > particular > > processor. > > I came across two results (given below) from Phoronix OSbench test suite > > which also show poor thread creation time for FreeBSD 11.x > > > > https://openbenchmarking.org/result/1804094-AR-1804096AR58 > > https://openbenchmarking.org/result/1807114-RA-WINDOWSSE47 > > > > In this look for "Create Threads" row. > > This list shows, that FreeBSD-12 is fast, only FreeBSD-11 shows the long > thread creation time (16000 vs 36 in whatever units), even though > FreeBSD-12 > was running with KPTI-Patches. > > This lets me think that the issue that caused these delays has been fixed, > but not merged to FreeBSD-11, yet.# > > I guess that the cause for the delays is the synchronization IPIs between > sockets, and that these might be very high due to some cores in deep sleep > modes (with corresponding long wake-up delays). > > Regards, STefan > From owner-freebsd-hackers@freebsd.org Mon Nov 19 12:49:12 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9C6FF1102AF6 for ; Mon, 19 Nov 2018 12:49:12 +0000 (UTC) (envelope-from steevanxperia@gmail.com) Received: from mail-oi1-x229.google.com (mail-oi1-x229.google.com [IPv6:2607:f8b0:4864:20::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D7CC874351 for ; Mon, 19 Nov 2018 12:49:11 +0000 (UTC) (envelope-from steevanxperia@gmail.com) Received: by mail-oi1-x229.google.com with SMTP id 192-v6so25131445oii.6 for ; Mon, 19 Nov 2018 04:49:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=8/IA1eCjJ+axOhBXJYYMPx7MB4rsevTdSLN+SZ54pFQ=; b=veNxxBsyQnjxoNkHwBNygKp1e2jeVxiJVvtEqciy/o2HtLGDSiDhCsZLpm5+1RIDGC z8yc8Twghfqdp9QNKoI92MEuxV1acIPcWou/jEhBayw9V8pQnkDP4hc3EKpO7pVTjmCD o+56hfgXg7C6nK+5crxRFSHdVMaKMn3ZdAJnxVG6lWyK46RaC241GU6lc2Wf/y/3mMYp s7KalKHOK2UQ9sQ5xgkbh/kAfMZLb/ZKVBNG4QVhuHW7s258+o7SYamOHnTWFaDa1xIL uWmvjO5UAXjEi7uGQX1avk9zQtYRd/ThlKfNAjnFPDhNJ0i2BDMLoif94ieVtpbee36I fBcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=8/IA1eCjJ+axOhBXJYYMPx7MB4rsevTdSLN+SZ54pFQ=; b=rzIPFPLKEjaECmjrbk04Q1Ro/X4zsXWGlUta5k3eYbzcS8xb1pjd0NPVnFwSY95GNi Qiyw8JmgQIAM7mrTIgVl6Px6pVu/5nJquyXRpWMOn+ZYt3QuacyQ4iNGY1/R59OqQm/V iNVbRtSJNy65cmHFp9+Frer8VtS0rBxYKAHNVkdZ7A9Qg6Wt29lJdVPQmIm+jgOxeeKD kAoisGzvTbC3vchfp1KPMxY6RpCoGS1tDGDA77hvJWzj2zmzamn4TxPE56KpXyNObnr1 qoyP58N4ZBZgX4axRw1yCaeWyuHzAYeVkIT6999bhE93jCq8X9ZXq8xPffxQl9mPSvaw 8xQQ== X-Gm-Message-State: AGRZ1gKSFxEGfZVZIzYcg8iuDRHtiARtFe0u6LI6MYbfTY3I5W+7TJpB Fjk2Emp/Gd3KMLdj9emYLzutJyIbObhAbiXsBfM= X-Google-Smtp-Source: AJdET5eNUdkNIjhs9OGioVGXGJA+7/c7ilGQ04vchonTasqU2lbIH/1B5MO4j57XvWJtkfAZzMPhj5BQK8fUnHLqXhU= X-Received: by 2002:aca:a57:: with SMTP id 84mr7344715oik.197.1542631751103; Mon, 19 Nov 2018 04:49:11 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Steevan Rodrigues Date: Mon, 19 Nov 2018 18:18:59 +0530 Message-ID: Subject: Re: Contigfree takes too much time that cuases PCIe driver unload to take up to 30 minutes To: rysto32@gmail.com Cc: freebsd-hackers@freebsd.org X-Rspamd-Queue-Id: D7CC874351 X-Spamd-Result: default: False [-3.83 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.99)[-0.994,0]; R_DKIM_ALLOW(-0.20)[gmail.com]; FROM_HAS_DN(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TO_DN_NONE(0.00)[]; NEURAL_HAM_SHORT(-0.96)[-0.963,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; RCVD_IN_DNSWL_NONE(0.00)[9.2.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; IP_SCORE(-0.86)[ipnet: 2607:f8b0::/32(-2.52), asn: 15169(-1.70), country: US(-0.09)] X-Rspamd-Server: mx1.freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Nov 2018 12:49:12 -0000 I tried FreeBSD 12 Beta 3 version on this server with Xeon Gold 5115 CPU. All these problems have disappeared. Actually contigfree was taking too much time on FreeBSD 11.x on my server with Xeon Gold 5115 CPU. In FreeBSD 12 Beta3 also contigfree still takes much more time compared to contigmalloc. However when I compare the values to FreeBSD 11.x number I can see huge improvement in FreeBSD 12 Beta 3 . Because of this contigfree issue my driver unload used to take 5 to 20 minutes in FreeBSD 11.x. Now my driver takes only a few seconds to load and a few seconds to unload in FreeBSD 12. BEta 3. Hence it looks like the problem was specific FreeBSD 11.x. Regards, Steevan On Sat, Oct 27, 2018 at 12:22 PM Steevan Rodrigues wrote: > > Yes , it is contigfree that is taking too much time. ( Contigfree is > taking almost 5x to 10x more time than contigmalloc ) > However, I see this issue only on a SuperMicro Server which has Xeon Gold > CPU with 10 cores ( dual CPU i.e total 20 cores). > > So I am wondering whether FreeBSD has performance issues on mutlicore ( > more than 16 cores) servers ? > Has anyone come across issues like this? > > Thanks > Steevan > > > > > On Fri, Oct 5, 2018 at 11:31 PM Ryan Stone wrote: > >> I'm not sure why configfree() would be taking a long time, but >> configmalloc() can be extraordinarily expensive when it needs to >> defragment memory to meet the request. Does your application really >> require a lot of physically contiguous memory? if you can restructure >> to not require contigmalloc() all -- maybe by using S/G DMA -- you may >> find your life significantly easier. >> > From owner-freebsd-hackers@freebsd.org Mon Nov 19 14:40:04 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8F0EE1105492 for ; Mon, 19 Nov 2018 14:40:04 +0000 (UTC) (envelope-from sblachmann@gmail.com) Received: from mail-ot1-x332.google.com (mail-ot1-x332.google.com [IPv6:2607:f8b0:4864:20::332]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8F64C77A17; Mon, 19 Nov 2018 14:40:03 +0000 (UTC) (envelope-from sblachmann@gmail.com) Received: by mail-ot1-x332.google.com with SMTP id a11so24169506otr.10; Mon, 19 Nov 2018 06:40:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=XfiBqUVncJFse1tDHMD0T4nyp9VP77uipHHTQ9L+5WE=; b=c7S331aZ8J4674ZZMNhQoVk432VkmTodS3QHzii2pRwh7m1TZtJyByhwajo3KR2iee y5hWcdbEVR0GFHGHWZiQEuqppp04cMmn7GrT9b+KyQBkHdAYEghVW8ENg8t0CPYomTVB +ytmPO2YSdqEtyL9stjd4bbjxuZQuABHaZGbOF7POTYgkUxjTn7b1X342iw9lZm3KQSG regdbRIJUv5i+xgidSBVlSSaHflBNHM20JlUxEz81pGiwkSGTa2CO2Ur6AMR88g9rzml F/iJ8jwn4W5DTiG3skr9u3ilvVNfnmZBDTVYDOyYo8Lt5lgkkaXTBf53WMjNd8ZA0sRv 3q4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=XfiBqUVncJFse1tDHMD0T4nyp9VP77uipHHTQ9L+5WE=; b=VMm1YTp+ZlKS+OvbAVTWIDEuxy72NO1iiHvRBe9UJZc369qlYeSWMnlB8MiFIt7klw U7207GeOfWPi6z/ZF7IcY9uU2cmMkNRCq9fEJoAH0Pm9Rh9NcCKN63U1h2QcHeSyurry Dfd2etU1qlxzE0vbDuu3/ovFqL4NNsTjrjCGWy1G814IP2JxbfBbDB7eBBXNsOk9xHp4 R3zzPJFGkVG69k8uQV4q+YigCEwZP1vOUoQFHGJ4ghyb48XUGAoEpbQpWnwcnTGHdmB8 T747Mafg97NOZvcwLqOB6uOBQy0lhiE4LpdHSB60txhAky0lwtWSEyTgM8/qwZilk7jo C48w== X-Gm-Message-State: AGRZ1gIbOiOeq4vhX7NoYNYbTMnElzhLPZ3TiYdcxKKDqKMVgl1Nk50x yCCiJlJ+WFFCQPRmjf5PGz3tFjlceIe2cH2uQYWIltut X-Google-Smtp-Source: AJdET5cnMn6SsgNIWPa88nOsnbycpV/mwW2D1adM/T9D3BXhjr185Rj+BSN9iBw3jJCj21IOnenxLVCvnpxQgTgJfIA= X-Received: by 2002:a9d:775a:: with SMTP id t26mr12450952otl.32.1542638402717; Mon, 19 Nov 2018 06:40:02 -0800 (PST) MIME-Version: 1.0 Received: by 2002:ac9:3052:0:0:0:0:0 with HTTP; Mon, 19 Nov 2018 06:40:01 -0800 (PST) In-Reply-To: References: <201811182205.wAIM543W036241@slippy.cwsent.com> From: Stefan Blachmann Date: Mon, 19 Nov 2018 15:40:01 +0100 Message-ID: Subject: Re: 13-CURRENT: several GB swap being used despite plenty of free RAM To: Wojciech Puchar Cc: Cy Schubert , Mark Millard , Rebecca Cran , freebsd-hackers Hackers , Mark Johnston , Ian Lepore Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 8F64C77A17 X-Spamd-Result: default: False [-3.82 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; R_DKIM_ALLOW(-0.20)[gmail.com]; FROM_HAS_DN(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; MIME_GOOD(-0.10)[text/plain]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; RCPT_COUNT_SEVEN(0.00)[7]; RCVD_IN_DNSWL_NONE(0.00)[2.3.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.95)[-0.953,0]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; IP_SCORE(-0.85)[ipnet: 2607:f8b0::/32(-2.49), asn: 15169(-1.69), country: US(-0.09)] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Nov 2018 14:40:04 -0000 On 11/19/18, Wojciech Puchar wrote: >> can go a long way. Having said that, it's been a while since I've had >> to do this. The updates made to ZFS ARC and NUMA have allowed me to >> rely on the algorithms baked into FreeBSD. My 8 GB systems have >> performed rather well. > > 8GB is LOT LOT LOT of memory. > This might indeed be a cue. Actually, I *never* had such annoying swap lag issues, until I replaced my old computers (8GB) with newer ones (24-48GB). With this increased memory, the ZFS ARC issue became apparent, causing massive problems when ARC used more than 40 of the 48GB, starving programs from memory, but this can be tuned easily. And the "preemptive swapping" issue became apparent, too. Letting programs idle for some time caused swapping out. This led to a very bad user experience... often when one changes to another desktop/virtual screen to another program, there is a delay of swapping in, which in some cases could last for seconds when hundreds of megabytes were swapped in again, in spite of some tens of gigabytes of memory that never have been used since system boot. > and my servers that have LOTS of httpd servers each for one webpage which are usually rarely visited. I wondered why my test server had a delayed response time the first few times I call a page from it after having it idled for some time. To me it looks like that the threads apache keeps cached in memory for fast response times got swapped out and the server response is slow until all these have become "memory resident" again. As the bad response times due to unnecessary preemptive swapping would induce Google downranking for sites that aren't visited sufficiently frequently enough to keep them from being swapped out, I consider this a very very serious problem. Disabling swap (either totally or for the affected programs/users/jails using rctl) and taking care that memory never runs out was the only way I was able to fix these issues. And I find very interesting that this seems to occur *only* on machines with *much* memory. The complaints about this swapping behavior I saw on the forums *always* regarded machines with at least 48GB of RAM. (I am not sure whether I saw reports regarding 32GB machines, so I would suggest investigating the issue on machines >=48GB) From owner-freebsd-hackers@freebsd.org Mon Nov 19 15:51:22 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8B0AF11078D0 for ; Mon, 19 Nov 2018 15:51:22 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A0EA37B95B for ; Mon, 19 Nov 2018 15:51:21 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (localhost [127.0.0.1]) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3) with ESMTP id wAJFpHO6062221; Mon, 19 Nov 2018 07:51:17 -0800 (PST) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: (from freebsd-rwg@localhost) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3/Submit) id wAJFpHn3062220; Mon, 19 Nov 2018 07:51:17 -0800 (PST) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201811191551.wAJFpHn3062220@pdx.rh.CN85.dnsmgr.net> Subject: Re: 13-CURRENT: several GB swap being used despite plenty of free RAM In-Reply-To: <201f06e9-54b8-32f5-e4ef-be694d511d93@multiplay.co.uk> To: Steven Hartland Date: Mon, 19 Nov 2018 07:51:17 -0800 (PST) CC: freebsd-hackers@freebsd.org X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-Rspamd-Queue-Id: A0EA37B95B X-Spamd-Result: default: False [0.78 / 15.00]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[dnsmgr.net]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(0.22)[0.216,0]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[cached: pdx.rh.CN85.dnsmgr.net]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-0.31)[-0.308,0]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:13868, ipnet:69.59.192.0/19, country:US]; MID_RHS_MATCH_FROM(0.00)[]; IP_SCORE(-0.02)[country: US(-0.09)] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Nov 2018 15:51:22 -0000 > It really isn't It is all relative. You have to take into perspective just what it is your "computing" and how much state it should need to do that "computation". Modern systems and software have become very wasteful as memory was "cheap". With 10nM getting the better of Intel, and DARPA pushing towards 5nM the cheapness factor is rapidly erroding. 8GB is a huge amount of memory if I am trying to play astroids on a 640x480 display, it is a drop in the bucket if I am trying to do fluid dynamics of a aircraft wing. > On 19/11/2018 10:24, Wojciech Puchar wrote: > >> can go a long way. Having said that, it's been a while since I've had > >> to do this. The updates made to ZFS ARC and NUMA have allowed me to > >> rely on the algorithms baked into FreeBSD. My 8 GB systems have > >> performed rather well. > > > > 8GB is LOT LOT LOT of memory. > > My diskless boot server laptop is running ZFS and now since updated to 12.0Beta4 NUMA too and is rather happy in its 2GB of memory. The work load fits niceless into this footprint. I probably would not want to fire up X11 and Firefox as a daily browswer on it, but the fact is it can also do that without too much pain. -- Rod Grimes rgrimes@freebsd.org From owner-freebsd-hackers@freebsd.org Mon Nov 19 17:41:18 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9BCCE110F1F5 for ; Mon, 19 Nov 2018 17:41:18 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic307-9.consmr.mail.gq1.yahoo.com (sonic307-9.consmr.mail.gq1.yahoo.com [98.137.64.33]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0A69481742 for ; Mon, 19 Nov 2018 17:41:17 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: wsEe1cwVM1mzvd37YjOoie0DKeN0C8.IomPi6w4mtuzzto52VHTPHEv2zsCvfqH MjfXp3um1TXN8NXvWjvfy1mhkMQSxMbxZn4A0SCh7NppZ8cd9XTpe6xq1Qyv5e.WfrbP5j7hg5tD MEx3qqcnnFNxsKyksvpX1sSZAXxTJnDIrcrmph63lKoEKRWq7bH7SWB9pUWMdEbeb7rBiO2b9SxJ VmETovH14axOjmYxLCxtEW3lgSW7b8B_ysRhh6QX35E4D2vOC58tmqlTilvVWdSGRkaUVOHTN_Td XuR58KLiE49n2dj4JKBPZPyxOWoiPZI_qxtFRwFNvCrf0RnHzCGLmRYhq72Pg64xfLoztiA9z64L vGvhFgS2doNQd2_U_mhyKB3kmEJaUzUbpx7z94xM.xP8cHW8ADq3aO1W04t9Pvbe2C.C2TfbzmON vlnHB4jofTJfZ.MwXCZFhdGbOKtIYcTU0OqVvpWsMe5FHaC_3rlMajSgEeXV1mQrHbS1paBmp6Q0 WUWRrbFKEk97DH0a_XFePZdvKN_CZJu3tvGqBUcPkYoqwhO4lUGimNtHEJuIAa3VTN25z4kRrxpa hbVlOT9AnYRKFyP9mb1OAywAcqtgZH2aTTPYKVHLaVbGWkd4wMBX_lyKCtcEFjl42QVInRXM37WZ RnBP3RQoEoSTMN.oYueI9GCxm.H_FrWVyL1RHVu1vf_IhNloPCunQhbVi7.E85SdwogWnNREEwnu QM6KolB1J_DyHJcIEpZlkuZwUcCGlijB.DKOSxQWeTPPbrn8SjkXbXVfiIYfTkjq8uCq55l5W7Th sW600uYTDCQJ3POZeMsOIeIM8wO.t_WS9PZK8UCGUxHU.sbiQkcyf9sb9QoK5gJtQRCGk2xQofDR uWSjQrxIpwe83n55lLowTIC5cmCT28b_8JBOWX._CyPdvk4CNlscX2DIkzkqgeflv7LJo6lYK5KR c8zlO2cmfZDbrKRJUtBC98LgIlTVsH7lUyIOHZ8ldjqTni4HR8G6qCMwNKEP_wb2PMpr46cbUPl6 bb2EFcc8JHd9C2vk8RbXoa3KVCrakf8HMDaJ9L6mjmMwDNrd0GKixOT0- Received: from sonic.gate.mail.ne1.yahoo.com by sonic307.consmr.mail.gq1.yahoo.com with HTTP; Mon, 19 Nov 2018 17:41:09 +0000 Received: from c-76-115-7-162.hsd1.or.comcast.net (EHLO [192.168.1.25]) ([76.115.7.162]) by smtp415.mail.gq1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 18d4660213a4acdc0052b74c7158b9f7; Mon, 19 Nov 2018 17:41:07 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 12.1 \(3445.101.1\)) Subject: Re: 13-CURRENT: several GB swap being used despite plenty of free RAM From: Mark Millard In-Reply-To: Date: Mon, 19 Nov 2018 09:41:06 -0800 Cc: Wojciech Puchar , Cy Schubert , Rebecca Cran , freebsd-hackers Hackers , Mark Johnston , Ian Lepore Content-Transfer-Encoding: quoted-printable Message-Id: References: <201811182205.wAIM543W036241@slippy.cwsent.com> To: Stefan Blachmann X-Mailer: Apple Mail (2.3445.101.1) X-Rspamd-Queue-Id: 0A69481742 X-Spamd-Result: default: False [-0.85 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com]; NEURAL_HAM_MEDIUM(-0.89)[-0.892,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; FREEMAIL_FROM(0.00)[yahoo.com]; NEURAL_SPAM_SHORT(0.31)[0.313,0]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; MX_GOOD(-0.01)[cached: mta6.am0.yahoodns.net]; RCVD_IN_DNSWL_NONE(0.00)[33.64.137.98.list.dnswl.org : 127.0.5.0]; RCPT_COUNT_SEVEN(0.00)[7]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; IP_SCORE(0.24)[ipnet: 98.137.64.0/21(0.72), asn: 36647(0.58), country: US(-0.09)]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Nov 2018 17:41:18 -0000 On 2018-Nov-19, at 06:40, Stefan Blachmann = wrote: > On 11/19/18, Wojciech Puchar wrote: >>> can go a long way. Having said that, it's been a while since I've = had >>> to do this. The updates made to ZFS ARC and NUMA have allowed me to >>> rely on the algorithms baked into FreeBSD. My 8 GB systems have >>> performed rather well. >>=20 >> 8GB is LOT LOT LOT of memory. >>=20 >=20 > This might indeed be a cue. > Actually, I *never* had such annoying swap lag issues, until I > replaced my old computers (8GB) with newer ones (24-48GB). >=20 > With this increased memory, the ZFS ARC issue became apparent, causing > massive problems when ARC used more than 40 of the 48GB, starving > programs from memory, but this can be tuned easily. >=20 > And the "preemptive swapping" issue became apparent, too. You mentioned ZFS. Have any reports of "preemptive swapping" been for a context where ZFS was not involved? This might always be part of the context, sort of like you are saying larger amounts of RAM are. My contexts do not use ZFS but I have one context with 96 GiBytes of RAM. It has no problem with pageouts/swapouts during idle, even when idle for days. Such activity is always well explained by process RAM use and dirty page handling that follows. No evidence of "preemptive swapping". But idle here means very little running: only running the processes the OS starts plus up to a few ssh sessions, nfs client/server included, ntpd included, ssh allowed, a couple of network protocol daemons enabled, sendmail enables set to "NO". (There are 32 "cpus".) The machines are not outward accessible and do not run X11 or other such. > Letting > programs idle for some time caused swapping out. Is there any evidence of growing RAM use during such generally idle times? Might there be some form of RAM leak in your workload or the OS (for how you are using it)? FreeBSD waits to start paging/swapping until the free RAM is low --by its criteria for low. The pageout/swapout is just a a step in the means of gaining more free RAM. Have you observed what is going on (RAM use) just before and during when pages-outs/swap-outs are first initiated? A description of that would be handy for folks trying to help you. > This led to a very > bad user experience... often when one changes to another > desktop/virtual screen to another program, there is a delay of > swapping in, which in some cases could last for seconds when hundreds > of megabytes were swapped in again, in spite of some tens of gigabytes > of memory that never have been used since system boot. >=20 >> and my servers that have LOTS of httpd servers each for one webpage = which are usually rarely visited. >=20 > I wondered why my test server had a delayed response time the first > few times I call a page from it after having it idled for some time. > To me it looks like that the threads apache keeps cached in memory for > fast response times got swapped out and the server response is slow > until all these have become "memory resident" again. >=20 > As the bad response times due to unnecessary preemptive swapping would > induce Google downranking for sites that aren't visited sufficiently > frequently enough to keep them from being swapped out, I consider this > a very very serious problem. >=20 >=20 > Disabling swap (either totally or for the affected > programs/users/jails using rctl) and taking care that memory never > runs out was the only way I was able to fix these issues. >=20 > And I find very interesting that this seems to occur *only* on > machines with *much* memory. > The complaints about this swapping behavior I saw on the forums > *always* regarded machines with at least 48GB of RAM. (I am not sure > whether I saw reports regarding 32GB machines, so I would suggest > investigating the issue on machines >=3D48GB) =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-hackers@freebsd.org Tue Nov 20 13:30:09 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5AB46112D08C for ; Tue, 20 Nov 2018 13:30:09 +0000 (UTC) (envelope-from soralx@cydem.org) Received: from smtp.triumf.ca (smtp.triumf.ca [142.90.100.195]) by mx1.freebsd.org (Postfix) with ESMTP id 367F36D74C for ; Tue, 20 Nov 2018 13:30:07 +0000 (UTC) (envelope-from soralx@cydem.org) Received: from mscad14 (mscad14.triumf.ca [142.90.115.36]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.triumf.ca (Postfix) with ESMTP id 13AA5F80C for ; Tue, 20 Nov 2018 05:30:01 -0800 (PST) Date: Tue, 20 Nov 2018 05:30:00 -0800 From: To: Subject: [bug] fsck refuses to repair damaged UFS using backup superblock Message-ID: <20181120053000.56fbee6b@mscad14> X-Mailer: Claws Mail 3.16.0 (GTK+ 2.24.32; amd64-portbld-freebsd9.3) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 367F36D74C X-Spamd-Result: default: False [2.78 / 15.00]; ARC_NA(0.00)[]; R_SPF_FAIL(1.00)[-all]; IP_SCORE(-0.02)[country: CA(-0.10)]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_SPAM_SHORT(0.56)[0.559,0]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; NEURAL_SPAM_MEDIUM(0.13)[0.133,0]; RCPT_COUNT_ONE(0.00)[1]; DMARC_NA(0.00)[cydem.org]; MX_GOOD(-0.01)[spamtrap.bmcorp.ca,spamtrap.sorokin.ca]; NEURAL_SPAM_LONG(0.61)[0.613,0]; FROM_NO_DN(0.00)[]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MID_RHS_NOT_FQDN(0.50)[]; ASN(0.00)[asn:36391, ipnet:142.90.0.0/16, country:CA]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Nov 2018 13:30:09 -0000 Howdy! Since send-pr(1) is now gone, I guess the next option is to send a message directly to the developers... Yesterday, I ran into a bug in fsck_ffs that gave me a little scare. Short story: on -CURRENT, fsck refuses to check a FS with a corrupted superblock, even when an alternate (backup) SB location is given. Long story. I've been testing a newly-built system based on an X399 platform with a 2950X CPU and an Optane 905P 480GB U.2 drive. The system ran a ~2-day old -CURRENT; when compiling newest world and kernel, I found the machine in a locked-up state. After a hard reset, boot failed because the root FS became corrupted & was not available: kernel: Superblock check-hash failed: recorded check-hash XXX != computed check-hash YYY I have not yet figured out why the corruption happened... bad hardware? bug in the NVMe driver? "OK", I thought, "No worries. We'll just boot using another disk, fsck the corrupted FS with a backup superblock, and be up in a moment". The machine was doing nothing but compiling, so no valuable data loss. So I did `dumpfs -m /dev/ada0p3` on the spare disk (which was the source for the new disk image, thus had almost identical partitions and filesystems) to get the FS details, then did `newfs -N [...] /dev/ada0p3` to find locations of superblock backups, then finally ran `fsck_ffs -b 192 /dev/nvd0p3` -- only to get the same "check- -hash failed" message, plus another strange message: "Can't open /dev/nvd0p3: [...]". Then fsck quits. Note that `fsck_ffs -b ...` on a FS with good superblock works OK. After fiddling with a debugger for a bit, I commented out the line "return (0);" in /usr/src/sbin/fsck_ffs/setup.c:136, recompiled fsck, and the FS was recovered successfully. What was actually happening: fsck's setup.c calls ufs_disk_fillout() from libufs' type.c, which in turn calls sbread() from the same library, which then calls sbget(disk->d_fd, &fs, -1) [[where '-1' is hard-coded to indicate the primary superblock]] that then simply invokes ffs_sbget from ffs kernel driver -- and this returns ENOENT, which eventually causes fsck to give up before even looking at the specified backup superblock. I don't know what exactly ufs_disk_fillout() does, but fortunately for me, fsck worked without the "sbread(disk)" part of that function having much luck on a disk with corrupted superblock. Also, I have a feeling that calling a kernel's ffs driver function when using fsck to fix a broken filesystem is not the best thing to do... Please CC, as I am not subscribed. -- [SorAlx] ridin' VN2000 Classic LT From owner-freebsd-hackers@freebsd.org Thu Nov 22 20:35:24 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 635DE1106C53 for ; Thu, 22 Nov 2018 20:35:24 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [IPv6:2a01:4f8:d12:604::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 7BADA85DDE for ; Thu, 22 Nov 2018 20:35:13 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13:0:0:0:5]) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id wAMKZ24K007962 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 22 Nov 2018 21:35:03 +0100 (CET) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: Received: from [10.58.0.4] (dadv@[10.58.0.4]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id wAMKZ2WL071843 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT) for ; Fri, 23 Nov 2018 03:35:02 +0700 (+07) (envelope-from eugen@grosbein.net) To: Freebsd hackers list From: Eugene Grosbein Subject: TRIM utility Message-ID: <7699de57-d903-1d61-ee42-062ed312b20d@grosbein.net> Date: Fri, 23 Nov 2018 03:34:49 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="BMWLwdDNWBlI0B1KElsFW6Prk6LfFDvG5" X-Spam-Status: No, score=0.3 required=5.0 tests=BAYES_00,LOCAL_FROM,SPF_PASS autolearn=no autolearn_force=no version=3.4.2 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * -0.0 SPF_PASS SPF: sender matches SPF record * 2.6 LOCAL_FROM From my domains X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on hz.grosbein.net X-Rspamd-Queue-Id: 7BADA85DDE X-Spamd-Result: default: False [-7.29 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; MX_INVALID(0.50)[greylisted]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; HAS_ATTACHMENT(0.00)[]; MIME_GOOD(-0.20)[multipart/signed,multipart/mixed,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_THREE(0.00)[3]; DMARC_NA(0.00)[grosbein.net]; TO_DN_ALL(0.00)[]; R_SPF_PERMFAIL(0.00)[]; NEURAL_HAM_SHORT(-0.95)[-0.954,0]; SIGNED_PGP(-2.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; IP_SCORE(-2.63)[ip: (-6.79), ipnet: 2a01:4f8::/29(-3.43), asn: 24940(-2.92), country: DE(-0.01)]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Nov 2018 20:35:24 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --BMWLwdDNWBlI0B1KElsFW6Prk6LfFDvG5 Content-Type: multipart/mixed; boundary="28tQcQm9tUPKBPJWQOQ20J0TXSTHRFKCN"; protected-headers="v1" From: Eugene Grosbein To: Freebsd hackers list Message-ID: <7699de57-d903-1d61-ee42-062ed312b20d@grosbein.net> Subject: TRIM utility --28tQcQm9tUPKBPJWQOQ20J0TXSTHRFKCN Content-Type: text/plain; charset=koi8-r Content-Transfer-Encoding: quoted-printable Hi! I found we have no utility capable to perform TRIM for the whole SSD devi= ce or arbitrary part of it, so I wrote simple one. I can't think of nice nam= e for it, so proposal are welcome! Draft name is "erase". I ask for pre-commit code review, too. The code is tested with TRIM-capable SSD and non-capable other devices. Currently it has four options, all of them are, hmm, optional: -b: to specify offset from the beginning of the device for trimmed region= instead of default 0; -l: to specify offset from the "-b" margin - length - for trimmed region = instead of whole device; -r rfile: for alternative way to specify length as length of referenced f= ile; -v: for verbose mode that shows final values for the beginning offset and= length. Later options override previous ones. Then it expects a list of device na= mes as arguments: erase ada0 erase /dev/ada0s1 erase -b 4096 -r file.img -v /dev/da0 /dev/da1 /dev/da2 The code: http://www.grosbein.net/freebsd/erase.c http://www.grosbein.net/freebsd/Makefile.erase My "mandoc" skills are very poor and English is not my native language, so any help with manual page creation will be appreciated. Eugene Grosbein P.S. I realized that our kernel-level TRIM support has no connection to c= am(4) nor to geom(4), so distinct utility instead of addition to camcontrol(8) or geom(8). --28tQcQm9tUPKBPJWQOQ20J0TXSTHRFKCN-- --BMWLwdDNWBlI0B1KElsFW6Prk6LfFDvG5 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQEcBAEBAgAGBQJb9xL0AAoJELDNGvImmIso2SkIAJnHBVSkCE3V9mcSSgrw+D6W q8viJSwMrgqeE/FrWp8XE5nWVEAVff2HcqcC05DRAW0ieuEAj0rCKWwbxrq/xvP0 lyhQc6nS0buX9yz5/FmwKCDOQ3AJ3mn5z9CXtg6aOhVre712eO5qnoIpIYoistsQ Fg/ku4KbKkU9DcfYwWMxhlotkNkcIFDSxisyQY/j5CvYAxjB/i5FfHYtI+ny7ki5 C1JbZoaOMjnGwZuZ05bj0r2cMtVnOZbxrBuKIXh5nf3BWY/6tSp5Sz30M7fcY7ge 3N/J+0/sJQnXLzxNoLb9639bQ/MSyx9mPNNeVTrJZaqVLu2+vZgBStLlF8YKRKU= =v8A5 -----END PGP SIGNATURE----- --BMWLwdDNWBlI0B1KElsFW6Prk6LfFDvG5-- From owner-freebsd-hackers@freebsd.org Thu Nov 22 23:20:24 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B65591133C23 for ; Thu, 22 Nov 2018 23:20:24 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-io1-xd35.google.com (mail-io1-xd35.google.com [IPv6:2607:f8b0:4864:20::d35]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C90206D43B for ; Thu, 22 Nov 2018 23:20:23 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-io1-xd35.google.com with SMTP id a3so7581160ioc.4 for ; Thu, 22 Nov 2018 15:20:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=ib7Roiw6HVomcORvmY9bkppGJJqna4XoYuIp+aOxH9M=; b=oeoasWoZsHz+fGNDMFrxvmRcXUMGOeBreASl6GpzqLPx6S8NSQxh9x8b/z9YaDiafa TyWMyosEFZskDExNDcBtWEpq3AJxWFVfomdoZCVZ29KgbvL7QrsxnvxBclvAxkVveKVs QKoP+6/n/+OF5w45w66fySyYk+j6Dt70TDk/zULjk+UxUTpjFgXJMfo9FInMGDnsrKIL g8AXM8XIJpv5Fa+hB42w0YhAVo72OOloQE5HQ7WjxHb0r62UTcuhAfARoZRHrvp3RapM fjxtFyumwbI3v7M77tUNFdntGu9syVAmkBPo0P6MLe60BFUwRcBlcyIJoagdhC/cjbND +/vg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=ib7Roiw6HVomcORvmY9bkppGJJqna4XoYuIp+aOxH9M=; b=Zfthgo08gYsxAsd0kpAtFnk2HevPsdIZ1Uyl/Ss592FYwJC1JPOtHjfU95SWsC8i7I qMdCMyusHedOVzJCm8+qF6SJHSkvnI1xCbxfLEQzoRRu/Voex/karmXJ2xynVWC8VkLs W1ZUZ33X75Pe5TXq2hpH+ZH1PWPDZN3w4wPU3v4NGWGfHy9pUAeHHrDOOD8/Vms8GnVq K5hEh0Td/o1BoxzRvRfpS9ahA0vJhvcTbc2VHu1O6yKaC5rRdfruz5CJ+OQ/J1ia7JAl SDsry16uoSSLC9xcRz8psCIor3Uh3ETN5wYw7poo0/7jzN8osplOHAWxH6hbZMQiGaR9 riwg== X-Gm-Message-State: AA+aEWZs3C2J1TiwG7/bv0aOEcC21FzbpBCdUawOK2o9/olYyaL/agDE xW4zKeTdaYFWcvxYhDYfLNCjQq4kz/8ol7pbZlQq6pe9 X-Google-Smtp-Source: AFSGD/WDn/AtuyQYkEJ66qJ7K2qNsLp59Yym6CUScHXX+bOfhT2Qu95YjONi2WiA03QG3Ok8CxdGJ3LGzt2IdIrkMUw= X-Received: by 2002:a6b:7809:: with SMTP id j9-v6mr9183689iom.299.1542928822690; Thu, 22 Nov 2018 15:20:22 -0800 (PST) MIME-Version: 1.0 References: <7699de57-d903-1d61-ee42-062ed312b20d@grosbein.net> In-Reply-To: <7699de57-d903-1d61-ee42-062ed312b20d@grosbein.net> From: Warner Losh Date: Thu, 22 Nov 2018 16:20:10 -0700 Message-ID: Subject: Re: TRIM utility To: Eugene Grosbein Cc: FreeBSD Hackers X-Rspamd-Queue-Id: C90206D43B X-Spamd-Result: default: False [-5.52 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_SHORT(-0.96)[-0.963,0]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[cached: ALT1.aspmx.l.google.com]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[5.3.d.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; RCVD_TLS_LAST(0.00)[]; IP_SCORE(-2.55)[ip: (-9.05), ipnet: 2607:f8b0::/32(-2.10), asn: 15169(-1.51), country: US(-0.09)]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Server: mx1.freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Nov 2018 23:20:24 -0000 On Thu, Nov 22, 2018, 1:36 PM Eugene Grosbein Hi! > > I found we have no utility capable to perform TRIM for the whole SSD device > or arbitrary part of it, so I wrote simple one. I can't think of nice name > for it, so proposal are welcome! Draft name is "erase". > > I ask for pre-commit code review, too. > The code is tested with TRIM-capable SSD and non-capable other devices. > > Currently it has four options, all of them are, hmm, optional: > > -b: to specify offset from the beginning of the device for trimmed region > instead of default 0; > -l: to specify offset from the "-b" margin - length - for trimmed region > instead of whole device; > -r rfile: for alternative way to specify length as length of referenced > file; > This seems really obscure and would be better handled by a stat command. -v: for verbose mode that shows final values for the beginning offset and > length. > > Later options override previous ones. Then it expects a list of device > names as arguments: > > erase ada0 > erase /dev/ada0s1 > erase -b 4096 -r file.img -v /dev/da0 /dev/da1 /dev/da2 > > The code: > > http://www.grosbein.net/freebsd/erase.c > http://www.grosbein.net/freebsd/Makefile.erase > > My "mandoc" skills are very poor and English is not my native language, > so any help with manual page creation will be appreciated. > > Eugene Grosbein > > P.S. I realized that our kernel-level TRIM support has no connection to > cam(4) nor to geom(4), > so distinct utility instead of addition to camcontrol(8) or geom(8). > "erase" is a really bad name. It's fraught with too many overloaded meanings. "trim" is likely the least bad name we can use. "delete" is similarly bad, even though that name is in the ioctl and in BIO_DELETE. "unmap" is the SCSI term, so it wouldn't be terrible, though the term is less widely used than "trim" is for this stuff. None of the other command names for more obscure technologies are general enough or widely enough known. "discard" is the Linux mount option, which seems less descriptive. Linux has a fstrim command, which does something kinda similar (it's a lot like fsck -E to erase unused parts of the filesystem), so there is some overlap. I couldn't find a dedicated command to do that, but if it does, we should follow that convention to reimplement. It feels to me to be a dd conv= option, but that isn't reflected in any implementation I could find, so that suggests I'm nuts (though sparse is a weak match). Warner > From owner-freebsd-hackers@freebsd.org Fri Nov 23 01:20:34 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D01DF113685B; Fri, 23 Nov 2018 01:20:33 +0000 (UTC) (envelope-from jhs@berklix.com) Received: from land.berklix.org (land.berklix.org [144.76.10.75]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "land.berklix.org", Issuer "land.berklix.org" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id D92BB706A7; Fri, 23 Nov 2018 01:20:31 +0000 (UTC) (envelope-from jhs@berklix.com) Received: from mart.js.berklix.net (p2E52CE62.dip0.t-ipconnect.de [46.82.206.98]) (authenticated bits=0) by land.berklix.org (8.15.2/8.15.2) with ESMTPSA id wAN1HgT1047482 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 23 Nov 2018 01:17:53 GMT (envelope-from jhs@berklix.com) Received: from fire.js.berklix.net (fire.js.berklix.net [192.168.91.41]) by mart.js.berklix.net (8.14.3/8.14.3) with ESMTP id wAN1Hcks051877; Fri, 23 Nov 2018 02:17:38 +0100 (CET) (envelope-from jhs@berklix.com) Received: from fire.js.berklix.net (localhost [127.0.0.1]) by fire.js.berklix.net (8.14.7/8.14.7) with ESMTP id wAN1HKAT037185; Fri, 23 Nov 2018 02:17:32 +0100 (CET) (envelope-from jhs@berklix.com) Message-Id: <201811230117.wAN1HKAT037185@fire.js.berklix.net> To: soralx@cydem.org cc: freebsd-hackers@freebsd.org, freebsd-fs@freebsd.org Subject: Re: [bug] fsck refuses to repair damaged UFS using backup superblock From: "Julian H. Stacey" Organization: http://berklix.eu BSD Unix Linux Consultants, Munich Germany User-agent: EXMH on FreeBSD http://berklix.eu/free/ X-From: http://www.berklix.eu/~jhs/ In-reply-to: Your message "Tue, 20 Nov 2018 05:30:00 -0800." <20181120053000.56fbee6b@mscad14> Date: Fri, 23 Nov 2018 02:17:20 +0100 X-Rspamd-Queue-Id: D92BB706A7 X-Spamd-Result: default: False [-2.39 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-0.98)[-0.978,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-0.95)[-0.953,0]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; DMARC_NA(0.00)[berklix.com]; AUTH_NA(1.00)[]; HAS_ORG_HEADER(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[land.berklix.com,slim.berklix.com]; NEURAL_HAM_SHORT(-0.88)[-0.882,0]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[98.206.82.46.zen.spamhaus.org : 127.0.0.10]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:24940, ipnet:144.76.0.0/16, country:DE]; RCVD_TLS_LAST(0.00)[]; IP_SCORE(-0.47)[ipnet: 144.76.0.0/16(0.56), asn: 24940(-2.91), country: DE(-0.01)] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Nov 2018 01:20:34 -0000 Hi soralx@cydem.org, Added cc: to ensure file system specialists see this. Reference: > From: > Date: Tue, 20 Nov 2018 05:30:00 -0800 soralx@cydem.org wrote: > > Howdy! > > Since send-pr(1) is now gone, I guess the next option is to send a > message directly to the developers... > > Yesterday, I ran into a bug in fsck_ffs that gave me a little scare. > > Short story: on -CURRENT, fsck refuses to check a FS with a corrupted > superblock, even when an alternate (backup) SB location is given. > > Long story. I've been testing a newly-built system based on an X399 > platform with a 2950X CPU and an Optane 905P 480GB U.2 drive. The > system ran a ~2-day old -CURRENT; when compiling newest world and > kernel, I found the machine in a locked-up state. After a hard reset, > boot failed because the root FS became corrupted & was not available: > kernel: Superblock check-hash failed: recorded check-hash XXX != computed check-hash YYY > > I have not yet figured out why the corruption happened... bad hardware? > bug in the NVMe driver? > > "OK", I thought, "No worries. We'll just boot using another disk, fsck > the corrupted FS with a backup superblock, and be up in a moment". > The machine was doing nothing but compiling, so no valuable data loss. > > So I did `dumpfs -m /dev/ada0p3` on the spare disk (which was the > source for the new disk image, thus had almost identical partitions > and filesystems) to get the FS details, then did `newfs -N [...] > /dev/ada0p3` to find locations of superblock backups, then finally > ran `fsck_ffs -b 192 /dev/nvd0p3` -- only to get the same "check- > -hash failed" message, plus another strange message: "Can't open > /dev/nvd0p3: [...]". Then fsck quits. > Note that `fsck_ffs -b ...` on a FS with good superblock works OK. > > After fiddling with a debugger for a bit, I commented out the line > "return (0);" in /usr/src/sbin/fsck_ffs/setup.c:136, recompiled fsck, > and the FS was recovered successfully. > > What was actually happening: fsck's setup.c calls ufs_disk_fillout() > from libufs' type.c, which in turn calls sbread() from the same > library, which then calls sbget(disk->d_fd, &fs, -1) [[where '-1' > is hard-coded to indicate the primary superblock]] that then simply > invokes ffs_sbget from ffs kernel driver -- and this returns ENOENT, > which eventually causes fsck to give up before even looking at the > specified backup superblock. > > I don't know what exactly ufs_disk_fillout() does, but fortunately > for me, fsck worked without the "sbread(disk)" part of that function > having much luck on a disk with corrupted superblock. Also, I have a > feeling that calling a kernel's ffs driver function when using fsck > to fix a broken filesystem is not the best thing to do... > > Please CC, as I am not subscribed. > > -- > [SorAlx] ridin' VN2000 Classic LT > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > Cheers, Julian -- Julian Stacey, Computer Consultant, Systems Engineer, BSD Linux Unix, Munich. Brexit referendum stole 3,700,000 votes from Brits abroad, inc. 700,000 in EU UK PM lied it's democratic in Article 50 http://exitbrexit.uk/brexit/#lie Campaign lies, criminal funded; Markets, jobs & pound down; New Referendum! From owner-freebsd-hackers@freebsd.org Fri Nov 23 03:02:02 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 708C411385EC for ; Fri, 23 Nov 2018 03:02:02 +0000 (UTC) (envelope-from khanzf@gmail.com) Received: from mail-io1-xd2e.google.com (mail-io1-xd2e.google.com [IPv6:2607:f8b0:4864:20::d2e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 61F4072B8F for ; Fri, 23 Nov 2018 03:01:37 +0000 (UTC) (envelope-from khanzf@gmail.com) Received: by mail-io1-xd2e.google.com with SMTP id m19so7821299ioh.3 for ; Thu, 22 Nov 2018 19:01:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=5805ECcuEwKcCgjBwm4GT9ks6ngXUWCI7oAcYMBg7Mg=; b=JMqdGCQ1J7jMC9B3iO1rhSFK0QxOBtp0b1mwK9Tb4m5wkoV9BUHWmaQVQoScSJJQJ4 45BqMaQiHY+LnYUHeAnQjgFq0AkgogaiCAkT3fzk7BxTc/iGjJmj2LH9xiQ8NKGzW01s c7TQmFZm+tQtqmLTW9Xig1lqnHY4v5wcdPXvwlV1O3hBEDid2Yrl8pZSKO0nfoiJKoeI zCxAxupH7v1/D98PIPUoumfK2l9+uR/fvEAUISB8usr/QEduCticz+NTM5YRZuUCX1gt +k9ME+SqPYdP7M18j/3pnkT+InynXDx+RG/essBk7ApA4L1EJ1r3W42esyBCvEwZLHGz rSJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=5805ECcuEwKcCgjBwm4GT9ks6ngXUWCI7oAcYMBg7Mg=; b=FdAk8PFsSKmou9M52M4JvMbbIlhadm1XVAkj1F3SRjOtSPTrBDc8JTr8v1G9ta21RA mDMETCLAGesIFZhUxaNy8DsMP5X3g7A/fJNi0G2qo53R+txqVu2fLvYp06OdMgGR1THn MNtqG6UwlTxRRTVbyUPdzYZupaWTezpjWnA8MCSj4qop+gue850bAa8ii3SnEGwgz2/N 237uUxpsLEqBHpqzOGhw8EeDH6KTTuLR9H2f3XbKW67p4QO4P2a5kKBhIVm/1JbG1OoB mxF4gZcP3XwxDiPIGhtQ3+vVdckNf2iAifx3usMsoiGmR+ZMkVahdgo3xKfk8yn7IeNS dVzA== X-Gm-Message-State: AA+aEWa3wOJXzllcSqLQ8Ud4csUjkRcY2+U4jEuorqPHez/anwsA75SU 4tfMBBCePDf5Gx4k8rnahFtS8fjtZgO2XCrnPtWg5g== X-Google-Smtp-Source: AFSGD/VWy55NgvmBXY5c3GbbCdFFipl1po60HOhXYLFl7IL84KMh+Nqxd5Dj5Rm+Tmekq8x5+tD3943SuPtGG1gCOPI= X-Received: by 2002:a5d:8497:: with SMTP id t23mr2802167iom.11.1542942096474; Thu, 22 Nov 2018 19:01:36 -0800 (PST) MIME-Version: 1.0 From: Farhan Khan Date: Thu, 22 Nov 2018 22:01:18 -0500 Message-ID: Subject: fetch(3) getting HTTP headers? To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 61F4072B8F X-Spamd-Result: default: False [-4.65 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[e.2.d.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(-1.65)[ip: (-4.56), ipnet: 2607:f8b0::/32(-2.09), asn: 15169(-1.51), country: US(-0.09)]; NEURAL_HAM_SHORT(-0.99)[-0.993,0]; FROM_EQ_ENVFROM(0.00)[]; SUBJECT_ENDS_QUESTION(1.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Nov 2018 03:02:02 -0000 Hi all, It seems that fetch(3) does not have an interface to provide the HTTP headers from a request. If not, would this be a worthwhile library to expand to add that ability? Also, it strikes me as odd that fetch(3) uses environmental variables to set header values, rather than through a function argument. I ask because I am working on a project that uses the HTTP library and would prefer to use the native FreeBSD libraries before expanding to a 3rd party library. Thoughts? -- Farhan Khan PGP Fingerprint: B28D 2726 E2BC A97E 3854 5ABE 9A9F 00BC D525 16EE From owner-freebsd-hackers@freebsd.org Fri Nov 23 06:59:58 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2F5CD113D17E for ; Fri, 23 Nov 2018 06:59:58 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [IPv6:2a01:4f8:d12:604::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 24978795A7 for ; Fri, 23 Nov 2018 06:59:46 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13:0:0:0:5]) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id wAN6xX39012927 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 23 Nov 2018 07:59:34 +0100 (CET) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: imp@bsdimp.com Received: from [10.58.0.4] (dadv@[10.58.0.4]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id wAN6xWqk078040 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Fri, 23 Nov 2018 13:59:32 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: TRIM utility To: Warner Losh References: <7699de57-d903-1d61-ee42-062ed312b20d@grosbein.net> Cc: FreeBSD Hackers From: Eugene Grosbein Message-ID: <6975ff4e-6383-e52c-3a11-d35b95cca114@grosbein.net> Date: Fri, 23 Nov 2018 13:59:31 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=0.3 required=5.0 tests=BAYES_00,LOCAL_FROM,SPF_PASS autolearn=no autolearn_force=no version=3.4.2 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * -0.0 SPF_PASS SPF: sender matches SPF record * 2.6 LOCAL_FROM From my domains X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on hz.grosbein.net X-Rspamd-Queue-Id: 24978795A7 X-Spamd-Result: default: False [-5.17 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; MX_INVALID(0.50)[greylisted]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[grosbein.net]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; R_SPF_PERMFAIL(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-0.97)[-0.973,0]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; IP_SCORE(-2.60)[ip: (-6.60), ipnet: 2a01:4f8::/29(-3.48), asn: 24940(-2.90), country: DE(-0.01)]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Nov 2018 06:59:58 -0000 23.11.2018 6:20, Warner Losh wrote: >> I found we have no utility capable to perform TRIM for the whole SSD device >> or arbitrary part of it, so I wrote simple one. I can't think of nice name >> for it, so proposal are welcome! Draft name is "erase". >> >> I ask for pre-commit code review, too. >> The code is tested with TRIM-capable SSD and non-capable other devices. >> >> Currently it has four options, all of them are, hmm, optional: >> >> -b: to specify offset from the beginning of the device for trimmed region >> instead of default 0; >> -l: to specify offset from the "-b" margin - length - for trimmed region >> instead of whole device; >> -r rfile: for alternative way to specify length as length of referenced >> file; >> > > This seems really obscure and would be better handled by a stat command. This is inspired by truncate(1) having same option that saves extra call to stat. Forgot to note, that options -b and -l allow suffixes [K|k|M|m|G|g|T|t] just like truncate's option -s does. > "erase" is a really bad name. It's fraught with too many overloaded > meanings. "trim" is likely the least bad name we can use. I'm fine with this, renamed. > Linux has a fstrim command, which does something kinda similar (it's a lot > like fsck -E to erase unused parts of the filesystem), so there is some > overlap. I couldn't find a dedicated command to do that, but if it does, we > should follow that convention to reimplement. Well, they have http://man7.org/linux/man-pages/man8/blkdiscard.8.html I don't like the name, though. It's too complicated to pronounce and too long same time. I can rename -b option to -o to match blkdiscard's, if this matters. From owner-freebsd-hackers@freebsd.org Fri Nov 23 08:34:32 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 222F8113F2BF for ; Fri, 23 Nov 2018 08:34:32 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: from puchar.net (puchar.net [194.1.144.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "puchar.net", Issuer "puchar.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 2DE307C4D0 for ; Fri, 23 Nov 2018 08:34:30 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.15.2/8.15.2) with ESMTPS id wAN8YSYW039706 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 23 Nov 2018 09:34:28 +0100 (CET) (envelope-from puchar-wojtek@puchar.net) Received: from localhost (puchar-wojtek@localhost) by puchar.net (8.15.2/8.15.2/Submit) with ESMTP id wAN8YNBZ039703; Fri, 23 Nov 2018 09:34:23 +0100 (CET) (envelope-from puchar-wojtek@puchar.net) Date: Fri, 23 Nov 2018 09:34:23 +0100 (CET) From: Wojciech Puchar To: Eugene Grosbein cc: Freebsd hackers list Subject: Re: TRIM utility In-Reply-To: <7699de57-d903-1d61-ee42-062ed312b20d@grosbein.net> Message-ID: References: <7699de57-d903-1d61-ee42-062ed312b20d@grosbein.net> User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Rspamd-Queue-Id: 2DE307C4D0 X-Spamd-Result: default: False [-1.95 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.66)[-0.659,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-0.33)[-0.325,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[puchar.net]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[puchar.net]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[90.144.1.194.list.dnswl.org : 127.0.10.0]; NEURAL_HAM_SHORT(-0.66)[-0.663,0]; IP_SCORE(0.00)[country: PL(0.02)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:43476, ipnet:194.1.144.0/24, country:PL]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Nov 2018 08:34:32 -0000 thanks. very useful tool On Fri, 23 Nov 2018, Eugene Grosbein wrote: > Hi! > > I found we have no utility capable to perform TRIM for the whole SSD device > or arbitrary part of it, so I wrote simple one. I can't think of nice name > for it, so proposal are welcome! Draft name is "erase". > > I ask for pre-commit code review, too. > The code is tested with TRIM-capable SSD and non-capable other devices. > > Currently it has four options, all of them are, hmm, optional: > > -b: to specify offset from the beginning of the device for trimmed region instead of default 0; > -l: to specify offset from the "-b" margin - length - for trimmed region instead of whole device; > -r rfile: for alternative way to specify length as length of referenced file; > -v: for verbose mode that shows final values for the beginning offset and length. > > Later options override previous ones. Then it expects a list of device names as arguments: > > erase ada0 > erase /dev/ada0s1 > erase -b 4096 -r file.img -v /dev/da0 /dev/da1 /dev/da2 > > The code: > > http://www.grosbein.net/freebsd/erase.c > http://www.grosbein.net/freebsd/Makefile.erase > > My "mandoc" skills are very poor and English is not my native language, > so any help with manual page creation will be appreciated. > > Eugene Grosbein > > P.S. I realized that our kernel-level TRIM support has no connection to cam(4) nor to geom(4), > so distinct utility instead of addition to camcontrol(8) or geom(8). > > From owner-freebsd-hackers@freebsd.org Fri Nov 23 10:59:51 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AC6331142E31 for ; Fri, 23 Nov 2018 10:59:51 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [IPv6:2a01:4f8:d12:604::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id CFC9583702 for ; Fri, 23 Nov 2018 10:59:40 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13:0:0:0:5]) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id wANAxSvR014893 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 23 Nov 2018 11:59:29 +0100 (CET) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: lars.engels@0x20.net Received: from eg.sd.rdtc.ru (eugen@localhost [127.0.0.1]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTP id wANAxRV1081797; Fri, 23 Nov 2018 17:59:27 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: TRIM utility To: Lars Engels References: <7699de57-d903-1d61-ee42-062ed312b20d@grosbein.net> <6975ff4e-6383-e52c-3a11-d35b95cca114@grosbein.net> <20181123104205.GZ59358@e.0x20.net> Cc: Warner Losh , FreeBSD Hackers From: Eugene Grosbein Message-ID: <5BF7DD8F.1040007@grosbein.net> Date: Fri, 23 Nov 2018 17:59:27 +0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.4.0 MIME-Version: 1.0 In-Reply-To: <20181123104205.GZ59358@e.0x20.net> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=3.3 required=5.0 tests=BAYES_00, DATE_IN_FUTURE_96_Q, LOCAL_FROM, SPF_PASS autolearn=no autolearn_force=no version=3.4.2 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 3.0 DATE_IN_FUTURE_96_Q Date: is 4 days to 4 months after Received: * date * -0.0 SPF_PASS SPF: sender matches SPF record * 2.6 LOCAL_FROM From my domains X-Spam-Level: *** X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on hz.grosbein.net X-Rspamd-Queue-Id: CFC9583702 X-Spamd-Result: default: False [-5.13 / 15.00]; MX_INVALID(0.50)[greylisted]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; R_SPF_PERMFAIL(0.00)[]; NEURAL_HAM_SHORT(-0.97)[-0.973,0]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[grosbein.net]; TO_MATCH_ENVRCPT_SOME(0.00)[]; IP_SCORE(-2.56)[ip: (-6.42), ipnet: 2a01:4f8::/29(-3.48), asn: 24940(-2.90), country: DE(-0.02)] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Nov 2018 10:59:51 -0000 On 23.11.2018 17:42, Lars Engels wrote: > Would "hdtrim" or "ssdtrim" be better names? Just "trim" sounds better to me :-) From owner-freebsd-hackers@freebsd.org Fri Nov 23 11:12:57 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EE03511432CB for ; Fri, 23 Nov 2018 11:12:56 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.not-for.work (onlyone.not-for.work [IPv6:2a01:4f8:201:6350::2]) by mx1.freebsd.org (Postfix) with ESMTP id 5FBC783E96 for ; Fri, 23 Nov 2018 11:12:56 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from [192.168.23.186] (unknown [89.113.128.32]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.not-for.work (Postfix) with ESMTPSA id EBF681F42; Fri, 23 Nov 2018 14:12:54 +0300 (MSK) Reply-To: lev@FreeBSD.org Subject: Re: TRIM utility To: Eugene Grosbein , Freebsd hackers list References: <7699de57-d903-1d61-ee42-062ed312b20d@grosbein.net> From: Lev Serebryakov Openpgp: preference=signencrypt Autocrypt: addr=lev@FreeBSD.org; prefer-encrypt=mutual; keydata= xsFNBFKbGksBEADeguVs+XyJc3mL3iiOBqDd16wSk97YTJYOi4VsHsINzJr09oFvNDiaDBIi fLn2p8XcJvehcsF2GSgrfXfw+uK4O1jyNIKJmiYA0EtE+ZbRtvDrrE0w6Q8+SDeKA21SWh3Y vSQ0DJUontbgW55ER2CbEiIUTIn34uQ0kmESAaw/v5p/9ue8yPTmURvv130FqPFz8VPzltqL NxyGt54TxPfKAzAHEIwxlEZ63JOwzloKh1UDBExcsf9nJO08/TAVgR5UZ5njFBPzaaquhRoP qPJLEQQDqxPIlvMNtHKf7iIebE4BHeqgCdJA0BoiR6gpa0wlsZtdrTPK3n4wYSphLvGbhfOZ YW/hbcu7HYS/FImkVxB3iY17kcC1UTnx4ZaYeASPBGOOPbXky1lLfmDGWIFT//70yx+G17qD OZzF1SvJJhGvh6ilFYaWMX7T+nIp6Mcafc4D7AakXM+XdubNXOMlCJhzPcZ0skgAEnYV587w V7em5fDVwQccwvtfezzqKeJAU5TGiywBHSR5Svzk2FwRNf6M//hWkpq0SRR63iOhkHGOAEBi 69GfEIwH2/w24rLxP0E+Hqq8n+EWNkPatw1Mhcl5PKkdvGCjJUaGNMkpBffjyYo254JXRscR eEnwdIkJt4ErDvjb2/UrOFq31wWMOiLzJeVchAgvTHBMRfP9aQARAQABzShMZXYgU2VyZWJy eWFrb3YgPGxldkBzZXJlYnJ5YWtvdi5zcGIucnU+wsGCBBMBCAAsAhsDBwsJCAcDAgEGFQgC CQoLBBYCAwECHgECF4ACGQEFAlKbP8wFCQlmJwEACgkQ6rA8WL/cR4/6VBAAjRMyyX3PBFx/ HxyiIZ698EfwlWUua8Ft4crtrdK52m0qNkbBB9BH8xQgBHG32A1CwyzQnzxHgZuoOWMjh+Qq WJv7dmpM/q/c1GCJHhlPgewXrciTwpAamZILN071u+1GCPWwGRPzfQ/U+k63KJWx9ozf4doM WTTom6Cqcssi4J1u5kkt52a5ZRhsCK9pEVGilk36XTP9BakGrnMSIxF/NK4xeZVX2q+Nuqvf RchyofKXVgLEDLwb1cd/baLtBpDzy0PTN2Zl2lX4kOA6jwTKsqRya9A1Vui1KXwPh2XViTQ1 7Y3l5qg/M+sR73DohezP6bO6huOnLhty17jAqHPNlD6RonDo+j8uIlEg4iMSTN3MhzkBAu0Q pe3ucQ0o1767JiXN3fsNvRzSFhLVNDqPLce4uKlMogsbreXWvdgHGTN1ybOHGbybZnP77yHz uNBacbmG3vL/OLXMqwLdL2JXoiec4DmXjjCdhTBl5xLV9Hz/6VWKqElteg8QFVvHB3tHWzJ4 /rpiVEixytCIII6DS33BXZ0h2EOkK/6AYA2SJxy1vgOH4SZBtDBHoezmHV2nFnq5O0c7AuAB 7WPWgQG0sEwHQPZmg/baRGitRJnaxf/Gvf1DeD1x1VrcoVke2vwBcgDM3kugP8L9hsqic2D3 dI+gP76haeuvNNZr3y9L9zvOwU0EUpsaSwEQALRr3B+OjY/cnJPstz5CVsVWyEZtJtrNviZr tBgbkhlkPm98sEWR4+gbpyeufdYJengDjeGzMDKcLB7h5fICS/j6A8XdlJ40TlbPfNgb6OHa ebaIYKTJpXKR9sD7ZyGivYMofm0em40wGUX7BIkdkomaWj+wUiS0CdXU0FWDj9wv73+Eim+X zZyXeFgIPv97v+pET7DfwKkADOfrkW9s4OfvGVjd+wm35wc8EngQEz0qdPBxx74X7vZFAxlA SXu8gDBJGYt2Bkc3QwULnfeXrZJWgqNPR5o44gGu96yaiOFaN/C6CJtev5ZEX+0ZxbvsHHB7 Z5AtsRURKpZ4w5HFHGhzHtDtoAKgeZ/gbhTVXPHvNQR818eN+Nl5BV8BRF/8yhR6VlJb8GYw h8oKDeVGVYC34+raHZQAM9WoBnN7jlt4T9zzPwtmw5mIahGFgvw1KDr7OItN2ZgtZ20UYC5m Go602nmHq0aPbU6SwGi1xohrliNsKaaciYiMaVIGRQq8iGr9Fe2HlvaA3BpB275i/gCVlUdG y5XLAv+yQMUvn5Z7XVsMroxDk/O+ae1ElyBvKiKyfWGJXTg5XUukkkyQmfWPxWUGoNA1P/P4 GMHSu7/Rqe/7m4uPu/RyTTqsSjjKJdP9kBwEzvqPtXsVoZuShtrptRQJDYflhgE4qmKSMKen ABEBAAHCwWUEGAECAA8FAlKbGksCGwwFCRLMAwAACgkQ6rA8WL/cR48RkA//SNzeW3CI8KHx rA0aeHW6Nb5ieoqVRBGLyjBM06RX6vHB9v4dJL6Z+yV2jGN2s+XZX2HILbuTOwcTxGkI3xTT e0cDXVaF5K8R/liigUjtwuC2v/sWgoWyUmK1Cy9CPYdcXmFq6nESfkUe8DYiGOUULdHq5w63 F53yOZ72iXRBQBZgkhPtRFu4lPYIzOsMag9DIJ9CthR1r0ziqU/keb94Qt3l+aXK7CwGdY7X T4zUIMHNYsuAuyX+NJIXfsN68TT6m7QmlUwxPs13nxmoVQzm4ruV+hlQKh1MtbsjWRkNgPxF IPiqoAEhy8QoddlSvRTwL5Z7zFQiwMdiXU7toL8pfzj/zJR1jELXKMipijrt5MLrV8XX3OPN yZZvh95VIl8mv+iAqwSZUufd2EJnvj5TObB0eH+a+34NWf/XqA3fPjE6KHzmdnw9PZjPEjlx JCPECSs+6gse1+GaEfKYuXzB/ENe2ctlcfx5iQJXFc+/+zG/uU/JX/pXJHA12CUfB5g7lH6X BZIHvRo3VTCDjXgbF5xxDAe5V4exf8d4oSNjQIFLYxxN7zkvH89EN6RPfRgsWN7bYArCwfS9 MOgs9pFeCOewR6qieK150aoqNENGfKFXJup+5VVl6I0mU+j0rgVDZDht2/QgP/Tb4lGBe+ai pOGaK/GYNR+Ad6bUmokKsx4= Organization: FreeBSD Message-ID: Date: Fri, 23 Nov 2018 14:12:47 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.3.1 MIME-Version: 1.0 In-Reply-To: <7699de57-d903-1d61-ee42-062ed312b20d@grosbein.net> Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="na8lnN3rArlTlA1zSOCI88C75zPqM9GqR" X-Rspamd-Queue-Id: 5FBC783E96 X-Spamd-Result: default: False [2.13 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_SPAM_SHORT(0.93)[0.928,0]; NEURAL_SPAM_MEDIUM(0.87)[0.875,0]; NEURAL_SPAM_LONG(0.33)[0.325,0]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Nov 2018 11:12:57 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --na8lnN3rArlTlA1zSOCI88C75zPqM9GqR Content-Type: multipart/mixed; boundary="PX2k43ITM4H0N6Jtgjx0x2kjTYVWXAaW9"; protected-headers="v1" From: Lev Serebryakov Reply-To: lev@FreeBSD.org To: Eugene Grosbein , Freebsd hackers list Message-ID: Subject: Re: TRIM utility References: <7699de57-d903-1d61-ee42-062ed312b20d@grosbein.net> In-Reply-To: <7699de57-d903-1d61-ee42-062ed312b20d@grosbein.net> --PX2k43ITM4H0N6Jtgjx0x2kjTYVWXAaW9 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 22.11.2018 23:34, Eugene Grosbein wrote: > Currently it has four options, all of them are, hmm, optional: >=20 > -b: to specify offset from the beginning of the device for trimmed regi= on instead of default 0; > -l: to specify offset from the "-b" margin - length - for trimmed regio= n instead of whole device; > -r rfile: for alternative way to specify length as length of referenced= file; > -v: for verbose mode that shows final values for the beginning offset a= nd length. Please, add '-n' for dry-run, which could be useful together with '-v'. --=20 // Lev Serebryakov --PX2k43ITM4H0N6Jtgjx0x2kjTYVWXAaW9-- --na8lnN3rArlTlA1zSOCI88C75zPqM9GqR Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEE+W0coLX0MYtnSzMK6rA8WL/cR48FAlv34LBfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEY5 NkQxQ0EwQjVGNDMxOEI2NzRCMzMwQUVBQjAzQzU4QkZEQzQ3OEYACgkQ6rA8WL/c R4/kyw/+I1G6Jn+LsMRTlajQah5J59bSWki2yjPGzCt7PC5W30AQ6tQrwycp96DA VUvAFX0mtCOMXTlJGLrYr+YLYgQhhj89Qk0tEl/D/N+Z+HAakltOIu04e5rvuR5E xh3/tJE8Fuq8HwQ4rwRS20J/y3nGjpLaGA4jNAAR+4pv5nuk6xdoxk+r/eTahnJp lMORY+LLBnQdH2oEzN1y2tGheGysiPwA36A+JHThZhXzmee+t2otsYTZrGIyieaW dlEE5Pdugp2tIuTSe57mWb6NifBgdptyp9PvbrZPMeq3wlP9qNaj1cm8dZBsige6 CETXgPYc28hVWF6I2+Hj6stQs0mcpp3zb0a5bChm2wfpGEmoKMmJ7ZULMvSLqX3z v4jYiu/aPqt1aIx6JlRajOIRRMo/S+EdC+ozpRs47gqBueITkId/mirthtzzRtm3 ey8czl91o8YSU3IyO99Fcxwl6dnT52v2jai3eq2Lrhjzc6N6et/08Pgq4wmZpJSy p1vkHecPBTewBUr8/5e6jR+IM8ksDl4evgWkxlXSiLNJYp+1o++tbxSTTqdQ7dNi p+V/v3jLtlFfFePQDYhe1cX3gK8cyDvigCLoWTQUVbD/07Pkhu1VQW5rVeQ9CJgM C0k9Zc0sBYlXO3n87VO5pMWAA5mTrAOE3NmmxQn+DT83gjITqBc= =JFd5 -----END PGP SIGNATURE----- --na8lnN3rArlTlA1zSOCI88C75zPqM9GqR-- From owner-freebsd-hackers@freebsd.org Fri Nov 23 11:19:21 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DBB4211434FF for ; Fri, 23 Nov 2018 11:19:21 +0000 (UTC) (envelope-from phk@critter.freebsd.dk) Received: from phk.freebsd.dk (phk.freebsd.dk [130.225.244.222]) by mx1.freebsd.org (Postfix) with ESMTP id 9EECD84117; Fri, 23 Nov 2018 11:19:20 +0000 (UTC) (envelope-from phk@critter.freebsd.dk) Received: from critter.freebsd.dk (unknown [192.168.55.3]) by phk.freebsd.dk (Postfix) with ESMTP id 9EEBC14856; Fri, 23 Nov 2018 11:19:13 +0000 (UTC) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.15.2/8.15.2) with ESMTPS id wANBJCTs003878 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 23 Nov 2018 11:19:12 GMT (envelope-from phk@critter.freebsd.dk) Received: (from phk@localhost) by critter.freebsd.dk (8.15.2/8.15.2/Submit) id wANBJBtr003877; Fri, 23 Nov 2018 11:19:11 GMT (envelope-from phk) To: lev@FreeBSD.org cc: Eugene Grosbein , Freebsd hackers list Subject: Re: TRIM utility In-reply-to: From: "Poul-Henning Kamp" References: <7699de57-d903-1d61-ee42-062ed312b20d@grosbein.net> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <3875.1542971951.1@critter.freebsd.dk> Content-Transfer-Encoding: quoted-printable Date: Fri, 23 Nov 2018 11:19:11 +0000 Message-ID: <3876.1542971951@critter.freebsd.dk> X-Rspamd-Queue-Id: 9EECD84117 X-Spamd-Result: default: False [4.39 / 15.00]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; NEURAL_SPAM_SHORT(0.90)[0.900,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[freebsd.dk]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(0.93)[0.928,0]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[phk.freebsd.dk]; NEURAL_SPAM_LONG(0.90)[0.899,0]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[phk@phk.freebsd.dk,phk@critter.freebsd.dk]; RCVD_NO_TLS_LAST(0.10)[]; R_DKIM_NA(0.00)[]; IP_SCORE(0.38)[asn: 1835(1.88), country: EU(0.01)]; ASN(0.00)[asn:1835, ipnet:130.225.0.0/16, country:EU]; FROM_NEQ_ENVFROM(0.00)[phk@phk.freebsd.dk,phk@critter.freebsd.dk] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Nov 2018 11:19:22 -0000 -------- In message , Lev Serebry= akov = writes: >> Currently it has four options, all of them are, hmm, optional: Isn't this the kind of thing that dd(1) should learn about instead ? -- = Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe = Never attribute to malice what can adequately be explained by incompetence= . From owner-freebsd-hackers@freebsd.org Fri Nov 23 11:34:36 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DDF191143BD4 for ; Fri, 23 Nov 2018 11:34:35 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.not-for.work (onlyone.not-for.work [IPv6:2a01:4f8:201:6350::2]) by mx1.freebsd.org (Postfix) with ESMTP id 442AE84BC4 for ; Fri, 23 Nov 2018 11:34:35 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from [192.168.23.186] (unknown [89.113.128.32]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.not-for.work (Postfix) with ESMTPSA id 8AA5B1F4A; Fri, 23 Nov 2018 14:34:34 +0300 (MSK) Reply-To: lev@FreeBSD.org Subject: Re: TRIM utility To: Poul-Henning Kamp Cc: Eugene Grosbein , Freebsd hackers list References: <7699de57-d903-1d61-ee42-062ed312b20d@grosbein.net> <3876.1542971951@critter.freebsd.dk> From: Lev Serebryakov Openpgp: preference=signencrypt Autocrypt: addr=lev@FreeBSD.org; prefer-encrypt=mutual; keydata= xsFNBFKbGksBEADeguVs+XyJc3mL3iiOBqDd16wSk97YTJYOi4VsHsINzJr09oFvNDiaDBIi fLn2p8XcJvehcsF2GSgrfXfw+uK4O1jyNIKJmiYA0EtE+ZbRtvDrrE0w6Q8+SDeKA21SWh3Y vSQ0DJUontbgW55ER2CbEiIUTIn34uQ0kmESAaw/v5p/9ue8yPTmURvv130FqPFz8VPzltqL NxyGt54TxPfKAzAHEIwxlEZ63JOwzloKh1UDBExcsf9nJO08/TAVgR5UZ5njFBPzaaquhRoP qPJLEQQDqxPIlvMNtHKf7iIebE4BHeqgCdJA0BoiR6gpa0wlsZtdrTPK3n4wYSphLvGbhfOZ YW/hbcu7HYS/FImkVxB3iY17kcC1UTnx4ZaYeASPBGOOPbXky1lLfmDGWIFT//70yx+G17qD OZzF1SvJJhGvh6ilFYaWMX7T+nIp6Mcafc4D7AakXM+XdubNXOMlCJhzPcZ0skgAEnYV587w V7em5fDVwQccwvtfezzqKeJAU5TGiywBHSR5Svzk2FwRNf6M//hWkpq0SRR63iOhkHGOAEBi 69GfEIwH2/w24rLxP0E+Hqq8n+EWNkPatw1Mhcl5PKkdvGCjJUaGNMkpBffjyYo254JXRscR eEnwdIkJt4ErDvjb2/UrOFq31wWMOiLzJeVchAgvTHBMRfP9aQARAQABzShMZXYgU2VyZWJy eWFrb3YgPGxldkBzZXJlYnJ5YWtvdi5zcGIucnU+wsGCBBMBCAAsAhsDBwsJCAcDAgEGFQgC CQoLBBYCAwECHgECF4ACGQEFAlKbP8wFCQlmJwEACgkQ6rA8WL/cR4/6VBAAjRMyyX3PBFx/ HxyiIZ698EfwlWUua8Ft4crtrdK52m0qNkbBB9BH8xQgBHG32A1CwyzQnzxHgZuoOWMjh+Qq WJv7dmpM/q/c1GCJHhlPgewXrciTwpAamZILN071u+1GCPWwGRPzfQ/U+k63KJWx9ozf4doM WTTom6Cqcssi4J1u5kkt52a5ZRhsCK9pEVGilk36XTP9BakGrnMSIxF/NK4xeZVX2q+Nuqvf RchyofKXVgLEDLwb1cd/baLtBpDzy0PTN2Zl2lX4kOA6jwTKsqRya9A1Vui1KXwPh2XViTQ1 7Y3l5qg/M+sR73DohezP6bO6huOnLhty17jAqHPNlD6RonDo+j8uIlEg4iMSTN3MhzkBAu0Q pe3ucQ0o1767JiXN3fsNvRzSFhLVNDqPLce4uKlMogsbreXWvdgHGTN1ybOHGbybZnP77yHz uNBacbmG3vL/OLXMqwLdL2JXoiec4DmXjjCdhTBl5xLV9Hz/6VWKqElteg8QFVvHB3tHWzJ4 /rpiVEixytCIII6DS33BXZ0h2EOkK/6AYA2SJxy1vgOH4SZBtDBHoezmHV2nFnq5O0c7AuAB 7WPWgQG0sEwHQPZmg/baRGitRJnaxf/Gvf1DeD1x1VrcoVke2vwBcgDM3kugP8L9hsqic2D3 dI+gP76haeuvNNZr3y9L9zvOwU0EUpsaSwEQALRr3B+OjY/cnJPstz5CVsVWyEZtJtrNviZr tBgbkhlkPm98sEWR4+gbpyeufdYJengDjeGzMDKcLB7h5fICS/j6A8XdlJ40TlbPfNgb6OHa ebaIYKTJpXKR9sD7ZyGivYMofm0em40wGUX7BIkdkomaWj+wUiS0CdXU0FWDj9wv73+Eim+X zZyXeFgIPv97v+pET7DfwKkADOfrkW9s4OfvGVjd+wm35wc8EngQEz0qdPBxx74X7vZFAxlA SXu8gDBJGYt2Bkc3QwULnfeXrZJWgqNPR5o44gGu96yaiOFaN/C6CJtev5ZEX+0ZxbvsHHB7 Z5AtsRURKpZ4w5HFHGhzHtDtoAKgeZ/gbhTVXPHvNQR818eN+Nl5BV8BRF/8yhR6VlJb8GYw h8oKDeVGVYC34+raHZQAM9WoBnN7jlt4T9zzPwtmw5mIahGFgvw1KDr7OItN2ZgtZ20UYC5m Go602nmHq0aPbU6SwGi1xohrliNsKaaciYiMaVIGRQq8iGr9Fe2HlvaA3BpB275i/gCVlUdG y5XLAv+yQMUvn5Z7XVsMroxDk/O+ae1ElyBvKiKyfWGJXTg5XUukkkyQmfWPxWUGoNA1P/P4 GMHSu7/Rqe/7m4uPu/RyTTqsSjjKJdP9kBwEzvqPtXsVoZuShtrptRQJDYflhgE4qmKSMKen ABEBAAHCwWUEGAECAA8FAlKbGksCGwwFCRLMAwAACgkQ6rA8WL/cR48RkA//SNzeW3CI8KHx rA0aeHW6Nb5ieoqVRBGLyjBM06RX6vHB9v4dJL6Z+yV2jGN2s+XZX2HILbuTOwcTxGkI3xTT e0cDXVaF5K8R/liigUjtwuC2v/sWgoWyUmK1Cy9CPYdcXmFq6nESfkUe8DYiGOUULdHq5w63 F53yOZ72iXRBQBZgkhPtRFu4lPYIzOsMag9DIJ9CthR1r0ziqU/keb94Qt3l+aXK7CwGdY7X T4zUIMHNYsuAuyX+NJIXfsN68TT6m7QmlUwxPs13nxmoVQzm4ruV+hlQKh1MtbsjWRkNgPxF IPiqoAEhy8QoddlSvRTwL5Z7zFQiwMdiXU7toL8pfzj/zJR1jELXKMipijrt5MLrV8XX3OPN yZZvh95VIl8mv+iAqwSZUufd2EJnvj5TObB0eH+a+34NWf/XqA3fPjE6KHzmdnw9PZjPEjlx JCPECSs+6gse1+GaEfKYuXzB/ENe2ctlcfx5iQJXFc+/+zG/uU/JX/pXJHA12CUfB5g7lH6X BZIHvRo3VTCDjXgbF5xxDAe5V4exf8d4oSNjQIFLYxxN7zkvH89EN6RPfRgsWN7bYArCwfS9 MOgs9pFeCOewR6qieK150aoqNENGfKFXJup+5VVl6I0mU+j0rgVDZDht2/QgP/Tb4lGBe+ai pOGaK/GYNR+Ad6bUmokKsx4= Organization: FreeBSD Message-ID: Date: Fri, 23 Nov 2018 14:34:33 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.3.1 MIME-Version: 1.0 In-Reply-To: <3876.1542971951@critter.freebsd.dk> Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="gdLJKbQRaGEmFG8Z4NVmFup3ClJlWTBVy" X-Rspamd-Queue-Id: 442AE84BC4 X-Spamd-Result: default: False [1.57 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_SPAM_LONG(0.23)[0.227,0]; NEURAL_SPAM_SHORT(0.51)[0.514,0]; NEURAL_SPAM_MEDIUM(0.83)[0.829,0]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Nov 2018 11:34:36 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --gdLJKbQRaGEmFG8Z4NVmFup3ClJlWTBVy Content-Type: multipart/mixed; boundary="AmD7cpDuwRB8GxyC9ZIfJguyryonsv3qo"; protected-headers="v1" From: Lev Serebryakov Reply-To: lev@FreeBSD.org To: Poul-Henning Kamp Cc: Eugene Grosbein , Freebsd hackers list Message-ID: Subject: Re: TRIM utility References: <7699de57-d903-1d61-ee42-062ed312b20d@grosbein.net> <3876.1542971951@critter.freebsd.dk> In-Reply-To: <3876.1542971951@critter.freebsd.dk> --AmD7cpDuwRB8GxyC9ZIfJguyryonsv3qo Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 23.11.2018 14:19, Poul-Henning Kamp wrote: >>> Currently it has four options, all of them are, hmm, optional: > Isn't this the kind of thing that dd(1) should learn about instead ? One utility to done one thing very well? :-) dd(1) is way overloaded, IMHO. --=20 // Lev Serebryakov --AmD7cpDuwRB8GxyC9ZIfJguyryonsv3qo-- --gdLJKbQRaGEmFG8Z4NVmFup3ClJlWTBVy Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEE+W0coLX0MYtnSzMK6rA8WL/cR48FAlv35cpfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEY5 NkQxQ0EwQjVGNDMxOEI2NzRCMzMwQUVBQjAzQzU4QkZEQzQ3OEYACgkQ6rA8WL/c R48EThAArw/0MlN9wA7V4Ng/Fx783bqtoK1P++gJz3bHAAgCRz/228wIPq5WkO8c yRF/Wd19lCc8MwV9J8x7iLgUZV1YhN7EZcrxOodEQ4iHzUwgtfeh9PAWBRflKSlB oa+ymVdAKh4R/g4jv+pt90jPK5TK6lpurf9v18gtZdFoHhYLPSH+IaW/hGgOky7w tiVbuqMexmwGk+WaGk3NDTy9L/Hs3USa5S6UlbZFf4mdteIlJ/SWVfjzxRnr4IzT RO0eqvaJ9N87so+bjOl01w2Dj57Dx/NEIaAQbBpE30jgHjNr0n16qF4M+3DPxMX+ wj19vyCte0JraP1gYv5dIA9KEuXDug95J9UNWwbUxKUl4DBuZW44fDgG0cN/UNCW 3mMNU9DJhm9fEnx98G4To6AmMISm7+KctcV0/JfaENlJRBg8qKjCzyU6oqgi0eiV fi0O0qrfYA2LhLLjMpb5mNxrcCOGFQOjHH+mZO9uPWH88e0HoYmQybr7/oJzdctD FsaliIDudD5ueYVuHHklMnyY7oBXoITGYO6m5BfiaX+yYCrFvhhO1f5aj2t/3En5 /7HPAZ5WYgLadN4R1Ay5EBqz214gmRsVfzXEvpNUVEJjIsvqncUSS3iWIwCvJkzj LnHhmqZNFRtqLfWFuK+8FL8c4aGtfum6kPAacUqyD4q95oewmqA= =u8Pn -----END PGP SIGNATURE----- --gdLJKbQRaGEmFG8Z4NVmFup3ClJlWTBVy-- From owner-freebsd-hackers@freebsd.org Fri Nov 23 11:46:21 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2F16E114448B for ; Fri, 23 Nov 2018 11:46:21 +0000 (UTC) (envelope-from gljennjohn@gmail.com) Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3BC00858FC; Fri, 23 Nov 2018 11:46:20 +0000 (UTC) (envelope-from gljennjohn@gmail.com) Received: by mail-wr1-x42d.google.com with SMTP id q18so12041110wrx.9; Fri, 23 Nov 2018 03:46:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:from:to:cc:subject:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=7EDQ0FJVu6xtkRmWm/4IkBgsxCzom5r37nLVE4blHx8=; b=K8EZeM2U8tXwzWd4C7bQ48bcN1EyeLRoVQS3aKoCdf7G39GJ/vLYdI4N5uZDABhAZY p3ZPrVPpfp2xWrJbVm9K3Mi4wE+7KjDm7FTAcgbhT7Bhjfa+Rd3549JlCoJ+hjpe0O2Y IWEWosffNwVXnN6voWJwx4NbFlQEsZifBaT/LyRAQEvC2h/3+cnmjzNoYvhI/S+6Zaqe aO2QNdQdjIaIFE6ar19nhTc6s4uAodt0nQSWlgZsZw5KEJns6Y7xNVLQufcHMZUkWVGD gQP/MAl+5+FlmbsIPGnlklr1BMtagj5F2zIXycekVWn/h1niKI4UZdlN90LMjmgKoK7t 0tmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:reply-to:mime-version:content-transfer-encoding; bh=7EDQ0FJVu6xtkRmWm/4IkBgsxCzom5r37nLVE4blHx8=; b=j7/2UjvcPJ7i6PldAfNtkz0llrfvF6/g2J5+W/aYJZczn8U0y/sv08/4/JrPcneyst Og7uB0GZub9PtXIScUegu0iWVuvXO+t57Pm3iqbNCmCDFpc0CQefo0x7eUrtB2NaNlGW PN84LElJrsQgmaiGjTLCDzkR/KepVOcNnqQ+gCt2ymcGxWhfc1R183zSB4MJbkS9EBwz N138Ve8lyST+6ZEJrlZIEH/dywn1+yJw47wAXPEYZlTWi8eOmsPDzZvLnoSFcNlGpqWg m3tDWVS7oAey6FM/2GGNXIGI1QYoBhM+QnwTv1ksB5IkukjCKI19beJ1QJ8YYHU2OJvq l4NQ== X-Gm-Message-State: AA+aEWaV6z1ZkZ5huagA1E8TUU4fdcBOjdk4dm7YasIKG4j4DGvdwthO Q79eykl+0YsBMCRNEMQemyVXeWjH X-Google-Smtp-Source: AFSGD/UE1waaphJRGk8seTNn9PIsQ3Js2vCbcfefagiJ1OfG0WTiCe6Y7JZsMeDJSCf6uU7AVpSsdw== X-Received: by 2002:adf:a743:: with SMTP id e3mr13194260wrd.56.1542973579036; Fri, 23 Nov 2018 03:46:19 -0800 (PST) Received: from ernst.home (p5B02380D.dip0.t-ipconnect.de. [91.2.56.13]) by smtp.gmail.com with ESMTPSA id 5sm11357677wmw.8.2018.11.23.03.46.18 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 23 Nov 2018 03:46:18 -0800 (PST) Date: Fri, 23 Nov 2018 12:46:17 +0100 From: Gary Jennejohn To: Lev Serebryakov Cc: Eugene Grosbein , Freebsd hackers list Subject: Re: TRIM utility Message-ID: <20181123124617.4ef3ac43@ernst.home> In-Reply-To: References: <7699de57-d903-1d61-ee42-062ed312b20d@grosbein.net> Reply-To: gljennjohn@gmail.com X-Mailer: Claws Mail 3.16.0 (GTK+ 2.24.32; amd64-portbld-freebsd12.0) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 3BC00858FC X-Spamd-Result: default: False [-4.59 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; HAS_REPLYTO(0.00)[gljennjohn@gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; REPLYTO_ADDR_EQ_FROM(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.94)[-0.944,0]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; IP_SCORE(-0.64)[ipnet: 2a00:1450::/32(-1.61), asn: 15169(-1.50), country: US(-0.09)]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; FREEMAIL_REPLYTO(0.00)[gmail.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[d.2.4.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Nov 2018 11:46:21 -0000 On Fri, 23 Nov 2018 14:12:47 +0300 Lev Serebryakov wrote: > On 22.11.2018 23:34, Eugene Grosbein wrote: > > > Currently it has four options, all of them are, hmm, optional: > > > > -b: to specify offset from the beginning of the device for trimmed region instead of default 0; > > -l: to specify offset from the "-b" margin - length - for trimmed region instead of whole device; > > -r rfile: for alternative way to specify length as length of referenced file; > > -v: for verbose mode that shows final values for the beginning offset and length. > Please, add '-n' for dry-run, which could be useful together with '-v'. > +1 -- Gary Jennejohn From owner-freebsd-hackers@freebsd.org Fri Nov 23 10:49:44 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DFF251142C23 for ; Fri, 23 Nov 2018 10:49:43 +0000 (UTC) (envelope-from lars@e.0x20.net) Received: from mail.0x20.net (mail.0x20.net [46.251.251.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "0x20.net", Issuer "COMODO RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 52A31832BC for ; Fri, 23 Nov 2018 10:49:43 +0000 (UTC) (envelope-from lars@e.0x20.net) Received: from e.0x20.net (mail.0x20.net [46.251.251.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.0x20.net (Postfix) with ESMTPS id 073DC98744; Fri, 23 Nov 2018 11:42:07 +0100 (CET) Received: (from lars@localhost) by e.0x20.net (8.15.2/8.15.2/Submit) id wANAg5ds023519; Fri, 23 Nov 2018 11:42:05 +0100 (CET) (envelope-from lars) Date: Fri, 23 Nov 2018 11:42:05 +0100 From: Lars Engels To: Eugene Grosbein Cc: Warner Losh , FreeBSD Hackers Subject: Re: TRIM utility Message-ID: <20181123104205.GZ59358@e.0x20.net> References: <7699de57-d903-1d61-ee42-062ed312b20d@grosbein.net> <6975ff4e-6383-e52c-3a11-d35b95cca114@grosbein.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <6975ff4e-6383-e52c-3a11-d35b95cca114@grosbein.net> X-Editor: VIM - Vi IMproved 8.0 User-Agent: Mutt/1.10.1 (2018-07-13) X-Rspamd-Queue-Id: 52A31832BC X-Spamd-Result: default: False [0.88 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.36)[-0.356,0]; MX_INVALID(0.50)[cached]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; IP_SCORE(-0.12)[asn: 31400(-0.59), country: DE(-0.02)]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[0x20.net]; AUTH_NA(1.00)[]; NEURAL_HAM_LONG(-0.21)[-0.208,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; NEURAL_HAM_SHORT(-0.13)[-0.132,0]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[lars.engels@0x20.net,lars@e.0x20.net]; R_DKIM_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; ASN(0.00)[asn:31400, ipnet:46.251.251.0/24, country:DE]; FROM_NEQ_ENVFROM(0.00)[lars.engels@0x20.net,lars@e.0x20.net] X-Rspamd-Server: mx1.freebsd.org X-Mailman-Approved-At: Fri, 23 Nov 2018 11:47:43 +0000 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Nov 2018 10:49:44 -0000 On Fri, Nov 23, 2018 at 01:59:31PM +0700, Eugene Grosbein wrote: > 23.11.2018 6:20, Warner Losh wrote: > > >> I found we have no utility capable to perform TRIM for the whole SSD device > >> or arbitrary part of it, so I wrote simple one. I can't think of nice name > >> for it, so proposal are welcome! Draft name is "erase". > >> > >> I ask for pre-commit code review, too. > >> The code is tested with TRIM-capable SSD and non-capable other devices. > >> > >> Currently it has four options, all of them are, hmm, optional: > >> > >> -b: to specify offset from the beginning of the device for trimmed region > >> instead of default 0; > >> -l: to specify offset from the "-b" margin - length - for trimmed region > >> instead of whole device; > >> -r rfile: for alternative way to specify length as length of referenced > >> file; > >> > > > > This seems really obscure and would be better handled by a stat command. > > This is inspired by truncate(1) having same option that saves extra call to stat. > > Forgot to note, that options -b and -l allow suffixes [K|k|M|m|G|g|T|t] > just like truncate's option -s does. > > > "erase" is a really bad name. It's fraught with too many overloaded > > meanings. "trim" is likely the least bad name we can use. > > I'm fine with this, renamed. > > > Linux has a fstrim command, which does something kinda similar (it's a lot > > like fsck -E to erase unused parts of the filesystem), so there is some > > overlap. I couldn't find a dedicated command to do that, but if it does, we > > should follow that convention to reimplement. > > Well, they have http://man7.org/linux/man-pages/man8/blkdiscard.8.html > > I don't like the name, though. It's too complicated to pronounce and too long same time. Would "hdtrim" or "ssdtrim" be better names? From owner-freebsd-hackers@freebsd.org Fri Nov 23 12:15:18 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C0FE21145706 for ; Fri, 23 Nov 2018 12:15:18 +0000 (UTC) (envelope-from yaneurabeya@gmail.com) Received: from mail-pg1-x534.google.com (mail-pg1-x534.google.com [IPv6:2607:f8b0:4864:20::534]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0CA7C86995; Fri, 23 Nov 2018 12:15:18 +0000 (UTC) (envelope-from yaneurabeya@gmail.com) Received: by mail-pg1-x534.google.com with SMTP id s198so3032905pgs.2; Fri, 23 Nov 2018 04:15:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=rODfUGZH5PvdKXU6UZ6SAUO/af1yD1I7ttbjnpRDqaY=; b=RoKMjXIXUjh0o9Tqv2EvT5N2Q8jYHkraH9S0iC34cBaYtzX762qpBOdyOlR4ZXNedW C4oaweANHt7/r3uK02iHu5RyG/OejE7yz3KLAETj/3nFUDZl311y6jTletatl91STWma 56sHguPPdWUWDYl3PU5phR+w66wKLTVzMBXoxNebwGkgt88A/75P58FSNcQEiYjTvdp7 UNXsr1PTepBAz6lJFBQISD3pQ51ZywqFVJarpBO/F9Lz7D74mqdeGetWRi1g3V4DGoGE RJZz60+zGlWOdY6JYyvIInDOTGKoPXB1fGALAPlar6PmR6v+/BeHRDL44Fa9ZI0QH0YP DOdQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=rODfUGZH5PvdKXU6UZ6SAUO/af1yD1I7ttbjnpRDqaY=; b=qkhgtfbg53PrrrFUO3JB7MA7yCBWIhuD3rMzXqe0lWWniNMBmcCO+lCfBdZjN0uwVU eQAi4R4gpccOZuI+d5HPU9phZIsMI977ZdC3ySndu2DKKJzIPjZ+qO1f6sRAcAgqIUZh +h+b605YC4PrJOXfwc6C/96/kGTL6xsbigIn+xj3qgBuv48OtS0mc6wXRBFzKR9SY/SP 7pzZXchXG1eHZxBvfSvTN4x9UPyG8F9kwYk/pbZB1JJuRrcQ7ARbTFpGbv4JFw2jrXLU VbU3F6GK61x39+O4x2K28b75sZM4hieoVhQrTukfSCla3NSjJywlwy1SJzdjzMNju4V2 4d1w== X-Gm-Message-State: AA+aEWYXuz1sSRbOvo7Em7imxx4YXzQIet6TIK5kWDgyLUC55p7DIf7E dQZspTcg2zRpoGwvbAuETzCSr3uM X-Google-Smtp-Source: AFSGD/XsdedNMYGVyF+vek4ZJ0ZdY9F5Vx04rN0xLGls3SZ0j2m45aH7hPy+BWMzz35y/6lXSm2fSw== X-Received: by 2002:a63:c503:: with SMTP id f3mr13499180pgd.431.1542975316758; Fri, 23 Nov 2018 04:15:16 -0800 (PST) Received: from [192.168.20.13] (c-73-19-52-228.hsd1.wa.comcast.net. [73.19.52.228]) by smtp.gmail.com with ESMTPSA id t66sm41914239pfd.54.2018.11.23.04.15.15 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 23 Nov 2018 04:15:16 -0800 (PST) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (1.0) Subject: Re: TRIM utility From: Enji Cooper X-Mailer: iPhone Mail (16B92) In-Reply-To: Date: Fri, 23 Nov 2018 04:15:15 -0800 Cc: Poul-Henning Kamp , Freebsd hackers list , Eugene Grosbein Content-Transfer-Encoding: quoted-printable Message-Id: References: <7699de57-d903-1d61-ee42-062ed312b20d@grosbein.net> <3876.1542971951@critter.freebsd.dk> To: lev@FreeBSD.org X-Rspamd-Queue-Id: 0CA7C86995 X-Spamd-Result: default: False [-4.20 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; FREEMAIL_FROM(0.00)[gmail.com]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[4.3.5.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(-0.73)[ipnet: 2607:f8b0::/32(-2.07), asn: 15169(-1.50), country: US(-0.09)]; NEURAL_HAM_SHORT(-0.96)[-0.959,0]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Nov 2018 12:15:19 -0000 > On Nov 23, 2018, at 03:34, Lev Serebryakov wrote: >=20 > On 23.11.2018 14:19, Poul-Henning Kamp wrote: >=20 >>>> Currently it has four options, all of them are, hmm, optional: >> Isn't this the kind of thing that dd(1) should learn about instead ? > One utility to done one thing very well? :-) >=20 > dd(1) is way overloaded, IMHO. I agree that dd is super complex; while we could add conv=3Dtrim, that might= not be the right way. Would it make sense to have pushed down into a simple library (leveraging th= e logic used by dd), and adopt in several tools? Thanks so much for bringing this up Eugene :). -Enji= From owner-freebsd-hackers@freebsd.org Fri Nov 23 16:00:47 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 05D26114A863 for ; Fri, 23 Nov 2018 16:00:47 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-io1-xd36.google.com (mail-io1-xd36.google.com [IPv6:2607:f8b0:4864:20::d36]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5A3B069AFF for ; Fri, 23 Nov 2018 16:00:46 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-io1-xd36.google.com with SMTP id g8so9094859iop.10 for ; Fri, 23 Nov 2018 08:00:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=VI9V+Kg0Nhh0KGNaJL3LZ2Up55kxW0boBPrQ6xdBhDA=; b=rBi+kJ9HRl0p/WdQOfd68Hqq5R3WJ3tQn4EqPlbxJbbrWooeAcE0w3+vfv2Rp88DA4 1IgevB0I1WL3Kj2L06XbMHShH+vuXsCIlS1bsVWM6pSsWe5ANSuGXob2XMO0czZVuYwh 770bDme4IpsqCcANbGn1ZfvkNvvJRTad3TODYnzUtlXsE4zIn1yIN1TrOfWd3ml6Dw1i V9scg636aRvZP662rpwbkrFOjPrQ/Hr/1u47CcntxxPw0O2Y+NSKCkccDIuwXqWZ1QGZ dH03OcNW7JR4e4hiEo9iZlf9Yhphl2u/bAKrvc5GEJxroCg3UBXAvy9A0YPHG6sN6Rd9 HG6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=VI9V+Kg0Nhh0KGNaJL3LZ2Up55kxW0boBPrQ6xdBhDA=; b=R4TX5d94Rp4RCSrE3QP4XXT8yFACt11FWhwUk0k1/i1nnMvErrBmJiJ/PhjW1A8309 GwvVg4ZcDH8yLSeJ4fHHK9Z6broK0BAnPcK4KvLkTPjzDIQIvtbiAUsYvIvEReGMSylf yz716FapQFkhO5wfQU51DcnbdykVkvN0M0lmkeUj2c6kd4R8OcuEZaxf++y+9ljCXxGZ n0ZCMrxgCWOdYgFLofBtmwlqY9hE/fdSgSxIKA9FuBDSJLjVRxlS5o4CZJ9kagmKnEXD eYyOFEbyS2R09Cbrst46CAFngv/CTRGHoPwzRj10opMF3bjoBuuaxfZSJiUE4epX+Qs7 oLVg== X-Gm-Message-State: AA+aEWb/aBw4dXSz7TZFcurYOQXY5jhIJocEwH5/mqmo0h/jauFpyDqN I4VwdISFAo3E03AEMKa7HCknNzFppjWfDWVaX0Y7ky5R X-Google-Smtp-Source: AFSGD/U/iEjucXfpa1RlrxeCA0cs9AjS3/9JPRsldzSF6UYBupsLTKfMk+X9wH1XSDIIt8kP2cwG6lpiwIfdwiBCyXI= X-Received: by 2002:a5e:d808:: with SMTP id l8mr2002229iok.299.1542988845320; Fri, 23 Nov 2018 08:00:45 -0800 (PST) MIME-Version: 1.0 References: <7699de57-d903-1d61-ee42-062ed312b20d@grosbein.net> <6975ff4e-6383-e52c-3a11-d35b95cca114@grosbein.net> <20181123104205.GZ59358@e.0x20.net> <5BF7DD8F.1040007@grosbein.net> In-Reply-To: <5BF7DD8F.1040007@grosbein.net> From: Warner Losh Date: Fri, 23 Nov 2018 09:00:33 -0700 Message-ID: Subject: Re: TRIM utility To: Eugene Grosbein Cc: Lars Engels , FreeBSD Hackers X-Rspamd-Queue-Id: 5A3B069AFF X-Spamd-Result: default: False [-3.68 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.99)[-0.994,0]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-0.99)[-0.988,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[cached: ALT1.aspmx.l.google.com]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; RCVD_IN_DNSWL_NONE(0.00)[6.3.d.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; NEURAL_HAM_SHORT(-0.96)[-0.960,0]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; RCVD_TLS_LAST(0.00)[]; IP_SCORE(-0.73)[ipnet: 2607:f8b0::/32(-2.06), asn: 15169(-1.49), country: US(-0.09)]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Server: mx1.freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Nov 2018 16:00:47 -0000 On Fri, Nov 23, 2018, 3:59 AM Eugene Grosbein On 23.11.2018 17:42, Lars Engels wrote: > > > Would "hdtrim" or "ssdtrim" be better names? > > Just "trim" sounds better to me :-) > Trim applies to more than add. And never to hdd. Virtualization uses it. SD cards use it. NVMe drives use it (I'd argue the aren't really SSDs but something else). Warner > From owner-freebsd-hackers@freebsd.org Fri Nov 23 16:01:03 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 507A2114A889 for ; Fri, 23 Nov 2018 16:01:03 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8428B69CB3; Fri, 23 Nov 2018 16:01:02 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (localhost [127.0.0.1]) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3) with ESMTP id wANG0xGV083200; Fri, 23 Nov 2018 08:00:59 -0800 (PST) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: (from freebsd-rwg@localhost) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3/Submit) id wANG0wHc083199; Fri, 23 Nov 2018 08:00:58 -0800 (PST) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201811231600.wANG0wHc083199@pdx.rh.CN85.dnsmgr.net> Subject: Re: TRIM utility In-Reply-To: To: lev@freebsd.org Date: Fri, 23 Nov 2018 08:00:58 -0800 (PST) CC: Poul-Henning Kamp , Freebsd hackers list , Eugene Grosbein X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-Rspamd-Queue-Id: 8428B69CB3 X-Spamd-Result: default: False [-0.51 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.54)[-0.539,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-0.55)[-0.546,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[dnsmgr.net]; AUTH_NA(1.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[cached: pdx.rh.CN85.dnsmgr.net]; NEURAL_HAM_SHORT(-0.30)[-0.296,0]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:13868, ipnet:69.59.192.0/19, country:US]; MID_RHS_MATCH_FROM(0.00)[]; IP_SCORE(-0.02)[country: US(-0.09)] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Nov 2018 16:01:03 -0000 > On 23.11.2018 14:19, Poul-Henning Kamp wrote: > > >>> Currently it has four options, all of them are, hmm, optional: > > Isn't this the kind of thing that dd(1) should learn about instead ? > One utility to done one thing very well? :-) > > dd(1) is way overloaded, IMHO. I agree here, we do too much of trying to shoe horn things into existing utilities then we end up with a command parser that only a mother could love. trim, hdtrim, blktrim, camtrim, any of them are fine, fstrim is bad, this is not a filesystem op, too bad the next thing that comes along that is "trim" like well have to pick something other than trim. I might ask would it be horribly hard to access the "secure erase" feature from this utility? Or do we have another that can easily get at that function, that is usually the prefered vendor specific method to "trim" the complete drive, often restoring badly leveled SSD's to a performant and usable state. -- Rod Grimes rgrimes@freebsd.org From owner-freebsd-hackers@freebsd.org Fri Nov 23 16:03:10 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 61A9C114AB54 for ; Fri, 23 Nov 2018 16:03:10 +0000 (UTC) (envelope-from yuripv@yuripv.net) Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com [66.111.4.28]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AB9FB6A0EE for ; Fri, 23 Nov 2018 16:03:09 +0000 (UTC) (envelope-from yuripv@yuripv.net) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.nyi.internal (Postfix) with ESMTP id 21B512213F for ; Fri, 23 Nov 2018 11:03:01 -0500 (EST) Received: from mailfrontend2 ([10.202.2.163]) by compute5.internal (MEProxy); Fri, 23 Nov 2018 11:03:02 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yuripv.net; h=to :from:subject:message-id:date:mime-version:content-type; s=fm3; bh=f/5pT0MXidg6tEQlviCLfiqu4cseFxQVfmgD61dK4lE=; b=b61oPoerO4Bp 3K5j0cdTdfHnfkhxc++q8ZQJEWEAu2rtp56BNVMl9UzHNOIYZyVdcM3RNukeCqTi wYhmiGH5C0ooclDkJCJky8/X06pkQlDEKDU/zVUzsysTn6jXK1PAOTpFLH/Kr0nu WazK9/P25q1S93dCIEDgk7pU5YhWebR3Y2MBmwpRc7azOBTOFuTZhBlgDpfuNSQt gLHWQTNWx/jgDy9RIbouXaJrtmXLAQXEADIomBihtT7HDKkaUcjDwxVQ/okP5e2e 7a6Mu+VxPBwu6Hs54krsKReeihGRZayy/ahd7wVlsRXBvZnsgtwDmPnUFdFi0+Js T2xpWMmceg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-type:date:from:message-id :mime-version:subject:to:x-me-proxy:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; bh=f/5pT0MXidg6tEQlviCLfiqu4cseF xQVfmgD61dK4lE=; b=BoZFJ0ZGKHHgyd1CjgB7CwwXfZhNZqIqM8tMAzkt1Zw+M tZYmx2VaIPS7O5CaRf6+JcIYeyC8J0HdHNaRhyRfbZbAkESAE1C/hVnHCp510trf aL5lQEgA2xqg3/VReF+GfU3QdfWQPEFoVTlc4ox5xcbwvdWk6v3eqBnqNA7eQgA/ rI5q4T+G2uUPovUsx8ECftT5dH3B9nENztTfphjnH3q+1kOCRJ1IVTwqLgpCwXhE 8iqxt8Ru3VpcHH7HmIIQAOcvZkwwapdOEA+kLXwHnKO9eLwhhHnOZu2uleB2OU7p v9A2CXLxaFdZ2RZOWmjCthONAnGWIaSFX8qJPZYSA== X-ME-Sender: X-ME-Proxy: Received: from thor.yuripv.net (unknown [92.50.223.252]) by mail.messagingengine.com (Postfix) with ESMTPA id 7318A102E0 for ; Fri, 23 Nov 2018 11:03:00 -0500 (EST) To: freebsd-hackers From: Yuri Pankov Subject: regex, multibyte locales, and word boundaries Openpgp: preference=signencrypt Autocrypt: addr=yuripv@yuripv.net; keydata= xsBNBFu8u6IBCADB11gP0QwnorrHjqAtKLHKHNHskhy0s7jqJKfx0YqXgVBKGLJ9/mjLAz0F CBNvemHSDDTs0mEZ9cBKKi6cmsav6+UQgr//yai6hvXLBJqKchSFO4MhmdvBtsGFq1yKz5Zi uhjmimKyIpgBgvMdbgGbGq6cnSB2uEPmZuJr419SVRODOkXukU+F5WHgaHzDdHAIu1asCt2B +6msxqIqlFWcXyZyTGicTGGvC/PFIsVRUtD1dIJANTC876g7DTb7LZXWiWwJpSJ4GKMXMHVX Ct9BoQ4i3nhKbOxb6Io1wsy+NFyWsTJ9KYrxKKPJP3oG8BWb/cqlFqnE4eNSsiq2q7krABEB AAHNIFl1cmkgUGFua292IDx5dXJpcHZARnJlZUJTRC5vcmc+wsCUBBMBCgA+FiEE+Gq3PsPe LT4tL/9wk4vgf7Eq4WwFAlu9Cn0CGwMFCQWjmoAFCwkIBwMFFQoJCAsFFgMCAQACHgECF4AA CgkQk4vgf7Eq4WxuPQf9HccaDyusO1J+wDQNlp9/uU0cnIfjHAeG80xrAfN9Vnf1wO9T2/WI iYlIdK+KVnhSa/DeBuHq/asfpUbrOleTF0hzG39os+95DzuT9a/j5XeQGuBgNbpVB+10zR3I 5AagSQetHilcZtz65g9GTUuIxb+xDaBehFBjyYXApfNE6yY5IlzDZpM7MOOLLFm2mQwQ8yjS eZ4jA6qW6/QMXRTkmpC9EXIeWDuNgWBwszaFGR6oUIpl0mGmwdJkEKwUazt6OuoDilMNZefZ 0pVFZBhnE46vK+6FDDFZE3BkeHVnqvy2QGL/6uKhSHc0lChCEPHnhqz6v23MwcQ6ktVWzvBJ oM7ATQRbvLuiAQgAyood0Pd96wzY+GQPBYQUNkZZgYL8Di3AzyC94dFe4d/Mt/h4rIBUnFwA g7Ha05WGdW0V5A/RRxDcpwXL9Jf97hiQ5PI2hiAxNEz/DkAUafiGlPfwR5wKqysUyRiKJQ2o ctpvssdsoXXOgeLo1jA6ghda1jg/spjlsPlS5ZTpKx3GWuTybV/VDhmwKWZfGUzPBJeAgDTf BdW4PTFs1IvvC2KBlhnPgcLBUtTlAdXOEj4DLuXw+Fn7K/ckZdOn3aRANmE+wf4+f+UUgtLB NmbP7ZifyUX5RyddsnI+fZmtsUDHxCReNIWQ6TBUJmb21aoBIN6HEHJbY28ZSCmf5owuMwAR AQABwsB8BBgBCgAmFiEE+Gq3PsPeLT4tL/9wk4vgf7Eq4WwFAlu8u6ICGwwFCQWjmoAACgkQ k4vgf7Eq4WyA3AgAqgGTHKMVAS2WuNGuW9uI+YtY6ZbwmGG94fkOZbefgRSfO5Am+HSblA95 IdotvQa8VkFmvVjbnvaM8XmJG5H17m0GF3sVaJUbJ4euDnRrBPCr6KwRQQd83Svxkbdicvo7 J031FrkJZW8zD9DH4QgzJNTKPFrwx9v3DhD/8iyn9tGvnHepy7O24nY5hl6PacrgSgLVeir/ lUbueAC/gP1AWLv3gdw7b83J7rftWauimj/vpFMD8CDSyJNODgQ8DdM0TU4qjABWGMs9r2Rw QehNbYf5f/2QuW/Q5NGaRSNW2HS/cpp62XtTKmxj5wwk6EMbtNE/6WQpumfdmK2UGLjcJQ== Message-ID: <5166f3c9-d587-a245-df21-8e50f075a8cc@yuripv.net> Date: Fri, 23 Nov 2018 19:02:40 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.3.0 MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="p8IhWLes9jJtEyQe8bxwD0DPXoEUrQ9Zz" X-Rspamd-Queue-Id: AB9FB6A0EE X-Spamd-Result: default: False [-4.85 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(0.00)[+ip4:66.111.4.28]; HAS_ATTACHMENT(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[cached: in2-smtp.messagingengine.com]; DKIM_TRACE(0.00)[yuripv.net:+,messagingengine.com:+]; NEURAL_HAM_SHORT(-0.99)[-0.991,0]; SIGNED_PGP(-2.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[252.223.50.92.zen.spamhaus.org : 127.0.0.11]; IP_SCORE(-2.66)[ip: (-9.30), asn: 11403(-3.89), country: US(-0.09)]; RCVD_TLS_LAST(0.00)[]; RCVD_IN_DNSWL_LOW(-0.10)[28.4.111.66.list.dnswl.org : 127.0.5.1]; ASN(0.00)[asn:11403, ipnet:66.111.4.0/24, country:US]; MID_RHS_MATCH_FROM(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; ARC_NA(0.00)[]; RECEIVED_SPAMHAUS_XBL(3.00)[252.223.50.92.zen.spamhaus.org : 127.0.0.4]; R_DKIM_ALLOW(0.00)[yuripv.net,messagingengine.com]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-0.998,0]; MIME_GOOD(-0.20)[multipart/signed,multipart/mixed,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; DMARC_NA(0.00)[yuripv.net]; RCPT_COUNT_ONE(0.00)[1]; BAD_REP_POLICIES(0.10)[] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Nov 2018 16:03:10 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --p8IhWLes9jJtEyQe8bxwD0DPXoEUrQ9Zz Content-Type: multipart/mixed; boundary="07odA7TKf2mtNqjkFg4cktmrTFo1E2kIs"; protected-headers="v1" From: Yuri Pankov To: freebsd-hackers Message-ID: <5166f3c9-d587-a245-df21-8e50f075a8cc@yuripv.net> Subject: regex, multibyte locales, and word boundaries --07odA7TKf2mtNqjkFg4cktmrTFo1E2kIs Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable Hi, We have the following note in the BUGS section of regcomp(3): ---------------------------------------------------------------------- Word-boundary matching does not work properly in multibyte locales. ---------------------------------------------------------------------- It was added ages ago along with multibyte support in our regex implementation, though I can't think of any positive test case to see that the problem is real, and eventually fix it. I'm wondering if anyone has real life examples showing the bug? --07odA7TKf2mtNqjkFg4cktmrTFo1E2kIs-- --p8IhWLes9jJtEyQe8bxwD0DPXoEUrQ9Zz Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEE+Gq3PsPeLT4tL/9wk4vgf7Eq4WwFAlv4JLIACgkQk4vgf7Eq 4WxpGAgAqKQP7R+0Qbc7zGo6QCEfO37P4SG3H3o5pUGvdCOOweUCGLQQALS1cqww WUbgvpWYuMzYVNAhslURF/S1cV0v3nmzkH4vlksnJJ3vYJ0KVipkdsXNN6M5dvYj 5RU0g2EYyLingB1GCFvlazA1mjV7RZ/f91SNOX9fFIQC2u9IfSwmdnePyeDpym6M dR0SrDuO1iQGsuelKNXunTTRZ3oJq4PDFV5FXBg8qWj9jl3wVWXpUa1NERZfLhcr NadmcGMnwtGaxXcocNwjed7gTLoNQ4oYGML5b8i5a2bDO4mJcj8a+ZO3VKuwfuYT NUfCRIed7gQU2jigQWPLCNyUiftNEA== =tTwj -----END PGP SIGNATURE----- --p8IhWLes9jJtEyQe8bxwD0DPXoEUrQ9Zz-- From owner-freebsd-hackers@freebsd.org Fri Nov 23 18:42:05 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D779E114EADD for ; Fri, 23 Nov 2018 18:42:04 +0000 (UTC) (envelope-from bch@online.de) Received: from mout.kundenserver.de (mout.kundenserver.de [212.227.126.130]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "mout.kundenserver.de", Issuer "TeleSec ServerPass DE-2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A703D7017B for ; Fri, 23 Nov 2018 18:42:03 +0000 (UTC) (envelope-from bch@online.de) Received: from online.de ([195.201.29.72]) by mrelayeu.kundenserver.de (mreue010 [212.227.15.167]) with ESMTPSA (Nemesis) id 1MxV8b-1fTDYo1NEX-00xu8I for ; Fri, 23 Nov 2018 19:41:55 +0100 Received: by online.de (Postfix, from userid 1000) id 7F859587CA; Fri, 23 Nov 2018 19:41:54 +0100 (CET) From: Christian Barthel To: freebsd-hackers@freebsd.org Subject: top(1): adding grep like selection and system processes Date: Fri, 23 Nov 2018 19:41:54 +0100 Message-ID: <878t1j1vrh.fsf@x230.onfire.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (berkeley-unix) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Provags-ID: V03:K1:19Vb0lvzPUuwVP2zVjVU7rf6KdWKONg+VGkGKdMekZM9LP4KJjc V3v3ELuhtOrs5Qo92B/o7KKOYtkJCxpSpE0CjzVNZ7cDRys0037XfS5VV/feBcahPA9+X9S SsY/1I6FzyUZJ48vAjkY0tGxd3F8jd2OtZJW3+gU2oNxi8PWB3FBcNCpyzXS19cRLhwO1dS er6+h0+2hc316Af1k+6Gw== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:LX0COE2/tA8=:Mkmv2IJ+eGBkuGpdMyspcb 0aW9vxwnbIwJNxQctJj4GeTmjS6H3aWT8Eolh43Jz6fMB1GGPzrQjiO0Pi4KMOg6pqSV+4uEK k4DXUXX0rXXIb8Dy4aIGb5w6Bh7ZI9DVPDwFYPAQm0SLJRZV3sWXT/XVvgi1J/WQxCC7YtbXH m4oWErCnYPYP8G48nyWokTF+q1JHDrw65GBGDxuMWx0MPClWYBojAfqurD4PdKcEn8cq4Bf64 NAvgZLtpvB4lfIxVOi3YrV+sXTAhsbQAIuMA8ECW18AatgMm68XcSOhHUyNCVcexaZDS/3oLP 6jeqpzULgdgA9SWKUFDVDWMqta6/Q62m3n30Cazpr8Hqbs0hEZgQERDH37z10+dOJuGzMvQFI qds7BJxcFO03GbI9RD33Nd+I8aPLueoJxhXui0yP2Afsqth2u2SdTAdoezTp+7QzGFh7T8H4w aKd8bVUcawrrzEXsSmn2st3pUwIknK8NYqHCNRICGPHqMZN1Db/VPcF3pD0ZzneuInFBgW35S bdTusvQJ8RPTn4qdB/PMIT7DtF8hfi93AwOpt7N7OatVDHd/8Mp1m60gD8A0OMSZExYBbihbx iZbD0RvdaETkug6bKRRtCHy7141paX8YJc9TIT9iifUOJyzhRMs4A76ol9hVMGdBUYqvUYZWG QzKZ+t/rkePXoJSiNGQm27lIufmIQ0HMOS+WMoXqLTVxwxti/EsODiTspzxPLkbN3hiZhm9rC fp2edkLuq+po0HwUIdh/hoYa6bNjgtATyvSHjmmAvulho8C2p2gtgFgBTyKz9yhPZH91P+45c 8CHuzpA X-Rspamd-Queue-Id: A703D7017B X-Spamd-Result: default: False [1.51 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-0.66)[-0.655,0]; HAS_ATTACHMENT(0.00)[]; MIME_GOOD(-0.10)[multipart/mixed,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; AUTH_NA(1.00)[]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_NONE(0.00)[]; NEURAL_SPAM_SHORT(0.87)[0.866,0]; MX_GOOD(-0.01)[mx00.emig.kundenserver.de,mx01.emig.kundenserver.de]; RCVD_IN_DNSWL_NONE(0.00)[130.126.227.212.list.dnswl.org : 127.0.5.0]; NEURAL_SPAM_MEDIUM(0.34)[0.341,0]; R_SPF_NA(0.00)[]; DMARC_NA(0.00)[online.de]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:8560, ipnet:212.227.0.0/16, country:DE]; IP_SCORE(-0.03)[ipnet: 212.227.0.0/16(-0.30), asn: 8560(0.15), country: DE(-0.01)]; MIME_UNKNOWN(0.10)[text/x-diff] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Nov 2018 18:42:05 -0000 --=-=-= Content-Type: text/plain Hello, I've added two features that I (personally) found useful in top(1): A grep-like feature to filter command names in top(1) (inspired by the OpenBSD top version). One example may be to filter out all current virtual machines by using "top -g bhyve" or interactively with "g bhyve". (there was already a selection field ps.command; so, perhaps someone is already working on a similar feature?). It is not possible to filter with a regular expression. The second flag is selection of system processes with -K or interactively toggle between user and system processes. I want to ask if this is of general interest for FreeBSD users or if its useful for someone else here? I've tested it on FreeBSD 13.0-CURRENT. Christian --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=top-grep-sysproc.patch Content-Description: top(1) patch to add grep-like feature and system process selection. diff --git a/usr.bin/top/commands.c b/usr.bin/top/commands.c index 88f4b0867d4..082f5fcc9af 100644 --- a/usr.bin/top/commands.c +++ b/usr.bin/top/commands.c @@ -59,10 +59,12 @@ const struct command all_commands[] = {'H', "toggle the displaying of threads", false, CMD_thrtog}, {'h', "show this help text", true, CMD_help}, {'?', NULL, true, CMD_help}, + {'g', "grep command name", false, CMD_grep}, {'i', "toggle the displaying of idle processes", false, CMD_idletog}, {'I', NULL, false, CMD_idletog}, {'j', "toggle the displaying of jail ID", false, CMD_jidtog}, {'J', "display processes for only one jail (+ selects all jails)", false, CMD_jail}, + {'K', "toggle between system and user processes", false, CMD_kern}, {'k', "kill processes; send a signal to a list of processes", false, CMD_kill}, {'q', "quit" , true, CMD_quit}, {'m', "toggle the display between 'cpu' and 'io' modes", false, CMD_viewtog}, diff --git a/usr.bin/top/commands.h b/usr.bin/top/commands.h index 0071fbe62fc..75fe9756a56 100644 --- a/usr.bin/top/commands.h +++ b/usr.bin/top/commands.h @@ -46,6 +46,8 @@ enum cmd_id { CMD_order, CMD_pid , CMD_toggletid, + CMD_grep, + CMD_kern, }; struct command { diff --git a/usr.bin/top/machine.c b/usr.bin/top/machine.c index 374c9da0edf..2fb5b5221b7 100644 --- a/usr.bin/top/machine.c +++ b/usr.bin/top/machine.c @@ -803,6 +803,13 @@ get_process_info(struct system_info *si, struct process_select *sel, /* skip self */ continue; + if (sel->command && strncmp(pp->ki_comm, sel->command, COMMLEN)) + /* skip not matching commands */ + continue; + + if (!sel->user && !(pp->ki_flag & P_SYSTEM) && sel->pid == -1) + continue; + if (!sel->system && (pp->ki_flag & P_SYSTEM) && sel->pid == -1) /* skip system process */ continue; diff --git a/usr.bin/top/machine.h b/usr.bin/top/machine.h index c3c7777d910..40a36e947b9 100644 --- a/usr.bin/top/machine.h +++ b/usr.bin/top/machine.h @@ -63,6 +63,7 @@ struct process_select bool idle; /* show idle processes */ bool self; /* show self */ bool system; /* show system processes */ + bool user; /* show user processes */ bool thread; /* show threads */ bool thread_id; /* show thread ids */ #define TOP_MAX_UIDS 8 @@ -73,7 +74,7 @@ struct process_select bool swap; /* show swap usage */ bool kidle; /* show per-CPU idle threads */ int pid; /* only this pid (unless pid == -1) */ - const char *command; /* only this command (unless == NULL) */ + char *command; /* only this command (unless == NULL) */ }; /* routines defined by the machine dependent module */ diff --git a/usr.bin/top/top.c b/usr.bin/top/top.c index 2279479409b..3a30bf40d9a 100644 --- a/usr.bin/top/top.c +++ b/usr.bin/top/top.c @@ -271,6 +271,7 @@ main(int argc, char *argv[]) ps.idle = true; ps.self = true; ps.system = false; + ps.user = true; reset_uids(); ps.thread = false; ps.wcpu = 1; @@ -305,14 +306,20 @@ main(int argc, char *argv[]) optind = 1; } - while ((i = getopt_long(ac, av, "CSIHPabijJ:nquvzs:d:U:m:o:p:Ttw", longopts, NULL)) != EOF) + while ((i = getopt_long(ac, av, "CSIHKPabg:ijJ:nquvzs:d:U:m:o:p:Ttw", longopts, NULL)) != EOF) { switch(i) { case 'v': /* show version number */ errx(0, "version FreeBSD"); break; - + case 'g': + ps.command = strdup(optarg); + break; + case 'K': + ps.user = false; + ps.system = true; + break; case 'u': /* toggle uid/username display */ do_unames = !do_unames; break; @@ -870,7 +877,15 @@ main(int argc, char *argv[]) topn = newval; } break; - + case CMD_kern: + if (ps.user == false) { + ps.user = true; + ps.system = false; + } else { + ps.user = false; + ps.system = true; + } + break; case CMD_delay: /* new seconds delay */ new_message(MT_standout, "Seconds to delay: "); if ((i = readline(tempbuf1, 8, true)) > -1) @@ -1027,7 +1042,19 @@ main(int argc, char *argv[]) reset_display(); putchar('\r'); break; + case CMD_grep: + new_message(MT_standout, + "command name: "); + if (readline(tempbuf2, sizeof(tempbuf2), false) > 0) { + if (ps.command != NULL) + free(ps.command); + if (tempbuf2[0] == '+' && tempbuf2[1] == '\0') + ps.command = NULL; + else + ps.command = strdup(tempbuf2); + } + break; case CMD_jail: new_message(MT_standout, "Jail to show (+ for all): "); --=-=-=-- From owner-freebsd-hackers@freebsd.org Fri Nov 23 18:53:52 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3C58C114ED52 for ; Fri, 23 Nov 2018 18:53:52 +0000 (UTC) (envelope-from cse.cem@gmail.com) Received: from mail-io1-f42.google.com (mail-io1-f42.google.com [209.85.166.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 66B98705E7 for ; Fri, 23 Nov 2018 18:53:51 +0000 (UTC) (envelope-from cse.cem@gmail.com) Received: by mail-io1-f42.google.com with SMTP id v10so3767897ios.13 for ; Fri, 23 Nov 2018 10:53:51 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:reply-to :from:date:message-id:subject:to:cc; bh=AKR+nckBiz3rTVsJy7PApZ15EG4ZWHRiLvNlhhlY6aQ=; b=fFTdoDHtPd7+zuDf+EPJaKKwf7c4qoHHRKgIFug6u009mPTFmWb+dXmM2rwRrNIV1k PtD0JIip4YlASN/KYtdFPP3iB55EoUjWUJaVenwXGao2+03oabqxJ23sJgTq9LHhlltS UP588tkpSdCPR45CWbgS9Npql9sF8Mo9D3RUYrK4Hz2ZKa3eymDYgvooEeZkCYyPQakL AbSLMhpQ35V8iUjYJKTI5tdkyaqb4UuoGv48W2Phf9J/NR0g6gsKlLOsuu+VZvEDcQeF WJtAW8uBtUIONurGTOQ0TT6bfF/pV6bZBe0fGP5hxfBEXumehMDBlLpeaRrgNYv3D5+p pC9g== X-Gm-Message-State: AA+aEWY+ghmnWwjYujjX/7w2SH+WnYHQQ24o8y+w4GcIDsR4gAFtZ8kJ NFv9AOlQaYU4YCvPNAiYD9ENGx/i X-Google-Smtp-Source: AFSGD/W9fa7T+AlcmgxzOGF7ToxZR34EQmDMu7V5YJ2jTirrD5NHAjcbWKGinnilR9JA6cwBF1XEXw== X-Received: by 2002:a6b:b717:: with SMTP id h23mr11937085iof.14.1542999224968; Fri, 23 Nov 2018 10:53:44 -0800 (PST) Received: from mail-io1-f54.google.com (mail-io1-f54.google.com. [209.85.166.54]) by smtp.gmail.com with ESMTPSA id c102sm3544406itd.38.2018.11.23.10.53.44 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 23 Nov 2018 10:53:44 -0800 (PST) Received: by mail-io1-f54.google.com with SMTP id a3so9413343ioc.4 for ; Fri, 23 Nov 2018 10:53:44 -0800 (PST) X-Received: by 2002:a5d:94cc:: with SMTP id y12mr8353013ior.233.1542999224152; Fri, 23 Nov 2018 10:53:44 -0800 (PST) MIME-Version: 1.0 References: <878t1j1vrh.fsf@x230.onfire.org> In-Reply-To: <878t1j1vrh.fsf@x230.onfire.org> Reply-To: cem@freebsd.org From: Conrad Meyer Date: Fri, 23 Nov 2018 10:53:33 -0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: top(1): adding grep like selection and system processes To: bch@online.de Cc: "freebsd-hackers@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 66B98705E7 X-Spamd-Result: default: False [-3.85 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; HAS_REPLYTO(0.00)[cem@freebsd.org]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; REPLYTO_ADDR_EQ_FROM(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-0.83)[-0.831,0]; FORGED_SENDER(0.30)[cem@freebsd.org,csecem@gmail.com]; IP_SCORE(-1.01)[ipnet: 209.85.128.0/17(-3.48), asn: 15169(-1.49), country: US(-0.09)]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; TAGGED_FROM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_NEQ_ENVFROM(0.00)[cem@freebsd.org,csecem@gmail.com]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[42.166.85.209.list.dnswl.org : 127.0.5.0]; RCVD_TLS_LAST(0.00)[] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Nov 2018 18:53:52 -0000 Hi Christian, I've wanted these features in top many times. Sounds great to me. Take care, Conrad On Fri, Nov 23, 2018 at 10:44 AM Christian Barthel wrote: > > Hello, > > I've added two features that I (personally) found useful in top(1): > > A grep-like feature to filter command names in top(1) (inspired by the > OpenBSD top version). One example may be to filter out all current > virtual machines by using "top -g bhyve" or interactively with "g > bhyve". (there was already a selection field ps.command; so, perhaps > someone is already working on a similar feature?). It is not possible > to filter with a regular expression. > > The second flag is selection of system processes with -K or > interactively toggle between user and system processes. > > I want to ask if this is of general interest for FreeBSD users or if its > useful for someone else here? I've tested it on FreeBSD 13.0-CURRENT. > > Christian > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" From owner-freebsd-hackers@freebsd.org Fri Nov 23 19:10:19 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C1F38114F031; Fri, 23 Nov 2018 19:10:19 +0000 (UTC) (envelope-from jhs@berklix.com) Received: from land.berklix.org (land.berklix.org [144.76.10.75]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "land.berklix.org", Issuer "land.berklix.org" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id D69BA70A6D; Fri, 23 Nov 2018 19:10:17 +0000 (UTC) (envelope-from jhs@berklix.com) Received: from mart.js.berklix.net (p2E52CE62.dip0.t-ipconnect.de [46.82.206.98]) (authenticated bits=0) by land.berklix.org (8.15.2/8.15.2) with ESMTPSA id wANJ7eri013764 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 23 Nov 2018 19:07:44 GMT (envelope-from jhs@berklix.com) Received: from fire.js.berklix.net (fire.js.berklix.net [192.168.91.41]) by mart.js.berklix.net (8.14.3/8.14.3) with ESMTP id wANJ7ZKi058456; Fri, 23 Nov 2018 20:07:36 +0100 (CET) (envelope-from jhs@berklix.com) Received: from fire.js.berklix.net (localhost [127.0.0.1]) by fire.js.berklix.net (8.14.7/8.14.7) with ESMTP id wANJ7HYf058666; Fri, 23 Nov 2018 20:07:29 +0100 (CET) (envelope-from jhs@berklix.com) Message-Id: <201811231907.wANJ7HYf058666@fire.js.berklix.net> cc: soralx@cydem.org, freebsd-fs@freebsd.org, freebsd-hackers@freebsd.org Subject: Re: [bug] fsck refuses to repair damaged UFS using backup superblock From: "Julian H. Stacey" Organization: http://berklix.eu BSD Unix Linux Consultants, Munich Germany User-agent: EXMH on FreeBSD http://berklix.eu/free/ X-From: http://www.berklix.eu/~jhs/ In-reply-to: Your message "Fri, 23 Nov 2018 02:17:20 +0100." <201811230117.wAN1HKAT037185@fire.js.berklix.net> Date: Fri, 23 Nov 2018 20:07:17 +0100 X-Rspamd-Queue-Id: D69BA70A6D X-Spamd-Result: default: False [3.05 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-0.40)[-0.397,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_SPAM_SHORT(0.76)[0.756,0]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; DMARC_NA(0.00)[berklix.com]; AUTH_NA(1.00)[]; HAS_ORG_HEADER(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[cached: land.berklix.com]; NEURAL_SPAM_LONG(0.27)[0.267,0]; MISSING_TO(2.00)[]; R_SPF_NA(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[98.206.82.46.zen.spamhaus.org : 127.0.0.10]; R_DKIM_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:24940, ipnet:144.76.0.0/16, country:DE]; IP_SCORE(-0.46)[ipnet: 144.76.0.0/16(0.54), asn: 24940(-2.84), country: DE(-0.01)]; FROM_EQ_ENVFROM(0.00)[] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Nov 2018 19:10:19 -0000 "Julian H. Stacey" wrote: > Hi soralx@cydem.org, > Added cc: to ensure file system specialists see this. > > Please CC, as I am not subscribed. An over active spam filter bounced my CC to soralx@cydem.org so unless he/she thinks to subscribes, a chance any other CC may not be seen. Cheers, Julian -- Julian Stacey, Computer Consultant, Systems Engineer, BSD Linux Unix, Munich. Brexit referendum stole 3,700,000 votes from Brits abroad, inc. 700,000 in EU PM lied it's democratic in Article 50 http://exitbrexit.uk/brexit/#lie Campaign lies, criminal funded; Markets, jobs & pound down; New Referendum! From owner-freebsd-hackers@freebsd.org Fri Nov 23 22:42:00 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4947411333DA for ; Fri, 23 Nov 2018 22:42:00 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-io1-xd32.google.com (mail-io1-xd32.google.com [IPv6:2607:f8b0:4864:20::d32]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0597F77DC6 for ; Fri, 23 Nov 2018 22:41:58 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-io1-xd32.google.com with SMTP id l14so9587683ioj.5 for ; Fri, 23 Nov 2018 14:41:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=GsPYOcyxa1pH3/7XpC4OeJUByDW0kXPz2zx0fyHu+oA=; b=AGx/3n0fLWXX2sHj3HTM4JPI+CMevIOyx5PRjbe4h5s+El5uNN7OMcYUsABvQ+ae8T cK8rJrDOGOuBqL2j99EdsIPXWRsXag2e/g1HIWqxo5EUdZzQBQKQhzrB67jUPaQSkfYN CGDYrPT6QSFuSFMjdUPKTqh1FNRb7jSPX1QeZ8MDALBa2qk9oHeNkbqOQC4Y0ZKPdKLl g7nFqhLhaxItC9JFxYilbCvM7hQcHbDEFkZPFWjoE00grRPTfTfjCWwj7j+WLuZ8N8Bp tFfsz2qZdkoI8pti4Td6cR0sGagz6f7TYqJaqNuN0tmkh1Z0gl9IwNssRKwH9HgHN6+V L4Jg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=GsPYOcyxa1pH3/7XpC4OeJUByDW0kXPz2zx0fyHu+oA=; b=FCmgyAZ7jR9fwGA4gGqb5g65AbjPxnx9hxU7oJHFHan0VoaipmFwtRxXu35g9m4ldK 4KkyN2E3VVLzn4l2V1IbARxh07s7kshOXrS+iCi/7r2uIkWxMdFTvKVGcjmAosAFY/uh 1t+ap5v/eJ7F1Uc3/Z46H0XUlVoCXl5420rsVEu3o3L/LiDfQOiKQH9dHVVWcv01Ju4O PbBPHPv4PNLQ2SeLrvVJs5+ZjSNweA/1x+CN/qiTGK3j6u4tIFLWTad8Q945KGDpLcZY MIxTRo7LwfQSjwxqy8gWNx6pKvtI5Lc6HDV/WNjSkyS/TuXlIZP5xVnOZQoDXDDjb7l2 iytw== X-Gm-Message-State: AA+aEWYxNLYzxCIeLo4/jHwSLVAwVoAo5q+3F4+ZjZsrHDODZ1J84dju +zS3j6uwh14ZrRlWoJLyPWapZfeRf+ha+EwvEiJiMQ== X-Google-Smtp-Source: AFSGD/Us+lGDorAA9J7tThFiJeqmX8gFrNtBoyyin4AxEudrafFnEf1VtBZkCBKQZn1KTC7W6e9reUPkZ1GWqQU4IFo= X-Received: by 2002:a5e:d808:: with SMTP id l8mr3054314iok.299.1543012918187; Fri, 23 Nov 2018 14:41:58 -0800 (PST) MIME-Version: 1.0 References: <201811231600.wANG0wHc083199@pdx.rh.CN85.dnsmgr.net> In-Reply-To: <201811231600.wANG0wHc083199@pdx.rh.CN85.dnsmgr.net> From: Warner Losh Date: Fri, 23 Nov 2018 15:41:46 -0700 Message-ID: Subject: Re: TRIM utility To: "Rodney W. Grimes" Cc: Lev Serebryakov , FreeBSD Hackers , Poul-Henning Kamp , Eugene Grosbein X-Rspamd-Queue-Id: 0597F77DC6 X-Spamd-Result: default: False [-3.69 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.998,0]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-0.98)[-0.984,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; RCPT_COUNT_FIVE(0.00)[5]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; MX_GOOD(-0.01)[ALT1.aspmx.l.google.com,aspmx.l.google.com,ALT2.aspmx.l.google.com]; RCVD_IN_DNSWL_NONE(0.00)[2.3.d.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; NEURAL_HAM_SHORT(-0.98)[-0.983,0]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; RCVD_TLS_LAST(0.00)[]; IP_SCORE(-0.72)[ipnet: 2607:f8b0::/32(-2.02), asn: 15169(-1.48), country: US(-0.09)]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Server: mx1.freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Nov 2018 22:42:00 -0000 On Fri, Nov 23, 2018, 9:04 AM Rodney W. Grimes < freebsd-rwg@pdx.rh.cn85.dnsmgr.net wrote: > > On 23.11.2018 14:19, Poul-Henning Kamp wrote: > > > > >>> Currently it has four options, all of them are, hmm, optional: > > > Isn't this the kind of thing that dd(1) should learn about instead ? > > One utility to done one thing very well? :-) > > > > dd(1) is way overloaded, IMHO. > > I agree here, we do too much of trying to shoe horn things > into existing utilities then we end up with a command parser > that only a mother could love. > > trim, hdtrim, blktrim, camtrim, any of them > are fine, fstrim is bad, this is not a filesystem op, > too bad the next thing that comes > along that is "trim" like well have to pick > something other than trim. > Actually, you can now do the disk delete ioctl on a file range, and the putative trim program does that... but we've settled on trim I think. I might ask would it be horribly hard to access the > "secure erase" feature from this utility? Yes. It would. That's hard with the current storage stack to do via the disk interface. And often the underlying protocols do not support partial ranges. There is no good way to do this with buf/bio interface we have. So it is a really bad match all the way around. Or do we > have another that can easily get at that function, > that is usually the prefered vendor specific method > to "trim" the complete drive, often restoring badly > leveled SSD's to a performant and usable state. > Camcontrol already supports secure erase for both SCSI and ATA drives. And sanitize for SCSI (an alternative way to do the same thing to reset the ssd's FLT). It bypasses the disk interface and sends raw protocol commands via the pass interface. I do this all the time to rehab drives, do diagnosis of vendor issues or scrub ssds I'm sending to third parties. Warner -- > Rod Grimes > rgrimes@freebsd.org > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > From owner-freebsd-hackers@freebsd.org Sat Nov 24 01:25:06 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 30456113739C for ; Sat, 24 Nov 2018 01:25:06 +0000 (UTC) (envelope-from kaduk@mit.edu) Received: from dmz-mailsec-scanner-4.mit.edu (dmz-mailsec-scanner-4.mit.edu [18.9.25.15]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5566C7CFF2 for ; Sat, 24 Nov 2018 01:25:04 +0000 (UTC) (envelope-from kaduk@mit.edu) X-AuditID: 1209190f-4bbff700000047cf-c8-5bf8a73cbd20 Received: from mailhub-auth-4.mit.edu ( [18.7.62.39]) (using TLS with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by dmz-mailsec-scanner-4.mit.edu (Symantec Messaging Gateway) with SMTP id ED.93.18383.D37A8FB5; Fri, 23 Nov 2018 20:19:57 -0500 (EST) Received: from outgoing.mit.edu (OUTGOING-AUTH-1.MIT.EDU [18.9.28.11]) by mailhub-auth-4.mit.edu (8.14.7/8.9.2) with ESMTP id wAO1Jrsr017585; Fri, 23 Nov 2018 20:19:54 -0500 Received: from kduck.kaduk.org (24-107-191-124.dhcp.stls.mo.charter.com [24.107.191.124]) (authenticated bits=56) (User authenticated as kaduk@ATHENA.MIT.EDU) œby outgoing.mit.edu (8.14.7/8.12.4) with ESMTP id wAO1JnQl022135 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 23 Nov 2018 20:19:52 -0500 Date: Fri, 23 Nov 2018 19:19:49 -0600 From: Benjamin Kaduk To: soralx@cydem.org Cc: freebsd-hackers@freebsd.org Subject: Re: [bug] fsck refuses to repair damaged UFS using backup superblock Message-ID: <20181124011948.GD68416@kduck.kaduk.org> References: <20181120053000.56fbee6b@mscad14> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20181120053000.56fbee6b@mscad14> User-Agent: Mutt/1.9.1 (2017-09-22) X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrKIsWRmVeSWpSXmKPExsUixG6nrmu7/Ee0wbkNqhbbN/9jtHj49ACj A5PHm3nzmD1mfJrPEsAUxWWTkpqTWZZapG+XwJXR80qx4DdTxe+tnawNjFuZuhg5OSQETCQe vLrC1sXIxSEksIZJ4vKOc4wQzkZGiWdL9rFDOHeZJNru32QDaWERUJX4tfY0mM0moCLR0H2Z GcQWERCR+LLmKVicWUBe4uH2MywgtrCAr8SDH/+A4hwcvEDrdqwSBAkLCehKTL40F6yEV0BQ 4uTMJywQrVoSN/69ZAIpZxaQllj+jwMkzCmgJ9G2dyc7iC0qoCyxt+8Q+wRGgVlIumch6Z6F 0L2AkXkVo2xKbpVubmJmTnFqsm5xcmJeXmqRrolebmaJXmpK6SZGUIhySvLvYJzT4H2IUYCD UYmH14D5R7QQa2JZcWXuIUZJDiYlUd5b84FCfEn5KZUZicUZ8UWlOanFhxglOJiVRHi/OQDl eFMSK6tSi/JhUtIcLErivL9EHkcLCaQnlqRmp6YWpBbBZGU4OJQkeHmXATUKFqWmp1akZeaU IKSZODhBhvMADY8HqeEtLkjMLc5Mh8ifYtTleLfg/3RmIZa8/LxUKXFeG5AiAZCijNI8uDmg 1CKRvb/mFaM40FvCvA1Lgap4gGkJbtIroCVMQEvk538HWVKSiJCSamDccP6FS7CKrXt57k1L j8361zbvsVnX3TDl7g7J6SuXWjZ3zjCcNWEpo6HSzj/c3k/d1h/PcHP6Pue5cYed6nuDT7Om XDP4ejLtrvWam+EXIkzXi26pTaxefKD654GABvU3O/ad+Lpq8RR73rPJl2d+O/z92LXzz98o POdbGL3rYtypu5dtJd/dYFZiKc5INNRiLipOBAC6GqbTCAMAAA== X-Rspamd-Queue-Id: 5566C7CFF2 X-Spamd-Result: default: False [0.99 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.93)[-0.926,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:18.9.25.0/24]; NEURAL_HAM_LONG(-0.77)[-0.765,0]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; DMARC_NA(0.00)[mit.edu]; NEURAL_HAM_SHORT(-0.73)[-0.729,0]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_MED(-0.20)[15.25.9.18.list.dnswl.org : 127.0.11.2]; RCPT_COUNT_TWO(0.00)[2]; RCVD_ILLEGAL_CHARS(4.00)[]; MX_GOOD(-0.01)[dmz-mailsec-scanner-8.mit.edu,dmz-mailsec-scanner-5.mit.edu,dmz-mailsec-scanner-2.mit.edu,dmz-mailsec-scanner-7.mit.edu,dmz-mailsec-scanner-3.mit.edu,dmz-mailsec-scanner-1.mit.edu,dmz-mailsec-scanner-6.mit.edu,dmz-mailsec-scanner-4.mit.edu]; RECEIVED_SPAMHAUS_PBL(0.00)[124.191.107.24.zen.spamhaus.org : 127.0.0.10]; R_DKIM_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:3, ipnet:18.9.25.0/24, country:US]; IP_SCORE(-0.08)[asn: 3(-0.30), country: US(-0.09)]; FROM_EQ_ENVFROM(0.00)[] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Nov 2018 01:25:06 -0000 On Tue, Nov 20, 2018 at 05:30:00AM -0800, soralx@cydem.org wrote: > > Howdy! > > Since send-pr(1) is now gone, I guess the next option is to send a > message directly to the developers... I'm not sure where one would get that impression, given that https://www.freebsd.org/support/bugreports.html links to https://bugs.freebsd.org/bugzilla/enter_bug.cgi . -Ben From owner-freebsd-hackers@freebsd.org Sat Nov 24 07:30:09 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EC3E21144942 for ; Sat, 24 Nov 2018 07:30:08 +0000 (UTC) (envelope-from soralx@cydem.org) Received: from smtp.triumf.ca (smtp.triumf.ca [142.90.100.195]) by mx1.freebsd.org (Postfix) with ESMTP id ED4046C5B2 for ; Sat, 24 Nov 2018 07:30:07 +0000 (UTC) (envelope-from soralx@cydem.org) Received: from mscad14 (mscad14.triumf.ca [142.90.115.36]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.triumf.ca (Postfix) with ESMTP id 04A37F812; Fri, 23 Nov 2018 23:30:01 -0800 (PST) Date: Fri, 23 Nov 2018 23:30:00 -0800 From: To: Cc: Subject: Re: [bug] fsck refuses to repair damaged UFS using backup superblock Message-ID: <20181123233000.51f2af51@mscad14> In-Reply-To: <20181124011948.GD68416@kduck.kaduk.org> References: <20181120053000.56fbee6b@mscad14> <20181124011948.GD68416@kduck.kaduk.org> X-Mailer: Claws Mail 3.16.0 (GTK+ 2.24.32; amd64-portbld-freebsd9.3) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: ED4046C5B2 X-Spamd-Result: default: False [1.87 / 15.00]; ARC_NA(0.00)[]; R_SPF_FAIL(1.00)[-all]; NEURAL_HAM_MEDIUM(-0.06)[-0.057,0]; IP_SCORE(-0.02)[country: CA(-0.09)]; NEURAL_SPAM_SHORT(0.12)[0.119,0]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; DMARC_NA(0.00)[cydem.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[spamtrap.bmcorp.ca,spamtrap.sorokin.ca]; RCPT_COUNT_TWO(0.00)[2]; FROM_NO_DN(0.00)[]; NEURAL_SPAM_LONG(0.34)[0.338,0]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MID_RHS_NOT_FQDN(0.50)[]; ASN(0.00)[asn:36391, ipnet:142.90.0.0/16, country:CA]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Nov 2018 07:30:09 -0000 Ben, > > Since send-pr(1) is now gone, I guess the next option is to send a > > message directly to the developers... > I'm not sure where one would get that impression, given that > https://www.freebsd.org/support/bugreports.html links to > https://bugs.freebsd.org/bugzilla/enter_bug.cgi . I have tried the web-based bugzilla, but was greeted with a log-in page when I went to the bug report address & no form to report bugs. > -Ben -- [SorAlx] ridin' VN2000 Classic LT From owner-freebsd-hackers@freebsd.org Sat Nov 24 09:27:06 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CF64911489C3 for ; Sat, 24 Nov 2018 09:27:05 +0000 (UTC) (envelope-from miltonott@fastmail.com) Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com [66.111.4.28]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9B9AB707F2 for ; Sat, 24 Nov 2018 09:27:03 +0000 (UTC) (envelope-from miltonott@fastmail.com) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.nyi.internal (Postfix) with ESMTP id 0FDD521ACE for ; Sat, 24 Nov 2018 04:27:03 -0500 (EST) Received: from mailfrontend2 ([10.202.2.163]) by compute5.internal (MEProxy); Sat, 24 Nov 2018 04:27:03 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastmail.com; h= date:from:message-id:to:subject:in-reply-to:references; s=fm1; bh=NTR28HjY0Muz2V+WAJoO4PpY5Y7Vos8aCh9PTQbICbs=; b=lwuHgeaVBx8n i5FmavZ3HG/XhNdyNATPJdstYyx5b9VaDpA0NbSNJzD+3C4dKFUYf7QJzfvva/se rhHs4JPenZuS6lnDXzVO4DU40ILzV40KdzjC1m80EluaDOmSFB3N2DCoAaAMjzMs yRJl6nsoIurK8frYaZ8OJjX2YM954DsPbC1cQbh3WYNQBvgKLxyB2NCxG6jFJTi/ RULGON5J8sT/F+lOeQwq4mKrJtfN/0U/s9v9WaeestYgD9xbfR8UKHGmoPJkBS9I ArWmTjbb4vMN5VhncIXi5x3c+eMx5cvF1F1Xx857wOeBeYrM+BnY+NendpWY18yu iZAOKemFMw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=date:from:in-reply-to:message-id :references:subject:to:x-me-proxy:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; bh=NTR28HjY0Muz2V+WAJoO4PpY5Y7Vo s8aCh9PTQbICbs=; b=OpQ4GL1CAvx4pytSaHKvEtUWax7pqfd1WqAwHCrhu1nRx SaeJMQQA5eIcyFzkOhnhS4CZFd0g9kGAB5zgCLxsHV7or3DIWp6B/5k0mjVi49K5 DbHYD1P4v0Uk6KZubd3ptJa7wWlSS8oEJd2QwZmAo1i0hssFKOAo5UZq+JySh5WK ITo+6XunEX1zSBvnTrKoLYqMEHh8ZqkPUk10G/vX/JmzKvXr2oRzNMy+JtuAFSDj gV/6EY6rc52H6Mmt7Azwalfcef/kpjm7E4NF9gfBGMK1yke7Xhl0eSq5ZNYPyZ+R NojumzdIXcLJFjpPCDErRy2oJ/P+fDtJtIB7bXJ6Q== X-ME-Sender: X-ME-Proxy: Received: from euler.miltonslab.com (unknown [58.170.223.67]) by mail.messagingengine.com (Postfix) with ESMTPA id 1A8FE102E8 for ; Sat, 24 Nov 2018 04:27:01 -0500 (EST) Received: (from root@localhost) by euler.miltonslab.com (8.15.2/8.15.2) id wAO9QvOC003283; Sat, 24 Nov 2018 09:26:57 GMT (envelope-from miltonott) Date: Sat, 24 Nov 2018 09:26:57 GMT From: miltonott Message-Id: <201811240926.wAO9QvOC003283@euler.miltonslab.com> To: freebsd-hackers@freebsd.org Subject: Re: Adding support for MosChip 9912 PCIe (serial/parallel) cards In-Reply-To: 201811170804.wAH84AM1001846@euler.miltonslab.com References: 201811170804.wAH84AM1001846@euler.miltonslab.com X-Rspamd-Queue-Id: 9B9AB707F2 X-Spamd-Result: default: False [-7.73 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[fastmail.com,messagingengine.com]; RCVD_COUNT_FIVE(0.00)[5]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:66.111.4.28]; FREEMAIL_FROM(0.00)[fastmail.com]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; IP_SCORE(-3.63)[ip: (-9.46), ipnet: 66.111.4.0/24(-4.71), asn: 11403(-3.89), country: US(-0.09)]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_TRACE(0.00)[fastmail.com:+,messagingengine.com:+]; DMARC_POLICY_ALLOW(-0.50)[fastmail.com,none]; MX_GOOD(-0.01)[in2-smtp.messagingengine.com,in1-smtp.messagingengine.com,in2-smtp.messagingengine.com,in1-smtp.messagingengine.com,in2-smtp.messagingengine.com,in1-smtp.messagingengine.com,in2-smtp.messagingengine.com,in1-smtp.messagingengine.com]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.99)[-0.990,0]; RCVD_IN_DNSWL_LOW(-0.10)[28.4.111.66.list.dnswl.org : 127.0.5.1]; RECEIVED_SPAMHAUS_PBL(0.00)[67.223.170.58.zen.spamhaus.org : 127.0.0.11]; FREEMAIL_ENVFROM(0.00)[fastmail.com]; ASN(0.00)[asn:11403, ipnet:66.111.4.0/24, country:US]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[] X-Rspamd-Server: mx1.freebsd.org X-Mailman-Approved-At: Sat, 24 Nov 2018 11:15:47 +0000 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Nov 2018 09:27:06 -0000 on Tue Mar 6 07:27:36 UTC 2018 -Andre wrote: >> I am now using this diff to access the MCS9912. I am (mis)using >> puc to initialise whatever is needed for the printer port to work >> and let ppc attach to puc. As puc does not attach to single port >> devices, I have removed this check. uart works by simply adding >> the device. This is all quite ugly but it works... > miltonott wrote: > My lord, may I please see the device.hints file you are now using. I am now able to establish a login session on a serial port using the additions to /usr/src/sys/dev/uart/uart_bus_pci.c provided by -Andre on March 2018. The device appearance in the output of pciconf gives: uart2@pci0:3:0:0: class=0x070002 card=0x1000a000 chip=0x99129710 rev=0x00 hdr=0x00 vendor = 'MosChip Semiconductor Technology Ltd.' device = 'PCIe 9912 Multi-I/O Controller' class = simple comms subclass = UART uart3@pci0:3:0:1: class=0x070002 card=0x1000a000 chip=0x99129710 rev=0x00 hdr=0x00 vendor = 'MosChip Semiconductor Technology Ltd.' device = 'PCIe 9912 Multi-I/O Controller' class = simple comms subclass = UART /var/run/dmesg.boot now gives: uart2: port 0xe030-0xe037 mem 0xf7c05000-0xf7c05fff,0xf7c04000-0xf7c04fff irq 17 at device 0.0 on pci2 uart2: fast interrupt uart2: PPS capture mode: DCDinvalid random: harvesting attach, 8 bytes (4 bits) from uart2 uart3: port 0xe020-0xe027 mem 0xf7c03000-0xf7c03fff,0xf7c02000-0xf7c02fff irq 18 at device 0.1 on pci2 uart3: fast interrupt uart3: PPS capture mode: DCDinvalid $ /usr/bin/cu -l /dev/cuau3 gives: FreeBSD/amd64 (fallacy) (ttyu0) login: Thanking you -Andre. From owner-freebsd-hackers@freebsd.org Sat Nov 24 14:44:16 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4FAC11150803 for ; Sat, 24 Nov 2018 14:44:16 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: from puchar.net (puchar.net [194.1.144.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "puchar.net", Issuer "puchar.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 271357AFAA; Sat, 24 Nov 2018 14:44:14 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.15.2/8.15.2) with ESMTPS id wAOEiBSK020061 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sat, 24 Nov 2018 15:44:11 +0100 (CET) (envelope-from puchar-wojtek@puchar.net) Received: from localhost (puchar-wojtek@localhost) by puchar.net (8.15.2/8.15.2/Submit) with ESMTP id wAOEi5Ep020058; Sat, 24 Nov 2018 15:44:05 +0100 (CET) (envelope-from puchar-wojtek@puchar.net) Date: Sat, 24 Nov 2018 15:44:05 +0100 (CET) From: Wojciech Puchar To: Warner Losh cc: "Rodney W. Grimes" , FreeBSD Hackers , Poul-Henning Kamp , Lev Serebryakov , Eugene Grosbein Subject: Re: TRIM utility In-Reply-To: Message-ID: References: <201811231600.wANG0wHc083199@pdx.rh.CN85.dnsmgr.net> User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Rspamd-Queue-Id: 271357AFAA X-Spamd-Result: default: False [-2.05 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.84)[-0.842,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-0.89)[-0.886,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[puchar.net]; RCPT_COUNT_FIVE(0.00)[6]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[puchar.net]; NEURAL_HAM_SHORT(-0.01)[-0.015,0]; RCVD_IN_DNSWL_NONE(0.00)[90.144.1.194.list.dnswl.org : 127.0.10.0]; IP_SCORE(0.00)[country: PL(0.02)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:43476, ipnet:194.1.144.0/24, country:PL]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Nov 2018 14:44:16 -0000 > > Yes. It would. That's hard with the current storage stack to do via the > disk interface. And often the underlying protocols do not support partial > ranges. There is no good way to do this with buf/bio interface we have. So what is an actual difference between "secure erase" and trimming whole disk? > it is a really bad match all the way around. > > Or do we >> have another that can easily get at that function, >> that is usually the prefered vendor specific method >> to "trim" the complete drive, often restoring badly >> leveled SSD's to a performant and usable state. >> > > Camcontrol already supports secure erase for both SCSI and ATA drives. And > sanitize for SCSI (an alternative way to do the same thing to reset the > ssd's FLT). It bypasses the disk interface and sends raw protocol commands > via the pass interface. I do this all the time to rehab drives, do > diagnosis of vendor issues or scrub ssds I'm sending to third parties. > > Warner > > -- >> Rod Grimes >> rgrimes@freebsd.org >> _______________________________________________ >> freebsd-hackers@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers >> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" >> > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > > From owner-freebsd-hackers@freebsd.org Sat Nov 24 16:09:06 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6711D1152E81 for ; Sat, 24 Nov 2018 16:09:06 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from tensor.andric.com (tensor.andric.com [IPv6:2001:470:7a58:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "tensor.andric.com", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E13727F29E; Sat, 24 Nov 2018 16:09:05 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from [IPv6:2001:470:7a58::d1da:f838:7f36:60b7] (unknown [IPv6:2001:470:7a58:0:d1da:f838:7f36:60b7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by tensor.andric.com (Postfix) with ESMTPSA id 1DA371D7A1; Sat, 24 Nov 2018 17:09:04 +0100 (CET) From: Dimitry Andric Message-Id: Content-Type: multipart/signed; boundary="Apple-Mail=_7F4FE982-8BC1-4DE9-874D-A8A01FD3FE8B"; protocol="application/pgp-signature"; micalg=pgp-sha1 Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: TRIM utility Date: Sat, 24 Nov 2018 17:08:59 +0100 In-Reply-To: Cc: Warner Losh , FreeBSD Hackers , Poul-Henning Kamp , Lev Serebryakov , "Rodney W. Grimes" , Eugene Grosbein To: Wojciech Puchar References: <201811231600.wANG0wHc083199@pdx.rh.CN85.dnsmgr.net> X-Mailer: Apple Mail (2.3445.9.1) X-Rspamd-Queue-Id: E13727F29E X-Spamd-Result: default: False [2.28 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_SPAM_SHORT(0.64)[0.642,0]; NEURAL_SPAM_MEDIUM(0.86)[0.865,0]; NEURAL_SPAM_LONG(0.78)[0.777,0]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Nov 2018 16:09:06 -0000 --Apple-Mail=_7F4FE982-8BC1-4DE9-874D-A8A01FD3FE8B Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii On 24 Nov 2018, at 15:44, Wojciech Puchar wrote: >=20 >> Yes. It would. That's hard with the current storage stack to do via = the >> disk interface. And often the underlying protocols do not support = partial >> ranges. There is no good way to do this with buf/bio interface we = have. So >=20 > what is an actual difference between "secure erase" and trimming whole = disk? It depends a lot on the device. Some devices encrypt all blocks automatically, and a Secure Erase command might just throw away the key, not go over all the data and actively wipe it. Most SSDs will likely also trim all the blocks to be erased, since users have come to expect the SSD performance to go back to the 'out of box' level, after such an operation. If you are lucky, the manufacturer's documentation will explain the specifics, but don't count on it. :) -Dimitry --Apple-Mail=_7F4FE982-8BC1-4DE9-874D-A8A01FD3FE8B Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.2 iF0EARECAB0WIQR6tGLSzjX8bUI5T82wXqMKLiCWowUCW/l3mwAKCRCwXqMKLiCW o7nAAJ0Z3FrogUA0TA5b3k8KEPQ0jgoZ5QCeOOWf/uPoOyIaykndtARE8vw7xwI= =yOxB -----END PGP SIGNATURE----- --Apple-Mail=_7F4FE982-8BC1-4DE9-874D-A8A01FD3FE8B-- From owner-freebsd-hackers@freebsd.org Sat Nov 24 17:05:23 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 690CF110466C for ; Sat, 24 Nov 2018 17:05:23 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9983F82F7B; Sat, 24 Nov 2018 17:05:22 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (localhost [127.0.0.1]) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3) with ESMTP id wAOH5KJe088485; Sat, 24 Nov 2018 09:05:20 -0800 (PST) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: (from freebsd-rwg@localhost) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3/Submit) id wAOH5K8j088484; Sat, 24 Nov 2018 09:05:20 -0800 (PST) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201811241705.wAOH5K8j088484@pdx.rh.CN85.dnsmgr.net> Subject: Re: TRIM utility In-Reply-To: To: Dimitry Andric Date: Sat, 24 Nov 2018 09:05:20 -0800 (PST) CC: Wojciech Puchar , Warner Losh , FreeBSD Hackers , Poul-Henning Kamp , Lev Serebryakov , Eugene Grosbein X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-Rspamd-Queue-Id: 9983F82F7B X-Spamd-Result: default: False [0.47 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.08)[-0.078,0]; FROM_HAS_DN(0.00)[]; NEURAL_SPAM_SHORT(0.02)[0.019,0]; NEURAL_HAM_LONG(-0.34)[-0.341,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[dnsmgr.net]; AUTH_NA(1.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[cached: pdx.rh.CN85.dnsmgr.net]; RCPT_COUNT_SEVEN(0.00)[7]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:13868, ipnet:69.59.192.0/19, country:US]; MID_RHS_MATCH_FROM(0.00)[]; IP_SCORE(-0.02)[country: US(-0.09)] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Nov 2018 17:05:23 -0000 > On 24 Nov 2018, at 15:44, Wojciech Puchar wrote: > > > >> Yes. It would. That's hard with the current storage stack to do via the > >> disk interface. And often the underlying protocols do not support partial > >> ranges. There is no good way to do this with buf/bio interface we have. So > > > > what is an actual difference between "secure erase" and trimming whole disk? I wonder if any drive recognises a single trim command that covers the whole drive, or if that is even possible to do. Or if it recognizes that all blocks are infact in a free/trimmed state (should be easy if they have a total block inuse counter, just watch for it to hit 0). > It depends a lot on the device. Some devices encrypt all blocks > automatically, and a Secure Erase command might just throw away the key, > not go over all the data and actively wipe it. > > Most SSDs will likely also trim all the blocks to be erased, since users > have come to expect the SSD performance to go back to the 'out of box' > level, after such an operation. I do not think they actually "trim" anything, I am pretty sure they just do a full FDT re-initializse and keep the block use counters for future allocations, in effect this looks like you trimmed the whole drive but usually completed in just a fex seconds or less. Some manufactures actually recommend this procedure to revive a drives performance that has degraded over time. > > If you are lucky, the manufacturer's documentation will explain the > specifics, but don't count on it. :) And there is that problem :-( -- Rod Grimes rgrimes@freebsd.org From owner-freebsd-hackers@freebsd.org Sat Nov 24 17:16:04 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 057CF1104E0E for ; Sat, 24 Nov 2018 17:16:04 +0000 (UTC) (envelope-from ian@freebsd.org) Received: from outbound1.eu.mailhop.org (outbound1.eu.mailhop.org [52.28.251.132]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3D71F83A62 for ; Sat, 24 Nov 2018 17:16:03 +0000 (UTC) (envelope-from ian@freebsd.org) ARC-Seal: i=1; a=rsa-sha256; t=1543079754; cv=none; d=outbound.mailhop.org; s=arc-outbound20181012; b=D3CG3EG9WGhxlKhtXDnXnNjKiBb68c35d3vK8JXv5JcLqg0FoNTlRZtr4h8HW1Mgfru2vWCWhoKZu OndoPVy5Czc6WhvdR9zqN9h3Y37mGRaLT/AEW941bRC2LZXYsXeTRSbpgkP89n+V/DnvJaclDxpbQG H4vJpdzxF1Bx5yEd8uXd4oQTfB0pNY8uMK31jekvVM/g0vsYoLtrQhf47e/549Oa7Nexc61Sejn/PS wEJDzytXLy+RsIgQbjF6NBaUEoWlyv0UHhaOUoyITavA6eXzlvMC7GFt9D6l+ODAX0Vb+vsJDeyS1m Qb9tWAhWnt6mCEhCemm4+g3XVqvvIOg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=outbound.mailhop.org; s=arc-outbound20181012; h=content-transfer-encoding:mime-version:content-type:references:in-reply-to: date:cc:to:from:subject:message-id:dkim-signature:from; bh=wQkwjkU34N3gxGXRlTRbFm2uaeKhB/Eu2fiapvvuO7w=; b=KnM1skiSXj+4DgEMNnD1sdL0mJntx8ZT3MEllUk0OqXmrMVyNT1Dt0Jzga2c9hI0ZLLKzeEOdZIJC bVpQCnhO2vdQNa+gWONHnjq30QtWZuhKR+w63on8HUiXjwEIOv6VOH+hlJNlt2JSsRF4pvgdojM7AL EX31orFLGEXJhAsYg/web1AG/Qw/lL/pkazmkSUA3519FLxmPdTAIOy9RdforFv4VfuBtbVx71xOEj qHcfbotbsnsinTUEJb/0KU7o/P+KBbGpNC/BdW6CjW51HQySogA5vs6m8eUwPLT5d0hXEZo+Bh5ZHM NLZprLh0Vj/jBmREs7YPRSn17ziELvg== ARC-Authentication-Results: i=1; outbound3.eu.mailhop.org; spf=softfail smtp.mailfrom=freebsd.org smtp.remote-ip=67.177.211.60; dmarc=none header.from=freebsd.org; arc=none header.oldest-pass=0; DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=outbound.mailhop.org; s=dkim-high; h=content-transfer-encoding:mime-version:content-type:references:in-reply-to: date:cc:to:from:subject:message-id:from; bh=wQkwjkU34N3gxGXRlTRbFm2uaeKhB/Eu2fiapvvuO7w=; b=bHWg4jDB7ZpqfyhETVYZ0QQrRqIVEtamKaEl4eXR8GfQ0HTeaG/mgWsshJn2vgTL0WMrwPPFJf7pF exg//YyuH5ptmu0nv1hvRSGnglrgBmgpuabmmZE3cprL6VUYLUV1emX8PlAAFEgVWtQr5Jm6qXYnPx 2jfxlGWuSplbZbVDCgTxlBXSaJkHHVO8b3kdj1U3CDXL0zZB57lV2YL025AI4+2klpD3v+77Pc8YNL eRZz1hJ37cp9jnMOc6NX1uvbMuzqN8FBC6K1t1kp1r0v3NJkq3jP/4Qi9giNrbCkhK96Gvmhc8t/28 1gZM2tgR6GqyRoeT1y/Zz2xu+EYy3Jg== X-MHO-RoutePath: aGlwcGll X-MHO-User: 95cc4eb8-f00c-11e8-8a28-a1efd8da9a94 X-Report-Abuse-To: https://support.duocircle.com/support/solutions/articles/5000540958-duocircle-standard-smtp-abuse-information X-Originating-IP: 67.177.211.60 X-Mail-Handler: DuoCircle Outbound SMTP Received: from ilsoft.org (unknown [67.177.211.60]) by outbound3.eu.mailhop.org (Halon) with ESMTPSA id 95cc4eb8-f00c-11e8-8a28-a1efd8da9a94; Sat, 24 Nov 2018 17:15:50 +0000 (UTC) Received: from rev (rev [172.22.42.240]) by ilsoft.org (8.15.2/8.15.2) with ESMTP id wAOHFlnk025921; Sat, 24 Nov 2018 10:15:47 -0700 (MST) (envelope-from ian@freebsd.org) Message-ID: <1543079747.36737.6.camel@freebsd.org> Subject: Re: 13-CURRENT: several GB swap being used despite plenty of free RAM From: Ian Lepore To: Stefan Blachmann , Cy Schubert Cc: Wojciech Puchar , freebsd-hackers Hackers , Rebecca Cran , Mark Johnston Date: Sat, 24 Nov 2018 10:15:47 -0700 In-Reply-To: References: <201811180154.wAI1smhg049214@slippy.cwsent.com> Content-Type: text/plain; charset="ISO-8859-1" X-Mailer: Evolution 3.18.5.1 FreeBSD GNOME Team Port Mime-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 3D71F83A62 X-Spamd-Result: default: False [1.78 / 15.00]; local_wl_from(0.00)[freebsd.org]; NEURAL_SPAM_SHORT(0.68)[0.683,0]; NEURAL_SPAM_MEDIUM(0.60)[0.599,0]; NEURAL_SPAM_LONG(0.50)[0.497,0]; ASN(0.00)[asn:16509, ipnet:52.28.0.0/16, country:US] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Nov 2018 17:16:04 -0000 On Sun, 2018-11-18 at 13:11 +0100, Stefan Blachmann wrote: > The inconveniences that the new swapping strategy causes are a regular > topic in the FreeBSD forums. > > Desktop users complain about lagginess, server users complain of long > delays because server processes intended to be kept in memory for > quick response times got swapped out and need to be swapped in again, > resulting in outrageously poor server performance in spite of plenty > of unused memory. > > Turning off swap completely, as Cy Schubert suggests, is strongly > discouraged in the forums, as it can lead to kernel panicking because > of being unable to swap out in critical kernel memory shortage > situations, leading to the risk of  very serious filesystem > corruption. > > However, Cy Schubert is probably right when stating that the new > swapping strategy resembles the 1960s-1980s industry's main swapping > strategy. > The bad thing is now, that nowadays memory is no longer scarce and > people can dimension their memory such that under normal circumstances > there will never be any need to swap. > > So I guess the unwillingness of the developer team to add an option > like "NoPreemptiveSwapping", which disables swapping out as long as > there is free physical memory available, is of psychological nature. > > Lacking such an option, there is still the possibility to use rctl to > disable swapping for particular users, processes, jails etc to > mitigate the problems caused by the new swapping strategy to some > degree. > Well that was a nice little rant, I hope you feel better. Perhaps it has cleared your mind enough to eventually realize that I gave you the command to do exactly what you sarcastically suggest "the developer team" is unwilling to implement. It's still there, in the quoted text below, not quite completely obscured by the typical word soup of irrelevancies from the usual source which sidetracked this thread into a discussion about swapping, which isn't involved in any way (and served with a side of mixed top and bottom posting to make it almost impossible to follow the conversation). -- Ian > On 11/18/18, Cy Schubert wrote: > > > > In message , Mark > > Millard via f > > reebsd-hackers writes: > > > > > > On 2018-Nov-17, at 16:13, Mark Johnston > > > wrote: > > > > > > > > > > > > On Sat, Nov 17, 2018 at 04:59:48PM -0700, Ian Lepore wrote: > > > > > > > > > > On Sat, 2018-11-17 at 22:52 +0100, Wojciech Puchar wrote: > > > > > > > > > > > > freebsd will not swap with that lots of free ram. > > > > > > but it's 90GB free NOW, how about before? > > > > > > > > > > > Your information is outdated. For at least a couple years now > > > > > (since > > > > > approximately the 10.1 - 10.2 timeframe is my vague > > > > > estimate), freebsd > > > > > will page out application memory that hasn't been referenced > > > > > for some > > > > > time, even when the system has no shortage of free memory at > > > > > all. > > > > No, FreeBSD will only ever swap when there is a free page > > > > shortage. > > > > The > > > > difference is that we now slowly age unreferenced pages into > > > > the > > > > inactive queue, which makes them candidates for pageout and > > > > subsequent > > > > eviction.  With pageout_update_period=0, anonymous memory won't > > > > get > > > > paged out unless there's a shortage of inactive pages, or an > > > > application > > > > calls madvise(MADV_DONTNEED) on a range of memory (which moves > > > > any > > > > backing pages to the inactive queue). > > > Swapping is built on top of paging as I understand. The system > > > can page without swapping but can not swap without (effectively) > > > paging to implement the swapping, if I understand right. If I > > > understand right, swapped-out means that kernel stacks have > > > been written out and have to be loaded back in RAM before the > > > process/threads can even run. (I might not understand.) > > > > > > If I've got that right, are there distinctions here for > > > paging that is not part of swapping vs. actual swapping > > > (and its use of paging)? Saying that something does not > > > swap does not necessarily imply that it does not page: > > > it still could have paging activity that does not include > > > moving the kernel stacks for the process to backing media? > > This is generally the old-school definition, IIRC third year comp > > sci, > > original BSD definition, which was also the way IBM defined it for > > MVS. > > (IBM also virtually swapped out address spaces, not written to > > DASD, as > > a means to deny CPU cycles through the scheduler not given the > > chance > > to consider the tasks (threads) in the address space). > > > > You can disable swapping by setting vm.swap_enabled=0. > > > > > > > > > > > At times I have trouble interpreting when wording goes back > > > and forth between swapping and paging, both for the intended > > > meaning and for the technical implications. > > > > > > > > > > > > > > > > > The advice I was recently given to revert to the old behavior > > > > > is: > > > > > > > > > >   sysctl vm.pageout_update_period=0 > > > > > > > > > > I've been using it on a couple systems here for a few days > > > > > now, and so > > > > > far results are promising, I am no longer seeing gratuitous > > > > > swapfile > > > > > usage on systems that have so much free physical ram that > > > > > they should > > > > > never need to page anything out. I haven't yet pushed one of > > > > > those > > > > > systems hard enough to check what happens when they do need > > > > > to start > > > > > proactively paging out inactive memory due to shortages -- it > > > > > could be > > > > > that turning off the new behavior has downsides for some > > > > > workloads. > > > > > > > > > > > > === > > > Mark Millard > > > marklmi at yahoo.com > > > ( dsl-only.net went > > > away in early 2018-Mar) > > > > > > _______________________________________________ > > > freebsd-hackers@freebsd.org mailing list > > > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > > > To unsubscribe, send any mail to > > > "freebsd-hackers-unsubscribe@freebsd.org" > > > > > -- > > Cheers, > > Cy Schubert > > FreeBSD UNIX:     Web:  http://www.FreeBSD.org > > > > The need of the many outweighs the greed of the few. > > > > > > _______________________________________________ > > freebsd-hackers@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freeb > > sd.org" > > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd > .org" From owner-freebsd-hackers@freebsd.org Sat Nov 24 17:47:20 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 781A3110EFE6 for ; Sat, 24 Nov 2018 17:47:20 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Received: from smtp-out-no.shaw.ca (smtp-out-no.shaw.ca [64.59.134.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "Client", Issuer "CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 43FC68494C; Sat, 24 Nov 2018 17:47:19 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Received: from spqr.komquats.com ([70.67.125.17]) by shaw.ca with ESMTPA id Qc1MgftzeP7x2Qc1OgFtgS; Sat, 24 Nov 2018 10:47:11 -0700 X-Authority-Analysis: v=2.3 cv=X9GD11be c=1 sm=1 tr=0 a=VFtTW3WuZNDh6VkGe7fA3g==:117 a=VFtTW3WuZNDh6VkGe7fA3g==:17 a=8nJEP1OIZ-IA:10 a=JHtHm7312UAA:10 a=6I5d2MoRAAAA:8 a=YxBL1-UpAAAA:8 a=CjxXgO3LAAAA:8 a=YVOhz5M6AAAA:8 a=SLG1KRGDAAAA:8 a=MUQfBbhXRRlCCULTAwgA:9 a=YRp9IFY1wmEaPIZV:21 a=k3DeAcpCFCNX51e_:21 a=wPNLvfGTeEIA:10 a=IjZwj45LgO3ly-622nXo:22 a=Ia-lj3WSrqcvXOmTRaiG:22 a=sbbTL3E6IKcx-RquDtO-:22 a=-TBaU1e9WpdkKBzYXnwo:22 Received: from slippy.cwsent.com (slippy [10.1.1.91]) by spqr.komquats.com (Postfix) with ESMTPS id AD71A19C3; Sat, 24 Nov 2018 09:47:07 -0800 (PST) Received: from slippy.cwsent.com (localhost [127.0.0.1]) by slippy.cwsent.com (8.15.2/8.15.2) with ESMTP id wAOHl63U079943; Sat, 24 Nov 2018 09:47:06 -0800 (PST) (envelope-from Cy.Schubert@cschubert.com) Received: from slippy (cy@localhost) by slippy.cwsent.com (8.15.2/8.15.2/Submit) with ESMTP id wAOHl4Kx079905; Sat, 24 Nov 2018 09:47:05 -0800 (PST) (envelope-from Cy.Schubert@cschubert.com) Message-Id: <201811241747.wAOHl4Kx079905@slippy.cwsent.com> X-Authentication-Warning: slippy.cwsent.com: cy owned process doing -bs X-Mailer: exmh version 2.8.0 04/21/2012 with nmh-1.7.1 Reply-to: Cy Schubert From: Cy Schubert X-os: FreeBSD X-Sender: cy@cwsent.com X-URL: http://www.cschubert.com/ To: Ian Lepore cc: Stefan Blachmann , Cy Schubert , Wojciech Puchar , freebsd-hackers Hackers , Rebecca Cran , Mark Johnston Subject: Re: 13-CURRENT: several GB swap being used despite plenty of free RAM In-Reply-To: Message from Ian Lepore of "Sat, 24 Nov 2018 10:15:47 -0700." <1543079747.36737.6.camel@freebsd.org> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Date: Sat, 24 Nov 2018 09:47:04 -0800 X-CMAE-Envelope: MS4wfBe0jdoPj9lmqjH9MyXQ4YdAnInb8nFY/qe5mwbmWHesqWmT+1AlFKagwjwAsbsc4IWm1WqKkQxsfgzihxb2n917hk1x+Vgz0Q6tG66dtmvmMzj5/UO1 0ylZsSsbUX+PJidNgu5zoX2LvbiixtJ/4fZKU06Biq5HRUjnkOcoXRWaSHqM7JMbkYcYqBXJc+9yzKlF5sLu/luZTdgErmKejPOpbF90YDxE+jt3LmTsl7H8 HuP4DG1EjrHQ0IKasJr6tK83CQJyBuNwszsoRVObJaYG8GyD3FdSfzn2Zjaz9C5WCaz6AoweuiCWfWMv+q44rvu/h7niF1aTT6P29udGi14= X-Rspamd-Queue-Id: 43FC68494C X-Spamd-Result: default: False [-3.59 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_FIVE(0.00)[5]; HAS_REPLYTO(0.00)[Cy.Schubert@cschubert.com]; FROM_HAS_DN(0.00)[]; MV_CASE(0.50)[]; IP_SCORE(-0.96)[ipnet: 64.59.128.0/20(-2.60), asn: 6327(-2.08), country: CA(-0.09)]; MIME_GOOD(-0.10)[text/plain]; HAS_XAW(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; REPLYTO_EQ_FROM(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[cached: spqr.komquats.com]; NEURAL_HAM_SHORT(-0.93)[-0.929,0]; RCPT_COUNT_SEVEN(0.00)[7]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[17.125.67.70.zen.spamhaus.org : 127.0.0.11]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:6327, ipnet:64.59.128.0/20, country:CA]; FREEMAIL_CC(0.00)[gmail.com]; RCVD_TLS_LAST(0.00)[]; RCVD_IN_DNSWL_LOW(-0.10)[12.134.59.64.list.dnswl.org : 127.0.5.1] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Nov 2018 17:47:20 -0000 In message <1543079747.36737.6.camel@freebsd.org>, Ian Lepore writes: > On Sun, 2018-11-18 at 13:11 +0100, Stefan Blachmann wrote: > > The inconveniences that the new swapping strategy causes are a regular > > topic in the FreeBSD forums. > > > > Desktop users complain about lagginess, server users complain of long > > delays because server processes intended to be kept in memory for > > quick response times got swapped out and need to be swapped in again, > > resulting in outrageously poor server performance in spite of plenty > > of unused memory. > > > > Turning off swap completely, as Cy Schubert suggests, is strongly > > discouraged in the forums, as it can lead to kernel panicking because > > of being unable to swap out in critical kernel memory shortage > > situations, leading to the risk of  very serious filesystem > > corruption. > > > > However, Cy Schubert is probably right when stating that the new > > swapping strategy resembles the 1960s-1980s industry's main swapping > > strategy. > > The bad thing is now, that nowadays memory is no longer scarce and > > people can dimension their memory such that under normal circumstances > > there will never be any need to swap. > > > > So I guess the unwillingness of the developer team to add an option > > like "NoPreemptiveSwapping", which disables swapping out as long as > > there is free physical memory available, is of psychological nature. > > > > Lacking such an option, there is still the possibility to use rctl to > > disable swapping for particular users, processes, jails etc to > > mitigate the problems caused by the new swapping strategy to some > > degree. > > > > Well that was a nice little rant, I hope you feel better. Perhaps it > has cleared your mind enough to eventually realize that I gave you the > command to do exactly what you sarcastically suggest "the developer > team" is unwilling to implement. > > It's still there, in the quoted text below, not quite completely > obscured by the typical word soup of irrelevancies from the usual > source which sidetracked this thread into a discussion about swapping, > which isn't involved in any way (and served with a side of mixed top > and bottom posting to make it almost impossible to follow the > conversation). > > -- Ian There was some question about swapping, which I had answered. My point was: this technology has been around for decades and that almost all operating systems manage memory in a similar fashion -- I believe my answer was a little more subtle and wordy, put bluntly: this is CS 101 and everyone here should understand it. It doesn't pay to be subtle. Bluntly: it's virtually done the same everywhere. So what is the problem? -- Cheers, Cy Schubert FreeBSD UNIX: Web: http://www.FreeBSD.org The need of the many outweighs the greed of the few. > > > On 11/18/18, Cy Schubert wrote: > > > > > > In message , Mark > > > Millard via f > > > reebsd-hackers writes: > > > > > > > > On 2018-Nov-17, at 16:13, Mark Johnston > > > > wrote: > > > > > > > > > > > > > > > On Sat, Nov 17, 2018 at 04:59:48PM -0700, Ian Lepore wrote: > > > > > > > > > > > > On Sat, 2018-11-17 at 22:52 +0100, Wojciech Puchar wrote: > > > > > > > > > > > > > > freebsd will not swap with that lots of free ram. > > > > > > > but it's 90GB free NOW, how about before? > > > > > > > > > > > > > Your information is outdated. For at least a couple years now > > > > > > (since > > > > > > approximately the 10.1 - 10.2 timeframe is my vague > > > > > > estimate), freebsd > > > > > > will page out application memory that hasn't been referenced > > > > > > for some > > > > > > time, even when the system has no shortage of free memory at > > > > > > all. > > > > > No, FreeBSD will only ever swap when there is a free page > > > > > shortage. > > > > > The > > > > > difference is that we now slowly age unreferenced pages into > > > > > the > > > > > inactive queue, which makes them candidates for pageout and > > > > > subsequent > > > > > eviction.  With pageout_update_period=0, anonymous memory won't > > > > > get > > > > > paged out unless there's a shortage of inactive pages, or an > > > > > application > > > > > calls madvise(MADV_DONTNEED) on a range of memory (which moves > > > > > any > > > > > backing pages to the inactive queue). > > > > Swapping is built on top of paging as I understand. The system > > > > can page without swapping but can not swap without (effectively) > > > > paging to implement the swapping, if I understand right. If I > > > > understand right, swapped-out means that kernel stacks have > > > > been written out and have to be loaded back in RAM before the > > > > process/threads can even run. (I might not understand.) > > > > > > > > If I've got that right, are there distinctions here for > > > > paging that is not part of swapping vs. actual swapping > > > > (and its use of paging)? Saying that something does not > > > > swap does not necessarily imply that it does not page: > > > > it still could have paging activity that does not include > > > > moving the kernel stacks for the process to backing media? > > > This is generally the old-school definition, IIRC third year comp > > > sci, > > > original BSD definition, which was also the way IBM defined it for > > > MVS. > > > (IBM also virtually swapped out address spaces, not written to > > > DASD, as > > > a means to deny CPU cycles through the scheduler not given the > > > chance > > > to consider the tasks (threads) in the address space). > > > > > > You can disable swapping by setting vm.swap_enabled=0. > > > > > > > > > > > > > > > At times I have trouble interpreting when wording goes back > > > > and forth between swapping and paging, both for the intended > > > > meaning and for the technical implications. > > > > > > > > > > > > > > > > > > > > > The advice I was recently given to revert to the old behavior > > > > > > is: > > > > > > > > > > > >   sysctl vm.pageout_update_period=0 > > > > > > > > > > > > I've been using it on a couple systems here for a few days > > > > > > now, and so > > > > > > far results are promising, I am no longer seeing gratuitous > > > > > > swapfile > > > > > > usage on systems that have so much free physical ram that > > > > > > they should > > > > > > never need to page anything out. I haven't yet pushed one of > > > > > > those > > > > > > systems hard enough to check what happens when they do need > > > > > > to start > > > > > > proactively paging out inactive memory due to shortages -- it > > > > > > could be > > > > > > that turning off the new behavior has downsides for some > > > > > > workloads. > > > > > > > > > > > > > > > > === > > > > Mark Millard > > > > marklmi at yahoo.com > > > > ( dsl-only.net went > > > > away in early 2018-Mar) > > > > > > > > _______________________________________________ > > > > freebsd-hackers@freebsd.org mailing list > > > > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > > > > To unsubscribe, send any mail to > > > > "freebsd-hackers-unsubscribe@freebsd.org" > > > > > > > -- > > > Cheers, > > > Cy Schubert > > > FreeBSD UNIX:     Web:  http://www.FreeBSD.org > > > > > > The need of the many outweighs the greed of the few. > > > > > > > > > _______________________________________________ > > > freebsd-hackers@freebsd.org mailing list > > > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > > > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freeb > > > sd.org" > > > > > _______________________________________________ > > freebsd-hackers@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd > > .org" From owner-freebsd-hackers@freebsd.org Sat Nov 24 21:31:13 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F16F7113A29C for ; Sat, 24 Nov 2018 21:31:12 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-it1-x133.google.com (mail-it1-x133.google.com [IPv6:2607:f8b0:4864:20::133]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 36E238CFD8 for ; Sat, 24 Nov 2018 21:31:12 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-it1-x133.google.com with SMTP id g85so22943174ita.3 for ; Sat, 24 Nov 2018 13:31:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=kQOgAzY2B//rMDWtej74YMRdIuMuRlx7DV55WuU/suk=; b=H0o/zIZBjEAqWL8DoUIrAXvZccH0PzBr+Z34qDIhskcHSm/Qxt6T9jn9mgYIml3r/g fs9u3r0/GT927/rLb7RpAz8503+igNEaGSOuWIcOWqoBoUlsX0QpOUxovZ+6f8kUWmF5 yEb0Zu4skIJVF5WSXY81NCfhUz6tk5Xy1reFz8QvuxyOiIFcISRnI9wRlgNQF0Gn21Z1 WHPaPmRaL9woshZ3s67CAxWOQ25doFmC5mYrZnw3bQmnlzZrVVB4fGfevrnU735Nyie1 70upWL9xj8sn4DjzqTVDIXXLVMMi2/jDTxiADDZzhlUdjJdNWYQdxitOgpZEA9mASFGk HB8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=kQOgAzY2B//rMDWtej74YMRdIuMuRlx7DV55WuU/suk=; b=ZQD4lM4yviHCicA8580OZQz2pgS2G4035ynnk9MeJzARM79kQIhB+E9PS4JRpYhLU7 RfmuyN9jJrICY+OnDlVyoEkfAaSGU7zfWCyfMN1KcdZvXIwSJlHEj+IDCNPe5Pjr6z7v xQAi2d6ZsrhBCOIS0s8MTzfmrjVG9tDBrBnTxDLVyRgL/1znEKbmnrQ7vV0C8vimIqZY pkYyGAR3JjHEXHlvJ+6y0YxqE/B/F4niXfZL3mkogI/V60NH8gEga7rmdiDe5j7b49Lb yApsB8iwlAjvNpQq54DPC1xSv+UaxyrsEpk3NsRpo2x8e6vVkDlByeo8XqhT1WgQwAvk LTRA== X-Gm-Message-State: AA+aEWZCJ7BlND07QCHUVwkhDyDNy2dTNSRzoPcccgAZKc7ZkmysahQD 2n9qaLYa8dibUV9tmL6rNZ/RFJxlmipALcdxeuDHWw== X-Google-Smtp-Source: AFSGD/W4JFnCNRTtQF1O5x0EmcspuU7xA72tOFogswAIlCAkEgvB99gJgbrWYVnLl4pIvY004c7m3sOLNvnGRbJkWdU= X-Received: by 2002:a02:3da:: with SMTP id e87mr16909006jae.78.1543095071369; Sat, 24 Nov 2018 13:31:11 -0800 (PST) MIME-Version: 1.0 References: <201811231600.wANG0wHc083199@pdx.rh.CN85.dnsmgr.net> In-Reply-To: From: Warner Losh Date: Sat, 24 Nov 2018 14:31:00 -0700 Message-ID: Subject: Re: TRIM utility To: Dimitry Andric Cc: Wojciech Puchar , "freebsd-hackers@freebsd.org" , Poul-Henning Kamp , Lev Serebryakov , "Rodney W. Grimes" , Eugene Grosbein X-Rspamd-Queue-Id: 36E238CFD8 X-Spamd-Result: default: False [-4.98 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.98)[-0.983,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[cached: ALT1.aspmx.l.google.com]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; RCVD_IN_DNSWL_NONE(0.00)[3.3.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; RCPT_COUNT_SEVEN(0.00)[7]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; RCVD_TLS_LAST(0.00)[]; IP_SCORE(-1.99)[ip: (-6.48), ipnet: 2607:f8b0::/32(-1.93), asn: 15169(-1.43), country: US(-0.09)]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Server: mx1.freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Nov 2018 21:31:13 -0000 On Sat, Nov 24, 2018 at 9:09 AM Dimitry Andric wrote: > On 24 Nov 2018, at 15:44, Wojciech Puchar wrote: > > > >> Yes. It would. That's hard with the current storage stack to do via the > >> disk interface. And often the underlying protocols do not support > partial > >> ranges. There is no good way to do this with buf/bio interface we have. > So > > > > what is an actual difference between "secure erase" and trimming whole > disk? > > It depends a lot on the device. Some devices encrypt all blocks > automatically, and a Secure Erase command might just throw away the key, > not go over all the data and actively wipe it. > When there's a difference, sanitize is usually the variant that does this. In all the SSDs I've had problems with, however, both Secure Erase and Sanitize reset the NAND pool (which effectively causes all the NAND erase blocks to be erased). TRIM is a 'logical' command, not a physical one. It doesn't tell the drive to erase anything. It just marks the data as being free. Most drives will then garbage collect things because certain low-water threshold marks are hit causing the blocks to be erased, but TRIM is definitely not a command to erase. Also, Secure Erase and Sanitize both are more destructive an thorough than a simple TRIM in that they potentially remove extra metadata that you'd need to keep intact if you wanted to keep using the drive (such as the current encryption keys). > Most SSDs will likely also trim all the blocks to be erased, since userave > come to expect the SSD performance to go back to the 'out of box' > level, after such an operation. > Correct. I've not found any that don't, even though they are allowed to. It also maximizes future performance because there's no need to stop and wait for erase blocks to go through and an erase cycle before programming them. And there's some advantage, in some of the NAND technology nodes, to keeping things in the erased state a long as possible. > If you are lucky, the manufacturer's documentation will explain the > specifics, but don't count on it. :) > Very lucky :) Warner From owner-freebsd-hackers@freebsd.org Sat Nov 24 21:40:44 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BCBF0113A6BF for ; Sat, 24 Nov 2018 21:40:44 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-it1-x12e.google.com (mail-it1-x12e.google.com [IPv6:2607:f8b0:4864:20::12e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 118688D6A7 for ; Sat, 24 Nov 2018 21:40:44 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-it1-x12e.google.com with SMTP id v11so22252409itj.0 for ; Sat, 24 Nov 2018 13:40:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=iqpECedrwCsdc+fJ6yeYFZgtRa5/TkqxgN6Ucf6eAVs=; b=Hg8eeCsbf99WPXxtRxS2aH+uwsJuEudIEStG4pTPPOmzeFoyspMZLYO52waio5rijh EqmKZk9Wj4tYPH3qUhIy1Dl9P0csibl855NJDvRh6Oc5hhor0uikn/L7M55mdTctEuFX M+pBtBNu/zGnWSrJDsI38HPWISTh8lA/Tz+XxIfkwaWCnKpB1qLGsLlqjuW712cszHYP IHuSB7cM1Wxv6ug76n0UwPDoGjEzf8urwyPBCSSQtXOUNhal2/rJt2xsBOoAsgdw0jZN W312Uu+EumgVxrqpnHDbid/vYnMp71obnnGpa3wC5Em1XF5RdQ1tDyaiiEeUKlnbmDNT DQQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=iqpECedrwCsdc+fJ6yeYFZgtRa5/TkqxgN6Ucf6eAVs=; b=P1+oS34FcR9FiBdulzKG/l7C0/qIn7iXQZi4qtDAaJAjCMntTEmqDbDoZrzNplh+oW Nq/XqExlKTLTG6L+vhmlII3/qf91Mt9+/II/bMZnmJNBnvRHHnR9XVYISglcd0Pdg7z1 oboj7llCkDmws0WCxCEJyYIS0MLkWvo4YopQ17XZlptsuIRiqywFOUBGHUpzdYfPkjTN BuVYxN9Q9X8TO8WMSKArAe0KrjPMP7V1jgJ2b5dZ5D2/o0je1DgIkiQWxTbv/T7o8sP/ LrYrO5S8aE04kVSO3hHbDvIiLlnna7TfbtW5HrPTSmT7ZUccufIoEklsPEzidj1VnfK0 ZfWw== X-Gm-Message-State: AA+aEWY1r8i5ROjPpXQxcLUe4g6/AcQZ54jHn1+dLCgFKKlR5DRiDQFq 8hTG6eFANQnOnfQlvR7wEdQsSYdtQQW3qSdmDw8BkA== X-Google-Smtp-Source: AJdET5ePedSp3WibU3EwDcp7uzaVC5h+N42MICe7clq5xTrJPOMY+WzKK8Zk9O4+hHMC6+s6+yAmH34bOEG942A4+J8= X-Received: by 2002:a24:5f94:: with SMTP id r142-v6mr18502786itb.171.1543095643259; Sat, 24 Nov 2018 13:40:43 -0800 (PST) MIME-Version: 1.0 References: <201811241705.wAOH5K8j088484@pdx.rh.CN85.dnsmgr.net> In-Reply-To: <201811241705.wAOH5K8j088484@pdx.rh.CN85.dnsmgr.net> From: Warner Losh Date: Sat, 24 Nov 2018 14:40:32 -0700 Message-ID: Subject: Re: TRIM utility To: "Rodney W. Grimes" Cc: Dimitry Andric , Wojciech Puchar , "freebsd-hackers@freebsd.org" , Poul-Henning Kamp , Lev Serebryakov , Eugene Grosbein X-Rspamd-Queue-Id: 118688D6A7 X-Spamd-Result: default: False [-4.71 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.98)[-0.982,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; MX_GOOD(-0.01)[cached: ALT1.aspmx.l.google.com]; RCPT_COUNT_SEVEN(0.00)[7]; RCVD_IN_DNSWL_NONE(0.00)[e.2.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; RCVD_TLS_LAST(0.00)[]; IP_SCORE(-1.72)[ip: (-5.14), ipnet: 2607:f8b0::/32(-1.93), asn: 15169(-1.43), country: US(-0.09)]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Server: mx1.freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Nov 2018 21:40:45 -0000 On Sat, Nov 24, 2018 at 10:05 AM Rodney W. Grimes < freebsd-rwg@pdx.rh.cn85.dnsmgr.net> wrote: > > On 24 Nov 2018, at 15:44, Wojciech Puchar wrote: > > > > > >> Yes. It would. That's hard with the current storage stack to do via > the > > >> disk interface. And often the underlying protocols do not support > partial > > >> ranges. There is no good way to do this with buf/bio interface we > have. So > > > > > > what is an actual difference between "secure erase" and trimming whole > disk? > > I wonder if any drive recognises a single trim command > that covers the whole drive, or if that is even possible > to do. Or if it recognizes that all blocks are infact > in a free/trimmed state (should be easy if they have > a total block inuse counter, just watch for it to hit 0). > TRIM is a logical operation. And it's limited to ~32GB in ATA. Lots of TRIMs will trigger garbage collection, which is what causes the erase blocks to be physically erased so they can be used for future writes. > > > It depends a lot on the device. Some devices encrypt all blocks > > automatically, and a Secure Erase command might just throw away the key, > > not go over all the data and actively wipe it. > > > > Most SSDs will likely also trim all the blocks to be erased, since users > > have come to expect the SSD performance to go back to the 'out of box' > > level, after such an operation. > > I do not think they actually "trim" anything, I am pretty > sure they just do a full FDT re-initializse and keep the > block use counters for future allocations, in effect this > looks like you trimmed the whole drive but usually completed > in just a fex seconds or less. > TRIM and Secure Erase are different beasts. TRIM is a logical operation that unmaps a LBA to physical mapping and does nothing more (though that unmapping may cause other things to happen indirectly). TRIMs also have to conform to certain behavior characteristics spelled out in the standards, so at the very least trimming the whole drive requires that, for drives that report back 0's or 1's for trimmed sectors, it write out records that say the blocks were unmapped in case there's a power outage before it can get to erasing things. Such SSDs are somewhat rare these days. Most take more than a few seconds to trim the entire drive. At least so my experience has shown. > Some manufactures actually recommend this procedure to > revive a drives performance that has degraded over time. > p Most recommend secure erase to restore performance because it knows it can do certain operations cheaply than lots of incremental trims might make harder to know. > > > > If you are lucky, the manufacturer's documentation will explain the > > specifics, but don't count on it. :) > > And there is that problem :-( > Not really. It depends on what you are using the tool for. TRIM is just an operation that un-does mappings. Secure Erase and Sanitize make the drive's contents unavailable to the host (though maybe not determined adversaries with access to the NAND chips). The reason the specs are nailed down 100% is because there's a number of NAND cell / block failures that make guarantees impossible. And the vendor may not implement any feature to 'restore factory new speed' in its drives, either because it believes its FTL never slows down (super high end) or because they don't care (super low end). Warner Warner