From owner-freebsd-stable@freebsd.org Sun Jul 19 00:51:33 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 32D7D377BBE for ; Sun, 19 Jul 2020 00:51:33 +0000 (UTC) (envelope-from james.wright@digital-chaos.com) Received: from mout.kundenserver.de (mout.kundenserver.de [212.227.126.131]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mout.kundenserver.de", Issuer "TeleSec ServerPass Class 2 CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4B8RBW56Vrz3xBh for ; Sun, 19 Jul 2020 00:51:31 +0000 (UTC) (envelope-from james.wright@digital-chaos.com) Received: from [192.168.0.15] ([82.18.193.38]) by mrelayeu.kundenserver.de (mreue010 [212.227.15.163]) with ESMTPSA (Nemesis) id 1MD9nd-1k5W8e2S7F-0095zc for ; Sun, 19 Jul 2020 02:51:29 +0200 To: freebsd-stable@freebsd.org From: James Wright Subject: ls colour (COLORTERM / CLICOLOR) Message-ID: <557add91-44cf-c981-8965-9bab90498ea1@digital-chaos.com> Date: Sun, 19 Jul 2020 01:51:28 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Provags-ID: V03:K1:UaU7rxhLJ7zCF1OAcXODF/Mzvb/BXGROUQwlfjbAWCdvpZ5eSVn 8zyjMND30q48fhVz0c3VTv6fgVFBweTJ4rmIZTh2oFaECcHGShthlUMOTSn5PrObJzui6qt /5/88GsKiVtQ6F9TIwDHlXpVsBCSyQ5aF1v6NCdI0aGPwUHEv7OKj7+1bYh+hBfm/l0+dlw FCJr8WI4McJC3+JRvYcQw== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:C6XpyO6Hv5s=:sGBrv7imHavzKMk9CJQuXk Cw3zK1DyofN2600msyrDO+td7UUZNkNhdqFt+FbDJ3XWrwbCoWjNrwuaqb2UUzy9vQ3hfB4+g XVVwwOWUhWE1Kjux/TGLm4HAUe5mZTNoeXoG90jB8JrNaFijWhDBHTJzFOMiK6gLYal1jhSq7 56SyKrM01RsYM/f+Z1kg2y74ol0pfcmumjJA/jdZztw85HTfiA2iF920qNBLMrKSncC44C4vY yk6JDUsJo4v2Etrq9Jk9qWuNYkhp3yBotIi4ejRBvoyroke+CZl8XeukGVfh6hyld+CCsku0u KEcxMursDd6uu/3lwY7SFhrhTyY9N3hjcl5jeP88ie+XHsXPmMVnG6dpSrWtHxBXFkJ6VtInR J7qSj+kqrm70lJdgO7rStc7SNiUL1oXwnDkV3Sx7K2aUzzhUb6JlQNDn0AMe1PXk36kYU1OPa BwYlBzSBkBcFz8S4UguU6WFkSh8XYZCik/RCRMrjmIiX2qvZqmsNYC+EA00fmhXhSDWupyujv H4R2f/2qdf/xV6wWnUSCdyN6dOX9WE06lk90f0cNUaSCAHA91o8VMpuh5O3wnbbh+2V04haXM DRTQLqjnuqFmv0z5Eh30QgOmqhbz41jX3NIMb0P5A4J5ox/7H0fw7sDLL55KgrSKZiFxUDFpP 6rJE060KWG5ob9DIJA0kk8cAFWHDVDTFHpV8AlOcdB1z9RU5/aOHin5R4DR0TEfCqKcvljTUm pIByGwdrE/PzlDFP5YhANMIbMfVRZSCp/cd9QirLRpW4JbsUb6nV+ppV2MQH0/6duHTCIAy+t ZgneKSp5ZAYud8ECDGwVh6uRbRFdBjh7prGVPlEaB2V7tf3RykX4aQAtWGCWWFR+AE9m3CJRM Rx6CGw8A/VQNBPzHQXvA== X-Rspamd-Queue-Id: 4B8RBW56Vrz3xBh X-Spamd-Bar: ++ Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of james.wright@digital-chaos.com has no SPF policy when checking 212.227.126.131) smtp.mailfrom=james.wright@digital-chaos.com X-Spamd-Result: default: False [2.27 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; AUTH_NA(1.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_SPAM_MEDIUM(0.60)[0.603]; DMARC_NA(0.00)[digital-chaos.com]; NEURAL_HAM_SHORT(-0.03)[-0.034]; NEURAL_SPAM_LONG(0.80)[0.801]; RCVD_IN_DNSWL_NONE(0.00)[212.227.126.131:from]; R_SPF_NA(0.00)[no SPF record]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:8560, ipnet:212.227.0.0/16, country:DE]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[212.227.126.131:from] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Jul 2020 00:51:33 -0000    Updated to 12.1-STABLE r363215 a few days ago (previous build was circa 1st June) but seem to have lost "ls" colour output with "COLORTERM=yes" set in my env.   Setting "CLICOLOR=yes" seems to enable it again, however the man page states that setting either should work? From owner-freebsd-stable@freebsd.org Sun Jul 19 01:04:37 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D4FED37847A for ; Sun, 19 Jul 2020 01:04:37 +0000 (UTC) (envelope-from yuripv@yuripv.dev) Received: from new1-smtp.messagingengine.com (new1-smtp.messagingengine.com [66.111.4.221]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4B8RTc3h8Wz3yPM; Sun, 19 Jul 2020 01:04:36 +0000 (UTC) (envelope-from yuripv@yuripv.dev) Received: from compute2.internal (compute2.nyi.internal [10.202.2.42]) by mailnew.nyi.internal (Postfix) with ESMTP id 616DA580483; Sat, 18 Jul 2020 21:04:35 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute2.internal (MEProxy); Sat, 18 Jul 2020 21:04:35 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yuripv.dev; h= subject:to:references:cc:from:message-id:date:mime-version :in-reply-to:content-type:content-transfer-encoding; s=fm1; bh=l EEDmvMLDb1L/2MAVT5fCAgD8cCdtwpbGlmeYW0I2Oc=; b=Ks3S0fEmwUgmyT7NB p1RxYeKhAOI5lmGp1OG0nsaVKLANifWm8qO85QReCB9619RhpjPEtMESqZwxuV6w DRC1yN5hVYc654V5I6SbAl9sD7pc6BZquPETq+kza0jspzAoR94i7ubnc8OoEFWE KRkTocv8vQtbsPnvVQ1sDp3NKvVv8jClupOLLl7A/GWGJ35QzK9j13Hdb21gt6x3 tTJ0sQvVBcX9BvcmoONo8vxpfQkuUAgxjF9m7UncyfOfYABfpYOo2GvqAhN5F35Q byoXbUAUH4y9ZoY6c+HoIceyGqO+OgpQPK4gs1FuRjmQ9RBEVs8ZxkyctUMC+87P PthSg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; bh=lEEDmvMLDb1L/2MAVT5fCAgD8cCdtwpbGlmeYW0I2 Oc=; b=bgaHNsx/Ckr7/lTjSH08r5L0zl0oFFJt0Vjl+8tEdZh9ghgWdMLnSpwOz EotIfj9Ec0luBsSQ+AovIJ1/xMAFsvgCh0XM/VD5aLn64c38BZcZl6fgLt4YmWVr avBDxMAEr6U5vRmMfS6HChAH5Jek9o1vXz/SBpXjdpibqGLS5+jHYxQ//FLlNkaA jWjnzr0YNwbtVAPY0grx1sB4p2O7GAu6eLQhz7ejftX0gTJn4T3t9SGEue7TMwZ8 RJbVLysO8OKGCyyayo0H/9qUOpDH2YAEZWuUu1nodWZ1exTIEQ/XpI9/GhpCgwKM 87gbR3u8YUiCRUboJgjkG4IKld75Q== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduiedrgedtgdegfecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefuvfhfhffkffgfgggjtgfgsehtkeertddtfeejnecuhfhrohhmpegjuhhrihcu rfgrnhhkohhvuceohihurhhiphhvseihuhhrihhpvhdruggvvheqnecuggftrfgrthhtvg hrnhepveefvefgtefgueekfeejudeukeffjeeugfeiteetueelvddttefhleehvdejteev necuffhomhgrihhnpehfrhgvvggsshgurdhorhhgnecukfhppeeluddrvdegtddruddvge drudefjeenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhm peihuhhrihhpvheshihurhhiphhvrdguvghv X-ME-Proxy: Received: from [192.168.1.6] (unknown [91.240.124.137]) by mail.messagingengine.com (Postfix) with ESMTPA id 467F230600A3; Sat, 18 Jul 2020 21:04:34 -0400 (EDT) Subject: Re: ls colour (COLORTERM / CLICOLOR) To: James Wright , freebsd-stable@freebsd.org References: <557add91-44cf-c981-8965-9bab90498ea1@digital-chaos.com> Cc: Kyle Evans From: Yuri Pankov Message-ID: Date: Sun, 19 Jul 2020 04:04:33 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 In-Reply-To: <557add91-44cf-c981-8965-9bab90498ea1@digital-chaos.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4B8RTc3h8Wz3yPM X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yuripv.dev header.s=fm1 header.b=Ks3S0fEm; dkim=pass header.d=messagingengine.com header.s=fm3 header.b=bgaHNsx/; dmarc=none; spf=pass (mx1.freebsd.org: domain of yuripv@yuripv.dev designates 66.111.4.221 as permitted sender) smtp.mailfrom=yuripv@yuripv.dev X-Spamd-Result: default: False [-3.69 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yuripv.dev:s=fm1,messagingengine.com:s=fm3]; NEURAL_HAM_MEDIUM(-0.97)[-0.975]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:66.111.4.221]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[yuripv.dev]; NEURAL_HAM_LONG(-0.99)[-0.991]; RWL_MAILSPIKE_GOOD(0.00)[66.111.4.221:from]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[yuripv.dev:+,messagingengine.com:+]; NEURAL_HAM_SHORT(-1.12)[-1.124]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:11403, ipnet:66.111.4.0/24, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_IN_DNSWL_LOW(-0.10)[66.111.4.221:from] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Jul 2020 01:04:37 -0000 James Wright wrote: > >    Updated to 12.1-STABLE r363215 a few days ago (previous build was > circa 1st June) > but seem to have lost "ls" colour output with "COLORTERM=yes" set in my > env. > >   Setting "CLICOLOR=yes" seems to enable it again, however the man page > states that > setting either should work? It's https://svnweb.freebsd.org/base?view=revision&revision=361818. I'm a bit lost myself what was the intended end result, so CCing Kyle in case he can describe it better. From owner-freebsd-stable@freebsd.org Sun Jul 19 11:21:19 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id BDC7435B36A for ; Sun, 19 Jul 2020 11:21:19 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from vtr.rulingia.com (vtr.rulingia.com [IPv6:2001:19f0:5801:ebe:5400:1ff:fe53:30fd]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vtr.rulingia.com", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4B8j9B1tzwz4RNp for ; Sun, 19 Jul 2020 11:21:17 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from server.rulingia.com (ppp239-208.static.internode.on.net [59.167.239.208]) by vtr.rulingia.com (8.15.2/8.15.2) with ESMTPS id 06JBL7Xl060220 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Sun, 19 Jul 2020 21:21:13 +1000 (AEST) (envelope-from peter@rulingia.com) X-Bogosity: Ham, spamicity=0.000000 Received: from server.rulingia.com (localhost.rulingia.com [127.0.0.1]) by server.rulingia.com (8.15.2/8.15.2) with ESMTPS id 06JBL23Q015898 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Sun, 19 Jul 2020 21:21:02 +1000 (AEST) (envelope-from peter@server.rulingia.com) Received: (from peter@localhost) by server.rulingia.com (8.15.2/8.15.2/Submit) id 06JBL2i4015897 for freebsd-stable@freebsd.org; Sun, 19 Jul 2020 21:21:02 +1000 (AEST) (envelope-from peter) Date: Sun, 19 Jul 2020 21:21:02 +1000 From: Peter Jeremy To: freebsd-stable@freebsd.org Subject: Re: svn commit: r362848 - in stable/12/sys: net netinet sys Message-ID: <20200719112102.GA15535@server.rulingia.com> References: <202007011803.061I3cTs089322@repo.freebsd.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="T4sUOijqQbZv57TR" Content-Disposition: inline In-Reply-To: <202007011803.061I3cTs089322@repo.freebsd.org> X-PGP-Key: http://www.rulingia.com/keys/peter.pgp X-Rspamd-Queue-Id: 4B8j9B1tzwz4RNp X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of peter@rulingia.com designates 2001:19f0:5801:ebe:5400:1ff:fe53:30fd as permitted sender) smtp.mailfrom=peter@rulingia.com X-Spamd-Result: default: False [-4.53 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.99)[-0.994]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-0.95)[-0.950]; RCVD_COUNT_THREE(0.00)[3]; DMARC_NA(0.00)[rulingia.com]; NEURAL_HAM_SHORT(-0.18)[-0.182]; SIGNED_PGP(-2.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:20473, ipnet:2001:19f0:5800::/38, country:US]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Jul 2020 11:21:19 -0000 --T4sUOijqQbZv57TR Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable I'm sending this to -stable, rather than the src groups because I don't believe the problem is the commit itself, rather the commit has uncovered a latent problem elsewhere. On 2020-Jul-01 18:03:38 +0000, Michael Tuexen wrote: >Author: tuexen >Date: Wed Jul 1 18:03:38 2020 >New Revision: 362848 >URL: https://svnweb.freebsd.org/changeset/base/362848 > >Log: > MFC r353480: Use event handler in SCTP I have no idea how, but this update breaks booting amd64 for me (r362847 works and this doesn't). I have a custom kernel with ZFS but no SCTP so I have no real idea how this could break booting - presumably the eventhandler change has uncovered a bug somewhere else. The symptoms are that I get: Mounting from zfs:zroot/ROOT/r363310 failed with error 6; retrying for 3 mo= re seconds Mounting from zfs:zroot/ROOT/r363310 failed with error 6 (r363310 is where I was trying to update to and I didn't change the BE name as I was searching for the problem and error 6 is ENXIO). I tried to reproduce the problem with GENERIC but it hangs after displaying the EFI framebuffer information (I've seen that before and suspect it is a loader problem but haven't dug into it). Does anyone have any ideas? --=20 Peter Jeremy --T4sUOijqQbZv57TR Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEE2M6l8vfIeOACl4uUHZIUommfjLIFAl8ULIxfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEQ4 Q0VBNUYyRjdDODc4RTAwMjk3OEI5NDFEOTIxNEEyNjk5RjhDQjIACgkQHZIUommf jLJ+dxAAiOC2u6X2nZFuBYJbqJN4p88KW96UmmY13yMc10/5Y5U7myHHdiaigWCZ pqHX/OONeeXH/lpYLH10AH8fc1ptJ1GQIhVOGaGeCzeodbYQltMrFUooPLfsFyqw IuwPm9uxX4wgfMz7HYMPDYJG5FDOnA7bD1Ld7TygcnMHoODpekCS6Apmg6kBgkMS OGCqA0OimH4a0WZIQhpF6GfEqQugu2pSqdHnWeEhTHE1m+PqJfNte+/P8eYwfIW4 s8R66YaRE9y/LzDI2iZmy2J+Eo4Gr4MHKnNHSFGb/VGCohLIfWrp4sZ/ZOlAuomF Y4nixgJIr0NembHcncnFulP2TGkQBJ3JHS7Uz8g4jb8yP0xeqTWEJlIG7nyGjNQa P5RYxj1WTtVJMyPHGXX75IDc66KgQFpXeMQBCKkLwk4Zn9PX0Zo6dfgI2SbgUfnj wWOxD+0+Hs2EVS8QKiw9PBZd/xPfT5eRGtuzw80NRV1F9yAHyNTZJhRA839osvqn KnxRsZBvM36Jz8SRXa2vL/k1vig55kHnOMM8coyk8UOb8IcwvCP/07BSiWmv3SD5 4sZeO3aEXLHC3RaOtBnUXGBHwOjh2E2BVK7fvThj19eeBeNj3GjBvTTFq58RbGAs n8N0Lakfj0PdtHNFraN26qJICtQRNSOGI44oqN6gpuo4ln6zebY= =t/+G -----END PGP SIGNATURE----- --T4sUOijqQbZv57TR-- From owner-freebsd-stable@freebsd.org Sun Jul 19 11:48:37 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id ACB2035C030 for ; Sun, 19 Jul 2020 11:48:37 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4B8jmh5L0qz4TYd for ; Sun, 19 Jul 2020 11:48:36 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id 06JBmSv6014496 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sun, 19 Jul 2020 14:48:31 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 06JBmSv6014496 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id 06JBmS6J014493; Sun, 19 Jul 2020 14:48:28 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sun, 19 Jul 2020 14:48:28 +0300 From: Konstantin Belousov To: Peter Jeremy Cc: freebsd-stable@freebsd.org Subject: Re: svn commit: r362848 - in stable/12/sys: net netinet sys Message-ID: <20200719114828.GD44314@kib.kiev.ua> References: <202007011803.061I3cTs089322@repo.freebsd.org> <20200719112102.GA15535@server.rulingia.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20200719112102.GA15535@server.rulingia.com> X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on tom.home X-Rspamd-Queue-Id: 4B8jmh5L0qz4TYd X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=gmail.com (policy=none); spf=softfail (mx1.freebsd.org: 2001:470:d5e7:1::1 is neither permitted nor denied by domain of kostikbel@gmail.com) smtp.mailfrom=kostikbel@gmail.com X-Spamd-Result: default: False [0.86 / 15.00]; ARC_NA(0.00)[]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; NEURAL_SPAM_SHORT(0.04)[0.044]; MIME_GOOD(-0.10)[text/plain]; HAS_XAW(0.00)[]; R_SPF_SOFTFAIL(0.00)[~all:c]; NEURAL_SPAM_MEDIUM(0.29)[0.293]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_SPAM_LONG(0.52)[0.524]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; DMARC_POLICY_SOFTFAIL(0.10)[gmail.com : No valid SPF, No valid DKIM,none] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Jul 2020 11:48:37 -0000 On Sun, Jul 19, 2020 at 09:21:02PM +1000, Peter Jeremy wrote: > I'm sending this to -stable, rather than the src groups because I > don't believe the problem is the commit itself, rather the commit > has uncovered a latent problem elsewhere. > > On 2020-Jul-01 18:03:38 +0000, Michael Tuexen wrote: > >Author: tuexen > >Date: Wed Jul 1 18:03:38 2020 > >New Revision: 362848 > >URL: https://svnweb.freebsd.org/changeset/base/362848 > > > >Log: > > MFC r353480: Use event handler in SCTP > > I have no idea how, but this update breaks booting amd64 for me (r362847 > works and this doesn't). I have a custom kernel with ZFS but no SCTP so I > have no real idea how this could break booting - presumably the > eventhandler change has uncovered a bug somewhere else. > > The symptoms are that I get: > Mounting from zfs:zroot/ROOT/r363310 failed with error 6; retrying for 3 more seconds > Mounting from zfs:zroot/ROOT/r363310 failed with error 6 > > (r363310 is where I was trying to update to and I didn't change the BE > name as I was searching for the problem and error 6 is ENXIO). > > I tried to reproduce the problem with GENERIC but it hangs after > displaying the EFI framebuffer information (I've seen that before and > suspect it is a loader problem but haven't dug into it). > > Does anyone have any ideas? Did you checked that the physical devices where your ZFS pool is located, are detected, and that kernel messages for their drivers are as usual ? Overall, is there anything strange in the verbose dmesg ? From owner-freebsd-stable@freebsd.org Sun Jul 19 14:17:04 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8609536046E for ; Sun, 19 Jul 2020 14:17:04 +0000 (UTC) (envelope-from mike@sentex.net) Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [IPv6:2607:f3e0:0:3::19]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "pyroxene.sentex.ca", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4B8n3z6Ccdz4cqP for ; Sun, 19 Jul 2020 14:17:03 +0000 (UTC) (envelope-from mike@sentex.net) Received: from [IPv6:2607:f3e0:0:4:cd26:ee8a:a97e:89a8] ([IPv6:2607:f3e0:0:4:cd26:ee8a:a97e:89a8]) by pyroxene2a.sentex.ca (8.15.2/8.15.2) with ESMTPS id 06JEH24w058796 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO) for ; Sun, 19 Jul 2020 10:17:02 -0400 (EDT) (envelope-from mike@sentex.net) To: FreeBSD-STABLE Mailing List From: mike tancsa Subject: zfs meta data slowness Autocrypt: addr=mike@sentex.net; keydata= mQENBFywzOMBCACoNFpwi5MeyEREiCeHtbm6pZJI/HnO+wXdCAWtZkS49weOoVyUj5BEXRZP xflV2ib2hflX4nXqhenaNiia4iaZ9ft3I1ebd7GEbGnsWCvAnob5MvDZyStDAuRxPJK1ya/s +6rOvr+eQiXYNVvfBhrCfrtR/esSkitBGxhUkBjOti8QwzD71JVF5YaOjBAs7jZUKyLGj0kW yDg4jUndudWU7G2yc9GwpHJ9aRSUN8e/mWdIogK0v+QBHfv/dsI6zVB7YuxCC9Fx8WPwfhDH VZC4kdYCQWKXrm7yb4TiVdBh5kgvlO9q3js1yYdfR1x8mjK2bH2RSv4bV3zkNmsDCIxjABEB AAG0HW1pa2UgdGFuY3NhIDxtaWtlQHNlbnRleC5uZXQ+iQFUBBMBCAA+FiEEmuvCXT0aY6hs 4SbWeVOEFl5WrMgFAlywzOYCGwMFCQHhM4AFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQ eVOEFl5WrMhnPAf7Bf+ola0V9t4i8rwCMGvzkssGaxY/5zNSZO9BgSgfN0WzgmBEOy/3R4km Yn5KH94NltJYAAE5hqkFmAwK6psOqAR9cxHrRfU+gV2KO8pCDc6K/htkQcd/mclJYpCHp6Eq EVJOiAxcNaYuHZkeMdXDuvvI5Rk82VHk84BGgxIqIrhLlkguoPbXOOa+8c/Mpb1sRAGZEOuX EzKNC49+GS9gKW6ISbanyPsGEcFyP7GKMzcHBPf3cPrewZQZ6gBoNscasL6IJeAQDqzQAxbU GjO0qBSMRgnLXK7+DJlxrYdHGXqNbV6AYsmHJ6c2WWWiuRviFBqXinlgJ2FnYebZPAfWiQ== Message-ID: Date: Sun, 19 Jul 2020 10:17:02 -0400 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 4B8n3z6Ccdz4cqP X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of mike@sentex.net designates 2607:f3e0:0:3::19 as permitted sender) smtp.mailfrom=mike@sentex.net X-Spamd-Result: default: False [-0.32 / 15.00]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.58)[-0.577]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f3e0::/32]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; HFILTER_HELO_IP_A(1.00)[pyroxene2a.sentex.ca]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-0.65)[-0.646]; HFILTER_HELO_NORES_A_OR_MX(0.30)[pyroxene2a.sentex.ca]; TO_DN_ALL(0.00)[]; NEURAL_HAM_SHORT(-0.09)[-0.093]; DMARC_NA(0.00)[sentex.net]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Jul 2020 14:17:04 -0000 Are there any tweaks that can be done to speed up or improve zfs metadata performance ? I have a backup server with a lot of snapshots (40,000)  and just doing a listing can take a great deal of time.  Best case scenario is about 24 seconds, worst case, I have seen it up to 15 minutes.  (FreeBSD 12.1-STABLE r363078) ARC Efficiency:                                 79.33b         Cache Hit Ratio:                92.81%  73.62b         Cache Miss Ratio:               7.19%   5.71b         Actual Hit Ratio:               92.78%  73.60b         Data Demand Efficiency:         96.47%  461.91m         Data Prefetch Efficiency:       1.00%   262.73m         CACHE HITS BY CACHE LIST:           Anonymously Used:             0.01%   3.86m           Most Recently Used:           3.91%   2.88b           Most Frequently Used:         96.06%  70.72b           Most Recently Used Ghost:     0.01%   5.31m           Most Frequently Used Ghost:   0.01%   10.47m         CACHE HITS BY DATA TYPE:           Demand Data:                  0.61%   445.60m           Prefetch Data:                0.00%   2.63m           Demand Metadata:              99.36%  73.15b           Prefetch Metadata:            0.03%   21.00m         CACHE MISSES BY DATA TYPE:           Demand Data:                  0.29%   16.31m           Prefetch Data:                4.56%   260.10m           Demand Metadata:              95.02%  5.42b           Prefetch Metadata:            0.14%   7.75m Other than increase the metadata max, I havent really changed any tuneables ZFS Tunables (sysctl):         kern.maxusers                           4416         vm.kmem_size                            66691842048         vm.kmem_size_scale                      1         vm.kmem_size_min                        0         vm.kmem_size_max                        1319413950874         vfs.zfs.trim.max_interval               1         vfs.zfs.trim.timeout                    30         vfs.zfs.trim.txg_delay                  32         vfs.zfs.trim.enabled                    1         vfs.zfs.vol.immediate_write_sz          32768         vfs.zfs.vol.unmap_sync_enabled          0         vfs.zfs.vol.unmap_enabled               1         vfs.zfs.vol.recursive                   0         vfs.zfs.vol.mode                        1         vfs.zfs.version.zpl                     5         vfs.zfs.version.spa                     5000         vfs.zfs.version.acl                     1         vfs.zfs.version.ioctl                   7         vfs.zfs.debug                           0         vfs.zfs.super_owner                     0         vfs.zfs.immediate_write_sz              32768         vfs.zfs.sync_pass_rewrite               2         vfs.zfs.sync_pass_dont_compress         5         vfs.zfs.sync_pass_deferred_free         2         vfs.zfs.zio.dva_throttle_enabled        1         vfs.zfs.zio.exclude_metadata            0         vfs.zfs.zio.use_uma                     1         vfs.zfs.zio.taskq_batch_pct             75         vfs.zfs.zil_maxblocksize                131072         vfs.zfs.zil_slog_bulk                   786432         vfs.zfs.zil_nocacheflush                0         vfs.zfs.zil_replay_disable              0         vfs.zfs.cache_flush_disable             0         vfs.zfs.standard_sm_blksz               131072         vfs.zfs.dtl_sm_blksz                    4096         vfs.zfs.min_auto_ashift                 9         vfs.zfs.max_auto_ashift                 13         vfs.zfs.vdev.trim_max_pending           10000         vfs.zfs.vdev.bio_delete_disable         0         vfs.zfs.vdev.bio_flush_disable          0         vfs.zfs.vdev.def_queue_depth            32         vfs.zfs.vdev.queue_depth_pct            1000         vfs.zfs.vdev.write_gap_limit            4096         vfs.zfs.vdev.read_gap_limit             32768         vfs.zfs.vdev.aggregation_limit_non_rotating131072         vfs.zfs.vdev.aggregation_limit          1048576         vfs.zfs.vdev.initializing_max_active    1         vfs.zfs.vdev.initializing_min_active    1         vfs.zfs.vdev.removal_max_active         2         vfs.zfs.vdev.removal_min_active         1         vfs.zfs.vdev.trim_max_active            64         vfs.zfs.vdev.trim_min_active            1         vfs.zfs.vdev.scrub_max_active           2         vfs.zfs.vdev.scrub_min_active           1         vfs.zfs.vdev.async_write_max_active     10         vfs.zfs.vdev.async_write_min_active     1         vfs.zfs.vdev.async_read_max_active      3         vfs.zfs.vdev.async_read_min_active      1         vfs.zfs.vdev.sync_write_max_active      10         vfs.zfs.vdev.sync_write_min_active      10         vfs.zfs.vdev.sync_read_max_active       10         vfs.zfs.vdev.sync_read_min_active       10         vfs.zfs.vdev.max_active                 1000         vfs.zfs.vdev.async_write_active_max_dirty_percent60         vfs.zfs.vdev.async_write_active_min_dirty_percent30         vfs.zfs.vdev.mirror.non_rotating_seek_inc1         vfs.zfs.vdev.mirror.non_rotating_inc    0         vfs.zfs.vdev.mirror.rotating_seek_offset1048576         vfs.zfs.vdev.mirror.rotating_seek_inc   5         vfs.zfs.vdev.mirror.rotating_inc        0         vfs.zfs.vdev.trim_on_init               1         vfs.zfs.vdev.cache.bshift               16         vfs.zfs.vdev.cache.size                 0         vfs.zfs.vdev.cache.max                  16384         vfs.zfs.vdev.validate_skip              0         vfs.zfs.vdev.max_ms_shift               34         vfs.zfs.vdev.default_ms_shift           29         vfs.zfs.vdev.max_ms_count_limit         131072         vfs.zfs.vdev.min_ms_count               16         vfs.zfs.vdev.default_ms_count           200         vfs.zfs.txg.timeout                     5         vfs.zfs.space_map_ibs                   14         vfs.zfs.special_class_metadata_reserve_pct25         vfs.zfs.user_indirect_is_special        1         vfs.zfs.ddt_data_is_special             1         vfs.zfs.spa_allocators                  4         vfs.zfs.spa_min_slop                    134217728         vfs.zfs.spa_slop_shift                  5         vfs.zfs.spa_asize_inflation             24         vfs.zfs.deadman_enabled                 1         vfs.zfs.deadman_checktime_ms            5000         vfs.zfs.deadman_synctime_ms             1000000         vfs.zfs.debugflags                      0         vfs.zfs.recover                         0         vfs.zfs.spa_load_verify_data            1         vfs.zfs.spa_load_verify_metadata        1         vfs.zfs.spa_load_verify_maxinflight     10000         vfs.zfs.max_missing_tvds_scan           0         vfs.zfs.max_missing_tvds_cachefile      2         vfs.zfs.max_missing_tvds                0         vfs.zfs.spa_load_print_vdev_tree        0         vfs.zfs.ccw_retry_interval              300         vfs.zfs.check_hostid                    1         vfs.zfs.multihost_fail_intervals        10         vfs.zfs.multihost_import_intervals      20         vfs.zfs.multihost_interval              1000         vfs.zfs.mg_fragmentation_threshold      85         vfs.zfs.mg_noalloc_threshold            0         vfs.zfs.condense_pct                    200         vfs.zfs.metaslab_sm_blksz               4096         vfs.zfs.metaslab.bias_enabled           1         vfs.zfs.metaslab.lba_weighting_enabled  1         vfs.zfs.metaslab.fragmentation_factor_enabled1         vfs.zfs.metaslab.preload_enabled        1         vfs.zfs.metaslab.preload_limit          3         vfs.zfs.metaslab.unload_delay           8         vfs.zfs.metaslab.load_pct               50         vfs.zfs.metaslab.min_alloc_size         33554432         vfs.zfs.metaslab.df_free_pct            4         vfs.zfs.metaslab.df_alloc_threshold     131072         vfs.zfs.metaslab.debug_unload           0         vfs.zfs.metaslab.debug_load             0         vfs.zfs.metaslab.fragmentation_threshold70         vfs.zfs.metaslab.force_ganging          16777217         vfs.zfs.free_bpobj_enabled              1         vfs.zfs.free_max_blocks                 -1         vfs.zfs.zfs_scan_checkpoint_interval    7200         vfs.zfs.zfs_scan_legacy                 0         vfs.zfs.no_scrub_prefetch               0         vfs.zfs.no_scrub_io                     0         vfs.zfs.resilver_min_time_ms            3000         vfs.zfs.free_min_time_ms                1000         vfs.zfs.scan_min_time_ms                1000         vfs.zfs.scan_idle                       50         vfs.zfs.scrub_delay                     4         vfs.zfs.resilver_delay                  2         vfs.zfs.zfetch.array_rd_sz              1048576         vfs.zfs.zfetch.max_idistance            67108864         vfs.zfs.zfetch.max_distance             8388608         vfs.zfs.zfetch.min_sec_reap             2         vfs.zfs.zfetch.max_streams              8         vfs.zfs.prefetch_disable                0         vfs.zfs.delay_scale                     500000         vfs.zfs.delay_min_dirty_percent         60         vfs.zfs.dirty_data_sync_pct             20         vfs.zfs.dirty_data_max_percent          10         vfs.zfs.dirty_data_max_max              4294967296         vfs.zfs.dirty_data_max                  4294967296         vfs.zfs.max_recordsize                  1048576         vfs.zfs.default_ibs                     17         vfs.zfs.default_bs                      9         vfs.zfs.send_holes_without_birth_time   1         vfs.zfs.mdcomp_disable                  0         vfs.zfs.per_txg_dirty_frees_percent     5         vfs.zfs.nopwrite_enabled                1         vfs.zfs.dedup.prefetch                  1         vfs.zfs.dbuf_cache_lowater_pct          10         vfs.zfs.dbuf_cache_hiwater_pct          10         vfs.zfs.dbuf_metadata_cache_overflow    0         vfs.zfs.dbuf_metadata_cache_shift       6         vfs.zfs.dbuf_cache_shift                5         vfs.zfs.dbuf_metadata_cache_max_bytes   1025282816         vfs.zfs.dbuf_cache_max_bytes            2050565632         vfs.zfs.arc_min_prescient_prefetch_ms   6         vfs.zfs.arc_min_prefetch_ms             1         vfs.zfs.l2c_only_size                   0         vfs.zfs.mfu_ghost_data_esize            7778263552         vfs.zfs.mfu_ghost_metadata_esize        16851792896         vfs.zfs.mfu_ghost_size                  24630056448         vfs.zfs.mfu_data_esize                  3059418112         vfs.zfs.mfu_metadata_esize              28641792         vfs.zfs.mfu_size                        6399023104         vfs.zfs.mru_ghost_data_esize            2199812096         vfs.zfs.mru_ghost_metadata_esize        6289682432         vfs.zfs.mru_ghost_size                  8489494528         vfs.zfs.mru_data_esize                  22781456384         vfs.zfs.mru_metadata_esize              309155840         vfs.zfs.mru_size                        23847875584         vfs.zfs.anon_data_esize                 0         vfs.zfs.anon_metadata_esize             0         vfs.zfs.anon_size                       8556544         vfs.zfs.l2arc_norw                      1         vfs.zfs.l2arc_feed_again                1         vfs.zfs.l2arc_noprefetch                1         vfs.zfs.l2arc_feed_min_ms               200         vfs.zfs.l2arc_feed_secs                 1         vfs.zfs.l2arc_headroom                  2         vfs.zfs.l2arc_write_boost               8388608         vfs.zfs.l2arc_write_max                 8388608         vfs.zfs.arc_meta_strategy               1         vfs.zfs.arc_meta_limit                  15833624576         vfs.zfs.arc_free_target                 346902         vfs.zfs.arc_kmem_cache_reap_retry_ms    1000         vfs.zfs.compressed_arc_enabled          1         vfs.zfs.arc_grow_retry                  60         vfs.zfs.arc_shrink_shift                7         vfs.zfs.arc_average_blocksize           8192         vfs.zfs.arc_no_grow_shift               5         vfs.zfs.arc_min                         8202262528         vfs.zfs.arc_max                         39334498304         vfs.zfs.abd_chunk_size                  4096         vfs.zfs.abd_scatter_enabled             1 From owner-freebsd-stable@freebsd.org Mon Jul 20 10:41:10 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0AE4B3585C0; Mon, 20 Jul 2020 10:41:10 +0000 (UTC) (envelope-from lwhsu@freebsd.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2610:1c1:1:6074::16:84]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "freefall.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4B9JDP4lYLz3Wvw; Mon, 20 Jul 2020 10:41:09 +0000 (UTC) (envelope-from lwhsu@freebsd.org) Received: by freefall.freebsd.org (Postfix, from userid 1129) id 6F442B24C; Mon, 20 Jul 2020 10:41:09 +0000 (UTC) Date: Mon, 20 Jul 2020 10:41:09 +0000 From: Li-Wen Hsu To: freebsd-testing@freebsd.org Cc: freebsd-current@freebsd.org, freebsd-stable@freebsd.org Subject: FreeBSD CI Weekly Report 2020-07-19 Message-ID: <20200720104109.GA21360@freefall.freebsd.org> Reply-To: freebsd-testing@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Jul 2020 10:41:10 -0000 (Please send the followup to freebsd-testing@ and note Reply-To is set.) FreeBSD CI Weekly Report 2020-07-19 =================================== Here is a summary of the FreeBSD Continuous Integration results for the period from 2020-07-13 to 2020-07-19. During this period, we have: * 1699 builds (93.3% (-2.4) passed, 6.7% (+2.4) failed) of buildworld and buildkernel (GENERIC and LINT) were executed on aarch64, amd64, armv6, armv7, i386, mips, mips64, powerpc, powerpc64, powerpcspe, riscv64, sparc64 architectures for head, stable/12, stable/11 branches. * 191 test runs (88.0% (+1.3) passed, 12.0% (+0) unstable, 0% (-1.3) exception) were executed on amd64, i386, riscv64 architectures for head, stable/12, stable/11 branches. * 36 doc and www builds (100% (+0) passed) Test case status (on 2020-07-19 23:59): | Branch/Architecture | Total | Pass | Fail | Skipped | | ------------------- | -------- | --------- | ----- | ------- | | head/amd64 | 7859 (0) | 7768 (+1) | 0 (0) | 91 (-1) | | head/i386 | 7857 (0) | 7759 (+1) | 0 (0) | 98 (-1) | | 12-STABLE/amd64 | 7617 (0) | 7557 (0) | 0 (0) | 60 (0) | | 12-STABLE/i386 | 7615 (0) | 7550 (+3) | 0 (0) | 65 (-3) | | 11-STABLE/amd64 | 6912 (0) | 6861 (0) | 0 (0) | 51 (0) | | 11-STABLE/i386 | 6910 (0) | 6854 (0) | 0 (0) | 56 (0) | (The statistics from experimental jobs are omitted) If any of the issues found by CI are in your area of interest or expertise please investigate the PRs listed below. The latest web version of this report is available at https://hackmd.io/@FreeBSD-CI/report-20200719 and archive is available at https://hackmd.io/@FreeBSD-CI/ , any help is welcomed. ## Failing jobs * https://ci.freebsd.org/job/FreeBSD-head-amd64-gcc6_build/ ``` /usr/local/bin/x86_64-unknown-freebsd12.1-ld: /tmp/obj/workspace/src/amd64.amd64/lib/clang/liblldb/liblldb.a(IOHandlerCursesGUI.o): in function `curses::Window::Box(unsigned int, unsigned int)': /workspace/src/contrib/llvm-project/lldb/source/Core/IOHandlerCursesGUI.cpp:361: undefined reference to `box' /usr/local/bin/x86_64-unknown-freebsd12.1-ld: /workspace/src/contrib/llvm-project/lldb/source/Core/IOHandlerCursesGUI.cpp:361: undefined reference to `box' collect2: error: ld returned 1 exit status ``` From kevans@: one of ncurses' scripts that generates box and a bunch of other symbols is shooting blanks with gcc6, however it seems fine on gcc9. ## Regressions * lib.libexecinfo.backtrace_test.backtrace_fmt_basic starts failing on amd64 after r360915 https://bugs.freebsd.org/246537 * lib.msun.ctrig_test.test_inf_inputs starts failing after llvm10 import https://bugs.freebsd.org/244732 * Lock-order reversals triggered by tests under sys.net.if_lagg_test.* on i386 https://bugs.freebsd.org/244163 Discovered by newly endabled sys.net.* tests. ([r357857](https://svnweb.freebsd.org/changeset/base/357857)) * sys.net.if_lagg_test.lacp_linkstate_destroy_stress panics i386 kernel https://bugs.freebsd.org/244168 Discovered by newly endabled sys.net.* tests. ([r357857](https://svnweb.freebsd.org/changeset/base/357857)) Fix in review: https://reviews.freebsd.org/D25284 ## Failing and Flaky tests (from experimental jobs) * https://ci.freebsd.org/job/FreeBSD-head-amd64-dtrace_test/ * cddl.usr.sbin.dtrace.common.misc.t_dtrace_contrib.tst_dynopt_d * https://bugs.freebsd.org/237641 * https://ci.freebsd.org/job/FreeBSD-head-amd64-test_zfs/ * There are ~13 failing and ~109 skipped cases, including flakey ones, see https://ci.freebsd.org/job/FreeBSD-head-amd64-test_zfs/lastCompletedBuild/testReport/ for more details * Work for cleaning these failing cass are in progress * https://ci.freebsd.org/job/FreeBSD-head-amd64-test_ltp/ * Total 3749 tests, 2277 success, 647 failures, 825 skipped ## Disabled Tests * sys.fs.tmpfs.mount_test.large https://bugs.freebsd.org/212862 * sys.fs.tmpfs.link_test.kqueue https://bugs.freebsd.org/213662 * sys.kqueue.libkqueue.kqueue_test.main https://bugs.freebsd.org/233586 * sys.kern.ptrace_test.ptrace__PT_KILL_competing_stop https://bugs.freebsd.org/220841 * lib.libc.regex.exhaust_test.regcomp_too_big (i386 only) https://bugs.freebsd.org/237450 * sys.netinet.socket_afinet.socket_afinet_bind_zero https://bugs.freebsd.org/238781 * sys.netpfil.pf.names.names * sys.netpfil.pf.synproxy.synproxy https://bugs.freebsd.org/238870 * sys.kern.ptrace_test.ptrace__follow_fork_child_detached_unrelated_debugger https://bugs.freebsd.org/239292 * sys.kern.ptrace_test.ptrace__follow_fork_both_attached_unrelated_debugger https://bugs.freebsd.org/239397 * sys.kern.ptrace_test.ptrace__parent_sees_exit_after_child_debugger https://bugs.freebsd.org/239399 * sys.kern.ptrace_test.ptrace__follow_fork_parent_detached_unrelated_debugger https://bugs.freebsd.org/239425 * sys.sys.qmath_test.qdivq_s64q https://bugs.freebsd.org/240219 * sys.kern.ptrace_test.ptrace__getppid https://bugs.freebsd.org/240510 * lib.libc.sys.stat_test.stat_socket https://bugs.freebsd.org/240621 * lib.libarchive.functional_test.test_write_filter_zstd https://bugs.freebsd.org/240683 * lib.libcasper.services.cap_dns.dns_test.main https://bugs.freebsd.org/241435 * local.kyua.* (31 cases) & local.lutok.* (3 cases) on 11-i386 https://ci.freebsd.org/job/FreeBSD-stable-11-i386-test/2278/testReport/ * sys.kern.ptrace_test.ptrace__procdesc_reparent_wait_child https://bugs.freebsd.org/243605 * sys.kern.ptrace_test.ptrace__parent_wait_after_attach https://bugs.freebsd.org/244055 * sys.kern.ptrace_test.ptrace__parent_exits_before_child https://bugs.freebsd.org/244056 * sys.net.if_lagg_test.witness (i386) https://bugs.freebsd.org/244163 * PipePdfork.WildcardWait in sys.capsicum.capsicum-test.main https://bugs.freebsd.org/244165 * sys.net.if_lagg_test.lacp_linkstate_destroy_stress (i386) https://bugs.freebsd.org/244168 * sys.netinet6.frag6.frag6_07.frag6_07 https://bugs.freebsd.org/244170 * sys.netinet.fibs_test.udp_dontroute6 https://bugs.freebsd.org/244172 * sys.netpfil.pf.nat.exhaust https://bugs.freebsd.org/244703 * sys.geom.class.gate.ggate_test.ggated (i386) https://bugs.freebsd.org/244737 * sys.kern.sysv_test.msg https://bugs.freebsd.org/233649 ## Issues ### Cause build fails * https://bugs.freebsd.org/233769 Possible build race: ld: error: unable to find library -lgcc_s ### Cause kernel panics * https://bugs.freebsd.org/238870 sys.netpfil.pf.names.names and sys.netpfil.pf.synproxy.synproxy cause panic ### Open * https://bugs.freebsd.org/237403 Tests in sys/opencrypto should be converted to Python3 * https://bugs.freebsd.org/237641 Flakey test case: common.misc.t_dtrace_contrib.tst_dynopt_d * https://bugs.freebsd.org/237656 "Freed UMA keg (rtentry) was not empty (18 items). Lost 1 pages of memory." seen when running sys/netipsec tests * https://bugs.freebsd.org/238781 sys.netinet.socket_afinet.socket_afinet_bind_zero does not work when mac_portacl(4) loaded * https://bugs.freebsd.org/239292 Flakey test case: sys.kern.ptrace_test.ptrace__follow_fork_child_detached_unrelated_debugger * https://bugs.freebsd.org/239397 Flakey test case: sys.kern.ptrace_test.ptrace__follow_fork_both_attached_unrelated_debugger * https://bugs.freebsd.org/239399 Flakey test case: sys.kern.ptrace_test.ptrace__parent_sees_exit_after_child_debugger * https://bugs.freebsd.org/239425 Flakey test case: sys.kern.ptrace_test.ptrace__follow_fork_parent_detached_unrelated_debugger * https://bugs.freebsd.org/241662 Flakey test case: lib.libarchive.functional_test.test_fuzz_iso9660 * https://bugs.freebsd.org/246443 sys.net.if_clone_test.epair_stress sometimes exceeds timeout limit but not caught by kyua * https://bugs.freebsd.org/247510 sys.net.if_lagg_test.status_stress panics kernel on i386 ### Others * [Tickets related to testing@](https://preview.tinyurl.com/y9maauwg) From owner-freebsd-stable@freebsd.org Mon Jul 20 13:04:09 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id E3E8535DDBF for ; Mon, 20 Jul 2020 13:04:09 +0000 (UTC) (envelope-from ronald-lists@klop.ws) Received: from smtp-relay-int.realworks.nl (smtp-relay-int.realworks.nl [194.109.157.24]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4B9MPN0dKKz40ch for ; Mon, 20 Jul 2020 13:04:07 +0000 (UTC) (envelope-from ronald-lists@klop.ws) Date: Mon, 20 Jul 2020 15:04:03 +0200 (CEST) From: Ronald Klop To: FreeBSD-STABLE Mailing List Message-ID: <1949194763.1.1595250243575@localhost> In-Reply-To: References: Subject: Re: zfs meta data slowness MIME-Version: 1.0 X-Mailer: Realworks (517.349.cbedd3c6603) Importance: Normal X-Priority: 3 (Normal) X-Rspamd-Queue-Id: 4B9MPN0dKKz40ch X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of ronald-lists@klop.ws designates 194.109.157.24 as permitted sender) smtp.mailfrom=ronald-lists@klop.ws X-Spamd-Result: default: False [-0.97 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.71)[-0.709]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:194.109.157.0/24]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; DMARC_NA(0.00)[klop.ws]; NEURAL_HAM_LONG(-0.43)[-0.430]; RCPT_COUNT_ONE(0.00)[1]; TO_DN_ALL(0.00)[]; NEURAL_HAM_SHORT(-0.03)[-0.031]; HAS_X_PRIO_THREE(0.00)[3]; RCVD_IN_DNSWL_NONE(0.00)[194.109.157.24:from]; RCVD_COUNT_ZERO(0.00)[0]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:3265, ipnet:194.109.0.0/16, country:NL]; MID_RHS_NOT_FQDN(0.50)[]; RWL_MAILSPIKE_VERYGOOD(0.00)[194.109.157.24:from] Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.33 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Jul 2020 13:04:10 -0000 Hi, My first suggestion would be to remove a lot of snapshots. But that my not match your business case. Maybe you can provide more information about your setup: Amount of RAM, CPU? output of "zpool status" output of "zfs list" if possible to share Type of disks/ssds? What is the load of the system? I/O per second, etc. Do you use dedup, GELI? Something else special about the setup. output of "top -b" That kind of information. Regards, Ronald. Van: mike tancsa Datum: zondag, 19 juli 2020 16:17 Aan: FreeBSD-STABLE Mailing List Onderwerp: zfs meta data slowness > > Are there any tweaks that can be done to speed up or improve zfs > metadata performance ? I have a backup server with a lot of snapshots > (40,000) and just doing a listing can take a great deal of time. Best > case scenario is about 24 seconds, worst case, I have seen it up to 15 > minutes. (FreeBSD 12.1-STABLE r363078) > > > ARC Efficiency: 79.33b > Cache Hit Ratio: 92.81% 73.62b > Cache Miss Ratio: 7.19% 5.71b > Actual Hit Ratio: 92.78% 73.60b > > Data Demand Efficiency: 96.47% 461.91m > Data Prefetch Efficiency: 1.00% 262.73m > > CACHE HITS BY CACHE LIST: > Anonymously Used: 0.01% 3.86m > Most Recently Used: 3.91% 2.88b > Most Frequently Used: 96.06% 70.72b > Most Recently Used Ghost: 0.01% 5.31m > Most Frequently Used Ghost: 0.01% 10.47m > > CACHE HITS BY DATA TYPE: > Demand Data: 0.61% 445.60m > Prefetch Data: 0.00% 2.63m > Demand Metadata: 99.36% 73.15b > Prefetch Metadata: 0.03% 21.00m > > CACHE MISSES BY DATA TYPE: > Demand Data: 0.29% 16.31m > Prefetch Data: 4.56% 260.10m > Demand Metadata: 95.02% 5.42b > Prefetch Metadata: 0.14% 7.75m > > > Other than increase the metadata max, I havent really changed any tuneables > > > ZFS Tunables (sysctl): > kern.maxusers 4416 > vm.kmem_size 66691842048 > vm.kmem_size_scale 1 > vm.kmem_size_min 0 > vm.kmem_size_max 1319413950874 > vfs.zfs.trim.max_interval 1 > vfs.zfs.trim.timeout 30 > vfs.zfs.trim.txg_delay 32 > vfs.zfs.trim.enabled 1 > vfs.zfs.vol.immediate_write_sz 32768 > vfs.zfs.vol.unmap_sync_enabled 0 > vfs.zfs.vol.unmap_enabled 1 > vfs.zfs.vol.recursive 0 > vfs.zfs.vol.mode 1 > vfs.zfs.version.zpl 5 > vfs.zfs.version.spa 5000 > vfs.zfs.version.acl 1 > vfs.zfs.version.ioctl 7 > vfs.zfs.debug 0 > vfs.zfs.super_owner 0 > vfs.zfs.immediate_write_sz 32768 > vfs.zfs.sync_pass_rewrite 2 > vfs.zfs.sync_pass_dont_compress 5 > vfs.zfs.sync_pass_deferred_free 2 > vfs.zfs.zio.dva_throttle_enabled 1 > vfs.zfs.zio.exclude_metadata 0 > vfs.zfs.zio.use_uma 1 > vfs.zfs.zio.taskq_batch_pct 75 > vfs.zfs.zil_maxblocksize 131072 > vfs.zfs.zil_slog_bulk 786432 > vfs.zfs.zil_nocacheflush 0 > vfs.zfs.zil_replay_disable 0 > vfs.zfs.cache_flush_disable 0 > vfs.zfs.standard_sm_blksz 131072 > vfs.zfs.dtl_sm_blksz 4096 > vfs.zfs.min_auto_ashift 9 > vfs.zfs.max_auto_ashift 13 > vfs.zfs.vdev.trim_max_pending 10000 > vfs.zfs.vdev.bio_delete_disable 0 > vfs.zfs.vdev.bio_flush_disable 0 > vfs.zfs.vdev.def_queue_depth 32 > vfs.zfs.vdev.queue_depth_pct 1000 > vfs.zfs.vdev.write_gap_limit 4096 > vfs.zfs.vdev.read_gap_limit 32768 > vfs.zfs.vdev.aggregation_limit_non_rotating131072 > vfs.zfs.vdev.aggregation_limit 1048576 > vfs.zfs.vdev.initializing_max_active 1 > vfs.zfs.vdev.initializing_min_active 1 > vfs.zfs.vdev.removal_max_active 2 > vfs.zfs.vdev.removal_min_active 1 > vfs.zfs.vdev.trim_max_active 64 > vfs.zfs.vdev.trim_min_active 1 > vfs.zfs.vdev.scrub_max_active 2 > vfs.zfs.vdev.scrub_min_active 1 > vfs.zfs.vdev.async_write_max_active 10 > vfs.zfs.vdev.async_write_min_active 1 > vfs.zfs.vdev.async_read_max_active 3 > vfs.zfs.vdev.async_read_min_active 1 > vfs.zfs.vdev.sync_write_max_active 10 > vfs.zfs.vdev.sync_write_min_active 10 > vfs.zfs.vdev.sync_read_max_active 10 > vfs.zfs.vdev.sync_read_min_active 10 > vfs.zfs.vdev.max_active 1000 > vfs.zfs.vdev.async_write_active_max_dirty_percent60 > vfs.zfs.vdev.async_write_active_min_dirty_percent30 > vfs.zfs.vdev.mirror.non_rotating_seek_inc1 > vfs.zfs.vdev.mirror.non_rotating_inc 0 > vfs.zfs.vdev.mirror.rotating_seek_offset1048576 > vfs.zfs.vdev.mirror.rotating_seek_inc 5 > vfs.zfs.vdev.mirror.rotating_inc 0 > vfs.zfs.vdev.trim_on_init 1 > vfs.zfs.vdev.cache.bshift 16 > vfs.zfs.vdev.cache.size 0 > vfs.zfs.vdev.cache.max 16384 > vfs.zfs.vdev.validate_skip 0 > vfs.zfs.vdev.max_ms_shift 34 > vfs.zfs.vdev.default_ms_shift 29 > vfs.zfs.vdev.max_ms_count_limit 131072 > vfs.zfs.vdev.min_ms_count 16 > vfs.zfs.vdev.default_ms_count 200 > vfs.zfs.txg.timeout 5 > vfs.zfs.space_map_ibs 14 > vfs.zfs.special_class_metadata_reserve_pct25 > vfs.zfs.user_indirect_is_special 1 > vfs.zfs.ddt_data_is_special 1 > vfs.zfs.spa_allocators 4 > vfs.zfs.spa_min_slop 134217728 > vfs.zfs.spa_slop_shift 5 > vfs.zfs.spa_asize_inflation 24 > vfs.zfs.deadman_enabled 1 > vfs.zfs.deadman_checktime_ms 5000 > vfs.zfs.deadman_synctime_ms 1000000 > vfs.zfs.debugflags 0 > vfs.zfs.recover 0 > vfs.zfs.spa_load_verify_data 1 > vfs.zfs.spa_load_verify_metadata 1 > vfs.zfs.spa_load_verify_maxinflight 10000 > vfs.zfs.max_missing_tvds_scan 0 > vfs.zfs.max_missing_tvds_cachefile 2 > vfs.zfs.max_missing_tvds 0 > vfs.zfs.spa_load_print_vdev_tree 0 > vfs.zfs.ccw_retry_interval 300 > vfs.zfs.check_hostid 1 > vfs.zfs.multihost_fail_intervals 10 > vfs.zfs.multihost_import_intervals 20 > vfs.zfs.multihost_interval 1000 > vfs.zfs.mg_fragmentation_threshold 85 > vfs.zfs.mg_noalloc_threshold 0 > vfs.zfs.condense_pct 200 > vfs.zfs.metaslab_sm_blksz 4096 > vfs.zfs.metaslab.bias_enabled 1 > vfs.zfs.metaslab.lba_weighting_enabled 1 > vfs.zfs.metaslab.fragmentation_factor_enabled1 > vfs.zfs.metaslab.preload_enabled 1 > vfs.zfs.metaslab.preload_limit 3 > vfs.zfs.metaslab.unload_delay 8 > vfs.zfs.metaslab.load_pct 50 > vfs.zfs.metaslab.min_alloc_size 33554432 > vfs.zfs.metaslab.df_free_pct 4 > vfs.zfs.metaslab.df_alloc_threshold 131072 > vfs.zfs.metaslab.debug_unload 0 > vfs.zfs.metaslab.debug_load 0 > vfs.zfs.metaslab.fragmentation_threshold70 > vfs.zfs.metaslab.force_ganging 16777217 > vfs.zfs.free_bpobj_enabled 1 > vfs.zfs.free_max_blocks -1 > vfs.zfs.zfs_scan_checkpoint_interval 7200 > vfs.zfs.zfs_scan_legacy 0 > vfs.zfs.no_scrub_prefetch 0 > vfs.zfs.no_scrub_io 0 > vfs.zfs.resilver_min_time_ms 3000 > vfs.zfs.free_min_time_ms 1000 > vfs.zfs.scan_min_time_ms 1000 > vfs.zfs.scan_idle 50 > vfs.zfs.scrub_delay 4 > vfs.zfs.resilver_delay 2 > vfs.zfs.zfetch.array_rd_sz 1048576 > vfs.zfs.zfetch.max_idistance 67108864 > vfs.zfs.zfetch.max_distance 8388608 > vfs.zfs.zfetch.min_sec_reap 2 > vfs.zfs.zfetch.max_streams 8 > vfs.zfs.prefetch_disable 0 > vfs.zfs.delay_scale 500000 > vfs.zfs.delay_min_dirty_percent 60 > vfs.zfs.dirty_data_sync_pct 20 > vfs.zfs.dirty_data_max_percent 10 > vfs.zfs.dirty_data_max_max 4294967296 > vfs.zfs.dirty_data_max 4294967296 > vfs.zfs.max_recordsize 1048576 > vfs.zfs.default_ibs 17 > vfs.zfs.default_bs 9 > vfs.zfs.send_holes_without_birth_time 1 > vfs.zfs.mdcomp_disable 0 > vfs.zfs.per_txg_dirty_frees_percent 5 > vfs.zfs.nopwrite_enabled 1 > vfs.zfs.dedup.prefetch 1 > vfs.zfs.dbuf_cache_lowater_pct 10 > vfs.zfs.dbuf_cache_hiwater_pct 10 > vfs.zfs.dbuf_metadata_cache_overflow 0 > vfs.zfs.dbuf_metadata_cache_shift 6 > vfs.zfs.dbuf_cache_shift 5 > vfs.zfs.dbuf_metadata_cache_max_bytes 1025282816 > vfs.zfs.dbuf_cache_max_bytes 2050565632 > vfs.zfs.arc_min_prescient_prefetch_ms 6 > vfs.zfs.arc_min_prefetch_ms 1 > vfs.zfs.l2c_only_size 0 > vfs.zfs.mfu_ghost_data_esize 7778263552 > vfs.zfs.mfu_ghost_metadata_esize 16851792896 > vfs.zfs.mfu_ghost_size 24630056448 > vfs.zfs.mfu_data_esize 3059418112 > vfs.zfs.mfu_metadata_esize 28641792 > vfs.zfs.mfu_size 6399023104 > vfs.zfs.mru_ghost_data_esize 2199812096 > vfs.zfs.mru_ghost_metadata_esize 6289682432 > vfs.zfs.mru_ghost_size 8489494528 > vfs.zfs.mru_data_esize 22781456384 > vfs.zfs.mru_metadata_esize 309155840 > vfs.zfs.mru_size 23847875584 > vfs.zfs.anon_data_esize 0 > vfs.zfs.anon_metadata_esize 0 > vfs.zfs.anon_size 8556544 > vfs.zfs.l2arc_norw 1 > vfs.zfs.l2arc_feed_again 1 > vfs.zfs.l2arc_noprefetch 1 > vfs.zfs.l2arc_feed_min_ms 200 > vfs.zfs.l2arc_feed_secs 1 > vfs.zfs.l2arc_headroom 2 > vfs.zfs.l2arc_write_boost 8388608 > vfs.zfs.l2arc_write_max 8388608 > vfs.zfs.arc_meta_strategy 1 > vfs.zfs.arc_meta_limit 15833624576 > vfs.zfs.arc_free_target 346902 > vfs.zfs.arc_kmem_cache_reap_retry_ms 1000 > vfs.zfs.compressed_arc_enabled 1 > vfs.zfs.arc_grow_retry 60 > vfs.zfs.arc_shrink_shift 7 > vfs.zfs.arc_average_blocksize 8192 > vfs.zfs.arc_no_grow_shift 5 > vfs.zfs.arc_min 8202262528 > vfs.zfs.arc_max 39334498304 > vfs.zfs.abd_chunk_size 4096 > vfs.zfs.abd_scatter_enabled 1 > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > > > From owner-freebsd-stable@freebsd.org Mon Jul 20 13:26:32 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id ADC6235ED27 for ; Mon, 20 Jul 2020 13:26:32 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4B9MvD462Zz42N2 for ; Mon, 20 Jul 2020 13:26:32 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: from mail-qt1-f170.google.com (mail-qt1-f170.google.com [209.85.160.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) (Authenticated sender: kevans) by smtp.freebsd.org (Postfix) with ESMTPSA id 6B64019658 for ; Mon, 20 Jul 2020 13:26:32 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: by mail-qt1-f170.google.com with SMTP id w27so12889158qtb.7 for ; Mon, 20 Jul 2020 06:26:32 -0700 (PDT) X-Gm-Message-State: AOAM532fg3lRp7UHIqZ1DdhDpa9PzbDymGgq2dCXMD+IX2uymwcO2Bgh tVyfwwjzSIqZgoXewPi9eah2NlDrWayVSQYnuZo= X-Google-Smtp-Source: ABdhPJypVOA2bR2Wke2R+UXYYcfVvnywVau66cUSJvrieapTXGFyZm7fwIYbD7R8iOCRIYJPXqTFlhCYBsOdLqASsxM= X-Received: by 2002:ac8:464f:: with SMTP id f15mr22888549qto.211.1595251591961; Mon, 20 Jul 2020 06:26:31 -0700 (PDT) MIME-Version: 1.0 References: <557add91-44cf-c981-8965-9bab90498ea1@digital-chaos.com> In-Reply-To: <557add91-44cf-c981-8965-9bab90498ea1@digital-chaos.com> From: Kyle Evans Date: Mon, 20 Jul 2020 08:26:20 -0500 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: ls colour (COLORTERM / CLICOLOR) To: James Wright Cc: FreeBSD-STABLE Mailing List Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Jul 2020 13:26:32 -0000 On Sat, Jul 18, 2020 at 7:51 PM James Wright wrote: > > > Updated to 12.1-STABLE r363215 a few days ago (previous build was > circa 1st June) > but seem to have lost "ls" colour output with "COLORTERM=yes" set in my env. > > Setting "CLICOLOR=yes" seems to enable it again, however the man page > states that > setting either should work? > Hi, Indeed, sorry for the flip-flopping. The short version of the situation is that I had flipped ls(1) to --color=auto by default based on a misunderstanding of defaults elsewhere due to shell aliases that I hadn't realized were in use. The ls(1) binary is historically and almost universally configured for non-colored by default where color support exists, and you should instead use appropriate shell alias for ls=`ls -G` or `ls --color=auto`. I can see where the manpage could describe the differences a little better. CLICOLOR (On FreeBSD) historically meant that we'll enable color if the terminal supports it, and setting it would have the same effect as the above shell alias. COLORTERM is less aggressive and won't imply any specific --color option, you would still --color=auto to go with it for it to have any effect. Thanks, Kyle Evans From owner-freebsd-stable@freebsd.org Mon Jul 20 17:40:37 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3AB2A364ECA for ; Mon, 20 Jul 2020 17:40:37 +0000 (UTC) (envelope-from james.wright@digital-chaos.com) Received: from mout.kundenserver.de (mout.kundenserver.de [217.72.192.73]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mout.kundenserver.de", Issuer "TeleSec ServerPass Class 2 CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4B9TXN4f9xz4Ltv; Mon, 20 Jul 2020 17:40:35 +0000 (UTC) (envelope-from james.wright@digital-chaos.com) Received: from [192.168.0.15] ([82.18.193.38]) by mrelayeu.kundenserver.de (mreue107 [212.227.15.179]) with ESMTPSA (Nemesis) id 1M27ep-1jzVaS36jZ-002XuN; Mon, 20 Jul 2020 19:40:33 +0200 Subject: Re: ls colour (COLORTERM / CLICOLOR) To: Kyle Evans Cc: FreeBSD-STABLE Mailing List References: <557add91-44cf-c981-8965-9bab90498ea1@digital-chaos.com> From: James Wright Message-ID: <39ad4771-2c87-6e9c-b683-697a88b41ac9@digital-chaos.com> Date: Mon, 20 Jul 2020 18:40:36 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-Provags-ID: V03:K1:hQ9Zb8VatZN8eP7ePa2gKc35v/XJdtRsmRdI65yPolTOyqJvKo2 dkkpwm9s7hvwVhk2Qk3VwnYiMbXnrpteI/pXZy6ItbX/lQ5HXFYvOpzqkQnxtRqIxc8Kw/X WLwYZuYjy2PM1lxYAgpW9jOyuCy7X7rU8BwdD43QpSZUng1F0L4Zo6l58jVBu9HO5chRscM 91x6K+/RKX357BuXVrWBQ== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:3qQTwLzsYhE=:qLA4FDdvQEqyBWehjx2iEP fyjeUpzVWlNgDe1mKNu3FnO7S5sakRhTLQiiTG8Fb1zafJXQAqCW1TJacqyYsKrZ7IQ2vJZV6 pWawY4vQVPpnoMZb4p4ZryhlnyWMBkKYIP+KQaaaNX+wlMA7sVuFHWtD2a1XRRUUk/yFgr3NL fN2pe6Qxm6ePlq8vazqzCMRr8KzuWUXsKAR/wAS8qjXPcV9y817maJ+hO6pUcQJi+qxnQ5VSb BDtAEQrH74I9xMjR8PXzMyXNa7JiUoTT0XzI13KNMUjIURjZFVi82soXOY+1XPLI0HMAHYJAX FiUy5xSkmHoz2oKDcJYsmoA8CtKAadHmSRbaNaI22lY/LVZZQG/DQxqcqgG2YIoYp6xpsKbT2 8tCXYtjghL7Ps5pbMCGE6tC/AH7oIjKCdkMCm8f86tYmBgc5HXsRMT7Q5BbqbuWygZfJul56l joYQQ2opdLDsG6i9XI6kCrTks0O29aLowOxQwnBmfCzMTmPrImxy1Bmf8V0yV1O6+n5MbocWg 0CPv1Hkx7RfMHAq6s6f6zlBsxB/43HyEjn7NmK/AZmnE4/wPxSkNcFHJvTWrk3EVIpwG8ICgZ AtefzGt5NGbsnD+qpdoj+pncljykbKeKMle4qTcAUapl+Aoc0F8sxKTKETTXexT3W04wfHk0Q xHc0kLFDUJsNIUbLMJWLW31UvFYRBJq3vK6W7Xzt5dod4st+LMSW8azaDxbLVKPaO2h3XyUat 65Zy3uOqAIXzUVKHxqeHAVlw7gL9m8iI36yZ7zukN1zuumLBsDnO8Wo7P5Th5QJdb4CmncZKQ 0dDRpTAS+Pg7u6LL15Q4X7G6+UQ/yBdoKvXKi6zimChadnBpTwWSD9PX68lvvBJyF92igsqGF JxC2ZM1w853J+kE3xzqw== X-Rspamd-Queue-Id: 4B9TXN4f9xz4Ltv X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; ASN(0.00)[asn:8560, ipnet:217.72.192.0/20, country:DE]; REPLY(-4.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Jul 2020 17:40:37 -0000 On 20/07/2020 14:26, Kyle Evans wrote: > On Sat, Jul 18, 2020 at 7:51 PM James Wright > wrote: >> >> Updated to 12.1-STABLE r363215 a few days ago (previous build was >> circa 1st June) >> but seem to have lost "ls" colour output with "COLORTERM=yes" set in my env. >> >> Setting "CLICOLOR=yes" seems to enable it again, however the man page >> states that >> setting either should work? >> > Hi, > > Indeed, sorry for the flip-flopping. The short version of the > situation is that I had flipped ls(1) to --color=auto by default based > on a misunderstanding of defaults elsewhere due to shell aliases that > I hadn't realized were in use. The ls(1) binary is historically and > almost universally configured for non-colored by default where color > support exists, and you should instead use appropriate shell alias for > ls=`ls -G` or `ls --color=auto`. > > I can see where the manpage could describe the differences a little > better. CLICOLOR (On FreeBSD) historically meant that we'll enable > color if the terminal supports it, and setting it would have the same > effect as the above shell alias. COLORTERM is less aggressive and > won't imply any specific --color option, you would still --color=auto > to go with it for it to have any effect. > > Thanks, > > Kyle Evans Thank you for the clarifying the diferences between CLICOLOR and COLORTERM, that makes sense to me now. I'll set the shell alias and remove COLORTERM. Only raised this in case it was an unintended consequence of recent changes. :-) From owner-freebsd-stable@freebsd.org Mon Jul 20 19:15:06 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3E484367C2A for ; Mon, 20 Jul 2020 19:15:06 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [IPv6:2a01:4f8:c2c:26d8::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4B9WdP1px2z4T28 for ; Mon, 20 Jul 2020 19:15:04 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13:0:0:0:5]) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id 06KJEj0o096533 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 20 Jul 2020 19:14:46 GMT (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: mike@sentex.net Received: from [10.58.0.10] (dadvw [10.58.0.10]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id 06KJEn9T032327 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Tue, 21 Jul 2020 02:14:49 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: zfs meta data slowness To: mike tancsa , FreeBSD-STABLE Mailing List References: From: Eugene Grosbein Message-ID: Date: Tue, 21 Jul 2020 02:14:42 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=0.3 required=5.0 tests=BAYES_00,LOCAL_FROM, NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.2 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * -0.0 SPF_PASS SPF: sender matches SPF record * 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record * 2.6 LOCAL_FROM From my domains * -0.0 NICE_REPLY_A Looks like a legit reply (A) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on hz.grosbein.net X-Rspamd-Queue-Id: 4B9WdP1px2z4T28 X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=permerror (mx1.freebsd.org: domain of eugen@grosbein.net uses mechanism not recognized by this client) smtp.mailfrom=eugen@grosbein.net X-Spamd-Result: default: False [-0.82 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.27)[-0.269]; FROM_HAS_DN(0.00)[]; NEURAL_SPAM_SHORT(0.12)[0.121]; NEURAL_HAM_LONG(-0.57)[-0.570]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[grosbein.net]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; R_SPF_PERMFAIL(0.00)[empty SPF record]; RCPT_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Jul 2020 19:15:06 -0000 19.07.2020 21:17, mike tancsa wrote: > Are there any tweaks that can be done to speed up or improve zfs > metadata performance ? I have a backup server with a lot of snapshots > (40,000) and just doing a listing can take a great deal of time. Best > case scenario is about 24 seconds, worst case, I have seen it up to 15 > minutes. (FreeBSD 12.1-STABLE r363078) I believe you have to perform kernel profiling for your specific use case. From owner-freebsd-stable@freebsd.org Mon Jul 20 21:21:07 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C6C8B36A119 for ; Mon, 20 Jul 2020 21:21:07 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from vtr.rulingia.com (vtr.rulingia.com [IPv6:2001:19f0:5801:ebe:5400:1ff:fe53:30fd]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vtr.rulingia.com", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4B9ZQp4LWwz4ZpM for ; Mon, 20 Jul 2020 21:21:06 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from server.rulingia.com (ppp239-208.static.internode.on.net [59.167.239.208]) by vtr.rulingia.com (8.15.2/8.15.2) with ESMTPS id 06KLKpbh073719 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 21 Jul 2020 07:20:56 +1000 (AEST) (envelope-from peter@rulingia.com) X-Bogosity: Ham, spamicity=0.000000 Received: from server.rulingia.com (localhost.rulingia.com [127.0.0.1]) by server.rulingia.com (8.15.2/8.15.2) with ESMTPS id 06KLKi93057864 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 21 Jul 2020 07:20:44 +1000 (AEST) (envelope-from peter@server.rulingia.com) Received: (from peter@localhost) by server.rulingia.com (8.15.2/8.15.2/Submit) id 06KLKiNv057863; Tue, 21 Jul 2020 07:20:44 +1000 (AEST) (envelope-from peter) Date: Tue, 21 Jul 2020 07:20:44 +1000 From: Peter Jeremy To: Konstantin Belousov Cc: freebsd-stable@freebsd.org Subject: Re: svn commit: r362848 - in stable/12/sys: net netinet sys Message-ID: <20200720212044.GA55887@server.rulingia.com> References: <202007011803.061I3cTs089322@repo.freebsd.org> <20200719112102.GA15535@server.rulingia.com> <20200719114828.GD44314@kib.kiev.ua> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="/9DWx/yDrRhgMJTb" Content-Disposition: inline In-Reply-To: <20200719114828.GD44314@kib.kiev.ua> X-PGP-Key: http://www.rulingia.com/keys/peter.pgp X-Rspamd-Queue-Id: 4B9ZQp4LWwz4ZpM X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of peter@rulingia.com designates 2001:19f0:5801:ebe:5400:1ff:fe53:30fd as permitted sender) smtp.mailfrom=peter@rulingia.com X-Spamd-Result: default: False [-4.97 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.96)[-0.958]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-1.01)[-1.013]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; DMARC_NA(0.00)[rulingia.com]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.60)[-0.597]; RCPT_COUNT_TWO(0.00)[2]; SIGNED_PGP(-2.00)[]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:20473, ipnet:2001:19f0:5800::/38, country:US]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Jul 2020 21:21:07 -0000 --/9DWx/yDrRhgMJTb Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2020-Jul-19 14:48:28 +0300, Konstantin Belousov wr= ote: >On Sun, Jul 19, 2020 at 09:21:02PM +1000, Peter Jeremy wrote: >> I'm sending this to -stable, rather than the src groups because I >> don't believe the problem is the commit itself, rather the commit >> has uncovered a latent problem elsewhere. >>=20 >> On 2020-Jul-01 18:03:38 +0000, Michael Tuexen wrote: >> >Author: tuexen >> >Date: Wed Jul 1 18:03:38 2020 >> >New Revision: 362848 >> >URL: https://svnweb.freebsd.org/changeset/base/362848 >> > >> >Log: >> > MFC r353480: Use event handler in SCTP >>=20 >> I have no idea how, but this update breaks booting amd64 for me (r362847 >> works and this doesn't). I have a custom kernel with ZFS but no SCTP so= I >> have no real idea how this could break booting - presumably the >> eventhandler change has uncovered a bug somewhere else. >>=20 >> The symptoms are that I get: >> Mounting from zfs:zroot/ROOT/r363310 failed with error 6; retrying for 3= more seconds >> Mounting from zfs:zroot/ROOT/r363310 failed with error 6 >>=20 >> (r363310 is where I was trying to update to and I didn't change the BE >> name as I was searching for the problem and error 6 is ENXIO). >>=20 >> I tried to reproduce the problem with GENERIC but it hangs after >> displaying the EFI framebuffer information (I've seen that before and >> suspect it is a loader problem but haven't dug into it). I've confirmed that particular problem is bug 209821. I've disabled EFI and GENERIC r362848 boots and runs successfully. >> Does anyone have any ideas? > >Did you checked that the physical devices where your ZFS pool is located, >are detected, and that kernel messages for their drivers are as usual ? >Overall, is there anything strange in the verbose dmesg ? There's nothing obviously strange (in particular, I can see the physical boot/root disk) but the faulty kernel appears to have moved the msgbuf somewhere unexpected so it's not saved across reboots and I'm limited to eyeballing the messages via DDB. Since GENERIC worked, I did some more experimenting and tracked the problem down to a lack of "options ACPI_DMAR" in my kernel config. That makes more sense, though I have no idea why it suddenly became mandatory for my system. --=20 Peter Jeremy --/9DWx/yDrRhgMJTb Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEE2M6l8vfIeOACl4uUHZIUommfjLIFAl8WCqRfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEQ4 Q0VBNUYyRjdDODc4RTAwMjk3OEI5NDFEOTIxNEEyNjk5RjhDQjIACgkQHZIUommf jLLzuA//ccCK8h3qEGfVXs+JQPXtrL0FPq2GQ27iTFK3PhGUaBQoeJ08wBkhO8jE Z/7RYS14vwURWb24YGSCHk2RknHbay/WH/gpBwOU5aDD0zNG3r5JiSFcjy6/9sRL SYl5Dtt61FjxzAcLwUHUYv81aYND8g5oqznJVrvL9sf2h79/5IH23KY7y5WnhcXo ygGAwzNpPCBs8gk++BbhcJxX1XqLkbyk1Ty5NehYOXW8U8PtTSzKArhq3na2r1iD gwLJJznzWeaCOJHkVdy9l+cT+d4brSOWiMoHrmeSQofd/UT6MxCFDi+kLEHOSW22 Za2t18dBq8zltiBbcJXl+KgQ6KFrxZtEJSm1pruVkSSHXL+BdXjY9SxSvMbnJ0aD 1izpqCsf4huBcegTA9aOA00Z3wDfoSdExBxNiATq2qb98+EI7Lr0AIyN3FmjMjbj 8bIx4TFhJtw4Isctms/5APMZN+LpRFMMKm+7tYdxMVTFYQBgL10CYTeUQBpT3sre 3wj5CsT+s15sRBvxtezRpBtcc7ol0yGIUCELdrVJ1EBJc01NigYOA9t7sSfHixmd qgDzUg+JDuUFoSHJonIREVbNFWwxDANPsUTzmuYqbhL1VS0smCsYmKbuZgL/P/Bl UICxlcvpKmWhGj2ByrMaJB3WkU4Fm+TfwIVMzRC73Lvg3TYoweE= =pOkP -----END PGP SIGNATURE----- --/9DWx/yDrRhgMJTb-- From owner-freebsd-stable@freebsd.org Mon Jul 20 21:47:33 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0CC0F36AB43 for ; Mon, 20 Jul 2020 21:47:33 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4B9b1H1kRMz4cSw for ; Mon, 20 Jul 2020 21:47:30 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id 06KLlNLQ006666 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Tue, 21 Jul 2020 00:47:26 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 06KLlNLQ006666 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id 06KLlNhW006665; Tue, 21 Jul 2020 00:47:23 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 21 Jul 2020 00:47:23 +0300 From: Konstantin Belousov To: Peter Jeremy Cc: freebsd-stable@freebsd.org Subject: Re: svn commit: r362848 - in stable/12/sys: net netinet sys Message-ID: <20200720214723.GP44314@kib.kiev.ua> References: <202007011803.061I3cTs089322@repo.freebsd.org> <20200719112102.GA15535@server.rulingia.com> <20200719114828.GD44314@kib.kiev.ua> <20200720212044.GA55887@server.rulingia.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20200720212044.GA55887@server.rulingia.com> X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on tom.home X-Rspamd-Queue-Id: 4B9b1H1kRMz4cSw X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=gmail.com (policy=none); spf=softfail (mx1.freebsd.org: 2001:470:d5e7:1::1 is neither permitted nor denied by domain of kostikbel@gmail.com) smtp.mailfrom=kostikbel@gmail.com X-Spamd-Result: default: False [-0.53 / 15.00]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.65)[-0.651]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[text/plain]; HAS_XAW(0.00)[]; R_SPF_SOFTFAIL(0.00)[~all]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.21)[-0.208]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_SPAM_LONG(0.33)[0.333]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; FREEMAIL_ENVFROM(0.00)[gmail.com]; DMARC_POLICY_SOFTFAIL(0.10)[gmail.com : No valid SPF, No valid DKIM,none] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Jul 2020 21:47:33 -0000 On Tue, Jul 21, 2020 at 07:20:44AM +1000, Peter Jeremy wrote: > On 2020-Jul-19 14:48:28 +0300, Konstantin Belousov wrote: > >On Sun, Jul 19, 2020 at 09:21:02PM +1000, Peter Jeremy wrote: > >> I'm sending this to -stable, rather than the src groups because I > >> don't believe the problem is the commit itself, rather the commit > >> has uncovered a latent problem elsewhere. > >> > >> On 2020-Jul-01 18:03:38 +0000, Michael Tuexen wrote: > >> >Author: tuexen > >> >Date: Wed Jul 1 18:03:38 2020 > >> >New Revision: 362848 > >> >URL: https://svnweb.freebsd.org/changeset/base/362848 > >> > > >> >Log: > >> > MFC r353480: Use event handler in SCTP > >> > >> I have no idea how, but this update breaks booting amd64 for me (r362847 > >> works and this doesn't). I have a custom kernel with ZFS but no SCTP so I > >> have no real idea how this could break booting - presumably the > >> eventhandler change has uncovered a bug somewhere else. > >> > >> The symptoms are that I get: > >> Mounting from zfs:zroot/ROOT/r363310 failed with error 6; retrying for 3 more seconds > >> Mounting from zfs:zroot/ROOT/r363310 failed with error 6 > >> > >> (r363310 is where I was trying to update to and I didn't change the BE > >> name as I was searching for the problem and error 6 is ENXIO). > >> > >> I tried to reproduce the problem with GENERIC but it hangs after > >> displaying the EFI framebuffer information (I've seen that before and > >> suspect it is a loader problem but haven't dug into it). > > I've confirmed that particular problem is bug 209821. I've disabled > EFI and GENERIC r362848 boots and runs successfully. Did you mis-typed the PR number ? The referenced bug talks about very early hang, while your report said that kernel boots up to the point of mounting root. > > >> Does anyone have any ideas? > > > >Did you checked that the physical devices where your ZFS pool is located, > >are detected, and that kernel messages for their drivers are as usual ? > >Overall, is there anything strange in the verbose dmesg ? > > There's nothing obviously strange (in particular, I can see the physical > boot/root disk) but the faulty kernel appears to have moved the msgbuf > somewhere unexpected so it's not saved across reboots and I'm limited to > eyeballing the messages via DDB. > > Since GENERIC worked, I did some more experimenting and tracked the > problem down to a lack of "options ACPI_DMAR" in my kernel config. > That makes more sense, though I have no idea why it suddenly became > mandatory for my system. No, this does not make too much sense either, since DMAR is disabled by default. Did you enabled it ? BTW, you are using stable, right ? There were some code reorganization commits in HEAD moving DMAR code around, but they were not merged to stable. From owner-freebsd-stable@freebsd.org Tue Jul 21 10:31:47 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C005737E5B6 for ; Tue, 21 Jul 2020 10:31:47 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from vtr.rulingia.com (vtr.rulingia.com [IPv6:2001:19f0:5801:ebe:5400:1ff:fe53:30fd]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vtr.rulingia.com", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4B9vz56VLmz4RZj for ; Tue, 21 Jul 2020 10:31:45 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from server.rulingia.com (ppp239-208.static.internode.on.net [59.167.239.208]) by vtr.rulingia.com (8.15.2/8.15.2) with ESMTPS id 06LAVZNh080059 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 21 Jul 2020 20:31:40 +1000 (AEST) (envelope-from peter@rulingia.com) X-Bogosity: Ham, spamicity=0.000000 Received: from server.rulingia.com (localhost.rulingia.com [127.0.0.1]) by server.rulingia.com (8.15.2/8.15.2) with ESMTPS id 06LAVTLL083291 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 21 Jul 2020 20:31:29 +1000 (AEST) (envelope-from peter@server.rulingia.com) Received: (from peter@localhost) by server.rulingia.com (8.15.2/8.15.2/Submit) id 06LAVTPg083290; Tue, 21 Jul 2020 20:31:29 +1000 (AEST) (envelope-from peter) Date: Tue, 21 Jul 2020 20:31:29 +1000 From: Peter Jeremy To: Konstantin Belousov Cc: freebsd-stable@freebsd.org Subject: Re: svn commit: r362848 - in stable/12/sys: net netinet sys Message-ID: <20200721103129.GA58157@server.rulingia.com> References: <202007011803.061I3cTs089322@repo.freebsd.org> <20200719112102.GA15535@server.rulingia.com> <20200719114828.GD44314@kib.kiev.ua> <20200720212044.GA55887@server.rulingia.com> <20200720214723.GP44314@kib.kiev.ua> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="ew6BAiZeqk4r7MaW" Content-Disposition: inline In-Reply-To: <20200720214723.GP44314@kib.kiev.ua> X-PGP-Key: http://www.rulingia.com/keys/peter.pgp X-Rspamd-Queue-Id: 4B9vz56VLmz4RZj X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of peter@rulingia.com designates 2001:19f0:5801:ebe:5400:1ff:fe53:30fd as permitted sender) smtp.mailfrom=peter@rulingia.com X-Spamd-Result: default: False [-4.54 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.05)[-1.053]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-1.02)[-1.018]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; DMARC_NA(0.00)[rulingia.com]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.07)[-0.070]; RCPT_COUNT_TWO(0.00)[2]; SIGNED_PGP(-2.00)[]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:20473, ipnet:2001:19f0:5800::/38, country:US]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Jul 2020 10:31:47 -0000 --ew6BAiZeqk4r7MaW Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2020-Jul-21 00:47:23 +0300, Konstantin Belousov wr= ote: >On Tue, Jul 21, 2020 at 07:20:44AM +1000, Peter Jeremy wrote: >> On 2020-Jul-19 14:48:28 +0300, Konstantin Belousov = wrote: >> >On Sun, Jul 19, 2020 at 09:21:02PM +1000, Peter Jeremy wrote: >> >> The symptoms are that I get: >> >> Mounting from zfs:zroot/ROOT/r363310 failed with error 6; retrying fo= r 3 more seconds >> >> Mounting from zfs:zroot/ROOT/r363310 failed with error 6 >> >>=20 >> >> (r363310 is where I was trying to update to and I didn't change the BE >> >> name as I was searching for the problem and error 6 is ENXIO). >> >>=20 >> >> I tried to reproduce the problem with GENERIC but it hangs after >> >> displaying the EFI framebuffer information (I've seen that before and >> >> suspect it is a loader problem but haven't dug into it). >>=20 >> I've confirmed that particular problem is bug 209821. I've disabled >> EFI and GENERIC r362848 boots and runs successfully. >Did you mis-typed the PR number ? The referenced bug talks about very >early hang, while your report said that kernel boots up to the point of >mounting root. My failure was with a custom kernel. Once I narrowed the problem to a commit that seemed unrelated to my problem, I tried to boot a GENERIC kernel at r362848. The GENERIC kernel boot failed much earlier due to the EFI problem documented in PR 209821. When I disabled EFI, then the GENERIC kernel worked, showing that my problem was due to my custom kernel. >> Since GENERIC worked, I did some more experimenting and tracked the >> problem down to a lack of "options ACPI_DMAR" in my kernel config. >> That makes more sense, though I have no idea why it suddenly became >> mandatory for my system. >No, this does not make too much sense either, since DMAR is disabled >by default. Did you enabled it ? "options ACPI_DMAR" has been in GENERIC since you first submitted the DMAR code was in r257251. I haven't ever set the hw.dmar.enable=3D1 loader tunable but it's not at all obvious that a kernel built without "options ACPI_DMAR" is functionally equivalent to a kernel that has DMAR compiled in but disabled - there's a lot of IOMMU manipulation code that is purely conditional on ACPI_DMAR. That said, I'm not using virtualisation and haven't actually enabled DMAR in the loader so I suspect that I've only masked the real issue. I currently have INVARIANTS and WITNESS but will look into some of the more extensive debugging options. (It looks like I missed the addition of "options ACPI_DMAR" when I was updating my custom kernel config with the differences between r250963 and r259512 about 8 years ago, and it hasn't caused any obvious problems until now. Obviously, I need to do a more careful review of my custom kernel config against GENERIC/NOTES). >BTW, you are using stable, right ? There were some code reorganization >commits in HEAD moving DMAR code around, but they were not merged to >stable. I'm using 12-STABLE. --=20 Peter Jeremy --ew6BAiZeqk4r7MaW Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEE2M6l8vfIeOACl4uUHZIUommfjLIFAl8Ww/tfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEQ4 Q0VBNUYyRjdDODc4RTAwMjk3OEI5NDFEOTIxNEEyNjk5RjhDQjIACgkQHZIUommf jLJ//Q/+Oi96z1m8+AZVVuUJIdmINkWuxJJUqXBrsXZVPLZqlDNM5E3UbLq1VdmZ JpnFgeXX2Y93vF+p4lFc28rQTaXkHXHvFKvDKNdF45LlkHD/AhnrHzd/mtZW1IuX GotQgkqSLHs8thhS88kCh6Y0ZmFV3gYqwiWa5hdhWYi06VEz45ukUflc9dlsxb54 cYYYCi/999rdqZ9o0m+aR84M3XMBJoLeVDxFNVykplDmLATGLFw6igqlbXD52hCK KJ92pzmB0ia51cGr3tK0SDBzFfLe0Xirf/K6k8xN0Y/GUxVOcrvMYHW5m1V7hk/U +jP2qtp4yps8AboyYb0B4JCylFFIMiqMsHURuK5uMgPLrdyjIau717Hor606KK94 uDb6OA/zX/TZxGANYQAJWfdZJ3r9skDxzLuZYl1Ql9AVYICRGxmKScz0+IU7VJHz wDzEbydc/yvg4KPdewJPFmxCkypplZOL2y7SrOyAPenhTXmxlNOvez3GChBooIp4 Giu5zNVAI+YrVnyI2ORLbJSa1yKK4rixFl+zKhzK2qnFz+lUbRpJJ8y9mICGDi+q 1PKHOaqweaczCcg0enCLs2Wlie7FntbNldxxwPSU5/PPbJ6AxL4S4OZNmlwFQ0ke bXo0pxY79t30Ga4cI5O98FuR9+NisyG1nO7puIDVK8kpcnNqzDE= =ow9x -----END PGP SIGNATURE----- --ew6BAiZeqk4r7MaW-- From owner-freebsd-stable@freebsd.org Tue Jul 21 10:52:52 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 66B4737F013 for ; Tue, 21 Jul 2020 10:52:52 +0000 (UTC) (envelope-from zeising+freebsd@daemonic.se) Received: from mail.daemonic.se (mail.daemonic.se [IPv6:2607:f740:d:20::25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4B9wRQ3SRmz4TJh for ; Tue, 21 Jul 2020 10:52:50 +0000 (UTC) (envelope-from zeising+freebsd@daemonic.se) Received: from cid.daemonic.se (localhost [IPv6:::1]) by mail.daemonic.se (Postfix) with ESMTP id 4B9wRG1v3Rz3mjt; Tue, 21 Jul 2020 10:52:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=daemonic.se; h= content-transfer-encoding:content-language:content-type :content-type:in-reply-to:mime-version:user-agent:date:date :message-id:from:from:references:subject:subject:received :received; s=20151023; t=1595328761; bh=U2aKiUcGnB9tbDRi3kPVAGQm lgiaWYTGw91n+8MVHEk=; b=YP1wPGjoPhjRFAy7qmV9VS9VJ5Q/hBk1h/HrCDJJ +AEBOFgcxfgyPzWyZSv0pR15sLdJeyOQguc7zt4I/kia5RFc5BU2T+gHJ/nSU5PN kcSrqx2dqp875A0kMnWH/ynEpbNiRrsoFji6ZjqptKbdE9Bgo44LA4YgChAiUOOZ 1VM= X-Virus-Scanned: amavisd-new at daemonic.se Received: from mail.daemonic.se ([IPv6:::1]) (using TLS with cipher ECDHE-RSA-AES128-GCM-SHA256) by cid.daemonic.se (mailscanner.daemonic.se [IPv6:::1]) (amavisd-new, port 10587) with ESMTPS id Vb44q7kV--Tx; Tue, 21 Jul 2020 10:52:41 +0000 (UTC) Received: from garnet.daemonic.se (host-95-198-70-249.mobileonline.telia.com [95.198.70.249]) by mail.daemonic.se (Postfix) with ESMTPSA id 4B9wRF4Pn3z3lbm; Tue, 21 Jul 2020 10:52:41 +0000 (UTC) Subject: Re: drm i915kms on 12.1-STABLE (r363237) To: Nils Johannsen , freebsd-stable@freebsd.org References: From: Niclas Zeising Message-ID: <9ee4b0b9-50bf-c7cb-9581-bf8b5374a718@daemonic.se> Date: Tue, 21 Jul 2020 12:52:40 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4B9wRQ3SRmz4TJh X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=daemonic.se header.s=20151023 header.b=YP1wPGjo; dmarc=pass (policy=none) header.from=daemonic.se; spf=pass (mx1.freebsd.org: domain of zeising@daemonic.se designates 2607:f740:d:20::25 as permitted sender) smtp.mailfrom=zeising@daemonic.se X-Spamd-Result: default: False [-3.89 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; RCVD_COUNT_THREE(0.00)[4]; DKIM_TRACE(0.00)[daemonic.se:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[daemonic.se,none]; NEURAL_HAM_SHORT(-0.86)[-0.862]; FREEMAIL_TO(0.00)[gmx.de,freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:36236, ipnet:2607:f740:d::/48, country:US]; TAGGED_FROM(0.00)[freebsd]; MID_RHS_MATCH_FROM(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[95.198.70.249:received]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.01)[-1.009]; R_DKIM_ALLOW(-0.20)[daemonic.se:s=20151023]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.02)[-1.017]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Jul 2020 10:52:52 -0000 On 2020-07-17 11:49, Nils Johannsen wrote: > Hi all together, > yesterday I installed 12.1-STABLE [1] with kernel version `uname -K` 1201519 on my ThinkPad E490 with 'Intel UHD Graphics 620' [2]. > But if I install the 'drm-fbsd12.0-kmod' and load the driver `kldload /boot/modules/i915kms.ko` it will only show a black screen until I manually poweroff the notebook. > See the `/var/log/messages` [3] below. The 'drm-fbsd12.0-kmod' from pkg [4] behaves the sames as if I build it manually from ports. On 12.1-RELEASE it worked just fine. > Has anybody a hint for me, what might be wrong and how I should deal with it? > Regards, Nils > > > [1] FreeBSD-12.1-STABLE-amd64-20200702-r362880-memstick.img > > [2] pciconf -lv > ``` > vgapci0@pci0:0:2:0: class=0x030000 card=0x507217aa chip=0x3ea08086 rev=0x00 hdr=0x00 > vendor = 'Intel Corporation' > device = 'UHD Graphics 620 (Whiskey Lake)' > class = display > subclass = VGA > ``` Hi! Whiskey Lake isn't supported by drm-fbsd12.0-kmod. You need to upgrade to 13-CURRENT and use drm-devel-kmod. Regards -- Niclas From owner-freebsd-stable@freebsd.org Tue Jul 21 19:37:35 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0B708365D88 for ; Tue, 21 Jul 2020 19:37:35 +0000 (UTC) (envelope-from mike@sentex.net) Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [IPv6:2607:f3e0:0:3::19]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "pyroxene.sentex.ca", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BB84s5FXxz47ck for ; Tue, 21 Jul 2020 19:37:33 +0000 (UTC) (envelope-from mike@sentex.net) Received: from [IPv6:2607:f3e0:0:4:79db:7301:d591:b3bc] ([IPv6:2607:f3e0:0:4:79db:7301:d591:b3bc]) by pyroxene2a.sentex.ca (8.15.2/8.15.2) with ESMTPS id 06LJbWqo091687 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Tue, 21 Jul 2020 15:37:32 -0400 (EDT) (envelope-from mike@sentex.net) Subject: Re: zfs meta data slowness To: Ronald Klop , FreeBSD-STABLE Mailing List References: <1949194763.1.1595250243575@localhost> From: mike tancsa Autocrypt: addr=mike@sentex.net; keydata= mQENBFywzOMBCACoNFpwi5MeyEREiCeHtbm6pZJI/HnO+wXdCAWtZkS49weOoVyUj5BEXRZP xflV2ib2hflX4nXqhenaNiia4iaZ9ft3I1ebd7GEbGnsWCvAnob5MvDZyStDAuRxPJK1ya/s +6rOvr+eQiXYNVvfBhrCfrtR/esSkitBGxhUkBjOti8QwzD71JVF5YaOjBAs7jZUKyLGj0kW yDg4jUndudWU7G2yc9GwpHJ9aRSUN8e/mWdIogK0v+QBHfv/dsI6zVB7YuxCC9Fx8WPwfhDH VZC4kdYCQWKXrm7yb4TiVdBh5kgvlO9q3js1yYdfR1x8mjK2bH2RSv4bV3zkNmsDCIxjABEB AAG0HW1pa2UgdGFuY3NhIDxtaWtlQHNlbnRleC5uZXQ+iQFUBBMBCAA+FiEEmuvCXT0aY6hs 4SbWeVOEFl5WrMgFAlywzOYCGwMFCQHhM4AFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQ eVOEFl5WrMhnPAf7Bf+ola0V9t4i8rwCMGvzkssGaxY/5zNSZO9BgSgfN0WzgmBEOy/3R4km Yn5KH94NltJYAAE5hqkFmAwK6psOqAR9cxHrRfU+gV2KO8pCDc6K/htkQcd/mclJYpCHp6Eq EVJOiAxcNaYuHZkeMdXDuvvI5Rk82VHk84BGgxIqIrhLlkguoPbXOOa+8c/Mpb1sRAGZEOuX EzKNC49+GS9gKW6ISbanyPsGEcFyP7GKMzcHBPf3cPrewZQZ6gBoNscasL6IJeAQDqzQAxbU GjO0qBSMRgnLXK7+DJlxrYdHGXqNbV6AYsmHJ6c2WWWiuRviFBqXinlgJ2FnYebZPAfWiQ== Message-ID: <975657af-ccac-bbd1-e22b-86270c624226@sentex.net> Date: Tue, 21 Jul 2020 15:37:33 -0400 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 In-Reply-To: <1949194763.1.1595250243575@localhost> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 4BB84s5FXxz47ck X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of mike@sentex.net designates 2607:f3e0:0:3::19 as permitted sender) smtp.mailfrom=mike@sentex.net X-Spamd-Result: default: False [-0.83 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.997]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f3e0::/32]; NEURAL_HAM_LONG(-0.99)[-0.989]; MIME_GOOD(-0.10)[text/plain]; HFILTER_HELO_IP_A(1.00)[pyroxene2a.sentex.ca]; HFILTER_HELO_NORES_A_OR_MX(0.30)[pyroxene2a.sentex.ca]; DMARC_NA(0.00)[sentex.net]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_SPAM_SHORT(0.16)[0.160]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Jul 2020 19:37:35 -0000 Hi,     Thanks for the response. Reply in line On 7/20/2020 9:04 AM, Ronald Klop wrote: > Hi, > > My first suggestion would be to remove a lot of snapshots. But that my > not match your business case. As its a backup server, its sort of the point to have all those snapshots. > Maybe you can provide more information about your setup: > Amount of RAM, CPU? 64G, Xeon(R) CPU E3-1240 v6 @ 3.70GHz > output of "zpool status" # zpool status -x all pools are healthy > output of "zfs list" if possible to share its a big list # zfs list | wc      824    4120  107511 > Type of disks/ssds? old school Device Model:     WDC WD80EFAX-68KNBN0 > What is the load of the system? I/O per second, etc. its not cpu bound, disks are sometimes running at 100% based on gstat, but not always > Do you use dedup, GELI? no and no > Something else special about the setup. > output of "top -b" > ports are right now being built in a VM, but the problem (zrepl hanging) and zfs list -t snapshots taking forever happens regardless   PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND  4439 root         12  40   20  6167M  5762M kqread   3 535:13 200.00% bhyve 98783 root          2  21    0    16M  5136K hdr->b   4   0:01   1.95% zfs 76489 root         21  23    0   738M    54M uwait    1   2:18   0.88% zrepl 98784 root          1  21    0    13M  3832K piperd   3   0:01   0.59% zfs 99563 root          1  20    0    13M  4136K zio->i   4   0:00   0.39% zfs 16136 root         18  25    0   705M    56M uwait    3  29:58   0.00% zrepl-freebsd-amd64  1845 root          1  20    0    12M  3772K nanslp   7   5:54   0.00% ossec-syscheckd  1567 root          1  20    0    11M  2744K select   0   2:22   0.00% syslogd  1737 root         32  20    0    11M  2844K rpcsvc   6   1:40   0.00% nfsd  1660 root          1 -52   r0    11M    11M nanslp   5   1:18   0.00% watchdogd  1434 root          1  20    0  9988K   988K select   3   0:27   0.00% devd  2435 mdtancsa      1  20    0    20M  8008K select   0   0:21   0.00% sshd  1754 root          3  20    0    18M  3556K select   1   0:11   0.00% apcupsd  5917 root          1  20    0    11M  2672K select   2   0:06   0.00% script  1449 _pflogd       1  20    0    12M  3572K bpf      3   0:05   0.00% pflogd     ---Mike > That kind of information. > > Regards, > Ronald. > > > Van: mike tancsa > Datum: zondag, 19 juli 2020 16:17 > Aan: FreeBSD-STABLE Mailing List > Onderwerp: zfs meta data slowness >> >> Are there any tweaks that can be done to speed up or improve zfs >> metadata performance ? I have a backup server with a lot of snapshots >> (40,000)  and just doing a listing can take a great deal of time.  Best >> case scenario is about 24 seconds, worst case, I have seen it up to 15 >> minutes.  (FreeBSD 12.1-STABLE r363078) >> >> >> ARC Efficiency:                                 79.33b >>         Cache Hit Ratio:                92.81%  73.62b >>         Cache Miss Ratio:               7.19%   5.71b >>         Actual Hit Ratio:               92.78%  73.60b >> >>         Data Demand Efficiency:         96.47%  461.91m >>         Data Prefetch Efficiency:       1.00%   262.73m >> >>         CACHE HITS BY CACHE LIST: >>           Anonymously Used:             0.01%   3.86m >>           Most Recently Used:           3.91%   2.88b >>           Most Frequently Used:         96.06%  70.72b >>           Most Recently Used Ghost:     0.01%   5.31m >>           Most Frequently Used Ghost:   0.01%   10.47m >> >>         CACHE HITS BY DATA TYPE: >>           Demand Data:                  0.61%   445.60m >>           Prefetch Data:                0.00%   2.63m >>           Demand Metadata:              99.36%  73.15b >>           Prefetch Metadata:            0.03%   21.00m >> >>         CACHE MISSES BY DATA TYPE: >>           Demand Data:                  0.29%   16.31m >>           Prefetch Data:                4.56%   260.10m >>           Demand Metadata:              95.02%  5.42b >>           Prefetch Metadata:            0.14%   7.75m >> >> >> Other than increase the metadata max, I havent really changed any >> tuneables >> >> >> ZFS Tunables (sysctl): >>         kern.maxusers                           4416 >>         vm.kmem_size                            66691842048 >>         vm.kmem_size_scale                      1 >>         vm.kmem_size_min                        0 >>         vm.kmem_size_max                        1319413950874 >>         vfs.zfs.trim.max_interval               1 >>         vfs.zfs.trim.timeout                    30 >>         vfs.zfs.trim.txg_delay                  32 >>         vfs.zfs.trim.enabled                    1 >>         vfs.zfs.vol.immediate_write_sz          32768 >>         vfs.zfs.vol.unmap_sync_enabled          0 >>         vfs.zfs.vol.unmap_enabled               1 >>         vfs.zfs.vol.recursive                   0 >>         vfs.zfs.vol.mode                        1 >>         vfs.zfs.version.zpl                     5 >>         vfs.zfs.version.spa                     5000 >>         vfs.zfs.version.acl                     1 >>         vfs.zfs.version.ioctl                   7 >>         vfs.zfs.debug                           0 >>         vfs.zfs.super_owner                     0 >>         vfs.zfs.immediate_write_sz              32768 >>         vfs.zfs.sync_pass_rewrite               2 >>         vfs.zfs.sync_pass_dont_compress         5 >>         vfs.zfs.sync_pass_deferred_free         2 >>         vfs.zfs.zio.dva_throttle_enabled        1 >>         vfs.zfs.zio.exclude_metadata            0 >>         vfs.zfs.zio.use_uma                     1 >>         vfs.zfs.zio.taskq_batch_pct             75 >>         vfs.zfs.zil_maxblocksize                131072 >>         vfs.zfs.zil_slog_bulk                   786432 >>         vfs.zfs.zil_nocacheflush                0 >>         vfs.zfs.zil_replay_disable              0 >>         vfs.zfs.cache_flush_disable             0 >>         vfs.zfs.standard_sm_blksz               131072 >>         vfs.zfs.dtl_sm_blksz                    4096 >>         vfs.zfs.min_auto_ashift                 9 >>         vfs.zfs.max_auto_ashift                 13 >>         vfs.zfs.vdev.trim_max_pending           10000 >>         vfs.zfs.vdev.bio_delete_disable         0 >>         vfs.zfs.vdev.bio_flush_disable          0 >>         vfs.zfs.vdev.def_queue_depth            32 >>         vfs.zfs.vdev.queue_depth_pct            1000 >>         vfs.zfs.vdev.write_gap_limit            4096 >>         vfs.zfs.vdev.read_gap_limit             32768 >>         vfs.zfs.vdev.aggregation_limit_non_rotating131072 >>         vfs.zfs.vdev.aggregation_limit          1048576 >>         vfs.zfs.vdev.initializing_max_active    1 >>         vfs.zfs.vdev.initializing_min_active    1 >>         vfs.zfs.vdev.removal_max_active         2 >>         vfs.zfs.vdev.removal_min_active         1 >>         vfs.zfs.vdev.trim_max_active            64 >>         vfs.zfs.vdev.trim_min_active            1 >>         vfs.zfs.vdev.scrub_max_active           2 >>         vfs.zfs.vdev.scrub_min_active           1 >>         vfs.zfs.vdev.async_write_max_active     10 >>         vfs.zfs.vdev.async_write_min_active     1 >>         vfs.zfs.vdev.async_read_max_active      3 >>         vfs.zfs.vdev.async_read_min_active      1 >>         vfs.zfs.vdev.sync_write_max_active      10 >>         vfs.zfs.vdev.sync_write_min_active      10 >>         vfs.zfs.vdev.sync_read_max_active       10 >>         vfs.zfs.vdev.sync_read_min_active       10 >>         vfs.zfs.vdev.max_active                 1000 >>         vfs.zfs.vdev.async_write_active_max_dirty_percent60 >>         vfs.zfs.vdev.async_write_active_min_dirty_percent30 >>         vfs.zfs.vdev.mirror.non_rotating_seek_inc1 >>         vfs.zfs.vdev.mirror.non_rotating_inc    0 >>         vfs.zfs.vdev.mirror.rotating_seek_offset1048576 >>         vfs.zfs.vdev.mirror.rotating_seek_inc   5 >>         vfs.zfs.vdev.mirror.rotating_inc        0 >>         vfs.zfs.vdev.trim_on_init               1 >>         vfs.zfs.vdev.cache.bshift               16 >>         vfs.zfs.vdev.cache.size                 0 >>         vfs.zfs.vdev.cache.max                  16384 >>         vfs.zfs.vdev.validate_skip              0 >>         vfs.zfs.vdev.max_ms_shift               34 >>         vfs.zfs.vdev.default_ms_shift           29 >>         vfs.zfs.vdev.max_ms_count_limit         131072 >>         vfs.zfs.vdev.min_ms_count               16 >>         vfs.zfs.vdev.default_ms_count           200 >>         vfs.zfs.txg.timeout                     5 >>         vfs.zfs.space_map_ibs                   14 >>         vfs.zfs.special_class_metadata_reserve_pct25 >>         vfs.zfs.user_indirect_is_special        1 >>         vfs.zfs.ddt_data_is_special             1 >>         vfs.zfs.spa_allocators                  4 >>         vfs.zfs.spa_min_slop                    134217728 >>         vfs.zfs.spa_slop_shift                  5 >>         vfs.zfs.spa_asize_inflation             24 >>         vfs.zfs.deadman_enabled                 1 >>         vfs.zfs.deadman_checktime_ms            5000 >>         vfs.zfs.deadman_synctime_ms             1000000 >>         vfs.zfs.debugflags                      0 >>         vfs.zfs.recover                         0 >>         vfs.zfs.spa_load_verify_data            1 >>         vfs.zfs.spa_load_verify_metadata        1 >>         vfs.zfs.spa_load_verify_maxinflight     10000 >>         vfs.zfs.max_missing_tvds_scan           0 >>         vfs.zfs.max_missing_tvds_cachefile      2 >>         vfs.zfs.max_missing_tvds                0 >>         vfs.zfs.spa_load_print_vdev_tree        0 >>         vfs.zfs.ccw_retry_interval              300 >>         vfs.zfs.check_hostid                    1 >>         vfs.zfs.multihost_fail_intervals        10 >>         vfs.zfs.multihost_import_intervals      20 >>         vfs.zfs.multihost_interval              1000 >>         vfs.zfs.mg_fragmentation_threshold      85 >>         vfs.zfs.mg_noalloc_threshold            0 >>         vfs.zfs.condense_pct                    200 >>         vfs.zfs.metaslab_sm_blksz               4096 >>         vfs.zfs.metaslab.bias_enabled           1 >>         vfs.zfs.metaslab.lba_weighting_enabled  1 >>         vfs.zfs.metaslab.fragmentation_factor_enabled1 >>         vfs.zfs.metaslab.preload_enabled        1 >>         vfs.zfs.metaslab.preload_limit          3 >>         vfs.zfs.metaslab.unload_delay           8 >>         vfs.zfs.metaslab.load_pct               50 >>         vfs.zfs.metaslab.min_alloc_size         33554432 >>         vfs.zfs.metaslab.df_free_pct            4 >>         vfs.zfs.metaslab.df_alloc_threshold     131072 >>         vfs.zfs.metaslab.debug_unload           0 >>         vfs.zfs.metaslab.debug_load             0 >>         vfs.zfs.metaslab.fragmentation_threshold70 >>         vfs.zfs.metaslab.force_ganging          16777217 >>         vfs.zfs.free_bpobj_enabled              1 >>         vfs.zfs.free_max_blocks                 -1 >>         vfs.zfs.zfs_scan_checkpoint_interval    7200 >>         vfs.zfs.zfs_scan_legacy                 0 >>         vfs.zfs.no_scrub_prefetch               0 >>         vfs.zfs.no_scrub_io                     0 >>         vfs.zfs.resilver_min_time_ms            3000 >>         vfs.zfs.free_min_time_ms                1000 >>         vfs.zfs.scan_min_time_ms                1000 >>         vfs.zfs.scan_idle                       50 >>         vfs.zfs.scrub_delay                     4 >>         vfs.zfs.resilver_delay                  2 >>         vfs.zfs.zfetch.array_rd_sz              1048576 >>         vfs.zfs.zfetch.max_idistance            67108864 >>         vfs.zfs.zfetch.max_distance             8388608 >>         vfs.zfs.zfetch.min_sec_reap             2 >>         vfs.zfs.zfetch.max_streams              8 >>         vfs.zfs.prefetch_disable                0 >>         vfs.zfs.delay_scale                     500000 >>         vfs.zfs.delay_min_dirty_percent         60 >>         vfs.zfs.dirty_data_sync_pct             20 >>         vfs.zfs.dirty_data_max_percent          10 >>         vfs.zfs.dirty_data_max_max              4294967296 >>         vfs.zfs.dirty_data_max                  4294967296 >>         vfs.zfs.max_recordsize                  1048576 >>         vfs.zfs.default_ibs                     17 >>         vfs.zfs.default_bs                      9 >>         vfs.zfs.send_holes_without_birth_time   1 >>         vfs.zfs.mdcomp_disable                  0 >>         vfs.zfs.per_txg_dirty_frees_percent     5 >>         vfs.zfs.nopwrite_enabled                1 >>         vfs.zfs.dedup.prefetch                  1 >>         vfs.zfs.dbuf_cache_lowater_pct          10 >>         vfs.zfs.dbuf_cache_hiwater_pct          10 >>         vfs.zfs.dbuf_metadata_cache_overflow    0 >>         vfs.zfs.dbuf_metadata_cache_shift       6 >>         vfs.zfs.dbuf_cache_shift                5 >>         vfs.zfs.dbuf_metadata_cache_max_bytes   1025282816 >>         vfs.zfs.dbuf_cache_max_bytes            2050565632 >>         vfs.zfs.arc_min_prescient_prefetch_ms   6 >>         vfs.zfs.arc_min_prefetch_ms             1 >>         vfs.zfs.l2c_only_size                   0 >>         vfs.zfs.mfu_ghost_data_esize            7778263552 >>         vfs.zfs.mfu_ghost_metadata_esize        16851792896 >>         vfs.zfs.mfu_ghost_size                  24630056448 >>         vfs.zfs.mfu_data_esize                  3059418112 >>         vfs.zfs.mfu_metadata_esize              28641792 >>         vfs.zfs.mfu_size                        6399023104 >>         vfs.zfs.mru_ghost_data_esize            2199812096 >>         vfs.zfs.mru_ghost_metadata_esize        6289682432 >>         vfs.zfs.mru_ghost_size                  8489494528 >>         vfs.zfs.mru_data_esize                  22781456384 >>         vfs.zfs.mru_metadata_esize              309155840 >>         vfs.zfs.mru_size                        23847875584 >>         vfs.zfs.anon_data_esize                 0 >>         vfs.zfs.anon_metadata_esize             0 >>         vfs.zfs.anon_size                       8556544 >>         vfs.zfs.l2arc_norw                      1 >>         vfs.zfs.l2arc_feed_again                1 >>         vfs.zfs.l2arc_noprefetch                1 >>         vfs.zfs.l2arc_feed_min_ms               200 >>         vfs.zfs.l2arc_feed_secs                 1 >>         vfs.zfs.l2arc_headroom                  2 >>         vfs.zfs.l2arc_write_boost               8388608 >>         vfs.zfs.l2arc_write_max                 8388608 >>         vfs.zfs.arc_meta_strategy               1 >>         vfs.zfs.arc_meta_limit                  15833624576 >>         vfs.zfs.arc_free_target                 346902 >>         vfs.zfs.arc_kmem_cache_reap_retry_ms    1000 >>         vfs.zfs.compressed_arc_enabled          1 >>         vfs.zfs.arc_grow_retry                  60 >>         vfs.zfs.arc_shrink_shift                7 >>         vfs.zfs.arc_average_blocksize           8192 >>         vfs.zfs.arc_no_grow_shift               5 >>         vfs.zfs.arc_min                         8202262528 >>         vfs.zfs.arc_max                         39334498304 >>         vfs.zfs.abd_chunk_size                  4096 >>         vfs.zfs.abd_scatter_enabled             1 >> >> _______________________________________________ >> freebsd-stable@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-stable >> To unsubscribe, send any mail to >> "freebsd-stable-unsubscribe@freebsd.org" >> >> >> > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > From owner-freebsd-stable@freebsd.org Wed Jul 22 05:05:06 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D45D93712F5 for ; Wed, 22 Jul 2020 05:05:06 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670063.outbound.protection.outlook.com [40.107.67.63]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BBNgj6fzlz4fxN for ; Wed, 22 Jul 2020 05:05:05 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=VXZmwwi7E4N4I1RPi42KZp2PO6zUO/6XmKSw7/TtHBCapaXD3QCInqrbaH8Ssho6TY58BXAyMpXxjW2AMZqZpcudHB/rFZ83Eh8ePC/oE+ihrH5YFbh4bckFI/O6lzS6TmQyHbh0jkUV5xfHTGecwvoE7YHg0x09LbKzojCOfsAKl3+lJjpMXlYQxVU8yNm2/ST8QUQViRgv+TfS0dclvOdwsdxT6qdxuCtiohnwN5YwFU9ROZDZ4XDAvZn6PTVFiKW7DC69OZZ2INXgsRO+v8YlJIz+rc+rPL+q/xXAtq4/TqPGz0GNkDvNuamfYHO75MGUc9WFINp2/eB1QGRqGw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=SQ2y+RrkDDmKaj7gS26+JvD6Sk2uqVO9zMaSwX5R6Pk=; b=mFs8R72lrc35pAeQT+vVsp14D6Owg+rbvg8Ars45gp2m2juJ/IA3CYUK7W5diEyLIkQ9/1s8CeSUId/Nsz6VLlvxCsE1w6Dhj06T5O71u+hjcZodfl/wl86+M8IGz+4eW2wsfiZv9tsqflpYnPKFwtVFgNxfrRNNojp8/GpcorMtJtOeRxz0IlS4FiTDUfxQWa///cKzI6jmDeMuHcQWcX4j9hko6Ieqh1lBUsyzAPXq8Xv2fA4Dq43uD43Aa1P2dw/Kz6mjpvx4zKtKj39zC7e2+jAIOgooAfzPDFn7K2BCE3F89zKxT9kXaWhIS/XKv49NSTUnzuL9AfZQeo5iDQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=SQ2y+RrkDDmKaj7gS26+JvD6Sk2uqVO9zMaSwX5R6Pk=; b=Us8pInEDIZBLFx5SpPEgcEe8uU6/xpGoQ29snSHFplx6U8bH15KTWP0pgcCvOW2Y/Q4sypCbQUekL+HR1pnokQAcvLyyEx+erqpFHS0O6iDiAcojvIGyO5uZWU2eGI1mkmMWRXv1PTomE6opOymwWLLkx4pCxcbliEYF83BmAEBJWcH33Ql7Q4iHfjudmcyNMBC1mAbo9zb2HoGTF3pTJgqWe9aoFk39Efe83ws8O35QCmRKVxWnBayHRITg7xWsrYBZkUNAdHczeb13N23DyEL97Ecvg671K6q+AEdrKXrekNWIyKxGk+vI7lyFBx9PPU9PfX115/ssCQlwHxVm5g== Received: from QB1PR01MB3364.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:38::14) by QB1PR01MB3363.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:3d::32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3216.22; Wed, 22 Jul 2020 05:05:03 +0000 Received: from QB1PR01MB3364.CANPRD01.PROD.OUTLOOK.COM ([fe80::60f3:4ca2:8a4a:1e91]) by QB1PR01MB3364.CANPRD01.PROD.OUTLOOK.COM ([fe80::60f3:4ca2:8a4a:1e91%7]) with mapi id 15.20.3195.028; Wed, 22 Jul 2020 05:04:55 +0000 From: Rick Macklem To: mike tancsa , Ronald Klop , FreeBSD-STABLE Mailing List Subject: Re: zfs meta data slowness Thread-Topic: zfs meta data slowness Thread-Index: AQHWX5Zn3RfLm17QbU6StnM4JI9lQKkTCiOb Date: Wed, 22 Jul 2020 05:04:54 +0000 Message-ID: References: <1949194763.1.1595250243575@localhost>, <975657af-ccac-bbd1-e22b-86270c624226@sentex.net> In-Reply-To: <975657af-ccac-bbd1-e22b-86270c624226@sentex.net> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: d76e3997-abe7-4ead-a6f0-08d82dfcc60c x-ms-traffictypediagnostic: QB1PR01MB3363: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:4502; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: hD8ZX1vcEX2270TutNoExzoqMF+ixQaL3IuxCE3Z57wbzZkeST7WMCIXpzwVEahIaPasw66kmUkPwzDp8ZX0UxYjBciOaL/YZRPmwkdoostkEwSAu92QCQcM+x8i110t361bermObblI6YoEwLlLl5VbheFnt62d+s4VxSQIUuUtvOEzUEuuQ0vJPvcLol/fii4Pbw3W6fbUX0VgmRh4S9bOKw4/xJrfpNqrf9hL66e7SNzV+AIeLWZfkIuy6T1Z8VpumHJoSrYr1sKkK8bo0Wb0iyHPezFJvo5zHaUqK67pVbcf+s14L9V4NQNxpDFamF56VP21CU99gAUHZERHuQY4LBNok2a5RlNHqyVGPGJJOwfiCucXSb+CpN8u+lqBmIOl69Ppcuulg3P13wzQvQ== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:QB1PR01MB3364.CANPRD01.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFTY:; SFS:(376002)(136003)(39830400003)(346002)(366004)(396003)(66556008)(66446008)(66476007)(52536014)(64756008)(3480700007)(66946007)(55016002)(76116006)(91956017)(9686003)(186003)(5660300002)(30864003)(966005)(33656002)(110136005)(478600001)(86362001)(71200400001)(6506007)(83380400001)(2906002)(7696005)(8676002)(316002)(786003)(8936002); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: dnvMHzHE8o1grTpGPcYugAOHlPrBrr8K5TLMQ8E3ZTyvvDQI2mixrrQOY+kwViDEFnsqwXao/E/D+tNNahPW64CwldKqUV94e+uNTBBsZfwQBbjxibaVTN8XaZ2+lvZF5xLaUj280k2XG412uVM6fHIQwY4NkbUcfGr09HHMZZSXczMTNXBeS3kxJsk0YZjw29ciig55k6CLTgY8uaV8gI+JYG0Xf5YmvbPV/t3wHo4DOlfyp/1QVIpcBT02v+18yPuUXG7oNKGNmD2B4kz9ri+D6MyHYw2w71y67aOuZIOA9NLujUIGSHTdCxdLQuIy+7Gcf9LZT64j69Fb9PgMyCSqoVKdzsCIu15nRe39ZuySKYIx0tMvebFSGC7tS9ZrlqIQogIRWUR8xlK9+tgBOnmS1RtzmjdJwnP//e14RLGaU1L+CUKUe3s28vDxTvmlfu6iYTJzLzg+props6Vye68V/AjaE8yEepUZemG1EfqyOgkBgLw8rYwAaXPserCCt+8O8bVWol1Cq/p0TUhZrnhqMEV+0F8ByQzjX0lT7L9TOGGJjM94iwfTAH1OahG6 x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: QB1PR01MB3364.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: d76e3997-abe7-4ead-a6f0-08d82dfcc60c X-MS-Exchange-CrossTenant-originalarrivaltime: 22 Jul 2020 05:04:54.9642 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: TR6HVl0tae4op8qiuIUPbbG++U2go26YcT1dBA6h6DQIzakxNynDuMN6O1ysQK6bTrl6r+kNFCaYSFeApQvKMA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: QB1PR01MB3363 X-Rspamd-Queue-Id: 4BBNgj6fzlz4fxN X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=uoguelph.ca header.s=selector1 header.b=Us8pInED; dmarc=none; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.67.63 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-5.38 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-0.998]; R_DKIM_ALLOW(-0.20)[uoguelph.ca:s=selector1]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.67.63:from]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; NEURAL_HAM_LONG(-0.98)[-0.982]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[uoguelph.ca]; DWL_DNSWL_LOW(-1.00)[uoguelph.ca:dkim]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[uoguelph.ca:+]; NEURAL_HAM_SHORT(-0.80)[-0.801]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; RCVD_IN_DNSWL_LOW(-0.10)[40.107.67.63:from] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Jul 2020 05:05:06 -0000 mike tancsa wrote:=0A= >Hi,=0A= > Thanks for the response. Reply in line=0A= >=0A= >On 7/20/2020 9:04 AM, Ronald Klop wrote:=0A= >> Hi,=0A= >>=0A= >> My first suggestion would be to remove a lot of snapshots. But that my= =0A= >> not match your business case.=0A= >=0A= >As its a backup server, its sort of the point to have all those snapshots.= =0A= I'm the last guy who should be commenting on ZFS, since I never use it.=0A= However, it is my understanding that ZFS "pseudo automounts" each=0A= snapshot when you go there, so I think that might be what is taking=0A= so long (ie. not really meta data).=0A= =0A= Of course I have no idea what might speed that up. I would be=0A= tempted to look in ZFS for the "snapshot mounting code", in=0A= case I could find an obvious problem...=0A= =0A= rick=0A= =0A= =0A= > Maybe you can provide more information about your setup:=0A= > Amount of RAM, CPU?=0A= 64G, Xeon(R) CPU E3-1240 v6 @ 3.70GHz=0A= > output of "zpool status"=0A= # zpool status -x=0A= =0A= all pools are healthy=0A= =0A= =0A= > output of "zfs list" if possible to share=0A= =0A= its a big list=0A= =0A= # zfs list | wc=0A= 824 4120 107511=0A= =0A= =0A= > Type of disks/ssds?=0A= old school Device Model: WDC WD80EFAX-68KNBN0=0A= > What is the load of the system? I/O per second, etc.=0A= its not cpu bound, disks are sometimes running at 100% based on gstat,=0A= but not always=0A= > Do you use dedup, GELI?=0A= =0A= no and no=0A= =0A= =0A= > Something else special about the setup.=0A= > output of "top -b"=0A= >=0A= =0A= ports are right now being built in a VM, but the problem (zrepl hanging)=0A= and zfs list -t snapshots taking forever happens regardless=0A= =0A= PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU=0A= COMMAND=0A= 4439 root 12 40 20 6167M 5762M kqread 3 535:13 200.00% bhyv= e=0A= 98783 root 2 21 0 16M 5136K hdr->b 4 0:01 1.95% zfs= =0A= 76489 root 21 23 0 738M 54M uwait 1 2:18 0.88% zrep= l=0A= 98784 root 1 21 0 13M 3832K piperd 3 0:01 0.59% zfs= =0A= 99563 root 1 20 0 13M 4136K zio->i 4 0:00 0.39% zfs= =0A= 16136 root 18 25 0 705M 56M uwait 3 29:58 0.00%=0A= zrepl-freebsd-amd64=0A= 1845 root 1 20 0 12M 3772K nanslp 7 5:54 0.00%=0A= ossec-syscheckd=0A= 1567 root 1 20 0 11M 2744K select 0 2:22 0.00%=0A= syslogd=0A= 1737 root 32 20 0 11M 2844K rpcsvc 6 1:40 0.00% nfsd= =0A= 1660 root 1 -52 r0 11M 11M nanslp 5 1:18 0.00%=0A= watchdogd=0A= 1434 root 1 20 0 9988K 988K select 3 0:27 0.00% devd= =0A= 2435 mdtancsa 1 20 0 20M 8008K select 0 0:21 0.00% sshd= =0A= 1754 root 3 20 0 18M 3556K select 1 0:11 0.00%=0A= apcupsd=0A= 5917 root 1 20 0 11M 2672K select 2 0:06 0.00%=0A= script=0A= 1449 _pflogd 1 20 0 12M 3572K bpf 3 0:05 0.00%=0A= pflogd=0A= =0A= ---Mike=0A= =0A= > That kind of information.=0A= >=0A= > Regards,=0A= > Ronald.=0A= >=0A= >=0A= > Van: mike tancsa =0A= > Datum: zondag, 19 juli 2020 16:17=0A= > Aan: FreeBSD-STABLE Mailing List =0A= > Onderwerp: zfs meta data slowness=0A= >>=0A= >> Are there any tweaks that can be done to speed up or improve zfs=0A= >> metadata performance ? I have a backup server with a lot of snapshots=0A= >> (40,000) and just doing a listing can take a great deal of time. Best= =0A= >> case scenario is about 24 seconds, worst case, I have seen it up to 15= =0A= >> minutes. (FreeBSD 12.1-STABLE r363078)=0A= >>=0A= >>=0A= >> ARC Efficiency: 79.33b=0A= >> Cache Hit Ratio: 92.81% 73.62b=0A= >> Cache Miss Ratio: 7.19% 5.71b=0A= >> Actual Hit Ratio: 92.78% 73.60b=0A= >>=0A= >> Data Demand Efficiency: 96.47% 461.91m=0A= >> Data Prefetch Efficiency: 1.00% 262.73m=0A= >>=0A= >> CACHE HITS BY CACHE LIST:=0A= >> Anonymously Used: 0.01% 3.86m=0A= >> Most Recently Used: 3.91% 2.88b=0A= >> Most Frequently Used: 96.06% 70.72b=0A= >> Most Recently Used Ghost: 0.01% 5.31m=0A= >> Most Frequently Used Ghost: 0.01% 10.47m=0A= >>=0A= >> CACHE HITS BY DATA TYPE:=0A= >> Demand Data: 0.61% 445.60m=0A= >> Prefetch Data: 0.00% 2.63m=0A= >> Demand Metadata: 99.36% 73.15b=0A= >> Prefetch Metadata: 0.03% 21.00m=0A= >>=0A= >> CACHE MISSES BY DATA TYPE:=0A= >> Demand Data: 0.29% 16.31m=0A= >> Prefetch Data: 4.56% 260.10m=0A= >> Demand Metadata: 95.02% 5.42b=0A= >> Prefetch Metadata: 0.14% 7.75m=0A= >>=0A= >>=0A= >> Other than increase the metadata max, I havent really changed any=0A= >> tuneables=0A= >>=0A= >>=0A= >> ZFS Tunables (sysctl):=0A= >> kern.maxusers 4416=0A= >> vm.kmem_size 66691842048=0A= >> vm.kmem_size_scale 1=0A= >> vm.kmem_size_min 0=0A= >> vm.kmem_size_max 1319413950874=0A= >> vfs.zfs.trim.max_interval 1=0A= >> vfs.zfs.trim.timeout 30=0A= >> vfs.zfs.trim.txg_delay 32=0A= >> vfs.zfs.trim.enabled 1=0A= >> vfs.zfs.vol.immediate_write_sz 32768=0A= >> vfs.zfs.vol.unmap_sync_enabled 0=0A= >> vfs.zfs.vol.unmap_enabled 1=0A= >> vfs.zfs.vol.recursive 0=0A= >> vfs.zfs.vol.mode 1=0A= >> vfs.zfs.version.zpl 5=0A= >> vfs.zfs.version.spa 5000=0A= >> vfs.zfs.version.acl 1=0A= >> vfs.zfs.version.ioctl 7=0A= >> vfs.zfs.debug 0=0A= >> vfs.zfs.super_owner 0=0A= >> vfs.zfs.immediate_write_sz 32768=0A= >> vfs.zfs.sync_pass_rewrite 2=0A= >> vfs.zfs.sync_pass_dont_compress 5=0A= >> vfs.zfs.sync_pass_deferred_free 2=0A= >> vfs.zfs.zio.dva_throttle_enabled 1=0A= >> vfs.zfs.zio.exclude_metadata 0=0A= >> vfs.zfs.zio.use_uma 1=0A= >> vfs.zfs.zio.taskq_batch_pct 75=0A= >> vfs.zfs.zil_maxblocksize 131072=0A= >> vfs.zfs.zil_slog_bulk 786432=0A= >> vfs.zfs.zil_nocacheflush 0=0A= >> vfs.zfs.zil_replay_disable 0=0A= >> vfs.zfs.cache_flush_disable 0=0A= >> vfs.zfs.standard_sm_blksz 131072=0A= >> vfs.zfs.dtl_sm_blksz 4096=0A= >> vfs.zfs.min_auto_ashift 9=0A= >> vfs.zfs.max_auto_ashift 13=0A= >> vfs.zfs.vdev.trim_max_pending 10000=0A= >> vfs.zfs.vdev.bio_delete_disable 0=0A= >> vfs.zfs.vdev.bio_flush_disable 0=0A= >> vfs.zfs.vdev.def_queue_depth 32=0A= >> vfs.zfs.vdev.queue_depth_pct 1000=0A= >> vfs.zfs.vdev.write_gap_limit 4096=0A= >> vfs.zfs.vdev.read_gap_limit 32768=0A= >> vfs.zfs.vdev.aggregation_limit_non_rotating131072=0A= >> vfs.zfs.vdev.aggregation_limit 1048576=0A= >> vfs.zfs.vdev.initializing_max_active 1=0A= >> vfs.zfs.vdev.initializing_min_active 1=0A= >> vfs.zfs.vdev.removal_max_active 2=0A= >> vfs.zfs.vdev.removal_min_active 1=0A= >> vfs.zfs.vdev.trim_max_active 64=0A= >> vfs.zfs.vdev.trim_min_active 1=0A= >> vfs.zfs.vdev.scrub_max_active 2=0A= >> vfs.zfs.vdev.scrub_min_active 1=0A= >> vfs.zfs.vdev.async_write_max_active 10=0A= >> vfs.zfs.vdev.async_write_min_active 1=0A= >> vfs.zfs.vdev.async_read_max_active 3=0A= >> vfs.zfs.vdev.async_read_min_active 1=0A= >> vfs.zfs.vdev.sync_write_max_active 10=0A= >> vfs.zfs.vdev.sync_write_min_active 10=0A= >> vfs.zfs.vdev.sync_read_max_active 10=0A= >> vfs.zfs.vdev.sync_read_min_active 10=0A= >> vfs.zfs.vdev.max_active 1000=0A= >> vfs.zfs.vdev.async_write_active_max_dirty_percent60=0A= >> vfs.zfs.vdev.async_write_active_min_dirty_percent30=0A= >> vfs.zfs.vdev.mirror.non_rotating_seek_inc1=0A= >> vfs.zfs.vdev.mirror.non_rotating_inc 0=0A= >> vfs.zfs.vdev.mirror.rotating_seek_offset1048576=0A= >> vfs.zfs.vdev.mirror.rotating_seek_inc 5=0A= >> vfs.zfs.vdev.mirror.rotating_inc 0=0A= >> vfs.zfs.vdev.trim_on_init 1=0A= >> vfs.zfs.vdev.cache.bshift 16=0A= >> vfs.zfs.vdev.cache.size 0=0A= >> vfs.zfs.vdev.cache.max 16384=0A= >> vfs.zfs.vdev.validate_skip 0=0A= >> vfs.zfs.vdev.max_ms_shift 34=0A= >> vfs.zfs.vdev.default_ms_shift 29=0A= >> vfs.zfs.vdev.max_ms_count_limit 131072=0A= >> vfs.zfs.vdev.min_ms_count 16=0A= >> vfs.zfs.vdev.default_ms_count 200=0A= >> vfs.zfs.txg.timeout 5=0A= >> vfs.zfs.space_map_ibs 14=0A= >> vfs.zfs.special_class_metadata_reserve_pct25=0A= >> vfs.zfs.user_indirect_is_special 1=0A= >> vfs.zfs.ddt_data_is_special 1=0A= >> vfs.zfs.spa_allocators 4=0A= >> vfs.zfs.spa_min_slop 134217728=0A= >> vfs.zfs.spa_slop_shift 5=0A= >> vfs.zfs.spa_asize_inflation 24=0A= >> vfs.zfs.deadman_enabled 1=0A= >> vfs.zfs.deadman_checktime_ms 5000=0A= >> vfs.zfs.deadman_synctime_ms 1000000=0A= >> vfs.zfs.debugflags 0=0A= >> vfs.zfs.recover 0=0A= >> vfs.zfs.spa_load_verify_data 1=0A= >> vfs.zfs.spa_load_verify_metadata 1=0A= >> vfs.zfs.spa_load_verify_maxinflight 10000=0A= >> vfs.zfs.max_missing_tvds_scan 0=0A= >> vfs.zfs.max_missing_tvds_cachefile 2=0A= >> vfs.zfs.max_missing_tvds 0=0A= >> vfs.zfs.spa_load_print_vdev_tree 0=0A= >> vfs.zfs.ccw_retry_interval 300=0A= >> vfs.zfs.check_hostid 1=0A= >> vfs.zfs.multihost_fail_intervals 10=0A= >> vfs.zfs.multihost_import_intervals 20=0A= >> vfs.zfs.multihost_interval 1000=0A= >> vfs.zfs.mg_fragmentation_threshold 85=0A= >> vfs.zfs.mg_noalloc_threshold 0=0A= >> vfs.zfs.condense_pct 200=0A= >> vfs.zfs.metaslab_sm_blksz 4096=0A= >> vfs.zfs.metaslab.bias_enabled 1=0A= >> vfs.zfs.metaslab.lba_weighting_enabled 1=0A= >> vfs.zfs.metaslab.fragmentation_factor_enabled1=0A= >> vfs.zfs.metaslab.preload_enabled 1=0A= >> vfs.zfs.metaslab.preload_limit 3=0A= >> vfs.zfs.metaslab.unload_delay 8=0A= >> vfs.zfs.metaslab.load_pct 50=0A= >> vfs.zfs.metaslab.min_alloc_size 33554432=0A= >> vfs.zfs.metaslab.df_free_pct 4=0A= >> vfs.zfs.metaslab.df_alloc_threshold 131072=0A= >> vfs.zfs.metaslab.debug_unload 0=0A= >> vfs.zfs.metaslab.debug_load 0=0A= >> vfs.zfs.metaslab.fragmentation_threshold70=0A= >> vfs.zfs.metaslab.force_ganging 16777217=0A= >> vfs.zfs.free_bpobj_enabled 1=0A= >> vfs.zfs.free_max_blocks -1=0A= >> vfs.zfs.zfs_scan_checkpoint_interval 7200=0A= >> vfs.zfs.zfs_scan_legacy 0=0A= >> vfs.zfs.no_scrub_prefetch 0=0A= >> vfs.zfs.no_scrub_io 0=0A= >> vfs.zfs.resilver_min_time_ms 3000=0A= >> vfs.zfs.free_min_time_ms 1000=0A= >> vfs.zfs.scan_min_time_ms 1000=0A= >> vfs.zfs.scan_idle 50=0A= >> vfs.zfs.scrub_delay 4=0A= >> vfs.zfs.resilver_delay 2=0A= >> vfs.zfs.zfetch.array_rd_sz 1048576=0A= >> vfs.zfs.zfetch.max_idistance 67108864=0A= >> vfs.zfs.zfetch.max_distance 8388608=0A= >> vfs.zfs.zfetch.min_sec_reap 2=0A= >> vfs.zfs.zfetch.max_streams 8=0A= >> vfs.zfs.prefetch_disable 0=0A= >> vfs.zfs.delay_scale 500000=0A= >> vfs.zfs.delay_min_dirty_percent 60=0A= >> vfs.zfs.dirty_data_sync_pct 20=0A= >> vfs.zfs.dirty_data_max_percent 10=0A= >> vfs.zfs.dirty_data_max_max 4294967296=0A= >> vfs.zfs.dirty_data_max 4294967296=0A= >> vfs.zfs.max_recordsize 1048576=0A= >> vfs.zfs.default_ibs 17=0A= >> vfs.zfs.default_bs 9=0A= >> vfs.zfs.send_holes_without_birth_time 1=0A= >> vfs.zfs.mdcomp_disable 0=0A= >> vfs.zfs.per_txg_dirty_frees_percent 5=0A= >> vfs.zfs.nopwrite_enabled 1=0A= >> vfs.zfs.dedup.prefetch 1=0A= >> vfs.zfs.dbuf_cache_lowater_pct 10=0A= >> vfs.zfs.dbuf_cache_hiwater_pct 10=0A= >> vfs.zfs.dbuf_metadata_cache_overflow 0=0A= >> vfs.zfs.dbuf_metadata_cache_shift 6=0A= >> vfs.zfs.dbuf_cache_shift 5=0A= >> vfs.zfs.dbuf_metadata_cache_max_bytes 1025282816=0A= >> vfs.zfs.dbuf_cache_max_bytes 2050565632=0A= >> vfs.zfs.arc_min_prescient_prefetch_ms 6=0A= >> vfs.zfs.arc_min_prefetch_ms 1=0A= >> vfs.zfs.l2c_only_size 0=0A= >> vfs.zfs.mfu_ghost_data_esize 7778263552=0A= >> vfs.zfs.mfu_ghost_metadata_esize 16851792896=0A= >> vfs.zfs.mfu_ghost_size 24630056448=0A= >> vfs.zfs.mfu_data_esize 3059418112=0A= >> vfs.zfs.mfu_metadata_esize 28641792=0A= >> vfs.zfs.mfu_size 6399023104=0A= >> vfs.zfs.mru_ghost_data_esize 2199812096=0A= >> vfs.zfs.mru_ghost_metadata_esize 6289682432=0A= >> vfs.zfs.mru_ghost_size 8489494528=0A= >> vfs.zfs.mru_data_esize 22781456384=0A= >> vfs.zfs.mru_metadata_esize 309155840=0A= >> vfs.zfs.mru_size 23847875584=0A= >> vfs.zfs.anon_data_esize 0=0A= >> vfs.zfs.anon_metadata_esize 0=0A= >> vfs.zfs.anon_size 8556544=0A= >> vfs.zfs.l2arc_norw 1=0A= >> vfs.zfs.l2arc_feed_again 1=0A= >> vfs.zfs.l2arc_noprefetch 1=0A= >> vfs.zfs.l2arc_feed_min_ms 200=0A= >> vfs.zfs.l2arc_feed_secs 1=0A= >> vfs.zfs.l2arc_headroom 2=0A= >> vfs.zfs.l2arc_write_boost 8388608=0A= >> vfs.zfs.l2arc_write_max 8388608=0A= >> vfs.zfs.arc_meta_strategy 1=0A= >> vfs.zfs.arc_meta_limit 15833624576=0A= >> vfs.zfs.arc_free_target 346902=0A= >> vfs.zfs.arc_kmem_cache_reap_retry_ms 1000=0A= >> vfs.zfs.compressed_arc_enabled 1=0A= >> vfs.zfs.arc_grow_retry 60=0A= >> vfs.zfs.arc_shrink_shift 7=0A= >> vfs.zfs.arc_average_blocksize 8192=0A= >> vfs.zfs.arc_no_grow_shift 5=0A= >> vfs.zfs.arc_min 8202262528=0A= >> vfs.zfs.arc_max 39334498304=0A= >> vfs.zfs.abd_chunk_size 4096=0A= >> vfs.zfs.abd_scatter_enabled 1=0A= >>=0A= >> _______________________________________________=0A= >> freebsd-stable@freebsd.org mailing list=0A= >> https://lists.freebsd.org/mailman/listinfo/freebsd-stable=0A= >> To unsubscribe, send any mail to=0A= >> "freebsd-stable-unsubscribe@freebsd.org"=0A= >>=0A= >>=0A= >>=0A= > _______________________________________________=0A= > freebsd-stable@freebsd.org mailing list=0A= > https://lists.freebsd.org/mailman/listinfo/freebsd-stable=0A= > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"= =0A= >=0A= _______________________________________________=0A= freebsd-stable@freebsd.org mailing list=0A= https://lists.freebsd.org/mailman/listinfo/freebsd-stable=0A= To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"= =0A= From owner-freebsd-stable@freebsd.org Wed Jul 22 05:29:37 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id F40D2372278 for ; Wed, 22 Jul 2020 05:29:36 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [IPv6:2a01:4f8:c2c:26d8::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BBPCz5Hphz3RwG for ; Wed, 22 Jul 2020 05:29:35 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13:0:0:0:5]) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id 06M5TAie026213 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 22 Jul 2020 05:29:12 GMT (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: mike@sentex.net Received: from [10.58.0.10] (dadvw [10.58.0.10]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id 06M5TCV7049576 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Wed, 22 Jul 2020 12:29:13 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: zfs meta data slowness To: mike tancsa , Ronald Klop , FreeBSD-STABLE Mailing List References: <1949194763.1.1595250243575@localhost> <975657af-ccac-bbd1-e22b-86270c624226@sentex.net> From: Eugene Grosbein Message-ID: <9ddce60a-9d91-fa69-0e11-54342cb67890@grosbein.net> Date: Wed, 22 Jul 2020 12:29:04 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <975657af-ccac-bbd1-e22b-86270c624226@sentex.net> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=0.3 required=5.0 tests=BAYES_00,LOCAL_FROM, NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.2 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record * -0.0 SPF_PASS SPF: sender matches SPF record * 2.6 LOCAL_FROM From my domains * -0.0 NICE_REPLY_A Looks like a legit reply (A) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on hz.grosbein.net X-Rspamd-Queue-Id: 4BBPCz5Hphz3RwG X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=permerror (mx1.freebsd.org: domain of eugen@grosbein.net uses mechanism not recognized by this client) smtp.mailfrom=eugen@grosbein.net X-Spamd-Result: default: False [-1.24 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.78)[-0.778]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_SPAM_SHORT(0.50)[0.496]; NEURAL_HAM_LONG(-0.85)[-0.855]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[grosbein.net]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; R_SPF_PERMFAIL(0.00)[empty SPF record]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Jul 2020 05:29:37 -0000 22.07.2020 2:37, mike tancsa wrote: >> Something else special about the setup. >> output of "top -b" >> > > ports are right now being built in a VM, but the problem (zrepl hanging) > and zfs list -t snapshots taking forever happens regardless > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU > COMMAND > 4439 root 12 40 20 6167M 5762M kqread 3 535:13 200.00% bhyve > 98783 root 2 21 0 16M 5136K hdr->b 4 0:01 1.95% zfs > 76489 root 21 23 0 738M 54M uwait 1 2:18 0.88% zrepl > 98784 root 1 21 0 13M 3832K piperd 3 0:01 0.59% zfs > 99563 root 1 20 0 13M 4136K zio->i 4 0:00 0.39% zfs You need top -SHPI From owner-freebsd-stable@freebsd.org Wed Jul 22 11:24:34 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1E3E3379922 for ; Wed, 22 Jul 2020 11:24:34 +0000 (UTC) (envelope-from ronald-lists@klop.ws) Received: from smtp-relay-int.realworks.nl (smtp-relay-int.realworks.nl [194.109.157.24]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4BBY5X5SS6z43PM for ; Wed, 22 Jul 2020 11:24:32 +0000 (UTC) (envelope-from ronald-lists@klop.ws) Date: Wed, 22 Jul 2020 13:24:28 +0200 (CEST) From: Ronald Klop To: FreeBSD-STABLE Mailing List Message-ID: <1473979028.2.1595417068381@localhost> In-Reply-To: <975657af-ccac-bbd1-e22b-86270c624226@sentex.net> References: <1949194763.1.1595250243575@localhost> <975657af-ccac-bbd1-e22b-86270c624226@sentex.net> Subject: Re: zfs meta data slowness MIME-Version: 1.0 X-Mailer: Realworks (518.363.e9e4e461e05) Importance: Normal X-Priority: 3 (Normal) X-Rspamd-Queue-Id: 4BBY5X5SS6z43PM X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of ronald-lists@klop.ws designates 194.109.157.24 as permitted sender) smtp.mailfrom=ronald-lists@klop.ws X-Spamd-Result: default: False [-0.70 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.53)[-0.529]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:194.109.157.0/24]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; DMARC_NA(0.00)[klop.ws]; NEURAL_HAM_LONG(-0.84)[-0.843]; RCPT_COUNT_ONE(0.00)[1]; TO_DN_ALL(0.00)[]; HAS_X_PRIO_THREE(0.00)[3]; RCVD_IN_DNSWL_NONE(0.00)[194.109.157.24:from]; NEURAL_SPAM_SHORT(0.47)[0.468]; RCVD_COUNT_ZERO(0.00)[0]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:3265, ipnet:194.109.0.0/16, country:NL]; MID_RHS_NOT_FQDN(0.50)[]; RWL_MAILSPIKE_VERYGOOD(0.00)[194.109.157.24:from] Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.33 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Jul 2020 11:24:34 -0000 Van: mike tancsa Datum: dinsdag, 21 juli 2020 21:37 Aan: Ronald Klop , FreeBSD-STABLE Mailing List Onderwerp: Re: zfs meta data slowness > > Hi, > Thanks for the response. Reply in line > > On 7/20/2020 9:04 AM, Ronald Klop wrote: > > Hi, > > > > My first suggestion would be to remove a lot of snapshots. But that my > > not match your business case. > > As its a backup server, its sort of the point to have all those snapshots. > > > > Maybe you can provide more information about your setup: > > Amount of RAM, CPU? > 64G, Xeon(R) CPU E3-1240 v6 @ 3.70GHz > > output of "zpool status" > # zpool status -x > > all pools are healthy > That is nice to know. Instead of zpool status -x, the output of "zpool status" is very interesting. And "zpool list" also. That gives the reader information about your setup, which helps thinking along about the possible cause. But as somebody else mentioned. Profiling the kernel might be the best thing to do. Dtrace can be used for it. But I don't know these commands by heart. If I remember correctly there is an optimization for "zfs list -o name". This is much faster because it does not get extra information from the disks. See: https://svnweb.freebsd.org/base?view=revision&revision=230438 Regards, Ronald. > > > output of "zfs list" if possible to share > > its a big list > > # zfs list | wc > 824 4120 107511 > > > > Type of disks/ssds? > old school Device Model: WDC WD80EFAX-68KNBN0 > > What is the load of the system? I/O per second, etc. > its not cpu bound, disks are sometimes running at 100% based on gstat, > but not always > > Do you use dedup, GELI? > > no and no > > > > Something else special about the setup. > > output of "top -b" > > > > ports are right now being built in a VM, but the problem (zrepl hanging) > and zfs list -t snapshots taking forever happens regardless > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU > COMMAND > 4439 root 12 40 20 6167M 5762M kqread 3 535:13 200.00% bhyve > 98783 root 2 21 0 16M 5136K hdr->b 4 0:01 1.95% zfs > 76489 root 21 23 0 738M 54M uwait 1 2:18 0.88% zrepl > 98784 root 1 21 0 13M 3832K piperd 3 0:01 0.59% zfs > 99563 root 1 20 0 13M 4136K zio->i 4 0:00 0.39% zfs > 16136 root 18 25 0 705M 56M uwait 3 29:58 0.00% > zrepl-freebsd-amd64 > 1845 root 1 20 0 12M 3772K nanslp 7 5:54 0.00% > ossec-syscheckd > 1567 root 1 20 0 11M 2744K select 0 2:22 0.00% > syslogd > 1737 root 32 20 0 11M 2844K rpcsvc 6 1:40 0.00% nfsd > 1660 root 1 -52 r0 11M 11M nanslp 5 1:18 0.00% > watchdogd > 1434 root 1 20 0 9988K 988K select 3 0:27 0.00% devd > 2435 mdtancsa 1 20 0 20M 8008K select 0 0:21 0.00% sshd > 1754 root 3 20 0 18M 3556K select 1 0:11 0.00% > apcupsd > 5917 root 1 20 0 11M 2672K select 2 0:06 0.00% > script > 1449 _pflogd 1 20 0 12M 3572K bpf 3 0:05 0.00% > pflogd > > ---Mike > > > That kind of information. > > > > Regards, > > Ronald. > > > > > > Van: mike tancsa > > Datum: zondag, 19 juli 2020 16:17 > > Aan: FreeBSD-STABLE Mailing List > > Onderwerp: zfs meta data slowness > >> > >> Are there any tweaks that can be done to speed up or improve zfs > >> metadata performance ? I have a backup server with a lot of snapshots > >> (40,000) and just doing a listing can take a great deal of time. Best > >> case scenario is about 24 seconds, worst case, I have seen it up to 15 > >> minutes. (FreeBSD 12.1-STABLE r363078) > >> > >> > >> ARC Efficiency: 79.33b > >> Cache Hit Ratio: 92.81% 73.62b > >> Cache Miss Ratio: 7.19% 5.71b > >> Actual Hit Ratio: 92.78% 73.60b > >> > >> Data Demand Efficiency: 96.47% 461.91m > >> Data Prefetch Efficiency: 1.00% 262.73m > >> > >> CACHE HITS BY CACHE LIST: > >> Anonymously Used: 0.01% 3.86m > >> Most Recently Used: 3.91% 2.88b > >> Most Frequently Used: 96.06% 70.72b > >> Most Recently Used Ghost: 0.01% 5.31m > >> Most Frequently Used Ghost: 0.01% 10.47m > >> > >> CACHE HITS BY DATA TYPE: > >> Demand Data: 0.61% 445.60m > >> Prefetch Data: 0.00% 2.63m > >> Demand Metadata: 99.36% 73.15b > >> Prefetch Metadata: 0.03% 21.00m > >> > >> CACHE MISSES BY DATA TYPE: > >> Demand Data: 0.29% 16.31m > >> Prefetch Data: 4.56% 260.10m > >> Demand Metadata: 95.02% 5.42b > >> Prefetch Metadata: 0.14% 7.75m > >> > >> > >> Other than increase the metadata max, I havent really changed any > >> tuneables > >> > >> > >> ZFS Tunables (sysctl): > >> kern.maxusers 4416 > >> vm.kmem_size 66691842048 > >> vm.kmem_size_scale 1 > >> vm.kmem_size_min 0 > >> vm.kmem_size_max 1319413950874 > >> vfs.zfs.trim.max_interval 1 > >> vfs.zfs.trim.timeout 30 > >> vfs.zfs.trim.txg_delay 32 > >> vfs.zfs.trim.enabled 1 > >> vfs.zfs.vol.immediate_write_sz 32768 > >> vfs.zfs.vol.unmap_sync_enabled 0 > >> vfs.zfs.vol.unmap_enabled 1 > >> vfs.zfs.vol.recursive 0 > >> vfs.zfs.vol.mode 1 > >> vfs.zfs.version.zpl 5 > >> vfs.zfs.version.spa 5000 > >> vfs.zfs.version.acl 1 > >> vfs.zfs.version.ioctl 7 > >> vfs.zfs.debug 0 > >> vfs.zfs.super_owner 0 > >> vfs.zfs.immediate_write_sz 32768 > >> vfs.zfs.sync_pass_rewrite 2 > >> vfs.zfs.sync_pass_dont_compress 5 > >> vfs.zfs.sync_pass_deferred_free 2 > >> vfs.zfs.zio.dva_throttle_enabled 1 > >> vfs.zfs.zio.exclude_metadata 0 > >> vfs.zfs.zio.use_uma 1 > >> vfs.zfs.zio.taskq_batch_pct 75 > >> vfs.zfs.zil_maxblocksize 131072 > >> vfs.zfs.zil_slog_bulk 786432 > >> vfs.zfs.zil_nocacheflush 0 > >> vfs.zfs.zil_replay_disable 0 > >> vfs.zfs.cache_flush_disable 0 > >> vfs.zfs.standard_sm_blksz 131072 > >> vfs.zfs.dtl_sm_blksz 4096 > >> vfs.zfs.min_auto_ashift 9 > >> vfs.zfs.max_auto_ashift 13 > >> vfs.zfs.vdev.trim_max_pending 10000 > >> vfs.zfs.vdev.bio_delete_disable 0 > >> vfs.zfs.vdev.bio_flush_disable 0 > >> vfs.zfs.vdev.def_queue_depth 32 > >> vfs.zfs.vdev.queue_depth_pct 1000 > >> vfs.zfs.vdev.write_gap_limit 4096 > >> vfs.zfs.vdev.read_gap_limit 32768 > >> vfs.zfs.vdev.aggregation_limit_non_rotating131072 > >> vfs.zfs.vdev.aggregation_limit 1048576 > >> vfs.zfs.vdev.initializing_max_active 1 > >> vfs.zfs.vdev.initializing_min_active 1 > >> vfs.zfs.vdev.removal_max_active 2 > >> vfs.zfs.vdev.removal_min_active 1 > >> vfs.zfs.vdev.trim_max_active 64 > >> vfs.zfs.vdev.trim_min_active 1 > >> vfs.zfs.vdev.scrub_max_active 2 > >> vfs.zfs.vdev.scrub_min_active 1 > >> vfs.zfs.vdev.async_write_max_active 10 > >> vfs.zfs.vdev.async_write_min_active 1 > >> vfs.zfs.vdev.async_read_max_active 3 > >> vfs.zfs.vdev.async_read_min_active 1 > >> vfs.zfs.vdev.sync_write_max_active 10 > >> vfs.zfs.vdev.sync_write_min_active 10 > >> vfs.zfs.vdev.sync_read_max_active 10 > >> vfs.zfs.vdev.sync_read_min_active 10 > >> vfs.zfs.vdev.max_active 1000 > >> vfs.zfs.vdev.async_write_active_max_dirty_percent60 > >> vfs.zfs.vdev.async_write_active_min_dirty_percent30 > >> vfs.zfs.vdev.mirror.non_rotating_seek_inc1 > >> vfs.zfs.vdev.mirror.non_rotating_inc 0 > >> vfs.zfs.vdev.mirror.rotating_seek_offset1048576 > >> vfs.zfs.vdev.mirror.rotating_seek_inc 5 > >> vfs.zfs.vdev.mirror.rotating_inc 0 > >> vfs.zfs.vdev.trim_on_init 1 > >> vfs.zfs.vdev.cache.bshift 16 > >> vfs.zfs.vdev.cache.size 0 > >> vfs.zfs.vdev.cache.max 16384 > >> vfs.zfs.vdev.validate_skip 0 > >> vfs.zfs.vdev.max_ms_shift 34 > >> vfs.zfs.vdev.default_ms_shift 29 > >> vfs.zfs.vdev.max_ms_count_limit 131072 > >> vfs.zfs.vdev.min_ms_count 16 > >> vfs.zfs.vdev.default_ms_count 200 > >> vfs.zfs.txg.timeout 5 > >> vfs.zfs.space_map_ibs 14 > >> vfs.zfs.special_class_metadata_reserve_pct25 > >> vfs.zfs.user_indirect_is_special 1 > >> vfs.zfs.ddt_data_is_special 1 > >> vfs.zfs.spa_allocators 4 > >> vfs.zfs.spa_min_slop 134217728 > >> vfs.zfs.spa_slop_shift 5 > >> vfs.zfs.spa_asize_inflation 24 > >> vfs.zfs.deadman_enabled 1 > >> vfs.zfs.deadman_checktime_ms 5000 > >> vfs.zfs.deadman_synctime_ms 1000000 > >> vfs.zfs.debugflags 0 > >> vfs.zfs.recover 0 > >> vfs.zfs.spa_load_verify_data 1 > >> vfs.zfs.spa_load_verify_metadata 1 > >> vfs.zfs.spa_load_verify_maxinflight 10000 > >> vfs.zfs.max_missing_tvds_scan 0 > >> vfs.zfs.max_missing_tvds_cachefile 2 > >> vfs.zfs.max_missing_tvds 0 > >> vfs.zfs.spa_load_print_vdev_tree 0 > >> vfs.zfs.ccw_retry_interval 300 > >> vfs.zfs.check_hostid 1 > >> vfs.zfs.multihost_fail_intervals 10 > >> vfs.zfs.multihost_import_intervals 20 > >> vfs.zfs.multihost_interval 1000 > >> vfs.zfs.mg_fragmentation_threshold 85 > >> vfs.zfs.mg_noalloc_threshold 0 > >> vfs.zfs.condense_pct 200 > >> vfs.zfs.metaslab_sm_blksz 4096 > >> vfs.zfs.metaslab.bias_enabled 1 > >> vfs.zfs.metaslab.lba_weighting_enabled 1 > >> vfs.zfs.metaslab.fragmentation_factor_enabled1 > >> vfs.zfs.metaslab.preload_enabled 1 > >> vfs.zfs.metaslab.preload_limit 3 > >> vfs.zfs.metaslab.unload_delay 8 > >> vfs.zfs.metaslab.load_pct 50 > >> vfs.zfs.metaslab.min_alloc_size 33554432 > >> vfs.zfs.metaslab.df_free_pct 4 > >> vfs.zfs.metaslab.df_alloc_threshold 131072 > >> vfs.zfs.metaslab.debug_unload 0 > >> vfs.zfs.metaslab.debug_load 0 > >> vfs.zfs.metaslab.fragmentation_threshold70 > >> vfs.zfs.metaslab.force_ganging 16777217 > >> vfs.zfs.free_bpobj_enabled 1 > >> vfs.zfs.free_max_blocks -1 > >> vfs.zfs.zfs_scan_checkpoint_interval 7200 > >> vfs.zfs.zfs_scan_legacy 0 > >> vfs.zfs.no_scrub_prefetch 0 > >> vfs.zfs.no_scrub_io 0 > >> vfs.zfs.resilver_min_time_ms 3000 > >> vfs.zfs.free_min_time_ms 1000 > >> vfs.zfs.scan_min_time_ms 1000 > >> vfs.zfs.scan_idle 50 > >> vfs.zfs.scrub_delay 4 > >> vfs.zfs.resilver_delay 2 > >> vfs.zfs.zfetch.array_rd_sz 1048576 > >> vfs.zfs.zfetch.max_idistance 67108864 > >> vfs.zfs.zfetch.max_distance 8388608 > >> vfs.zfs.zfetch.min_sec_reap 2 > >> vfs.zfs.zfetch.max_streams 8 > >> vfs.zfs.prefetch_disable 0 > >> vfs.zfs.delay_scale 500000 > >> vfs.zfs.delay_min_dirty_percent 60 > >> vfs.zfs.dirty_data_sync_pct 20 > >> vfs.zfs.dirty_data_max_percent 10 > >> vfs.zfs.dirty_data_max_max 4294967296 > >> vfs.zfs.dirty_data_max 4294967296 > >> vfs.zfs.max_recordsize 1048576 > >> vfs.zfs.default_ibs 17 > >> vfs.zfs.default_bs 9 > >> vfs.zfs.send_holes_without_birth_time 1 > >> vfs.zfs.mdcomp_disable 0 > >> vfs.zfs.per_txg_dirty_frees_percent 5 > >> vfs.zfs.nopwrite_enabled 1 > >> vfs.zfs.dedup.prefetch 1 > >> vfs.zfs.dbuf_cache_lowater_pct 10 > >> vfs.zfs.dbuf_cache_hiwater_pct 10 > >> vfs.zfs.dbuf_metadata_cache_overflow 0 > >> vfs.zfs.dbuf_metadata_cache_shift 6 > >> vfs.zfs.dbuf_cache_shift 5 > >> vfs.zfs.dbuf_metadata_cache_max_bytes 1025282816 > >> vfs.zfs.dbuf_cache_max_bytes 2050565632 > >> vfs.zfs.arc_min_prescient_prefetch_ms 6 > >> vfs.zfs.arc_min_prefetch_ms 1 > >> vfs.zfs.l2c_only_size 0 > >> vfs.zfs.mfu_ghost_data_esize 7778263552 > >> vfs.zfs.mfu_ghost_metadata_esize 16851792896 > >> vfs.zfs.mfu_ghost_size 24630056448 > >> vfs.zfs.mfu_data_esize 3059418112 > >> vfs.zfs.mfu_metadata_esize 28641792 > >> vfs.zfs.mfu_size 6399023104 > >> vfs.zfs.mru_ghost_data_esize 2199812096 > >> vfs.zfs.mru_ghost_metadata_esize 6289682432 > >> vfs.zfs.mru_ghost_size 8489494528 > >> vfs.zfs.mru_data_esize 22781456384 > >> vfs.zfs.mru_metadata_esize 309155840 > >> vfs.zfs.mru_size 23847875584 > >> vfs.zfs.anon_data_esize 0 > >> vfs.zfs.anon_metadata_esize 0 > >> vfs.zfs.anon_size 8556544 > >> vfs.zfs.l2arc_norw 1 > >> vfs.zfs.l2arc_feed_again 1 > >> vfs.zfs.l2arc_noprefetch 1 > >> vfs.zfs.l2arc_feed_min_ms 200 > >> vfs.zfs.l2arc_feed_secs 1 > >> vfs.zfs.l2arc_headroom 2 > >> vfs.zfs.l2arc_write_boost 8388608 > >> vfs.zfs.l2arc_write_max 8388608 > >> vfs.zfs.arc_meta_strategy 1 > >> vfs.zfs.arc_meta_limit 15833624576 > >> vfs.zfs.arc_free_target 346902 > >> vfs.zfs.arc_kmem_cache_reap_retry_ms 1000 > >> vfs.zfs.compressed_arc_enabled 1 > >> vfs.zfs.arc_grow_retry 60 > >> vfs.zfs.arc_shrink_shift 7 > >> vfs.zfs.arc_average_blocksize 8192 > >> vfs.zfs.arc_no_grow_shift 5 > >> vfs.zfs.arc_min 8202262528 > >> vfs.zfs.arc_max 39334498304 > >> vfs.zfs.abd_chunk_size 4096 > >> vfs.zfs.abd_scatter_enabled 1 > >> > >> _______________________________________________ > >> freebsd-stable@freebsd.org mailing list > >> https://lists.freebsd.org/mailman/listinfo/freebsd-stable > >> To unsubscribe, send any mail to > >> "freebsd-stable-unsubscribe@freebsd.org" > >> > >> > >> > > _______________________________________________ > > freebsd-stable@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > > > From owner-freebsd-stable@freebsd.org Wed Jul 22 12:56:57 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 72DC637BEC9 for ; Wed, 22 Jul 2020 12:56:57 +0000 (UTC) (envelope-from mike@sentex.net) Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [IPv6:2607:f3e0:0:3::19]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "pyroxene.sentex.ca", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BBb885RRfz48sg for ; Wed, 22 Jul 2020 12:56:56 +0000 (UTC) (envelope-from mike@sentex.net) Received: from [IPv6:2607:f3e0:0:4:8125:6e09:8b8c:7875] ([IPv6:2607:f3e0:0:4:8125:6e09:8b8c:7875]) by pyroxene2a.sentex.ca (8.15.2/8.15.2) with ESMTPS id 06MCusSG069039 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Wed, 22 Jul 2020 08:56:54 -0400 (EDT) (envelope-from mike@sentex.net) Subject: Re: zfs meta data slowness To: Rick Macklem , Ronald Klop , FreeBSD-STABLE Mailing List References: <1949194763.1.1595250243575@localhost> <975657af-ccac-bbd1-e22b-86270c624226@sentex.net> From: mike tancsa Autocrypt: addr=mike@sentex.net; keydata= mQENBFywzOMBCACoNFpwi5MeyEREiCeHtbm6pZJI/HnO+wXdCAWtZkS49weOoVyUj5BEXRZP xflV2ib2hflX4nXqhenaNiia4iaZ9ft3I1ebd7GEbGnsWCvAnob5MvDZyStDAuRxPJK1ya/s +6rOvr+eQiXYNVvfBhrCfrtR/esSkitBGxhUkBjOti8QwzD71JVF5YaOjBAs7jZUKyLGj0kW yDg4jUndudWU7G2yc9GwpHJ9aRSUN8e/mWdIogK0v+QBHfv/dsI6zVB7YuxCC9Fx8WPwfhDH VZC4kdYCQWKXrm7yb4TiVdBh5kgvlO9q3js1yYdfR1x8mjK2bH2RSv4bV3zkNmsDCIxjABEB AAG0HW1pa2UgdGFuY3NhIDxtaWtlQHNlbnRleC5uZXQ+iQFUBBMBCAA+FiEEmuvCXT0aY6hs 4SbWeVOEFl5WrMgFAlywzOYCGwMFCQHhM4AFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQ eVOEFl5WrMhnPAf7Bf+ola0V9t4i8rwCMGvzkssGaxY/5zNSZO9BgSgfN0WzgmBEOy/3R4km Yn5KH94NltJYAAE5hqkFmAwK6psOqAR9cxHrRfU+gV2KO8pCDc6K/htkQcd/mclJYpCHp6Eq EVJOiAxcNaYuHZkeMdXDuvvI5Rk82VHk84BGgxIqIrhLlkguoPbXOOa+8c/Mpb1sRAGZEOuX EzKNC49+GS9gKW6ISbanyPsGEcFyP7GKMzcHBPf3cPrewZQZ6gBoNscasL6IJeAQDqzQAxbU GjO0qBSMRgnLXK7+DJlxrYdHGXqNbV6AYsmHJ6c2WWWiuRviFBqXinlgJ2FnYebZPAfWiQ== Message-ID: <963d17ae-d3bb-6824-4ef8-dbe70dd13791@sentex.net> Date: Wed, 22 Jul 2020 08:56:54 -0400 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 4BBb885RRfz48sg X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of mike@sentex.net designates 2607:f3e0:0:3::19 as permitted sender) smtp.mailfrom=mike@sentex.net X-Spamd-Result: default: False [-1.05 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.92)[-0.917]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2607:f3e0::/32]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; HFILTER_HELO_IP_A(1.00)[pyroxene2a.sentex.ca]; HFILTER_HELO_NORES_A_OR_MX(0.30)[pyroxene2a.sentex.ca]; DMARC_NA(0.00)[sentex.net]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; NEURAL_HAM_SHORT(-0.13)[-0.132]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Jul 2020 12:56:57 -0000 On 7/22/2020 1:04 AM, Rick Macklem wrote: > mike tancsa wrote: >> Hi, >> Thanks for the response. Reply in line >> >> On 7/20/2020 9:04 AM, Ronald Klop wrote: >>> Hi, >>> >>> My first suggestion would be to remove a lot of snapshots. But that my >>> not match your business case. >> As its a backup server, its sort of the point to have all those snapshots. > I'm the last guy who should be commenting on ZFS, since I never use it. > However, it is my understanding that ZFS "pseudo automounts" each > snapshot when you go there, so I think that might be what is taking > so long (ie. not really meta data). > Thanks Rick, in this case, its just listing snapshots from the command line zfs list -t snapshot that is taking forever. Best case scenario when the box boots up after running it once, it will take about 25 seconds but when the box is taking in zfs streams, it really slows down and can be anywhere up to 30min 0{backup4}# time zfs list -t snapshot > /tmp/snap.out 1.839u 23.211s 3:11.69 13.0%    71+178k 2504801+38io 0pf+0w 0{backup4}# time zfs list -t snapshot > /tmp/snap.out 1.817u 23.612s 0:25.47 99.8%    71+178k 2472088+38io 0pf+0w 0{backup4}# time zfs list -t snapshot > /tmp/snap.out 2.040u 23.314s 0:25.40 99.8%    71+177k 2472105+38io 0pf+0w 0{backup4}# From owner-freebsd-stable@freebsd.org Wed Jul 22 13:02:30 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id CC2D937C381 for ; Wed, 22 Jul 2020 13:02:30 +0000 (UTC) (envelope-from mike@sentex.net) Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [IPv6:2607:f3e0:0:3::19]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "pyroxene.sentex.ca", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BBbGZ1tKhz49JN for ; Wed, 22 Jul 2020 13:02:30 +0000 (UTC) (envelope-from mike@sentex.net) Received: from [IPv6:2607:f3e0:0:4:8125:6e09:8b8c:7875] ([IPv6:2607:f3e0:0:4:8125:6e09:8b8c:7875]) by pyroxene2a.sentex.ca (8.15.2/8.15.2) with ESMTPS id 06MD2RsQ069415 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Wed, 22 Jul 2020 09:02:27 -0400 (EDT) (envelope-from mike@sentex.net) Subject: Re: zfs meta data slowness To: Eugene Grosbein , Ronald Klop , FreeBSD-STABLE Mailing List References: <1949194763.1.1595250243575@localhost> <975657af-ccac-bbd1-e22b-86270c624226@sentex.net> <9ddce60a-9d91-fa69-0e11-54342cb67890@grosbein.net> From: mike tancsa Autocrypt: addr=mike@sentex.net; keydata= mQENBFywzOMBCACoNFpwi5MeyEREiCeHtbm6pZJI/HnO+wXdCAWtZkS49weOoVyUj5BEXRZP xflV2ib2hflX4nXqhenaNiia4iaZ9ft3I1ebd7GEbGnsWCvAnob5MvDZyStDAuRxPJK1ya/s +6rOvr+eQiXYNVvfBhrCfrtR/esSkitBGxhUkBjOti8QwzD71JVF5YaOjBAs7jZUKyLGj0kW yDg4jUndudWU7G2yc9GwpHJ9aRSUN8e/mWdIogK0v+QBHfv/dsI6zVB7YuxCC9Fx8WPwfhDH VZC4kdYCQWKXrm7yb4TiVdBh5kgvlO9q3js1yYdfR1x8mjK2bH2RSv4bV3zkNmsDCIxjABEB AAG0HW1pa2UgdGFuY3NhIDxtaWtlQHNlbnRleC5uZXQ+iQFUBBMBCAA+FiEEmuvCXT0aY6hs 4SbWeVOEFl5WrMgFAlywzOYCGwMFCQHhM4AFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQ eVOEFl5WrMhnPAf7Bf+ola0V9t4i8rwCMGvzkssGaxY/5zNSZO9BgSgfN0WzgmBEOy/3R4km Yn5KH94NltJYAAE5hqkFmAwK6psOqAR9cxHrRfU+gV2KO8pCDc6K/htkQcd/mclJYpCHp6Eq EVJOiAxcNaYuHZkeMdXDuvvI5Rk82VHk84BGgxIqIrhLlkguoPbXOOa+8c/Mpb1sRAGZEOuX EzKNC49+GS9gKW6ISbanyPsGEcFyP7GKMzcHBPf3cPrewZQZ6gBoNscasL6IJeAQDqzQAxbU GjO0qBSMRgnLXK7+DJlxrYdHGXqNbV6AYsmHJ6c2WWWiuRviFBqXinlgJ2FnYebZPAfWiQ== Message-ID: <2267c96f-b988-df1e-887b-77dc9b2a3e32@sentex.net> Date: Wed, 22 Jul 2020 09:02:27 -0400 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 In-Reply-To: <9ddce60a-9d91-fa69-0e11-54342cb67890@grosbein.net> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 4BBbGZ1tKhz49JN X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of mike@sentex.net designates 2607:f3e0:0:3::19 as permitted sender) smtp.mailfrom=mike@sentex.net X-Spamd-Result: default: False [-0.99 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.91)[-0.911]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2607:f3e0::/32]; NEURAL_HAM_LONG(-1.00)[-0.996]; MIME_GOOD(-0.10)[text/plain]; HFILTER_HELO_IP_A(1.00)[pyroxene2a.sentex.ca]; HFILTER_HELO_NORES_A_OR_MX(0.30)[pyroxene2a.sentex.ca]; DMARC_NA(0.00)[sentex.net]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; NEURAL_HAM_SHORT(-0.09)[-0.088]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Jul 2020 13:02:30 -0000 On 7/22/2020 1:29 AM, Eugene Grosbein wrote: > 22.07.2020 2:37, mike tancsa wrote: > >>> Something else special about the setup. >>> output of "top -b" >>> >> ports are right now being built in a VM, but the problem (zrepl hanging) >> and zfs list -t snapshots taking forever happens regardless >> >> PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU >> COMMAND >> 4439 root 12 40 20 6167M 5762M kqread 3 535:13 200.00% bhyve >> 98783 root 2 21 0 16M 5136K hdr->b 4 0:01 1.95% zfs >> 76489 root 21 23 0 738M 54M uwait 1 2:18 0.88% zrepl >> 98784 root 1 21 0 13M 3832K piperd 3 0:01 0.59% zfs >> 99563 root 1 20 0 13M 4136K zio->i 4 0:00 0.39% zfs > You need top -SHPI last pid: 66981;  load averages:  3.44,  3.25,  3.34  up 1+22:49:34    08:58:06 2056 threads:  12 running, 1988 sleeping, 3 zombie, 53 waiting CPU 0:  7.7% user,  0.1% nice, 15.0% system,  0.7% interrupt, 76.5% idle CPU 1:  9.9% user,  0.1% nice, 16.6% system,  0.1% interrupt, 73.2% idle CPU 2: 10.0% user,  0.1% nice, 17.5% system,  0.4% interrupt, 71.9% idle CPU 3: 10.3% user,  0.1% nice, 21.2% system,  0.1% interrupt, 68.2% idle CPU 4:  9.7% user,  0.1% nice, 15.6% system,  0.4% interrupt, 74.3% idle CPU 5: 10.2% user,  0.1% nice, 21.3% system,  0.1% interrupt, 68.3% idle CPU 6:  9.7% user,  0.1% nice, 16.6% system,  0.5% interrupt, 73.1% idle CPU 7: 10.1% user,  0.1% nice, 21.3% system,  0.1% interrupt, 68.4% idle Mem: 4236M Active, 19G Inact, 283M Laundry, 37G Wired, 3248K Buf, 1667M Free ARC: 25G Total, 9939M MFU, 12G MRU, 397M Anon, 573M Header, 2143M Other      20G Compressed, 29G Uncompressed, 1.43:1 Ratio Swap: 20G Total, 20G Free   PID USERNAME    PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND  4439 root        123   20  6167M  5856M CPU2     2  20.2H 100.00% bhyve{vcpu 0}  4439 root        122   20  6167M  5856M CPU3     3  20.2H  99.17% bhyve{vcpu 1}    11 root        155 ki31     0B   128K CPU0     0  35.9H  65.38% idle{idle: cpu0}    11 root        155 ki31     0B   128K CPU4     4  34.8H  63.38% idle{idle: cpu4} 66956 root         90    0    61M    54M CPU5     5   0:09  62.70% zfs    11 root        155 ki31     0B   128K CPU6     6  34.3H  58.06% idle{idle: cpu6}    11 root        155 ki31     0B   128K RUN      2  33.7H  54.49% idle{idle: cpu2}    11 root        155 ki31     0B   128K RUN      1  34.3H  53.76% idle{idle: cpu1}    11 root        155 ki31     0B   128K RUN      3  32.0H  53.47% idle{idle: cpu3}    11 root        155 ki31     0B   128K CPU7     7  32.0H  50.68% idle{idle: cpu7}    11 root        155 ki31     0B   128K RUN      5  32.0H  48.29% idle{idle: cpu5}    56 root        -12    -     0B  5168K -        1   5:49   9.67% zpool-zbackup2{zio_write_issue_3}    56 root        -12    -     0B  5168K -        3   5:48   9.57% zpool-zbackup2{zio_write_issue_5}    56 root        -12    -     0B  5168K -        4   5:48   9.47% zpool-zbackup2{zio_write_issue_4}    56 root        -12    -     0B  5168K -        7   5:48   9.47% zpool-zbackup2{zio_write_issue_2}    56 root        -12    -     0B  5168K -        0   5:49   9.38% zpool-zbackup2{zio_write_issue_1}    56 root        -12    -     0B  5168K -        1   5:48   9.38% zpool-zbackup2{zio_write_issue_0} 63392 root         23    0   729M    32M uwait    0   0:01   4.20% zrepl{zrepl} From owner-freebsd-stable@freebsd.org Wed Jul 22 13:27:19 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0A5E837C869 for ; Wed, 22 Jul 2020 13:27:19 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [IPv6:2a01:4f8:c2c:26d8::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BBbqB0rhbz4BdM for ; Wed, 22 Jul 2020 13:27:17 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13:0:0:0:5]) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id 06MDR2gu030258 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 22 Jul 2020 13:27:03 GMT (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: mike@sentex.net Received: from [10.58.0.10] (dadvw [10.58.0.10]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id 06MDR5o3053661 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Wed, 22 Jul 2020 20:27:05 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: zfs meta data slowness To: mike tancsa , Ronald Klop , FreeBSD-STABLE Mailing List References: <1949194763.1.1595250243575@localhost> <975657af-ccac-bbd1-e22b-86270c624226@sentex.net> <9ddce60a-9d91-fa69-0e11-54342cb67890@grosbein.net> <2267c96f-b988-df1e-887b-77dc9b2a3e32@sentex.net> From: Eugene Grosbein Message-ID: <097a63f1-3b25-3916-cdb9-1d4cf0d088fd@grosbein.net> Date: Wed, 22 Jul 2020 20:26:56 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <2267c96f-b988-df1e-887b-77dc9b2a3e32@sentex.net> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=0.3 required=5.0 tests=BAYES_00,LOCAL_FROM, NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.2 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record * -0.0 SPF_PASS SPF: sender matches SPF record * 2.6 LOCAL_FROM From my domains * -0.0 NICE_REPLY_A Looks like a legit reply (A) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on hz.grosbein.net X-Rspamd-Queue-Id: 4BBbqB0rhbz4BdM X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=permerror (mx1.freebsd.org: domain of eugen@grosbein.net uses mechanism not recognized by this client) smtp.mailfrom=eugen@grosbein.net X-Spamd-Result: default: False [-1.74 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.62)[-0.616]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-0.80)[-0.801]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[grosbein.net]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; R_SPF_PERMFAIL(0.00)[empty SPF record]; NEURAL_HAM_SHORT(-0.23)[-0.226]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE]; RCVD_TLS_ALL(0.00)[]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Jul 2020 13:27:19 -0000 22.07.2020 20:02, mike tancsa wrote: > > On 7/22/2020 1:29 AM, Eugene Grosbein wrote: >> 22.07.2020 2:37, mike tancsa wrote: >> >>>> Something else special about the setup. >>>> output of "top -b" >>>> >>> ports are right now being built in a VM, but the problem (zrepl hanging) >>> and zfs list -t snapshots taking forever happens regardless >>> >>> PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU >>> COMMAND >>> 4439 root 12 40 20 6167M 5762M kqread 3 535:13 200.00% bhyve >>> 98783 root 2 21 0 16M 5136K hdr->b 4 0:01 1.95% zfs >>> 76489 root 21 23 0 738M 54M uwait 1 2:18 0.88% zrepl >>> 98784 root 1 21 0 13M 3832K piperd 3 0:01 0.59% zfs >>> 99563 root 1 20 0 13M 4136K zio->i 4 0:00 0.39% zfs >> You need top -SHPI > > last pid: 66981; load averages: 3.44, 3.25, 3.34 up 1+22:49:34 > 08:58:06 > 2056 threads: 12 running, 1988 sleeping, 3 zombie, 53 waiting > CPU 0: 7.7% user, 0.1% nice, 15.0% system, 0.7% interrupt, 76.5% idle > CPU 1: 9.9% user, 0.1% nice, 16.6% system, 0.1% interrupt, 73.2% idle > CPU 2: 10.0% user, 0.1% nice, 17.5% system, 0.4% interrupt, 71.9% idle > CPU 3: 10.3% user, 0.1% nice, 21.2% system, 0.1% interrupt, 68.2% idle > CPU 4: 9.7% user, 0.1% nice, 15.6% system, 0.4% interrupt, 74.3% idle > CPU 5: 10.2% user, 0.1% nice, 21.3% system, 0.1% interrupt, 68.3% idle > CPU 6: 9.7% user, 0.1% nice, 16.6% system, 0.5% interrupt, 73.1% idle > CPU 7: 10.1% user, 0.1% nice, 21.3% system, 0.1% interrupt, 68.4% idle > Mem: 4236M Active, 19G Inact, 283M Laundry, 37G Wired, 3248K Buf, 1667M Free > ARC: 25G Total, 9939M MFU, 12G MRU, 397M Anon, 573M Header, 2143M Other > 20G Compressed, 29G Uncompressed, 1.43:1 Ratio > Swap: 20G Total, 20G Free > > PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 4439 root 123 20 6167M 5856M CPU2 2 20.2H 100.00% > bhyve{vcpu 0} > 4439 root 122 20 6167M 5856M CPU3 3 20.2H 99.17% > bhyve{vcpu 1} > 11 root 155 ki31 0B 128K CPU0 0 35.9H 65.38% > idle{idle: cpu0} > 11 root 155 ki31 0B 128K CPU4 4 34.8H 63.38% > idle{idle: cpu4} > 66956 root 90 0 61M 54M CPU5 5 0:09 62.70% zfs > 11 root 155 ki31 0B 128K CPU6 6 34.3H 58.06% > idle{idle: cpu6} > 11 root 155 ki31 0B 128K RUN 2 33.7H 54.49% > idle{idle: cpu2} > 11 root 155 ki31 0B 128K RUN 1 34.3H 53.76% > idle{idle: cpu1} > 11 root 155 ki31 0B 128K RUN 3 32.0H 53.47% > idle{idle: cpu3} > 11 root 155 ki31 0B 128K CPU7 7 32.0H 50.68% > idle{idle: cpu7} > 11 root 155 ki31 0B 128K RUN 5 32.0H 48.29% > idle{idle: cpu5} > 56 root -12 - 0B 5168K - 1 5:49 9.67% > zpool-zbackup2{zio_write_issue_3} > 56 root -12 - 0B 5168K - 3 5:48 9.57% > zpool-zbackup2{zio_write_issue_5} > 56 root -12 - 0B 5168K - 4 5:48 9.47% > zpool-zbackup2{zio_write_issue_4} > 56 root -12 - 0B 5168K - 7 5:48 9.47% > zpool-zbackup2{zio_write_issue_2} > 56 root -12 - 0B 5168K - 0 5:49 9.38% > zpool-zbackup2{zio_write_issue_1} > 56 root -12 - 0B 5168K - 1 5:48 9.38% > zpool-zbackup2{zio_write_issue_0} > 63392 root 23 0 729M 32M uwait 0 0:01 4.20% > zrepl{zrepl} It's hard to read due to wrapping plus, it is truncated. Maybe additional flag -d3 or similar will help, combined with dedirection < /dev/null > top.out so it won't use size of your terminal to wrap/truncate output. Also, make sure you invoke top while "zfs" command is running. Also, procstat -kk for pid of "zfs" command would be useful (but may occur pretty long). I suppose it blocks waiting for some kernel lock and procstat would show details. From owner-freebsd-stable@freebsd.org Fri Jul 24 09:43:51 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3A763377B7D for ; Fri, 24 Jul 2020 09:43:51 +0000 (UTC) (envelope-from Alexander@leidinger.net) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 4BCkmQ4wBZz4JNb for ; Fri, 24 Jul 2020 09:43:50 +0000 (UTC) (envelope-from Alexander@leidinger.net) Received: by mailman.nyi.freebsd.org (Postfix) id A7020377996; Fri, 24 Jul 2020 09:43:50 +0000 (UTC) Delivered-To: stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A5B9237784F for ; Fri, 24 Jul 2020 09:43:50 +0000 (UTC) (envelope-from Alexander@leidinger.net) Received: from mailgate.Leidinger.net (bastille.leidinger.net [89.238.82.207]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4BCkmP3y9hz4JKs for ; Fri, 24 Jul 2020 09:43:43 +0000 (UTC) (envelope-from Alexander@leidinger.net) Received: from outgoing.leidinger.net (p5b165853.dip0.t-ipconnect.de [91.22.88.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-256) server-digest SHA256) (Client did not present a certificate) by mailgate.Leidinger.net (Postfix) with ESMTPSA id 1D1FB4844; Fri, 24 Jul 2020 11:43:37 +0200 (CEST) Received: from webmail.leidinger.net (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-256) server-digest SHA256) (Client did not present a certificate) by outgoing.leidinger.net (Postfix) with ESMTPS id 54C271D82; Fri, 24 Jul 2020 11:43:35 +0200 (CEST) Date: Fri, 24 Jul 2020 11:43:35 +0200 Message-ID: <20200724114335.Horde.qXv0qmzOIxJll_4bX4-3J-5@webmail.leidinger.net> From: Alexander Leidinger To: jails@freebsd.org, stable@freebsd.org, curent@freebsd.org Subject: Start of "Container Orchestration" wiki page Accept-Language: de,en Content-Type: multipart/signed; boundary="=_mVADpE8OEH3mDC2A_YcbQ7P"; protocol="application/pgp-signature"; micalg=pgp-sha1 MIME-Version: 1.0 X-Rspamd-Queue-Id: 4BCkmP3y9hz4JKs X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.21 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[leidinger.net:s=outgoing-alex]; NEURAL_HAM_MEDIUM(-1.01)[-1.015]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-1.01)[-1.008]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[leidinger.net:+]; DMARC_POLICY_ALLOW(-0.50)[leidinger.net,quarantine]; NEURAL_HAM_SHORT(-0.09)[-0.087]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:34240, ipnet:89.238.64.0/18, country:DE]; RCVD_TLS_ALL(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[91.22.88.83:received] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Jul 2020 09:43:51 -0000 This message is in MIME format and has been PGP signed. --=_mVADpE8OEH3mDC2A_YcbQ7P Content-Type: text/plain; charset=utf-8; format=flowed; DelSp=Yes Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi, I have started a wiki page wich talks about container orchestration. https://wiki.freebsd.org/ContainerOrchestration It is not yet connected to the front-page (anyone with a good idea=20=20 where=20it fits in the frontpage: feel free to add it there). My goal of this page (I'm open to extend or change the goal) is to=20=20 break=20up a little bit the image of a container (=3D a concept) being=20= =20 equal=20to docker (an implementation of the concept + an orchestration=20= =20 tool),=20and to show what we already have in this area. So this is=20=20 partly=20PR/marketing, and partly knowledge transfer. I have added the tools I'm aware of, and for those which I had in use=20=20 or=20was quickly able to grab the info I was adding to the tools I=20=20 use/know. It=20would be nice if some people would chime in and help a little bit=20= =20 out=20with content and ideas there. Bye, Alexander. --=20 http://www.Leidinger.net=20Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF http://www.FreeBSD.org netchild@FreeBSD.org : PGP 0x8F31830F9F2772BF --=_mVADpE8OEH3mDC2A_YcbQ7P Content-Type: application/pgp-signature Content-Description: Digitale PGP-Signatur Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAABAgAGBQJfGq1HAAoJEBINsJsD+NiGgMYP/1TR+DSObP61Mp78hF81z5jt keTTTSj6Od4ljTm6TpLlqvwUwrXkrEqyImdcsOigUckZpV2GnTDRB2YhEGmeWo2r Yh/UCo8qHqYAluyOoSruOlmpn6KICS79EvCITeJ0blygWaoV5mxAm1TDeZyxy63E lVV5kyZ2eEpIgY1WP/8CYZMibjM1MiGZ41PmnJvfXVoZl2/grSgtg+BgpZwcCJon Myg/0DrdRVJfebBGuHBo3B7nB78jIrGcjrGPDAaXDl4+Ilo9I/QaloA6QgRgqXH/ vML8eKA1phAopuZ/hbal4zONibxGgVXi/GUR6Z9vybenyumtEg9CDpGILtfi9kXC 1qPr5Sh0cpJh+j+9Hy03L06ZltkVJonWdLacLPjUvKHiu5O7UV1lzIay1DeMjQia 8245SiIzlvs8aIie55OM/2SmGOVKK5wgd9a2v4weKl9MbZizw1CtKDuyhIeG9SUY SmXG576JUpvDyc77nHU2mZ5V0S/PdRMEoTVB8/IfekIjZ0bch8rS6nOiRBI4r4WE qYsFwEscqM+brWTydK+viLk2gkQGC3HYHSwYmEl6GQhsEptnKSK4GCEZ1i1FKah0 u5zxqGvV3TFUjAqpQ2t59H8TRTLkJeMXrSyZfUzMKrcMLBAyjaEieF7vodow9bQO bEEkM8MBI0l+pBvH9FvE =M099 -----END PGP SIGNATURE----- --=_mVADpE8OEH3mDC2A_YcbQ7P-- From owner-freebsd-stable@freebsd.org Sat Jul 25 20:05:22 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4E13636E88E for ; Sat, 25 Jul 2020 20:05:22 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BDcW61Mycz4bWR for ; Sat, 25 Jul 2020 20:05:22 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from mousie.catspoiler.org (unknown [76.212.85.177]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) (Authenticated sender: truckman) by smtp.freebsd.org (Postfix) with ESMTPSA id 9E7EB33C48 for ; Sat, 25 Jul 2020 20:05:21 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Date: Sat, 25 Jul 2020 13:05:19 -0700 (PDT) From: Don Lewis Subject: 11-STABLE and 12-STABLE build failures To: freebsd-stable@FreeBSD.org Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; CHARSET=us-ascii Content-Disposition: INLINE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Jul 2020 20:05:22 -0000 I'm seeing this failure when building 11-STABLE and 12-STABLE poudriere jails: --- localedef.full --- cc -O2 -pipe -fno-common -I. -I/var/poudriere/jails/12STABLEamd64/usr/src/usr.bin/localedef -I/var/poudriere/jails/12STABLEamd64/usr/src/lib/libc/locale -I/var/poudriere/jails/12STABLEamd64/usr/src/lib/libc/stdtime -g -std=gnu99 -Qunused-arguments -I/usr/obj/var/poudriere/jails/12STABLEamd64/usr/src/amd64.amd64/tmp/legacy/usr/include -static -L/usr/obj/var/poudriere/jails/12STABLEamd64/usr/src/amd64.amd64/tmp/legacy/usr/lib -o localedef.full charmap.o collate.o ctype.o localedef.o messages.o monetary.o numeric.o parser.o scanner.o time.o wide.o -legacy ld: error: undefined symbol: yydebug >>> referenced by localedef.c:276 (/var/poudriere/jails/12STABLEamd64/usr/src/usr.bin/localedef/localedef.c:276) >>> localedef.o:(main) cc: error: linker command failed with exit code 1 (use -v to see invocation) *** [localedef.full] Error code 1 I hadn't done a build in a while, so I don't know how long this has been broken. My builds last night failed and things are still broken right now. From owner-freebsd-stable@freebsd.org Sat Jul 25 20:13:30 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5F68C36E9DD for ; Sat, 25 Jul 2020 20:13:30 +0000 (UTC) (envelope-from jonc@chen.org.nz) Received: from mail-oo1-xc42.google.com (mail-oo1-xc42.google.com [IPv6:2607:f8b0:4864:20::c42]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BDchT1K1Wz4c1H for ; Sat, 25 Jul 2020 20:13:27 +0000 (UTC) (envelope-from jonc@chen.org.nz) Received: by mail-oo1-xc42.google.com with SMTP id y9so2470186oot.9 for ; Sat, 25 Jul 2020 13:13:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chen-org-nz.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=Tv05j5FEcGd9DYf/a9kO5lzTCcbZYl38Sy7h7g0SVCk=; b=v4DJdwX2a6PaVPicHrBzyY2zL1/nnrP9xcLe+quAh+VvK5JCyxOQiD27sxSFG/BKif 68ymHccIaRNHD/b9z+TO2+fbGtJZ3tJk7eW2+xXEmvbZNWMyCHkc903f4jx8jCkcSsBd wwN/07+StOGJnPlrOBtz+FJ4gJo/S4YsWj3Q3T0okTgN4t89Shba4apHgYJu8AQfw78j JkiZYdTlqYSy3rYrxDdmNmhbTsxCvowngmJUobAbe6XeS+c9OIg7ZHcr/Zk7jmKzkQMV nPghWZIEr+a9PLMJckvQxGyX1ZWyZAkyZ024Ls4k759uqW2cCKEwhPDJJGBRFweNP9+u zm5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=Tv05j5FEcGd9DYf/a9kO5lzTCcbZYl38Sy7h7g0SVCk=; b=NZFN3lsf/2/aKJr0eIrB3QHE8MNZqBVrLXLMe3BGPNncCEVpdCA8uHxCW3R8cMjPIZ nkfX+5tqqMFRnx0+3n2Zk9hC6ZW+3vvz+VfAh6uWbgBAajbm8NyXd5VjSI5ixjBvkJo/ 2/okYc4PX/szPlSPCvLAPXwhOJddAmOL9aLaZe+zhoWlyFl62GdFzF6KrIdhSsPeVVK4 poIEiQb97pRULlrSQo9NTycG5f7IXMirFi/3WH5XHf2Q8mjrCD94aTHSRbnQ5GNatkhN rFsltFEn35eulzrv8ESdVTiovkfGLjxJyDEjAFRIHdxKS3mSCZ/LMZGU1uFSuB3qFpyU PuEA== X-Gm-Message-State: AOAM530Xtmve855vaF0QZLWh0sgJHQ4dA/RUxYhULmLo9cF55f7T6fYy 9BtfFRMJ9yYRL8NMM1VA2hXVSWqMWZKg7fEX5i6VIQ== X-Google-Smtp-Source: ABdhPJyg/ACVKfbuBNxhO3d4cdk5+QKbLBLsGcosps3yNnJ0a8/cyHpif+QMCXGVB5C77uVhTpTcuxRmpSoZ2soX53c= X-Received: by 2002:a4a:88e6:: with SMTP id q35mr15294284ooh.10.1595708006795; Sat, 25 Jul 2020 13:13:26 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Jonathan Chen Date: Sun, 26 Jul 2020 08:13:11 +1200 Message-ID: Subject: Re: 11-STABLE and 12-STABLE build failures To: Don Lewis Cc: freebsd-stable@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 4BDchT1K1Wz4c1H X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=chen-org-nz.20150623.gappssmtp.com header.s=20150623 header.b=v4DJdwX2; dmarc=none; spf=softfail (mx1.freebsd.org: 2607:f8b0:4864:20::c42 is neither permitted nor denied by domain of jonc@chen.org.nz) smtp.mailfrom=jonc@chen.org.nz X-Spamd-Result: default: False [-2.63 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.83)[-0.831]; R_DKIM_ALLOW(-0.20)[chen-org-nz.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-0.99)[-0.987]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; DMARC_NA(0.00)[chen.org.nz]; R_SPF_SOFTFAIL(0.00)[~all]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[chen-org-nz.20150623.gappssmtp.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::c42:from]; NEURAL_HAM_SHORT(-0.51)[-0.509]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Jul 2020 20:13:30 -0000 On Sun, 26 Jul 2020 at 08:05, Don Lewis wrote: > > I'm seeing this failure when building 11-STABLE and 12-STABLE poudriere > jails: > > --- localedef.full --- > cc -O2 -pipe -fno-common -I. -I/var/poudriere/jails/12STABLEamd64/usr/src= /usr.bin/localedef -I/var/poudriere/jails/12STABLEamd64/usr/src/lib/libc/lo= cale -I/var/poudriere/jails/12STABLEamd64/usr/src/lib/libc/stdtime -g -std= =3Dgnu99 -Qunused-arguments -I/usr/obj/var/poudriere/jails/12STABLEamd64/us= r/src/amd64.amd64/tmp/legacy/usr/include -static -L/usr/obj/var/poudriere= /jails/12STABLEamd64/usr/src/amd64.amd64/tmp/legacy/usr/lib -o localedef.fu= ll charmap.o collate.o ctype.o localedef.o messages.o monetary.o numeric.o = parser.o scanner.o time.o wide.o -legacy > ld: error: undefined symbol: yydebug > >>> referenced by localedef.c:276 (/var/poudriere/jails/12STABLEamd64/usr= /src/usr.bin/localedef/localedef.c:276) > >>> localedef.o:(main) > cc: error: linker command failed with exit code 1 (use -v to see invocati= on) > *** [localedef.full] Error code 1 > > I hadn't done a build in a while, so I don't know how long this has been > broken. My builds last night failed and things are still broken right > now. It must be something recent, as my last build of 12-STABLE at r363443 (~23 Jul) succeeded. Cheers. --=20 Jonathan Chen From owner-freebsd-stable@freebsd.org Sat Jul 25 20:33:52 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1338436F71E for ; Sat, 25 Jul 2020 20:33:52 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qk1-x72e.google.com (mail-qk1-x72e.google.com [IPv6:2607:f8b0:4864:20::72e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BDd7y0krMz4csP for ; Sat, 25 Jul 2020 20:33:49 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qk1-x72e.google.com with SMTP id d14so11900997qke.13 for ; Sat, 25 Jul 2020 13:33:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=Fz5cvh32w6mCjcM6XK9+tZkM1RDvb2PYGJBIwadpFFo=; b=XRCl9I9Ck8S5Dnb3ntTid8sltlQzeaXA/vX5iu8aUYE7oDdw6d/vwWyG+mrniKDJAw Bzu0WtXLS+FWFwa4S4Zd/vU/sFdAGWUYHpnC+rzOJ7007hfLcHo+OdqnAbAig1vC9bew tEcEf31k1sAs2IacafUCbtZkIsB66GEYdMOniSFJsdQXeJ4O8HmuWvAppXvxOr8bBj+/ rgq002wpRwGhL6+zZ94oGZPcVP36rohuPU6RDSHYZR//xR+UpwWWNsv2DZYjhzC2Mujh MIp9TIphcGImdAr9Y7NIo7hrmJtEAUXBnEgWRMc3VSgJwHiiwG8DjcqSZnKlvRt3GhgT tz0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Fz5cvh32w6mCjcM6XK9+tZkM1RDvb2PYGJBIwadpFFo=; b=dH1fbwzqT8nW9wbqp4TsEzZNjR/pErhujc2lEhN+wVmLOv9tt3/CbAr43qDn10kl88 MkgL645GhYRtZXmhWyDu/3ffqlRRuUt7Dd72csFKgSgTj9dyDBRbcdaRcGzP8RPJj/TX xiu728ObD09XzURJ826AMhRnl71ReEeCC1s2y0zc5SuIs097uETcxxNee6pfUezpM+Ni gk4X8uO1hbnlkep+G5kewpC7q9h5ddOkSQCfKAAGp8vhbmJiTSe8UIQuN9e3oJMC512u fbIziEdvYcnup+gxkHRKSORTiyvuZZ3eRNDomjrxqKmFqna/PgRY4QwW48R3arZ3vR4Y +Kqw== X-Gm-Message-State: AOAM531iux7WYHc3/XjxWrxdZN3Ur6gTWPRqjLWNmwSi8Pq1zCoxS368 Wmxd7zToE8qzIAbqg+xnN+XAh+9frZG0UMbPVV1fOg== X-Google-Smtp-Source: ABdhPJzJMhSM8lJwXHcN9iyjsrLa0t8PBtXuDWHg7oFWHB/IFA1YME+bc4zg8HC9xxG2W8ok2AVGSEKdUmPcVJXbLKA= X-Received: by 2002:a37:66d7:: with SMTP id a206mr16312602qkc.495.1595709228745; Sat, 25 Jul 2020 13:33:48 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Warner Losh Date: Sat, 25 Jul 2020 14:33:37 -0600 Message-ID: Subject: Re: 11-STABLE and 12-STABLE build failures To: Don Lewis Cc: FreeBSD-STABLE Mailing List X-Rspamd-Queue-Id: 4BDd7y0krMz4csP X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20150623.gappssmtp.com header.s=20150623 header.b=XRCl9I9C; dmarc=none; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2607:f8b0:4864:20::72e) smtp.mailfrom=wlosh@bsdimp.com X-Spamd-Result: default: False [-2.34 / 15.00]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com:s=20150623]; NEURAL_HAM_MEDIUM(-0.89)[-0.887]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-0.97)[-0.968]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::72e:from]; NEURAL_HAM_SHORT(-0.49)[-0.489]; R_SPF_NA(0.00)[no SPF record]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_COUNT_TWO(0.00)[2]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.33 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Jul 2020 20:33:52 -0000 Liby.a was retired. Maybe there is some dangling references? On Sat, Jul 25, 2020, 2:05 PM Don Lewis wrote: > I'm seeing this failure when building 11-STABLE and 12-STABLE poudriere > jails: > > --- localedef.full --- > cc -O2 -pipe -fno-common -I. > -I/var/poudriere/jails/12STABLEamd64/usr/src/usr.bin/localedef > -I/var/poudriere/jails/12STABLEamd64/usr/src/lib/libc/locale > -I/var/poudriere/jails/12STABLEamd64/usr/src/lib/libc/stdtime -g -std=gnu99 > -Qunused-arguments > -I/usr/obj/var/poudriere/jails/12STABLEamd64/usr/src/amd64.amd64/tmp/legacy/usr/include > -static > -L/usr/obj/var/poudriere/jails/12STABLEamd64/usr/src/amd64.amd64/tmp/legacy/usr/lib > -o localedef.full charmap.o collate.o ctype.o localedef.o messages.o > monetary.o numeric.o parser.o scanner.o time.o wide.o -legacy > ld: error: undefined symbol: yydebug > >>> referenced by localedef.c:276 > (/var/poudriere/jails/12STABLEamd64/usr/src/usr.bin/localedef/localedef.c:276) > >>> localedef.o:(main) > cc: error: linker command failed with exit code 1 (use -v to see > invocation) > *** [localedef.full] Error code 1 > > I hadn't done a build in a while, so I don't know how long this has been > broken. My builds last night failed and things are still broken right > now. > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > From owner-freebsd-stable@freebsd.org Sat Jul 25 20:42:13 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id EA74236FB3D for ; Sat, 25 Jul 2020 20:42:13 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BDdKd60zfz4dVD; Sat, 25 Jul 2020 20:42:13 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from mousie.catspoiler.org (unknown [76.212.85.177]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) (Authenticated sender: truckman) by smtp.freebsd.org (Postfix) with ESMTPSA id 50C6B33D6C; Sat, 25 Jul 2020 20:42:13 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Date: Sat, 25 Jul 2020 13:42:11 -0700 (PDT) From: Don Lewis Subject: Re: 11-STABLE and 12-STABLE build failures To: Warner Losh cc: FreeBSD-STABLE Mailing List In-Reply-To: Message-ID: References: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; CHARSET=us-ascii Content-Disposition: INLINE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Jul 2020 20:42:14 -0000 On 25 Jul, Warner Losh wrote: > Liby.a was retired. Maybe there is some dangling references? # grep yydebug * localedef.c: yydebug = 0; localedef.h:extern int yydebug; I see the same in the 13-CURRENT source and it builds successfully. > On Sat, Jul 25, 2020, 2:05 PM Don Lewis wrote: > >> I'm seeing this failure when building 11-STABLE and 12-STABLE poudriere >> jails: >> >> --- localedef.full --- >> cc -O2 -pipe -fno-common -I. >> -I/var/poudriere/jails/12STABLEamd64/usr/src/usr.bin/localedef >> -I/var/poudriere/jails/12STABLEamd64/usr/src/lib/libc/locale >> -I/var/poudriere/jails/12STABLEamd64/usr/src/lib/libc/stdtime -g -std=gnu99 >> -Qunused-arguments >> -I/usr/obj/var/poudriere/jails/12STABLEamd64/usr/src/amd64.amd64/tmp/legacy/usr/include >> -static >> -L/usr/obj/var/poudriere/jails/12STABLEamd64/usr/src/amd64.amd64/tmp/legacy/usr/lib >> -o localedef.full charmap.o collate.o ctype.o localedef.o messages.o >> monetary.o numeric.o parser.o scanner.o time.o wide.o -legacy >> ld: error: undefined symbol: yydebug >> >>> referenced by localedef.c:276 >> (/var/poudriere/jails/12STABLEamd64/usr/src/usr.bin/localedef/localedef.c:276) >> >>> localedef.o:(main) >> cc: error: linker command failed with exit code 1 (use -v to see >> invocation) >> *** [localedef.full] Error code 1 >> >> I hadn't done a build in a while, so I don't know how long this has been >> broken. My builds last night failed and things are still broken right >> now. >> >> _______________________________________________ >> freebsd-stable@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-stable >> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" >> From owner-freebsd-stable@freebsd.org Sat Jul 25 20:46:14 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 2D4D536FC69 for ; Sat, 25 Jul 2020 20:46:14 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BDdQG0RVTz4dWW; Sat, 25 Jul 2020 20:46:14 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from mousie.catspoiler.org (unknown [76.212.85.177]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) (Authenticated sender: truckman) by smtp.freebsd.org (Postfix) with ESMTPSA id 6516734152; Sat, 25 Jul 2020 20:46:13 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Date: Sat, 25 Jul 2020 13:46:13 -0700 (PDT) From: Don Lewis Subject: Re: 11-STABLE and 12-STABLE build failures To: Warner Losh cc: FreeBSD-STABLE Mailing List In-Reply-To: Message-ID: References: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; CHARSET=us-ascii Content-Disposition: INLINE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Jul 2020 20:46:14 -0000 On 25 Jul, Don Lewis wrote: > On 25 Jul, Warner Losh wrote: >> Liby.a was retired. Maybe there is some dangling references? > > # grep yydebug * > localedef.c: yydebug = 0; > localedef.h:extern int yydebug; > > I see the same in the 13-CURRENT source and it builds successfully. If I diff the two source trees, I see that the 13-CURRENT references are guarded by #if YYDEBUG. From owner-freebsd-stable@freebsd.org Sat Jul 25 20:50:48 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 77E8337002B for ; Sat, 25 Jul 2020 20:50:48 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BDdWX2dYLz4dtR; Sat, 25 Jul 2020 20:50:48 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from mousie.catspoiler.org (unknown [76.212.85.177]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) (Authenticated sender: truckman) by smtp.freebsd.org (Postfix) with ESMTPSA id B1DDB33D6F; Sat, 25 Jul 2020 20:50:47 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Date: Sat, 25 Jul 2020 13:50:47 -0700 (PDT) From: Don Lewis Subject: Re: 11-STABLE and 12-STABLE build failures To: Warner Losh cc: FreeBSD-STABLE Mailing List In-Reply-To: Message-ID: References: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; CHARSET=us-ascii Content-Disposition: INLINE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Jul 2020 20:50:48 -0000 On 25 Jul, Don Lewis wrote: > On 25 Jul, Don Lewis wrote: >> On 25 Jul, Warner Losh wrote: >>> Liby.a was retired. Maybe there is some dangling references? >> >> # grep yydebug * >> localedef.c: yydebug = 0; >> localedef.h:extern int yydebug; >> >> I see the same in the 13-CURRENT source and it builds successfully. > > If I diff the two source trees, I see that the 13-CURRENT references are > guarded by #if YYDEBUG. Looks like we need this MFC: %svn log -c 362569 ------------------------------------------------------------------------ r362569 | jkim | 2020-06-23 19:08:08 -0700 (Tue, 23 Jun 2020) | 2 lines Fix build with recent byacc. ------------------------------------------------------------------------ From owner-freebsd-stable@freebsd.org Sat Jul 25 20:59:47 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D414A370445 for ; Sat, 25 Jul 2020 20:59:47 +0000 (UTC) (envelope-from yuripv@yuripv.dev) Received: from wnew4-smtp.messagingengine.com (wnew4-smtp.messagingengine.com [64.147.123.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4BDdjv0HGrz4fLR; Sat, 25 Jul 2020 20:59:46 +0000 (UTC) (envelope-from yuripv@yuripv.dev) Received: from compute2.internal (compute2.nyi.internal [10.202.2.42]) by mailnew.west.internal (Postfix) with ESMTP id 6164395B; Sat, 25 Jul 2020 16:59:45 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Sat, 25 Jul 2020 16:59:45 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yuripv.dev; h= subject:to:cc:references:from:message-id:date:mime-version :in-reply-to:content-type:content-transfer-encoding; s=fm1; bh=C iXxURZJolQAXW8Q9CoPav2Jcz9OG/6expCgE7TvKTY=; b=hVBzX4Xdt5YpueL3v zultNveLcSGYLsiGfuvbW5R2z4lic8T8LEQoPB7fuBoDVMQ9FZoCOVt+ZRrt3IcA uNTNAmCeTupYYx5xqCtDURSkVbXUjDHQ6oX766IL79FajFU4uxoA0BN6CNkhPnXj jD5ifWfLPCLO+m6TtgyEvYIbatPKou2vE/Nqwhb2DsB5AjIq+p5OUMjUKx9tQK3d HLUB7h1j6U0m2ygIXg+0Q40urdEmqPcjJ76PEt1pqvPLjdTdVF3tzU4u5sqRc3bW GmzF2/Uu4mMuBCtmgDtC9qvCDqV8zD+cCLVL11lIj+jVIvAah+hw1nCdfSdo/x9P n0WDQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; bh=CiXxURZJolQAXW8Q9CoPav2Jcz9OG/6expCgE7TvK TY=; b=mEqRO0iJNVuZrdRgaQX/+v/GaVg2FUC1W6eD9K0z3qrw3UJQcIKdSKYIX +49QsLHrpV15uLYZ1ZP3/24N5Allya61bznYGJXMHpGOwTTijCHcX+SpS1IT8vxw jVcbnYUbKae9zXUxasZC6FCvZWPo1jFZzWa8/IA3+yAWJlEkER3fer9CI3VbltpV gZwKUZYl2coWk+rCHeTVeLQY4IWvD2Qrue61iAo4d+022XDx2aJaAIpB/QTPb5um Fw3OYf6q4zSdFNdwApiAPe0jsO9y3ukYk6BRiGwCAV5zCwbzC9hxZ0bMrCNTAWUo +ZyUVHq1QJMR9bBlINgNx/A7VDqDw== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduiedrheehgdduheejucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepuffvfhfhkffffgggjggtgfesthejredttdefjeenucfhrhhomhepjghurhhi ucfrrghnkhhovhcuoeihuhhrihhpvheshihurhhiphhvrdguvghvqeenucggtffrrghtth gvrhhnpeffhedvkeegieejveelvdffgfevueevgffgjefhleeuueevtedvvddvgfeiteff teenucfkphepudekhedrvdeggedrvdduhedrleenucevlhhushhtvghrufhiiigvpedtne curfgrrhgrmhepmhgrihhlfhhrohhmpeihuhhrihhpvheshihurhhiphhvrdguvghv X-ME-Proxy: Received: from [192.168.1.6] (unknown [185.244.215.9]) by mail.messagingengine.com (Postfix) with ESMTPA id 5DFC73280060; Sat, 25 Jul 2020 16:59:43 -0400 (EDT) Subject: Re: 11-STABLE and 12-STABLE build failures To: Don Lewis , Warner Losh Cc: FreeBSD-STABLE Mailing List References: From: Yuri Pankov Message-ID: <515e9746-7d30-341b-7e6a-50e0fa29b9ff@yuripv.dev> Date: Sat, 25 Jul 2020 23:59:40 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4BDdjv0HGrz4fLR X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yuripv.dev header.s=fm1 header.b=hVBzX4Xd; dkim=pass header.d=messagingengine.com header.s=fm3 header.b=mEqRO0iJ; dmarc=none; spf=pass (mx1.freebsd.org: domain of yuripv@yuripv.dev designates 64.147.123.18 as permitted sender) smtp.mailfrom=yuripv@yuripv.dev X-Spamd-Result: default: False [-3.85 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yuripv.dev:s=fm1,messagingengine.com:s=fm3]; NEURAL_HAM_MEDIUM(-1.03)[-1.026]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip4:64.147.123.18]; NEURAL_HAM_LONG(-0.97)[-0.967]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[yuripv.dev]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yuripv.dev:+,messagingengine.com:+]; NEURAL_HAM_SHORT(-1.25)[-1.253]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:11403, ipnet:64.147.123.0/24, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_IN_DNSWL_LOW(-0.10)[64.147.123.18:from] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Jul 2020 20:59:47 -0000 Don Lewis wrote: > On 25 Jul, Don Lewis wrote: >> On 25 Jul, Don Lewis wrote: >>> On 25 Jul, Warner Losh wrote: >>>> Liby.a was retired. Maybe there is some dangling references? >>> >>> # grep yydebug * >>> localedef.c: yydebug = 0; >>> localedef.h:extern int yydebug; >>> >>> I see the same in the 13-CURRENT source and it builds successfully. >> >> If I diff the two source trees, I see that the 13-CURRENT references are >> guarded by #if YYDEBUG. > > Looks like we need this MFC: > > %svn log -c 362569 > ------------------------------------------------------------------------ > r362569 | jkim | 2020-06-23 19:08:08 -0700 (Tue, 23 Jun 2020) | 2 lines > > Fix build with recent byacc. > > ------------------------------------------------------------------------ Just checked that I see this too trying to build stable/12 on recent head, failing while bootstrapping tools where we use the host system binaries. From owner-freebsd-stable@freebsd.org Sat Jul 25 23:12:04 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8B6293736DE for ; Sat, 25 Jul 2020 23:12:04 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BDhfX3F1Dz3X4y; Sat, 25 Jul 2020 23:12:04 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from mousie.catspoiler.org (unknown [76.212.85.177]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) (Authenticated sender: truckman) by smtp.freebsd.org (Postfix) with ESMTPSA id B587035340; Sat, 25 Jul 2020 23:12:03 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Date: Sat, 25 Jul 2020 16:12:01 -0700 (PDT) From: Don Lewis Subject: Re: 11-STABLE and 12-STABLE build failures To: Yuri Pankov cc: Warner Losh , FreeBSD-STABLE Mailing List In-Reply-To: <515e9746-7d30-341b-7e6a-50e0fa29b9ff@yuripv.dev> Message-ID: References: <515e9746-7d30-341b-7e6a-50e0fa29b9ff@yuripv.dev> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; CHARSET=us-ascii Content-Disposition: INLINE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Jul 2020 23:12:04 -0000 On 25 Jul, Yuri Pankov wrote: > Don Lewis wrote: >> On 25 Jul, Don Lewis wrote: >>> On 25 Jul, Don Lewis wrote: >>>> On 25 Jul, Warner Losh wrote: >>>>> Liby.a was retired. Maybe there is some dangling references? >>>> >>>> # grep yydebug * >>>> localedef.c: yydebug = 0; >>>> localedef.h:extern int yydebug; >>>> >>>> I see the same in the 13-CURRENT source and it builds successfully. >>> >>> If I diff the two source trees, I see that the 13-CURRENT references are >>> guarded by #if YYDEBUG. >> >> Looks like we need this MFC: >> >> %svn log -c 362569 >> ------------------------------------------------------------------------ >> r362569 | jkim | 2020-06-23 19:08:08 -0700 (Tue, 23 Jun 2020) | 2 lines >> >> Fix build with recent byacc. >> >> ------------------------------------------------------------------------ > > Just checked that I see this too trying to build stable/12 on recent > head, failing while bootstrapping tools where we use the host system > binaries. I just did the MFC, so it should be fixed now.