From owner-freebsd-fs@freebsd.org Tue Jun 20 17:16:35 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EA321D9D5C7 for ; Tue, 20 Jun 2017 17:16:35 +0000 (UTC) (envelope-from Aaron.Caza@ca.weatherford.com) Received: from NAM01-BN3-obe.outbound.protection.outlook.com (mail-bn3nam01on0085.outbound.protection.outlook.com [104.47.33.85]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2E75A670C4 for ; Tue, 20 Jun 2017 17:16:34 +0000 (UTC) (envelope-from Aaron.Caza@ca.weatherford.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weatherford.onmicrosoft.com; s=selector1-ca-weatherford-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=toPZ4M6HGVap0fEigG2yG+UVU+5CY7tJwEMtbuq7Xz0=; b=IGOe0CbScQrwGTyB06D3HoMBUwzzPXjSgXpwudOJMVYB5m2Trr8HStQEeRysudOzLe9x8a/mSdujicGBlw7jC1O5h8k8f4dsm28HHVBdK3dSpnGvc6fGfp16cIcbhO3fs0aTwkXHEnno0JSuPNz73NL5nIBbG6RC8Q/PZ8OVb7s= Received: from BLUPR0301CA0010.namprd03.prod.outlook.com (10.162.113.148) by MWHPR03MB3166.namprd03.prod.outlook.com (10.174.174.35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1178.14; Tue, 20 Jun 2017 17:16:31 +0000 Received: from BY2FFO11FD028.protection.gbl (2a01:111:f400:7c0c::120) by BLUPR0301CA0010.outlook.office365.com (2a01:111:e400:5259::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1199.15 via Frontend Transport; Tue, 20 Jun 2017 17:16:31 +0000 Authentication-Results: spf=pass (sender IP is 23.103.226.20) smtp.mailfrom=ca.weatherford.com; freebsd.org; dkim=none (message not signed) header.d=none;freebsd.org; dmarc=bestguesspass action=none header.from=ca.weatherford.com; Received-SPF: Pass (protection.outlook.com: domain of ca.weatherford.com designates 23.103.226.20 as permitted sender) receiver=protection.outlook.com; client-ip=23.103.226.20; helo=032-smtp-out.weatherford.com; Received: from 032-smtp-out.weatherford.com (23.103.226.20) by BY2FFO11FD028.mail.protection.outlook.com (10.1.15.217) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1157.12 via Frontend Transport; Tue, 20 Jun 2017 17:16:30 +0000 Received: from DM2PR58MB013.032d.mgd.msft.net (141.251.110.81) by DM2PR58MB015.032d.mgd.msft.net (141.251.110.83) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1178.14; Tue, 20 Jun 2017 17:16:29 +0000 Received: from DM2PR58MB013.032d.mgd.msft.net ([141.251.110.81]) by DM2PR58MB013.032d.mgd.msft.net ([141.251.110.81]) with mapi id 15.01.1178.018; Tue, 20 Jun 2017 17:16:29 +0000 From: "Caza, Aaron" To: "freebsd-fs@freebsd.org" Subject: RE: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs Thread-Topic: RE: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs Thread-Index: AdLp6PSQo+GaVjnpTieZ/TkgLGTt0Q== Date: Tue, 20 Jun 2017 17:16:29 +0000 Message-ID: <36ec1fb476e647a78c19463eea493859@DM2PR58MB013.032d.mgd.msft.net> Accept-Language: en-CA, en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: x-originating-ip: [141.251.205.196] x-ms-publictraffictype: Email X-MS-TrafficTypeDiagnostic: DM2PR58MB015:|MWHPR03MB3166: X-MS-Office365-Filtering-Correlation-Id: 6c2d23ae-64f3-4235-aec1-08d4b800181e MIME-Version: 1.0 X-OrganizationHeadersPreserved: DM2PR58MB015.032d.mgd.msft.net X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: =?us-ascii?Q?CIP:23.103.226.20; IPV:NLI; CTRY:; EFV:NLI; SFV:NSPM; SFS:(100090?= =?us-ascii?Q?20)(39850400002)(39400400002)(39840400002)(39450400003)(3986?= =?us-ascii?Q?0400002)(39410400002)(2980300002)(438002)(199003)(189002)(37?= =?us-ascii?Q?7454003)(13464003)(24454002)(9170700003)(45080400002)(773600?= =?us-ascii?Q?2)(102836003)(356003)(3846002)(6116002)(189998001)(108616004?= =?us-ascii?Q?)(260700001)(106466001)(790700001)(19618635001)(72206003)(47?= =?us-ascii?Q?8600001)(966005)(1680700002)(15974865002)(5890100001)(247360?= =?us-ascii?Q?03)(84326002)(2900100001)(2501003)(86146001)(861006)(6692600?= =?us-ascii?Q?2)(86362001)(7696004)(54356999)(22756006)(5660300001)(678660?= =?us-ascii?Q?02)(6916009)(7906003)(5640700003)(606005)(99936001)(733005)(?= =?us-ascii?Q?54896002)(55016002)(54556002)(38730400002)(8936002)(6306002)?= =?us-ascii?Q?(236005)(512954002)(8676002)(81166006)(9686003)(53946003)(11?= =?us-ascii?Q?0136004)(53936002)(6246003)(2351001)(2906002)(53546009)(6959?= =?us-ascii?Q?6002)(42882006)(229853002)(66066001)(33646002)(50986999)(562?= =?us-ascii?Q?774006)(299355004);DIR:OUT;SFP:1101;SCL:1;SRVR:MWHPR03MB3166?= =?us-ascii?Q?;H:032-smtp-out.weatherford.com;FPR:;SPF:Pass;MLV:ovrnspm;A:?= =?us-ascii?Q?0;MX:1;PTR:InfoDomainNonexistent;LANG:en;?= X-Microsoft-Exchange-Diagnostics: 1; BY2FFO11FD028; 1:daGLGdk0yxliVuXU9ExqUlBf9qZwXnrUa8XrarlwQzW8MFIlwJODFAzHAnkr22suGxnMTjX20Y8mLUbNdgLCD55Bz53IZvYwrxRxKD6R8in4HOukNuEmaHnsjIgLozZtxNL0B/l/zJ+ZG6b8X1gPgMrs41vPKnyx2sk+YLpkNzrnISS+uFdwvl1oaZ0jy12me/446jHNWDXQD9OBgHLv2qh+I/wZMzO8VxBQmL3HCOBezeg1u+5LrKpjoH0NUVpab2q0jofPRTkzyHTgxSS6Gq5LccWlvwoNOW1aK6rpWNzMAbXchzNg6zImNMvSy2/wAWVQSiEjwp7Qt1tNT1wOGmbwWAk0aVa9jpBvPG/gprYZc2aXuP4hX2YaQEJXMRC4TBa57HUA9LwyKEyB2BMrIE5eSpeVgdJNEpNcDUaMS4lmWOiIuXP+6K/OdUycqX6WOIHC4iMZqzJalPYp6hnzU1XiQqwRxrM16tM0e4ny5DuV4lAg+xyw4Y5zs2sg7l5v666d14TYQIjKOBx6uhLd5p7LOlmvWXvEujfE/U1P/6kwrBGorQ14mkYrNbmPei0UiZ3/OpNeBke/6FLNwRsdZWESzyVrv6EMvkb46biibvZpyaz3q625LOrmaK/EZ8hA+zv+Txt54TCbgkf60qBQBRwVEeWyJ7GaIMb5hlcX789sxTAPqEnIdWZlESi78UgqQ55I8GfzN7GBfUKSsMDJ9IC/ezRMdFEz0Kz3ya24W96zGAtNYxkqzd2Obo0KtJhXrAtKNEP3PGh8h7MjijRKxV0ry4TAsgMC2PNrcUAV4YQ+806Bwz32wGEB2oaL1kDL8RxZy+DkKwDi6Gx4Q/sK0N36Q99Vla9ZQtP0KuxPJI2V8nmetbx3/mwaBs/7TZdZ X-CrossPremisesHeadersPromoted: BY2FFO11FD028.protection.gbl X-CrossPremisesHeadersFiltered: BY2FFO11FD028.protection.gbl X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(300000500055)(300135000095)(300000501055)(300135300095)(300000502055)(300135100095)(22001)(8251501002)(2017030254075)(300000503055)(300135400095)(49563074)(201703131423075)(201703031133081)(201702281549075)(300000504055)(300135200095)(300000505055)(300135600095)(300000506048)(300135500095); SRVR:MWHPR03MB3166; X-Microsoft-Exchange-Diagnostics: 1; MWHPR03MB3166; 3:999/iWb2t1u4OBhtvd4my13GP5nQx7PeXZS+z2oPnXjfES5m2sdf/q1J3CLy8b2zgC9jzDWu1Kifp7r7WErgHvf3IONBdA6b7v6UgChSgm2l9rwcq0L6rOqPm5fJ8VBl3GYQHhyguqeL4Cngz7vPgm9CmisNvA0kzuim8YHWB7fDI8g4F5aIXXOFZ+B34iA+XYWyq3ZX38Sp0CpQXZI53bTUPSVBFLoc20e7vSWrxVJvt0DNQ2Tq0PxRiFWh6DWu5G9YFB9ru6WjyTsFHZvtM65Flo2Acy1SKpd9TkwMEY6JxV6ELvNuAcaTP0zH3Tc12kyQf6SJe169OiqFCIoRTCMdRrLjo0+0mlxRcdzBPvtjyZWG7UoirpinHcjd0V3TkpZfs+L+EXkE2LHEqkI0XSX9HZLwTZ+3h2UG+4Co6j1VxDWedzOShVWiiyow05KUaBUVVA2pysF+rdeBlM7/NAatB+Y9MA3pVaWdpFbnowSv6dJ4zjNdM/xd+S0pGqZpXoeNSJWsWeSGoSdfKXPLQfX5pG4CPVpDTwJIeUb5Qxc6Q39tBzNhkmpbGrMWP8N9j00obF2KIKBhdd+DUK9DJFRS/Rm4ojMZq/vY+oQaADpQhF3OWSNGOUmGKmUi1/rrkXYIJuxVMnDM8tQgBgN0Z+KmtXmq+xsZgoY3yggxCc/Br9/pT6I9rtFTp4+A7ColfG+BBEBBY/7r2Zd7sf7bxrMwvdF0tLpn9Dp6OWpgVAP8GRA35jzhEjTfljMaEkQk9cS29UvGzpN5n92kQoFaUuWGJ21zRluue3X/yS/Utsm5mlUx5INRPhZF+QbX1cbMU8CBXWWXa7h4m+NdSlZXnKjMu3puaYwBUCjTowAOZv/DNI4xVqTkoMfAarYVIkDmIVb+UIyw1EP1oiprAlBkq9O4NUIPKbvjRZrRSg1KOhzkspClmoLP6keAV+KMJeh2 X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; MWHPR03MB3166; 25:WYi1Bs1uQxzg8v1t69IYQaLRqNQ+OdHtlQmKp4UOt?= =?us-ascii?Q?b/+jnIGnH0uxt7Q/9jwQAiYvKiZx2uVxO+72cac2rd7G3sSwZZdrV/lDNFWv?= =?us-ascii?Q?FWZTg5juIkjat1a0y215oVaHvzv7HwaEg5iOaxGeVXTjA7ldIQf7qTqwP4vq?= =?us-ascii?Q?5q6LSAiKWFJ3sKzS5tDwCfV//DfYl6phD37Bw+fAYNEljkCqxyjxpaxvTYkr?= =?us-ascii?Q?qG96C1+kOkzwobAgFv45AhtkkfWBhPyJPPhqijiuy5VcpbJl3Lga16dd1g1f?= =?us-ascii?Q?my5XIujNe+QoibnjvIk009LJKV3WhUrxAjHlH+dmVt5E2Fr2zd+WtHRVrsLD?= =?us-ascii?Q?fFfa+KW+fcd1/yZArWKVuj+m3C5ZNNFchESm4HPKlbe3x2gKJvsQeu4TuhNE?= =?us-ascii?Q?jWXLCnEm0C5YIhb5M9MPGwc7qwTNrX+lb4MLA6/TFmLIMwavGFBbYz9L6OHM?= =?us-ascii?Q?9IX5+Kry6eNwW19L2MogHxXoZ6vNaL4yEZYpDPm7yQgZK1TUqnzZNwFQXKIJ?= =?us-ascii?Q?5AIlA4FO2KCxT3LW+MENclMbPprHDfMMRDxle0/6oM4ZDXz8OVWGa6fXii9m?= =?us-ascii?Q?x2x41zzlhfHyOS9YNcW2t86WAMkD+axOiAFVT0V3J6Bc5Zfo7nNjSEJGa0jS?= =?us-ascii?Q?WAbAsPYxTR48z6ywiy1AffQqbEsXXOHSejfitOUaoMls3Vg8M3rZqgxE1It4?= =?us-ascii?Q?O/Heh4O/OtMUhqgegEdRgPqf5WO+puSKZgwmb11wlTx4SLnSDkGGN1SsQdsG?= =?us-ascii?Q?iKPXNbGc2AYqYZ8StkXaIlkTEPABZ5FOlMCRnpxHDOEq0X9dNycG23modcnj?= =?us-ascii?Q?NSn+n1R1nSLRi1M3HQGO2gJg6xcSLgEBDp22050uww050gvxr50F4TdXkRtE?= =?us-ascii?Q?IWT+Qury8SWFEGzOkOo0T7cIOW1tDunzxA15jvY7Phwk1DZSIIGVb5mFvS3r?= =?us-ascii?Q?+bKmZfEmEK1ZV57N0J/2k1XSJzqUC3UDJyhIc1cGg=3D=3D?= X-Microsoft-Exchange-Diagnostics: 1; MWHPR03MB3166; 31:KXOCy+WxsgPROjhD+9pEN3VzM5GY8fRgDBWUmfYHOZZfCqI7mYln9DFNgY0CbviqqX+saSnmyslSVZk+RqpeqPSE3MUUo44gb4JTxhsHE159iZt3SfwGEQKWB5udVt4uKcEGKJdiNBw0DAjG1snkvm2Z7SPciH9U1sFx1mUTpYpS+k4CyIvW6sdRYsPMLxMmIpTN9eBTA6RLviqGTRO0FlP0yszzbCIpZz2F0Q6iAvobEiJ/SdMbDbxE0IeonCirMsA6AoR18rEXldxupjmg2a5Vq6K6YflvcQQkCagkljpeugHx/hydkX95JkoFMHjhtzmqrpvssQdpO5SFECqiGsgON9NzIOk3UYYQA70lLge9BjpOuzbAMyruhUKhK9U5IJzuvjBK2Rf5LcnVHzgmfL/xoFoDhDyeuJhjPHwZBQ7XRCUxlCxfSPDr1FRjwEqebAHZAmyPO+Elxtc2UOlZpoixud0ZTjXN+A1gtE2R794GPOQpmOoUSuDuTEvO5gNIigSsQIS2fzXav9hrkKoojaj48Sej33xCyi7Q/djzTL66U1QPya2c4X5eLH6u9eXttNd6fjJS2k/nXKtDuu6CkK+Y0PS4k0XU9p58IQcIjpeVA/1VPQ42kLNVXcfGypHpA7zrBTMsnejAMsOjPvDNi5Fe96jcYvqMJeh2Av9MFtHWkfZRY7pHuVDyDltwMo2zmCcIh0voU0WfbDFOYCqvIFAqDfcr+L1Q79xwB3i+T9UjirUX9NbdLt8cnar/JyCH X-Microsoft-Exchange-Diagnostics: 1; MWHPR03MB3166; 20:x0IY2DWHFZ0k4aCMJdIGqfzHXHMxyJBcdDblqBePUgSar9O1EjDK/yfjZeIKki2xYf1fLIw7or7EwOT3KIrXnmbegxY6j8poVGANjqwznzRLUc8pg2122MkvGVlUm5ch6jNh2ykEw8e2jehTYlB3MP8Obh1unr5S38oFWEdgiTvJkgM4Pv7vW+nVuTTlHXkFrBMO14pZqtkASFv9ja58jkgPuqVS20K4WXa9vI14SMdyumHlmHEfU0UNHHENtKct1wGg3HN3zkztYlD/gHMqE7VTXKvzpVzhnP4kXLiXHqxoJspwijV0Z5dCipxy/gMrXZWwjKL9SCRSaFFYkQh79U0ThEW1pARrcZUdd8svLzYq3lmDbnzeKaib9YQ7GpdrUktsGUU4D/+vc0U6W5v5kdoU5mwWkL+w4hqkqgib2HTc/TIfhnD7kSjkS3akOwBZLyTcN2gXabv8mZ6P/cKkuY77ShNK7KyPDwwFQ3kbXAXBxulpxdyf6PnT6OzoIzsL X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(158342451672863)(84480959824636)(209352067349851)(192374486261705)(31418570063057)(59266950573212)(128460861657000)(254730959083279)(86561027422486)(21748063052155); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(102415395)(11241501159)(6040450)(601004)(2401047)(13018025)(8121501046)(5005006)(13016025)(10201501046)(100000703101)(100105400095)(93006095)(93004095)(3002001)(6055026)(6041248)(20161123562025)(20161123564025)(20161123558100)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123560025)(20161123555025)(6072148)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:MWHPR03MB3166; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:MWHPR03MB3166; X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; MWHPR03MB3166; 4:/+PaV3mKbP69u3VkD9HUjx59AzUViaX/Z0D8m8aq/U?= =?us-ascii?Q?YnHInMzRpfQAKfexK/uo+J7dtaA22ByuIKMOoQbHq/SNzplRIR5OEh91hMox?= =?us-ascii?Q?WFJACVWDbqobre/Rp7KKfHLhUBdhmJvEMl3+UMTsEweQtKophc17g3TzWUjw?= =?us-ascii?Q?dmeeYxInhzmWCbEAyAErVSWkAKBNS5UE9LBOebc94TfRNjVEMRBSRSCUspyp?= =?us-ascii?Q?gwyq1or5pfYEEiZHOXODn4rSIE1UczjGFUgqLYAT7S/C2veonTiknDTLAC7L?= =?us-ascii?Q?mccHgol6isIhmcBEKCajnPZ2igXiucFmQnhDKawkVpGuqLb6An6zmCHxlqHX?= =?us-ascii?Q?KuezPPfSzjmn3kQ33z9fy3MWaFndV/FRbSVZUpGxZztqO5uJcpD+pOPjz3xq?= =?us-ascii?Q?JmlPfoAVMSR2Xz9mmuK7mxDE8rADzl4eWfxSoE2fHu8BYA9s8aoRS0b9jnW7?= =?us-ascii?Q?TxZJWI9sHRVqJZZ79ags+lar+K8uyyg6yVliWm0Ddkvf25xWmDkLrgfpArf9?= =?us-ascii?Q?ytDZ4z1a/9Yq6fp0DISth3limYr4OF6IYmGEd197ZSn4gx7Cm9Jchm2Cdvdx?= =?us-ascii?Q?WaI5JbNG9SVnYAGvh90ovkIlq7aC0UAVx6dzTbOm4DhBuJ2t/sKwh3/HgT5h?= =?us-ascii?Q?WRtXNi5dAtqrItb0bs4MG4Y7j+J65Zxk0e3rrWUA5QuWXRJQKh2ii1KEF8jM?= =?us-ascii?Q?n0BerccdGMsLAof0fo4Ff7NVnw6y/d1eeWfvDGhBJ5wDesFl9B3aLOO1cgH3?= =?us-ascii?Q?TWJ8Jp8dXGebKI3XDGEyQW9IbQqLMiDQe61Knt9Bu56RrsSbeqVP5fdCGRin?= =?us-ascii?Q?Seql23kG0Nv4zFCPwd8jKUS1xvAtUodlSvI8wG4xvWTfcGIL5QaNX3pGBg9R?= =?us-ascii?Q?XqTWqAIIuJLKjvcMCGEaOVPy+BeSTIQ45nDrSG2JVUXHlGaAnS0+C1SDzUjK?= =?us-ascii?Q?CpslktEi3t6BArOXxa+AQuOZZVLJgxxSi8VS3f+ZKQJLPo4w2orOipLT+s4s?= =?us-ascii?Q?4Y/praU3HYYebz57wgILTAXxWG+7dSGUpBpSE3UPk02oSAcybiqW+4Ud0orw?= =?us-ascii?Q?5SAXb6bHS5xp0q3ae9A86bcqa2ubHXC1vkZMtn+2PRVeCvGjAHZaL2ZwatBg?= =?us-ascii?Q?6CCq9sz3Y++DmV7N205/6WcLmo78dvrOECEGWVlNH6gwU3a6BjCLSZhM3Vyg?= =?us-ascii?Q?4Lkq9VA1sO+aC+OMp1RA1v8QdZLiO3bDUKASE/Ff2LSjYMj1X3JEFvNc7e41?= =?us-ascii?Q?BRQdCX4ns6tVZDY6oXVOepP6A5iRCHLZOgImyjVunGXl9J5BrJa9paVCS3pk?= =?us-ascii?Q?wWMtn0yDZHsbEOam1ROdYLfQIDV6CLCrw72m8Zsd87lhiFeM5d//SVSizEZz?= =?us-ascii?Q?clXihfuAPgniQ5BDoYr4ATsnts3tTX6xpF1HJF3iUA3OlnERkeMIfIJQ7/0v?= =?us-ascii?Q?bURfOd/IMDSfDtp/MIc+cYKT4jNPOeyMavc21oct/GXOhcr8JDleZ2A3BTCL?= =?us-ascii?Q?A/wZy1Dmqujmy902uhjOd5BPYmeEYqRynNGdjtpmHUFklcGk5dk68+lBRY2m?= =?us-ascii?Q?QzIV9n2l5gQpaCS96RwSemcTKftemMM7MXCcWZiiMCmb9kd918y3aeJFjtFQ?= =?us-ascii?Q?uVxRHZ5k++rvhuJxBg2g=3D=3D?= X-Forefront-PRVS: 03449D5DD1 X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; MWHPR03MB3166; 23:keRT6SWjoQenjVGt4oSBTxCjlvp986sYl+eMTuTG+?= =?us-ascii?Q?Mp1EeazzUjVW77maj4sUrfoBpuMtk0uYfeDRJ1g2yw8bdlHoATy1pyZaH8Bn?= =?us-ascii?Q?enfu8L88gyGE3QzIRjYvIlxQJbm9wM/+ANI2+Kftlgt1vIZlDAfF5j6BKr5N?= =?us-ascii?Q?YVvnKOTgauSP2th3SbIrj+Ufa/h1QNU3REqhYT95OCotghM0aZpRWgRjS34z?= =?us-ascii?Q?vjQ669dazk6SKg8Z9Yn7ELV1RWPnGpo8EiQAoXLhpc28Q0kYkyDw0YeFww/Z?= =?us-ascii?Q?ep3CHerL8tNyYyorSDfbcKbolEcmNVPMbMlh2X2ql8hyO50lo4KguwsXLzz9?= =?us-ascii?Q?XKvHYhWocVGNfkkEa49gGvdlGRPVcyufNJif3ZpjZYYW2LjtWsOiYuuSQ79T?= =?us-ascii?Q?/itlbO+qfMvB1hxE05TwvSZNKGIc+59Oot7GtUayk5KL9sm89QM9jx6eroGp?= =?us-ascii?Q?7kKemg7F7E4bQSYHz4fmhewbgtOoS1UmiLyzbwS376O9d9VYfay0h6/RIMTF?= =?us-ascii?Q?fwp9KWMpSJsbR784bcSFROaTw4zHfPTSCIzFj41hobDgFOuqu02lQrU7kynv?= =?us-ascii?Q?SPGMnD20QebY1OPWfKibnmlA+DUR4HMMWKtUqVnK5QaAshgf8F00k5HYsPCG?= =?us-ascii?Q?CLh5NcFN24MjKgQeEyc41XnlO6TDYy4xlXMkRjS6bK0B7wV8WmPpgL2AesKa?= =?us-ascii?Q?nuZQUYrDgAZJINFY5ckBZx3uZiTqtMEI3eIWqRNIIDRd+M30TNmP//mHXbhb?= =?us-ascii?Q?SnNiGhU0CSUbVJGvmKqLZEC7+OzVCQcTAivZRDHFAJ8ehmZLam01mzWoLze2?= =?us-ascii?Q?uii95cKvqy45jP/YOcQ7fhWl4G5HxuaFpddhuQMEQ6aWC8RyA6GU61urbYfx?= =?us-ascii?Q?ZM2vgep+LAdxf6zORWPhHl5y0bpZoO8bP5OQjzJg2LEWNqJMMri8amfxkmj8?= =?us-ascii?Q?JDrO54usRBa3PSUQhAnGrA8qcdQk0FbmPShHHWsBuS+4YO9FxY7MEut9ikKQ?= =?us-ascii?Q?iK5kVTzyqW21G0JZKhYhvI5aMkp0UJaxeghZyJ1JyU/KAMwmJihQI9sAQtdp?= =?us-ascii?Q?uJxQKbq2zerR7lL0PjJn/MFrI+MQCgyny8rWO//l6/laR4GcM8yUysu1HDmx?= =?us-ascii?Q?L3C15H+GqIJVawKd5En8LrT23c3eyfMMZGg1jMo7NPJhTFuqBZhyY6OsSmD7?= =?us-ascii?Q?eOU31EuwqVF3RIA8RsrO2tFOfuKmjMEZeKP626dTQFeLmeR98rP81gSceUsg?= =?us-ascii?Q?m192kiNO5wYPcldMLrE0OgBqAn9FL8CkKJy0XmoKRXAeA1X7xbTeZbWGhATb?= =?us-ascii?Q?quc41rpW1RowzVH5W21YvNLQXo/Cd44t3L5XHs0iMjUnqhNViCaUa3nl+G4F?= =?us-ascii?Q?wezjCtQwr2skDTpFyJkpg1ocFJuqlqh7XKQs+MG66iomgN0jacaAPvHsZmOI?= =?us-ascii?Q?7ZOLUfCFH/lBNx7NuOq/I7ainYcj/+6IyVBEgkWiLcmMF0jJO6Fu7nK51ytC?= =?us-ascii?Q?W9qpGLaAujFh6z8IPLH2QCTuzdkOLwvvVXytzEfvwTRpkxQnN2SOBc1jkIpQ?= =?us-ascii?Q?jxvZ0eIUzj0F3NQks26PtA7on7164jEiLPO3D2agwTkE0HZbVnLYbknDbke5?= =?us-ascii?Q?67KUTMwOQG9+MNkM5AV7AOA52P5yEzV9C7PAb/yux4OkCNgj8uV9BrO6y+kb?= =?us-ascii?Q?pvqOZ4JbbNWzMWdkWs0n8MjEAiZRmmSgwA6u46xMUu/4Lp+96g9BG/9kDQjw?= =?us-ascii?Q?c5dP1Hf4dvJQ7e07AvUVs6EOCoaXdip47LBUQjBjsUcWWaxRuz9Wyiuzu1WR?= =?us-ascii?Q?3Vu6nJ1sPyeWZooS703u+upOFr2Bf9D2g=3D=3D?= X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; MWHPR03MB3166; 6:sZU1vRPVJcl1hf8T+ZOnEay5ScVKcCE+RtQ9USmLRo?= =?us-ascii?Q?ZB5YtsB+f+6YLv0IVRYCyMzknITR+MwrJt6buWuOo3jUww8fOLknajT5GD1c?= =?us-ascii?Q?KjLgnIEq8VmCBMN/W61QAQkZjTtdnRWDjuqk+VCqOupy2pXpWo2L9FlJhSYo?= =?us-ascii?Q?9rltGbIBn2iIxFxWQpe5aubSkIt9GBIXqbdo6P4cH6FHdYyfj1wgZo4gA3dr?= =?us-ascii?Q?RZ079jc7q6vXcR6OXmfyJMzvz4odv2n1EI6ZY3ntOq+l5PRnRT+pn9KqxOOT?= =?us-ascii?Q?z6x1hw42Rb8aMRnsbt4cMizxg5/WUdh8xbnnjdXXBGRetiAxD81WcSUc5jcd?= =?us-ascii?Q?JtiiLY64BegrlEIC4BbcViUp2bnG2k3cPlHu3RWNemdEZFlXl//z1thCqEa/?= =?us-ascii?Q?LUlZPLfm1M4odtczUdtsmb3rLleGKweNno60yYm3P4hWHgh4wyLDQZV+POXC?= =?us-ascii?Q?SCNOnDmcbE7H4EhoxU5nIZtXXzuJcIOpH7cxzFkyE98My/gWrlE7keaeX/M3?= =?us-ascii?Q?73Hqjm+ZsCpEvIkGDFcfKjqgPuRZ1sfHekwtkk6KQAKmZ2KbYz2EtVcmiGrF?= =?us-ascii?Q?rlQOS8uXjjk028sRYTtodjhb5JO5GyTXtcIE2PNOygH0FtT0xuxhCVgKjD4L?= =?us-ascii?Q?esrWCe6svZ7lueRjxIn45G1QZ1qn7I9CUbQnAbNEg8UofIFcLy8vfDMuutZW?= =?us-ascii?Q?q0tbP84GDEz9QQQaFVr95DKS+cKlxYUA6vAtntzbKi+iXLo6SgUGvUJ/QLxe?= =?us-ascii?Q?IrGfjxflWBggPp9umj3R3WTcrcao7vWfbt8tnctH9dXPhzZxglEb3n+F0YCX?= =?us-ascii?Q?yH8LYcQaysM9W7C0sLHsJQDC5Z8tXHObA/K77IoD0dmmibPJW9/8uhCV8Jt5?= =?us-ascii?Q?uVFfBnsSYrCpVQ3yIQaejARPIyFRhQRMP6eVkZzDX1xJUwc0JUG1vav4lSYR?= =?us-ascii?Q?CZonXtZp0PRWGN+7dKiQYW3T5LfiOe65/72Me1iBSOPuElESRHMwde1JdMfs?= =?us-ascii?Q?WZT/qBmhMgYaWsZuxHaebm?= X-Microsoft-Exchange-Diagnostics: 1; MWHPR03MB3166; 5:cYVwUNY/eiNe2Mh4tHTAaAhoP91ftXQuv0yykhJpQeJzKzDqz1G57HEjPgS6YOEAxyB2AqZFito8YBYCrAO42QB+EiYCGovgag0za5uXIXX8WyiwqrAqSCME1bhI9+NWflle1wGHaIRGJZkgEi2Zi2m6SSwK9MbFxIwNQDz9ALNBHFsp7kzhJGLtad2+rRvjYCDtEIkolj6PcbJWoq+soS6Ojk9EgLXXb3bfz2omhPYq1izrSAO4dGUaEP4Z7EGwlNJYLUTlvNBNJ7gh7iUM51C93ZW6/D4/3L8AQxrT0qUTofalgzSGG43f8Nc2ZOX98BzeC14hyjCDiyC6edOf26UKHXxgdsOQuO2gv+K4w2znj4E0u11bXQHfHVhpq2bOImUIifihVLluU7Ai/gEtcBrwPL4u4OUyn0A2KmibC5YQiSLlr0o83RMR4Nkgh7OkHdWn5+yAygQ2PGUMPSdQBB82/V2aeNUNKoUWP65YjRjL1Pfls97fhkx4FaZM22S1; 24:+w431Io4i0FNMPWAC2BaU1ptsQbGSKv5qQao4K4YnMJw7ckeJBpbUE/GzSo6O3uHWTFX1uqjgqapqAsx/MnPhgTNfeTa8rVHUtRtTHpoq1o= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; MWHPR03MB3166; 7:Wb2l//aVe5wO3Gjta0+7rvbpG8/p/wfQPaibrtJ5RvzQJIo7Vd9oBzdNODBFFypccNfLZR6ZdVP5Aw4raroc/X43zdTG06TnV0BOokWDGkMQcpNHmOXY5ZeOrMAWagJP17s6P4iTMWB8fVm+SR3ZCfGk2XNa3KCr7tk8F9HUvIbWkaMIp79uKIUOPkXHY6m04MOLoiiU9VcCjIrRve3sAEOCV+NouUbuUZfUEwvkhNgyotfdlg7C5FoayEPH9+5vebc/1ve8RE1XDQqfhd9X9wKCC9JA5fPsEVp6opkFridlff6CVNlxiidlNBxwllcF8ntxx5I+3uHTpsWKMdDygTn/c76SCyDUy+LGhQS1CneSeuFySnidkyCqQod3kU4HomVTHWk4R5bqsgOKIYgZXlhUWQt7KsCSd/vN61NdRbQ4Dui79avCdnRLj/VQvkbwEMAyXkqT/wXfPYjkCxY3m8gmBBGusCFUWKAkw3zx0sl99V+8lkvViq9SM1Ym3kpoFFQs+Kkki07s+bFpkPGDspeYW07g7KwxCA+UO6Brng32NuXlYG4BIaWmT5fif+YbNEIuIpC411XQNLKeiOj8blJbF/v5GzgWnNkWz3og3mdUlVXG7fG8Yo7JK8vZYHayhwivMlpMT7k8Wcuu7GsMAsBmGPLuBDhiv4Sqbv+rV0xQvtlXL3vibeKUwsiOrc7MM+Hp2R+dTv9055kT2/OJeOSstforjMIy54yTBoHqU2TvEAHbXAhwFRJaCG358g3yuHZ+JaggiuUye9lpo4hYpneTgqMH4qBWK5uMr5xwdO8= X-OriginatorOrg: ca.weatherford.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Jun 2017 17:16:30.9321 (UTC) X-MS-Exchange-CrossTenant-Id: dd63fb60-07f6-4d96-8d40-ebeca61a524e X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=dd63fb60-07f6-4d96-8d40-ebeca61a524e; Ip=[23.103.226.20]; Helo=[032-smtp-out.weatherford.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR03MB3166 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Jun 2017 17:16:36 -0000 > -----Original Message----- > From: Steven Hartland [mailto:killing@multiplay.co.uk] > Sent: Monday, June 19, 2017 7:32 PM > To: freebsd-fs@freebsd.org > Subject: Re: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs > > On 20/06/2017 01:57, Caza, Aaron wrote: > >> vfs.zfs.min_auto_ashift is a sysctl only its not a tuneable, so settin= g it in /boot/loader.conf won't have any effect. > >> > >> There's no need for it to be a tuneable as it only effects vdevs when = they are created, which an only be done once the system is running. > >> > > The bsdinstall install script itself set vfs.zfs.min_auto_shift=3D12 in= /boot/loader.conf yet, as you say, this doesn't do anything. As a user, t= his is a bit confusing to see it in /boot/loader.conf but do a 'sysctl -a |= grep min_auto_ashift' and see 'vfs.zfs.min_auto_ashift: 9' so felt it was = worth mentioning. > Absolutely, patch is in review here: > https://reviews.freebsd.org/D11278 Thanks for taking care of this Steve - appreciated. > > > >> You don't explain why you believe there is degrading performance? > > As I related in my post, my previous FreeBSD 11-Stable setup using this= same hardware, I was seeing 950MB/s after bootup. I've been posting to th= e freebsd-hackers list, but have moved to freebsd-fs list as this seemingly= has something to do with FreeBSD+ZFS behavior and user Jov had previously = cross-posted to this list for me: > > https://docs.freebsd.org/cgi/getmsg.cgi?fetch=3D2905+0+archive/2017/fr > > ee > > bsd-fs/20170618.freebsd-fs > > > > I've been using FreeBSD+ZFS ever since FreeBSD 9.0, admittedly, with a = different zpool layout which is essentially as follows: > > adaXp1 - gptboot loader > > adaXp2 - 1GB UFS partition > > adaXp3 - UFS with UUID labeled partition hosting a GEOM ELI > > layer using NULL encryption to emulate 4k sectors (done before > > ashift was an > > option) > > > > So, adaXp3 would show up as something like the following: > > > > /dev/gpt/b62feb20-554b-11e7-989b-000bab332ee8 > > /dev/gpt/b62feb20-554b-11e7-989b-000bab332ee8.eli > > > > Then, the zpool mirrored pair would be something like the following: > > > > pool: wwbase > > state: ONLINE > > scan: none requested > > config: > > > > NAME STATE RE= AD WRITE CKSUM > > wwbase ONLINE = 0 0 0 > > mirror-0 ONLINE = 0 0 0 > > gpt/b62feb20-554b-11e7-989b-000bab332ee8.eli ONLINE = 0 0 0 > > gpt/4c596d40-554c-11e7-beb1-002590766b41.eli ONLINE = 0 0 0 > > > > Using the above zpool configuration on this same hardware on FreeBSD > > 11-Stable, I was seeing read speeds of 950MB/s using dd (dd > > if=3D/testdb/test of=3D/dev/null bs=3D1m). However, after anywhere fro= m 5 > > to 24 hours, performance would degrade down to less than 100MB/s for > > unknown reasons - server was essentially idle so it's a > mystery to me why this occurs. I'm seeing this behavior on FreeBSD > 10.3R amd64 up through FreeBSD11.0 Stable. As I wasn't making any headwa= y in resolving this, I opted today to use the FreeBSD11.1 Beta 2 memstick i= mage to create a basic FreeBSD 11.1 Beta 2 amd64 Auto(ZFS) installation to = see if this would resolve the original issue I was having as I would be us= ing ZFS-on-root and vfs.zfs.min_auto_ashift=3D12 instead of my own emulatio= n as described above. However, instead of seeing the 950MB/s that I expect= ed - which it what I see it with my alternative emulation - I'm seeing 450M= B/s. I've yet to determine if this zpool setup as done by the bsdinstall s= cript > will suffer from the original performance degradation I observed. > > > >> What is the exact dd command your running as that can have a huge impa= ct on performance. > > dd if=3D/testdb/test of=3D/dev/null bs=3D1m > > > > Note that file /testdb/test is 16GB, twice the size of ram available in= this system. The /testdb directory is a ZFS file system with recordsize= =3D8k, chosen as ultimately it's intended to host a PostgreSQL database whi= ch uses an 8k page size. > > > > My understanding is that a ZFS mirrored pool with two drives can read f= rom both drives at the same time hence double the speed. This is what I've= actually observed ever since I first started using this in FreeBSD 9.0 wit= h the GEOM ELI 4k sector emulation. This is actually my first time using F= reeBSD's native installer's Auto(ZFS) setup > with 4k sectors emulated usin= g vfs.zfs.min_auto_ashift=3D12. As it's a ZFS mirrored pool, I still expec= ted it to be able to read at double-speed as it does with the GEOM ELI 4k s= ector emulation; however, it does not. > > >> On 19/06/2017 23:14, Caza, Aaron wrote: >>> I've been having a problem with FreeBSD ZFS SSD performance inexplicab= ly degrading after < 24 hours uptime as described in a separate e-mail thr= ead. In an effort to get down to basics, I've now performed a ZFS-on-Root = install of FreeBSD 11.1 Beta 2 amd64 using the default Auto(ZFS) install us= ing the default 4k sector emulation (vfs.zfs.min_auto_ashift=3D3D12) settin= g (no swap, not encrypted). >>> >>> Firstly, the vfs.zfs.min_auto_ashift=3D3D12 is set correctly in the /bo= ot=3D/loader.conf file, but doesn't appear to work because when I log in an= d do "systctl -a | grep min_auto_ashift" it's set to 9 and not 12 as expect= ed. I tried setting it to vfs.zfs.min_auto_ashift=3D3D"12" in /boot/loader= .conf but that didn't make any difference so I finally just added it to /et= c/sysctl.conf where it seems to work. So, something needs to be changed to= make this functionaly work correctly. >>> >>> Next, after reboot I was expecting somewhere in the neighborhood of 950= MB/s from the ZFS mirrored zpool of 2 Samsung 850 Pro 256GB SSDs that I'm u= sing as I was previously seeing this before with my previous FreeBSD 11-Sta= ble setup which, admittedly, is a different from the way the bsdinstall scr= ipt does it. However, I'm seeing half that on bootup. >>> >>> Performance result: >>> Starting 'dd' test of large file...please wait >>> 16000+0 records in >>> 16000+0 records out >>> 16777216000 bytes transferred in 37.407043 secs (448504207 >>> bytes/sec) > Can you show the output from gstat -pd during this DD please. dT: 1.001s w: 1.000s L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d = %busy Name 0 4318 4318 34865 0.0 0 0 0.0 0 0 0.0= 14.2| ada0 0 4402 4402 35213 0.0 0 0 0.0 0 0 0.0= 14.4| ada1 dT: 1.002s w: 1.000s L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d = %busy Name 1 4249 4249 34136 0.0 0 0 0.0 0 0 0.0= 14.1| ada0 0 4393 4393 35287 0.0 0 0 0.0 0 0 0.0= 14.5| ada1 Every now and again, I was seeing d/s hit, which I understand to be TRIM op= erations - it would briefly show 16 then go back to 0. test@f111beta2:~ # dd if=3D/testdb/test of=3D/dev/null bs=3D1m 16000+0 records in 16000+0 records out 16777216000 bytes transferred in 43.447343 secs (386150561 bytes/sec) test@= f111beta2:~ # uptime 9:54AM up 19:38, 2 users, load averages: 2.92, 1.01,= 0.44 root@f111beta2:~ # dd if=3D/testdb/test of=3D/dev/null bs=3D1m 16000+0 records in 16000+0 records out 16777216000 bytes transferred in 236.097011 secs (71060688 bytes/sec) test@= f111beta2:~ # uptime 10:36AM up 20:20, 2 users, load averages: 0.90, 0.62,= 0.36 As can be seen in the above 'dd' test results, I'm back to seeing the origi= nal issue I reported on freebsd-hackers - performance degrading after < 24 = hours of uptime going from ~386MB/sec to ~71MB/sec inexpicably - this serve= r isn't doing anything other than running this test hourly. Please note in the gstat -pd output above, this was after the performance d= egradation hit. Prior to this, I was seeing %busy of ~60%. In this partic= ular instance, the performance degradation hit ~20hrs into the test but I'v= e see it hit as soon as ~5hrs. Previously, Allan Jude had advised zfs.vfs.trim.enabled=3D0 to see if this = changed the behavior. I did this; however, it had no impact - but that was= when I was using the GEOM ELI 4k sector emulation and not the ashift 4k se= ctor emulation. The GEOM ELI 4k sector emulation does not appear to work w= ith TRIM operations as gstat -d in that case always stayed at 0 ops/s. I c= an try disabling trim, but did not want to reboot the server to restart the= test in case there was some additional info worth capturing. I have captured an hourly log that can be provided containing the initial d= msg, zpool status, zfs list, zfs get all along with an hourly capture of th= e results of running the above 'dd' test with associated zfs-stats -a and s= ysctl -a output though it's currently 2.8MB hence too large to post to this= list. Also, there seems to be a problem with my freebsd-fs subscription as I'm no= t getting e-mail notifications despite having submitted a subscription requ= est so apologies for my slow responses. -- Aaron Aaron Caza Senior Server Developer Weatherford SLS Canada R&D Group Weatherford | 1620 27 Ave NE | #124B | Calgary | AB | T2E 8W4 Direct +1 (403) 693-7773 Aaron.Caza@ca.weatherford.com | www.w= eatherford.com [cid:image001.jpg@01D27566.E8E4ABE0] [cid:image002.jpg@01D27566.E8E4ABE0] [cid:image003.jpg@01D27566.E8E4ABE0] [cid:image004.jpg@01D27566.E8E4ABE0] [cid:image005.jpg@01D27566.E8E4ABE0] This message may contain confidential and privileged information. If it ha= s been sent to you in error, please reply to advise the sender of the error= and then immediately delete it. If you are not the intended recipient, do= not read, copy, disclose or otherwise use this message. The sender discla= ims any liability for such unauthorized use. PLEASE NOTE that all incoming= e-mails sent to Weatherford e-mail accounts will be archived and may be sc= anned by us and/or by external service providers to detect and prevent thre= ats to our systems, investigate illegal or inappropriate behavior, and/or e= liminate unsolicited promotional e-mails("spam"). This process could resul= t in deletion of a legitimate e-mail before it is read by its intended reci= pient at our organization. Moreover, based on the scanning results, the fu= ll text of e-mails and attachments may be made available to Weatherford sec= urity and other personnel for review and appropriate action. If you have a= ny concerns about this process, please contact us at dataprivacy@weatherfor= d.com. This message may contain confidential and privileged information. If it has= been sent to you in error, please reply to advise the sender of the error = and then immediately delete it. If you are not the intended recipient, do n= ot read, copy, disclose or otherwise use this message. The sender disclaims= any liability for such unauthorized use. PLEASE NOTE that all incoming e-m= ails sent to Weatherford e-mail accounts will be archived and may be scanne= d by us and/or by external service providers to detect and prevent threats = to our systems, investigate illegal or inappropriate behavior, and/or elimi= nate unsolicited promotional e-mails (spam). This process could result in d= eletion of a legitimate e-mail before it is read by its intended recipient = at our organization. Moreover, based on the scanning results, the full text= of e-mails and attachments may be made available to Weatherford security a= nd other personnel for review and appropriate action. If you have any conce= rns about this process, please contact us at dataprivacy@weatherford.com.