From nobody Thu May 2 12:29:56 2024 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VVYD90nSJz5JxcJ for ; Thu, 2 May 2024 12:30:01 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [IPv6:2607:f3e0:0:1::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smarthost1.sentex.ca", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4VVYD84bntz44V6 for ; Thu, 2 May 2024 12:30:00 +0000 (UTC) (envelope-from mike@sentex.net) Authentication-Results: mx1.freebsd.org; none Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [199.212.134.19]) by smarthost1.sentex.ca (8.17.1/8.16.1) with ESMTPS id 442CTxu8030979 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=FAIL); Thu, 2 May 2024 08:29:59 -0400 (EDT) (envelope-from mike@sentex.net) Received: from [IPV6:2607:f3e0:0:4:bd4c:f42f:eabc:4f29] ([IPv6:2607:f3e0:0:4:bd4c:f42f:eabc:4f29]) by pyroxene2a.sentex.ca (8.18.1/8.15.2) with ESMTPS id 442CTv2N004147 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Thu, 2 May 2024 08:29:58 -0400 (EDT) (envelope-from mike@sentex.net) Message-ID: <77e203b3-c555-408b-9634-c452cb3a57ac@sentex.net> Date: Thu, 2 May 2024 08:29:56 -0400 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-stable@freebsd.org Sender: owner-freebsd-stable@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: how to tell if TRIM is working To: Matthew Grooms , stable@freebsd.org References: <5e1b5097-c1c0-4740-a491-63c709d01c25@sentex.net> <67721332-fa1d-4b3c-aa57-64594ad5d77a@shrew.net> Content-Language: en-US From: mike tancsa Autocrypt: addr=mike@sentex.net; keydata= xsBNBFywzOMBCACoNFpwi5MeyEREiCeHtbm6pZJI/HnO+wXdCAWtZkS49weOoVyUj5BEXRZP xflV2ib2hflX4nXqhenaNiia4iaZ9ft3I1ebd7GEbGnsWCvAnob5MvDZyStDAuRxPJK1ya/s +6rOvr+eQiXYNVvfBhrCfrtR/esSkitBGxhUkBjOti8QwzD71JVF5YaOjBAs7jZUKyLGj0kW yDg4jUndudWU7G2yc9GwpHJ9aRSUN8e/mWdIogK0v+QBHfv/dsI6zVB7YuxCC9Fx8WPwfhDH VZC4kdYCQWKXrm7yb4TiVdBh5kgvlO9q3js1yYdfR1x8mjK2bH2RSv4bV3zkNmsDCIxjABEB AAHNHW1pa2UgdGFuY3NhIDxtaWtlQHNlbnRleC5uZXQ+wsCOBBMBCAA4FiEEmuvCXT0aY6hs 4SbWeVOEFl5WrMgFAl+pQfkCGwMFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQeVOEFl5W rMiN6ggAk3H5vk8QnbvGbb4sinxZt/wDetgk0AOR9NRmtTnPaW+sIJEfGBOz47Xih+f7uWJS j+uvc9Ewn2Z7n8z3ZHJlLAByLVLtcNXGoRIGJ27tevfOaNqgJHBPbFOcXCBBFTx4MYMM4iAZ cDT5vsBTSaM36JZFtHZBKkuFEItbA/N8ZQSHKdTYMIA7A3OCLGbJBqloQ8SlW4MkTzKX4u7R yefAYQ0h20x9IqC5Ju8IsYRFacVZconT16KS81IBceO42vXTN0VexbVF2rZIx3v/NT75r6Vw 0FlXVB1lXOHKydRA2NeleS4NEG2vWqy/9Boj0itMfNDlOhkrA/0DcCurMpnpbM7ATQRcsMzk AQgA1Dpo/xWS66MaOJLwA28sKNMwkEk1Yjs+okOXDOu1F+0qvgE8sVmrOOPvvWr4axtKRSG1 t2QUiZ/ZkW/x/+t0nrM39EANV1VncuQZ1ceIiwTJFqGZQ8kb0+BNkwuNVFHRgXm1qzAJweEt RdsCMohB+H7BL5LGCVG5JaU0lqFU9pFP40HxEbyzxjsZgSE8LwkI6wcu0BLv6K6cLm0EiHPO l5G8kgRi38PS7/6s3R8QDsEtbGsYy6O82k3zSLIjuDBwA9GRaeigGppTxzAHVjf5o9KKu4O7 gC2KKVHPegbXS+GK7DU0fjzX57H5bZ6komE5eY4p3oWT/CwVPSGfPs8jOwARAQABwsB2BBgB CAAgFiEEmuvCXT0aY6hs4SbWeVOEFl5WrMgFAl+pQfkCGwwACgkQeVOEFl5WrMiVqwf9GwU8 c6cylknZX8QwlsVudTC8xr/L17JA84wf03k3d4wxP7bqy5AYy7jboZMbgWXngAE/HPQU95NM aukysSnknzoIpC96XZJ0okLBXVS6Y0ylZQ+HrbIhMpuQPoDweoF5F9wKrsHRoDaUK1VR706X rwm4HUzh7Jk+auuMYfuCh0FVlFBEuiJWMLhg/5WCmcRfiuB6F59ZcUQrwLEZeNhF2XJV4KwB Tlg7HCWO/sy1foE5noaMyACjAtAQE9p5kGYaj+DuRhPdWUTsHNuqrhikzIZd2rrcMid+ktb0 NvtvswzMO059z1YGMtGSqQ4srCArju+XHIdTFdiIYbd7+jeehg== In-Reply-To: <67721332-fa1d-4b3c-aa57-64594ad5d77a@shrew.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Scanned-By: MIMEDefang 2.86 X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA] X-Rspamd-Queue-Id: 4VVYD84bntz44V6 On 5/1/2024 4:24 PM, Matthew Grooms wrote: > On 5/1/24 14:38, mike tancsa wrote: >> Kind of struggling to check if TRIM is actually working or not with >> my SSDs on RELENG_14 in ZFS. >> >> On a pool that has almost no files on it (capacity at 0% out of 3TB), >> should not >> >> zpool -w trim be almost instant after a couple of runs ? >> Instead it seems to always take about 10min to complete. >> >> Looking at the stats, >> >> kstat.zfs.tortank1.misc.iostats.trim_bytes_failed: 0 >> kstat.zfs.tortank1.misc.iostats.trim_extents_failed: 0 >> kstat.zfs.tortank1.misc.iostats.trim_bytes_skipped: 2743435264 >> kstat.zfs.tortank1.misc.iostats.trim_extents_skipped: 253898 >> kstat.zfs.tortank1.misc.iostats.trim_bytes_written: 14835526799360 >> kstat.zfs.tortank1.misc.iostats.trim_extents_written: 1169158 >> >> what and why are bytes being skipped ? >> >> One of the drives for example I had a hard time seeing evidence of >> this at the disk level while fiddling with TRIM recently. It appeared >> that at least some counters are driver and operation specific. For >> example, the da driver appears to update counters in some paths but >> not others. I assume that ada is different. There is a bug report for >> da, but haven't seen any feedback ... > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=277673 > > You could try to run gstat with the -d flag during the time period > when the delete operations are expected to occur. That should give you > an idea of what's happening at the disk level in real time but may not > offer more info than you're already seeing. > It *seems* to be doing something.  What I dont understand is why if I run it once, do nothing (no writes / snapshots etc), and then run trim again, it seems to be doing something with gstat even though there should not be anything to mark as being trimmed ? dT: 1.002s  w: 1.000s  L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w    d/s kBps   ms/d   %busy Name     0   1254      0      0    0.0    986   5202    2.0    244 8362733    4.5   55.6  ada0    12   1242      0      0    0.0   1012   5218    1.9    206 4972041    6.0   63.3  ada2    12   1242      0      0    0.0   1012   5218    1.9    206 4972041    6.0   63.3  ada2p1     0   4313      0      0    0.0   1024   5190    0.8   3266 6463815    0.4   62.8  ada3     0   1254      0      0    0.0    986   5202    2.0    244 8362733    4.5   55.6  ada0p1     0   4238      0      0    0.0    960   4874    0.7   3254 6280362    0.4   59.8  ada5     0   4313      0      0    0.0   1024   5190    0.8   3266 6463815    0.4   62.8  ada3p1     0   4238      0      0    0.0    960   4874    0.7   3254 6280362    0.4   59.8  ada5p1 dT: 1.001s  w: 1.000s  L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w    d/s kBps   ms/d   %busy Name     2   2381      0      0    0.0   1580   9946    0.9    767 5990286    1.8   70.0  ada0     2   2801      0      0    0.0   1540   9782    0.9   1227 11936510    1.0   65.2  ada2     2   2801      0      0    0.0   1540   9782    0.9   1227 11936510    1.0   65.2  ada2p1     0   2072      0      0    0.0   1529   9566    0.8    509 12549587    2.1   57.0  ada3     2   2381      0      0    0.0   1580   9946    0.9    767 5990286    1.8   70.0  ada0p1     0   2042      0      0    0.0   1517   9427    0.6    491 12549535    1.9   52.4  ada5     0   2072      0      0    0.0   1529   9566    0.8    509 12549587    2.1   57.0  ada3p1     0   2042      0      0    0.0   1517   9427    0.6    491 12549535    1.9   52.4  ada5p1 dT: 1.002s  w: 1.000s  L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w    d/s kBps   ms/d   %busy Name     2   1949      0      0    0.0   1094   5926    1.2    827 11267200    1.8   78.8  ada0     0   2083      0      0    0.0   1115   6034    0.7    939 16537981    1.4   67.2  ada2     0   2083      0      0    0.0   1115   6034    0.7    939 16537981    1.4   67.2  ada2p1     2   2525      0      0    0.0   1098   5914    0.8   1399 16021615    1.1   79.3  ada3     2   1949      0      0    0.0   1094   5926    1.2    827 11267200    1.8   78.8  ada0p1    12   2471      0      0    0.0   1018   5399    1.0   1425 15395566    1.1   80.5  ada5     2   2525      0      0    0.0   1098   5914    0.8   1399 16021615    1.1   79.3  ada3p1    12   2471      0      0    0.0   1018   5399    1.0   1425 15395566    1.1   80.5  ada5p1 The ultimate problem is that after a while with a lot of writes, the disk performance will be toast until I do a manual trim -f of the disk :(   this is most notable on consumer WD SSDs.  I havent done any extensive tests with Samsung SSDs to see if there are performance penalties or not. It might be that they are just better at masking the problem.  I dont see the same issue with ZFS on Linux with the same disks / hardware I have an open PR in https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=277992 that I think might actually have 2 separate problems.     ---Mike >> e.g. here was one disk in the pool that was taking a long time for >> each zpool trim >> >> # time trim -f /dev/ada1 >> trim /dev/ada1 offset 0 length 1000204886016 >> 0.000u 0.057s 1:29.33 0.0%      5+184k 0+0io 0pf+0w >> and then if I re-run it >> #  time trim -f /dev/ada1 >> trim /dev/ada1 offset 0 length 1000204886016 >> 0.000u 0.052s 0:04.15 1.2%      1+52k 0+0io 0pf+0w >> >> 90 seconds and then 4 seconds after that. >> > > -Matthew >