Date: Wed, 04 Mar 2015 16:09:29 +0000 From: Steven Hartland <killing@multiplay.co.uk> To: freebsd-geom@freebsd.org Subject: Re: Trim on gmirrored SSDs is slow and results in inresponsive system Message-ID: <54F72E39.2070105@multiplay.co.uk> In-Reply-To: <54F721BC.5070204@bytecamp.net> References: <54F6FE2E.60303@bytecamp.net> <54F707CC.6070105@multiplay.co.uk> <54F7147F.8070206@bytecamp.net> <54F718D9.1080201@multiplay.co.uk> <54F721BC.5070204@bytecamp.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On 04/03/2015 15:16, Robert Schulze wrote:
> Hi,
>
> Am 04.03.2015 um 15:38 schrieb Steven Hartland:
>>>> Do you have one disk which has really slow TRIM?
>>>>
>>> please note, it does not depend on the disk, it is related to gmirror.
>> Yes but gmirror will wait for all disks to complete the delete before
>> the top level delete is marked as complete, so if you have an issue with
>> one of the underlying devices, be that cabling issue or disk on the way
>> out, this could be your issue.
> Both disks seem to be OK, see further down.
>
>>>> Also how does gstat -d -p compare between gmirror and none gmirror
>>>> installs on the same machine?
>>> Deleting on a non-mirrored UFS does not influence the system, because
>>> the BIO_DELETE calls are processed extremely fast (~ 1 sec with 1GB
>>> file), with a high number of d/s (~ 100k).
>> Can you get snapshot of both in action please
> I've collected data via
>
> $ iostat -w0.5 ada0 ada1
>
> directly after unlinking a 2 GB file for all possible cases:
>
> 1) non mirrored on ada0:
>
> tty ada0 ada1 cpu
> tin tout KB/t tps MB/s KB/t tps MB/s us ni sy in id
> 1 154 111.45 241 26.20 14.73 12 0.17 0 0 5 0 95
> 0 662 32.00 122437 3826.17 0.00 0 0.00 0 0 7 0 92
> 0 124 0.00 0 0.00 0.00 0 0.00 0 0 0 0 100
>
> 2) non mirrored on ada1:
>
> tty ada0 ada1 cpu
> tin tout KB/t tps MB/s KB/t tps MB/s us ni sy in id
> 3 2876 12.78 4 0.05 108.38 190 20.14 0 0 4 0 96
> 0 672 0.00 0 0.00 32.00 120446 3763.94 0 0 8 0 92
> 0 430 0.00 0 0.00 0.00 0 0.00 0 0 0 0 100
>
> 3) gmirrored with ada0+ada1:
>
> tty ada0 ada1 cpu
> tin tout KB/t tps MB/s KB/t tps MB/s us ni sy in id
> 7 174 110.65 927 100.13 126.38 728 89.82 0 0 3 0 97
> 0 684 32.00 5629 175.90 32.00 5629 175.90 0 0 26 0 74
> 0 459 32.00 3587 112.08 32.00 3615 112.96 0 0 22 0 78
> 0 393 32.00 2192 68.49 32.00 2169 67.79 0 0 21 0 79
> 0 346 32.00 2102 65.67 32.00 2100 65.61 0 0 20 0 80
> 0 338 32.00 5391 168.47 32.00 4793 149.79 0 0 20 0 80
> 0 391 32.00 4461 139.39 32.00 5735 179.22 0 0 23 0 77
> 0 438 32.00 3008 93.99 32.00 2374 74.19 0 0 25 0 75
> 0 406 32.00 2141 66.90 32.00 2100 65.61 0 0 24 0 76
> 0 367 32.00 1825 57.04 32.00 1825 57.04 0 0 23 0 77
> 0 340 32.00 1835 57.33 32.00 1806 56.45 0 0 25 0 75
> 0 321 32.00 1517 47.40 32.00 1485 46.41 0 0 23 0 77
> 0 289 32.00 1376 43.00 32.00 1385 43.29 0 0 27 0 73
> 0 273 32.00 1271 39.71 32.00 1291 40.35 0 0 23 0 77
> 0 267 32.00 1220 38.12 32.00 1197 37.42 0 0 23 0 77
> 0 314 32.00 5665 177.04 32.00 5196 162.36 0 0 20 0 80
> 0 412 31.98 5568 173.88 31.98 6147 191.98 0 0 22 0 78
> 0 492 32.00 5530 172.81 32.00 5426 169.57 0 0 23 0 77
> 0 459 32.00 2505 78.30 32.00 2549 79.65 0 0 27 0 73
> 0 393 32.00 1940 60.62 32.00 1942 60.68 0 0 22 0 78
> 0 367 32.00 1882 58.80 32.00 1883 58.86 0 0 19 0 81
> 0 379 32.00 5076 158.63 32.00 5477 171.15 0 0 20 0 80
> 0 611 32.00 4432 138.51 32.00 3951 123.48 0 0 20 0 80
> 0 380 32.00 3062 95.69 32.00 2254 70.43 0 0 29 0 71
> 0 370 32.00 2365 73.90 32.00 1912 59.74 0 0 24 0 76
> 0 345 32.00 1940 60.62 32.00 1825 57.05 0 0 24 0 76
> 0 329 32.00 1229 38.40 32.00 2380 74.38 0 0 19 0 81
> 0 372 32.00 1692 52.89 32.00 1587 49.59 0 0 26 0 74
> 0 415 32.00 1588 49.64 32.00 1496 46.74 0 0 25 0 75
> 0 346 32.00 1456 45.50 32.00 1429 44.66 0 0 26 0 74
> 0 346 32.00 1355 42.33 32.00 1283 40.10 0 0 24 0 76
> 0 356 32.00 1155 36.09 32.00 1501 46.92 0 0 25 0 75
> 0 385 32.00 1177 36.78 32.00 1091 34.08 0 0 23 0 77
> 0 393 32.00 1805 56.39 32.00 1823 56.98 0 0 24 0 76
> 0 365 32.00 2021 63.15 32.00 2064 64.50 0 0 26 0 74
> 0 344 32.00 1868 58.39 32.00 1938 60.56 0 0 25 0 75
> 0 380 32.00 1897 59.27 32.00 1936 60.50 0 0 27 0 73
> 0 344 32.00 1774 55.45 32.00 1778 55.57 0 0 23 0 77
> 0 334 32.00 1674 52.31 32.00 1693 52.91 0 0 25 0 75
> 0 329 32.00 2254 70.43 32.00 2428 75.89 0 0 19 0 81
> 0 406 32.00 6331 197.85 32.00 6337 198.02 0 0 24 0 76
> 0 408 32.00 2966 92.69 32.00 2925 91.40 0 0 25 0 75
> 0 353 32.00 2280 71.25 32.00 2167 67.73 0 0 25 0 75
> 0 575 32.00 5051 157.84 32.00 4844 151.38 0 0 18 0 82
> 0 421 32.00 3801 118.79 32.00 4171 130.33 0 0 23 0 77
> 0 397 32.00 466 14.56 32.00 470 14.68 0 0 3 0 97
> 0 344 0.00 0 0.00 0.00 0 0.00 0 0 0 0 100
>
> 4) gmirrored with only one component active (ada0)
>
> tty ada0 ada1 cpu
> tin tout KB/t tps MB/s KB/t tps MB/s us ni sy in id
> 1 132 124.10 176 21.34 15.00 10 0.15 0 0 4 0 96
> 0 643 32.00 50662 1583.18 0.00 0 0.00 0 0 19 0 81
> 0 458 32.00 11102 346.93 0.00 0 0.00 0 0 18 0 82
> 0 287 32.00 4051 126.58 0.00 0 0.00 0 0 17 0 83
> 0 244 32.00 3500 109.38 0.00 0 0.00 0 0 18 0 82
> 0 256 32.00 3358 104.93 0.00 0 0.00 0 0 19 0 81
> 0 244 32.00 3010 94.06 0.00 0 0.00 0 0 18 0 82
> 0 260 32.00 2710 84.68 0.00 0 0.00 0 0 18 0 82
> 0 257 32.00 2524 78.88 0.00 0 0.00 0 0 18 0 82
> 0 234 32.00 2345 73.29 0.00 0 0.00 0 0 17 0 82
> 0 245 32.00 2185 68.28 0.00 0 0.00 0 0 19 0 81
> 0 239 32.00 2072 64.73 0.00 0 0.00 0 0 17 0 82
> 0 244 32.00 1970 61.56 0.00 0 0.00 0 0 18 0 81
> 0 243 32.00 1878 58.70 0.00 0 0.00 0 0 18 0 82
> 0 241 32.00 1799 56.22 0.00 0 0.00 0 0 19 0 81
> 0 231 32.00 1928 60.25 0.00 0 0.00 0 0 18 0 82
> 0 254 32.00 2442 76.30 0.00 0 0.00 0 0 15 0 85
> 0 252 32.00 2457 76.77 0.00 0 0.00 0 0 12 0 88
> 0 246 32.00 2461 76.89 0.00 0 0.00 0 0 13 0 87
> 0 235 32.00 2472 77.24 0.00 0 0.00 0 0 12 0 88
> 0 194 32.00 2491 77.83 0.00 0 0.00 0 0 13 0 87
> 0 201 32.00 2517 78.65 0.00 0 0.00 0 0 13 0 87
> 0 429 32.00 2545 79.54 0.00 0 0.00 0 0 12 0 88
> 0 231 31.99 9419 294.23 0.00 0 0.00 0 0 12 0 88
> 0 303 0.00 0 0.00 0.00 0 0.00 0 0 0 0 100
>
> 5) gmirrored with only one component active (ada1)
>
> tty ada0 ada1 cpu
> tin tout KB/t tps MB/s KB/t tps MB/s us ni sy in id
> 1 153 11.75 5 0.06 100.91 298 29.40 0 0 5 0 95
> 0 635 0.00 0 0.00 32.00 31614 987.94 0 0 19 0 81
> 0 372 0.00 0 0.00 32.00 20236 632.38 0 0 17 0 83
> 0 331 0.50 2 0.00 32.00 16922 528.74 0 0 15 0 85
> 0 350 0.00 0 0.00 32.00 20795 649.84 0 0 17 0 83
> 0 278 0.00 0 0.00 32.00 17748 554.63 0 0 15 0 85
> 0 276 0.00 0 0.00 32.00 14885 465.16 0 0 8 0 92
> 0 272 0.00 0 0.00 0.00 0 0.00 0 0 0 0 100
Transaction sizes look like they are the same but the tps from gmirror
is significantly lower.
As iostat doesn't properly understand deletes it would still be nice to
see that gstat -d -p output as that will also give us queue depth.
Try backing out r268816 and see if that has any impact.
Having a quick look through the code it looks like there is an
accounting issue with sc_writes as it will register deletes on queue but
not completion which will prevent g_mirror_idle working correctly.
Regards
Steve
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?54F72E39.2070105>