Date: Wed, 08 Jul 2015 11:37:09 -0500 From: Karl Denninger <karl@denninger.net> To: freebsd-fs@freebsd.org Subject: Re: Speeding up resilvering Message-ID: <559D51B5.3050709@denninger.net> In-Reply-To: <559D4DD3.10903@infracaninophile.co.uk> References: <559D2AB6.5070007@FreeBSD.org> <559D2C7C.6060201@denninger.net> <559D3167.1000705@infracaninophile.co.uk> <559D3236.1060102@denninger.net> <559D3380.8050703@sentex.net> <559D4DD3.10903@infracaninophile.co.uk>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a cryptographically signed message in MIME format. --------------ms030104030200090303040803 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On 7/8/2015 11:20, Matthew Seaman wrote: > On 07/08/15 15:28, Mike Tancsa wrote: >>> Look at the IO saturation on the disk channel(s) involved with either= >>>> systat -vm or iostat. If the channel is saturated then there's noth= ing >>>> you can do in terms of tuning; the question then turns to why actual= I/O >>>> performance is so poor and has to be addressed there. >> I had one server that was taking ages, and it turned out to be the >> controller, not the disk that was hosed. As Karl suggested, take a loo= k >> at the throughput on gstat. Is anyone disk or groups of disks lagging >> far behind on write speeds ? In my case, it was a very obvious and >> glaring outlier. > Thanks for the suggestions. I've been playing with gstat et al, and as= > far as I can tell, all the drives are behaving reasonably well. I'm > certainly getting 90-100% capacity (mostly reads) continually on the > original drives whilst the new one (mostly writes) seems to go in > bursts. Which is pretty much what I'd expect when resilvering a RAIDZ.= > > So, having exhausted that, I actually sat down and timed what progress > it was making rather more carefully. Turns out my impression that > applying the sysctl tweaks I mentioned previously had little effect was= > wrong. Current projection is about 50h total to do the resilvering, > which is much, much better than the approx 12days I was expecting > previously. In fact, that's pretty much inline with what I'd expect > from this hardware. > > Cheers, > > Matthew > > > Be aware that when a resilver /starts /it does a number of very small and diverse I/O operations. Throughput during that time _*stinks*_, and as a result in the first few minutes to half-hour or so (with a very large vdev) you will see utterly ridiculous projections of completion tim= e. Once that part of the process completes (usually within a few minutes on modest to moderate size vdevs, but it will be longer on very large ones) the performance level of the resilver will go up a lot and the projected completion time will become far more reasonable. --=20 Karl Denninger karl@denninger.net <mailto:karl@denninger.net> /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms030104030200090303040803 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIGXzCC BlswggRDoAMCAQICASkwDQYJKoZIhvcNAQELBQAwgZAxCzAJBgNVBAYTAlVTMRAwDgYDVQQI EwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM TEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExIjAgBgkqhkiG9w0BCQEWE0N1ZGEg U3lzdGVtcyBMTEMgQ0EwHhcNMTUwNDIxMDIyMTU5WhcNMjAwNDE5MDIyMTU5WjBaMQswCQYD VQQGEwJVUzEQMA4GA1UECBMHRmxvcmlkYTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1zIExMQzEe MBwGA1UEAxMVS2FybCBEZW5uaW5nZXIgKE9DU1ApMIICIjANBgkqhkiG9w0BAQEFAAOCAg8A MIICCgKCAgEAuYRY+EB2mGtZ3grlVO8TmnEvduVFA/IYXcCmNSOC1q+pTVjylsjcHKBcOPb9 TP1KLxdWP+Q1soSORGHlKw2/HcVzShDW5WPIKrvML+Ry0XvIvNBu9adTiCsA9nci4Cnf98XE hVpenER0qbJkBUOGT1rP4iAcfjet0lEgzPEnm+pAxv6fYSNp1WqIY9u0b1pkQiaWrt8hgNOc rJOiLbc8CeQ/DBP6rUiQjYNO9/aPNauEtHkNNfR9RgLSfGUdZuOCmJqnIla1HsrZhA5p69Bv /e832BKiNPaH5wF6btAiPpTr2sRhwQO8/IIxcRX1Vxd1yZbjYtJGw+9lwEcWRYAmoxkzKLPi S6Zo/6z5wgNpeK1H+zOioMoZIczgI8BlX1iHxqy/FAvm4PHPnC8s+BLnJLwr+jvMNHm82QwL J9hC5Ho8AnFU6TkCuq+P2V8/clJVqnBuvTUKhYMGSm4mUp+lAgR4L+lwIEqSeWVsxirIcE7Z OKkvI7k5x3WeE3+c6w74L6PfWVAd84xFlo9DKRdU9YbkFuFZPu21fi/LmE5brImB5P+jdqnK eWnVwRq+RBFLy4kehCzMXooitAwgP8l/JJa9VDiSyd/PAHaVGiat2vCdDh4b8cFL7SV6jPA4 k0MgGUA/6Et7wDmhZmCigggr9K6VQCx8jpKB3x1NlNNiaWECAwEAAaOB9DCB8TA3BggrBgEF BQcBAQQrMCkwJwYIKwYBBQUHMAGGG2h0dHA6Ly9jdWRhc3lzdGVtcy5uZXQ6ODg4ODAJBgNV HRMEAjAAMBEGCWCGSAGG+EIBAQQEAwIFoDALBgNVHQ8EBAMCBeAwLAYJYIZIAYb4QgENBB8W HU9wZW5TU0wgR2VuZXJhdGVkIENlcnRpZmljYXRlMB0GA1UdDgQWBBTFHJQt6cloXBdG1Pv1 o2YgH+7lWTAfBgNVHSMEGDAWgBQkcZudhX383d29sMqSlAOh+tNtNTAdBgNVHREEFjAUgRJr YXJsQGRlbm5pbmdlci5uZXQwDQYJKoZIhvcNAQELBQADggIBAE9/dxi2YqjCYYhiybp4GKcm 7tBVa/GLW+qcHPcoT4dqmqghlLz8+iUH+HCJjRQATVGyMEnvISOKFVHC6aZIG+Sg7J8bfS4+ fjKDi9smRH2VPPx3bV8+yFYRNroMGHaPHZB/Xctmmvc+PZ9O2W7rExgrODtxIOB3Zs6wkYf+ ty+9r1KmTHlV+rRHI6timH1uiyFE3cPi1taAEBxf0851cJV8k40PGF8G48ewnq8SY9sCf5cv liXbpdgU+I4ND5BuTjg63WS32zuhLd1VSuH3ZC/QbcncMX5W3oLXmcQP5/5uTiBJy74kdPtG MSZ9rXwZPwNxP/8PXMSR7ViaFvjUkf4bJlyENFa2PGxLk4EUzOuO7t3brjMlQW1fuInfG+ko 3tVxko20Hp0tKGPe/9cOxBVBZeZH/VgpZn3cLculGzZjmdh2fqAQ6kv9Z9AVOG1+dq0c1zt8 2zm+Oi1pikGXkfz5UJq60psY6zbX25BuEZkthO/qiS4pxjxb7gQkS0rTEHTy+qv0l3QVL0wa NAT74Zaj7l5DEW3qdQQ0dtVieyvptg9CxkfQJE3JyBMb0zBj9Qhc5/hbTfhSlHzZMEbUuIyx h9vxqFAmGzfB1/WfOKkiNHChkpPW8ZeH9yPeDBKvrgZ96dREHFoVkDk7Vpw5lSM+tFOfdyLg xxhb/RZVUDeUMYIE4zCCBN8CAQEwgZYwgZAxCzAJBgNVBAYTAlVTMRAwDgYDVQQIEwdGbG9y aWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBMTEMxHDAa BgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExIjAgBgkqhkiG9w0BCQEWE0N1ZGEgU3lzdGVt cyBMTEMgQ0ECASkwCQYFKw4DAhoFAKCCAiEwGAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAc BgkqhkiG9w0BCQUxDxcNMTUwNzA4MTYzNzA5WjAjBgkqhkiG9w0BCQQxFgQUXlQIRyU8E7oX TkaE5I0JUnF50XUwbAYJKoZIhvcNAQkPMV8wXTALBglghkgBZQMEASowCwYJYIZIAWUDBAEC MAoGCCqGSIb3DQMHMA4GCCqGSIb3DQMCAgIAgDANBggqhkiG9w0DAgIBQDAHBgUrDgMCBzAN BggqhkiG9w0DAgIBKDCBpwYJKwYBBAGCNxAEMYGZMIGWMIGQMQswCQYDVQQGEwJVUzEQMA4G A1UECBMHRmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3Rl bXMgTExDMRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhND dWRhIFN5c3RlbXMgTExDIENBAgEpMIGpBgsqhkiG9w0BCRACCzGBmaCBljCBkDELMAkGA1UE BhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExEjAQBgNVBAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQ Q3VkYSBTeXN0ZW1zIExMQzEcMBoGA1UEAxMTQ3VkYSBTeXN0ZW1zIExMQyBDQTEiMCAGCSqG SIb3DQEJARYTQ3VkYSBTeXN0ZW1zIExMQyBDQQIBKTANBgkqhkiG9w0BAQEFAASCAgA4VGYV JD17B2xeT8x9fAawSjdp1wtlzdQXVOULaa5CQF3VEM9sI8crtynceHfzeH8OrQLiCZdA2U4z 0v2mu8YxSR6U79t49/aPCHqw6vJvYd5AZjNH/1DPPcu8c81YaM/tZI3R7bg5RiDXFcx0Z+hY wutsOkKLn7ZvVAB3H031BlBZI7r38cBQBAIQ6843S1OJAxyNGlwfOxc6k4p4SxJjfdaz/YHz X6NY6nxDR+AKJ5g3uhgs8LnwqTQUbiAovsLTrQZwxgUk7z8PZUQN6eKs7gUh4cz8Aw/xsck6 +Lq6qBrJrFrygBSxUyK6kflyN8lbGNJyb/fTWEr0voVX81Nyp8htewtp+wa+ov2XU8JKcTOv FDJhHkNSvKyPrkn7gHGyLL/JRpoP8X9qncAjJ2ABj7p1LLPYMbMQzs3LRjXlX8DgIOB+c+iX D3MfXmwNHAamg1fsCjD339YY/HR7bRrLAxATdopQ3/RzaBcfngtTBLWYOisDQau9AAeAXU03 PlIepUQsxU4mzam5UXPIA138npmT0IojlURL+Stn6OiVuNNGiTv/fSwr9+GYm0nTWAubdy/c OxK6ZcTJK1nj7mfTS6Rxfo3IhGjZTJP9T3hbSyGOT3uKRyP53yWa2D5Ocanqhr8EOpghUN+9 c7VL8vKASZomZAtkXd9m34oaFAcQ9gAAAAAAAA== --------------ms030104030200090303040803--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?559D51B5.3050709>