Date: Tue, 18 Mar 2014 06:06:09 -0500 From: Karl Denninger <karl@denninger.net> To: freebsd-fs@freebsd.org Subject: Re: Reoccurring ZFS performance problems [RESOLVED] Message-ID: <532828A1.6080605@denninger.net> In-Reply-To: <c5a61ac31b121de10f9b065967fe1ae3@mail.mikej.com> References: <531E2406.8010301@denninger.net> <5320A0E8.2070406@denninger.net> <5322E64E.8020009@denninger.net> <CA%2BD9QhtguPKD9zQ35246LkMt5gTU6MJ%2BZBigoznk7FHQ4R0nhA@mail.gmail.com> <CA%2BD9Qht_7dW6gahTFKz7B9%2BpgJemkFoPLLcASmPvCfGytxF8cQ@mail.gmail.com> <53236BF3.9060500@denninger.net> <CA%2BD9QhstDyetoA5HdwyMA1BOyafMJ%2BbrxDEvdNHUXFPa3YGPtg@mail.gmail.com> <c5a61ac31b121de10f9b065967fe1ae3@mail.mikej.com>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a cryptographically signed message in MIME format. --------------ms080003040007000606060803 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: quoted-printable On 3/18/2014 5:26 AM, mikej wrote: > On 2014-03-14 19:04, Matthias Gamsjager wrote: >> Much better thx :) >> >> Will this patch be review by some kernel devs and merged? >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > I am a little surprised this thread has been so quiet. I have been > running with this patch and my desktop is more pleasant when memory > demands are great - no more swapping - and wired no longer grows > uncontrollable. > > Is more review coming the silence is deffining. > It makes an utterly-enormous difference here. This is what one of my "nasty-busy" servers looks like this morning=20 (it's got a very busy blog on it along with other things, and is=20 pretty-quiet right now -- but it won't be in a couple of hours) 1 users Load 0.22 0.25 0.21 Mar 18 05:55 Mem:KB REAL VIRTUAL VN PAGER SWAP P= AGER Tot Share Tot Share Free in out in = out Act 4238440 31700 7953812 53652 2993908 count All 16025k 39644 8680436 249960 pages Proc: Interrup= ts r p d s w Csw Trp Sys Int Sof Flt ioflt 2083 to= tal 204 7321 1498 6416 665 313 707 207 cow 12 ua= rt0 4 428 zfod 20 uh= ci0 16 0.4%Sys 0.1%Intr 0.6%User 0.0%Nice 99.0%Idle ozfod pc= m0 17 | | | | | | | | | | %ozfod ehc= i0 uhci > daefr uhc= i1 21 dtbuf 417 prcfr 455 uh= ci3 ehci Namei Name-cache Dir-cache 485892 desvn 1197 totfr 16 twa= 0 30 Calls hits % hits % 136934 numvn react 994 cp= u0:timer 8063 8009 99 121473 frevn pdwak 42 mp= s0 256 871 pdpgs 15 em= 0:rx 0 Disks ada0 da0 da1 da2 da3 da4 da5 intrn 20 em0= :tx 0 KB/t 0.00 20.46 19.92 0.00 0.00 22.06 44.21 17177460 wire em0= :link tps 0 7 7 0 0 7 11 2131860 act 45 em1= :rx 0 MB/s 0.00 0.15 0.15 0.00 0.00 0.15 0.47 2158808 inact 38 em1= :tx 0 %busy 0 7 7 0 0 0 0 7512 cache em1= :link 2986396 free ah= ci0:ch0 buf 16 cp= u1:timer 23 cp= u11:time 17 cp= u5:timer 13 cp= u9:timer 44 cp= u4:timer 35 cp= u15:time 26 cp= u6:timer 16 cp= u14:time 28 cp= u7:timer 23 cp= u13:time 23 cp= u3:timer 43 cp= u10:time 50 cp= u2:timer 29 cp= u12:time 40 cp= u8:timer Here's the ARC cache.... [karl@NewFS ~]$ zfs-stats -A ------------------------------------------------------------------------ ZFS Subsystem Report Tue Mar 18 05:56:42 2014 ------------------------------------------------------------------------ ARC Summary: (HEALTHY) Memory Throttle Count: 0 ARC Misc: Deleted: 1.55m Recycle Misses: 66.33k Mutex Misses: 1.55k Evict Skips: 4.14m ARC Size: 60.01% 13.40 GiB Target Size: (Adaptive) 60.01% 13.40 GiB Min Size (Hard Limit): 12.50% 2.79 GiB Max Size (High Water): 8:1 22.33 GiB ARC Size Breakdown: Recently Used Cache Size: 79.13% 10.60 GiB Frequently Used Cache Size: 20.87% 2.80 GiB ARC Hash Breakdown: Elements Max: 1.34m Elements Current: 62.76% 840.43k Collisions: 7.02m Chain Max: 13 Chains: 247.65k ------------------------------------------------------------------------ Note the scale-down from the maximum -- this is with: [karl@NewFS ~]$ sysctl -a|grep percent vfs.zfs.arc_freepage_percent_target: 10 My test machine has a lot less memory in it and there the default (25%)=20 appears to be a good value. Before this delta was put on the code this system would have tried to=20 grab the entire 22GB to the exclusion of anything else. What I used to=20 do is limit it to 16GB via arc_max which was fine in the mornings and=20 overnight, but during the day it didn't cut it -- and there was no way=20 to change it without a reboot either. This particular machine has 24GB=20 of RAM in it and provides services both externally and internally=20 (separate interfaces.) How efficient is the cache? [karl@NewFS ~]$ zfs-stats -E ------------------------------------------------------------------------ ZFS Subsystem Report Tue Mar 18 05:59:01 2014 ------------------------------------------------------------------------ ARC Efficiency: 81.13m Cache Hit Ratio: 97.84% 79.38m Cache Miss Ratio: 2.16% 1.75m Actual Hit Ratio: 69.81% 56.64m Data Demand Efficiency: 99.09% 50.37m Data Prefetch Efficiency: 28.77% 1.46m CACHE HITS BY CACHE LIST: Anonymously Used: 28.48% 22.61m Most Recently Used: 6.81% 5.40m Most Frequently Used: 64.54% 51.23m Most Recently Used Ghost: 0.03% 24.86k Most Frequently Used Ghost: 0.13% 104.39k CACHE HITS BY DATA TYPE: Demand Data: 62.88% 49.91m Prefetch Data: 0.53% 419.73k Demand Metadata: 8.28% 6.57m Prefetch Metadata: 28.31% 22.47m CACHE MISSES BY DATA TYPE: Demand Data: 26.03% 456.20k Prefetch Data: 59.29% 1.04m Demand Metadata: 9.84% 172.53k Prefetch Metadata: 4.84% 84.81k ------------------------------------------------------------------------ --=20 -- Karl karl@denninger.net --------------ms080003040007000606060803 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIFTzCC BUswggQzoAMCAQICAQgwDQYJKoZIhvcNAQEFBQAwgZ0xCzAJBgNVBAYTAlVTMRAwDgYDVQQI EwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM TEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExLzAtBgkqhkiG9w0BCQEWIGN1c3Rv bWVyLXNlcnZpY2VAY3VkYXN5c3RlbXMubmV0MB4XDTEzMDgyNDE5MDM0NFoXDTE4MDgyMzE5 MDM0NFowWzELMAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExFzAVBgNVBAMTDkthcmwg RGVubmluZ2VyMSEwHwYJKoZIhvcNAQkBFhJrYXJsQGRlbm5pbmdlci5uZXQwggIiMA0GCSqG SIb3DQEBAQUAA4ICDwAwggIKAoICAQC5n2KBrBmG22nVntVdvgKCB9UcnapNThrW1L+dq6th d9l4mj+qYMUpJ+8I0rTbY1dn21IXQBoBQmy8t1doKwmTdQ59F0FwZEPt/fGbRgBKVt3Quf6W 6n7kRk9MG6gdD7V9vPpFV41e+5MWYtqGWY3ScDP8SyYLjL/Xgr+5KFKkDfuubK8DeNqdLniV jHo/vqmIgO+6NgzPGPgmbutzFQXlxUqjiNAAKzF2+Tkddi+WKABrcc/EqnBb0X8GdqcIamO5 SyVmuM+7Zdns7D9pcV16zMMQ8LfNFQCDvbCuuQKMDg2F22x5ekYXpwjqTyfjcHBkWC8vFNoY 5aFMdyiN/Kkz0/kduP2ekYOgkRqcShfLEcG9SQ4LQZgqjMpTjSOGzBr3tOvVn5LkSJSHW2Z8 Q0dxSkvFG2/lsOWFbwQeeZSaBi5vRZCYCOf5tRd1+E93FyQfpt4vsrXshIAk7IK7f0qXvxP4 GDli5PKIEubD2Bn+gp3vB/DkfKySh5NBHVB+OPCoXRUWBkQxme65wBO02OZZt0k8Iq0i4Rci WV6z+lQHqDKtaVGgMsHn6PoeYhjf5Al5SP+U3imTjF2aCca1iDB5JOccX04MNljvifXgcbJN nkMgrzmm1ZgJ1PLur/ADWPlnz45quOhHg1TfUCLfI/DzgG7Z6u+oy4siQuFr9QT0MQIDAQAB o4HWMIHTMAkGA1UdEwQCMAAwEQYJYIZIAYb4QgEBBAQDAgWgMAsGA1UdDwQEAwIF4DAsBglg hkgBhvhCAQ0EHxYdT3BlblNTTCBHZW5lcmF0ZWQgQ2VydGlmaWNhdGUwHQYDVR0OBBYEFHw4 +LnuALyLA5Cgy7T5ZAX1WzKPMB8GA1UdIwQYMBaAFF3U3hpBZq40HB5VM7B44/gmXiI0MDgG CWCGSAGG+EIBAwQrFilodHRwczovL2N1ZGFzeXN0ZW1zLm5ldDoxMTQ0My9yZXZva2VkLmNy bDANBgkqhkiG9w0BAQUFAAOCAQEAZ0L4tQbBd0hd4wuw/YVqEBDDXJ54q2AoqQAmsOlnoxLO 31ehM/LvrTIP4yK2u1VmXtUumQ4Ao15JFM+xmwqtEGsh70RRrfVBAGd7KOZ3GB39FP2TgN/c L5fJKVxOqvEnW6cL9QtvUlcM3hXg8kDv60OB+LIcSE/P3/s+0tEpWPjxm3LHVE7JmPbZIcJ1 YMoZvHh0NSjY5D0HZlwtbDO7pDz9sZf1QEOgjH828fhtborkaHaUI46pmrMjiBnY6ujXMcWD pxtikki0zY22nrxfTs5xDWGxyrc/cmucjxClJF6+OYVUSaZhiiHfa9Pr+41okLgsRB0AmNwE f6ItY3TI8DGCBQowggUGAgEBMIGjMIGdMQswCQYDVQQGEwJVUzEQMA4GA1UECBMHRmxvcmlk YTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3RlbXMgTExDMRwwGgYD VQQDExNDdWRhIFN5c3RlbXMgTExDIENBMS8wLQYJKoZIhvcNAQkBFiBjdXN0b21lci1zZXJ2 aWNlQGN1ZGFzeXN0ZW1zLm5ldAIBCDAJBgUrDgMCGgUAoIICOzAYBgkqhkiG9w0BCQMxCwYJ KoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNDAzMTgxMTA2MDlaMCMGCSqGSIb3DQEJBDEW BBTcOO7UeJ4/nbtSto5QA6XRUEXPejBsBgkqhkiG9w0BCQ8xXzBdMAsGCWCGSAFlAwQBKjAL BglghkgBZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3DQMCAgFA MAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMIG0BgkrBgEEAYI3EAQxgaYwgaMwgZ0xCzAJBgNV BAYTAlVTMRAwDgYDVQQIEwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoT EEN1ZGEgU3lzdGVtcyBMTEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExLzAtBgkq hkiG9w0BCQEWIGN1c3RvbWVyLXNlcnZpY2VAY3VkYXN5c3RlbXMubmV0AgEIMIG2BgsqhkiG 9w0BCRACCzGBpqCBozCBnTELMAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExEjAQBgNV BAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1zIExMQzEcMBoGA1UEAxMTQ3Vk YSBTeXN0ZW1zIExMQyBDQTEvMC0GCSqGSIb3DQEJARYgY3VzdG9tZXItc2VydmljZUBjdWRh c3lzdGVtcy5uZXQCAQgwDQYJKoZIhvcNAQEBBQAEggIAI0JBocgBiPhaaUMnDh3WPOrhQIiY SQO21PI8XyJdcPGfl5aAC//PA2LZFUuh2dAclIijuelHygj7SHkLZwtskQ7ZMn1QF/SD0SOF PW7zkSZUcJVGlF/nSWzJyD9x+dVdDHTM+hcatiCovgIH3ua7N28FXlRr5zSrvxNi12XZsdsN v35TiF76Vr4kRY0rKAvEeXaEjGgWgERgCNwawF65yInebFkRzZTSKJgxRaFi+7EzuTXDPhd8 qLLZoVa9yAeCHACGXS2yx04lBMCdpAZNbELWnASftLuk058ZNoRsgrnCiTzNtSoypeRNESRE 5X1XIrIqCSv6JRagC97JVpdEUe5EI2xcbaPzXobGCKVqq+3/WZMPKbsVk9dpak5siEiUMadC rmMKPoBlJoFXcxKHOFONSre3bYVF8rqy85zOKOSl2VHBoBSa1cUInp5nTBItGW8mMyXAxGUw s6A08WGJacFZCw0sioSQ3O2KzwWDkUWiv8AC+osmSyJjeVOH7UgxSAy61DbdePzlL+RLjN+m pIJ611l2JdWPEK3izuEF9Jht6aYv5xAH2/U7lRMWJwouPNnDFui7jZW7B1/dIceDgAJ5/JOC hPPpGZy+epqTWL3Y2/qUEIcmdCiNg3ea32ok9e2idPkCBjtGR0vo+ZFEQ4B9O34uScqYXJ0U azlwyOwAAAAAAAA= --------------ms080003040007000606060803--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?532828A1.6080605>