Date: Wed, 19 Mar 2014 05:59:49 -0500 From: Karl Denninger <karl@denninger.net> To: Adrian Chadd <adrian@freebsd.org> Cc: Alan Cox <alc@freebsd.org>, "freebsd-hackers@freebsd.org" <freebsd-hackers@freebsd.org> Subject: Re: Tracking down what has inact pages locked up Message-ID: <532978A5.3050707@denninger.net> In-Reply-To: <CAJ-VmomFVv_xUwhgcnEpaHX%2BZesXdmR4GVr4rKSjA2dJK687NQ@mail.gmail.com> References: <53260B36.2070409@denninger.net> <201403181505.47349.jhb@freebsd.org> <5328A024.6050901@denninger.net> <201403181730.02471.jhb@freebsd.org> <532910D1.3010704@denninger.net> <CAJ-VmomFVv_xUwhgcnEpaHX%2BZesXdmR4GVr4rKSjA2dJK687NQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a cryptographically signed message in MIME format. --------------ms030305030907060707060007 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: quoted-printable Yes, but I will have to be strategic on this -- probably have to wait=20 for the weekend as this particular box is one that will generate a lot=20 of screaming if I reboot it to turn in-kernel NAT back on (say much less = play on the console in single-user mode) during the work week :-) I did that and didn't see anything sucking up the RAM -- which was what=20 got my attention. On 3/18/2014 11:53 PM, Adrian Chadd wrote: > Can you dump out vmstat -z during the leak and once you've unloaded it?= > > > -a > > > On 18 March 2014 20:36, Karl Denninger <karl@denninger.net> wrote: >> On 3/18/2014 4:30 PM, John Baldwin wrote: >>> On Tuesday, March 18, 2014 3:36:04 pm Karl Denninger wrote: >>>> On 3/18/2014 2:05 PM, John Baldwin wrote: >>>>> On Sunday, March 16, 2014 4:36:06 pm Karl Denninger wrote: >>>>>> Is there a reasonable way to determine who or what has that memory= >>>>>> locked up -- and thus why the vm system is not demoting that space= into >>>>>> the cache bucket so it can be freed (which, if my understanding is= >>>>>> correct, should be happening long before now!) >>>>> I have a hackish thing (for 8.x, might work on 10.x) to let you fig= ure >>>>> out >>>>> what is using up RAM. This should perhaps go into the base system = at >>>>> some >>>>> point. >>>>> >>>>> Grab the bits at http://people.freebsd.org/~jhb/vm_objects/ >>>>> >>>>> You will want to build the kld first and use 'make load' to load it= =2E It >>>>> adds >>>>> a new sysctl that dumps info about all the VM objects in the system= =2E >>>>> You can >>>>> then build the 'vm_objects' tool and run it. It can take a while t= o run >>>>> if >>>>> you have NFS mounts, so I typically save its output to a file first= and >>>>> then >>>>> use sort on the results. sort -n will show you the largest consume= r of >>>>> RAM, >>>>> sort -n -k 3 will show you the largest consumer of inactive pages. = Note >>>>> that 'df' and 'ph' objects are anonymous, and that filename paths a= ren't >>>>> always reliable, but this can still be useful. >>>>> >>>> Thanks. >>>> >>>> I suspect the cause of the huge inact consumption is a RAM leak in t= he >>>> NAT code in IPFW. It was not occurring in 9.2-STABLE, but is on >>>> 10.0-STABLE, and reverting to natd in userland stops it -- which >>>> pretty-well isolates where it's coming from. >>> Memory for in-kernel NAT should be wired pages, not inactive. >> Yeah, should be. :-) >> >> But..... it managed to lock up 19GB of the 24GB the system has in inac= t >> pages over 12 hours, and dropping the system to single user and unload= ing >> the modules did not release the RAM...... which is why the question (o= n how >> to track down what the hell is going on.) >> >> Changing the config back to natd as opposed to in-kernel NAT, however,= made >> the problem disappear. >> >> -- >> -- Karl >> karl@denninger.net >> >> > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.o= rg" > > > %SPAMBLOCK-SYS: Matched [@freebsd.org+], message ok --=20 -- Karl karl@denninger.net --------------ms030305030907060707060007 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIFTzCC BUswggQzoAMCAQICAQgwDQYJKoZIhvcNAQEFBQAwgZ0xCzAJBgNVBAYTAlVTMRAwDgYDVQQI EwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM TEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExLzAtBgkqhkiG9w0BCQEWIGN1c3Rv bWVyLXNlcnZpY2VAY3VkYXN5c3RlbXMubmV0MB4XDTEzMDgyNDE5MDM0NFoXDTE4MDgyMzE5 MDM0NFowWzELMAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExFzAVBgNVBAMTDkthcmwg RGVubmluZ2VyMSEwHwYJKoZIhvcNAQkBFhJrYXJsQGRlbm5pbmdlci5uZXQwggIiMA0GCSqG SIb3DQEBAQUAA4ICDwAwggIKAoICAQC5n2KBrBmG22nVntVdvgKCB9UcnapNThrW1L+dq6th d9l4mj+qYMUpJ+8I0rTbY1dn21IXQBoBQmy8t1doKwmTdQ59F0FwZEPt/fGbRgBKVt3Quf6W 6n7kRk9MG6gdD7V9vPpFV41e+5MWYtqGWY3ScDP8SyYLjL/Xgr+5KFKkDfuubK8DeNqdLniV jHo/vqmIgO+6NgzPGPgmbutzFQXlxUqjiNAAKzF2+Tkddi+WKABrcc/EqnBb0X8GdqcIamO5 SyVmuM+7Zdns7D9pcV16zMMQ8LfNFQCDvbCuuQKMDg2F22x5ekYXpwjqTyfjcHBkWC8vFNoY 5aFMdyiN/Kkz0/kduP2ekYOgkRqcShfLEcG9SQ4LQZgqjMpTjSOGzBr3tOvVn5LkSJSHW2Z8 Q0dxSkvFG2/lsOWFbwQeeZSaBi5vRZCYCOf5tRd1+E93FyQfpt4vsrXshIAk7IK7f0qXvxP4 GDli5PKIEubD2Bn+gp3vB/DkfKySh5NBHVB+OPCoXRUWBkQxme65wBO02OZZt0k8Iq0i4Rci WV6z+lQHqDKtaVGgMsHn6PoeYhjf5Al5SP+U3imTjF2aCca1iDB5JOccX04MNljvifXgcbJN nkMgrzmm1ZgJ1PLur/ADWPlnz45quOhHg1TfUCLfI/DzgG7Z6u+oy4siQuFr9QT0MQIDAQAB o4HWMIHTMAkGA1UdEwQCMAAwEQYJYIZIAYb4QgEBBAQDAgWgMAsGA1UdDwQEAwIF4DAsBglg hkgBhvhCAQ0EHxYdT3BlblNTTCBHZW5lcmF0ZWQgQ2VydGlmaWNhdGUwHQYDVR0OBBYEFHw4 +LnuALyLA5Cgy7T5ZAX1WzKPMB8GA1UdIwQYMBaAFF3U3hpBZq40HB5VM7B44/gmXiI0MDgG CWCGSAGG+EIBAwQrFilodHRwczovL2N1ZGFzeXN0ZW1zLm5ldDoxMTQ0My9yZXZva2VkLmNy bDANBgkqhkiG9w0BAQUFAAOCAQEAZ0L4tQbBd0hd4wuw/YVqEBDDXJ54q2AoqQAmsOlnoxLO 31ehM/LvrTIP4yK2u1VmXtUumQ4Ao15JFM+xmwqtEGsh70RRrfVBAGd7KOZ3GB39FP2TgN/c L5fJKVxOqvEnW6cL9QtvUlcM3hXg8kDv60OB+LIcSE/P3/s+0tEpWPjxm3LHVE7JmPbZIcJ1 YMoZvHh0NSjY5D0HZlwtbDO7pDz9sZf1QEOgjH828fhtborkaHaUI46pmrMjiBnY6ujXMcWD pxtikki0zY22nrxfTs5xDWGxyrc/cmucjxClJF6+OYVUSaZhiiHfa9Pr+41okLgsRB0AmNwE f6ItY3TI8DGCBQowggUGAgEBMIGjMIGdMQswCQYDVQQGEwJVUzEQMA4GA1UECBMHRmxvcmlk YTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3RlbXMgTExDMRwwGgYD VQQDExNDdWRhIFN5c3RlbXMgTExDIENBMS8wLQYJKoZIhvcNAQkBFiBjdXN0b21lci1zZXJ2 aWNlQGN1ZGFzeXN0ZW1zLm5ldAIBCDAJBgUrDgMCGgUAoIICOzAYBgkqhkiG9w0BCQMxCwYJ KoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNDAzMTkxMDU5NDlaMCMGCSqGSIb3DQEJBDEW BBS1OPnAj/kRF+VQafXhr2nsfOL4bzBsBgkqhkiG9w0BCQ8xXzBdMAsGCWCGSAFlAwQBKjAL BglghkgBZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3DQMCAgFA MAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMIG0BgkrBgEEAYI3EAQxgaYwgaMwgZ0xCzAJBgNV BAYTAlVTMRAwDgYDVQQIEwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoT EEN1ZGEgU3lzdGVtcyBMTEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExLzAtBgkq hkiG9w0BCQEWIGN1c3RvbWVyLXNlcnZpY2VAY3VkYXN5c3RlbXMubmV0AgEIMIG2BgsqhkiG 9w0BCRACCzGBpqCBozCBnTELMAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExEjAQBgNV BAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1zIExMQzEcMBoGA1UEAxMTQ3Vk YSBTeXN0ZW1zIExMQyBDQTEvMC0GCSqGSIb3DQEJARYgY3VzdG9tZXItc2VydmljZUBjdWRh c3lzdGVtcy5uZXQCAQgwDQYJKoZIhvcNAQEBBQAEggIAecWwGzmEMsHKMSm/y+ampTnvCkRC 5bbA/mvnYU33noUCxYZSWHSj4Nnwzlr5AKUCJXElvIBGaQl/81dg/Ssp2qkRy9SwxI0Fm6ar StyXsst0LR9i2Xwm+9Utn/T8JyflL8hM8M14dFbrU+bjI7MblEIgJtArl1yLiZnt1O5klj+W KQ661ldu/5f9NyD6DHA2q1jf409fJusOd6m32DxZmhS0TobD5IGR/P442tnnQxNfVDxcCMLi B0jDTpExgBYcBl1qdzK1bJ7J6KSgPbV3lYSrowJIRB6gaXJlXgJuIssnfcn1RL2CuIB3SUQK ArhZ+6hUswREbXXp3CX5ZvlBUlN8oScIWGhZpJlxtVpRJj5GW4fpnOL7/vltJhbrcEG5G/hd LjmBoqy2p5DdEhSyxyTazE8wxmw+0wgJYx0rqBeQ+kot5gnl3Y2/IBOm4QW0a9s6Mq0W0GO1 9ybkB8TCJZ186g9JNJFFS1dNpEWn+NHiQV+TVpDGZFuD4jAs7R1wdP/YFvc6xjvjI10ERjEc +i2LsfRSDEn7tbPFsa8W6BzDFNTMPpMN6qUqxr93TrZihMplHIscHZetZJNVZsePuvn3+JEA 1XdCz2UnRP2qdruIxnpRqAtALL/PUufcxutMuK6v9pqjryoF+CrgheFRPr0qVAHw7oS9e2aO E9jjPy0AAAAAAAA= --------------ms030305030907060707060007--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?532978A5.3050707>