Date: Tue, 25 Mar 2003 17:54:28 -0800 From: "Andrew Kinney" <andykinney@advantagecom.net> To: Igor Sysoev <is@rambler-co.ru> Cc: freebsd-hackers@FreeBSD.ORG Subject: Re: shared mem and panics when out of PV Entries Message-ID: <3E8097D4.22759.A3FC35@localhost> In-Reply-To: <Pine.BSF.4.21.0303251451040.476-100000@is> References: <3E7F0F65.7994.11617945@localhost>
next in thread | previous in thread | raw e-mail | index | archive | help
On 25 Mar 2003, at 17:56, Igor Sysoev wrote: > > So, what's the best approach to limiting memory shared via fork() or > > reducing PV Entry usage by that memory? Is there something I can do > > with the kernel config or sysctl to accomplish this? > > No, as far as I know there's no way to do it. > The irony is that you do not need the most of these PV entries because > you are not swaping. > My thoughts exactly. I suppose not all that many people run well used web servers with 4GB of RAM, so there wouldn't be any reason for this issue to come up on a regular basis. I'm going to expose my newbness here with respect to BSD memory management, but could the number of files served and filesystem caching have something to do with the PV Entry usage by Apache? We've got around 1.2 million files served by this Apache. Could it be that the extensive PV Entry usage has something to do with that? Obviously, not all are accessed all the time, but it wouldn't take a very large percentage of them being accessed to cause issues if filesystem caching is in any way related to PV Entry usage by Apache. I remember reading somewhere (sorry, didn't keep track of the link) that someone running a heavily used Squid proxy had a very similar issue with running out of PV Entries as they got more and more files in the cache. Squid is basically a modified Apache with proxy caching turned on. > I think you should try to decrease memory that shared between Apache > processes. If you can not change scripts then the single method is to > decrease number of Apache processes while keeping to handle the > current workload: 1) disable keepalive if it enabled; 2) set Apache > behind reverse-proxy server that frees Apache processes > as soon as proxy get the whole responses. We had keepalive set to the default of "on" (at least default for this install) with the default keepalive timeout of 15 seconds. Dropping the keepalive timeout down to 3 seconds has dramatically reduced the number of Apache processes required to serve the load. With the new settings, we're averaging 30 to 80 Apache processes, which is much more manageable in terms of memory usage, though we weren't anywhere near running out of physical RAM prior to this. We're servicing a little over 1000 requests per minute, which by some standards isn't a huge amount. We're still seeing quite heavy PV Entry usage, though. The reduced number of Apache processes (by more than half) doesn't seem to have appreciably reduced PV Entry usage versus the previous settings, so I suspect I may have been wrong about memory sharing as the culprit for the PV Entry usage. This observation may just be coincidence, but the average PV Entry usage seems to have gone up by a couple million entries since the changes to the Apache config. Time will tell if the PV Entries are still getting hit hard enough to cause panics due to running out of them. They're supposed to get forcibly recycled at 90% utilization from what I see in the kernel code, so if we never get above 90% utilization I guess I could consider the issue resolved. What other things in Apache (besides memory sharing via PHP and/or mod_perl) could generate PV Entry usage on a massive scale? Sincerely, Andrew Kinney President and Chief Technology Officer Advantagecom Networks, Inc. http://www.advantagecom.net
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3E8097D4.22759.A3FC35>