Date: Mon, 24 Mar 2003 01:57:32 -0800 From: "Andrew Kinney" <andykinney@advantagecom.net> To: freebsd-hackers@freebsd.org Subject: shared mem and panics when out of PV Entries Message-ID: <3E7E660C.10419.ECBF5C7@localhost>
next in thread | raw e-mail | index | archive | help
Hello, (very long message with background information on the issue follows) The machine discussed in this email is tracking RELENG_4_7. The machine is a well loaded web/mail/ftp server with dual Athlon MP 2000+ CPUs, 4GB of RAM, and 8GB of available swap space. The most it has ever swapped was 292KB (yes, KB not MB) and the CPUs are about 60% idle most of the time. I don't believe we're pushing the hardware all that hard given the specifications of the system. We're getting panics ("page fault" panic traced back to pmap_insert_entry) because we're running out of PV Entries. I've bumped up PMAP_SHPGPERPROC to 400, then 800, and then 1500 over the last several months in an attempt to solve this problem by increasing the PV Entries limit (per the warning in pmap_collect). Each time, we still bumped the limit and got panics due to running out of PV Entries. I've verified that maxed PV Entries are indeed the cause of the panics by logging sysctl vm.zone. Our PV Entry limit is now 11113502 (from sysctl vm.zone) and we're still running into this limit. With 1GB KVA space, I really shouldn't take PMAP_SHPGPERPROC much higher since each PV Entry takes 28 bytes of KVA space and with 11113502 PV Entries, that is nearly 300MB of KVA space used just for PV Entries. Also, from past experience, if you set PMAP_SHPGPERPROC too high, the system will not boot. I'm not 100% sure, but I believe that would happen right around PMAP_SHPGPERPROC=1600 from my calculations. Now, I could increase KVA space by way of increasing KVA_PAGES and presumably continue increasing PMAP_SHPGPERPROC until the problem goes away, but per a previous discussion, there are problems (related to pthreads) with increasing KVA_PAGES in FreeBSD 4.7. This has apparently been fixed in FreeBSD 4.8. Since moving to FreeBSD 4.8 isn't an option I can exercise in the near term, I'd like to approach this problem from a different angle and possibly solve it without needing to increase KVA_PAGES. In my opinion, increasing KVA_PAGES would only be a short term bandaid for the PV Entries issue. I'll eventually need to increase KVA_PAGES for a different reason, though, because sysctl vm.kvm_free does hit "0" sometimes after the system has been running for awhile, but I'll tackle that issue if/when it becomes a stability issue. The source of the PV Entry usage is shared memory in Apache (PHP and/or mod_perl). We know it's Apache memory sharing that causes the problem because nearly all PV Entries are freed when Apache is stopped and the problem is worst right after Apache starts when you have a lot of "clean" memory segments getting shared (10 to 11 million PV Entries used). After Apache has run for awhile and the shared memory segments become unshared due to them being "dirtied", the number of PV Entries used declines and settles in at a much lower number (3 to 5 million PV Entries used most days). Apache on this machine likes to hover right around 256 children processes, so you could see how requiring duplicate PV Entries per shared memory segment per Apache process could add up quite quickly. Under ideal circumstances, we'd correct our PHP/PERL code to avoid running into the issue at all, but this is customer code that we don't have much control over since we allow users to select and run their own CGI scripts without our intervention. So, for this instance, we have to solve it at the system level by bullet-proofing our system as much as possible. I've read that setting sysctl kern.ipc.shm_use_phys to "1" will cause shared memory segments not to use PV Entries at all. (See http://www.geocrawler.com/archives/3/159/2002/6/50/9031353/ ) However, in the real world with FreeBSD 4.7, setting kern.ipc.shm_use_phys to "1" seems to have no visible effect on the number of PV Entries consumed by Apache memory sharing. We're currently running sysctl kern.ipc.shm_use_phys=1 and are still seeing the heavy PV Entry usage after rebooting (sysctl set on boot) with no apparent difference in usage. My thoughts at this point are leaning towards setting some limits on shared memory (kern.ipc.shmmax, kern.ipc.shmseg, kern.ipc.shmall, and kern.ipc.shmmax sysctl's) to limit the number of PV Entries that Apache can consume by way of shared memory. Is that a viable approach to this problem? What kind of gotchas and caveats should I watch out for in taking this approach? Any recommendations on which one to tweak first? If I'm headed in completely the wrong direction to solve this PV Entries limit issue, I'd appreciate any alternative approaches to solving the problem that anyone is willing to offer. If you need any further information about our settings or usage, please let me know. I've been as thorough as possible in this email without getting overly verbose, but in complex issues like this I recognize that I may not have included all the information needed for the experts to make a fair assessment of the issue and suggest work arounds. Also, though I believe that the highly technical nature of this message made it suitable for posting to freebsd- hackers, if it would be better suited to a different list, please point me in the right direction. Thanks in advance for any advice or assistance you can offer. Sincerely, Andrew Kinney President and Chief Technology Officer Advantagecom Networks, Inc. http://www.advantagecom.net To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3E7E660C.10419.ECBF5C7>