Date: Mon, 29 Jul 2002 20:55:31 -1000 From: Clifton Royston <cliftonr@lava.net> To: Anders Nordby <anders@FreeBSD.org> Cc: hackers@FreeBSD.ORG Subject: Re: squid and datasize kernel problems Message-ID: <20020729205531.B29846@lava.net> In-Reply-To: <bulk.43165.20020729205438@hub.freebsd.org>; from owner-freebsd-hackers-digest@FreeBSD.ORG on Mon, Jul 29, 2002 at 08:54:40PM -0700 References: <bulk.43165.20020729205438@hub.freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, Jul 29, 2002 at 08:54:40PM -0700, freebsd-hackers-digest wrote: > Date: Mon, 29 Jul 2002 15:51:08 +0200 > From: Anders Nordby <anders@FreeBSD.org> > Subject: Re: squid and datasize kernel problems (was: openoffice stack and datasize kernel problems) > > On Sat, Jun 08, 2002 at 10:40:54AM -0700, Matthew Dillon wrote: > > <options MAXDSIZ="(1024*1024*1024)" > > <options MAXSSIZ="(256*1024*1024)" > > > > But it doesn't make much sense. A 64 megabyte stack ought > > to be plenty, as should the default 512MB data size. > > > > It should be noted that mmap() uses whatever VM space > > remains after MAXDSIZ and MAXSSIZ have been reserved, so > > increasing MAXDSIZ reduces the amount of VM available > > to mmap(). Still, a 1 GB MAXDSIZ should not result in > > system utilities / servers running out of mmap() space! > > Userland has 3G of VM space to play with. > > I'm running into some similar issues when trying to make Squid eat as > much as possible of the 4 GB memory I have installed in a Compaq > ProLiant DL 380 G2. At first, Squid seems to die and restart when trying > to allocate memory above 512 MB. By tuning MAXDSIZ, I have made it use > up to around 2 GB. If I set MAXDSIZ (I now do it in loader.conf with > kern.maxdsiz) above around 2950 MB, init starts failing upon boot: > > init in malloc(): error mmap(2) failed, check limits > init in malloc(): warning: recursive call > > Does anyone have any clues on how to overcome this? I'll be trying out > the the dlmalloc library that is distributed with Squid, but I suppose > I do need a 4 GB maximum data size to be able to make Squid actually use > 4 GB. Is this possible, or am I being totally foolish? > > Any hint very appreciated. :-) I just this weekend got done upgrading a 3.5-current box to RELENG-4.6.1, to run Squid on. Also upgraded to the latest Squid from /usr/ports, a big step up from the old version we were running. One of the reasons I originally picked FreeBSD for the Squid server, based on advice from the gurus on the Squid users list is because the unified memory management in FreeBSD makes *all* unallocated RAM available for disk buffering, so that you benefit from the full amount of RAM even if it's not allocated to the Squid process. As of a couple years ago, the best advice on Squid was to *not* try to make it buffer everything in the squid process's RAM, but let it do its two-tiered RAM/disk caching with a "moderate" amount of RAM (e.g. 128MB) and let the OS decide what "disk" content to buffer with the remainder. I doubt that's changed. Squid performance is also claimed to benefit substantially (up to 4x throughput) from choosing the diskd storage type. However, for this to work you'll need to rebuild your kernel to expand the Sys V message queues and Sys V shared memory allocation to some expanded voodoo values. See <http://www.squid-cache.org/Doc/FAQ/FAQ-22.html#ss22.6>; and <http://www.squid-cache.org/Doc/FAQ/FAQ-22.html#ss22.7>; I did this yesterday afternoon and it's looking good so far under light usage. Also enable softupdates, and mount your cache drives with noatime to help performance. For more help, try www.squid-cache.org and ask on the squid-users mailing list at: <http://www.squid-cache.org/mailing-lists.html>; which used to have plenty of Squid experts and Squid-on-FreeBSD experts; I'm far from either. -- Clifton -- Clifton Royston -- LavaNet Systems Architect -- cliftonr@lava.net "What do we need to make our world come alive? What does it take to make us sing? While we're waiting for the next one to arrive..." - Sisters of Mercy To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20020729205531.B29846>