Date: Fri, 30 May 2008 19:07:06 -0700 From: "Brad Penoff" <penoff@cs.ubc.ca> To: "Joshua Isom" <jrisom@gmail.com> Cc: Bill Moran <wmoran@potentialtech.com>, freebsd-questions@freebsd.org Subject: Re: resident memory limit Message-ID: <b89c3c270805301907x6f0b3a24n464ef533e4da99b0@mail.gmail.com> In-Reply-To: <b41a96fb2bc6a8090ea3ceefc930392e@gmail.com> References: <b89c3c270805201320i36955dccv9c643543e1b3c291@mail.gmail.com> <20080520165437.36413c46.wmoran@potentialtech.com> <b89c3c270805201439nf8adbd4s954a80bb82c72ba@mail.gmail.com> <b89c3c270805210143l2019442am233aeeb8748fc776@mail.gmail.com> <20080521090919.e3e777ec.wmoran@potentialtech.com> <b89c3c270805211027o12fa20c8jb3fd6a635e764888@mail.gmail.com> <20080521133645.e327ffdf.wmoran@potentialtech.com> <b41a96fb2bc6a8090ea3ceefc930392e@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, May 21, 2008 at 11:46 AM, Joshua Isom <jrisom@gmail.com> wrote: > > On May 21, 2008, at 12:36 PM, Bill Moran wrote: > >> In response to "Brad Penoff" <penoff@cs.ubc.ca>: >>> >>> On Wed, May 21, 2008 at 6:09 AM, Bill Moran <wmoran@potentialtech.com> >>> wrote: >>>> >>>> In response to "Brad Penoff" <penoff@cs.ubc.ca>: >>>> >>>>> On Tue, May 20, 2008 at 2:39 PM, Brad Penoff <penoff@cs.ubc.ca> wrote: >>>>>> >>>>>> On Tue, May 20, 2008 at 1:54 PM, Bill Moran <wmoran@potentialtech.com> >>>>>> wrote: >>>>>>> >>>>>>> In response to "Brad Penoff" <penoff@cs.ubc.ca>: >>>>>>>> >>>>>>>> I have an application that runs on Linux or Mac OS X but seems to >>>>>>>> have >>>>>>>> a problem when I run on FreeBSD (6.3 or 7). The issue is the memory >>>>>>>> footprint for the application (osubw_sctpclien below) is quite >>>>>>>> large; >>>>>>>> on Linux it can be as much as 950 MB in resident memory, according >>>>>>>> to >>>>>>>> top. However, on FreeBSD I start to get ENOMEM always around the >>>>>>>> time >>>>>>>> my resident memory size is about 200 MB. >>>>>>>> >>>>>>>> I read a few posts and have seen people fixing their problems by >>>>>>>> adjusting kern.maxdsiz in /boot/loader.conf and/or by adding a swap >>>>>>>> file. I've tried both and for my application, it still seems to be >>>>>>>> limited to 200 MB resident memory regardless of maxdsize and swap >>>>>>>> file >>>>>>>> setting. I wrote a toy application (malloctest below) that calls >>>>>>>> malloc in a while(1) and breaks once it gets ENOMEM (doing another >>>>>>>> while(1) so it doesn't exit); this application's memory size in top >>>>>>>> always matches the kern.maxdsiz setting, however it has a very low >>>>>>>> resident memory number, according to top. >>>>>>> >>>>>>> Have a look at /etc/login.conf and the associated man pages. >>>>>>> >>>>> >>>>> BTW, we've seen the exact behavior on FreeBSD 7 as well (6.3 was >>>>> reported here). We've tried on different hardware as well, and keep >>>>> getting haunted by this resident memory limit that we don't know how >>>>> to set. >>>>> >>>>> Any idea why, in the data I originally reported, I can allocate >>>>> kern.maxdsiz + swap (see SIZE from top output) for malloc(1 MB) in a >>>>> while loop, yet the top value for RES is always really low? >>>>> >>>>> How come, in contrast, my application starts to report ENOMEM when >>>>> SIZE is 203 MB and RES is 201 MB? This is why I titled the thread >>>>> asking about an unknown (to me ;-) limit for resident memory... >>>> >>>> It's called memory overcommit. If the OS thinks it _might_ be able >>>> to get you the memory, it will allow it. You only actually use the >>>> memory when you start putting data in it (hence the difference between >>>> SIZE and RES) Add a statement to fill up the malloc()ed memory with >>>> some sort of data in your loop, and you'll see different behaviour. >>>> >>>> As to what's limiting your application, I'm not sure. What does the >>>> output of 'ulimit -a' say? >>>> >>> >>> Thanks again for your time. >>> >>> >>> With the default loader.conf, my "limit -a" output is: >>> >>> Resource limits (current): >>> cputime infinity secs >>> filesize infinity kB >>> datasize 524288 kB >>> stacksize 65536 kB >>> coredumpsize infinity kB >>> memoryuse infinity kB >>> memorylocked infinity kB >>> maxprocesses 5547 >>> openfiles 11095 >>> sbsize infinity bytes >>> vmemoryuse infinity kB >>> >>> My application starts getting ENOMEM when I have 201 MB of resident >>> memory. >>> >>> >>> >>> When I change my loader.conf to match the 2 GB of physical memory that I >>> have: >>> kern.maxdsiz="2147483648" >>> kern.maxssiz="2147483648" >>> kern.dfldsiz="2147483648" >>> >>> ...and reboot, then my "limit -a" output is: >>> >>> Resource limits (current): >>> cputime infinity secs >>> filesize infinity kB >>> datasize 2097152 kB >>> stacksize 2097152 kB >>> coredumpsize infinity kB >>> memoryuse infinity kB >>> memorylocked infinity kB >>> maxprocesses 5547 >>> openfiles 11095 >>> sbsize infinity bytes >>> vmemoryuse infinity kB >>> >>> >>> However, the application still seems to max out at 201 MB of resident >>> memory. >>> >>> >>> People suggest to fix my login.conf but the memory related fields are >>> set to unlimited... Any ideas where this 201 MB limit of resident >>> memory comes from? >> >> That's pretty strange. If I had to guess, I would guess that there is no >> 201M limit, but that you're hitting some other limit that just happens to >> predictably occur at 201M with that program. >> >> I'm kind of grasping at straws here, so hopefully I won't lead you on a >> wild goose chase, but I would look next at putting some debugging in >> /etc/malloc.conf and seeing if you get any useful information from it >> (see the malloc man page). From there, possibly a ktrace of the process. >> Hopefully you have source code for the program and can compile it with >> debugging and run it under gdb. >> >> -- >> Bill Moran >> http://www.potentialtech.com >> _______________________________________________ >> freebsd-questions@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-questions >> To unsubscribe, send any mail to >> "freebsd-questions-unsubscribe@freebsd.org" >> >> I'm still playing around with malloc.conf and ktrace, searching for answers (I'll report if I find any) but in the meantime, I'll try to answer your questions... > > Here's a question I haven't seen asked yet. How much memory is it trying to allocate? If it can't get everything it's asking for it can fail. This same application runs on both Mac OS X and Linux. On Linux, the maximum memory demands shown in "top" were VIRT=955MB RES=929MB. This was actually on a machine with only 1 GB of RAM. This same program starts getting ENOMEM on FreeBSD on a machine with 2 GB of RAM when RES is 201 MB; the same occurs on FreeBSD on a machine with 1 GB of RAM (same HW as Linux). On FreeBSD, I can run a dummy program that calls malloc in a loop essential until the value of SIZE on "top" (same as VIRT in Linux) is kern.maxdsiz (which I set to 2GB in loader.conf). > Also, how is the application being started? There could be some setting in the shell startup that's putting a limit. I run it as root which uses /bin/csh . I did a chsh as root to bash and retried the same app but still reached the same magic 201 MB limit. > Is it a native FreeBSD program or is it a linux executable running under simulation? It's a native FreeBSD program. brad
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?b89c3c270805301907x6f0b3a24n464ef533e4da99b0>