Date: Fri, 26 Mar 1999 05:46:42 -0800 From: David Greenman <dg@root.com> To: graeme@echidna.com Cc: freebsd-questions@FreeBSD.ORG Subject: Re: Caching of large files in memory Message-ID: <199903261346.FAA01367@implode.root.com> In-Reply-To: Your message of "Fri, 26 Mar 1999 07:47:23 EST." <36FB81DB.10E5760B@echidna.com>
next in thread | previous in thread | raw e-mail | index | archive | help
What is probably happening is that there is not quite enough memory to cache the whole file. Even if you're 1 page (4K) short, if you're reading the entire file then the LRU replacement will reclaim the oldest read (first part) of the file in order to cache the end of the file. When you go to read the file again and the system wants to cache the first part of the file, then the next chunk is reclaimed to cache it, ultimately resulting in it needing to be re-read again. ...sort of like the system chasing it's tail but never being able to catch it because there isn't enough memory to contain the whole thing. The solution? Memory is cheap, so I'd buy more. :-) -DG David Greenman Co-founder/Principal Architect, The FreeBSD Project >I have a moderately busy web server (2.2.7S/Apache - a few hundred thousand hits >a day). It has been running fine with 128MB of memory. > >Recently the memory was expanded to 256MB, and about the same time, we started >running a task that scans a large file (about 112MB) on each execution. This >task also requires reading two smaller files of 12 and 20MB. So that's a total >of 144MB, but there's an extra 128MB available, and the system seemed to cache >the smaller files OK with 128MB memory. > >At first, it was clear all the files were being cached in memory. In the first >execution after an idle period, the files might be read from disk, but then no >disk activity was evident on repeated executions. > >However, after the server had been running about a day, things shifted so that >the large file is evidently being read more-or-less entire from disk every time. >This is true even after repeated, frequent reads of the file. Since the time the >server was rebooted, named has been building a cache from reverse name >resolution (about 8MB now, it will grow bigger yet), but otherwise I'm not aware >of any changes. > >Presently top shows > >Mem: 90M Active, 46M Inact, 26M Wired, 87M Cache, 8318K Buf, 784K Free > > >I'm wondering if anything can be done to encourage caching of the large file. > >I'm also curious as to why the system seems to read the large file entire, >rather than caching at least some major part of it. > > >I do have the option of breaking the large file into smaller pieces, the largest >of which would be about 12MB. I'm wondering if this was done, and all the pieces >were accessed frequently, would the system then cache most or all of these files >in memory? > > >Alternatively, I'd be prepared to expand memory still further, but only if I >could be sure it would result in the files being reliably cached. > > >-- >Graeme Tait - Echidna >graeme@echidna.com > > > >To Unsubscribe: send mail to majordomo@FreeBSD.org >with "unsubscribe freebsd-questions" in the body of the message To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-questions" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199903261346.FAA01367>