Date: Thu, 23 Sep 1999 09:33:08 -0500 (CDT) From: Kevin Day <toasty@dragondata.com> To: dillon@apollo.backplane.com (Matthew Dillon) Cc: toasty@dragondata.com (Kevin Day), dcs@newsguy.com (Daniel C. Sobral), hackers@FreeBSD.ORG Subject: Re: Idea: disposable memory Message-ID: <199909231433.JAA61714@celery.dragondata.com> In-Reply-To: <199909230904.CAA23644@apollo.backplane.com> from "Matthew Dillon" at Sep 23, 1999 02:04:22 AM
next in thread | previous in thread | raw e-mail | index | archive | help
> > > :> > Thoughts? > :> > :> man madvise? > :> > : > :Yeah, but MADV_FREE doesn't really do what I need. I have no idea if the > :system actually did free my ram or not. I want to hang on to the data, but > :if more ram is needed, then it can be discarded, but I need to know that it > :did, so that I can recreate it. Checking every time I blit an object to see > :if the page is zero'ed won't work. > : > :Kevin > > madvise ... MADV_DONTNEED is what you want. The data will remain mapped > until the system reuses it, at which point it reverts to zero-fill. > > The system will reuse the data fairly quickly, even if the system is > not all that loaded. > > You can lock the page back in simply by writing to something in the page. > > The system implements this madvise feature by marking the pages clean. > If you happen to write to the page before the system reuses it, it of > course gets redirtied. If you don't and the system reuses the page, > it goes bye bye (turns into zero-fill) from the point of view of your > process. > Either I'm not properly explaining what I want, or I'm misunderstanding you guys. :) Let me try to rexplain. The way things work now for us is that I mmap() decompressed video files in, and send them straight to our blitter. This works well since the system will nicely keep as much as is possible in ram, and rather than swap this out, it'll just discard what it is in memory if more ram is needed. When I need that bit of data again, it's fetched back off the disk for me. I'm now playing with compressed data streams. The decompression is slow, so I'd like to cache the *decompressed* version of these files. I end up allocating large amounts of ram in one process to cache the decompressed data. This is a disavantage over the above scenario, since now the system swaps out my decompressed data when more ram is needed elsewhere. Swapping out then swapping back in my decompressed data is about 4x slower than just re-reading my compressed stream and decompressing it again. Why don't I just allocate a predefined amount of memory and use that for a cache all the time? Most of the time we have about 20MB free on our system. Sometimes we end up with about 2MB free though, and what's happening now is that I start paging out data that I could recreate in less time than the page-in/page-out takes. What I want is to maintain a largish in-memory cache of decompressed data, and tell the kernel that "I'm still using this memory, but if memory runs short somewhere else, you're allowed to take it back from me. This ram has approximately the same priority as data in the disk cache". However, since I could decide to re-use anything in my cache at any time, I *need* to know if the kernel took some of this back from me, so I can then recreate it when needed. I could see two ways (from a userland perspective) of doing this. Either I have to check (somehow) every time I use a region of ram to make sure it's still there by some syscall, or the kernel somehow tells me that I need to get rid of x amount of ram, and it's up to me to decide what I want to throw away. I don't think MADV_FREE is what I want, since it makes my memory go away very quickly, *and* I have no way of knowing that the kernel did it. Make any more sense? Kevin To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199909231433.JAA61714>