Date: Thu, 23 Sep 1999 10:31:17 -0700 (PDT) From: Julian Elischer <julian@whistle.com> To: Matthew Dillon <dillon@apollo.backplane.com> Cc: Kevin Day <toasty@dragondata.com>, "Daniel C. Sobral" <dcs@newsguy.com>, hackers@FreeBSD.ORG Subject: Re: Idea: disposable memory Message-ID: <Pine.BSF.3.95.990923102054.2329A-100000@current1.whistle.com> In-Reply-To: <199909231654.JAA28326@apollo.backplane.com>
next in thread | previous in thread | raw e-mail | index | archive | help
I think what is needed is something similar to what we used to use at TFS. A device driver that controled a large number of pages. it had ioclts to allocate 'buffers' from these pages. each buffer was given a handle by whichthe user process refered to it. multiple processes could refer to them. the kernel could dealocate them, in which case an attempt to use that handle in an ioclt (all operatiosn were via ioctls) would return an error and the user knew it was no longer valid. device drivers were taught to be able to take a UIO of physical addresses for IO, and ioctls were added to these devices which took a buffer handle as an argument. e.g. "write buffer 0x34823 to disk location 0x125434" (the length was implied by the length of the buffer..) ioctls included.. (protocol stack). send buffer xyz to address 12.34.56.78 (buffer device) allocate a buffer of size x. (buffer device) free buffer y (buffer device) allow discard of buffer z (can discard if short of ram) (disk driver) write buffer z to location A. (disk driver) read buffer z from location A (sized by buffer size) (there was also a protocol which delivered a received buffer as a result of a recvmesg() call in the auxhiliary data) there was reference counting so that a buffer could never be freed until all users had released it. (e.g. disk requwsts held a reference, as did send requests). you could also mmap buffers into address space. julian On Thu, 23 Sep 1999, Matthew Dillon wrote: > :I'm now playing with compressed data streams. The decompression is slow, so > :I'd like to cache the *decompressed* version of these files. I end up > :allocating large amounts of ram in one process to cache the decompressed > :data. This is a disavantage over the above scenario, since now the system > :swaps out my decompressed data when more ram is needed elsewhere. Swapping > :out then swapping back in my decompressed data is about 4x slower than just > :re-reading my compressed stream and decompressing it again. > : > :Why don't I just allocate a predefined amount of memory and use that for a > :cache all the time? Most of the time we have about 20MB free on our system. > :Sometimes we end up with about 2MB free though, and what's happening now is > :that I start paging out data that I could recreate in less time than the > :page-in/page-out takes. > > Hmm. Well, you can check whether the memory has been swapped out with > mincore(), and then MADV_FREE it to get rid of it (MADV_FREE'ing something > that has been swapped out frees the swap and turns it back into zero-fill). > That doesn't get rid of the swapout bandwidth, though. > > I think, ultimately, you need to manage the memory used for your cache > manually. That means using mlock() and munlock() to lock your cache into > memory. For example, choose a cache size that you believe the system > can support without going bonkers, like 5MB. mmap() 5MB of ram and > mlock() it into memory. From that point on until you munlock() it or > exit, the memory will not be swapped out. > > If the purpose of the box is to maintain the flow of video, then the cache > is a critical resource and should be treated as such. > > -Matt > Matthew Dillon > <dillon@backplane.com> > > :Kevin > > > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-hackers" in the body of the message > To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?Pine.BSF.3.95.990923102054.2329A-100000>