Date: Fri, 4 Dec 1998 14:28:29 -0600 (CST) From: Kevin Day <toasty@home.dragondata.com> To: dillon@apollo.backplane.com (Matthew Dillon) Cc: hackers@FreeBSD.ORG Subject: Re: Nonblocking page fetching Message-ID: <199812042028.OAA25409@home.dragondata.com> In-Reply-To: <199812041934.LAA16719@apollo.backplane.com> from Matthew Dillon at "Dec 4, 1998 11:34:35 am"
next in thread | previous in thread | raw e-mail | index | archive | help
>
> :I have an application where I'm streaming large amounts of data from disk,
> :and throwing it on the screen. (Playing a movie, essentially). I'm mmapping
> :the region that i'm playing, and just start memcpy'ing each frame into the
> :renderer's buffer. This is very time critical, so running out of buffered
> :data isn't good.
> :
> :I needed a way to trick the kernel into bringing pages off of disk without
> :making my process block waiting for them. Essentially, if I'm playing frame
> :10, I'd better already have frames 10-15 in ram, and should start bringing
> :frame 16 in, while 10 is playing. (I tried keeping a 4-6 frame buffer)
> :...
> :
> :What would be very nice is a syscall that could tell the vm system that a
> :page will be needed shortly, so bring it in as soon as possible. Sort of
> :like madvising with WILL_NEED, but a much stronger hint.
>
> The facility already exists in the system. In fact, it even exists in
> -stable.
>
> mmap() the region as per normal.
>
> madvise() the region to be MADV_WILLNEED
> madvise() the region to be MADV_SEQUENTIAL
>
> The kernel will read-ahead. Now, it will not read-ahead quickly enough
> to avoid blocking, but you may be able to tune the kernel specifically
> to handle your situation. Take a look at the code in vm/vm_fault.c,
> around line 351. Also look at the VM_FAULT_READ_AHEAD and
> VM_FAULT_READ_BEHIND defines.
>
> The key to making this work in a full-streaming application is to
> issue I/O for N pages ahead but to not fault-in one page in the center
> of that group. You then take a page fault when you hit that page, but
> since it is in the buffer cache it doesn't block and the kernel takes
> the opportunity to issue read-aheads for the next N pages (of which N/2
> already probably in the buffer cache). If you make N big enough, you
> are all set.
I tried this initially, and had a few problems:
1) madvise seems slow
2) I need a *huge* amount to be prefetched when we first start, but
the amount varies considerably from movie to movie. I need something tunable
on the fly. (N changes with use)
>
> Note, however, that disk I/O bandwidth is much less then memory bandwidth
> and probably much less then video-write bandwidth. If you have limited
> memory to buffer the data, you *will* block at some point.
It's not so much bandwidth, but latency. My movie's bandwidth is much less
than the disk bandwidth, i just can never stop to wait for things to be
brought in.
Kevin
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199812042028.OAA25409>