Date: Thu, 6 Dec 2001 20:55:48 +0100 (CET) From: Rodolphe Ortalo <ortalo@laas.fr> To: Terry Lambert <tlambert2@mindspring.com> Cc: Nicolas Souchu <nsouch@fr.alcove.com>, freebsd-hackers@freebsd.org, KGI Devel <kgi-develop@lists.sourceforge.net> Subject: Re: kld VM pager Message-ID: <Pine.LNX.4.21.0112061909180.384-100000@tempest.rod.fr> In-Reply-To: <3C0F9B0D.4640D043@mindspring.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, 6 Dec 2001, Terry Lambert wrote:
> Nicolas Souchu wrote:
> > > From my reading of the code, all it is is a pseudo device that
> > > permits you to establish/remove memory mappings for various regions
> > > of the graphics card memory, which may or may not be apertured,
> > > into process address space, as a file buffer mapping.
> [ ... ]
> > As you consider it later, the VM is supposed to move the aperture window
> > if the hardware does not support linear frame buffering (VGA 64k window
> > for example). This is a minimal feature for the so-called mmio resources.
> > Some other kind of resources may be mapped as well, especially acceleration
> > resources like DMA, graphic pipelines... to provide the user application a
> > uniform API (memory mapping) for any kind of hw capacities.
>
> If the intent is just to turn non-linear into linear from the
> perspective of an applicaiton, and the boundary is always 4K
> or more, and it's not an access aperture that needs the card
> to be sent a control message to expose a different region in
> the host visible window, then you just need to make a bunch
> of different mappings to make non-linear physical memory into
> linear virtual memory.
In this case (not the general case), you usually *also* need to update
some hw registers (send control message in your wording) in the graphic
board chipset to have it make available the specific (e.g. 4k) card memory
area. (Sort of paged->linear mapping.) So the actual card driver needs to
register a fault handler and the kernel subsystem (KGI "driver") should
call it when needed. Needless to say that this is really bad for
performance and mostly useful only for old chipsets (e.g. to have a VGA
board present a linear 256ko memory area to applications via a 32k
window...). Definitely not the general case nowadays...
> I'm not sure if I understand you, though. Is all of the card
> memory visible in physical memory, or is only part of the card
> memory visible in physical memory? Forget virtual for a second.
Most of the time, with modern and common PC graphic cards, the entire card
physical memory can be visible in physical memory. However, this is a
modern evolution, old or advanced graphics hardware usually plays *a lot*
with the graphic memori(es) adressing wires.
Also the (KGI-)kernel (sometimes) would like to restrict the accessible
range, possibly *with* hardware help (IIRC advanced graphic hardware
provide hw control for windows, or stereo displays).
> If only a part of the card memory is visible in physical memory,
> then you will have to do fault handling, as I first suggested.
>
> FreeBSD doesn't support external pagers, per se. You might be
> able to hook it in, but really, it's not expected. You really
> want to look at /sys/vm/device_pager.c.
IMHO, this is not the most general case, especially for framebuffer
accesses which are usually linear<->linear (ie classical) mappings.
*However*, VGA-only drivers are still extremely useful as most cards are
VGA-compatible; e.g. at boot time on an otherwise unsupported card... IIRC
our basic VGA (especially VGA-text) driver takes advantage of this kind of
mechanism (SetOffset hook in the exported memory resource).
> The main problem, if there is really an aperture you have to
> move around, is that there's an implicit asumption that you have
> all the memory mapped, and the "put" is as simple as just
> writing to memory, so you aren't going to get the write fault
> that gets handled by a fault handler that you need.
[...]
> I find it incredibly hard to believe that anyone would build a
> video card that would be so slow to access, but I could easily
> be wrong here.
VGA text mode, even incredibly slow, is still much more convenient than a
black screen (well, unless you can have a real VT100 somewhere and the
ability to configure it with the eyes closed :-).
But you are right, this is not the general case. Let's summarize the
various kind of MMIO resources available via KGI: PAGED_PAGED,
LINEAR_PAGED, PAGED_LINEAR and LINEAR_LINEAR.
Amon all of them, the LINEAR_LINEAR case is clearly the simplest and most
useful one. However, even if the other mappings are tricky and potentially
inefficient, in the past, we have found useful to have them available:
that's why they were implemented.
The example which come to my mind is usually the VGA-text case: provide
access to 256k of VGA memory via a single 32k window. It allows several
things: have 4 VGA-compatible boards running (the 32k "window" is usually
in a fixed adress range: 0xA0000-0xBFFFF), have simple access to the text
font ("upper" part of the 256ko memory, modulo some mangling that the
driver does silently) and finally let the console code access a 32k (or
64k) area as a linear area.
I'm sure other cases could arise - as I said graphics hardware tends to
play a lot with the memory adressing and I would not be surprised it
they invented something new in the future...
[...]
> > But, I don't want any kind of buffering. /dev/graphic is just a Unix-like
> > file entrypoint to pass orders to the VM.
>
> You have buffering, anyway, if you have an aperture mapping that
> is changing. Minimally, you have to take a fault on the write
> to any memory not currently exposed in the aperture, remap the
> aperture, and then restart the operation with the fault having
> been satisfied. So you will need to remap the pages as having
> or not having "backing store" in the form of memory within the
> aperture window.
That's the kind of thing that we need I think. Is this what you call
"implenting an external pager"?
[...]
> > What would handle my VM faults in this case?
>
> It would have to be your code, remapping the aperture.
>
> Do video cards like this really exist?
For the "main" (framebuffer )memory, not much anymore. But once we started
implemented the various MMIO mappings I mentionned earlier, we started to
use memory (re)mapping for other things to provide applications access to
cards resource as-if they were available in conventional physical memory.
For example:
* the 256x3 bytes palette area (usually inside the DAC subsystem,
readable and writable, but not directly addressable by the CPU);
* the cursor picture (on some boards it is in the fb area, on some others
it is stored in special memory in the Ramdac);
* some monitors have an internal gamma correction lookup table;
* stereo devices and some OpenGL cards provide hw switching between
buffers (but the application usually wants to have both of them mapped in
memory);
* on-board microcode (the programmable WARP setup engine on the Matrox-es
for example, well its ucode is undocumented for the moment so it's
not urgent...).
Finally, as the VM fault handler is potentially the fastest way to
transfer data from userspace to kernelspace, we also used it to provide a
"virtual write-only FIFO" to applications, to allow them to send display
lists (graphics commands) to the hardware as fast as possible without
incurring the overhead of copying from userspace. With this accelerator
FIFO abstraction, we map a page in userspace and the application fills it
linearly, then it faults at the end of this page. KGI unmap the page,
sends it to the driver for (asynchronous) execution, and remap another
fresh page to the userspace program to replace the stolen one so that the
application can continue to push display lists for execution (until some
limit on the number of allowed in-process pages is hit).
In this case, the backing store of the memory is conventional memory
(well, DMA- or AGP-capable); but, from the point of view of the
application, this is write-only memory (content is not preserved).
Maybe this is not the ideal system (it surely looks like a hack when you
dig the details) - in fact there were several discussions on the KGI list
to replace this system with something different. But it's running now, and
IMO, pretty decently (to be honest, I may be the only one to think this
except the original author). Alternatives are welcome, but they would
require more work from our part (not a problem btw).
In any case, all these are derivates usages, in the first place access to
VM fault handlers is needed for the PAGED_{LINEAR,PAGED} cases of MMIO
mapping that I mentionned initially. Again, not useful for many boards,
hopefully less and less useful in the future, but unfortunatly experience
showed that it is useful.
[...]
> I think you will need to understand what the code wants before
> you ask for help. ;^).
Of course. :-) But given the intricacies of graphics hardware, it also
helps to understand first the available "hooks" existing in the operating
system to try to take advantage of them without reshaping the whole VM
subsystem. (OS designers and implementors usually do not like very much
when one starts hacking everywhere in their virtual memory management code
to add "convenient" hooks; I don't understand why... :-)) )
> > > what Linux does to achieve the effect, assuming that the examples
> > > aren't sufficient by themselves. Let one of us know; I'm personally
> > > very intersted in seeing this get into FreeBSD, considering 386BSD
> > > first started working on something like this back at the end of 1992.
> >
> > You're speaking about this kind of VM handling or KGI?
>
> KGI.
TNX. And thank you very much for your answers to Nicolas btw. I hope we
will deserve your interest.
Personnally, I do not think we are ready yet for kernel inclusion (any
kernel). But, with long time attempts, we have identified several core
mechanisms that *seem* to be needed from an OS to provide a pretty general
framework for graphic cards driver programming.
If OSes can offer the framework for us to implement these mechanisms, and
if designers of these OSes generally think that such framework or such
mechanisms do not have an overall negative impact on the OS, we may end up
with something useful...
Rodolphe
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?Pine.LNX.4.21.0112061909180.384-100000>