Skip site navigation (1)Skip section navigation (2)
Date:      Thu, 12 Apr 2007 14:32:06 +1000 (EST)
From:      Bruce Evans <bde@zeta.org.au>
To:        Jung-uk Kim <jkim@freebsd.org>
Cc:        freebsd-emulation@freebsd.org, Andriy Gapon <avg@icyb.net.ua>, freebsd-ports@freebsd.org, Juergen Lock <nox@jelal.kn-bremen.de>
Subject:   Re: kqemu+amd64: fpudna in kernel mode
Message-ID:  <20070412132450.K66414@delplex.bde.org>
In-Reply-To: <200704111307.17455.jkim@FreeBSD.org>
References:  <461CFD4B.4020801@icyb.net.ua> <200704111307.17455.jkim@FreeBSD.org>

next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, 11 Apr 2007, Jung-uk Kim wrote:

> On Wednesday 11 April 2007 11:22 am, Andriy Gapon wrote:
>> I get swarms of messages "fpudna in kernel mode" in system log when
>> I run qemu with kqemu on FreeBSD 6.2 amd64 host with Windows XP
>> 32-bit guest. I noticed that someone reported this problem a while
>> ago:
>> http://lists.freebsd.org/pipermail/freebsd-emulation/2006-February/
>> 001800.html
>
> This happens because we do not allow FP instructions in kernel mode as
> the error message suggested.  Executing FP instructions in kernel
> mode requires expensive save/restore cycles for FP states.

Yes, kqemu is violating an invariant.  This should cause a panic, but
only causes a printf.  In my kernel, it causes a debugger trap for the
kernel T_DNA in trap() and sometimes a panic in npxdna() when a
related invariant is violated, but I don't use kqemu and have never
seen these happen.  The panic in npxdna() rotted to a printf plus
bogus error handling in rev.1.131 of npx.c.

There may be no problem with kqemu, but it is or was hard to tell,
since sources were unavailable and no one provided debugging info.  I
think kqemu doesn't use the FPU directly, but in does a software
interrupt (int $0x10) to get to the T_DNA trap handler to emulate the
corresponding trap (int $0x10 not in the instruction stream) that
userland accessing the (not available) FPU would cause.  This would
give a perfect emulation in a fairly MI way if the software interrupt
were actually supported, but even then it is slower than calling a
(very MD) function that does the trap handling.  In FreeBSD, calling
npxdna() (fpudna() on amd64) directly would almost work, since npxdna()
has no args -- in particular it doesn't depend on the trap frame.  It
does some (slightly broken due to races) invariants checking, so any
hackish direct calls to it would have to avoid calling it if the
invariant are not satified.  The invariant is just that T_DNA doesn't
happen if the thread already owns the FPU.  In that case, kqemu shouldn't
be doing a real or fake DNA trap to force the thread to own the FPU.
Note that checking that the thread owns the FPU is nontrivial due to
interrupts, and that even if it is done correctly, there is a race
with interrupts (ithreads or preemptions trigger by interrupts) after
becoming the owner, so there is no guarantee that a thread owns the
FPU after doing a real or fake T_DNA to (try to) force the thread to
own the FPU -- the ownership may change immediately after it is gained,
before returning to userland or whatever actually uses the FPU.  This
is normally handled by simply repeating the T_DNA trap until the race
is not lost.  An emulator might need be careful with interrupts to
get or implement the normal handling.  The "slightly broken due to
races" invariants checking in npxdna() is from not being careful with
interrupts.  Hmm, rev.1.131 made this more than slightly broken, since
when the invariant PCPU_GET(fpcurthread) != NULL is violated due to
PCPU_GET(fpcurthread) being curthread, the fixup does a delicate
state change without keeping interrupts masked, so the new invariant
PCPU_GET(fpcurthread) == curthread that goes with the T_DNA trap being
turned off may be violated just before we turn the T_DNA trap off.

Apart from the races, I think doing int $0x10's in the kernel is safe
if and only if it is done near the syscall level.  Then the result is
the same as if userland did T_DNA's just before entering the kernel.
(BTW, note that userland can't do software int $0x10's since these are
privileged.  AFAIR, there is no way to deny gate privilege via software
interrupts to the kernel (ring 0).  This usually isn't a problem since
making software interrupts in the kernel is just foot shooting.)

>> As I understand "fpudna" means "FPU device not available" and the
>> message comes from here:
>> http://fxr.watson.org/fxr/source/amd64/amd64/trap.c
>>
>> on T_DNA ("device not avaialble") _kernel_ fault.
>
> You can ignore the complaints for now until we come up with better FP
> handling in kernel.

He won't be able to ignore them when the handling is to panic :-).
The other message "fpudna: fpucurthread == curthread %d times" is
something to worry about now.  Just printing that message enlarges the
race window significantly.

> Windows guest is not supported by full virtualization mode, AFAIK.
>
> http://www.qemu.net/kqemu-doc.html
>
> BTW, it doesn't give you much performance gain as you may think:
>
> 'Best performances are achieved with Linux 2.4 kernels. Linux 2.6
> works but the performance gains are small.'
>
> Even if we make it work with FreeBSD, I would expect similar results.

Is that because 2.6 is already better?  Is it a 2.6 host or target?

Bruce



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20070412132450.K66414>