From owner-freebsd-emulation@FreeBSD.ORG Thu Apr 12 04:32:17 2007 Return-Path: X-Original-To: freebsd-emulation@freebsd.org Delivered-To: freebsd-emulation@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 4987516A407; Thu, 12 Apr 2007 04:32:17 +0000 (UTC) (envelope-from bde@zeta.org.au) Received: from mailout2.pacific.net.au (mailout2-3.pacific.net.au [61.8.2.226]) by mx1.freebsd.org (Postfix) with ESMTP id CAEEF13C4C3; Thu, 12 Apr 2007 04:32:16 +0000 (UTC) (envelope-from bde@zeta.org.au) Received: from mailproxy2.pacific.net.au (mailproxy2.pacific.net.au [61.8.2.163]) by mailout2.pacific.net.au (Postfix) with ESMTP id A64A410D33A; Thu, 12 Apr 2007 14:32:09 +1000 (EST) Received: from katana.zip.com.au (katana.zip.com.au [61.8.7.246]) by mailproxy2.pacific.net.au (Postfix) with ESMTP id 08EF42741C; Thu, 12 Apr 2007 14:32:12 +1000 (EST) Date: Thu, 12 Apr 2007 14:32:06 +1000 (EST) From: Bruce Evans X-X-Sender: bde@delplex.bde.org To: Jung-uk Kim In-Reply-To: <200704111307.17455.jkim@FreeBSD.org> Message-ID: <20070412132450.K66414@delplex.bde.org> References: <461CFD4B.4020801@icyb.net.ua> <200704111307.17455.jkim@FreeBSD.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-emulation@freebsd.org, Andriy Gapon , freebsd-ports@freebsd.org, Juergen Lock Subject: Re: kqemu+amd64: fpudna in kernel mode X-BeenThere: freebsd-emulation@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Development of Emulators of other operating systems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Apr 2007 04:32:17 -0000 On Wed, 11 Apr 2007, Jung-uk Kim wrote: > On Wednesday 11 April 2007 11:22 am, Andriy Gapon wrote: >> I get swarms of messages "fpudna in kernel mode" in system log when >> I run qemu with kqemu on FreeBSD 6.2 amd64 host with Windows XP >> 32-bit guest. I noticed that someone reported this problem a while >> ago: >> http://lists.freebsd.org/pipermail/freebsd-emulation/2006-February/ >> 001800.html > > This happens because we do not allow FP instructions in kernel mode as > the error message suggested. Executing FP instructions in kernel > mode requires expensive save/restore cycles for FP states. Yes, kqemu is violating an invariant. This should cause a panic, but only causes a printf. In my kernel, it causes a debugger trap for the kernel T_DNA in trap() and sometimes a panic in npxdna() when a related invariant is violated, but I don't use kqemu and have never seen these happen. The panic in npxdna() rotted to a printf plus bogus error handling in rev.1.131 of npx.c. There may be no problem with kqemu, but it is or was hard to tell, since sources were unavailable and no one provided debugging info. I think kqemu doesn't use the FPU directly, but in does a software interrupt (int $0x10) to get to the T_DNA trap handler to emulate the corresponding trap (int $0x10 not in the instruction stream) that userland accessing the (not available) FPU would cause. This would give a perfect emulation in a fairly MI way if the software interrupt were actually supported, but even then it is slower than calling a (very MD) function that does the trap handling. In FreeBSD, calling npxdna() (fpudna() on amd64) directly would almost work, since npxdna() has no args -- in particular it doesn't depend on the trap frame. It does some (slightly broken due to races) invariants checking, so any hackish direct calls to it would have to avoid calling it if the invariant are not satified. The invariant is just that T_DNA doesn't happen if the thread already owns the FPU. In that case, kqemu shouldn't be doing a real or fake DNA trap to force the thread to own the FPU. Note that checking that the thread owns the FPU is nontrivial due to interrupts, and that even if it is done correctly, there is a race with interrupts (ithreads or preemptions trigger by interrupts) after becoming the owner, so there is no guarantee that a thread owns the FPU after doing a real or fake T_DNA to (try to) force the thread to own the FPU -- the ownership may change immediately after it is gained, before returning to userland or whatever actually uses the FPU. This is normally handled by simply repeating the T_DNA trap until the race is not lost. An emulator might need be careful with interrupts to get or implement the normal handling. The "slightly broken due to races" invariants checking in npxdna() is from not being careful with interrupts. Hmm, rev.1.131 made this more than slightly broken, since when the invariant PCPU_GET(fpcurthread) != NULL is violated due to PCPU_GET(fpcurthread) being curthread, the fixup does a delicate state change without keeping interrupts masked, so the new invariant PCPU_GET(fpcurthread) == curthread that goes with the T_DNA trap being turned off may be violated just before we turn the T_DNA trap off. Apart from the races, I think doing int $0x10's in the kernel is safe if and only if it is done near the syscall level. Then the result is the same as if userland did T_DNA's just before entering the kernel. (BTW, note that userland can't do software int $0x10's since these are privileged. AFAIR, there is no way to deny gate privilege via software interrupts to the kernel (ring 0). This usually isn't a problem since making software interrupts in the kernel is just foot shooting.) >> As I understand "fpudna" means "FPU device not available" and the >> message comes from here: >> http://fxr.watson.org/fxr/source/amd64/amd64/trap.c >> >> on T_DNA ("device not avaialble") _kernel_ fault. > > You can ignore the complaints for now until we come up with better FP > handling in kernel. He won't be able to ignore them when the handling is to panic :-). The other message "fpudna: fpucurthread == curthread %d times" is something to worry about now. Just printing that message enlarges the race window significantly. > Windows guest is not supported by full virtualization mode, AFAIK. > > http://www.qemu.net/kqemu-doc.html > > BTW, it doesn't give you much performance gain as you may think: > > 'Best performances are achieved with Linux 2.4 kernels. Linux 2.6 > works but the performance gains are small.' > > Even if we make it work with FreeBSD, I would expect similar results. Is that because 2.6 is already better? Is it a 2.6 host or target? Bruce