Date: Wed, 23 Sep 2009 14:05:43 +0300 From: Kostik Belousov <kostikbel@gmail.com> To: Larry Baird <lab@gta.com> Cc: freebsd-xen@freebsd.org Subject: Re: XEN 5.5.0 and clflush Message-ID: <20090923110543.GC47688@deviant.kiev.zoral.com.ua> In-Reply-To: <20090922171918.GA86010@gta.com> References: <20090922123401.GB29391@gta.com> <20090922131034.GV47688@deviant.kiev.zoral.com.ua> <5078471e0909221003g43a125f4s99a1f841616bb184@mail.gmail.com> <20090922171918.GA86010@gta.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--vHRy1SDQH4LgMqtx Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Sep 22, 2009 at 01:19:18PM -0400, Larry Baird wrote: > > > I think I will have to disable CLFLUSH support for intel CPUs when > > > self-snoop > > > is not reported. > > > > > > > > That's the kinda weird part about this though... It's not triggering an > > Invalid Instruction, but a GPF. Looking at AMD's description of how CLF= LUSH > > is supposed to work, I don't see why it's faulting with what looks like= a > > valid address. > >=20 > > While this is probably far outside the scope of what their entry-level > > support techs will understand, I can try raising this as a bug with Cit= rix > > under our support contract if you're confident that this is broken on X= en's > > end. > I keep wondering about comments having to do with AMD processor. The > native processor is an an Intel Xeon. Is this piece of the puzzle > important? Ok, the whole story (mostly to be able to point out an URL with description until the issue is sorted out). FreeBSD got ability to specify page caching attributes in the page table entries, so called Page Attribute Table (PAT). The physical to linear page mapping may specify the caching attribute, like WB (write-back), UC (uncacheable), WT (write-through) and so on. The procedure to set the caching attribute for the page involves removing old mapping, flushing TLB, creating new mapping, and then flushing the cache line that could be allocated for the remmapped physical memory. Flushing the cache may be performed by one of the three mechanisms: - do nothing; this is acceptable if CPU has so called Self-Snoop feature. For future discussion it is important to note that the feature is declared in the feature bitmap returned by CPUID instruction. - do the series of CLFLUSH instructions over the linear region. This method should be used if CPU has CLFLUSH support, again reported by CPUID. - do the WBINVD instruction that flushes the whole CPU cache. The methods above are ordered from most efficient (do nothing) to very time-consuming, and FreeBSD falls to the next method if previous is not supported. Real Intel CPUs (almost) all have all three methods implemented, and we do nothing to flush cache since self snoop is there. AMD CPUs, at least older models, do not have self-snoop, I believe, but do support CLFLUSH. I got no reports of problems caused by CLFLUSH on the AMD. If I manually disable self-snoop on Intel CPU, I get a reversed trap when doing CLFLUSH over the mapping for APIC registers page. I do not know the cause for this, my unfounded speculation is that there is some cache line allocated for it, and it is written out on CLFLUSH. Speculation is flimsy since BIOS should have programmed MTRR so that APIC is UC mode. Now, add the XEN to the picture. When whole machine virtualization (how it is called by XEN ?) runs GENERIC/i386 kernel, it uses so called VT feature of recent Intel CPUs, that, among other, gives the hypervisor the ability to intercept CPUID instruction. XEN seems to modify reported CPU features, removing self-snoop. As result. we fall back to CLFLUSH method. XEN seems to translate a fault caused by cache write-back to the APIC registrers, into #gpf. This is the issue that you get on XEN. So, I do not know why Intels cause trap on CLFLUSH, this may be either some stupidness in my clflush code, or an additional factor that I did not taken care of yet. Patch I sent disables CLFLUSH unconditionally, that is bad for performance on AMD. --vHRy1SDQH4LgMqtx Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (FreeBSD) iEYEARECAAYFAkq6AQYACgkQC3+MBN1Mb4igqQCfYQpu1Hx5xiVCpxMR6s1Uyips +RYAoIbkJzO8Kf8XIIzftK4HKwBC05MT =bCkf -----END PGP SIGNATURE----- --vHRy1SDQH4LgMqtx--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20090923110543.GC47688>