Date: Sun, 05 Feb 2012 22:48:12 -0700 From: Ian Lepore <freebsd@damnhippie.dyndns.org> To: John-Mark Gurney <jmg@funkthat.com> Cc: freebsd-arm@freebsd.org Subject: Re: Performance of SheevaPlug on 8-stable Message-ID: <1328507292.3546.67.camel@revolution.hippie.lan> In-Reply-To: <20120204234319.GR52468@funkthat.com> References: <1327980703.1662.240.camel@revolution.hippie.lan> <F48E21E0-129A-418A-B147-7D5FB01160A8@bsdimp.com> <1328025245.1662.289.camel@revolution.hippie.lan> <5FB4965A-66C9-4C99-8B61-5AC605F9ECC5@bsdimp.com> <1328030999.1662.324.camel@revolution.hippie.lan> <20120204234319.GR52468@funkthat.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sat, 2012-02-04 at 15:43 -0800, John-Mark Gurney wrote:
> Ian Lepore wrote this message on Tue, Jan 31, 2012 at 10:29 -0700:
> > On Tue, 2012-01-31 at 09:37 -0700, Warner Losh wrote:
> > > On Jan 31, 2012, at 8:54 AM, Ian Lepore wrote:
> > >
> > > > On Mon, 2012-01-30 at 22:39 -0700, Warner Losh wrote:
> > > >> Hi Ian,
> > > >>
> > > >> Do you have any data on what 9.0 does?
> > > >>
> > > >> Warner
> > > >
> > > > No. Do you have reason to believe it will be different than 8.x?
> > > >
> > > > It would be a major effort right now to get anything later than 8.2
> > > > built and running on one of our arm platforms. Maybe not as hard as the
> > > > 6.2 -> 8.2 conversion was, but we're still carrying a lot of diffs from
> > > > stock FreeBSD that have to be analyzed and merged by hand. Actually
> > > > before that can even happen I'd have to grab a snapshot of 9.0 and do an
> > > > svn->Hg conversion to even be able to start merging the diffs (and I'm
> > > > hardly an Hg expert, but those in the company who are let me know last
> > > > week that they're just as busy as me, and I'm on my own for this kind of
> > > > work). It's work I want to do, but I suspect it's going to happen later
> > > > rather than sooner because product deadlines are beginning to loom and
> > > > my ability to spend most of my time working on the OS side of things is
> > > > waning.
> > > >
> > > > If there are some specific changes you've got in mind that affect this
> > > > problem I might be able to backport and test them faster than I could
> > > > get a full 9.0 or -current build environment working, just point me at
> > > > them.
> > >
> > > I thought that we'd done a root cause of this and had put a fix into the vm system. Lemme look...
> > >
> > > ------------------------------------------------------------------------
> > > r224049 | marcel | 2011-07-14 20:11:26 -0600 (Thu, 14 Jul 2011) | 2 lines
> > >
> > > In pmap_protect(), don't call vm_page_dirty() if the page is unmanaged.
> > >
> > >
> > > ------------------------------------------------------------------------
> > > r221844 | cognet | 2011-05-13 09:54:12 -0600 (Fri, 13 May 2011) | 4 lines
> > >
> > > In pmap_change_wiring(), use the right argument for pmap_modify_pv().
> > > It only worked because the only consumer calls pmap_change_wiring() to remove
> > > the wiring.
> > >
> > > ------------------------------------------------------------------------
> > > r212507 | cognet | 2010-09-12 14:46:32 -0600 (Sun, 12 Sep 2010) | 5 lines
> > >
> > > In pmap_remove_all(), do not decrease pm_stats.wired_count if the mapping was
> > > wired, as it's been done later in pmap_nuke_pv().
> > >
> > > Submitted by: Mark Tinguely
> > >
> > >
> > > ------------------------------------------------------------------------
> > > r209223 | cognet | 2010-06-15 16:16:02 -0600 (Tue, 15 Jun 2010) | 4 lines
> > >
> > > Turn off cache if there's more than one kernel mapping, and one is writable.
> > >
> > > Submitted by: Mark Tinguely
> > >
> > > ------------------------------------------------------------------------
> > > r205028 | raj | 2010-03-11 14:16:54 -0700 (Thu, 11 Mar 2010) | 12 lines
> > >
> > > Fix ARM cache handling yet more.
> > >
> > > 1) vm_machdep.c: remove the dangling allocations so they do not
> > > un-necessarily turn off the cache upon consecutive access.
> > >
> > > 2) busdma_machdep.c: remove the same amount than shadow mapped.
> > >
> > > Reported by: Maks Verver
> > > Submitted by: Mark Tinguely
> > > Reviewed by: Grzegorz Bernacki
> > > MFC after: 3 days
> > >
> > > ------------------------------------------------------------------------
> > > r203637 | raj | 2010-02-07 13:48:57 -0700 (Sun, 07 Feb 2010) | 19 lines
> > >
> > > Improve checking whether an ARM VA has a valid mapping before performing cache
> > > sync.
> > >
> > > VIPT/PIPT caches need valid VA-PA mapping in PTE for a cache operation to
> > > succeed (unlike VIVT). Prior to this fix pmap was using l2pte_valid() for that
> > > check, but this is not sufficient as the function merely checks if a PTE
> > > exists (there can be existing but _invalid_ entries in the table).
> > >
> > > A new pmap_has_valid_mapping() routine is introduced to do this job right by
> > > checking proper PTE flags.
> > >
> > > Among other potential problems this cures coherency issues with L2 caches on
> > > MV-78100.
> > >
> > > Submitted by: Grzegorz Bernacki, Piotr Ziecik
> > > Reviewed, tested by: marcel
> > > Obtained from: Semihalf
> > > MFC after: 1 week
> > >
> > >
> > > Only the last two have MFC, so you can start there and see which of these changes are in...
> > >
> > > Just thought you might have a reference board that would be easy to test...
> > >
> > > Warner
> >
> > I think we may have all those changes incorporated except perhaps
> > r224049; I'll make sure of that.
> >
> > r209223 is the change that exposed this situation.
> >
> > I'm skeptical that any of the changes you cite (or any change at all in
> > the pmap layer) will fix the problem, because the problem seems to be
> > rooted in the fact that the vfs buffer cache establishes a kva mapping
> > of the buffer pages with the protections set to READ|WRITE|EXEC and
> > leaves that mapping in place as long as the buffer is in the cache, and
> > r209223 says that as long as there are multiple mappings of a page with
> > at least one writable, that page's i-cache and d-cache bits stay off.
> > (The multiple mappings being the one for the buffer cache that includes
> > write access and one or more READ|EXEC mappings made by pmap() when the
> > executable or library is loaded/relocated.)
> >
> > If my analysis is correct (and I'm fairly sure, if not 100% positive,
> > that it is), then it seems to me that the only fix available is going to
> > be at the vfs layer, and it's going to involve dropping the write access
> > to the pages in the buffer cache once any physical IO and/or uio
> > operations needing write access are completed.
> >
> > Even if I could figure out a patchset to fix the problem, it's going to
> > need a lot of input from the vm gurus to answer questions such as what
> > the performance impact will be to non-VIVT platforms that don't need
> > this extra work done. If the extra work is expensive enough (and I'm
> > not sure I could evaluate that properly) it may need to be conditional
> > on whether the platform needs it. I'm also vaguely uneasy with all this
> > on a purely philosphical level, since this could end up basically
> > infecting MI code with a platform-specific concept.
>
> What is an easy to figure out if a system we have is effected by this
> issue? I have a GW2348-4 board running FreeBSD 9.0-RC1 w/ some minor
> modifications to get pf to work...
>
> I think the system is effected by this since userland seems really slow..
>
> Thanks.
>
Okay, I've just confirmed that the problem as I described it above still
exists in 9.0 ...
# uname -a
FreeBSD 9.0-RELEASE FreeBSD 9.0-RELEASE #7: Mon Feb 6 04:38:59
UTC 2012
root@revolution.hippie.lan:/usr/obj/arm.arm/usr/src/sys/TFLEX
arm
# /usr/tsc/bin/testsimple
Elapsed 0.782453
# /usr/tsc/bin/testsimple
Elapsed 0.782539
# cat /usr/tsc/bin/testsimple >/dev/null
# /usr/tsc/bin/testsimple
Elapsed 10.090336
# /usr/tsc/bin/testsimple
Elapsed 10.090611
The testsimple program is just a little loop that repeatedly assigns a
volatile variable (had to fool the optimizer into generating some code).
I statically linked it to avoid any variability based on the race
between paging and readahead when loading shared libs. I'm just showing
here that the base problem still exists: when executable pages get into
the vfs buffer cache they (semi-)permanently lose their i-cache bit.
I also applied my patches from the start of this thread and re-tested
and they appear to still be a usable workaround for the problem in 9.
But they are just a workaround, we need to figure out a real fix for
this. (I had to enable the #if in ffs_read() as well as the one in
ffs_write() for this test, since my quick 'cat' test is reading the file
to get it into the cache.)
-- Ian
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1328507292.3546.67.camel>