Date: Thu, 28 Jan 2010 20:26:56 -0800 From: Randall Stewart <rrs@lakerest.net> To: Neel Natu <neelnatu@gmail.com> Cc: freebsd-mips@freebsd.org Subject: Re: Code review: groundwork for SMP Message-ID: <66207A08-F691-4603-A6C5-9C675414C91E@lakerest.net> In-Reply-To: <dffe84831001281401n7c9fb64bjb38260943448f315@mail.gmail.com> References: <dffe84831001262336l1978797g8b12fab815f4eb52@mail.gmail.com> <20100128.132114.1004138037722505681.imp@bsdimp.com> <dffe84831001281401n7c9fb64bjb38260943448f315@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Neel:
Now overall my first reaction to this was.. hey this is
a cool way to do this.
But I have been thinking about this a bit more and I am now
having second thoughts.
So lets see the old way was to have an array of
pcpup[MAXCPU]. The down side to this is it may cause
cache line tweaks and you are limited by the MAXCPU of course.
Now we can get around the cache line tweaks by simply padding
out the structure to the cache line size.. so thats probably not
an issue...
The MAXCPU is not extendable without a recompile.. but thats also
true for your method. Though sexy it has a very big down side that
I think calls for us to think real hard before going this route.
It burns up TLB entries. Ok that does not sound so bad on the
surface but wait lets think about this.. and I am going to
speak in terms of XLR... but other mips processors may have
the same issue.
1) I have 8 cores per cpu pack.
2) Each core has 4 "threads" which are kinda hyper threads, their
own register set, there own everything accept they share a pipeline
and get scheduled when one of the others are blocked.
3) This means I still need a pcpup per thread.
4) Now I have 64 TLB entries for every CPU complex. I can have them
16 per thread OR 64 shared amongst all threads.
5) This means I dedicate 4 of my 64 TLB entries for your pcpup entries.
Now if I am busy, when I need TLB entries most, I am pretty sure all 4
of those
pcpup entries will need to be active and can't be pulled out. So I
have lost
over 6% of my TLB entries for this method.
a) I am still bound to MAXCPU .. so there is no dynamics here.. not
that there needs to be
b) I loose one of the most precious and sparse resources in a mips
processor.
c) And I really don't gain much over just having an array of pcpup
structures indexed
by my CPUID separated by a pad of a cache line each.
I really would vote that we DO NOT do this. Instead lets stick with an
array and
pad it up to a cache line size if its not already..
Other wise you burn up 4 entries for me (and maybe other platforms)
and get
very little gain. I would rather find a way to get a superpage mapped
for
the entire kernel + some and hardwire it in...
I am not trying to be negative here because I do think its a real sexy
way to
access the pcpups.. it makes it real transparent... and on the XLP
where I
would have a lot more TLB entries thats cool.. but I have to think of
the
existing hardware and its very very small TLB's which is one of the
main things limiting performance.
So overall I say we do NOT do this..
R
On Jan 28, 2010, at 2:01 PM, Neel Natu wrote:
> Forwarding to freebsd-mips as suggested by Warner.
>
> This is a patch to provide access to pcpu structures for SMP kernels.
>
> The basic idea is to use a the same virtual address as a window onto
> distinct physical memory locations - one per processor. The physical
> address that you access through this mapping depends on which cpu you
> are currently executing on. We can now use the virtual address on any
> processor to access its per-cpu area.
>
> The details are:
>
> 1. The virtual address for 'struct pcpu *pcpup' is obtained by
> stealing 2 pages worth of KVA in pmap_bootstrap().
>
> 2. The mapping from the constant virtual address to a distinct
> physical page is done in cpu_pcpu_init() through a wired TLB entry.
>
> 3. A side-effect of this is that we reserve 2 pages worth of memory
> for the pcpu but in reality it needs much less than that. The unused
> memory is now used as the boot stack for the BSP and APs.
>
> I also cleaned up locore.S to remove the SMP-specific bits from it. I
> plan to use a separate mpboot.S for the AP bootstrap.
>
> Please review.
>
> The patch is available here:
> http://people.freebsd.org/~neel/smp_groundwork.diff
>
> best
> Neel
> _______________________________________________
> freebsd-mips@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-mips
> To unsubscribe, send any mail to "freebsd-mips-
> unsubscribe@freebsd.org"
>
------------------------------
Randall Stewart
803-317-4952 (cell)
803-345-0391(direct)
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?66207A08-F691-4603-A6C5-9C675414C91E>