Date: Tue, 6 Aug 2002 02:38:17 -0400 From: Jake Burkholder <jake@locore.ca> To: Luigi Rizzo <rizzo@icir.org> Cc: Peter Wemm <peter@wemm.org>, Terry Lambert <tlambert2@mindspring.com>, smp@FreeBSD.ORG Subject: Re: how to create per-cpu variables in SMP kernels ? Message-ID: <20020806023816.D76014@locore.ca> In-Reply-To: <20020805230556.C26751@iguana.icir.org>; from rizzo@icir.org on Mon, Aug 05, 2002 at 11:05:56PM -0700 References: <20020805015340.A17716@iguana.icir.org> <20020805221415.B0C732A7D6@canning.wemm.org> <20020805230556.C26751@iguana.icir.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Apparently, On Mon, Aug 05, 2002 at 11:05:56PM -0700,
Luigi Rizzo said words to the effect of;
> Hi Peter,
> thanks for the explaination.
> I still have a few doubts on this (let's restrict to the -current
> case where the code seems more readable):
>
> --- MINOR DETAIL ---
>
> * I wonder why the macro __PCPU_GET() in sys/i386/include/pcpu.h
> cannot store directly into __result for operand sizes of 1,2,4
> instead of going through a temporary variable. I.e. what would
> be wrong in having
>
> #define __PCPU_GET(name) ({ \
> __pcpu_type(name) __result; \
> \
> if (sizeof(__result) == 1) { \
> __asm __volatile("movb %%fs:%1,%0" \
> : "=r" (__result) \
> : "m" (*(u_char *)(__pcpu_offset(name)))); \
> } else if (sizeof(__result) == 2) { \
>
> Probably the same holds for __PCPU_SET().
The code has to work for all types; if __result is a struct timeval,
having it as an output in the asm statement doesn't compile. It all
gets optimized out anyway.
>
> --- OVERALL IMPLEMENTATION OF THE PER-CPU DATA ---
>
> Partly following Terry's description, i thought an arrangement
> like the following could be relatively simple to implement and not
> require any recourse to assembly code, does not impact the compiler's
> ability to do optimizations, and does not require an extra
> segment descriptor to access the struct pcpu.
>
> It relies on the following variables, my_pcpu to access the
> pcpu data of the local processor, all_pcpu to view all pcpu
> data (including our own, at a different mapping in vm space):
>
> struct pcpu *my_pcpu;
>
> struct pcpu *all_pcpu[MAXCPU]; /* XXX volatile */
>
> Early in the boot process we allocate MAXCPU physical pages,
> and MAXCPU+1 entries in the VM space. Individual pcpu structs
> go at the beginning of each of the physical pages, and the
> VM -> physical mapping of the first MAXCPU VM entries is the
> same for all processors. Then all_pcpu[i] can be initialized
> with a pointer to the beginning of the i-th VM page.
>
> The MAXCPU+1-th VM entry maps differently on each CPU,
> so that it effectively permits access to the per-cpu data.
> my_pcpu can be initialized with a pointer to the MAXCPU+1-th VM page.
>
> At this point, curproc and all other per-cpu variables for the
> local CPU can be accessed through
>
> my_pcpu->curproc
>
> and similar, whereas we can get to other cpu's data with
>
> all_pcpu[i]->curproc
>
> without the need for using %fs or special assembly language to
> access these fields.
>
> Then we can discuss how/where to put "volatile" keywords.
> In principle, all references through all_pcpu[] should be
> readonly and treated as volatile, with perhaps the exception of
> some section of code at machine startup. On the contrary we could
> safely assume that references through my_pcpu are non-volatile
> as the local processor should be the only one to mess with them
>
> Anything wrong with this description ?
This doesn't work because the page directory is per-process, not per-cpu.
To implement this you would need a fixed page directory entry which pointed
to a different page table page on each cpu, which mapped the different
per-cpu pages to the same virtual address. If 2 processes which shared
page directories were running concurrently on 2 cpus, they would both
see the same per-cpu data (one of then would get the wrong struct pcpu).
Basically the struct pcpu's cannot all be mapped to the same virtual
address.
Jake
>
> cheers
> luigi
>
> On Mon, Aug 05, 2002 at 03:14:15PM -0700, Peter Wemm wrote:
> ...
> > Sort-of. There is both a compile time issue and a runtime issue.
> >
> > Using the %fs:variable segment overrides doesn't make a lot of difference,
> > but the compiler is effectively wired so that they are treated as volatile.
> ...
>
> To Unsubscribe: send mail to majordomo@FreeBSD.org
> with "unsubscribe freebsd-smp" in the body of the message
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-smp" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20020806023816.D76014>