Date: Thu, 28 Jan 2010 21:28:51 -0800 From: Randall Stewart <rrs@lakerest.net> To: Juli Mallett <jmallett@FreeBSD.org> Cc: freebsd-mips@FreeBSD.org, Neel Natu <neelnatu@gmail.com> Subject: Re: Code review: groundwork for SMP Message-ID: <85D9D383-29A3-4F09-A2FE-61E4EA85CE9B@lakerest.net> In-Reply-To: <eaa228be1001282040r416f3764lde577786347a4d5e@mail.gmail.com> References: <dffe84831001262336l1978797g8b12fab815f4eb52@mail.gmail.com> <20100128.132114.1004138037722505681.imp@bsdimp.com> <dffe84831001281401n7c9fb64bjb38260943448f315@mail.gmail.com> <66207A08-F691-4603-A6C5-9C675414C91E@lakerest.net> <eaa228be1001282040r416f3764lde577786347a4d5e@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Jan 28, 2010, at 8:40 PM, Juli Mallett wrote: > On Thu, Jan 28, 2010 at 20:26, Randall Stewart <rrs@lakerest.net> =20 > wrote: >> It burns up TLB entries. Ok that does not sound so bad on the >> surface but wait lets think about this.. and I am going to >> speak in terms of XLR... but other mips processors may have >> the same issue. >> >> 1) I have 8 cores per cpu pack. >> 2) Each core has 4 "threads" which are kinda hyper threads, their >> own register set, there own everything accept they share a pipeline >> and get scheduled when one of the others are blocked. >> 3) This means I still need a pcpup per thread. >> 4) Now I have 64 TLB entries for every CPU complex. I can have them >> 16 per thread OR 64 shared amongst all threads. >> 5) This means I dedicate 4 of my 64 TLB entries for your pcpup =20 >> entries. > > So on your systems threads share the TLB? Wired TLB entries can't be > pulled out (in the case of the kernel stack it's basically > catastrophic for that to happen.) A compromise if your TLB entries > are really at a premium is to use a single large entry (using, say, a > single 32k page) that contains both PCPU and the kernel stack, or a > page which has pointers to pcpu data, the kernel stack, etc. I seem > to recall seeing a port of FreeBSD that used the same storage for the > kernel stack and PCPU data, but I could be mistaken. Which means you have a big array that you are offsetting. I was even thinking get a LARGE entry.. one that is say 8 Meg for the kernel.. covering all text/data etc... with this new super page stuff. of course I have never looked into how its implemented.. Going back to your idea. is that not the same thing as having an index into an array. Yes, you pay an index reference for every access .. or at least one to setup a pointer.. but I think that it much cheaper than a TLB miss is... (words for imp to think about)... > > There are other trade-offs available, of course. If we don't use the > gp for accessing small data, we can keep a pointer to the pcpu data of > a CPU in gp whenever the kernel is running, and then PCPU accesses are > just a madder of loading from offset+gp, which is very quick =97 = faster > than the wired TLB entry mechanism, unless you use a virtual address > for the pcpu in which case it can be painful. As there are more > things like VIMAGE, the amount of small global data in the kernel is > going to fall and making gp a pcpu pointer makes more sense. My old > port used -G0 and I still disable use of the gp in my non-FreeBSD MIPS > work =97 I think NetBSD used to but I haven't noticed what FreeBSD = does. > This is an interesting idea... need to think about it more ;-) > More curiosity than anything (since I don't seem to be able to get an > RMI system to develop on): if the threads are sharing the TLB, how are > updates to TLB-related fields synchronized? How do you atomically > increase the wired count of the TLB? How does 'tlbwr' work? Do you > have to use special instructions when you're sharing the TLB that are > XLR-specific? I can't tell you how the hardware works.. I can either have the TLB divided into 16 entries per thread OR enable a special global register and get 64 entries that all threads see. In any case I just don't see that this has that much gain... its sexy.. but I think its a tradeoff like everything.. R > > Juli. > ------------------------------ Randall Stewart 803-317-4952 (cell) 803-345-0391(direct)
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?85D9D383-29A3-4F09-A2FE-61E4EA85CE9B>