From owner-freebsd-mips@FreeBSD.ORG Fri Jan 29 05:07:53 2010 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 072C21065670 for ; Fri, 29 Jan 2010 05:07:53 +0000 (UTC) (envelope-from juli@clockworksquid.com) Received: from mail-pz0-f176.google.com (mail-pz0-f176.google.com [209.85.222.176]) by mx1.freebsd.org (Postfix) with ESMTP id DAE518FC12 for ; Fri, 29 Jan 2010 05:07:52 +0000 (UTC) Received: by pzk6 with SMTP id 6so1109810pzk.3 for ; Thu, 28 Jan 2010 21:07:52 -0800 (PST) MIME-Version: 1.0 Sender: juli@clockworksquid.com Received: by 10.142.117.1 with SMTP id p1mr230919wfc.185.1264740052089; Thu, 28 Jan 2010 20:40:52 -0800 (PST) In-Reply-To: <66207A08-F691-4603-A6C5-9C675414C91E@lakerest.net> References: <20100128.132114.1004138037722505681.imp@bsdimp.com> <66207A08-F691-4603-A6C5-9C675414C91E@lakerest.net> From: Juli Mallett Date: Thu, 28 Jan 2010 20:40:32 -0800 X-Google-Sender-Auth: 04ed61989917d238 Message-ID: To: Randall Stewart Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Cc: freebsd-mips@freebsd.org, Neel Natu Subject: Re: Code review: groundwork for SMP X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Jan 2010 05:07:53 -0000 On Thu, Jan 28, 2010 at 20:26, Randall Stewart wrote: > It burns up TLB entries. Ok that does not sound so bad on the > surface but wait lets think about this.. and I am going to > speak in terms of XLR... but other mips processors may have > the same issue. > > 1) I have 8 cores per cpu pack. > 2) Each core has 4 "threads" which are kinda hyper threads, their > own register set, there own everything accept they share a pipeline > and get scheduled when one of the others are blocked. > 3) This means I still need a pcpup per thread. > 4) Now I have 64 TLB entries for every CPU complex. I can have them > 16 per thread OR 64 shared amongst all threads. > 5) This means I dedicate 4 of my 64 TLB entries for your pcpup entries. So on your systems threads share the TLB? Wired TLB entries can't be pulled out (in the case of the kernel stack it's basically catastrophic for that to happen.) A compromise if your TLB entries are really at a premium is to use a single large entry (using, say, a single 32k page) that contains both PCPU and the kernel stack, or a page which has pointers to pcpu data, the kernel stack, etc. I seem to recall seeing a port of FreeBSD that used the same storage for the kernel stack and PCPU data, but I could be mistaken. There are other trade-offs available, of course. If we don't use the gp for accessing small data, we can keep a pointer to the pcpu data of a CPU in gp whenever the kernel is running, and then PCPU accesses are just a madder of loading from offset+gp, which is very quick =97 faster than the wired TLB entry mechanism, unless you use a virtual address for the pcpu in which case it can be painful. As there are more things like VIMAGE, the amount of small global data in the kernel is going to fall and making gp a pcpu pointer makes more sense. My old port used -G0 and I still disable use of the gp in my non-FreeBSD MIPS work =97 I think NetBSD used to but I haven't noticed what FreeBSD does. More curiosity than anything (since I don't seem to be able to get an RMI system to develop on): if the threads are sharing the TLB, how are updates to TLB-related fields synchronized? How do you atomically increase the wired count of the TLB? How does 'tlbwr' work? Do you have to use special instructions when you're sharing the TLB that are XLR-specific? Juli.