From owner-freebsd-mips@FreeBSD.ORG Fri Jan 29 04:27:07 2010 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D875E1065676 for ; Fri, 29 Jan 2010 04:27:07 +0000 (UTC) (envelope-from rrs@lakerest.net) Received: from lakerest.net (unknown [IPv6:2001:240:585:2:213:d4ff:fef3:2d8d]) by mx1.freebsd.org (Postfix) with ESMTP id 8D2B38FC0C for ; Fri, 29 Jan 2010 04:27:07 +0000 (UTC) Received: from [192.168.2.175] (pool-96-249-204-75.snfcca.dsl-w.verizon.net [96.249.204.75]) (authenticated bits=0) by lakerest.net (8.14.3/8.14.3) with ESMTP id o0T4R2ct099668 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Thu, 28 Jan 2010 23:27:04 -0500 (EST) (envelope-from rrs@lakerest.net) Message-Id: <66207A08-F691-4603-A6C5-9C675414C91E@lakerest.net> From: Randall Stewart To: Neel Natu In-Reply-To: Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v936) Date: Thu, 28 Jan 2010 20:26:56 -0800 References: <20100128.132114.1004138037722505681.imp@bsdimp.com> X-Mailer: Apple Mail (2.936) Cc: freebsd-mips@freebsd.org Subject: Re: Code review: groundwork for SMP X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Jan 2010 04:27:08 -0000 Neel: Now overall my first reaction to this was.. hey this is a cool way to do this. But I have been thinking about this a bit more and I am now having second thoughts. So lets see the old way was to have an array of pcpup[MAXCPU]. The down side to this is it may cause cache line tweaks and you are limited by the MAXCPU of course. Now we can get around the cache line tweaks by simply padding out the structure to the cache line size.. so thats probably not an issue... The MAXCPU is not extendable without a recompile.. but thats also true for your method. Though sexy it has a very big down side that I think calls for us to think real hard before going this route. It burns up TLB entries. Ok that does not sound so bad on the surface but wait lets think about this.. and I am going to speak in terms of XLR... but other mips processors may have the same issue. 1) I have 8 cores per cpu pack. 2) Each core has 4 "threads" which are kinda hyper threads, their own register set, there own everything accept they share a pipeline and get scheduled when one of the others are blocked. 3) This means I still need a pcpup per thread. 4) Now I have 64 TLB entries for every CPU complex. I can have them 16 per thread OR 64 shared amongst all threads. 5) This means I dedicate 4 of my 64 TLB entries for your pcpup entries. Now if I am busy, when I need TLB entries most, I am pretty sure all 4 of those pcpup entries will need to be active and can't be pulled out. So I have lost over 6% of my TLB entries for this method. a) I am still bound to MAXCPU .. so there is no dynamics here.. not that there needs to be b) I loose one of the most precious and sparse resources in a mips processor. c) And I really don't gain much over just having an array of pcpup structures indexed by my CPUID separated by a pad of a cache line each. I really would vote that we DO NOT do this. Instead lets stick with an array and pad it up to a cache line size if its not already.. Other wise you burn up 4 entries for me (and maybe other platforms) and get very little gain. I would rather find a way to get a superpage mapped for the entire kernel + some and hardwire it in... I am not trying to be negative here because I do think its a real sexy way to access the pcpups.. it makes it real transparent... and on the XLP where I would have a lot more TLB entries thats cool.. but I have to think of the existing hardware and its very very small TLB's which is one of the main things limiting performance. So overall I say we do NOT do this.. R On Jan 28, 2010, at 2:01 PM, Neel Natu wrote: > Forwarding to freebsd-mips as suggested by Warner. > > This is a patch to provide access to pcpu structures for SMP kernels. > > The basic idea is to use a the same virtual address as a window onto > distinct physical memory locations - one per processor. The physical > address that you access through this mapping depends on which cpu you > are currently executing on. We can now use the virtual address on any > processor to access its per-cpu area. > > The details are: > > 1. The virtual address for 'struct pcpu *pcpup' is obtained by > stealing 2 pages worth of KVA in pmap_bootstrap(). > > 2. The mapping from the constant virtual address to a distinct > physical page is done in cpu_pcpu_init() through a wired TLB entry. > > 3. A side-effect of this is that we reserve 2 pages worth of memory > for the pcpu but in reality it needs much less than that. The unused > memory is now used as the boot stack for the BSP and APs. > > I also cleaned up locore.S to remove the SMP-specific bits from it. I > plan to use a separate mpboot.S for the AP bootstrap. > > Please review. > > The patch is available here: > http://people.freebsd.org/~neel/smp_groundwork.diff > > best > Neel > _______________________________________________ > freebsd-mips@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-mips > To unsubscribe, send any mail to "freebsd-mips- > unsubscribe@freebsd.org" > ------------------------------ Randall Stewart 803-317-4952 (cell) 803-345-0391(direct)