From owner-freebsd-new-bus@FreeBSD.ORG Wed Feb 24 04:35:19 2010 Return-Path: Delivered-To: freebsd-new-bus@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CDADB106566B; Wed, 24 Feb 2010 04:35:19 +0000 (UTC) (envelope-from rajatjain@juniper.net) Received: from exprod7og124.obsmtp.com (exprod7og124.obsmtp.com [64.18.2.26]) by mx1.freebsd.org (Postfix) with ESMTP id CCD708FC16; Wed, 24 Feb 2010 04:35:18 +0000 (UTC) Received: from source ([66.129.224.36]) (using TLSv1) by exprod7ob124.postini.com ([64.18.6.12]) with SMTP ID DSNKS4SshmVUaDSyL+FTm5ptlGPCuZEBf9AD@postini.com; Tue, 23 Feb 2010 20:35:19 PST Received: from gaugeboson.jnpr.net (10.209.194.17) by P-EMHUB03-HQ.jnpr.net (172.24.192.37) with Microsoft SMTP Server id 8.1.393.1; Tue, 23 Feb 2010 20:32:39 -0800 Received: from emailbng3.jnpr.net ([10.209.194.27]) by gaugeboson.jnpr.net with Microsoft SMTPSVC(6.0.3790.3959); Wed, 24 Feb 2010 10:02:36 +0530 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-Class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Date: Wed, 24 Feb 2010 10:02:35 +0530 Message-ID: <8506939B503B404A84BBB12293FC45F606B88E55@emailbng3.jnpr.net> In-Reply-To: <24099271.347187.1266964555857.JavaMail.root@vms061.mailsrvcs.net> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Re: Strategy for PCI resource management (for supporting hot-plug) Thread-Index: Acq02NJfpnyCYOSsRvKUNhQqcZ9VZQAMFnBg References: <24099271.347187.1266964555857.JavaMail.root@vms061.mailsrvcs.net> From: Rajat Jain To: Sergey Babkin , X-OriginalArrivalTime: 24 Feb 2010 04:32:36.0576 (UTC) FILETIME=[6403EE00:01CAB50A] Cc: freebsd-ia32@freebsd.org, freebsd-new-bus@freebsd.org, freebsd-ppc@freebsd.org, freebsd-arch@freebsd.org Subject: RE: Re: Strategy for PCI resource management (for supporting hot-plug) X-BeenThere: freebsd-new-bus@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD's new-bus architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Feb 2010 04:35:19 -0000 Hello Sergey, > A kind of stupid question: should the reserve amounts depend on the level > of the bridge? > Perhaps the priidges closer to the root should get more reserves. Perhaps > it doesn't > matter so much durin gthe initial enumeration but ma matter latter after a > hot plug. One clarification perhaps I did not give in my proposal. We would reserve resources (bus numbers / memory / IO) only for bridges that are CAPABLE of HOT-PLUG. The rest of the bridges would get their usual share of resources.=20 Now, the same amount of reserved resources gets assigned to each HOT-PLUG capable bridge, irrespective of at which level it is in hierarchy. This is because no matter where it is, there is equal probability of a new card being plugged in at ANY of those slots.=20 The only problem as you say is when we plug in a PCI card, which has a HOT-PLUGGABLE SLOT on it (on which we can plug in more cards). (This is because a bridge wants extra reserved resources only when it is hot-plug capable). Do such devices exist? Since theoretically possible, but practically extremely rare, I say we do not support this case. Comments? Thanks, Rajat Jain > -----Original Message----- > From: Sergey Babkin [mailto:babkin@verizon.net] > Sent: Wednesday, February 24, 2010 4:06 AM > To: jhb@freebsd.org > Cc: freebsd-arch@freebsd.org; Rajat Jain; freebsd-ia32@freebsd.org; > freebsd-new-bus@freebsd.org; freebsd-ppc@freebsd.org > Subject: Re: Re: Strategy for PCI resource management (for supporting hot- > plug) >=20 > (Sorry, if the email comes out looking weird, I want to give another try > to see if the provider has > fixed the formatting issues i nthe web interface or not). >=20 > On Tuesday 23 February 2010 2:16:40 am Rajat Jain wrote: > > > > Hi, > > > > I'm trying to add PCI-E hotplug support to the FreeBSD. As a first step > > for the PCI-E hotplug support, I'm trying to decide on a resource > > management / allocation strategy for the PCI memory / IO and the bus > > numbers. Can you please comment on the following approach that I am > > considering for resource allocation: > > > > PROBLEM STATEMENT: > > ------------------ > > Given a memory range [A->B], IO range [C->D], and limited (256) bus > > numbers, enumerate the PCI tree of a system, leaving enough "holes" in > > between to allow addition of future devices. > > > > PROPOSED STRATEGY: > > ------------------ > > 1) When booting, start enumerating in a depth-first-search order. While > > enumeration, always keep track of: > > > > * The next bus number (x) that can be allocated > > > > * The next Memory space pointer (A + y) starting which allocation can > > be > > done. ("y" is the memory already allocated). > > > > * The next IO Space pointer (C + z) starting which allocation can be > > done. > > ("z" is the IO space already allocated). > > > > Keep incrementing the above as the resources are allocated. > > > > 2) Allocate bus numbers sequentially while traversing down from root to > > a leaf node (end point). When going down traversing a bridge: > > > > * Allocate the next available bus number (x) to the secondary bus of > > bridge. > > > > * Temporarily mark the subordinate bridge as 0xFF (to allow discovery > > of > > maximum buses). > > > > * Temporarily assign all the remaining available memory space to bridge > > > > [(A+x) -> B]. Ditto for IO space. > > > > 3) When a leaf node (End point) is reached, allocate the memory / IO > > resource requested by the device, and increment the pointers. > > > > 4) While passing a bridge in the upward direction, tweak the bridge > > registers such that its resources are ONLY ENOUGH to address the needs > > of all the PCI tree below it, and if it has its own internal memory > > mapped registers, some memory for it as well. > > > > The above is the standard depth-first algorithm for resource allocation. > > Here is the addition to support hot-plug: > > > > At each bridge that supports hot-plug, in addition to the resources that > > would have normally been allocated to this bridge, additionally > > pre-allocate and assign to bridge (in anticipation of any new devices > > that may be added later): > > > > a) "RSRVE_NUM_BUS" number of busses, to cater to any bridges, PCI trees > > present on the device plugged. > > > > b) "RSRVE_MEM" amount of memory space, to cater to all the PCI devices > > that > > may be attached later on. > > > > c) "RESRVE_IO" amount of IO space, to cater to all PCI devices that may > > be > > attached later on. >=20 > A kind of stupid question: should the reserve amounts depend on the level > of the bridge? > Perhaps the priidges closer to the root should get more reserves. Perhaps > it doesn't > matter so much durin gthe initial enumeration but ma matter latter after a > hot plug. >=20 > Suppose we have the Bridge B1 that gets RSRVE resources attached to it > during > the initial enumeration. Then someone comes and hot-plugs a bridge B2 > under B1. > B2 then I guess will also try to get a reserve of RSRVE resources for > itself, so it would > take the whole original reserve of B1 to itself. If someone comes later > and tries > to hot-plug another bridge B3 under B1, that bridge would not get any > resources > and the plugging would fail. >=20 > -SB