From owner-freebsd-new-bus@FreeBSD.ORG Tue Feb 23 07:19:04 2010 Return-Path: Delivered-To: freebsd-new-bus@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D14291065670; Tue, 23 Feb 2010 07:19:04 +0000 (UTC) (envelope-from rajatjain@juniper.net) Received: from exprod7og124.obsmtp.com (exprod7og124.obsmtp.com [64.18.2.26]) by mx1.freebsd.org (Postfix) with ESMTP id 3DCE18FC12; Tue, 23 Feb 2010 07:19:04 +0000 (UTC) Received: from source ([66.129.224.36]) (using TLSv1) by exprod7ob124.postini.com ([64.18.6.12]) with SMTP ID DSNKS4OBZ34W5IL1/4ymY+6O/BE5E9sukt0M@postini.com; Mon, 22 Feb 2010 23:19:04 PST Received: from emailbng1.jnpr.net (10.209.194.15) by P-EMHUB02-HQ.jnpr.net (172.24.192.36) with Microsoft SMTP Server id 8.1.393.1; Mon, 22 Feb 2010 23:16:44 -0800 Received: from emailbng3.jnpr.net ([10.209.194.27]) by emailbng1.jnpr.net with Microsoft SMTPSVC(6.0.3790.3959); Tue, 23 Feb 2010 12:46:41 +0530 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-Class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Date: Tue, 23 Feb 2010 12:46:40 +0530 Message-ID: <8506939B503B404A84BBB12293FC45F606B88C39@emailbng3.jnpr.net> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Strategy for PCI resource management (for supporting hot-plug) Thread-Index: Acq0WCUiblKr9+OlQeafGqGQtTgEow== From: Rajat Jain To: , X-OriginalArrivalTime: 23 Feb 2010 07:16:41.0758 (UTC) FILETIME=[25C9BBE0:01CAB458] Cc: freebsd-ia32@freebsd.org, freebsd-ppc@freebsd.org Subject: Strategy for PCI resource management (for supporting hot-plug) X-BeenThere: freebsd-new-bus@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD's new-bus architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Feb 2010 07:19:04 -0000 Hi, I'm trying to add PCI-E hotplug support to the FreeBSD. As a first step for the PCI-E hotplug support, I'm trying to decide on a resource management / allocation strategy for the PCI memory / IO and the bus numbers. Can you please comment on the following approach that I am considering for resource allocation: PROBLEM STATEMENT: ------------------ Given a memory range [A->B], IO range [C->D], and limited (256) bus numbers, enumerate the PCI tree of a system, leaving enough "holes" in between to allow addition of future devices. PROPOSED STRATEGY: ------------------ 1) When booting, start enumerating in a depth-first-search order. While enumeration, always keep track of: * The next bus number (x) that can be allocated * The next Memory space pointer (A + y) starting which allocation can be=20 done. ("y" is the memory already allocated). * The next IO Space pointer (C + z) starting which allocation can be done. ("z" is the IO space already allocated). Keep incrementing the above as the resources are allocated. 2) Allocate bus numbers sequentially while traversing down from root to a leaf node (end point). When going down traversing a bridge: * Allocate the next available bus number (x) to the secondary bus of=20 bridge. * Temporarily mark the subordinate bridge as 0xFF (to allow discovery of=20 maximum buses). * Temporarily assign all the remaining available memory space to bridge [(A+x) -> B]. Ditto for IO space. 3) When a leaf node (End point) is reached, allocate the memory / IO resource requested by the device, and increment the pointers.=20 4) While passing a bridge in the upward direction, tweak the bridge registers such that its resources are ONLY ENOUGH to address the needs of all the PCI tree below it, and if it has its own internal memory mapped registers, some memory for it as well. The above is the standard depth-first algorithm for resource allocation. Here is the addition to support hot-plug: At each bridge that supports hot-plug, in addition to the resources that would have normally been allocated to this bridge, additionally pre-allocate and assign to bridge (in anticipation of any new devices that may be added later): a) "RSRVE_NUM_BUS" number of busses, to cater to any bridges, PCI trees=20 present on the device plugged. b) "RSRVE_MEM" amount of memory space, to cater to all the PCI devices that=20 may be attached later on. c) "RESRVE_IO" amount of IO space, to cater to all PCI devices that may be=20 attached later on. Please note that the above RSRVE* are constants defining the amount of resources to be set aside for /below each HOT-PLUGGABLE bridge; their values may be tweaked via a compile time option or via a sysctl.=20 FEW COMMENTS ------------ =20 1) The strategy is fairly generic and tweak-able since it does not waste a lot of resources (The developer neds to pick up a smart bvalue for howmuch resources to reserve at each hot-pluggable slot): * The reservations shall be done only for hot-pluggable bridges * The developer can tweak the values (even disable it) for how much=20 Resources shall be allocated for each hot-pluggable bridge. =20 2) One point of debate is what happens if there are too much resource demands in the system (too many devices or the developer configures too many resources to be allocated for each hot-pluggable devices). For e.g. consider that while enumeration we find that all the resources are already allocated, while there are more devices that need resources. So do we simply do not enumerate them? Etc... Overall, how does the above look? Thanks & Best Regards, Rajat Jain