Date: Tue, 23 Feb 2010 16:35:55 -0600 (CST) From: Sergey Babkin <babkin@verizon.net> To: jhb@freebsd.org Cc: freebsd-ia32@freebsd.org, freebsd-new-bus@freebsd.org, freebsd-ppc@freebsd.org, freebsd-arch@freebsd.org Subject: Re: Re: Strategy for PCI resource management (for supporting hot-plug) Message-ID: <24099271.347187.1266964555857.JavaMail.root@vms061.mailsrvcs.net>
next in thread | raw e-mail | index | archive | help
(Sorry, if the email comes out looking weird, I want to give another try to see if the provider has
fixed the formatting issues i nthe web interface or not).
On Tuesday 23 February 2010 2:16:40 am Rajat Jain wrote:
>
> Hi,
>
> I'm trying to add PCI-E hotplug support to the FreeBSD. As a first step
> for the PCI-E hotplug support, I'm trying to decide on a resource
> management / allocation strategy for the PCI memory / IO and the bus
> numbers. Can you please comment on the following approach that I am
> considering for resource allocation:
>
> PROBLEM STATEMENT:
> ------------------
> Given a memory range [A->B], IO range [C->D], and limited (256) bus
> numbers, enumerate the PCI tree of a system, leaving enough "holes" in
> between to allow addition of future devices.
>
> PROPOSED STRATEGY:
> ------------------
> 1) When booting, start enumerating in a depth-first-search order. While
> enumeration, always keep track of:
>
> * The next bus number (x) that can be allocated
>
> * The next Memory space pointer (A + y) starting which allocation can
> be
> done. ("y" is the memory already allocated).
>
> * The next IO Space pointer (C + z) starting which allocation can be
> done.
> ("z" is the IO space already allocated).
>
> Keep incrementing the above as the resources are allocated.
>
> 2) Allocate bus numbers sequentially while traversing down from root to
> a leaf node (end point). When going down traversing a bridge:
>
> * Allocate the next available bus number (x) to the secondary bus of
> bridge.
>
> * Temporarily mark the subordinate bridge as 0xFF (to allow discovery
> of
> maximum buses).
>
> * Temporarily assign all the remaining available memory space to bridge
>
> [(A+x) -> B]. Ditto for IO space.
>
> 3) When a leaf node (End point) is reached, allocate the memory / IO
> resource requested by the device, and increment the pointers.
>
> 4) While passing a bridge in the upward direction, tweak the bridge
> registers such that its resources are ONLY ENOUGH to address the needs
> of all the PCI tree below it, and if it has its own internal memory
> mapped registers, some memory for it as well.
>
> The above is the standard depth-first algorithm for resource allocation.
> Here is the addition to support hot-plug:
>
> At each bridge that supports hot-plug, in addition to the resources that
> would have normally been allocated to this bridge, additionally
> pre-allocate and assign to bridge (in anticipation of any new devices
> that may be added later):
>
> a) "RSRVE_NUM_BUS" number of busses, to cater to any bridges, PCI trees
> present on the device plugged.
>
> b) "RSRVE_MEM" amount of memory space, to cater to all the PCI devices
> that
> may be attached later on.
>
> c) "RESRVE_IO" amount of IO space, to cater to all PCI devices that may
> be
> attached later on.
A kind of stupid question: should the reserve amounts depend on the level of the bridge?
Perhaps the priidges closer to the root should get more reserves. Perhaps it doesn't
matter so much durin gthe initial enumeration but ma matter latter after a hot plug.
Suppose we have the Bridge B1 that gets RSRVE resources attached to it during
the initial enumeration. Then someone comes and hot-plugs a bridge B2 under B1.
B2 then I guess will also try to get a reserve of RSRVE resources for itself, so it would
take the whole original reserve of B1 to itself. If someone comes later and tries
to hot-plug another bridge B3 under B1, that bridge would not get any resources
and the plugging would fail.
-SB
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?24099271.347187.1266964555857.JavaMail.root>