Date: Wed, 24 Feb 2010 10:02:35 +0530 From: Rajat Jain <rajatjain@juniper.net> To: Sergey Babkin <babkin@verizon.net>, <jhb@freebsd.org> Cc: freebsd-ia32@freebsd.org, freebsd-new-bus@freebsd.org, freebsd-ppc@freebsd.org, freebsd-arch@freebsd.org Subject: RE: Re: Strategy for PCI resource management (for supporting hot-plug) Message-ID: <8506939B503B404A84BBB12293FC45F606B88E55@emailbng3.jnpr.net> In-Reply-To: <24099271.347187.1266964555857.JavaMail.root@vms061.mailsrvcs.net> References: <24099271.347187.1266964555857.JavaMail.root@vms061.mailsrvcs.net>
next in thread | previous in thread | raw e-mail | index | archive | help
Hello Sergey,
> A kind of stupid question: should the reserve amounts depend on the
level
> of the bridge?
> Perhaps the priidges closer to the root should get more reserves.
Perhaps
> it doesn't
> matter so much durin gthe initial enumeration but ma matter latter
after a
> hot plug.
One clarification perhaps I did not give in my proposal. We would
reserve resources (bus numbers / memory / IO) only for bridges that are
CAPABLE of HOT-PLUG. The rest of the bridges would get their usual share
of resources.=20
Now, the same amount of reserved resources gets assigned to each
HOT-PLUG capable bridge, irrespective of at which level it is in
hierarchy. This is because no matter where it is, there is equal
probability of a new card being plugged in at ANY of those slots.=20
The only problem as you say is when we plug in a PCI card, which has a
HOT-PLUGGABLE SLOT on it (on which we can plug in more cards). (This is
because a bridge wants extra reserved resources only when it is hot-plug
capable). Do such devices exist? Since theoretically possible, but
practically extremely rare, I say we do not support this case.
Comments?
Thanks,
Rajat Jain
> -----Original Message-----
> From: Sergey Babkin [mailto:babkin@verizon.net]
> Sent: Wednesday, February 24, 2010 4:06 AM
> To: jhb@freebsd.org
> Cc: freebsd-arch@freebsd.org; Rajat Jain; freebsd-ia32@freebsd.org;
> freebsd-new-bus@freebsd.org; freebsd-ppc@freebsd.org
> Subject: Re: Re: Strategy for PCI resource management (for supporting
hot-
> plug)
>=20
> (Sorry, if the email comes out looking weird, I want to give another
try
> to see if the provider has
> fixed the formatting issues i nthe web interface or not).
>=20
> On Tuesday 23 February 2010 2:16:40 am Rajat Jain wrote:
> >
> > Hi,
> >
> > I'm trying to add PCI-E hotplug support to the FreeBSD. As a first
step
> > for the PCI-E hotplug support, I'm trying to decide on a resource
> > management / allocation strategy for the PCI memory / IO and the bus
> > numbers. Can you please comment on the following approach that I am
> > considering for resource allocation:
> >
> > PROBLEM STATEMENT:
> > ------------------
> > Given a memory range [A->B], IO range [C->D], and limited (256) bus
> > numbers, enumerate the PCI tree of a system, leaving enough "holes"
in
> > between to allow addition of future devices.
> >
> > PROPOSED STRATEGY:
> > ------------------
> > 1) When booting, start enumerating in a depth-first-search order.
While
> > enumeration, always keep track of:
> >
> > * The next bus number (x) that can be allocated
> >
> > * The next Memory space pointer (A + y) starting which allocation
can
> > be
> > done. ("y" is the memory already allocated).
> >
> > * The next IO Space pointer (C + z) starting which allocation can
be
> > done.
> > ("z" is the IO space already allocated).
> >
> > Keep incrementing the above as the resources are allocated.
> >
> > 2) Allocate bus numbers sequentially while traversing down from root
to
> > a leaf node (end point). When going down traversing a bridge:
> >
> > * Allocate the next available bus number (x) to the secondary bus
of
> > bridge.
> >
> > * Temporarily mark the subordinate bridge as 0xFF (to allow
discovery
> > of
> > maximum buses).
> >
> > * Temporarily assign all the remaining available memory space to
bridge
> >
> > [(A+x) -> B]. Ditto for IO space.
> >
> > 3) When a leaf node (End point) is reached, allocate the memory / IO
> > resource requested by the device, and increment the pointers.
> >
> > 4) While passing a bridge in the upward direction, tweak the bridge
> > registers such that its resources are ONLY ENOUGH to address the
needs
> > of all the PCI tree below it, and if it has its own internal memory
> > mapped registers, some memory for it as well.
> >
> > The above is the standard depth-first algorithm for resource
allocation.
> > Here is the addition to support hot-plug:
> >
> > At each bridge that supports hot-plug, in addition to the resources
that
> > would have normally been allocated to this bridge, additionally
> > pre-allocate and assign to bridge (in anticipation of any new
devices
> > that may be added later):
> >
> > a) "RSRVE_NUM_BUS" number of busses, to cater to any bridges, PCI
trees
> > present on the device plugged.
> >
> > b) "RSRVE_MEM" amount of memory space, to cater to all the PCI
devices
> > that
> > may be attached later on.
> >
> > c) "RESRVE_IO" amount of IO space, to cater to all PCI devices that
may
> > be
> > attached later on.
>=20
> A kind of stupid question: should the reserve amounts depend on the
level
> of the bridge?
> Perhaps the priidges closer to the root should get more reserves.
Perhaps
> it doesn't
> matter so much durin gthe initial enumeration but ma matter latter
after a
> hot plug.
>=20
> Suppose we have the Bridge B1 that gets RSRVE resources attached to
it
> during
> the initial enumeration. Then someone comes and hot-plugs a bridge B2
> under B1.
> B2 then I guess will also try to get a reserve of RSRVE resources for
> itself, so it would
> take the whole original reserve of B1 to itself. If someone comes
later
> and tries
> to hot-plug another bridge B3 under B1, that bridge would not get any
> resources
> and the plugging would fail.
>=20
> -SB
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?8506939B503B404A84BBB12293FC45F606B88E55>