From owner-freebsd-arch@FreeBSD.ORG Fri Feb 12 04:30:53 2010 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DE4CE106566C; Fri, 12 Feb 2010 04:30:52 +0000 (UTC) (envelope-from rajatjain@juniper.net) Received: from exprod7og121.obsmtp.com (exprod7og121.obsmtp.com [64.18.2.20]) by mx1.freebsd.org (Postfix) with ESMTP id 1DCB18FC08; Fri, 12 Feb 2010 04:30:51 +0000 (UTC) Received: from source ([66.129.224.36]) (using TLSv1) by exprod7ob121.postini.com ([64.18.6.12]) with SMTP ID DSNKS3TZe8WzYsKOhhr+zRsc0sdv7dwTQCFw@postini.com; Thu, 11 Feb 2010 20:30:52 PST Received: from emailbng1.jnpr.net (10.209.194.15) by P-EMHUB03-HQ.jnpr.net (172.24.192.37) with Microsoft SMTP Server id 8.1.393.1; Thu, 11 Feb 2010 20:29:22 -0800 Received: from emailbng3.jnpr.net ([10.209.194.27]) by emailbng1.jnpr.net with Microsoft SMTPSVC(6.0.3790.3959); Fri, 12 Feb 2010 09:59:19 +0530 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-Class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Date: Fri, 12 Feb 2010 09:59:19 +0530 Message-ID: <8506939B503B404A84BBB12293FC45F606A74F94@emailbng3.jnpr.net> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Understanding PCI enumeration for hot-plug Thread-Index: Acqrm/FL8jWF2VBsS6yi1Nozfl/z7Q== From: Rajat Jain To: , X-OriginalArrivalTime: 12 Feb 2010 04:29:19.0868 (UTC) FILETIME=[F1CFABC0:01CAAB9B] Cc: freebsd-ia32@freebsd.org, bms@freebsd.org, freebsd-ppc@freebsd.org Subject: Understanding PCI enumeration for hot-plug X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Feb 2010 04:30:53 -0000 Hello, I need to support build hot-plug in a FreeBSD based system where I know in advance the devices (and the PCI hierarchy on them) that can be hot-plugged into the system.=20 I'm trying to understand the PCI enumeration / re-enumeration process in the context of hot-plug / unplug. I'm assuming the case where the firmware is really dumb and the kernel needs to manage / allocate all PCI resources.=20 I'm going through the PCI specs (and PCI-to-PCI bridge specs) and here is what I think needs to be done when ever PCI / PCIe devices are added / removed from the system. I would appreciate if some one could please confirm my understanding and point out if I am missing something: PCI-REOURCE / BUS-NUMBER MGMT =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D [IFF NECESSARY] PCI configuration space of all the bridges needs to be re-written, right from the immediate parent of the device being removed until the host bridge in order to ensure that: a) Each parent bridge has secondary and subordinate bus range set so as to include all the bus numbers below it. This is required to forward configuration transactions. b) Each parent bridge has IO base and IO limit set so as to include all the IO address space below it. c) Each parent bridge has Memory base and Memory limit set so as to include all the Memory address space below it. d) Each parent bridge has Pre-fetch Memory base and Memory limit set so as to include all the Pre-fetch Memory address space below it. Note-1: The reason a/b/c/d above are marked "IFF NECESSARY" is that we can avoid all the above work if we can pre-allocate the above resources for future devices, and set these parent bridges ranges accordingly.=20 IN other words, consider a system where we know in advance, the PCI device tree on the devices that can be hot-plugged into the system. Here we can set aside PCI bus numbers and IO / Mem / Prefetchable memory ranges for them in advance. And thus configure the parent bridges to already include those ranges. Thus later when the devices are added, none of the parent bridges will need to be re-programmed. Is my understanding correct? DEVICE DETECTION & INITIALIZATION =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D e) Upon detection of the device (By attempting to read its config space), the most important item is to program its BARs to appropriate address spaces as requested by the device. The BARs need to be programmed such that they are included in the appropriate Base / limit registers of all the bridges upstream. Correct? Again, if we've used the strategy specified in Note-1, we can simply use the range we've already aside for this device. MY QUESTIONS =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D 1) Is my above understanding correct? 2) Does anything else also needs to be done in order to make it work? 3) As I said I'm trying to make it fast and optimize for the case where I know the devices [thus the PCI tree] that can be plugged in. Will my strategy specified in Note-1 work? So all I need to configure is my newly detected devices / bridges and not the existing ones... Thanks in Advance, Rajat Jain