Date: Mon, 23 Jul 2012 08:24:17 -0600 From: Warner Losh <imp@bsdimp.com> To: Julian Elischer <julian@freebsd.org> Cc: FreeBSD Current <current@freebsd.org> Subject: Re: PCIe hotplug Message-ID: <EB3C2E43-CA98-4783-AA36-7232F7B93D54@bsdimp.com> In-Reply-To: <500D010A.5080808@freebsd.org> References: <500A0E24.80101@freebsd.org> <EABF0570-55F1-4758-B0FF-62561FFAC4EF@samsco.org> <20120722231234.6f748d05@kan.dyndns.org> <F1592617-FBD9-4D2A-80DA-BC8CF5D96F87@bsdimp.com> <500D010A.5080808@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Jul 23, 2012, at 1:45 AM, Julian Elischer wrote: > On 7/22/12 9:11 PM, Warner Losh wrote: >> On Jul 22, 2012, at 9:12 PM, Alexander Kabaev wrote: >>=20 >>> On Sun, 22 Jul 2012 20:22:33 -0600 >>> Scott Long <scottl@samsco.org> wrote: >>>=20 >>>> On Jul 20, 2012, at 8:04 PM, Julian Elischer wrote: >>>>=20 >>>>> Is anyone looking at PCIe hotplug support? >>>>>=20 >>>>> I'm especially interested if anyone has a strategy for device >>>>> re-insertion and reassociating the reinserted device with its old >>>>> device_t so that it gets the same unit number.. (assumes access to >>>>> a serial number or similar) Even if it is put back into a = different >>>>> slot. >>>>>=20 >>>> Would the PCI system be responsible for figuring out this serial >>>> number? I don't think that it can, but it's a question to answer, = I >>>> guess. If it can't then it's up to the driver to generate a unique >>>> cookie that would be stored by the PCI subsystem. This cookie = would >>>> have to be based off of data that can be retrieved from the PCI >>>> config space and/or VPD space, since anything more would require >>>> resource allocation, which is only allowed in the DEV_ATTACH phase, >>>> and once you've hit that phase you've already pretty much sealed = the >>>> deal on unit number assignment. >>>>=20 >>>> So what would probably happen is that the PCI layer provides a ring >>>> buffer of cookie storage and a set of accessors for the drivers. = The >>>> cookies would map to a key-value pair with the device unit name and >>>> number. During probe, a driver can look at PCI config space and >>>> generate a cookie. That cookie can then be communicated up to the >>>> PCI layer for storage. Maybe the driver calls a match routine that >>>> returns a unit number on match and a store on failure, then the >>>> driver calls a set_unit_number accessor. Only the driver that wins >>>> the bid would win the unit number reassignment or cookie storage. = Or >>>> maybe the driver passes the cookie up as part of its return code, = and >>>> the match and unit assignment happens automatically. Drivers that >>>> don't want to participate in this simply wouldn't, and everything >>>> would continue to operate the same way. The two sticky parts are >>>> rogue/buggy drivers that abuse the api and cause a flood of cookies >>>> to be generated, and questions on when a unit number is eligible = for >>>> reuse. For the first one, a ring buffer of cookies would solve the >>>> immediate problem, but you might still have some risk of drivers >>>> selectively wrapping the buffer for whatever accidental or evil >>>> purpose. For the second problem, maybe a unit number stays >>>> persistent only if the PCIe hot remove mechanism requests it, and >>>> then only until the ring-buffer wraps. >>>>=20 >>>> Scott >>>>=20 >>> I do not think the whole problem as depicted by Julian is even worth >>> solving. Why keeping any data for the device that might _never_ come >>> back? What if the device hierarchy just starts from the PCI-e and >>> extends upwards and user still holds on to some vestiges of a = previous >>> device chain (say, by keeping a character control device sharing the >>> same unit number open, common practice)? Reusing unit number is much >>> trickier then, and might not be even possible. So, before one jumps >>> into 'how', can we agree on 'why' first? When device goes away, it = is >>> not just this device's device_t that is disappearing, it is a whole >>> tree rooted at that device. I see no point in trying to reconstruct >>> that. >> There's a reason that PC Card and CardBus never supported this at = all. The assumption was that reconnecting devices is so cheap that it = isn't worth the bother. This is true for all but some specialized = devices today: network information is easy to reconstruct, storage = drives are easy to reconfigure (since we already fail all in-flight = transactions when the device goes away), etc. I can see some advantage = to having storage cope, but there already geom classes that can help = people code when drives can go away. >>=20 >>> PCI-e hotplug proper is very much orthogonal to the question of unit >>> numbering and IS worth supporting. >> Yes. totally agreed. >=20 > I'm not saying that it's vitally important but was wondering if people = had a strategy for it.. > i.e. is it a question worth worrying about? >=20 > In a separate forum Warner and I (yeah I know I'm answering Warner, = but I'm addressing the others) discussed the feasibility of surviving = an "oops pulled the wrong card" event with regards to a particular flash = memory card. I was just carrying that forwards as a thought experiment = (There is actually a strategy that sounds feasible). >=20 > The problem of getting a serial number out of the BAR space during = probe is also possibly solvable in our case but the question of how long = to remember a device is legitimate an My answer would be that > 1/ a particular driver would be able to specify whether it could = handle this, and > 2/ it might be limited to some pragmatic number such as 16 or 32, or a = time limit. Actually, for the case where there's expensive state to reconstruct, = you'd likely want to keep the state around for an estimated time it = would take to reconstruct it. I think that this may necessarily have to = be done outside the framework of newbus, or you'd need a new = mechanism/state that the driver could request as part of its detach = routine that says effectively 'keep this around'. Any later probe on = insert would have to be able to find this and tell newbus about it. = However, that does start to get ugly in a hurry. It would be better, imho, that if a driver wanted to keep this info = around, it would have to take responsibility for finding any such = pending state and cope appropriately. In the case of storage devices, = you'd be able to have those storage volumes on a separate bus, and then = you'd have all the control that you need. But then you start wondering = into issues with what to do with the pending transactions, what to do = with new transactions, etc before giving up. Then again, all these are nice to haves, especially since we don't have = pcie hot plug at all right now. Warner=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?EB3C2E43-CA98-4783-AA36-7232F7B93D54>