Date: Sat, 2 Mar 2013 01:46:34 -0800 From: Adrian Chadd <adrian@freebsd.org> To: freebsd-wireless@freebsd.org Subject: [RFC] net80211 TX, take 4 (final) Message-ID: <CAJ-VmomnbLh9OdpNVL1=ahkuN6Y_LGY%2BZLbW-PK=mTAZducLCA@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
Hi, So here's where I'm at: * There's the serialisation of frames that have 802.11 state attached - eg, the encap path for normal VAP TX, for management frame TX, mesh, FF TX, etc; * There's then the serialisation of _dispatching those frames in order to the driver_; * The former will be too hard to serialise via a queue - at least until I get a whole bunch more time and can rewrite all of the different frame TX dispatch bits; * And we don't have an easy way to dispatch frames in order to the underlying driver - all we have is if_transmit() which hands a frame to the driver; it doesn't at all enforce ordering. So what I think I'm going to settle on: Immediate: * Create an IC TX serialisation lock which (for now) serialises both the IC transmission and the VAP TX state; * Grab the IC TX Lock when it's time to encapsulate an 802.3 frame and push it either onto a driver queue (via parent->if_transmit()); * If the driver does deferred transmission (ie, if it calls if_start to drain the parent ifnet queue after TX completion or reset) then it should also grab the IC TX lock too, just to ensure that driver dispatch is done in-order - but this isn't strictly needed, see below. Now, the why: * There's plenty of paths into TX - most notably the various places that create raw management frames to dispatch; * I don't mind that various places are grabbing 802.11 state (ie sequence numbers) and pushing things in-order to the driver - but they're doing it via ic_raw_xmit() rather than via the driver transmit method; * And we have to serialise frames from the point we assign 802.11 encapsulation state all the way through to queuing to the driver - AND the driver has to dequeue/handle these in order; * .. and if_transmit() by itself doesn't give us that. Now, I don't want to invent a new queue method. What I want to do is get this working correct now, and worry about fixing it when I'm at BSDCan and can discuss this with other networking people. What I'd like to do later on, once this is in -HEAD: * I'd like to create a new mbuf tag - that holds "802.11 TX state". This for now will include the TX BPF parameters but it can grow to include other things. * I'd then like to create a way to encapsulate a list of mbuf LISTS - right now the bufring and the mbuf packet lists use m_nextpkt - but net80211 creates fragment lists to pass to the driver by using m_nextpkt. So right now if we TX fragments we silently leak mbufs. It sucks, but that's what I have to deal with. * Next, the VAP transmit path and anything else calling parent->if_transmit() (with what I hope is an encapsulated, in-order frame) should queue mbuf lists into an ic staging queue; * Next, create a single-threaded driver dispatch method - likely a taskqueue for now - that walks the ic staging queue and calls the driver transmit method; * Then I'd like all the places that call ic->ic_raw_xmit() to instead create a new frame and push it into the ic staging queue, maintaining the correct sequence number ordering; * .. and teach the ic dispatch task to call either parent->if_transmit() or ic->ic_raw_xmit() depending upon if the frame was raw or not. * Once that's done, I can unwind that lock back to be per-VAP or per-STA, as now we're not using it as a "driver queue ordering" enforcing lock. _THEN_: * Migrate all wifi drivers to use if_transmit() instead of if_start(), so mbuf lists can be handed to the driver; * I'd like to teach all the wifi drivers about the "802.11 TX state" tag - and if an mbuf appears with the mbuf tag, it's passed through the driver raw method; * .. and then _I can kill ic_raw_xmit() entirely_. What I'd really really like to do: The main shortcoming here is the lack of ordered driver dispatching. Right now we call parent->if_transmit() which will either directly dispatch to the driver or queue via IF_ENQUEUE(), then call if_start()). What I really really want to do here is separate out driver queuing from driver dispatch - I want to hold a VAP or STA TX lock whilst setting up the 802.11 encapsulation and whilst I push it into this queue - and then I want to release the lock and call the driver dispatch routine. The driver is then responsible for dequeuing and handling frames in order. Right now there's no way to guarantee the driver handles things in the right order without holding a lock across calling parent->if_transmit(), which totally sucks. I'll talk with the networking cabal about this as I think they've been thinking about the larger scale requirements for this kind of queue discipline/management. So - with this final plan in mind - what do people think? I think I've got all the basic things down pat. Thanks, Adrian
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJ-VmomnbLh9OdpNVL1=ahkuN6Y_LGY%2BZLbW-PK=mTAZducLCA>