Date: Fri, 28 Nov 2014 08:54:23 +0100 From: Hans Petter Selasky <hps@selasky.org> To: Adrian Chadd <adrian@freebsd.org> Cc: "K. Macy" <kmacy@freebsd.org>, FreeBSD Current <freebsd-current@freebsd.org>, Michael Tuexen <tuexen@freebsd.org>, Navdeep Parhar <np@freebsd.org>, Lawrence Stewart <lstewart@freebsd.org>, Luigi Rizzo <rizzo@iet.unipi.it> Subject: Re: [RFC] Removal of M_FLOWID flag from m_flags [WAS: Add support for hardware transmit rate limiting queues] Message-ID: <54782A2F.3070103@selasky.org> In-Reply-To: <CAJ-Vmon0rtyej_pGJ1bptRfmcAV0SfsZ3KRRW7yPrf6REyrMFQ@mail.gmail.com> References: <546CE948.2070105@selasky.org> <CAHM0Q_NwfjH-XqwTaRiZdntAiT=73-dETogQRWeyhEZQm-Ky2Q@mail.gmail.com> <546D0CE3.602@selasky.org> <54775B0E.3000004@selasky.org> <CAJ-Vmo=qc255RsqQB7UJv%2Bo7CzNGG_Mfny_XRk2=jJaUwwSu8g@mail.gmail.com> <CAJ-Vmo=VG6_og-nYYwLkfHh-DOZQf6NM=aUzaBGY890vCAqTVg@mail.gmail.com> <CAJ-VmomoN9cQiZTq5NDV%2BCAPypN=1BvzO2fc%2BruAtL81AsRhFw@mail.gmail.com> <54775E81.2070505@selasky.org> <CAJ-Vmon0rtyej_pGJ1bptRfmcAV0SfsZ3KRRW7yPrf6REyrMFQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 11/27/14 18:33, Adrian Chadd wrote: > On 27 November 2014 at 09:25, Hans Petter Selasky <hps@selasky.org> wrote: >> On 11/27/14 18:20, Adrian Chadd wrote: >>> >>> On 27 November 2014 at 09:18, Adrian Chadd <adrian@freebsd.org> wrote: >>>> >>>> Erk - SCTP folk - are you using the mbuf flowid field for something >>>> SCTP specific? >>> >>> >>> erk - yes, you are. >>> >>> It seems we're going to run into "what exactly should flowid be used >>> for" problems. >>> >> >> Hi, >> >> If the flowid has special meaning inside the SCTP, please define an own >> "rsstype" for it. As far as I could see, it is only used to spread the >> traffic on the network adapter and on the CPU cores. > > I'm more worried about at what layers we're going to be using the > flowid values and that various pieces may stomp over other pieces. > > With the RSS stuff enabled, the IPv4 (and soon IPv6) input paths will > re-hash an input frame if it doesn't have an RSS hash; it'll then > overwrite whatever flowid value is in the mbuf. This is so no matter > what the NIC hands us or what de-encapsulation hands us, we will > always put that flow on into the right RSS bucket and a consistent > CPU. This should mostly alleviate any out-of-order issues seen with > internet-facing machines where things handle fragments and such. Hi, I'm not changing anything in the receive direction regarding the flowid. It should be exactly the same as before, except M_HASHTYPE_NONE which now indicates that flowid is not set. > > Then for the output side of things, it'll have to do a software RSS > hash on frames that don't have an RSS hash. Right now I assume in the > TCP path that the inp has an RSS flow setup in the inp and that the > TCP timers and other assorted stuff uses the inp details. > > For the UDP output side of things I currently always re-calculate the > RSS hash and stamp the mbuf with it before we send it out, again so it > ends up on the same output RSS bucket and thus CPU / NIC queue. > > If the inp ends up with a different flowid (eg a hardware ring flowid) then: > > * it won't match up on the receive side, so the whole RSS lock > contention avoidance thing can't happen; > * there's a known mapping for RSS bucket -> CPU IDs (since we're not > doing RSS bucket -> CPU ID reassignment yet) - but not for other > flowid types to CPU IDs. Not necessarily. Would could make a standard, that the lower X-bits of the flowid, always indicate RX/TX ring pair and the CPU core, and then the upper 32-X bits are free to use for other purposes. > > In general I think the tidyup patch looks fine and I'll do some more > RSS testing once it's committed. (But you did sneak in your new hash > type in the diff :-) I'll put the new hash type in a separate patch. It doesn't belong there - you're right. > > I think we then need to actually plan out how this stuff should hold > together before we all rush into it or we'll end up with a mess of > components that can't actually interact together. I don't want to have > to explain to people that they can't use SCTP, RSS and hardware ring / > flow assignment at the same time. It's just going to be painful in the > long run. Do you have code not committed which plan to use the flowid in new areas of the FreeBSD kernel? I would like to have a list of usages for the flowid field? --HPS
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?54782A2F.3070103>