Date: Sat, 28 Mar 2015 16:58:53 -0700 From: Adrian Chadd <adrian@freebsd.org> To: Slawa Olhovchenkov <slw@zxy.spb.ru> Cc: "freebsd-hackers@freebsd.org" <freebsd-hackers@freebsd.org> Subject: Re: irq cpu binding Message-ID: <CAJ-VmokSHHm3kMwz=bp7VbgZwADD2_pEr27NdzUfkGq1U=x_sw@mail.gmail.com> In-Reply-To: <20150328234116.GJ23643@zxy.spb.ru> References: <20150328194959.GE23643@zxy.spb.ru> <CAJ-Vmo=1rzB%2BYNGNuV9s=FnSse7FL7S42OSS4u-PzUs74b850A@mail.gmail.com> <20150328201219.GF23643@zxy.spb.ru> <CAJ-Vmo=wecgoVYcS14gsOnT86p=HEMdao65aXTi7jLfVVyOELg@mail.gmail.com> <20150328221621.GG23643@zxy.spb.ru> <CAJ-Vmomd6Z5Ou7cvV1Kg4m=X2907507hqKMWiz6ssZ45Pi_-Dg@mail.gmail.com> <20150328224634.GH23643@zxy.spb.ru> <CAJ-VmokwGgHGP6AjBcGbyJShBPX6dyJjjNeCBcjxLi1obaiRtQ@mail.gmail.com> <20150328230533.GI23643@zxy.spb.ru> <CAJ-VmongWE_z7Rod8-SoFmyiLqiTbHtSaAwjgAs05L_Z3jrWXA@mail.gmail.com> <20150328234116.GJ23643@zxy.spb.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi, * It turns out that fragments were being 100% handled out of order (compared to non-fragments in the same stream) when doing fragment reassembly, because the current system was assuming direct dispatch netisr and not checking any packet contents for whether they're on the wrong CPU. I checked. It's not noticable unless you go digging, but it's absolutely happening. That's why I spun a lot of cycles looking at the IP fragment reassembly path and which methods get called on the frames as they're reinjected. * We're going to have modify drivers, because the way drivers currently assign interrupts, pick CPUs for queues, auto-select how many queues to use, etc is all completely adhoc and not consistent. So yeah, we're going to change the drivers and they're going to be consistent and configurable. That way you can choose how you want to distribute work and pin or not pin things - and it's not done adhoc differently in each driver. Even igb, ixgbe and cxgbe differ in how they implement these three things. * For RSS, there'll be a consistent configuration for what the hardware is doing with hashing, rather than it being driver dependent. Again, otherwise you may end up with some NICs doing 2-tuple hashing where others are doing 4-tuple hashing, and behaviour changes dramatically based on what NIC you're using. * For applications - I'm not sure yet, but at the minimum the librss API I have vaguely sketched out and coded up in a git branch lets you pull out the list of buckets and which CPU it's on. I'm going to extend that a bit more, but it should be enough for things like nginx to say "ok, start up one nginx process per RSS bucket, and here's the CPU set for it to bind to." You said it has worker groups - that's great; I want that to be auto configured. -adrian
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJ-VmokSHHm3kMwz=bp7VbgZwADD2_pEr27NdzUfkGq1U=x_sw>