Date: Fri, 11 Apr 2014 14:30:44 -0400 From: Patrick Kelsey <kelsey@ieee.org> To: Adrian Chadd <adrian@freebsd.org> Cc: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org>, hiren panchasara <hiren.panchasara@gmail.com> Subject: Re: netisr observations Message-ID: <CAD44qMUVLLw0UNTgaTZ74=Ktq46ROT9E%2BssrHznHPhqujScBkA@mail.gmail.com> In-Reply-To: <CAJ-Vmo=TUVwuoWJeTCYvC-2sYvLRh%2BevACukS%2BaNHOaz9hwkrA@mail.gmail.com> References: <CALCpEUHhUkZ9b=2ynaN5-MkxOObs%2BO4RTsUhmhcMeC-WDnAxKg@mail.gmail.com> <CAJ-Vmo=TUVwuoWJeTCYvC-2sYvLRh%2BevACukS%2BaNHOaz9hwkrA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, Apr 11, 2014 at 2:48 AM, Adrian Chadd <adrian@freebsd.org> wrote: > [snip] > > So, hm, the thing that comes to mind is the flowid. What's the various > flowid's for flows? Are they all mapping to CPU 3 somehow The output of netstat -Q shows IP dispatch is set to default, which is direct (NETISR_DISPATCH_DIRECT). That means each IP packet will be processed on the same CPU that the Ethernet processing for that packet was performed on, so CPU selection for IP packets will not be based on flowid. The output of netstat -Q shows Ethernet dispatch is set to direct (NETISR_DISPATCH_DIRECT if you wind up reading the code), so the Ethernet processing for each packet will take place on the same CPU that the driver receives that packet on. For the igb driver with queues autoconfigured and msix enabled, as the sysctl output shows you have, the driver will create a number of queues subject to device limitations, msix message limitations, and the number of CPUs in the system, establish a separate interrupt handler for each one, and bind each of those interrupt handlers to a separate CPU. It also creates a separate single-threaded taskqueue for each queue. Each queue interrupt handler sends work to its associated taskqueue when the interrupt fires. Those taskqueues are where the Ethernet packets are received and processed by the driver. The question is where those taskqueue threads will be run. I don't see anything in the driver that makes an attempt to bind those taskqueue threads to specific CPUs, so really the location of all of the packet processing is up to the scheduler (i.e., arbitrary). The summary is: 1. the hardware schedules each received packet to one of its queues and raises the interrupt for that queue 2. that queue interrupt is serviced on the same CPU all the time, which is different from the CPUs for all other queues on that interface 3. the interrupt handler notifies the corresponding task queue, which runs its task in a thread on whatever CPU the scheduler chooses 4. that task dispatches the packet for Ethernet processing via netisr, which processes it on whatever the current CPU is 5. Ethernet processing dispatches that packet for IP processing via netisr, which processes it on whatever the current CPU is You might want to try changing the default netisr dispatch policy to 'deferred' (sysctl net.isr.dispatch=deferred). If you do that, the Ethernet processing will still happen on an arbitrary CPU chosen by the scheduler, but the IP processing should then get mapped to a CPU based on the flowid assigned by the driver. Since igb assigns flowids based on received queue number, all IP (and above) processing for that packet should then be performed on the same CPU the queue interrupt was bound to. Unfortunately, I don't have a system with igb interfaces to try that on. -Patrick
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAD44qMUVLLw0UNTgaTZ74=Ktq46ROT9E%2BssrHznHPhqujScBkA>