Skip site navigation (1)Skip section navigation (2)
Date:      Tue, 20 Oct 2015 18:31:47 -0700
From:      Maxim Sobolev <sobomax@FreeBSD.org>
To:        John Baldwin <jhb@freebsd.org>
Cc:        FreeBSD Net <freebsd-net@freebsd.org>
Subject:   Re: Some MSI are not routed correctly
Message-ID:  <CAH7qZfsU2hOcsfh-a9k1WrNZ6MdfhSvP40hKWs4ZnRo9TnC=zQ@mail.gmail.com>
In-Reply-To: <1608354.LQmTMSsd5C@ralph.baldwin.cx>
References:  <CAH7qZfscsmvU6E5d-VXqN_xz_Bjs7i6K0izd%2Bv8X2nm6Q9s7Aw@mail.gmail.com> <1608354.LQmTMSsd5C@ralph.baldwin.cx>

next in thread | previous in thread | raw e-mail | index | archive | help
Here you go:

$ sudo procstat -S 11
  PID    TID COMM             TDNAME           CPU CSID CPU MASK
   11 100026 intr             swi3: vm           0    1 0-23
   11 100027 intr             swi4: clock        0    1 0-23
   11 100028 intr             swi4: clock        0    1 0-23
   11 100029 intr             swi4: clock        0    1 0-23
   11 100030 intr             swi4: clock        0    1 0-23
   11 100031 intr             swi4: clock        0    1 0-23
   11 100032 intr             swi4: clock        0    1 0-23
   11 100033 intr             swi4: clock        0    1 0-23
   11 100034 intr             swi4: clock        0    1 0-23
   11 100035 intr             swi4: clock        0    1 0-23
   11 100036 intr             swi4: clock        0    1 0-23
   11 100037 intr             swi4: clock        0    1 0-23
   11 100038 intr             swi4: clock        0    1 0-23
   11 100039 intr             swi4: clock        0    1 0-23
   11 100040 intr             swi4: clock        0    1 0-23
   11 100041 intr             swi4: clock        0    1 0-23
   11 100042 intr             swi4: clock        0    1 0-23
   11 100043 intr             swi4: clock        0    1 0-23
   11 100044 intr             swi4: clock        0    1 0-23
   11 100045 intr             swi4: clock        0    1 0-23
   11 100046 intr             swi4: clock        0    1 0-23
   11 100047 intr             swi4: clock        0    1 0-23
   11 100048 intr             swi4: clock        0    1 0-23
   11 100049 intr             swi4: clock        0    1 0-23
   11 100050 intr             swi4: clock        0    1 0-23
   11 100051 intr             swi1: netisr 0    18    1 0-23
   11 100067 intr             swi6: task queue  21    1 0-23
   11 100068 intr             swi6: Giant task  15    1 0-23
   11 100070 intr             swi5: fast taskq   0    1 0-23
   11 100073 intr             irq264: igb0:que   0    1 0
   11 100075 intr             irq265: igb0:que   1    1 1
   11 100077 intr             irq266: igb0:que   2    1 2
   11 100079 intr             irq267: igb0:que   3    1 3
   11 100081 intr             irq268: igb0:lin   6    1 0-23
   11 100082 intr             irq269: igb1:que   4    1 4
   11 100084 intr             irq270: igb1:que   5    1 5
   11 100086 intr             irq271: igb1:que   6    1 6
   11 100088 intr             irq272: igb1:que   7    1 7
   11 100090 intr             irq273: igb1:lin   7    1 0-23
   11 100091 intr             irq274: ahci0      0    1 0-23
   11 100092 intr             irq19: xhci0       5    1 0-23
   11 100097 intr             irq18: ehci0 ehc   4    1 0-23
   11 100102 intr             irq275: igb2:que   0    1 8
   11 100104 intr             irq276: igb2:que   0    1 9
   11 100106 intr             irq277: igb2:que   0    1 10
   11 100108 intr             irq278: igb2:que   0    1 11
   11 100110 intr             irq279: igb2:lin   0    1 0-23
   11 100111 intr             irq280: igb3:que   0    1 12
   11 100113 intr             irq281: igb3:que   0    1 13
   11 100115 intr             irq282: igb3:que   0    1 14
   11 100117 intr             irq283: igb3:que   0    1 15
   11 100119 intr             irq284: igb3:lin   0    1 0-23
   11 100125 intr             irq285: mrsas0    11    1 0-23
   11 100126 intr             irq286: mrsas0    14    1 0-23
   11 100127 intr             irq287: mrsas0    13    1 0-23
   11 100128 intr             irq288: mrsas0    14    1 0-23
   11 100129 intr             irq289: mrsas0    15    1 0-23
   11 100130 intr             irq290: mrsas0    16    1 0-23
   11 100131 intr             irq291: mrsas0    17    1 0-23
   11 100132 intr             irq292: mrsas0    18    1 0-23
   11 100133 intr             irq293: mrsas0    19    1 0-23
   11 100134 intr             irq294: mrsas0    20    1 0-23
   11 100135 intr             irq295: mrsas0    21    1 0-23
   11 100136 intr             irq296: mrsas0    22    1 0-23
   11 100137 intr             irq297: mrsas0    23    1 0-23
   11 100138 intr             irq298: mrsas0     0    1 0-23
   11 100139 intr             irq299: mrsas0     4    1 0-23
   11 100140 intr             irq300: mrsas0     2    1 0-23
   11 100142 intr             swi0: uart uart    0    1 0-23


On Mon, Oct 19, 2015 at 2:03 PM, John Baldwin <jhb@freebsd.org> wrote:

> On Thursday, October 08, 2015 07:33:27 AM Maxim Sobolev wrote:
> > Hi John & others,
> >
> > We've came across a weird MSI routing issue on one of our newest dual
> > E5-2690v3 (haswell) Supermicro X10DRL-i boxes running latest 10.2-p4. It
> is
> > fitted with dual port Intel I350 card, in addition to the built-in I210
> > chip that is not used. The hw.igb.num_queues is set to 4, and the driver
> > reports binding to the CPUs 0-3 for the first port and CPUs 4-7 for the
> > second, however when verified with top -P under the load, interrupts are
> > only delivered to the CPUs 0-3, no interrupt time is recorded on the CPUs
> > 4-7. systat -vm shows that all 8 queues are firing interrupts, so my
> guess
> > that for whatever reason bus_bind_intr() is not doing what's expected to
> do
> > for half of those interrupts.
> >
> > What's interesting is that on a similar box (same chassis/mobo/cpu) but
> > equipped with the quad-port X540-AT2 10Gig card, interrupts are routed
> > properly. The latter is running with hw.ix.num_queues="3".
> >
> > pcib2: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
> > pci0: <ACPI PCI bus> on pcib2
> > pcib3: <ACPI PCI-PCI bridge> irq 26 at device 1.0 on pci0
> > pci1: <ACPI PCI bus> on pcib3
> > igb0: <Intel(R) PRO/1000 Network Connection version - 2.4.0> mem
> > 0xc7200000-0xc72fffff,0xc7304000-0xc7307fff irq 26 at device 0.0 on pci1
> > igb0: Using MSIX interrupts with 5 vectors
> > igb0: Ethernet address: a0:36:9f:76:af:20
> > igb0: Bound queue 0 to cpu0
> > igb0: Bound queue 1 to cpu1
> > igb0: Bound queue 2 to cpu2
> > igb0: Bound queue 3 to cpu3
> > igb0: netmap queues/slots: TX 4/4096, RX 4/4096
> > igb1: <Intel(R) PRO/1000 Network Connection version - 2.4.0> mem
> > 0xc7100000-0xc71fffff,0xc7300000-0xc7303fff irq 28 at device 0.1 on pci1
> > igb1: Using MSIX interrupts with 5 vectors
> > igb1: Ethernet address: a0:36:9f:76:af:21
> > igb1: Bound queue 0 to cpu4
> > igb1: Bound queue 1 to cpu5
> > igb1: Bound queue 2 to cpu6
> > igb1: Bound queue 3 to cpu7
> > igb1: netmap queues/slots: TX 4/4096, RX 4/4096
> >
> > pcib2: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
> > pci0: <ACPI PCI bus> on pcib2
> > pcib3: <ACPI PCI-PCI bridge> irq 26 at device 1.0 on pci0
> > pci1: <ACPI PCI bus> on pcib3
> > pcib4: <ACPI PCI-PCI bridge> irq 32 at device 2.0 on pci0
> > pci2: <ACPI PCI bus> on pcib4
> > pcib5: <ACPI PCI-PCI bridge> irq 40 at device 3.0 on pci0
> > pci3: <ACPI PCI bus> on pcib5
> > ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.8.3>
> port
> > 0x6020-0x603f mem 0xc7c00000-0xc7dfffff,0xc7e04000-0xc7e07fff irq 40 at
> > device 0.0 on pci3
> > ix0: Using MSIX interrupts with 4 vectors
> > ix0: Bound queue 0 to cpu 0
> > ix0: Bound queue 1 to cpu 1
> > ix0: Bound queue 2 to cpu 2
> > ix0: Ethernet address: 0c:c4:7a:5e:be:64
> > ix0: PCI Express Bus: Speed 5.0GT/s Width x8
> > ix0: netmap queues/slots: TX 3/4096, RX 3/4096
> > ix1: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.8.3>
> port
> > 0x6000-0x601f mem 0xc7a00000-0xc7bfffff,0xc7e00000-0xc7e03fff irq 44 at
> > device 0.1 on pci3
> > ix1: Using MSIX interrupts with 4 vectors
> > ix1: Bound queue 0 to cpu 3
> > ix1: Bound queue 1 to cpu 4
> > ix1: Bound queue 2 to cpu 5
> > ix1: Ethernet address: 0c:c4:7a:5e:be:65
> > ix1: PCI Express Bus: Speed 5.0GT/s Width x8
> > ix1: netmap queues/slots: TX 3/4096, RX 3/4096
> >
> > Some extra debug is here:
> >
> > http://sobomax.sippysoft.com/haswell_bug/bad.dmesg
> > http://sobomax.sippysoft.com/haswell_bug/lstopo_bad.png
> > http://sobomax.sippysoft.com/haswell_bug/systat_vm_bad.png
> > http://sobomax.sippysoft.com/haswell_bug/top_P_bad.png
> >
> > http://sobomax.sippysoft.com/haswell_bug/good.dmesg
> > http://sobomax.sippysoft.com/haswell_bug/lstopo_good.png
> > http://sobomax.sippysoft.com/haswell_bug/systat_vm_good.png
> > http://sobomax.sippysoft.com/haswell_bug/top_P_good.png
> >
> > Any ideas on how to debug that further are welcome. The box in the
> > production, but we can remove traffic during off-peak to run some
> > test/debug code on.
>
> Can you get procstat -S output for the interrupt threads?  (Usually
> interrupt
> threads are in pid 12, so 'procstat -S 12' would suffice.)
>
> --
> John Baldwin
>
>



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAH7qZfsU2hOcsfh-a9k1WrNZ6MdfhSvP40hKWs4ZnRo9TnC=zQ>