From owner-freebsd-net@freebsd.org Wed Oct 21 01:31:50 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 18E0EA19769 for ; Wed, 21 Oct 2015 01:31:50 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: from mail-wi0-x232.google.com (mail-wi0-x232.google.com [IPv6:2a00:1450:400c:c05::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 91571338 for ; Wed, 21 Oct 2015 01:31:49 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: by wicll6 with SMTP id ll6so69615507wic.0 for ; Tue, 20 Oct 2015 18:31:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sippysoft_com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=V705hSiPRkXnu6/nUOKwFKgz8X0tV3gQfzM+GdS0m38=; b=ngQm+6zw7emVASS3MI39+PhHLBDeb49u6TaGLvUME/nn/ffBwvSzah05y5ui4VPM37 K6wko60MrXcFSHfOP6DyFi4KdSQKcq97ZjDOaAzs5vdj8rZnExn7MqYdPABGf+/vAE91 D1xFz2MXYPwd2quvtHTioZqc17K5obSEwyKoBXvOSTw/XRnRnOfkPg1LDB70uDc7SET0 OASPlIzdNCPX/lH2E9+G4AHQG7RgkOZT/4B33rmZ3b8J6rad5opcCK1IO3hXMSpSUHGC rhNDiH8fUj/mr9DJigJEoCVBavqKPyHne81pkLdf/+v9eCt7d3eQB1ALEhvwhbnRVxFW SLag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=V705hSiPRkXnu6/nUOKwFKgz8X0tV3gQfzM+GdS0m38=; b=gx4eNTAUWm/ZO8Yu1FRDSkmP8yXk0G11o3FNmIY5uHbq2XLSxRuVWOSHanSfqpnSH+ 5Fs0t8Fg+TvmijpfT2bzE7BocsU2fcaTV7vl0Elvf9cFo08TXedwD9XtCE1JWnv1HtSa j822vwsxlcTufChONf2EqriryL8Dp81ZLOEJ9gYkSHN+6L2V2sdkhhtYvadXxXJQeutR od6g8+A56TOt4sRSG2BXL0itbfFvpgN1IAsLya9VLs9cgRWUQROAb+uckegYnfcyXmO7 StiyKtCvgMf9R0HHg/UmgRIlUSCZs2HdfQPxZImco6OAxzmyCgGO2/3wynUZQPusfZXP UFbA== X-Gm-Message-State: ALoCoQmHPqYrVXa2NLUwPnon+RDmiamIWaTk9+ryNBLmjfj86FYuiNJaC1pso9jFuFXEUDj/GYHh MIME-Version: 1.0 X-Received: by 10.180.72.177 with SMTP id e17mr29438440wiv.8.1445391107699; Tue, 20 Oct 2015 18:31:47 -0700 (PDT) Sender: sobomax@sippysoft.com Received: by 10.27.11.228 with HTTP; Tue, 20 Oct 2015 18:31:47 -0700 (PDT) In-Reply-To: <1608354.LQmTMSsd5C@ralph.baldwin.cx> References: <1608354.LQmTMSsd5C@ralph.baldwin.cx> Date: Tue, 20 Oct 2015 18:31:47 -0700 X-Google-Sender-Auth: COULeuZRMgTv_hswFpUVyAgCQj8 Message-ID: Subject: Re: Some MSI are not routed correctly From: Maxim Sobolev To: John Baldwin Cc: FreeBSD Net Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Oct 2015 01:31:50 -0000 Here you go: $ sudo procstat -S 11 PID TID COMM TDNAME CPU CSID CPU MASK 11 100026 intr swi3: vm 0 1 0-23 11 100027 intr swi4: clock 0 1 0-23 11 100028 intr swi4: clock 0 1 0-23 11 100029 intr swi4: clock 0 1 0-23 11 100030 intr swi4: clock 0 1 0-23 11 100031 intr swi4: clock 0 1 0-23 11 100032 intr swi4: clock 0 1 0-23 11 100033 intr swi4: clock 0 1 0-23 11 100034 intr swi4: clock 0 1 0-23 11 100035 intr swi4: clock 0 1 0-23 11 100036 intr swi4: clock 0 1 0-23 11 100037 intr swi4: clock 0 1 0-23 11 100038 intr swi4: clock 0 1 0-23 11 100039 intr swi4: clock 0 1 0-23 11 100040 intr swi4: clock 0 1 0-23 11 100041 intr swi4: clock 0 1 0-23 11 100042 intr swi4: clock 0 1 0-23 11 100043 intr swi4: clock 0 1 0-23 11 100044 intr swi4: clock 0 1 0-23 11 100045 intr swi4: clock 0 1 0-23 11 100046 intr swi4: clock 0 1 0-23 11 100047 intr swi4: clock 0 1 0-23 11 100048 intr swi4: clock 0 1 0-23 11 100049 intr swi4: clock 0 1 0-23 11 100050 intr swi4: clock 0 1 0-23 11 100051 intr swi1: netisr 0 18 1 0-23 11 100067 intr swi6: task queue 21 1 0-23 11 100068 intr swi6: Giant task 15 1 0-23 11 100070 intr swi5: fast taskq 0 1 0-23 11 100073 intr irq264: igb0:que 0 1 0 11 100075 intr irq265: igb0:que 1 1 1 11 100077 intr irq266: igb0:que 2 1 2 11 100079 intr irq267: igb0:que 3 1 3 11 100081 intr irq268: igb0:lin 6 1 0-23 11 100082 intr irq269: igb1:que 4 1 4 11 100084 intr irq270: igb1:que 5 1 5 11 100086 intr irq271: igb1:que 6 1 6 11 100088 intr irq272: igb1:que 7 1 7 11 100090 intr irq273: igb1:lin 7 1 0-23 11 100091 intr irq274: ahci0 0 1 0-23 11 100092 intr irq19: xhci0 5 1 0-23 11 100097 intr irq18: ehci0 ehc 4 1 0-23 11 100102 intr irq275: igb2:que 0 1 8 11 100104 intr irq276: igb2:que 0 1 9 11 100106 intr irq277: igb2:que 0 1 10 11 100108 intr irq278: igb2:que 0 1 11 11 100110 intr irq279: igb2:lin 0 1 0-23 11 100111 intr irq280: igb3:que 0 1 12 11 100113 intr irq281: igb3:que 0 1 13 11 100115 intr irq282: igb3:que 0 1 14 11 100117 intr irq283: igb3:que 0 1 15 11 100119 intr irq284: igb3:lin 0 1 0-23 11 100125 intr irq285: mrsas0 11 1 0-23 11 100126 intr irq286: mrsas0 14 1 0-23 11 100127 intr irq287: mrsas0 13 1 0-23 11 100128 intr irq288: mrsas0 14 1 0-23 11 100129 intr irq289: mrsas0 15 1 0-23 11 100130 intr irq290: mrsas0 16 1 0-23 11 100131 intr irq291: mrsas0 17 1 0-23 11 100132 intr irq292: mrsas0 18 1 0-23 11 100133 intr irq293: mrsas0 19 1 0-23 11 100134 intr irq294: mrsas0 20 1 0-23 11 100135 intr irq295: mrsas0 21 1 0-23 11 100136 intr irq296: mrsas0 22 1 0-23 11 100137 intr irq297: mrsas0 23 1 0-23 11 100138 intr irq298: mrsas0 0 1 0-23 11 100139 intr irq299: mrsas0 4 1 0-23 11 100140 intr irq300: mrsas0 2 1 0-23 11 100142 intr swi0: uart uart 0 1 0-23 On Mon, Oct 19, 2015 at 2:03 PM, John Baldwin wrote: > On Thursday, October 08, 2015 07:33:27 AM Maxim Sobolev wrote: > > Hi John & others, > > > > We've came across a weird MSI routing issue on one of our newest dual > > E5-2690v3 (haswell) Supermicro X10DRL-i boxes running latest 10.2-p4. It > is > > fitted with dual port Intel I350 card, in addition to the built-in I210 > > chip that is not used. The hw.igb.num_queues is set to 4, and the driver > > reports binding to the CPUs 0-3 for the first port and CPUs 4-7 for the > > second, however when verified with top -P under the load, interrupts are > > only delivered to the CPUs 0-3, no interrupt time is recorded on the CPUs > > 4-7. systat -vm shows that all 8 queues are firing interrupts, so my > guess > > that for whatever reason bus_bind_intr() is not doing what's expected to > do > > for half of those interrupts. > > > > What's interesting is that on a similar box (same chassis/mobo/cpu) but > > equipped with the quad-port X540-AT2 10Gig card, interrupts are routed > > properly. The latter is running with hw.ix.num_queues="3". > > > > pcib2: port 0xcf8-0xcff on acpi0 > > pci0: on pcib2 > > pcib3: irq 26 at device 1.0 on pci0 > > pci1: on pcib3 > > igb0: mem > > 0xc7200000-0xc72fffff,0xc7304000-0xc7307fff irq 26 at device 0.0 on pci1 > > igb0: Using MSIX interrupts with 5 vectors > > igb0: Ethernet address: a0:36:9f:76:af:20 > > igb0: Bound queue 0 to cpu0 > > igb0: Bound queue 1 to cpu1 > > igb0: Bound queue 2 to cpu2 > > igb0: Bound queue 3 to cpu3 > > igb0: netmap queues/slots: TX 4/4096, RX 4/4096 > > igb1: mem > > 0xc7100000-0xc71fffff,0xc7300000-0xc7303fff irq 28 at device 0.1 on pci1 > > igb1: Using MSIX interrupts with 5 vectors > > igb1: Ethernet address: a0:36:9f:76:af:21 > > igb1: Bound queue 0 to cpu4 > > igb1: Bound queue 1 to cpu5 > > igb1: Bound queue 2 to cpu6 > > igb1: Bound queue 3 to cpu7 > > igb1: netmap queues/slots: TX 4/4096, RX 4/4096 > > > > pcib2: port 0xcf8-0xcff on acpi0 > > pci0: on pcib2 > > pcib3: irq 26 at device 1.0 on pci0 > > pci1: on pcib3 > > pcib4: irq 32 at device 2.0 on pci0 > > pci2: on pcib4 > > pcib5: irq 40 at device 3.0 on pci0 > > pci3: on pcib5 > > ix0: > port > > 0x6020-0x603f mem 0xc7c00000-0xc7dfffff,0xc7e04000-0xc7e07fff irq 40 at > > device 0.0 on pci3 > > ix0: Using MSIX interrupts with 4 vectors > > ix0: Bound queue 0 to cpu 0 > > ix0: Bound queue 1 to cpu 1 > > ix0: Bound queue 2 to cpu 2 > > ix0: Ethernet address: 0c:c4:7a:5e:be:64 > > ix0: PCI Express Bus: Speed 5.0GT/s Width x8 > > ix0: netmap queues/slots: TX 3/4096, RX 3/4096 > > ix1: > port > > 0x6000-0x601f mem 0xc7a00000-0xc7bfffff,0xc7e00000-0xc7e03fff irq 44 at > > device 0.1 on pci3 > > ix1: Using MSIX interrupts with 4 vectors > > ix1: Bound queue 0 to cpu 3 > > ix1: Bound queue 1 to cpu 4 > > ix1: Bound queue 2 to cpu 5 > > ix1: Ethernet address: 0c:c4:7a:5e:be:65 > > ix1: PCI Express Bus: Speed 5.0GT/s Width x8 > > ix1: netmap queues/slots: TX 3/4096, RX 3/4096 > > > > Some extra debug is here: > > > > http://sobomax.sippysoft.com/haswell_bug/bad.dmesg > > http://sobomax.sippysoft.com/haswell_bug/lstopo_bad.png > > http://sobomax.sippysoft.com/haswell_bug/systat_vm_bad.png > > http://sobomax.sippysoft.com/haswell_bug/top_P_bad.png > > > > http://sobomax.sippysoft.com/haswell_bug/good.dmesg > > http://sobomax.sippysoft.com/haswell_bug/lstopo_good.png > > http://sobomax.sippysoft.com/haswell_bug/systat_vm_good.png > > http://sobomax.sippysoft.com/haswell_bug/top_P_good.png > > > > Any ideas on how to debug that further are welcome. The box in the > > production, but we can remove traffic during off-peak to run some > > test/debug code on. > > Can you get procstat -S output for the interrupt threads? (Usually > interrupt > threads are in pid 12, so 'procstat -S 12' would suffice.) > > -- > John Baldwin > >