Date: Thu, 14 Oct 2010 16:36:02 -0700 From: Jack Vogel <jfvogel@gmail.com> To: Terry Kennedy <TERRY@tmk.com> Cc: freebsd-stable@freebsd.org Subject: Re: Bogus "igb1: Could not setup receive structures" in 8-STABLE Message-ID: <AANLkTimAiQH9Esu=YUkf%2BAF9JX5ZTYhPn-Q1C7kbwxSY@mail.gmail.com> In-Reply-To: <01NT1YE1I98Q008KN2@tmk.com> References: <01NT1YE1I98Q008KN2@tmk.com>
next in thread | previous in thread | raw e-mail | index | archive | help
The problem is mbuf resources, the driver is autoconfiguring the number of queues based on the number of cores, on newer systems with lots of them this is outstripping the mbuf resource pool. I have decided to hard limit the queues to 8, you can fix the number manually by searching for num_queues in if_igb.c and setting it to something other than 0 for now. I am at work on a number of issues with igb and em right now which is why there has not been an MFC yet. Questions to me, Jack On Thu, Oct 14, 2010 at 3:26 PM, Terry Kennedy <TERRY@tmk.com> wrote: > I've run across a strange problem with the igb driver in 8-STABLE - when > I try to do anything with the second igb interface, I get one or more > "igb1: > Could not setup receive structures" error messages. > > This can be reproduced as simply as booting in single-user mode with an > empty /boot/loader.conf and doing: > > ifconfig igb0 up > ifconfig igb1 up > > I've tried to track this down, and as far as I can see, this is from some > change introduced between 8.1-RELEASE (igb 1.9.5) and the current 8-STABLE > (igb 2.0.1). When I try it when booting from the 8.1-RELEASE amd64 DVD, I > can bring up both interfaces. When I try it with an 8-STABLE kernel, I get > the error and igb1 is missing the "RUNNING" flag: > > igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000 > > options=1bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4> > ether 00:25:90:xx:xx:bc > inet6 fe80::225:90ff:fe02:xxbc%igb0 prefixlen 64 scopeid 0x1 > nd6 options=3<PERFORMNUD,ACCEPT_RTADV> > media: Ethernet 1000baseT <full-duplex> > status: active > igb1: flags=8803<UP,BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 9000 > > options=1bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4> > ether 00:25:90:xx:xx:bd > inet6 fe80::225:90ff:fe02:xxbd%igb1 prefixlen 64 tentative scopeid > 0x2 > nd6 options=3<PERFORMNUD,ACCEPT_RTADV> > media: Ethernet 1000baseT <full-duplex> > status: active > > I see 3 mbuf_jumbo_page allocation failures from: > > (1:20) rz1m:/sys/dev/e1000# vmstat -z | grep -v 0\$ > ITEM SIZE LIMIT USED FREE REQUESTS > FAILURES > > 64 Bucket: 536, 0, 263, 3, 263, > 106 > 128 Bucket: 1048, 0, 523, 2, 525, > 139 > mbuf_jumbo_page: 4096, 12800, 12307, 493, 30343, > 3 > > which correspond to the 3 "igb1: Could not setup receive structures" mess- > ages. If I try another "ifconfig igb1 up", I get another console message > and > the counter goes to 4. If I bump the kern.ipc.nmbjumbop sysctl to a larger > value, like 15000, I get the same error message when trying to work with > the > igb1 device, so I don't think it is a "real" error but indicates a problem > in the driver. > > This is on a Supermicro X8DTH-iF, BIOS 2.0a (latest) with a dual on-board > 82576. The dev.igb sysctl's for the two ports (excluding 0 values) are at- > tached. Note that most of the igb1 values are zero: > > (0:34) rz1m:/sys/dev/e1000# sysctl -a | grep igb.1 | grep -v ": 0" > dev.igb.1.%desc: Intel(R) PRO/1000 Network Connection version - 2.0.1 > dev.igb.1.%driver: igb > dev.igb.1.%location: slot=0 function=1 > dev.igb.1.%pnpinfo: vendor=0x8086 device=0x10c9 subvendor=0x15d9 > subdevice=0x0400 class=0x020000 > dev.igb.1.%parent: pci1 > dev.igb.1.nvm: -1 > dev.igb.1.flow_control: 3 > dev.igb.1.enable_aim: 1 > dev.igb.1.rx_processing_limit: 100 > dev.igb.1.link_irq: 1 > dev.igb.1.device_control: 13632065 > dev.igb.1.extended_int_mask: 2147483648 > dev.igb.1.fc_high_water: 47488 > dev.igb.1.fc_low_water: 47472 > > (0:35) rz1m:/sys/dev/e1000# sysctl -a | grep igb.0 | grep -v ": 0" > dev.igb.0.%desc: Intel(R) PRO/1000 Network Connection version - 2.0.1 > dev.igb.0.%driver: igb > dev.igb.0.%location: slot=0 function=0 > dev.igb.0.%pnpinfo: vendor=0x8086 device=0x10c9 subvendor=0x15d9 > subdevice=0x0400 class=0x020000 > dev.igb.0.%parent: pci1 > dev.igb.0.nvm: -1 > dev.igb.0.flow_control: 3 > dev.igb.0.enable_aim: 1 > dev.igb.0.rx_processing_limit: 100 > dev.igb.0.link_irq: 4 > dev.igb.0.device_control: 1087373889 > dev.igb.0.rx_control: 67338274 > dev.igb.0.interrupt_mask: 4 > dev.igb.0.extended_int_mask: 2147484671 > dev.igb.0.fc_high_water: 47488 > dev.igb.0.fc_low_water: 47472 > dev.igb.0.queue0.txd_head: 823 > dev.igb.0.queue0.txd_tail: 823 > dev.igb.0.queue0.tx_packets: 1402 > dev.igb.0.queue0.rxd_head: 319 > dev.igb.0.queue0.rxd_tail: 318 > dev.igb.0.queue0.rx_packets: 319 > dev.igb.0.queue0.rx_bytes: 69075 > dev.igb.0.queue1.txd_head: 2 > dev.igb.0.queue1.txd_tail: 2 > dev.igb.0.queue1.tx_packets: 1 > dev.igb.0.queue1.rxd_head: 52 > dev.igb.0.queue1.rxd_tail: 51 > dev.igb.0.queue1.rx_packets: 52 > dev.igb.0.queue1.rx_bytes: 6369 > dev.igb.0.queue2.txd_head: 27 > dev.igb.0.queue2.txd_tail: 27 > dev.igb.0.queue2.tx_packets: 13 > dev.igb.0.queue2.rxd_head: 72 > dev.igb.0.queue2.rxd_tail: 71 > dev.igb.0.queue2.rx_packets: 72 > dev.igb.0.queue2.rx_bytes: 8789 > dev.igb.0.queue3.txd_head: 177 > dev.igb.0.queue3.txd_tail: 177 > dev.igb.0.queue3.tx_packets: 64 > dev.igb.0.queue3.rxd_head: 88 > dev.igb.0.queue3.rxd_tail: 87 > dev.igb.0.queue3.rx_packets: 88 > dev.igb.0.queue3.rx_bytes: 16454 > dev.igb.0.queue4.rxd_head: 21 > dev.igb.0.queue4.rxd_tail: 20 > dev.igb.0.queue4.rx_packets: 21 > dev.igb.0.queue4.rx_bytes: 3547 > dev.igb.0.queue5.txd_head: 19 > dev.igb.0.queue5.txd_tail: 19 > dev.igb.0.queue5.tx_packets: 9 > dev.igb.0.queue5.rxd_head: 120 > dev.igb.0.queue5.rxd_tail: 119 > dev.igb.0.queue5.rx_packets: 120 > dev.igb.0.queue5.rx_bytes: 35518 > dev.igb.0.queue6.txd_head: 21 > dev.igb.0.queue6.txd_tail: 21 > dev.igb.0.queue6.tx_packets: 10 > dev.igb.0.queue6.rxd_head: 21 > dev.igb.0.queue6.rxd_tail: 20 > dev.igb.0.queue6.rx_packets: 21 > dev.igb.0.queue6.rx_bytes: 2876 > dev.igb.0.queue7.txd_head: 100 > dev.igb.0.queue7.txd_tail: 100 > dev.igb.0.queue7.tx_packets: 50 > dev.igb.0.queue7.rxd_head: 74 > dev.igb.0.queue7.rxd_tail: 73 > dev.igb.0.queue7.rx_packets: 1098 > dev.igb.0.queue7.rx_bytes: 116918 > dev.igb.0.queue8.txd_head: 11 > dev.igb.0.queue8.txd_tail: 11 > dev.igb.0.queue8.tx_packets: 6 > dev.igb.0.queue8.rxd_head: 25 > dev.igb.0.queue8.rxd_tail: 24 > dev.igb.0.queue8.rx_packets: 25 > dev.igb.0.queue8.rx_bytes: 3698 > dev.igb.0.mac_stats.total_pkts_recvd: 3138 > dev.igb.0.mac_stats.good_pkts_recvd: 1815 > dev.igb.0.mac_stats.bcast_pkts_recvd: 383 > dev.igb.0.mac_stats.mcast_pkts_recvd: 114 > dev.igb.0.mac_stats.rx_frames_64: 217 > dev.igb.0.mac_stats.rx_frames_65_127: 673 > dev.igb.0.mac_stats.rx_frames_128_255: 779 > dev.igb.0.mac_stats.rx_frames_256_511: 61 > dev.igb.0.mac_stats.rx_frames_512_1023: 46 > dev.igb.0.mac_stats.rx_frames_1024_1522: 39 > dev.igb.0.mac_stats.good_octets_recvd: 270378 > dev.igb.0.mac_stats.good_octets_txd: 252216 > dev.igb.0.mac_stats.total_pkts_txd: 1554 > dev.igb.0.mac_stats.good_pkts_txd: 1554 > dev.igb.0.mac_stats.bcast_pkts_txd: 36 > dev.igb.0.mac_stats.mcast_pkts_txd: 65 > dev.igb.0.mac_stats.tx_frames_64: 32 > dev.igb.0.mac_stats.tx_frames_65_127: 219 > dev.igb.0.mac_stats.tx_frames_128_255: 1222 > dev.igb.0.mac_stats.tx_frames_256_511: 55 > dev.igb.0.mac_stats.tx_frames_512_1023: 19 > dev.igb.0.mac_stats.tx_frames_1024_1522: 7 > dev.igb.0.interrupts.asserts: 3350 > dev.igb.0.interrupts.rx_pkt_timer: 1815 > dev.igb.0.interrupts.tx_abs_timer: 1815 > dev.igb.0.interrupts.tx_queue_empty: 1554 > dev.igb.0.host.rx_good_bytes: 270378 > dev.igb.0.host.tx_good_bytes: 252216 > > Both ports are cabled to the same Cisco switch. If I swap the cables at > the FreeBSD end between igb0 and igb1, the problem stays with igb1 so I > don't think it is the switch. > > I have 3 identical systems, all of which exhibit this same issue. Un- > fortunately, I don't have any other hardware with dual igb's. > > I can give a developer root access as well as a web-based remote console > if needed to track this down. > > Terry Kennedy http://www.tmk.com > terry@tmk.com New York, NY USA >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?AANLkTimAiQH9Esu=YUkf%2BAF9JX5ZTYhPn-Q1C7kbwxSY>