Skip site navigation (1)Skip section navigation (2)
Date:      Thu, 17 Jul 2014 15:49:02 +0800
From:      Xu Zhe <peterxu@cyphy.net>
To:        freebsd-net@freebsd.org
Cc:        Javen Wu <javenwu@cyphy.net>, Jason Zhang <jasonzhang@cyphy.net>
Subject:   Network unstability issue with ixgbe driver (ix0 local_faults non zero)
Message-ID:  <53C77FEE.9000707@cyphy.net>

next in thread | raw e-mail | index | archive | help
Hi, Freebsd developers,

We are encountering network problem on Freebsd (version 8.2), with Intel X540T
10g card and ixgbe 2.5.15 (also tried a older version 2.5.8) driver. First, we
found the problem when SSH always fails due to timed out. Then we found that
it is possibly a generic network issue rather than SSH problem.

We found non-zero local_faults and remote_faults in sysctl:

# sysctl -a | grep ix.0
dev.ix.0.%desc: Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.5.8
dev.ix.0.%driver: ix
dev.ix.0.%location: slot=0 function=0 handle=\_SB_.PCI0.NPE3.TGBE
dev.ix.0.%pnpinfo: vendor=0x8086 device=0x1528 subvendor=0x152d
subdevice=0x899f class=0x020000
dev.ix.0.%parent: pci3
dev.ix.0.fc: 3
dev.ix.0.enable_aim: 1
dev.ix.0.advertise_speed: 0
dev.ix.0.ts: 0
dev.ix.0.dropped: 0
dev.ix.0.mbuf_defrag_failed: 0
dev.ix.0.watchdog_events: 0
dev.ix.0.link_irq: 4
dev.ix.0.queue0.interrupt_rate: 55555
dev.ix.0.queue0.irqs: 1491075
dev.ix.0.queue0.txd_head: 604
dev.ix.0.queue0.txd_tail: 604
dev.ix.0.queue0.tso_tx: 154
dev.ix.0.queue0.no_tx_dma_setup: 0
dev.ix.0.queue0.no_desc_avail: 0
dev.ix.0.queue0.tx_packets: 948089
dev.ix.0.queue0.rxd_head: 620
dev.ix.0.queue0.rxd_tail: 619
dev.ix.0.queue0.rx_packets: 7799404
dev.ix.0.queue0.rx_bytes: 11075537104
dev.ix.0.queue0.rx_copies: 111468
dev.ix.0.queue0.lro_queued: 7788218
dev.ix.0.queue0.lro_flushed: 968958
dev.ix.0.queue1.interrupt_rate: 100000
dev.ix.0.queue1.irqs: 90817
dev.ix.0.queue1.txd_head: 1800
dev.ix.0.queue1.txd_tail: 1800
dev.ix.0.queue1.tso_tx: 2
dev.ix.0.queue1.no_tx_dma_setup: 0
dev.ix.0.queue1.no_desc_avail: 0
dev.ix.0.queue1.tx_packets: 32468
dev.ix.0.queue1.rxd_head: 1802
dev.ix.0.queue1.rxd_tail: 1801
dev.ix.0.queue1.rx_packets: 40714
dev.ix.0.queue1.rx_bytes: 4527395
dev.ix.0.queue1.rx_copies: 38784
dev.ix.0.queue1.lro_queued: 38668
dev.ix.0.queue1.lro_flushed: 38486
dev.ix.0.queue2.interrupt_rate: 71428
dev.ix.0.queue2.irqs: 28625
dev.ix.0.queue2.txd_head: 349
dev.ix.0.queue2.txd_tail: 349
dev.ix.0.queue2.tso_tx: 1
dev.ix.0.queue2.no_tx_dma_setup: 0
dev.ix.0.queue2.no_desc_avail: 0
dev.ix.0.queue2.tx_packets: 6981
dev.ix.0.queue2.rxd_head: 1952
dev.ix.0.queue2.rxd_tail: 1951
dev.ix.0.queue2.rx_packets: 6048
dev.ix.0.queue2.rx_bytes: 947930
dev.ix.0.queue2.rx_copies: 5241
dev.ix.0.queue2.lro_queued: 4846
dev.ix.0.queue2.lro_flushed: 4760
dev.ix.0.queue3.interrupt_rate: 500000
dev.ix.0.queue3.irqs: 54879
dev.ix.0.queue3.txd_head: 504
dev.ix.0.queue3.txd_tail: 504
dev.ix.0.queue3.tso_tx: 10
dev.ix.0.queue3.no_tx_dma_setup: 0
dev.ix.0.queue3.no_desc_avail: 0
dev.ix.0.queue3.tx_packets: 18406
dev.ix.0.queue3.rxd_head: 449
dev.ix.0.queue3.rxd_tail: 448
dev.ix.0.queue3.rx_packets: 20929
dev.ix.0.queue3.rx_bytes: 2572540
dev.ix.0.queue3.rx_copies: 20297
dev.ix.0.queue3.lro_queued: 19218
dev.ix.0.queue3.lro_flushed: 19102
dev.ix.0.queue4.interrupt_rate: 500000
dev.ix.0.queue4.irqs: 22609
dev.ix.0.queue4.txd_head: 1370
dev.ix.0.queue4.txd_tail: 1370
dev.ix.0.queue4.tso_tx: 1
dev.ix.0.queue4.no_tx_dma_setup: 0
dev.ix.0.queue4.no_desc_avail: 0
dev.ix.0.queue4.tx_packets: 3518
dev.ix.0.queue4.rxd_head: 1622
dev.ix.0.queue4.rxd_tail: 1621
dev.ix.0.queue4.rx_packets: 3670
dev.ix.0.queue4.rx_bytes: 474745
dev.ix.0.queue4.rx_copies: 3014
dev.ix.0.queue4.lro_queued: 2174
dev.ix.0.queue4.lro_flushed: 2171
dev.ix.0.queue5.interrupt_rate: 100000
dev.ix.0.queue5.irqs: 366375
dev.ix.0.queue5.txd_head: 833
dev.ix.0.queue5.txd_tail: 833
dev.ix.0.queue5.tso_tx: 326797
dev.ix.0.queue5.no_tx_dma_setup: 0
dev.ix.0.queue5.no_desc_avail: 0
dev.ix.0.queue5.tx_packets: 531092
dev.ix.0.queue5.rxd_head: 57
dev.ix.0.queue5.rxd_tail: 56
dev.ix.0.queue5.rx_packets: 796729
dev.ix.0.queue5.rx_bytes: 108295068
dev.ix.0.queue5.rx_copies: 582757
dev.ix.0.queue5.lro_queued: 795369
dev.ix.0.queue5.lro_flushed: 258290
dev.ix.0.queue6.interrupt_rate: 100000
dev.ix.0.queue6.irqs: 26775
dev.ix.0.queue6.txd_head: 1146
dev.ix.0.queue6.txd_tail: 1146
dev.ix.0.queue6.tso_tx: 13
dev.ix.0.queue6.no_tx_dma_setup: 0
dev.ix.0.queue6.no_desc_avail: 0
dev.ix.0.queue6.tx_packets: 5469
dev.ix.0.queue6.rxd_head: 1077
dev.ix.0.queue6.rxd_tail: 1076
dev.ix.0.queue6.rx_packets: 9269
dev.ix.0.queue6.rx_bytes: 6631479
dev.ix.0.queue6.rx_copies: 4878
dev.ix.0.queue6.lro_queued: 8054
dev.ix.0.queue6.lro_flushed: 4260
dev.ix.0.queue7.interrupt_rate: 55555
dev.ix.0.queue7.irqs: 243399
dev.ix.0.queue7.txd_head: 66
dev.ix.0.queue7.txd_tail: 66
dev.ix.0.queue7.tso_tx: 5
dev.ix.0.queue7.no_tx_dma_setup: 0
dev.ix.0.queue7.no_desc_avail: 0
dev.ix.0.queue7.tx_packets: 121101
dev.ix.0.queue7.rxd_head: 130
dev.ix.0.queue7.rxd_tail: 129
dev.ix.0.queue7.rx_packets: 127106
dev.ix.0.queue7.rx_bytes: 15197119
dev.ix.0.queue7.rx_copies: 118192
dev.ix.0.queue7.lro_queued: 125622
dev.ix.0.queue7.lro_flushed: 125138
dev.ix.0.mac_stats.crc_errs: 0
dev.ix.0.mac_stats.ill_errs: 0
dev.ix.0.mac_stats.byte_errs: 0
dev.ix.0.mac_stats.short_discards: 0
dev.ix.0.mac_stats.local_faults: 7              <=============== HERE
dev.ix.0.mac_stats.remote_faults: 1
dev.ix.0.mac_stats.rec_len_errs: 0
dev.ix.0.mac_stats.xon_txd: 0
dev.ix.0.mac_stats.xon_recvd: 0
dev.ix.0.mac_stats.xoff_txd: 0
dev.ix.0.mac_stats.xoff_recvd: 0
dev.ix.0.mac_stats.total_octets_rcvd: 11249450018
dev.ix.0.mac_stats.good_octets_rcvd: 11249396646
dev.ix.0.mac_stats.total_pkts_rcvd: 8804445
dev.ix.0.mac_stats.good_pkts_rcvd: 8803850
dev.ix.0.mac_stats.mcast_pkts_rcvd: 9311
dev.ix.0.mac_stats.bcast_pkts_rcvd: 1908
dev.ix.0.mac_stats.rx_frames_64: 18132
dev.ix.0.mac_stats.rx_frames_65_127: 759186
dev.ix.0.mac_stats.rx_frames_128_255: 116641
dev.ix.0.mac_stats.rx_frames_256_511: 686728
dev.ix.0.mac_stats.rx_frames_512_1023: 67041
dev.ix.0.mac_stats.rx_frames_1024_1522: 7156122
dev.ix.0.mac_stats.recv_undersized: 0
dev.ix.0.mac_stats.recv_fragmented: 0
dev.ix.0.mac_stats.recv_oversized: 0
dev.ix.0.mac_stats.recv_jabberd: 0
dev.ix.0.mac_stats.management_pkts_rcvd: 11219
dev.ix.0.mac_stats.management_pkts_drpd: 0
dev.ix.0.mac_stats.checksum_errs: 0
dev.ix.0.mac_stats.good_octets_txd: 20162287794
dev.ix.0.mac_stats.total_pkts_txd: 14419225
dev.ix.0.mac_stats.good_pkts_txd: 14419225
dev.ix.0.mac_stats.bcast_pkts_txd: 621
dev.ix.0.mac_stats.mcast_pkts_txd: 0
dev.ix.0.mac_stats.management_pkts_txd: 0
dev.ix.0.mac_stats.tx_frames_64: 12833
dev.ix.0.mac_stats.tx_frames_65_127: 549847
dev.ix.0.mac_stats.tx_frames_128_255: 80184
dev.ix.0.mac_stats.tx_frames_256_511: 631975
dev.ix.0.mac_stats.tx_frames_512_1023: 116264
dev.ix.0.mac_stats.tx_frames_1024_1522: 13028122

Does any one know what does local_faults/remot_faults mean here? Does this
means there is a hardware error? (We tried to find the adaptor manual, but
there is no detail on IXGBE_MLFC [0x04034] register)

Any suggestion on how to diagnose this problem is welcomed too.

Thanks in advance!
Peter



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?53C77FEE.9000707>