Date: Sun, 19 Sep 2010 08:27:52 -0400 From: Mike Tancsa <mike@sentex.net> To: freebsd-stable@freebsd.org Subject: em nic getting wedged under high load Message-ID: <201009191228.o8JCRxWj052684@lava.sentex.ca>
next in thread | raw e-mail | index | archive | help
At first we thought it might be the NIC, but we have since changed
out the nic, motherboard and power supply. Under high loads, we are
seeing the nic become wedged to the point where we have to reboot the
box to unwedge it
This is an intel board (S3420GPX) from Intel with 8G of RAM, and i5,
amd64 running RELENG_8. We first noticed it when we upgraded to an
i7 board. Thinking it was hardware, we changed it out to the i5 and
are still seeing the nic wedging under high nfs load.
em0@pci0:0:25:0: class=0x020000 card=0x34ec8086
chip=0x10ef8086 rev=0x05 hdr=0x00
vendor = 'Intel Corporation'
class = network
subclass = ethernet
cap 01[c8] = powerspec 2 supports D0 D3 current D0
cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message
cap 13[e0] = PCI Advanced Features: FLR TP
em1@pci0:9:0:0: class=0x020000 card=0x34ec8086 chip=0x10d38086
rev=0x00 hdr=0x00
vendor = 'Intel Corporation'
device = 'Intel 82574L Gigabit Ethernet Controller (82574L)'
class = network
subclass = ethernet
cap 01[c8] = powerspec 2 supports D0 D3 current D0
cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message
cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1)
cap 11[a0] = MSI-X supports 5 messages in map 0x1c
vmstat -i
interrupt total rate
irq16: siis0 510107 228
irq18: arcmsr0 479177 214
irq19: twa0 16297 7
irq21: ehci0 3390 1
irq23: ehci1 4628 2
cpu0: timer 4469679 1999
irq256: em0 1183006 529
irq257: em1 6120960 2738
irq258: ahci0 507884 227
cpu1: timer 4461694 1996
cpu2: timer 4461618 1996
cpu3: timer 4461832 1996
Total 26680272 11937
In its wedged state, the nic's stats show
dev.em.1.%desc: Intel(R) PRO/1000 Network Connection 7.0.5
dev.em.1.%driver: em
dev.em.1.%location: slot=0 function=0 handle=\_SB_.PCI0.PEX4.HART
dev.em.1.%pnpinfo: vendor=0x8086 device=0x10d3 subvendor=0x8086
subdevice=0x34ec class=0x020000
dev.em.1.%parent: pci9
dev.em.1.nvm: -1
dev.em.1.rx_int_delay: 0
dev.em.1.tx_int_delay: 66
dev.em.1.rx_abs_int_delay: 66
dev.em.1.tx_abs_int_delay: 66
dev.em.1.rx_processing_limit: 100
dev.em.1.link_irq: 0
dev.em.1.mbuf_alloc_fail: 0
dev.em.1.cluster_alloc_fail: 0
dev.em.1.dropped: 0
dev.em.1.tx_dma_fail: 0
dev.em.1.fc_high_water: 18432
dev.em.1.fc_low_water: 16932
dev.em.1.mac_stats.excess_coll: 0
dev.em.1.mac_stats.symbol_errors: 0
dev.em.1.mac_stats.sequence_errors: 0
dev.em.1.mac_stats.defer_count: 0
dev.em.1.mac_stats.missed_packets: 41522
dev.em.1.mac_stats.recv_no_buff: 19
dev.em.1.mac_stats.recv_errs: 0
dev.em.1.mac_stats.crc_errs: 0
dev.em.1.mac_stats.alignment_errs: 0
dev.em.1.mac_stats.coll_ext_errs: 0
dev.em.1.mac_stats.rx_overruns: 41398
dev.em.1.mac_stats.watchdog_timeouts: 0
dev.em.1.mac_stats.xon_recvd: 0
dev.em.1.mac_stats.xon_txd: 0
dev.em.1.mac_stats.xoff_recvd: 0
dev.em.1.mac_stats.xoff_txd: 0
dev.em.1.mac_stats.total_pkts_recvd: 95229129
dev.em.1.mac_stats.good_pkts_recvd: 95187607
dev.em.1.mac_stats.bcast_pkts_recvd: 79244
dev.em.1.mac_stats.mcast_pkts_recvd: 0
dev.em.1.mac_stats.rx_frames_64: 93680
dev.em.1.mac_stats.rx_frames_65_127: 1516349
dev.em.1.mac_stats.rx_frames_128_255: 4464941
dev.em.1.mac_stats.rx_frames_256_511: 4024
dev.em.1.mac_stats.rx_frames_512_1023: 2096067
dev.em.1.mac_stats.rx_frames_1024_1522: 87012546
dev.em.1.mac_stats.good_octets_recvd: 0
dev.em.1.mac_stats.good_octest_txd: 0
dev.em.1.mac_stats.total_pkts_txd: 66775098
dev.em.1.mac_stats.good_pkts_txd: 66775098
dev.em.1.mac_stats.bcast_pkts_txd: 509
dev.em.1.mac_stats.mcast_pkts_txd: 7
dev.em.1.mac_stats.tx_frames_64: 48038472
dev.em.1.mac_stats.tx_frames_65_127: 13402833
dev.em.1.mac_stats.tx_frames_128_255: 5324413
dev.em.1.mac_stats.tx_frames_256_511: 957
dev.em.1.mac_stats.tx_frames_512_1023: 319
dev.em.1.mac_stats.tx_frames_1024_1522: 8104
dev.em.1.mac_stats.tso_txd: 1069
dev.em.1.mac_stats.tso_ctx_fail: 0
dev.em.1.interrupts.asserts: 0
dev.em.1.interrupts.rx_pkt_timer: 0
dev.em.1.interrupts.rx_abs_timer: 0
dev.em.1.interrupts.tx_pkt_timer: 0
dev.em.1.interrupts.tx_abs_timer: 0
dev.em.1.interrupts.tx_queue_empty: 0
dev.em.1.interrupts.tx_queue_min_thresh: 0
dev.em.1.interrupts.rx_desc_min_thresh: 0
dev.em.1.interrupts.rx_overrun: 0
dev.em.1.host.breaker_tx_pkt: 0
dev.em.1.host.host_tx_pkt_discard: 0
dev.em.1.host.rx_pkt: 0
dev.em.1.host.breaker_rx_pkts: 0
dev.em.1.host.breaker_rx_pkt_drop: 0
dev.em.1.host.tx_good_pkt: 0
dev.em.1.host.breaker_tx_pkt_drop: 0
dev.em.1.host.rx_good_bytes: 0
dev.em.1.host.tx_good_bytes: 0
dev.em.1.host.length_errors: 0
dev.em.1.host.serdes_violation_pkt: 0
dev.em.1.host.header_redir_missed: 0
--------------------------------------------------------------------
Mike Tancsa, tel +1 519 651 3400
Sentex Communications, mike@sentex.net
Providing Internet since 1994 www.sentex.net
Cambridge, Ontario Canada www.sentex.net/mike
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201009191228.o8JCRxWj052684>