Date: Mon, 4 Jul 2011 21:32:11 -0400 (EDT) From: Charles Sprickman <spork@bway.net> To: freebsd-net@freebsd.org Subject: bce packet loss Message-ID: <alpine.OSX.2.00.1107042113000.2407@freemac>
next in thread | raw e-mail | index | archive | help
Hello,
We're running a few 8.1-R servers with Broadcom bce interfaces (Dell R510)
and I'm seeing occasional packet loss on them (enough that it trips nagios
now and then). Cabling seems fine as neither the switch nor the sysctl
info for the device show any errors/collisions/etc, however there is one
odd one, which is "dev.bce.1.stat_IfHCInBadOctets: 539369". See [1] below
for full sysctl output. The switch shows no errors but for "Dropped
packets 683868".
pciconf output is also below. [2]
By default, the switch had flow control set to "on". I also let it run
with "auto". In both cases, the drops continued to increment. I'm now
running with flow control off to see if that changes anything.
I do see some correlation between cpu usage and drops - I have cpu usage
graphed in nagios and cacti is graphing the drops on the dell switch.
There's no signs of running out of mbufs or similar.
So given that limited info, is there anything I can look at to track this
down? Anything stand out in the stats sysctl exposes? Two things are
standing out for me - the number of changes in bce regarding flow control
that are not in 8.1, and the correlation between cpu load and the drops.
What other information can I provide?
Thanks,
Charles
[1] [root@h23 /home/spork]# sysctl -a |grep bce.1
dev.bce.1.%desc: Broadcom NetXtreme II BCM5716 1000Base-T (C0)
dev.bce.1.%driver: bce
dev.bce.1.%location: slot=0 function=1
dev.bce.1.%pnpinfo: vendor=0x14e4 device=0x163b subvendor=0x1028
subdevice=0x02f1 class=0x020000
dev.bce.1.%parent: pci1
dev.bce.1.l2fhdr_error_count: 0
dev.bce.1.mbuf_alloc_failed_count: 282
dev.bce.1.fragmented_mbuf_count: 2748
dev.bce.1.dma_map_addr_rx_failed_count: 0
dev.bce.1.dma_map_addr_tx_failed_count: 5
dev.bce.1.unexpected_attention_count: 0
dev.bce.1.stat_IfHcInOctets: 62708651108
dev.bce.1.stat_IfHCInBadOctets: 539369
dev.bce.1.stat_IfHCOutOctets: 434264587173
dev.bce.1.stat_IfHCOutBadOctets: 0
dev.bce.1.stat_IfHCInUcastPkts: 533441918
dev.bce.1.stat_IfHCInMulticastPkts: 3108746
dev.bce.1.stat_IfHCInBroadcastPkts: 1314905
dev.bce.1.stat_IfHCOutUcastPkts: 640961970
dev.bce.1.stat_IfHCOutMulticastPkts: 26
dev.bce.1.stat_IfHCOutBroadcastPkts: 8909
dev.bce.1.stat_emac_tx_stat_dot3statsinternalmactransmiterrors: 0
dev.bce.1.stat_Dot3StatsCarrierSenseErrors: 0
dev.bce.1.stat_Dot3StatsFCSErrors: 0
dev.bce.1.stat_Dot3StatsAlignmentErrors: 0
dev.bce.1.stat_Dot3StatsSingleCollisionFrames: 0
dev.bce.1.stat_Dot3StatsMultipleCollisionFrames: 0
dev.bce.1.stat_Dot3StatsDeferredTransmissions: 0
dev.bce.1.stat_Dot3StatsExcessiveCollisions: 0
dev.bce.1.stat_Dot3StatsLateCollisions: 0
dev.bce.1.stat_EtherStatsCollisions: 0
dev.bce.1.stat_EtherStatsFragments: 0
dev.bce.1.stat_EtherStatsJabbers: 0
dev.bce.1.stat_EtherStatsUndersizePkts: 0
dev.bce.1.stat_EtherStatsOversizePkts: 0
dev.bce.1.stat_EtherStatsPktsRx64Octets: 34048797
dev.bce.1.stat_EtherStatsPktsRx65Octetsto127Octets: 431844366
dev.bce.1.stat_EtherStatsPktsRx128Octetsto255Octets: 25946173
dev.bce.1.stat_EtherStatsPktsRx256Octetsto511Octets: 39936369
dev.bce.1.stat_EtherStatsPktsRx512Octetsto1023Octets: 2296565
dev.bce.1.stat_EtherStatsPktsRx1024Octetsto1522Octets: 3931392
dev.bce.1.stat_EtherStatsPktsRx1523Octetsto9022Octets: 0
dev.bce.1.stat_EtherStatsPktsTx64Octets: 60122571
dev.bce.1.stat_EtherStatsPktsTx65Octetsto127Octets: 221041349
dev.bce.1.stat_EtherStatsPktsTx128Octetsto255Octets: 40177071
dev.bce.1.stat_EtherStatsPktsTx256Octetsto511Octets: 24099944
dev.bce.1.stat_EtherStatsPktsTx512Octetsto1023Octets: 44493532
dev.bce.1.stat_EtherStatsPktsTx1024Octetsto1522Octets: 251036438
dev.bce.1.stat_EtherStatsPktsTx1523Octetsto9022Octets: 0
dev.bce.1.stat_XonPauseFramesReceived: 61778
dev.bce.1.stat_XoffPauseFramesReceived: 76315
dev.bce.1.stat_OutXonSent: 0
dev.bce.1.stat_OutXoffSent: 0
dev.bce.1.stat_FlowControlDone: 0
dev.bce.1.stat_MacControlFramesReceived: 0
dev.bce.1.stat_XoffStateEntered: 0
dev.bce.1.stat_IfInFramesL2FilterDiscards: 145832
dev.bce.1.stat_IfInRuleCheckerDiscards: 0
dev.bce.1.stat_IfInFTQDiscards: 0
dev.bce.1.stat_IfInMBUFDiscards: 0
dev.bce.1.stat_IfInRuleCheckerP4Hit: 4448215
dev.bce.1.stat_CatchupInRuleCheckerDiscards: 0
dev.bce.1.stat_CatchupInFTQDiscards: 0
dev.bce.1.stat_CatchupInMBUFDiscards: 0
dev.bce.1.stat_CatchupInRuleCheckerP4Hit: 0
dev.bce.1.com_no_buffers: 0
[2] pciconf -lvb
bce1@pci0:1:0:1: class=0x020000 card=0x02f11028 chip=0x163b14e4
rev=0x20 hdr=0x00
vendor = 'Broadcom Corporation'
class = network
subclass = ethernet
bar [10] = type Memory, range 64, base 0xdc000000, size 33554432,
enabled
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?alpine.OSX.2.00.1107042113000.2407>