Date: Tue, 5 Jul 2011 18:41:47 -0400 (EDT) From: Charles Sprickman <spork@bway.net> To: freebsd-net@freebsd.org Subject: Re: bce packet loss Message-ID: <alpine.OSX.2.00.1107051758160.2407@freemac> In-Reply-To: <alpine.OSX.2.00.1107042113000.2407@freemac> References: <alpine.OSX.2.00.1107042113000.2407@freemac>
next in thread | previous in thread | raw e-mail | index | archive | help
Just adding a bit more information inline: On Mon, 4 Jul 2011, Charles Sprickman wrote: > Hello, > > We're running a few 8.1-R servers with Broadcom bce interfaces (Dell R510) > and I'm seeing occasional packet loss on them (enough that it trips nagios > now and then). Cabling seems fine as neither the switch nor the sysctl info > for the device show any errors/collisions/etc, however there is one odd one, > which is "dev.bce.1.stat_IfHCInBadOctets: 539369". See [1] below for full > sysctl output. The switch shows no errors but for "Dropped packets 683868". > > pciconf output is also below. [2] > > By default, the switch had flow control set to "on". I also let it run with > "auto". In both cases, the drops continued to increment. I'm now running > with flow control off to see if that changes anything. With flow control set to "off" on the switch I am still seeing dropped packets. The number shown by the switch after resetting the counters tracks "dev.bce.1.stat_IfHCInBadOctets" very closely. Neither the switch nor the host (in netstat) are reporting any errors, collisions, etc. I'm seeing less correlation between cpu utilization and drops, which is unfortunate. In fact, looking at all the graphs I have, such as disk activity, system load, traffic on the interface and a handful of others, I'm seeing no strong correlation in the last 24 hours or so between any system activity and the drops. Google isn't turn up much more than source code hits on "stat_IfHCInBadOctets", and from what I can tell, this is something that's just reading something off the card, so I'm having little luck figuring out just what a "bad octet" is in this context. Thanks, Charles > I do see some correlation between cpu usage and drops - I have cpu usage > graphed in nagios and cacti is graphing the drops on the dell switch. There's > no signs of running out of mbufs or similar. > > So given that limited info, is there anything I can look at to track this > down? Anything stand out in the stats sysctl exposes? Two things are > standing out for me - the number of changes in bce regarding flow control > that are not in 8.1, and the correlation between cpu load and the drops. > > What other information can I provide? > > Thanks, > > Charles > > [1] [root@h23 /home/spork]# sysctl -a |grep bce.1 > dev.bce.1.%desc: Broadcom NetXtreme II BCM5716 1000Base-T (C0) > dev.bce.1.%driver: bce > dev.bce.1.%location: slot=0 function=1 > dev.bce.1.%pnpinfo: vendor=0x14e4 device=0x163b subvendor=0x1028 > subdevice=0x02f1 class=0x020000 > dev.bce.1.%parent: pci1 > dev.bce.1.l2fhdr_error_count: 0 > dev.bce.1.mbuf_alloc_failed_count: 282 > dev.bce.1.fragmented_mbuf_count: 2748 > dev.bce.1.dma_map_addr_rx_failed_count: 0 > dev.bce.1.dma_map_addr_tx_failed_count: 5 > dev.bce.1.unexpected_attention_count: 0 > dev.bce.1.stat_IfHcInOctets: 62708651108 > dev.bce.1.stat_IfHCInBadOctets: 539369 > dev.bce.1.stat_IfHCOutOctets: 434264587173 > dev.bce.1.stat_IfHCOutBadOctets: 0 > dev.bce.1.stat_IfHCInUcastPkts: 533441918 > dev.bce.1.stat_IfHCInMulticastPkts: 3108746 > dev.bce.1.stat_IfHCInBroadcastPkts: 1314905 > dev.bce.1.stat_IfHCOutUcastPkts: 640961970 > dev.bce.1.stat_IfHCOutMulticastPkts: 26 > dev.bce.1.stat_IfHCOutBroadcastPkts: 8909 > dev.bce.1.stat_emac_tx_stat_dot3statsinternalmactransmiterrors: 0 > dev.bce.1.stat_Dot3StatsCarrierSenseErrors: 0 > dev.bce.1.stat_Dot3StatsFCSErrors: 0 > dev.bce.1.stat_Dot3StatsAlignmentErrors: 0 > dev.bce.1.stat_Dot3StatsSingleCollisionFrames: 0 > dev.bce.1.stat_Dot3StatsMultipleCollisionFrames: 0 > dev.bce.1.stat_Dot3StatsDeferredTransmissions: 0 > dev.bce.1.stat_Dot3StatsExcessiveCollisions: 0 > dev.bce.1.stat_Dot3StatsLateCollisions: 0 > dev.bce.1.stat_EtherStatsCollisions: 0 > dev.bce.1.stat_EtherStatsFragments: 0 > dev.bce.1.stat_EtherStatsJabbers: 0 > dev.bce.1.stat_EtherStatsUndersizePkts: 0 > dev.bce.1.stat_EtherStatsOversizePkts: 0 > dev.bce.1.stat_EtherStatsPktsRx64Octets: 34048797 > dev.bce.1.stat_EtherStatsPktsRx65Octetsto127Octets: 431844366 > dev.bce.1.stat_EtherStatsPktsRx128Octetsto255Octets: 25946173 > dev.bce.1.stat_EtherStatsPktsRx256Octetsto511Octets: 39936369 > dev.bce.1.stat_EtherStatsPktsRx512Octetsto1023Octets: 2296565 > dev.bce.1.stat_EtherStatsPktsRx1024Octetsto1522Octets: 3931392 > dev.bce.1.stat_EtherStatsPktsRx1523Octetsto9022Octets: 0 > dev.bce.1.stat_EtherStatsPktsTx64Octets: 60122571 > dev.bce.1.stat_EtherStatsPktsTx65Octetsto127Octets: 221041349 > dev.bce.1.stat_EtherStatsPktsTx128Octetsto255Octets: 40177071 > dev.bce.1.stat_EtherStatsPktsTx256Octetsto511Octets: 24099944 > dev.bce.1.stat_EtherStatsPktsTx512Octetsto1023Octets: 44493532 > dev.bce.1.stat_EtherStatsPktsTx1024Octetsto1522Octets: 251036438 > dev.bce.1.stat_EtherStatsPktsTx1523Octetsto9022Octets: 0 > dev.bce.1.stat_XonPauseFramesReceived: 61778 > dev.bce.1.stat_XoffPauseFramesReceived: 76315 > dev.bce.1.stat_OutXonSent: 0 > dev.bce.1.stat_OutXoffSent: 0 > dev.bce.1.stat_FlowControlDone: 0 > dev.bce.1.stat_MacControlFramesReceived: 0 > dev.bce.1.stat_XoffStateEntered: 0 > dev.bce.1.stat_IfInFramesL2FilterDiscards: 145832 > dev.bce.1.stat_IfInRuleCheckerDiscards: 0 > dev.bce.1.stat_IfInFTQDiscards: 0 > dev.bce.1.stat_IfInMBUFDiscards: 0 > dev.bce.1.stat_IfInRuleCheckerP4Hit: 4448215 > dev.bce.1.stat_CatchupInRuleCheckerDiscards: 0 > dev.bce.1.stat_CatchupInFTQDiscards: 0 > dev.bce.1.stat_CatchupInMBUFDiscards: 0 > dev.bce.1.stat_CatchupInRuleCheckerP4Hit: 0 > dev.bce.1.com_no_buffers: 0 > > [2] pciconf -lvb > bce1@pci0:1:0:1: class=0x020000 card=0x02f11028 chip=0x163b14e4 > rev=0x20 hdr=0x00 > vendor = 'Broadcom Corporation' > class = network > subclass = ethernet > bar [10] = type Memory, range 64, base 0xdc000000, size 33554432, > enabled >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?alpine.OSX.2.00.1107051758160.2407>