Date: Mon, 08 Sep 2014 22:14:36 -0300 From: Marcelo Gondim <gondim@bsdinfo.com.br> To: =?UTF-8?B?IkluZy4gQWxlxaEgTmVjaHbDoXRhbCI=?= <alesnechvatal@videon-znojmo.cz>, 'Eric Joyner' <ricera10@gmail.com> Cc: 'Jack F Vogel' <jfv@freebsd.org>, 'FreeBSD Net' <freebsd-net@freebsd.org>, 'Adrian Chadd' <adrian@freebsd.org> Subject: Re: ixgbe CRITICAL: ECC ERROR!! Please Reboot!! Message-ID: <540E547C.9030703@bsdinfo.com.br> In-Reply-To: <002001cfcba0$4812c7f0$d83857d0$@videon-znojmo.cz> References: <5408F23C.2030309@bsdinfo.com.br> <CAJ-VmomOECsnRt98ULijSkj9JmwnaLEnZ1ny7fqazsM3WJDNvA@mail.gmail.com> <54091607.9010100@bsdinfo.com.br> <5409CA44.8070203@bsdinfo.com.br> <CAJ-VmomnzTfHzw2i0C01kmfg1rKLePcTHhZqbPWsAezg%2Be9q0g@mail.gmail.com> <540A1A3E.2040306@bsdinfo.com.br> <540A8200.5010404@bsdinfo.com.br> <CA%2Bb0zg9%2Bhhz6sCzrTAHjhc5VzTbDWdnQ%2BBcfdMPsyTGsPeo3gQ@mail.gmail.com> <540DFAE3.8070001@bsdinfo.com.br> <002001cfcba0$4812c7f0$d83857d0$@videon-znojmo.cz>
next in thread | previous in thread | raw e-mail | index | archive | help
On 08/09/2014 17:05, Ing. Aleš Nechvátal wrote: > I was experiencing similar behaviour on a machine acting as a gateway. It > has one X520-SR2, both SPF+ ports used to connet to another machines with > the same NICs. One of the NICs suddenly stopped to pass data through and the > the ifconfing down up cycle was needed to make it work again, that in better > case. In the worse case, the machine rebooted without leaving any message in > syslog (it's on a remote site). This happened in intervals of 5 mins. up to > 1 hour (I am not sure about the exact correlation between traffic going > through and the number of these events, but during traffic peaks, it seems > to have happened more often). > > First I checked the mbufs, but didn't see any problem. Then I tried to > disable TSO (on both sides of the links), but without observing any > improvement. > I was playing with different versions of the kernel, the newer the kernel (I > went up to 9.3 STABLE, didn't try 10 or HEAD) the more instability. It > seemed, that the most stable was 9.1 PRERELEASE with 2.4.8 driver (at least > I didn't see reboots, a cron script could keep the gateway passing data by > shutting down and bringing up the interfaces). Now I am running 9.3 RELEASE > r271227M with the 2.5.25 driver from Intel's website. This configuration was > not stable either, until I discovered high interrupt rate on each queue (the > sum on all queues reached over 400k in some moments). I turned off the > limit of interrupts by setting hw.intr_storm_threshold=0 (previously set to > 150000). Since the following reboot (involuntary, btw.) the gateway keeps > running smoothly for 7 hours now with peak throughput of approx. 1.6 Gbit/s. > > The sum of interrupt rates on all your queues looks quite high too, so there > could be some similarity and disabling the mentioned limit could help. > > My sysctl.conf: > kern.ipc.somaxconn=1024 > net.inet.icmp.icmplim=100 > net.inet.tcp.blackhole=2 > net.inet.udp.blackhole=1 > net.inet.tcp.sendspace=262144 > net.inet.tcp.recvspace=262144 > net.inet.ip.fastforwarding=1 > kern.ipc.maxsockbuf=16777216 > net.inet.tcp.sendbuf_max=16777216 > net.inet.tcp.recvbuf_max=16777216 > hw.intr_storm_threshold=0 > > #sysctl -a dev.ix.1 I changed the value of "hw.intr_storm_threshold" to 0 and yet the interface stopped working. :( my sysctl.conf: net.inet.ip.forwarding=1 net.inet.ip.fastforwarding=1 net.inet6.ip6.forwarding=1 kern.ipc.somaxconn=4096 net.inet.tcp.syncookies=1 net.inet.ip.redirect=1 net.inet.ip.accept_sourceroute=0 net.inet.ip.sourceroute=0 net.inet.tcp.drop_synfin=1 net.inet.udp.blackhole=1 net.inet.tcp.blackhole=2 security.bsd.see_other_uids=0 net.inet.ip.fw.dyn_buckets=65536 net.inet.ip.fw.dyn_max=65536 hw.intr_storm_threshold=0 net.inet.ip.dummynet.pipe_slot_limit=800 net.inet.icmp.icmplim=2000 Cheers, > > dev.ix.1.%desc: Intel(R) PRO/10GbE PCI-Express Network Driver, Version - > 2.5.25 > dev.ix.1.%driver: ix > dev.ix.1.%location: slot=0 function=1 > dev.ix.1.%pnpinfo: vendor=0x8086 device=0x10fb subvendor=0x8086 > subdevice=0x7a11 class=0x020000 > dev.ix.1.%parent: pci1 > dev.ix.1.fc: 3 > dev.ix.1.enable_aim: 1 > dev.ix.1.advertise_speed: 0 > dev.ix.1.dropped: 0 > dev.ix.1.mbuf_defrag_failed: 0 > dev.ix.1.watchdog_events: 0 > dev.ix.1.link_irq: 2 > dev.ix.1.queue0.interrupt_rate: 31250 > dev.ix.1.queue0.irqs: 152182132 > dev.ix.1.queue0.txd_head: 526 > dev.ix.1.queue0.txd_tail: 526 > dev.ix.1.queue0.tso_tx: 2 > dev.ix.1.queue0.no_tx_dma_setup: 0 > dev.ix.1.queue0.no_desc_avail: 119373 > dev.ix.1.queue0.tx_packets: 274418209 > dev.ix.1.queue0.rxd_head: 441 > dev.ix.1.queue0.rxd_tail: 440 > dev.ix.1.queue0.rx_packets: 103852222 > dev.ix.1.queue0.rx_bytes: 25483398737 > dev.ix.1.queue0.rx_copies: 87237192 > dev.ix.1.queue0.lro_queued: 0 > dev.ix.1.queue0.lro_flushed: 0 > dev.ix.1.queue1.interrupt_rate: 31250 > dev.ix.1.queue1.irqs: 152973172 > dev.ix.1.queue1.txd_head: 1282 > dev.ix.1.queue1.txd_tail: 1282 > dev.ix.1.queue1.tso_tx: 50 > dev.ix.1.queue1.no_tx_dma_setup: 0 > dev.ix.1.queue1.no_desc_avail: 114840 > dev.ix.1.queue1.tx_packets: 271827311 > dev.ix.1.queue1.rxd_head: 495 > dev.ix.1.queue1.rxd_tail: 494 > dev.ix.1.queue1.rx_packets: 103292003 > dev.ix.1.queue1.rx_bytes: 20681070454 > dev.ix.1.queue1.rx_copies: 89490795 > dev.ix.1.queue1.lro_queued: 0 > dev.ix.1.queue1.lro_flushed: 0 > dev.ix.1.queue2.interrupt_rate: 21739 > dev.ix.1.queue2.irqs: 81502157 > dev.ix.1.queue2.txd_head: 313 > dev.ix.1.queue2.txd_tail: 313 > dev.ix.1.queue2.tso_tx: 50 > dev.ix.1.queue2.no_tx_dma_setup: 0 > dev.ix.1.queue2.no_desc_avail: 111923 > dev.ix.1.queue2.tx_packets: 23189156 > dev.ix.1.queue2.rxd_head: 349 > dev.ix.1.queue2.rxd_tail: 348 > dev.ix.1.queue2.rx_packets: 94457797 > dev.ix.1.queue2.rx_bytes: 18053240855 > dev.ix.1.queue2.rx_copies: 82430878 > dev.ix.1.queue2.lro_queued: 0 > dev.ix.1.queue2.lro_flushed: 0 > dev.ix.1.queue3.interrupt_rate: 83333 > dev.ix.1.queue3.irqs: 81678068 > dev.ix.1.queue3.txd_head: 1990 > dev.ix.1.queue3.txd_tail: 1990 > dev.ix.1.queue3.tso_tx: 3 > dev.ix.1.queue3.no_tx_dma_setup: 0 > dev.ix.1.queue3.no_desc_avail: 110545 > dev.ix.1.queue3.tx_packets: 21258249 > dev.ix.1.queue3.rxd_head: 1544 > dev.ix.1.queue3.rxd_tail: 1543 > dev.ix.1.queue3.rx_packets: 99140619 > dev.ix.1.queue3.rx_bytes: 19476001717 > dev.ix.1.queue3.rx_copies: 86228489 > dev.ix.1.queue3.lro_queued: 0 > dev.ix.1.queue3.lro_flushed: 0 > dev.ix.1.mac_stats.crc_errs: 0 > dev.ix.1.mac_stats.ill_errs: 0 > dev.ix.1.mac_stats.byte_errs: 0 > dev.ix.1.mac_stats.short_discards: 0 > dev.ix.1.mac_stats.local_faults: 0 > dev.ix.1.mac_stats.remote_faults: 2 > dev.ix.1.mac_stats.rec_len_errs: 0 > dev.ix.1.mac_stats.xon_txd: 0 > dev.ix.1.mac_stats.xon_recvd: 0 > dev.ix.1.mac_stats.xoff_txd: 0 > dev.ix.1.mac_stats.xoff_recvd: 0 > dev.ix.1.mac_stats.total_octets_rcvd: 85345731961 > dev.ix.1.mac_stats.good_octets_rcvd: 85345731961 > dev.ix.1.mac_stats.total_pkts_rcvd: 400734734 > dev.ix.1.mac_stats.good_pkts_rcvd: 400734734 > dev.ix.1.mac_stats.mcast_pkts_rcvd: 0 > dev.ix.1.mac_stats.bcast_pkts_rcvd: 0 > dev.ix.1.mac_stats.rx_frames_64: 190985269 > dev.ix.1.mac_stats.rx_frames_65_127: 142879679 > dev.ix.1.mac_stats.rx_frames_128_255: 18114498 > dev.ix.1.mac_stats.rx_frames_256_511: 5882972 > dev.ix.1.mac_stats.rx_frames_512_1023: 7407080 > dev.ix.1.mac_stats.rx_frames_1024_1522: 35465236 > dev.ix.1.mac_stats.recv_undersized: 0 > dev.ix.1.mac_stats.recv_fragmented: 0 > dev.ix.1.mac_stats.recv_oversized: 0 > dev.ix.1.mac_stats.recv_jabberd: 0 > dev.ix.1.mac_stats.management_pkts_rcvd: 0 > dev.ix.1.mac_stats.management_pkts_drpd: 0 > dev.ix.1.mac_stats.checksum_errs: 462621 > dev.ix.1.mac_stats.good_octets_txd: 705061464268 > dev.ix.1.mac_stats.total_pkts_txd: 590658400 > dev.ix.1.mac_stats.good_pkts_txd: 590658400 > dev.ix.1.mac_stats.bcast_pkts_txd: 2907 > dev.ix.1.mac_stats.mcast_pkts_txd: 0 > dev.ix.1.mac_stats.management_pkts_txd: 0 > dev.ix.1.mac_stats.tx_frames_64: 24533923 > dev.ix.1.mac_stats.tx_frames_65_127: 65221733 > dev.ix.1.mac_stats.tx_frames_128_255: 24274012 > dev.ix.1.mac_stats.tx_frames_256_511: 12092320 > dev.ix.1.mac_stats.tx_frames_512_1023: 10358523 > dev.ix.1.mac_stats.tx_frames_1024_1522: 454177889 > >> -----Original Message----- >> From: owner-freebsd-net@freebsd.org [mailto:owner-freebsd- >> net@freebsd.org] On Behalf Of Marcelo Gondim >> Sent: Monday, September 08, 2014 8:52 PM >> To: Eric Joyner >> Cc: Jack F Vogel; FreeBSD Net; Adrian Chadd >> Subject: Re: ixgbe CRITICAL: ECC ERROR!! Please Reboot!! >> >> On 08/09/2014 15:38, Eric Joyner wrote: >>> Getting local / remote faults is strange; are these stats from after you >>> rebooted? >> When the crash happens I just do: >> >> # ifconfig ix0 down; ifconfig ix0 up; ifconfig ix1 down; ifconfig up ixi1 >> >> After that the two interface ports return to work. >> >> In no time I had to restart the server. >> >> # uptime >> 3:51PM up 11 days, 15:44, 2 users, load averages: 5.36, 5.68, 5.69 >> >> Tomorrow will be going to the Datacenter to check on the temperature of >> the network interface. As the crashes occur when traffic is high, I >> believe that the temperature increases because processing is high. >> >> >>> --- >>> - Eric Joyner >>> >>> On Fri, Sep 5, 2014 at 8:39 PM, Marcelo Gondim <gondim@bsdinfo.com.br> >>> wrote: >>> >>>> On 05/09/2014 17:17, Marcelo Gondim wrote: >>>> >>>>> On 05/09/2014 16:49, Adrian Chadd wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> But is the airflow in the unit sufficient? >>>>>> >>>>>> I had this problem at a previous job - the box was running fine, the >>>>>> room was very cold, but the internal fans in the server were set to >>>>>> "be very quiet". It wasn't enough to keep the ixgbe NICs happy. I had >>>>>> to change the fan settings to "just always run full speed". >>>>>> >>>>>> The fan temperature feedback loop was based on sensors on the CPU, >>>>>> _not_ on the peripherals. >>>>>> >>>>> Hi Adrian, >>>>> >>>>> Ummm. I'll check it and improve internal cooling. :) >>>>> She is not happy and I'm also not. rsrsrsr >>>>> >>>>> Cheers, >>>>> >>>> Besides the problem of heating of the network interface, I am putting >> some >>>> information here. Could you tell me if there is something strange or is > it >>>> normal? >>>> >>>> dev.ix.0.%desc: Intel(R) PRO/10GbE PCI-Express Network Driver, Version >> - >>>> 2.5.15 >>>> dev.ix.0.%driver: ix >>>> dev.ix.0.%location: slot=0 function=0 handle=\_SB_.PCI1.BR48.S3F0 >>>> dev.ix.0.%pnpinfo: vendor=0x8086 device=0x154d subvendor=0x8086 >>>> subdevice=0x7b11 class=0x020000 >>>> dev.ix.0.%parent: pci131 >>>> dev.ix.0.fc: 3 >>>> dev.ix.0.enable_aim: 1 >>>> dev.ix.0.advertise_speed: 0 >>>> dev.ix.0.dropped: 0 >>>> dev.ix.0.mbuf_defrag_failed: 0 >>>> dev.ix.0.watchdog_events: 0 >>>> dev.ix.0.link_irq: 121769 >>>> dev.ix.0.queue0.interrupt_rate: 5319 >>>> dev.ix.0.queue0.irqs: 7900830877 >>>> dev.ix.0.queue0.txd_head: 1037 >>>> dev.ix.0.queue0.txd_tail: 1037 >>>> dev.ix.0.queue0.tso_tx: 142 >>>> dev.ix.0.queue0.no_tx_dma_setup: 0 >>>> dev.ix.0.queue0.no_desc_avail: 0 >>>> dev.ix.0.queue0.tx_packets: 9725701450 >>>> dev.ix.0.queue0.rxd_head: 1175 >>>> dev.ix.0.queue0.rxd_tail: 1174 >>>> dev.ix.0.queue0.rx_packets: 13069276955 >>>> dev.ix.0.queue0.rx_bytes: 3391061018 >>>> dev.ix.0.queue0.rx_copies: 8574407 >>>> dev.ix.0.queue0.lro_queued: 0 >>>> dev.ix.0.queue0.lro_flushed: 0 >>>> dev.ix.0.queue1.interrupt_rate: 41666 >>>> dev.ix.0.queue1.irqs: 7681141208 >>>> dev.ix.0.queue1.txd_head: 219 >>>> dev.ix.0.queue1.txd_tail: 221 >>>> dev.ix.0.queue1.tso_tx: 57 >>>> dev.ix.0.queue1.no_tx_dma_setup: 0 >>>> dev.ix.0.queue1.no_desc_avail: 44334 >>>> dev.ix.0.queue1.tx_packets: 10196891433 >>>> dev.ix.0.queue1.rxd_head: 1988 >>>> dev.ix.0.queue1.rxd_tail: 1987 >>>> dev.ix.0.queue1.rx_packets: 13210132242 >>>> dev.ix.0.queue1.rx_bytes: 4317357059 >>>> dev.ix.0.queue1.rx_copies: 8131936 >>>> dev.ix.0.queue1.lro_queued: 0 >>>> dev.ix.0.queue1.lro_flushed: 0 >>>> dev.ix.0.queue2.interrupt_rate: 5319 >>>> dev.ix.0.queue2.irqs: 7647486080 >>>> dev.ix.0.queue2.txd_head: 761 >>>> dev.ix.0.queue2.txd_tail: 761 >>>> dev.ix.0.queue2.tso_tx: 409 >>>> dev.ix.0.queue2.no_tx_dma_setup: 0 >>>> dev.ix.0.queue2.no_desc_avail: 54207 >>>> dev.ix.0.queue2.tx_packets: 10161246425 >>>> dev.ix.0.queue2.rxd_head: 1874 >>>> dev.ix.0.queue2.rxd_tail: 1872 >>>> dev.ix.0.queue2.rx_packets: 13175551880 >>>> dev.ix.0.queue2.rx_bytes: 4472798418 >>>> dev.ix.0.queue2.rx_copies: 7488876 >>>> dev.ix.0.queue2.lro_queued: 0 >>>> dev.ix.0.queue2.lro_flushed: 0 >>>> dev.ix.0.queue3.interrupt_rate: 500000 >>>> dev.ix.0.queue3.irqs: 7641129521 >>>> dev.ix.0.queue3.txd_head: 2039 >>>> dev.ix.0.queue3.txd_tail: 2039 >>>> dev.ix.0.queue3.tso_tx: 9 >>>> dev.ix.0.queue3.no_tx_dma_setup: 0 >>>> dev.ix.0.queue3.no_desc_avail: 150346 >>>> dev.ix.0.queue3.tx_packets: 10619971896 >>>> dev.ix.0.queue3.rxd_head: 1055 >>>> dev.ix.0.queue3.rxd_tail: 1054 >>>> dev.ix.0.queue3.rx_packets: 13137835529 >>>> dev.ix.0.queue3.rx_bytes: 4063197306 >>>> dev.ix.0.queue3.rx_copies: 8188713 >>>> dev.ix.0.queue3.lro_queued: 0 >>>> dev.ix.0.queue3.lro_flushed: 0 >>>> dev.ix.0.queue4.interrupt_rate: 5319 >>>> dev.ix.0.queue4.irqs: 7439824996 >>>> dev.ix.0.queue4.txd_head: 26 >>>> dev.ix.0.queue4.txd_tail: 26 >>>> dev.ix.0.queue4.tso_tx: 553912 >>>> dev.ix.0.queue4.no_tx_dma_setup: 0 >>>> dev.ix.0.queue4.no_desc_avail: 0 >>>> dev.ix.0.queue4.tx_packets: 10658683718 >>>> dev.ix.0.queue4.rxd_head: 684 >>>> dev.ix.0.queue4.rxd_tail: 681 >>>> dev.ix.0.queue4.rx_packets: 13204786830 >>>> dev.ix.0.queue4.rx_bytes: 3700845239 >>>> dev.ix.0.queue4.rx_copies: 8193379 >>>> dev.ix.0.queue4.lro_queued: 0 >>>> dev.ix.0.queue4.lro_flushed: 0 >>>> dev.ix.0.queue5.interrupt_rate: 15151 >>>> dev.ix.0.queue5.irqs: 7456613396 >>>> dev.ix.0.queue5.txd_head: 603 >>>> dev.ix.0.queue5.txd_tail: 603 >>>> dev.ix.0.queue5.tso_tx: 17 >>>> dev.ix.0.queue5.no_tx_dma_setup: 0 >>>> dev.ix.0.queue5.no_desc_avail: 0 >>>> dev.ix.0.queue5.tx_packets: 10639139790 >>>> dev.ix.0.queue5.rxd_head: 404 >>>> dev.ix.0.queue5.rxd_tail: 403 >>>> dev.ix.0.queue5.rx_packets: 13144301293 >>>> dev.ix.0.queue5.rx_bytes: 3986784766 >>>> dev.ix.0.queue5.rx_copies: 8256195 >>>> dev.ix.0.queue5.lro_queued: 0 >>>> dev.ix.0.queue5.lro_flushed: 0 >>>> dev.ix.0.queue6.interrupt_rate: 125000 >>>> dev.ix.0.queue6.irqs: 7466940576 >>>> dev.ix.0.queue6.txd_head: 1784 >>>> dev.ix.0.queue6.txd_tail: 1784 >>>> dev.ix.0.queue6.tso_tx: 2001 >>>> dev.ix.0.queue6.no_tx_dma_setup: 0 >>>> dev.ix.0.queue6.no_desc_avail: 0 >>>> dev.ix.0.queue6.tx_packets: 9784312967 >>>> dev.ix.0.queue6.rxd_head: 395 >>>> dev.ix.0.queue6.rxd_tail: 394 >>>> dev.ix.0.queue6.rx_packets: 13103079970 >>>> dev.ix.0.queue6.rx_bytes: 3581485264 >>>> dev.ix.0.queue6.rx_copies: 7336569 >>>> dev.ix.0.queue6.lro_queued: 0 >>>> dev.ix.0.queue6.lro_flushed: 0 >>>> dev.ix.0.queue7.interrupt_rate: 5319 >>>> dev.ix.0.queue7.irqs: 7486391989 >>>> dev.ix.0.queue7.txd_head: 1549 >>>> dev.ix.0.queue7.txd_tail: 1549 >>>> dev.ix.0.queue7.tso_tx: 2052 >>>> dev.ix.0.queue7.no_tx_dma_setup: 0 >>>> dev.ix.0.queue7.no_desc_avail: 0 >>>> dev.ix.0.queue7.tx_packets: 9758335509 >>>> dev.ix.0.queue7.rxd_head: 1990 >>>> dev.ix.0.queue7.rxd_tail: 1986 >>>> dev.ix.0.queue7.rx_packets: 13141436559 >>>> dev.ix.0.queue7.rx_bytes: 3877881546 >>>> dev.ix.0.queue7.rx_copies: 8505244 >>>> dev.ix.0.queue7.lro_queued: 0 >>>> dev.ix.0.queue7.lro_flushed: 0 >>>> dev.ix.0.mac_stats.crc_errs: 159 >>>> dev.ix.0.mac_stats.ill_errs: 5 >>>> dev.ix.0.mac_stats.byte_errs: 10 >>>> dev.ix.0.mac_stats.short_discards: 0 >>>> dev.ix.0.mac_stats.local_faults: 213 >>>> dev.ix.0.mac_stats.remote_faults: 37 >>>> dev.ix.0.mac_stats.rec_len_errs: 0 >>>> dev.ix.0.mac_stats.xon_txd: 33623392288 >>>> dev.ix.0.mac_stats.xon_recvd: 0 >>>> dev.ix.0.mac_stats.xoff_txd: 110161 >>>> dev.ix.0.mac_stats.xoff_recvd: 0 >>>> dev.ix.0.mac_stats.total_octets_rcvd: 37967032023930 >>>> dev.ix.0.mac_stats.good_octets_rcvd: 37965112076546 >>>> dev.ix.0.mac_stats.total_pkts_rcvd: 105201314224 >>>> dev.ix.0.mac_stats.good_pkts_rcvd: 18446743122750685300 >>>> dev.ix.0.mac_stats.mcast_pkts_rcvd: 43835 >>>> dev.ix.0.mac_stats.bcast_pkts_rcvd: 960566 >>>> dev.ix.0.mac_stats.rx_frames_64: 454067 >>>> dev.ix.0.mac_stats.rx_frames_65_127: 75350664570 >>>> dev.ix.0.mac_stats.rx_frames_128_255: 4726321485 >>>> dev.ix.0.mac_stats.rx_frames_256_511: 2282382994 >>>> dev.ix.0.mac_stats.rx_frames_512_1023: 3293330145 >>>> dev.ix.0.mac_stats.rx_frames_1024_1522: 19541190234 >>>> dev.ix.0.mac_stats.recv_undersized: 0 >>>> dev.ix.0.mac_stats.recv_fragmented: 0 >>>> dev.ix.0.mac_stats.recv_oversized: 0 >>>> dev.ix.0.mac_stats.recv_jabberd: 4 >>>> dev.ix.0.mac_stats.management_pkts_rcvd: 0 >>>> dev.ix.0.mac_stats.management_pkts_drpd: 0 >>>> dev.ix.0.mac_stats.checksum_errs: 527716650 >>>> dev.ix.0.mac_stats.good_octets_txd: 81355567084088 >>>> dev.ix.0.mac_stats.total_pkts_txd: 81545370171 >>>> dev.ix.0.mac_stats.good_pkts_txd: 47921867720 >>>> dev.ix.0.mac_stats.bcast_pkts_txd: 298646 >>>> dev.ix.0.mac_stats.mcast_pkts_txd: 18446744040086169062 >>>> dev.ix.0.mac_stats.management_pkts_txd: 0 >>>> dev.ix.0.mac_stats.tx_frames_64: 18446744045326220993 >>>> dev.ix.0.mac_stats.tx_frames_65_127: 15629697019 >>>> dev.ix.0.mac_stats.tx_frames_128_255: 4141756940 >>>> dev.ix.0.mac_stats.tx_frames_256_511: 2195917491 >>>> dev.ix.0.mac_stats.tx_frames_512_1023: 2717769904 >>>> dev.ix.0.mac_stats.tx_frames_1024_1522: 51620056990 >>>> >>>> dev.ix.1.%desc: Intel(R) PRO/10GbE PCI-Express Network Driver, Version >> - >>>> 2.5.15 >>>> dev.ix.1.%driver: ix >>>> dev.ix.1.%location: slot=0 function=1 handle=\_SB_.PCI1.BR48.S3F1 >>>> dev.ix.1.%pnpinfo: vendor=0x8086 device=0x154d subvendor=0x8086 >>>> subdevice=0x7b11 class=0x020000 >>>> dev.ix.1.%parent: pci131 >>>> dev.ix.1.fc: 3 >>>> dev.ix.1.enable_aim: 1 >>>> dev.ix.1.advertise_speed: 0 >>>> dev.ix.1.dropped: 0 >>>> dev.ix.1.mbuf_defrag_failed: 0 >>>> dev.ix.1.watchdog_events: 0 >>>> dev.ix.1.link_irq: 46692 >>>> dev.ix.1.queue0.interrupt_rate: 15151 >>>> dev.ix.1.queue0.irqs: 2634341620 >>>> dev.ix.1.queue0.txd_head: 1101 >>>> dev.ix.1.queue0.txd_tail: 1101 >>>> dev.ix.1.queue0.tso_tx: 0 >>>> dev.ix.1.queue0.no_tx_dma_setup: 0 >>>> dev.ix.1.queue0.no_desc_avail: 4142 >>>> dev.ix.1.queue0.tx_packets: 4749431326 >>>> dev.ix.1.queue0.rxd_head: 812 >>>> dev.ix.1.queue0.rxd_tail: 811 >>>> dev.ix.1.queue0.rx_packets: 21135516 >>>> dev.ix.1.queue0.rx_bytes: 33229682 >>>> dev.ix.1.queue0.rx_copies: 119434 >>>> dev.ix.1.queue0.lro_queued: 0 >>>> dev.ix.1.queue0.lro_flushed: 0 >>>> dev.ix.1.queue1.interrupt_rate: 15151 >>>> dev.ix.1.queue1.irqs: 2616347302 >>>> dev.ix.1.queue1.txd_head: 797 >>>> dev.ix.1.queue1.txd_tail: 797 >>>> dev.ix.1.queue1.tso_tx: 0 >>>> dev.ix.1.queue1.no_tx_dma_setup: 0 >>>> dev.ix.1.queue1.no_desc_avail: 4129 >>>> dev.ix.1.queue1.tx_packets: 4730356550 >>>> dev.ix.1.queue1.rxd_head: 1345 >>>> dev.ix.1.queue1.rxd_tail: 1344 >>>> dev.ix.1.queue1.rx_packets: 20844457 >>>> dev.ix.1.queue1.rx_bytes: 35014827 >>>> dev.ix.1.queue1.rx_copies: 125167 >>>> dev.ix.1.queue1.lro_queued: 0 >>>> dev.ix.1.queue1.lro_flushed: 0 >>>> dev.ix.1.queue2.interrupt_rate: 5555 >>>> dev.ix.1.queue2.irqs: 2603166062 >>>> dev.ix.1.queue2.txd_head: 1428 >>>> dev.ix.1.queue2.txd_tail: 1432 >>>> dev.ix.1.queue2.tso_tx: 0 >>>> dev.ix.1.queue2.no_tx_dma_setup: 0 >>>> dev.ix.1.queue2.no_desc_avail: 34368 >>>> dev.ix.1.queue2.tx_packets: 4681339909 >>>> dev.ix.1.queue2.rxd_head: 1153 >>>> dev.ix.1.queue2.rxd_tail: 1152 >>>> dev.ix.1.queue2.rx_packets: 21263862 >>>> dev.ix.1.queue2.rx_bytes: 46251419 >>>> dev.ix.1.queue2.rx_copies: 125156 >>>> dev.ix.1.queue2.lro_queued: 0 >>>> dev.ix.1.queue2.lro_flushed: 0 >>>> dev.ix.1.queue3.interrupt_rate: 5319 >>>> dev.ix.1.queue3.irqs: 2658483423 >>>> dev.ix.1.queue3.txd_head: 179 >>>> dev.ix.1.queue3.txd_tail: 179 >>>> dev.ix.1.queue3.tso_tx: 0 >>>> dev.ix.1.queue3.no_tx_dma_setup: 0 >>>> dev.ix.1.queue3.no_desc_avail: 4189 >>>> dev.ix.1.queue3.tx_packets: 4811705668 >>>> dev.ix.1.queue3.rxd_head: 583 >>>> dev.ix.1.queue3.rxd_tail: 582 >>>> dev.ix.1.queue3.rx_packets: 20472051 >>>> dev.ix.1.queue3.rx_bytes: 29823943 >>>> dev.ix.1.queue3.rx_copies: 129450 >>>> dev.ix.1.queue3.lro_queued: 0 >>>> dev.ix.1.queue3.lro_flushed: 0 >>>> dev.ix.1.queue4.interrupt_rate: 5319 >>>> dev.ix.1.queue4.irqs: 2629295642 >>>> dev.ix.1.queue4.txd_head: 592 >>>> dev.ix.1.queue4.txd_tail: 594 >>>> dev.ix.1.queue4.tso_tx: 0 >>>> dev.ix.1.queue4.no_tx_dma_setup: 0 >>>> dev.ix.1.queue4.no_desc_avail: 4106 >>>> dev.ix.1.queue4.tx_packets: 4774679355 >>>> dev.ix.1.queue4.rxd_head: 259 >>>> dev.ix.1.queue4.rxd_tail: 258 >>>> dev.ix.1.queue4.rx_packets: 22561633 >>>> dev.ix.1.queue4.rx_bytes: 40196457 >>>> dev.ix.1.queue4.rx_copies: 135966 >>>> dev.ix.1.queue4.lro_queued: 0 >>>> dev.ix.1.queue4.lro_flushed: 0 >>>> dev.ix.1.queue5.interrupt_rate: 18518 >>>> dev.ix.1.queue5.irqs: 2628129098 >>>> dev.ix.1.queue5.txd_head: 1070 >>>> dev.ix.1.queue5.txd_tail: 1072 >>>> dev.ix.1.queue5.tso_tx: 0 >>>> dev.ix.1.queue5.no_tx_dma_setup: 0 >>>> dev.ix.1.queue5.no_desc_avail: 4127 >>>> dev.ix.1.queue5.tx_packets: 4750613588 >>>> dev.ix.1.queue5.rxd_head: 595 >>>> dev.ix.1.queue5.rxd_tail: 594 >>>> dev.ix.1.queue5.rx_packets: 21309940 >>>> dev.ix.1.queue5.rx_bytes: 29264004 >>>> dev.ix.1.queue5.rx_copies: 134128 >>>> dev.ix.1.queue5.lro_queued: 0 >>>> dev.ix.1.queue5.lro_flushed: 0 >>>> dev.ix.1.queue6.interrupt_rate: 125000 >>>> dev.ix.1.queue6.irqs: 2615444123 >>>> dev.ix.1.queue6.txd_head: 1008 >>>> dev.ix.1.queue6.txd_tail: 1008 >>>> dev.ix.1.queue6.tso_tx: 0 >>>> dev.ix.1.queue6.no_tx_dma_setup: 0 >>>> dev.ix.1.queue6.no_desc_avail: 4106 >>>> dev.ix.1.queue6.tx_packets: 4703519800 >>>> dev.ix.1.queue6.rxd_head: 1209 >>>> dev.ix.1.queue6.rxd_tail: 1208 >>>> dev.ix.1.queue6.rx_packets: 22783319 >>>> dev.ix.1.queue6.rx_bytes: 28091721 >>>> dev.ix.1.queue6.rx_copies: 131653 >>>> dev.ix.1.queue6.lro_queued: 0 >>>> dev.ix.1.queue6.lro_flushed: 0 >>>> dev.ix.1.queue7.interrupt_rate: 6024 >>>> dev.ix.1.queue7.irqs: 2644640254 >>>> dev.ix.1.queue7.txd_head: 1313 >>>> dev.ix.1.queue7.txd_tail: 1313 >>>> dev.ix.1.queue7.tso_tx: 0 >>>> dev.ix.1.queue7.no_tx_dma_setup: 0 >>>> dev.ix.1.queue7.no_desc_avail: 4106 >>>> dev.ix.1.queue7.tx_packets: 4824584039 >>>> dev.ix.1.queue7.rxd_head: 1463 >>>> dev.ix.1.queue7.rxd_tail: 1462 >>>> dev.ix.1.queue7.rx_packets: 21464418 >>>> dev.ix.1.queue7.rx_bytes: 38133966 >>>> dev.ix.1.queue7.rx_copies: 163180 >>>> dev.ix.1.queue7.lro_queued: 0 >>>> dev.ix.1.queue7.lro_flushed: 0 >>>> dev.ix.1.mac_stats.crc_errs: 0 >>>> dev.ix.1.mac_stats.ill_errs: 0 >>>> dev.ix.1.mac_stats.byte_errs: 0 >>>> dev.ix.1.mac_stats.short_discards: 0 >>>> dev.ix.1.mac_stats.local_faults: 0 >>>> dev.ix.1.mac_stats.remote_faults: 2 >>>> dev.ix.1.mac_stats.rec_len_errs: 0 >>>> dev.ix.1.mac_stats.xon_txd: 11207820571 >>>> dev.ix.1.mac_stats.xon_recvd: 0 >>>> dev.ix.1.mac_stats.xoff_txd: 3735958948 >>>> dev.ix.1.mac_stats.xoff_recvd: 0 >>>> dev.ix.1.mac_stats.total_octets_rcvd: 47828274751 >>>> dev.ix.1.mac_stats.good_octets_rcvd: 47225376551 >>>> dev.ix.1.mac_stats.total_pkts_rcvd: 177931572 >>>> dev.ix.1.mac_stats.good_pkts_rcvd: 18446743954331576567 >>>> dev.ix.1.mac_stats.mcast_pkts_rcvd: 7410 >>>> dev.ix.1.mac_stats.bcast_pkts_rcvd: 8907 >>>> dev.ix.1.mac_stats.rx_frames_64: 0 >>>> dev.ix.1.mac_stats.rx_frames_65_127: 134619798 >>>> dev.ix.1.mac_stats.rx_frames_128_255: 7996096 >>>> dev.ix.1.mac_stats.rx_frames_256_511: 3996945 >>>> dev.ix.1.mac_stats.rx_frames_512_1023: 5145938 >>>> dev.ix.1.mac_stats.rx_frames_1024_1522: 21331154 >>>> dev.ix.1.mac_stats.recv_undersized: 0 >>>> dev.ix.1.mac_stats.recv_fragmented: 0 >>>> dev.ix.1.mac_stats.recv_oversized: 0 >>>> dev.ix.1.mac_stats.recv_jabberd: 0 >>>> dev.ix.1.mac_stats.management_pkts_rcvd: 0 >>>> dev.ix.1.mac_stats.management_pkts_drpd: 0 >>>> dev.ix.1.mac_stats.checksum_errs: 219681 >>>> dev.ix.1.mac_stats.good_octets_txd: 38874487735713 >>>> dev.ix.1.mac_stats.total_pkts_txd: 38026147390 >>>> dev.ix.1.mac_stats.good_pkts_txd: 27377335167 >>>> dev.ix.1.mac_stats.bcast_pkts_txd: 67411 >>>> dev.ix.1.mac_stats.mcast_pkts_txd: 18446744063060778699 >>>> dev.ix.1.mac_stats.management_pkts_txd: 0 >>>> dev.ix.1.mac_stats.tx_frames_64: 18446744065500352065 >>>> dev.ix.1.mac_stats.tx_frames_65_127: 6755049420 >>>> dev.ix.1.mac_stats.tx_frames_128_255: 1914523761 >>>> dev.ix.1.mac_stats.tx_frames_256_511: 1038553757 >>>> dev.ix.1.mac_stats.tx_frames_512_1023: 1240890942 >>>> dev.ix.1.mac_stats.tx_frames_1024_1522: 24637516838 >>>> >>>> >>>> Cheers, >>>> >>>> >>>>>> -a >>>>>> >>>>>> >>>>>> On 5 September 2014 07:35, Marcelo Gondim >> <gondim@bsdinfo.com.br> wrote: >>>>>>> Hi Adrian, >>>>>>> >>>>>>> I confirmed with the support staff of the room where the server is, >>>>>>> that the >>>>>>> ambient temperature was normal. >>>>>>> >>>>>>> >>>>>>> On 04/09/2014 22:46, Marcelo Gondim wrote: >>>>>>> >>>>>>>> On 04/09/2014 20:48, Adrian Chadd wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> The only time this has happened to me is because the card >> overheated. >>>>>>>>> Can you check that? >>>>>>>>> >>>>>>>> Hi Adrian, >>>>>>>> >>>>>>>> The room where the equipment is located is very cold but I'll check > it >>>>>>>> out. >>>>>>>> Also seen at the time of the problem, a lot of dropped packets. >>>>>>>> >>>>>>>> # netstat -idn >>>>>>>> ... >>>>>>>> ix0 1500 <Link#9> a0:36:9f:2a:6d:ac 18446743423829095869 > 159 >>>>>>>> 750924631703 53285910688 0 0 0 >>>>>>>> ix0 - fe80::a236:9f fe80::a236:9fff:f 0 - - > 2 >>>>>>>> - - - >>>>>>>> ix1 1500 <Link#10> a0:36:9f:2a:6d:ae 18446743954328745465 > 0 >>>>>>>> 119550050209 20178077451 0 0 0 >>>>>>>> ix1 - fe80::a236:9f fe80::a236:9fff:f 0 - - > 1 >>>>>>>> - - - >>>>>>>> ... >>>>>>>> >>>>>>>> 119550050209 droped packets on ix1 and 750924631703 droped on ix0 >>>>>>>> >>>>>>>> Could be interesting I upgrade to10.1-PRERELEASE? >>>>>>>> Could there be a problem with the driver? >>>>>>>> >>>>>>>> Traffic on ix0: 1.4Gbps output / 600Mbps input >>>>>>>> Traffic on ix1: 1.2Gbps output >>>>>>>> >>>>>>>> PPS on ix0: 163Kpps output / 215Kpps input >>>>>>>> PPS on ix1: 131Kpps output >>>>>>>> >>>>>>>> Thanks for your help. >>>>>>>> >>>>>>>>> -a >>>>>>>>> >>>>>>>>> >>>>>>>>> On 4 September 2014 16:14, Marcelo Gondim >> <gondim@bsdinfo.com.br> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi All, >>>>>>>>>> >>>>>>>>>> I have an Intel X520-SR2and today was working when all traffic >>>>>>>>>> stopped. >>>>>>>>>> I looked in the logs and found this message: >>>>>>>>>> >>>>>>>>>> Sep 4 18:29:53 rt01 kernel: ix1: >>>>>>>>>> Sep 4 18:29:53 rt01 kernel: CRITICAL: ECC ERROR!! Please > Reboot!! >>>>>>>>>> # uname -a >>>>>>>>>> FreeBSD rt01.xxxxx.com.br 10.0-STABLE FreeBSD 10.0-STABLE #10 >>>>>>>>>> r267839: >>>>>>>>>> Thu >>>>>>>>>> Jul 10 15:35:04 BRT 2014 >>>>>>>>>> root@rt01.xxxxx.com.br:/usr/obj/usr/src/sys/GONDIM10 amd64 >>>>>>>>>> >>>>>>>>>> # netstat -m >>>>>>>>>> 98324/53476/151800 mbufs in use (current/cache/total) >>>>>>>>>> 98301/44951/143252/1014370 mbuf clusters in use >>>>>>>>>> (current/cache/total/max) >>>>>>>>>> 98301/44897 mbuf+clusters out of packet secondary zone in use >>>>>>>>>> (current/cache) >>>>>>>>>> 0/421/421/507184 4k (page size) jumbo clusters in use >>>>>>>>>> (current/cache/total/max) >>>>>>>>>> 0/0/0/150276 9k jumbo clusters in use (current/cache/total/max) >>>>>>>>>> 0/0/0/84530 16k jumbo clusters in use (current/cache/total/max) >>>>>>>>>> 221183K/104955K/326138K bytes allocated to network >>>>>>>>>> (current/cache/total) >>>>>>>>>> 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) >>>>>>>>>> 0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters) >>>>>>>>>> 0/0/0 requests for jumbo clusters delayed (4k/9k/16k) >>>>>>>>>> 0/0/0 requests for jumbo clusters denied (4k/9k/16k) >>>>>>>>>> 0 requests for sfbufs denied >>>>>>>>>> 0 requests for sfbufs delayed >>>>>>>>>> 0 requests for I/O initiated by sendfile >>>>>>>>>> >>>>>>>>>> Best regards,
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?540E547C.9030703>