Date: Mon, 31 Oct 2011 12:34:28 +0330 From: Hooman Fazaeli <hoomanfazaeli@gmail.com> To: Emil Muratov <gpm@hotplug.ru> Cc: freebsd-net@freebsd.org, Jack Vogel <jfvogel@gmail.com>, Jason Wolfe <nitroboost@gmail.com> Subject: Re: Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIX enabled Message-ID: <4EAE649C.6040106@gmail.com> In-Reply-To: <4EAE58BC.2090000@hotplug.ru> References: <CAAAm0r0RXEJo4UiKS=Ui0e5OQTg6sg-xcYf3mYB5%2Bvk8i8557w@mail.gmail.com> <4E8F157A.40702@sentex.net> <CAAAm0r2JH43Rct7UxQK2duH1p43Nepnj5mpb6bXo==DPayhJLg@mail.gmail.com> <4E8F51D4.1060509@sentex.net> <CACqU3MVwLaepFymZJkaVk6p=SpykGhqs=VYFjLh9fP9S=AxDhg@mail.gmail.com> <CAAAm0r1DKvoL9=Ket9up=4%2B5xiCzTTZJK99FhF9jcCA28B0M%2BA@mail.gmail.com> <CAAAm0r3XdsMHZh%2BP_NF-txZasdExzwZ8ymmGQgGhJQds0fOiBQ@mail.gmail.com> <CAAAm0r1iS3z-7CBJ=xYDf%2BJOA1Q2nU0O54Twbyb7FjvgWHjKVw@mail.gmail.com> <4EA7E203.3020306@sepehrs.com> <CAAAm0r3Nr2t8cCetPkFnLQ-3KwqHw_0SpqbtvYPRUkSP=9n8CA@mail.gmail.com> <4EA80818.3030504@sentex.net> <4EA80F88.4000400@hotplug.ru> <4EA82715.2000404@gmail.com> <4EA8FA40.7010504@hotplug.ru> <4EA91836.2040508@gmail.com> <4EAE58BC.2090000@hotplug.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
On 10/31/2011 11:43 AM, Emil Muratov wrote: > >> >> You may try these settings and see if they help: >> >> - hw.em.fc_setting=0 (in /boot/loader.conf) >> - hw.em.rxd="4096" (in /boot/loader.conf) >> - hw.em.txd="4096" (in /boot/loader.conf) >> - Fix speed and duplex at both link sides. After doing that, confirm on the freebsd >> box (with ifconfig) and the other device (with whatever command it provides) that >> the same speed and duplex is used by both devices. >> >> you also have high values for dev.em.0.rx/tx_[abs]_int_delay. If you >> have set them manually, remove them or replace them with these in loader.conf: >> >> hw.em.rx_int_delay=0 >> hw.em.tx_int_delay=66 >> hw.em.tx_abs_int_delay=66 >> hw.em.rx_abs_int_delay=66 >> >> these may be set via corresponding sysctls too. >> > > Still no luck with the above settings, I've got another lockups a couple of times. Here is the recent details > > > ================================= > 11.10.30-23:43:06 ... interface em0 is down... > we have Ierrs and no ingoing packets for 5 secs, interface em0 must be toggled > > 11:43PM up 1 day, 3:01, 2 users, load averages: 0.76, 0.64, 0.70 > > == vmstat -i == > interrupt total rate > irq18: ehci0 1145540 11 > irq22: nfe0 473895599 4872 > cpu0: timer 195004026 2005 > irq256: ahci0 12832958 131 > irq257: em0:rx 0 95571051 982 > irq258: em0:tx 0 88777545 912 > irq259: em0:link 946 0 > cpu3: timer 195003397 2005 > cpu1: timer 195003398 2005 > cpu2: timer 195003399 2005 > Total 1452237859 14932 > > == netstat -m == > 5424/1701/7125 mbufs in use (current/cache/total) > 719/1185/1904/51200 mbuf clusters in use (current/cache/total/max) > 719/582 mbuf+clusters out of packet secondary zone in use (current/cache) > 329/583/912/12800 4k (page size) jumbo clusters in use (current/cache/total/max) > 4095/342/4437/12800 9k jumbo clusters in use (current/cache/total/max) > 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) > 40978K/8205K/49183K bytes allocated to network (current/cache/total) > 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > 0/6663503/0 requests for jumbo clusters denied (4k/9k/16k) > 0/0/0 sfbufs in use (current/peak/max) > 0 requests for sfbufs denied > 0 requests for sfbufs delayed > 0 requests for I/O initiated by sendfile > 0 calls to protocol drain routines > > == netstat -ind == > Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll Drop > usbus 0 <Link#1> 0 0 0 0 0 0 0 > usbus 0 <Link#2> 0 0 0 0 0 0 0 > nfe0 1500 <Link#3> 00:25:22:21:86:89 196018201 0 0 350650768 0 0 664 > nfe0 1500 fe80::225:22f fe80::225:22ff:fe 0 - - 0 - - - > nfe0 1500 10.16.128.0/1 10.16.189.71 6 - - 29787707 - - - > em0 9000 <Link#4> 00:1b:21:ab:bf:4a 175676617 949 0 101627139 0 0 0 > em0 9000 192.168.168.0 192.168.168.1 7628423 - - 13654747 - - - > em0 9000 fe80::21b:21f fe80::21b:21ff:fe 45 - - 5747 - - - > em0 9000 2002:d5xx:xxx 2002:d5xx:xxxx:x: 153 - - 159 - - - > > Oct 30 23:43:06 ion kernel: Interface is RUNNING and INACTIVE > Oct 30 23:43:07 ion kernel: em0: hw tdh = 2656, hw tdt = 3271 > Oct 30 23:43:07 ion kernel: em0: hw rdh = 2112, hw rdt = 2111 > Oct 30 23:43:07 ion kernel: em0: Tx Queue Status = 1 > Oct 30 23:43:07 ion kernel: em0: TX descriptors avail = 3481 > Oct 30 23:43:07 ion kernel: em0: Tx Descriptors avail failure = 0 > Oct 30 23:43:07 ion kernel: em0: RX discarded packets = 0 > Oct 30 23:43:07 ion kernel: em0: RX Next to Check = 2112 > Oct 30 23:43:07 ion kernel: em0: RX Next to Refresh = 2111 > net.inet.ip.intr_queue_maxlen: 4096 > net.inet.ip.intr_queue_drops: 0 > dev.em.0.%desc: Intel(R) PRO/1000 Network Connection 7.2.3 > dev.em.0.%driver: em > dev.em.0.%location: slot=0 function=0 > dev.em.0.%pnpinfo: vendor=0x8086 device=0x10d3 subvendor=0x8086 subdevice=0xa01f class=0x020000 > dev.em.0.%parent: pci2 > dev.em.0.nvm: -1 > dev.em.0.debug: -1 > dev.em.0.rx_int_delay: 0 > dev.em.0.tx_int_delay: 66 > dev.em.0.rx_abs_int_delay: 66 > dev.em.0.tx_abs_int_delay: 66 > dev.em.0.rx_processing_limit: 100 > dev.em.0.flow_control: 0 > dev.em.0.eee_control: 0 > dev.em.0.link_irq: 956 > dev.em.0.mbuf_alloc_fail: 0 > dev.em.0.cluster_alloc_fail: 0 > dev.em.0.dropped: 0 > dev.em.0.tx_dma_fail: 1 > dev.em.0.rx_overruns: 0 > dev.em.0.watchdog_timeouts: 0 > dev.em.0.device_control: 1074790984 > dev.em.0.rx_control: 100827170 > dev.em.0.fc_high_water: 11264 > dev.em.0.fc_low_water: 9764 > dev.em.0.queue0.txd_head: 2656 > dev.em.0.queue0.txd_tail: 3274 > dev.em.0.queue0.tx_irq: 88769608 > dev.em.0.queue0.no_desc_avail: 0 > dev.em.0.queue0.rxd_head: 2112 > dev.em.0.queue0.rxd_tail: 2111 > dev.em.0.queue0.rx_irq: 95554873 > dev.em.0.mac_stats.excess_coll: 0 > dev.em.0.mac_stats.single_coll: 0 > dev.em.0.mac_stats.multiple_coll: 0 > dev.em.0.mac_stats.late_coll: 0 > dev.em.0.mac_stats.collision_count: 0 > dev.em.0.mac_stats.symbol_errors: 0 > dev.em.0.mac_stats.sequence_errors: 0 > dev.em.0.mac_stats.defer_count: 0 > dev.em.0.mac_stats.missed_packets: 959 > dev.em.0.mac_stats.recv_no_buff: 0 > dev.em.0.mac_stats.recv_undersize: 0 > dev.em.0.mac_stats.recv_fragmented: 0 > dev.em.0.mac_stats.recv_oversize: 0 > dev.em.0.mac_stats.recv_jabber: 0 > dev.em.0.mac_stats.recv_errs: 0 > dev.em.0.mac_stats.crc_errs: 0 > dev.em.0.mac_stats.alignment_errs: 0 > dev.em.0.mac_stats.coll_ext_errs: 0 > dev.em.0.mac_stats.xon_recvd: 0 > dev.em.0.mac_stats.xon_txd: 0 > dev.em.0.mac_stats.xoff_recvd: 0 > dev.em.0.mac_stats.xoff_txd: 0 > dev.em.0.mac_stats.total_pkts_recvd: 175670049 > dev.em.0.mac_stats.good_pkts_recvd: 175669090 > dev.em.0.mac_stats.bcast_pkts_recvd: 2928 > dev.em.0.mac_stats.mcast_pkts_recvd: 1082 > dev.em.0.mac_stats.rx_frames_64: 4539426 > dev.em.0.mac_stats.rx_frames_65_127: 10242159 > dev.em.0.mac_stats.rx_frames_128_255: 576238 > dev.em.0.mac_stats.rx_frames_256_511: 372297 > dev.em.0.mac_stats.rx_frames_512_1023: 2208708 > dev.em.0.mac_stats.rx_frames_1024_1522: 157730262 > dev.em.0.mac_stats.good_octets_recvd: 238712986216 > dev.em.0.mac_stats.good_octets_txd: 40512955499 > dev.em.0.mac_stats.total_pkts_txd: 108590895 > dev.em.0.mac_stats.good_pkts_txd: 108590895 > dev.em.0.mac_stats.bcast_pkts_txd: 4604 > dev.em.0.mac_stats.mcast_pkts_txd: 13969 > dev.em.0.mac_stats.tx_frames_64: 13905445 > dev.em.0.mac_stats.tx_frames_65_127: 69702633 > dev.em.0.mac_stats.tx_frames_128_255: 1442225 > dev.em.0.mac_stats.tx_frames_256_511: 755348 > dev.em.0.mac_stats.tx_frames_512_1023: 375982 > dev.em.0.mac_stats.tx_frames_1024_1522: 22409262 > dev.em.0.mac_stats.tso_txd: 4695644 > dev.em.0.mac_stats.tso_ctx_fail: 0 > dev.em.0.interrupts.asserts: 36 > dev.em.0.interrupts.rx_pkt_timer: 0 > dev.em.0.interrupts.rx_abs_timer: 0 > dev.em.0.interrupts.tx_pkt_timer: 0 > dev.em.0.interrupts.tx_abs_timer: 0 > dev.em.0.interrupts.tx_queue_empty: 0 > dev.em.0.interrupts.tx_queue_min_thresh: 0 > dev.em.0.interrupts.rx_desc_min_thresh: 0 > dev.em.0.interrupts.rx_overrun: 0 > > == ifconfig em0 == > > em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000 > description: LAN > options=219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC> > ether 00:1b:21:ab:bf:4a > inet 192.168.168.1 netmask 0xffffffc0 broadcast 192.168.168.63 > inet6 fe80::21b:21ff:feab:bf4a%em0 prefixlen 64 scopeid 0x4 > inet6 2002:d5xx:xxxx:1::1 prefixlen 64 > nd6 options=1<PERFORMNUD> > media: Ethernet 1000baseT (1000baseT <full-duplex>) > status: active > > > > > The patch I prepared for Jason contains a possible fix for your problem (Cc'ed). Can you give it a try? Before that, and to rule out other possibilities, you may disable MSIX and see if it helps: hw.em.enable_msix="0" (in /boot/loader.conf)
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4EAE649C.6040106>