Date: Mon, 10 Oct 2011 23:59:48 -0700 From: Jason Wolfe <nitroboost@gmail.com> To: freebsd-net@freebsd.org Subject: Re: Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIX enabled Message-ID: <CAAAm0r3XdsMHZh%2BP_NF-txZasdExzwZ8ymmGQgGhJQds0fOiBQ@mail.gmail.com> In-Reply-To: <CAAAm0r1DKvoL9=Ket9up=4%2B5xiCzTTZJK99FhF9jcCA28B0M%2BA@mail.gmail.com> References: <CAAAm0r0RXEJo4UiKS=Ui0e5OQTg6sg-xcYf3mYB5%2Bvk8i8557w@mail.gmail.com> <4E8F157A.40702@sentex.net> <CAAAm0r2JH43Rct7UxQK2duH1p43Nepnj5mpb6bXo==DPayhJLg@mail.gmail.com> <4E8F51D4.1060509@sentex.net> <CACqU3MVwLaepFymZJkaVk6p=SpykGhqs=VYFjLh9fP9S=AxDhg@mail.gmail.com> <CAAAm0r1DKvoL9=Ket9up=4%2B5xiCzTTZJK99FhF9jcCA28B0M%2BA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, Oct 7, 2011 at 2:14 PM, Jason Wolfe <nitroboost@gmail.com> wrote: > Bumping rx/tx descriptors to 2048 was actually for performance reasons and > not to try to get around the issue. I did some fairly in depth testing and > found under heavy load it performed the best with those settings. > > As mentioned on the other thread I'll re enable MSI-X on a few servers here > and collect uptime and the kernel msgbuf in addition. I'll bump the > descriptors down to 512 to try and increase our chances and compile the > driver with EM_MULTIQUEUE also. > > Jason > Hi again, I flipped MSI-X on across ~15 boxes, and I did manage to pick up a fresh report over the weekend. This server in particular was doing ~1.2Gb at the time of the hang, but within a few hours on each side of the issue it pushed 1.5Gb without a hiccup. I have a multiqueue driver/kernel ready to go out, but wanted to send a report over with the driver running in 'stock' form with the extra requested data to use as a baseline. I have the script pulling out libwrap denies, so it reported in blank with nothing of value. I'll send in the action again once I see the results of multiqueue. I bounced em0 because dropped packets incremented 485695 to 486120 and the interface is not incrementing packets out. 1:10PM up 2 days, 4:10, 0 users, load averages: 0.47, 0.58, 0.63 interrupt total rate irq3: uart1 2240 0 cpu0: timer 375578618 2000 irq256: em0:rx 0 19094110 101 irq257: em0:tx 0 1118621232 5956 irq258: em0:link 1 0 irq259: em1:rx 0 1349901405 7188 irq260: em1:tx 0 1118832931 5958 irq261: em1:link 6131 0 irq262: mps0 200567934 1068 cpu2: timer 375570455 2000 cpu1: timer 375570455 2000 cpu3: timer 375570447 2000 Total 5309315959 28273 54783/18267/73050 mbufs in use (current/cache/total) 6530/3470/10000/5602642 mbuf clusters in use (current/cache/total/max) 6530/1021 mbuf+clusters out of packet secondary zone in use (current/cache) 43733/1145/44878/2801320 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) 201687K/16086K/217774K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/0/0 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 0 calls to protocol drain routines Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll Drop em0 1500 <Link#1> 00:25:90:2b:d6:e5 20018958 0 0 5955729874 0 0 486980 em0 1500 fe80:1::225:9 fe80:1::225:90ff: 0 - - 3 - - - em1 1500 <Link#2> 00:25:90:2b:d6:e5 5310041739 29785 0 5896740880 0 0 291131 em1 1500 fe80:2::225:9 fe80:2::225:90ff: 0 - - 2 - - - lagg0 1500 <Link#5> 00:25:90:2b:d6:e5 5330031157 0 0 11850884004 778111 0 0 lagg0 1500 69.164.38.0/2 69.164.38.116 4999076547 - - 11852994906 - - - lagg0 1500 fe80:5::225:9 fe80:5::225:90ff: 4 - - 5 - - - lagg0 1500 2607:f4e8:310 2607:f4e8:310:12: 5331 - - 5334 - - - kern.msgbuf: Oct 10 13:10:04 cds1066 kernel: Interface is RUNNING and INACTIVE Oct 10 13:10:04 cds1066 kernel: em0: hw tdh = 944, hw tdt = 944 Oct 10 13:10:04 cds1066 kernel: em0: hw rdh = 1805, hw rdt = 1804 Oct 10 13:10:04 cds1066 kernel: em0: Tx Queue Status = 0 Oct 10 13:10:04 cds1066 kernel: em0: TX descriptors avail = 2048 Oct 10 13:10:04 cds1066 kernel: em0: Tx Descriptors avail failure = 0 Oct 10 13:10:04 cds1066 kernel: em0: RX discarded packets = 0 Oct 10 13:10:04 cds1066 kernel: em0: RX Next to Check = 1806 Oct 10 13:10:04 cds1066 kernel: em0: RX Next to Refresh = 1805 Oct 10 13:10:04 cds1066 kernel: Interface is RUNNING and INACTIVE Oct 10 13:10:04 cds1066 kernel: em1: hw tdh = 1329, hw tdt = 1329 Oct 10 13:10:04 cds1066 kernel: em1: hw rdh = 464, hw rdt = 463 Oct 10 13:10:04 cds1066 kernel: em1: Tx Queue Status = 1 Oct 10 13:10:04 cds1066 kernel: em1: TX descriptors avail = 1807 Oct 10 13:10:04 cds1066 kernel: em1: Tx Descriptors avail failure = 0 Oct 10 13:10:04 cds1066 kernel: em1: RX discarded packets = 0 Oct 10 13:10:04 cds1066 kernel: em1: RX Next to Check = 716 Oct 10 13:10:04 cds1066 kernel: em1: RX Next to Refresh = 749 net.inet.ip.intr_queue_maxlen: 512 net.inet.ip.intr_queue_drops: 0 dev.em.0.%desc: Intel(R) PRO/1000 Network Connection 7.2.3 dev.em.0.%driver: em dev.em.0.%location: slot=0 function=0 dev.em.0.%pnpinfo: vendor=0x8086 device=0x10d3 subvendor=0x15d9 subdevice=0x10d3 class=0x020000 dev.em.0.%parent: pci1 dev.em.0.nvm: -1 dev.em.0.debug: -1 dev.em.0.rx_int_delay: 0 dev.em.0.tx_int_delay: 66 dev.em.0.rx_abs_int_delay: 66 dev.em.0.tx_abs_int_delay: 66 dev.em.0.rx_processing_limit: 100 dev.em.0.flow_control: 3 dev.em.0.eee_control: 0 dev.em.0.link_irq: 1 dev.em.0.mbuf_alloc_fail: 0 dev.em.0.cluster_alloc_fail: 0 dev.em.0.dropped: 0 dev.em.0.tx_dma_fail: 0 dev.em.0.rx_overruns: 0 dev.em.0.watchdog_timeouts: 0 dev.em.0.device_control: 1074790984 dev.em.0.rx_control: 67141634 dev.em.0.fc_high_water: 18432 dev.em.0.fc_low_water: 16932 dev.em.0.queue0.txd_head: 944 dev.em.0.queue0.txd_tail: 944 dev.em.0.queue0.tx_irq: 1118621164 dev.em.0.queue0.no_desc_avail: 0 dev.em.0.queue0.rxd_head: 1970 dev.em.0.queue0.rxd_tail: 1969 dev.em.0.queue0.rx_irq: 19094262 dev.em.0.mac_stats.excess_coll: 0 dev.em.0.mac_stats.single_coll: 0 dev.em.0.mac_stats.multiple_coll: 0 dev.em.0.mac_stats.late_coll: 0 dev.em.0.mac_stats.collision_count: 0 dev.em.0.mac_stats.symbol_errors: 0 dev.em.0.mac_stats.sequence_errors: 0 dev.em.0.mac_stats.defer_count: 0 dev.em.0.mac_stats.missed_packets: 0 dev.em.0.mac_stats.recv_no_buff: 0 dev.em.0.mac_stats.recv_undersize: 0 dev.em.0.mac_stats.recv_fragmented: 0 dev.em.0.mac_stats.recv_oversize: 0 dev.em.0.mac_stats.recv_jabber: 0 dev.em.0.mac_stats.recv_errs: 0 dev.em.0.mac_stats.crc_errs: 0 dev.em.0.mac_stats.alignment_errs: 0 dev.em.0.mac_stats.coll_ext_errs: 0 dev.em.0.mac_stats.xon_recvd: 0 dev.em.0.mac_stats.xon_txd: 0 dev.em.0.mac_stats.xoff_recvd: 0 dev.em.0.mac_stats.xoff_txd: 0 dev.em.0.mac_stats.total_pkts_recvd: 20025749 dev.em.0.mac_stats.good_pkts_recvd: 20025749 dev.em.0.mac_stats.bcast_pkts_recvd: 20017848 dev.em.0.mac_stats.mcast_pkts_recvd: 958 dev.em.0.mac_stats.rx_frames_64: 20017879 dev.em.0.mac_stats.rx_frames_65_127: 1430 dev.em.0.mac_stats.rx_frames_128_255: 6324 dev.em.0.mac_stats.rx_frames_256_511: 115 dev.em.0.mac_stats.rx_frames_512_1023: 1 dev.em.0.mac_stats.rx_frames_1024_1522: 0 dev.em.0.mac_stats.good_octets_recvd: 1282182184 dev.em.0.mac_stats.good_octets_txd: 8285874985788 dev.em.0.mac_stats.total_pkts_txd: 5955738931 dev.em.0.mac_stats.good_pkts_txd: 5955738931 dev.em.0.mac_stats.bcast_pkts_txd: 5 dev.em.0.mac_stats.mcast_pkts_txd: 1255 dev.em.0.mac_stats.tx_frames_64: 21396178 dev.em.0.mac_stats.tx_frames_65_127: 329580959 dev.em.0.mac_stats.tx_frames_128_255: 4571211 dev.em.0.mac_stats.tx_frames_256_511: 8152970 dev.em.0.mac_stats.tx_frames_512_1023: 50636253 dev.em.0.mac_stats.tx_frames_1024_1522: 5541401360 dev.em.0.mac_stats.tso_txd: 0 dev.em.0.mac_stats.tso_ctx_fail: 0 dev.em.0.interrupts.asserts: 3 dev.em.0.interrupts.rx_pkt_timer: 0 dev.em.0.interrupts.rx_abs_timer: 0 dev.em.0.interrupts.tx_pkt_timer: 0 dev.em.0.interrupts.tx_abs_timer: 0 dev.em.0.interrupts.tx_queue_empty: 0 dev.em.0.interrupts.tx_queue_min_thresh: 0 dev.em.0.interrupts.rx_desc_min_thresh: 0 dev.em.0.interrupts.rx_overrun: 0 dev.em.1.%desc: Intel(R) PRO/1000 Network Connection 7.2.3 dev.em.1.%driver: em dev.em.1.%location: slot=0 function=0 dev.em.1.%pnpinfo: vendor=0x8086 device=0x10d3 subvendor=0x15d9 subdevice=0x10d3 class=0x020000 dev.em.1.%parent: pci2 dev.em.1.nvm: -1 dev.em.1.debug: -1 dev.em.1.rx_int_delay: 0 dev.em.1.tx_int_delay: 66 dev.em.1.rx_abs_int_delay: 66 dev.em.1.tx_abs_int_delay: 66 dev.em.1.rx_processing_limit: 100 dev.em.1.flow_control: 3 dev.em.1.eee_control: 0 dev.em.1.link_irq: 6115 dev.em.1.mbuf_alloc_fail: 0 dev.em.1.cluster_alloc_fail: 0 dev.em.1.dropped: 0 dev.em.1.tx_dma_fail: 0 dev.em.1.rx_overruns: 0 dev.em.1.watchdog_timeouts: 0 dev.em.1.device_control: 1074790984 dev.em.1.rx_control: 67141634 dev.em.1.fc_high_water: 18432 dev.em.1.fc_low_water: 16932 dev.em.1.queue0.txd_head: 1873 dev.em.1.queue0.txd_tail: 1873 dev.em.1.queue0.tx_irq: 1118847525 dev.em.1.queue0.no_desc_avail: 0 dev.em.1.queue0.rxd_head: 1703 dev.em.1.queue0.rxd_tail: 1702 dev.em.1.queue0.rx_irq: 1344597485 dev.em.1.mac_stats.excess_coll: 0 dev.em.1.mac_stats.single_coll: 0 dev.em.1.mac_stats.multiple_coll: 0 dev.em.1.mac_stats.late_coll: 0 dev.em.1.mac_stats.collision_count: 0 dev.em.1.mac_stats.symbol_errors: 0 dev.em.1.mac_stats.sequence_errors: 0 dev.em.1.mac_stats.defer_count: 0 dev.em.1.mac_stats.missed_packets: 29780 dev.em.1.mac_stats.recv_no_buff: 3183 dev.em.1.mac_stats.recv_undersize: 0 dev.em.1.mac_stats.recv_fragmented: 0 dev.em.1.mac_stats.recv_oversize: 5 dev.em.1.mac_stats.recv_jabber: 0 dev.em.1.mac_stats.recv_errs: 0 dev.em.1.mac_stats.crc_errs: 0 dev.em.1.mac_stats.alignment_errs: 0 dev.em.1.mac_stats.coll_ext_errs: 0 dev.em.1.mac_stats.xon_recvd: 0 dev.em.1.mac_stats.xon_txd: 0 dev.em.1.mac_stats.xoff_recvd: 0 dev.em.1.mac_stats.xoff_txd: 0 dev.em.1.mac_stats.total_pkts_recvd: 5310091665 dev.em.1.mac_stats.good_pkts_recvd: 5310061880 dev.em.1.mac_stats.bcast_pkts_recvd: 20016936 dev.em.1.mac_stats.mcast_pkts_recvd: 957 dev.em.1.mac_stats.rx_frames_64: 2266338548 dev.em.1.mac_stats.rx_frames_65_127: 2435691095 dev.em.1.mac_stats.rx_frames_128_255: 3776153 dev.em.1.mac_stats.rx_frames_256_511: 8561090 dev.em.1.mac_stats.rx_frames_512_1023: 26381841 dev.em.1.mac_stats.rx_frames_1024_1522: 569313153 dev.em.1.mac_stats.good_octets_recvd: 1204804170787 dev.em.1.mac_stats.good_octets_txd: 8179280664329 dev.em.1.mac_stats.total_pkts_txd: 5896783132 dev.em.1.mac_stats.good_pkts_txd: 5896783132 dev.em.1.mac_stats.bcast_pkts_txd: 779 dev.em.1.mac_stats.mcast_pkts_txd: 9 dev.em.1.mac_stats.tx_frames_64: 22403486 dev.em.1.mac_stats.tx_frames_65_127: 354651732 dev.em.1.mac_stats.tx_frames_128_255: 4629346 dev.em.1.mac_stats.tx_frames_256_511: 8217773 dev.em.1.mac_stats.tx_frames_512_1023: 47239698 dev.em.1.mac_stats.tx_frames_1024_1522: 5459641097 dev.em.1.mac_stats.tso_txd: 0 dev.em.1.mac_stats.tso_ctx_fail: 0 dev.em.1.interrupts.asserts: 5493 dev.em.1.interrupts.rx_pkt_timer: 0 dev.em.1.interrupts.rx_abs_timer: 0 dev.em.1.interrupts.tx_pkt_timer: 0 dev.em.1.interrupts.tx_abs_timer: 0 dev.em.1.interrupts.tx_queue_empty: 0 dev.em.1.interrupts.tx_queue_min_thresh: 0 dev.em.1.interrupts.rx_desc_min_thresh: 0 dev.em.1.interrupts.rx_overrun: 4 Jason
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAAAm0r3XdsMHZh%2BP_NF-txZasdExzwZ8ymmGQgGhJQds0fOiBQ>