Date: Thu, 20 Mar 2014 16:56:44 -0300 From: Christopher Forgeron <csforgeron@gmail.com> To: Rick Macklem <rmacklem@uoguelph.ca> Cc: freebsd-net@freebsd.org Subject: Re: 9.2 ixgbe tx queue hang Message-ID: <CAB2_NwAOmPtZjB03pdDiTK2OvQgqk-tYf83Jq4Ukt9jnZA8CNA@mail.gmail.com> In-Reply-To: <1159309884.25490921.1395282576806.JavaMail.root@uoguelph.ca> References: <CAB2_NwDG=gB1WCJ7JKTHpkJCrvPuAhipkn%2BvPyT%2BxXzOBrTGkg@mail.gmail.com> <1159309884.25490921.1395282576806.JavaMail.root@uoguelph.ca>
next in thread | previous in thread | raw e-mail | index | archive | help
BTW, When I have the problem, this is what I see from netstat -m 4080/2956/7036/6127254 mbuf clusters in use (current/cache/total/max) 4080/2636 mbuf+clusters out of packet secondary zone in use (current/cache) 0/50/50/3063627 4k (page size) jumbo clusters in use (current/cache/total/max) 32768/155/32923/907741 9k jumbo clusters in use (current/cache/total/max) 0/0/0/510604 16k jumbo clusters in use (current/cache/total/max) 312541K/9182K/321724K bytes allocated to network (current/cache/total) 34481/2600/4091 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters delayed (4k/9k/16k) 50/27433/0 requests for jumbo clusters denied (4k/9k/16k) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile It doesn't look that bad to me, other than all of the denied counts - But I can't see sysctl buffer numbers that look too low... For those who are interested, here is a dump of hw.ix and dev.ix.0 (I have ix.1 off) hw.ix.enable_aim: 1 hw.ix.max_interrupt_rate: 31250 hw.ix.rx_process_limit: 256 hw.ix.tx_process_limit: 256 hw.ix.enable_msix: 1 hw.ix.num_queues: 8 hw.ix.txd: 2048 hw.ix.rxd: 2048 2014-03-20 16:29:05.291 - INFO - dev.ix.0.%desc: Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.5.15 dev.ix.0.%driver: ix dev.ix.0.%location: slot=0 function=0 dev.ix.0.%pnpinfo: vendor=0x8086 device=0x10f8 subvendor=0x8086 subdevice=0x000c class=0x020000 dev.ix.0.%parent: pci5 dev.ix.0.fc: 3 dev.ix.0.enable_aim: 1 dev.ix.0.advertise_speed: 0 dev.ix.0.dropped: 0 dev.ix.0.mbuf_defrag_failed: 0 dev.ix.0.watchdog_events: 0 dev.ix.0.link_irq: 5 dev.ix.0.queue0.interrupt_rate: 500000 dev.ix.0.queue0.irqs: 452969 dev.ix.0.queue0.txd_head: 319 dev.ix.0.queue0.txd_tail: 319 dev.ix.0.queue0.tso_tx: 61107 dev.ix.0.queue0.no_tx_dma_setup: 0 dev.ix.0.queue0.no_desc_avail: 0 dev.ix.0.queue0.tx_packets: 257636 dev.ix.0.queue0.rxd_head: 531 dev.ix.0.queue0.rxd_tail: 530 dev.ix.0.queue0.rx_packets: 522771 dev.ix.0.queue0.rx_bytes: 1318022421 dev.ix.0.queue0.rx_copies: 224837 dev.ix.0.queue0.lro_queued: 424583 dev.ix.0.queue0.lro_flushed: 181580 dev.ix.0.queue1.interrupt_rate: 125000 dev.ix.0.queue1.irqs: 22756 dev.ix.0.queue1.txd_head: 1169 dev.ix.0.queue1.txd_tail: 1169 dev.ix.0.queue1.tso_tx: 0 dev.ix.0.queue1.no_tx_dma_setup: 0 dev.ix.0.queue1.no_desc_avail: 0 dev.ix.0.queue1.tx_packets: 23202 dev.ix.0.queue1.rxd_head: 337 dev.ix.0.queue1.rxd_tail: 336 dev.ix.0.queue1.rx_packets: 337 dev.ix.0.queue1.rx_bytes: 32988 dev.ix.0.queue1.rx_copies: 225 dev.ix.0.queue1.lro_queued: 335 dev.ix.0.queue1.lro_flushed: 320 dev.ix.0.queue2.interrupt_rate: 500000 dev.ix.0.queue2.irqs: 20256 dev.ix.0.queue2.txd_head: 1201 dev.ix.0.queue2.txd_tail: 1201 dev.ix.0.queue2.tso_tx: 0 dev.ix.0.queue2.no_tx_dma_setup: 0 dev.ix.0.queue2.no_desc_avail: 0 dev.ix.0.queue2.tx_packets: 20962 dev.ix.0.queue2.rxd_head: 1021 dev.ix.0.queue2.rxd_tail: 1020 dev.ix.0.queue2.rx_packets: 1021 dev.ix.0.queue2.rx_bytes: 99126 dev.ix.0.queue2.rx_copies: 891 dev.ix.0.queue2.lro_queued: 396 dev.ix.0.queue2.lro_flushed: 391 dev.ix.0.queue3.interrupt_rate: 71428 dev.ix.0.queue3.irqs: 25072 dev.ix.0.queue3.txd_head: 1465 dev.ix.0.queue3.txd_tail: 1465 dev.ix.0.queue3.tso_tx: 0 dev.ix.0.queue3.no_tx_dma_setup: 0 dev.ix.0.queue3.no_desc_avail: 0 dev.ix.0.queue3.tx_packets: 25726 dev.ix.0.queue3.rxd_head: 310 dev.ix.0.queue3.rxd_tail: 309 dev.ix.0.queue3.rx_packets: 310 dev.ix.0.queue3.rx_bytes: 36886 dev.ix.0.queue3.rx_copies: 150 dev.ix.0.queue3.lro_queued: 309 dev.ix.0.queue3.lro_flushed: 286 dev.ix.0.queue4.interrupt_rate: 500000 dev.ix.0.queue4.irqs: 21251 dev.ix.0.queue4.txd_head: 308 dev.ix.0.queue4.txd_tail: 308 dev.ix.0.queue4.tso_tx: 0 dev.ix.0.queue4.no_tx_dma_setup: 0 dev.ix.0.queue4.no_desc_avail: 0 dev.ix.0.queue4.tx_packets: 22090 dev.ix.0.queue4.rxd_head: 589 dev.ix.0.queue4.rxd_tail: 588 dev.ix.0.queue4.rx_packets: 589 dev.ix.0.queue4.rx_bytes: 57938 dev.ix.0.queue4.rx_copies: 558 dev.ix.0.queue4.lro_queued: 585 dev.ix.0.queue4.lro_flushed: 585 dev.ix.0.queue5.interrupt_rate: 41666 dev.ix.0.queue5.irqs: 20123 dev.ix.0.queue5.txd_head: 314 dev.ix.0.queue5.txd_tail: 314 dev.ix.0.queue5.tso_tx: 0 dev.ix.0.queue5.no_tx_dma_setup: 0 dev.ix.0.queue5.no_desc_avail: 0 dev.ix.0.queue5.tx_packets: 20618 dev.ix.0.queue5.rxd_head: 112 dev.ix.0.queue5.rxd_tail: 111 dev.ix.0.queue5.rx_packets: 112 dev.ix.0.queue5.rx_bytes: 10224 dev.ix.0.queue5.rx_copies: 84 dev.ix.0.queue5.lro_queued: 109 dev.ix.0.queue5.lro_flushed: 109 dev.ix.0.queue6.interrupt_rate: 71428 dev.ix.0.queue6.irqs: 18418 dev.ix.0.queue6.txd_head: 732 dev.ix.0.queue6.txd_tail: 732 dev.ix.0.queue6.tso_tx: 45 dev.ix.0.queue6.no_tx_dma_setup: 0 dev.ix.0.queue6.no_desc_avail: 0 dev.ix.0.queue6.tx_packets: 19137 dev.ix.0.queue6.rxd_head: 824 dev.ix.0.queue6.rxd_tail: 823 dev.ix.0.queue6.rx_packets: 824 dev.ix.0.queue6.rx_bytes: 92838 dev.ix.0.queue6.rx_copies: 583 dev.ix.0.queue6.lro_queued: 818 dev.ix.0.queue6.lro_flushed: 716 dev.ix.0.queue7.interrupt_rate: 62500 dev.ix.0.queue7.irqs: 17681 dev.ix.0.queue7.txd_head: 721 dev.ix.0.queue7.txd_tail: 721 dev.ix.0.queue7.tso_tx: 0 dev.ix.0.queue7.no_tx_dma_setup: 0 dev.ix.0.queue7.no_desc_avail: 0 dev.ix.0.queue7.tx_packets: 18067 dev.ix.0.queue7.rxd_head: 1407 dev.ix.0.queue7.rxd_tail: 1406 dev.ix.0.queue7.rx_packets: 1407 dev.ix.0.queue7.rx_bytes: 252631 dev.ix.0.queue7.rx_copies: 884 dev.ix.0.queue7.lro_queued: 1400 dev.ix.0.queue7.lro_flushed: 1390 dev.ix.0.mac_stats.crc_errs: 0 dev.ix.0.mac_stats.ill_errs: 0 dev.ix.0.mac_stats.byte_errs: 0 dev.ix.0.mac_stats.short_discards: 0 dev.ix.0.mac_stats.local_faults: 2 dev.ix.0.mac_stats.remote_faults: 3 dev.ix.0.mac_stats.rec_len_errs: 0 dev.ix.0.mac_stats.xon_txd: 0 dev.ix.0.mac_stats.xon_recvd: 0 dev.ix.0.mac_stats.xoff_txd: 0 dev.ix.0.mac_stats.xoff_recvd: 0 dev.ix.0.mac_stats.total_octets_rcvd: 1320732697 dev.ix.0.mac_stats.good_octets_rcvd: 1320713370 dev.ix.0.mac_stats.total_pkts_rcvd: 527648 dev.ix.0.mac_stats.good_pkts_rcvd: 527365 dev.ix.0.mac_stats.mcast_pkts_rcvd: 25 dev.ix.0.mac_stats.bcast_pkts_rcvd: 75 dev.ix.0.mac_stats.rx_frames_64: 128032 dev.ix.0.mac_stats.rx_frames_65_127: 100057 dev.ix.0.mac_stats.rx_frames_128_255: 115733 dev.ix.0.mac_stats.rx_frames_256_511: 1210 dev.ix.0.mac_stats.rx_frames_512_1023: 3075 dev.ix.0.mac_stats.rx_frames_1024_1522: 179258 dev.ix.0.mac_stats.recv_undersized: 0 dev.ix.0.mac_stats.recv_fragmented: 0 dev.ix.0.mac_stats.recv_oversized: 0 dev.ix.0.mac_stats.recv_jabberd: 0 dev.ix.0.mac_stats.management_pkts_rcvd: 0 dev.ix.0.mac_stats.management_pkts_drpd: 0 dev.ix.0.mac_stats.checksum_errs: 0 dev.ix.0.mac_stats.good_octets_txd: 2815129453 dev.ix.0.mac_stats.total_pkts_txd: 640355 dev.ix.0.mac_stats.good_pkts_txd: 640355 dev.ix.0.mac_stats.bcast_pkts_txd: 2 dev.ix.0.mac_stats.mcast_pkts_txd: 25 dev.ix.0.mac_stats.management_pkts_txd: 0 dev.ix.0.mac_stats.tx_frames_64: 39831 dev.ix.0.mac_stats.tx_frames_65_127: 166390 dev.ix.0.mac_stats.tx_frames_128_255: 72116 dev.ix.0.mac_stats.tx_frames_256_511: 2072 dev.ix.0.mac_stats.tx_frames_512_1023: 1339 dev.ix.0.mac_stats.tx_frames_1024_1522: 358607 and lastly, the default sysctl kern.ipc: kern.ipc.maxsockbuf: 2097152 kern.ipc.sockbuf_waste_factor: 8 kern.ipc.max_linkhdr: 16 kern.ipc.max_protohdr: 60 kern.ipc.max_hdr: 76 kern.ipc.max_datalen: 92 kern.ipc.maxmbufmem: 50194468864 kern.ipc.nmbclusters: 6127254 kern.ipc.nmbjumbop: 3063627 kern.ipc.nmbjumbo9: 2723223 kern.ipc.nmbjumbo16: 2042416 kern.ipc.nmbufs: 39214440 kern.ipc.maxpipekva: 1610002432 kern.ipc.pipekva: 147456 kern.ipc.pipefragretry: 0 kern.ipc.pipeallocfail: 0 kern.ipc.piperesizefail: 0 kern.ipc.piperesizeallowed: 1 kern.ipc.msgmax: 16384 kern.ipc.msgmni: 40 kern.ipc.msgmnb: 2048 kern.ipc.msgtql: 40 kern.ipc.msgssz: 8 kern.ipc.msgseg: 2048 kern.ipc.semmni: 50 kern.ipc.semmns: 340 kern.ipc.semmnu: 150 kern.ipc.semmsl: 340 kern.ipc.semopm: 100 kern.ipc.semume: 50 kern.ipc.semusz: 632 kern.ipc.semvmx: 32767 kern.ipc.semaem: 16384 kern.ipc.shmmax: 536870912 kern.ipc.shmmin: 1 kern.ipc.shmmni: 192 kern.ipc.shmseg: 128 kern.ipc.shmall: 131072 kern.ipc.shm_use_phys: 0 kern.ipc.shm_allow_removed: 0 kern.ipc.soacceptqueue: 128 kern.ipc.numopensockets: 79 kern.ipc.maxsockets: 3144540 kern.ipc.sendfile.readahead: 1
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAB2_NwAOmPtZjB03pdDiTK2OvQgqk-tYf83Jq4Ukt9jnZA8CNA>