From owner-freebsd-questions@FreeBSD.ORG Tue Nov 3 20:41:50 2009 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D8AFD106566B for ; Tue, 3 Nov 2009 20:41:50 +0000 (UTC) (envelope-from cosmic17@yandex.ru) Received: from forward9.yandex.ru (forward9.yandex.ru [77.88.61.48]) by mx1.freebsd.org (Postfix) with ESMTP id 27AE38FC14 for ; Tue, 3 Nov 2009 20:41:50 +0000 (UTC) Received: from webmail77.yandex.ru (webmail77.yandex.ru [77.88.60.181]) by forward9.yandex.ru (Yandex) with ESMTP id 4F0F74682C0 for ; Tue, 3 Nov 2009 23:27:05 +0300 (MSK) Received: from localhost (localhost [127.0.0.1]) by webmail77.yandex.ru (Yandex) with ESMTP id 2E4684301EE for ; Tue, 3 Nov 2009 23:27:05 +0300 (MSK) X-Yandex-Spam: 1 X-Yandex-Front: webmail77 X-Yandex-TimeMark: 1257280025 Received: from 34.87.vltele.com (34.87.vltele.com [79.174.34.87]) by mail.yandex.ru with HTTP; Tue, 03 Nov 2009 23:27:04 +0300 From: =?koi8-r?B?5M3VyMEg7snLz8zByg==?= To: freebsd-questions@freebsd.org MIME-Version: 1.0 Message-Id: <4631257280024@webmail77.yandex.ru> Date: Tue, 03 Nov 2009 23:27:04 +0300 X-Mailer: Yamail [ http://yandex.ru ] 5.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=koi8-r Subject: Packets delays X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Nov 2009 20:41:50 -0000 Hello! We have a problem... Dlink 3426 is core of our network. DGS 3612 - routers, which are connected to core (users). Core is connected to bridge. There are three interfaces igb0, igb1, igb2 on bridge: igb0 - local network igb1 - border1 igb2 - border2 Bridge is are shaper (ipfw). border1 and border2 are connected to bridge. Border1: FreeBSD 7.0-STABLE-200807 i386 Border2: FreeBSD 7.2-STABLE-200906 i386 Bridge: FreeBSD 7.2-STABLE-200906 amd64 Traffic is devided between two borders. We have not got any problems with border2. But on border1 we have: ping through border1 from users and from bridge ping ya.ru PING ya.ru (93.158.134.8): 56 data bytes 64 bytes from 93.158.134.8: icmp_seq=0 ttl=58 time=95.083 ms 64 bytes from 93.158.134.8: icmp_seq=1 ttl=58 time=102.876 ms 64 bytes from 93.158.134.8: icmp_seq=2 ttl=58 time=113.630 ms 64 bytes from 93.158.134.8: icmp_seq=3 ttl=58 time=90.352 ms 64 bytes from 93.158.134.8: icmp_seq=4 ttl=58 time=127.349 ms 64 bytes from 93.158.134.8: icmp_seq=5 ttl=58 time=115.942 ms 64 bytes from 93.158.134.8: icmp_seq=6 ttl=58 time=103.753 ms 64 bytes from 93.158.134.8: icmp_seq=7 ttl=58 time=124.507 ms But in border1 we have: ping ya.ru PING ya.ru (213.180.204.8): 56 data bytes 64 bytes from 213.180.204.8: icmp_seq=0 ttl=61 time=3.130 ms 64 bytes from 213.180.204.8: icmp_seq=1 ttl=61 time=2.909 ms 64 bytes from 213.180.204.8: icmp_seq=2 ttl=61 time=3.791 ms 64 bytes from 213.180.204.8: icmp_seq=3 ttl=61 time=2.953 ms 64 bytes from 213.180.204.8: icmp_seq=4 ttl=61 time=3.298 ms 64 bytes from 213.180.204.8: icmp_seq=5 ttl=61 time=3.096 ms bridge: 1). netstat -w1d -I igb0 input (igb0) output packets errs bytes packets errs bytes colls 32442 0 19375201 29681 0 23220291 0 31894 0 19334182 29124 0 23043942 0 31566 0 18885017 28558 0 22390016 0 31810 0 19200993 28768 0 22270687 0 31879 0 19347245 29145 0 22795440 0 31697 0 18997706 29274 0 22651927 0 32042 0 18963695 29634 0 23508103 0 30674 0 18123997 28432 0 22890282 0 31654 0 18519433 28860 0 22918799 0 31961 0 19145696 29418 0 23238031 0 32056 0 19053994 29723 0 23775215 0 32367 0 18952231 29771 0 23774054 0 2). netstat -w1d -I igb1 input (igb1) output packets errs bytes packets errs bytes colls 21378 0 19903174 21410 0 9866489 0 21444 0 19948761 21752 0 10266869 0 21435 0 19972984 21825 0 10289265 0 21318 0 19748715 21431 0 10238696 0 21606 0 19855988 21467 0 10483023 0 21825 0 19956737 21750 0 10452641 0 21665 0 19805016 21980 0 10711357 0 20937 0 18983521 21357 0 10506764 0 21241 0 19280484 21992 0 10713008 0 21469 0 19652542 21667 0 10391424 0 21235 0 19538130 21524 0 10757572 0 20779 0 18963762 21003 0 10525245 0 20853 0 18854491 21312 0 10559890 0 Those commands on border1: 1). internal netstat -w1d -I em1 input (em1) output packets errs bytes packets errs bytes colls 9862 0 8286463 8226 0 4172860 0 10060 0 8603460 7209 0 3184874 0 9290 0 7781648 7709 0 3672720 0 9720 0 8311391 7208 0 3545028 0 10256 0 8669129 8232 0 4678531 0 9509 0 8140150 7715 0 3878637 0 9554 0 8081766 7199 0 3512477 0 9426 0 8046217 7721 0 3867273 0 9795 0 8177803 7721 0 3729722 0 9637 0 8297993 7199 0 3766607 0 9671 0 8275210 7202 0 3510855 0 9451 0 8306122 7196 0 3535604 0 9324 0 8009044 7202 0 3092661 0 9715 0 8265237 7364 0 3575433 0 9908 0 8215978 7499 0 3911766 0 9776 0 8270432 8086 0 4178572 0 2). external netstat -w1d -I em0 input (em0) output packets errs bytes packets errs bytes colls 8253 0 4640677 9803 0 8495567 0 7897 0 4878726 10041 0 7322200 0 8368 0 4776220 9832 0 8565609 0 7650 0 4285985 9283 0 7717595 0 7891 0 4352473 9797 0 8134889 0 8606 0 5018218 9657 0 8289548 0 8460 0 4720658 10207 0 8397786 0 7543 0 4085187 9277 0 8059337 0 9376 0 6040976 11500 0 8860064 0 8181 0 4552212 10295 0 8582838 0 8774 0 5079266 10062 0 8494193 0 8193 0 4674008 10295 0 8520010 0 10170 0 7212068 10813 0 8596957 0 8499 0 4936674 10828 0 9090783 0 8416 0 4409607 10297 0 9050824 0 7613 0 3960242 9280 0 8231506 0 A few month ago border1 in this configuration worked succsessfully.. On border1 there is yandex driver for em. loader.conf: hw.em.rxd="4096" hw.em.txd="4096" /etc/sysctl.conf ΞΑ border1: net.inet.tcp.recvspace=262144 net.inet.tcp.sendspace=62144 net.inet.udp.recvspace=1048576 kern.ipc.maxsockbuf=4194304 kern.ipc.nmbclusters=262144 net.inet.ip.portrange.first=5700 kern.ipc.somaxconn=65535 net.inet.tcp.blackhole=2 net.inet.udp.blackhole=1 net.inet.icmp.icmplim=30 net.inet.ip.intr_queue_maxlen=4096 net.inet.tcp.delayed_ack=0 net.inet.tcp.delacktime=10 net.inet.tcp.newreno=0 net.inet.tcp.msl=2500 net.inet.ip.rtmaxcache=1024 net.inet.raw.recvspace=65536 net.inet.ip.dummynet.hash_size=65536 net.inet.ip.fw.dyn_ack_lifetime=60 net.inet.ip.fw.dyn_syn_lifetime=10 net.inet.ip.fw.dyn_fin_lifetime=10 net.inet.ip.fw.dyn_max=16192 net.inet.ip.fastforwarding=1 net.isr.direct=1 net.inet.icmp.drop_redirect=1 dev.em.0.rx_kthreads=4 dev.em.1.rx_kthreads=4 dev.em.2.rx_kthreads=4 I have made the same options in sysctl on border1 like on border2: kern.ipc.nmbclusters=262144 net.inet.icmp.icmplim=200 I have added a new option net.inet.tcp.sendbuf_max=16777216 net.inet.tcp.recvbuf_max=16777216 net.bpf.bufsize=4194304 net.bpf.maxbufsize=8388608 But have no ane good results. dmesg: Border2 external: em0: Excessive collisions = 0 em0: Sequence errors = 0 em0: Defer count = 0 em0: Missed Packets = 0 em0: Receive No Buffers = 0 em0: Receive Length Errors = 0 em0: Receive errors = 0 em0: Crc errors = 0 em0: Alignment errors = 0 em0: Collision/Carrier extension errors = 0 em0: RX overruns = 0 em0: watchdog timeouts = 0 em0: RX MSIX IRQ = 0 TX MSIX IRQ = 0 LINK MSIX IRQ = 0 em0: XON Rcvd = 0 em0: XON Xmtd = 0 em0: XOFF Rcvd = 0 em0: XOFF Xmtd = 0 em0: Good Packets Rcvd = 5018002561 em0: Good Packets Xmtd = 5133609771 em0: TSO Contexts Xmtd = 0 em0: TSO Contexts Failed = 0 Border1 internal: em0: Excessive collisions = 0 em0: Sequence errors = 0 em0: Defer count = 0 em0: Missed Packets = 9975 em0: Receive No Buffers = 7099 em0: Receive Length Errors = 0 em0: Receive errors = 0 em0: Crc errors = 0 em0: Alignment errors = 0 em0: Collision/Carrier extension errors = 0 em0: RX overruns = 1 em0: watchdog timeouts = 0 em0: RX MSIX IRQ = 0 TX MSIX IRQ = 0 LINK MSIX IRQ = 0 em0: XON Rcvd = 0 em0: XON Xmtd = 0 em0: XOFF Rcvd = 0 em0: XOFF Xmtd = 0 em0: Good Packets Rcvd = 2639750787 em0: Good Packets Xmtd = 2455117624 em0: TSO Contexts Xmtd = 18 em0: TSO Contexts Failed = 0 Border2 internal: em1: Excessive collisions = 0 em1: Sequence errors = 0 em1: Defer count = 0 em1: Missed Packets = 0 em1: Receive No Buffers = 0 em1: Receive Length Errors = 0 em1: Receive errors = 0 em1: Crc errors = 0 em1: Alignment errors = 0 em1: Collision/Carrier extension errors = 0 em1: RX overruns = 0 em1: watchdog timeouts = 0 em1: RX MSIX IRQ = 0 TX MSIX IRQ = 0 LINK MSIX IRQ = 0 em1: XON Rcvd = 0 em1: XON Xmtd = 0 em1: XOFF Rcvd = 0 em1: XOFF Xmtd = 0 em1: Good Packets Rcvd = 5189211824 em1: Good Packets Xmtd = 4995412107 em1: TSO Contexts Xmtd = 16447 em1: TSO Contexts Failed = 0 Border1 internal: em1: Excessive collisions = 0 em1: Sequence errors = 0 em1: Defer count = 0 em1: Missed Packets = 0 em1: Receive No Buffers = 0 em1: Receive Length Errors = 0 em1: Receive errors = 0 em1: Crc errors = 0 em1: Alignment errors = 0 em1: Collision/Carrier extension errors = 0 em1: RX overruns = 0 em1: watchdog timeouts = 0 em1: RX MSIX IRQ = 0 TX MSIX IRQ = 0 LINK MSIX IRQ = 0 em1: XON Rcvd = 0 em1: XON Xmtd = 0 em1: XOFF Rcvd = 0 em1: XOFF Xmtd = 0 em1: Good Packets Rcvd = 2469872680 em1: Good Packets Xmtd = 2578357516 em1: TSO Contexts Xmtd = 14174 em1: TSO Contexts Failed = 0 Bridge: internal igb0: Adapter hardware address = 0xffffff00012f0528 igb0: CTRL = 0xc00241 RCTL = 0x801a igb0: Packet buffer = Tx=0k Rx=0k igb0: Flow control watermarks high = 63488 low = 61988 igb0: Queue(0) tdh = 154, tdt = 154 igb0: no descriptors avail event = 0 igb0: TX(0) MSIX IRQ Handled = 26582786007 igb0: TX(0) Packets sent = 69466117227 igb0: Queue(1) tdh = 0, tdt = 0 igb0: no descriptors avail event = 0 igb0: TX(1) MSIX IRQ Handled = 0 igb0: TX(1) Packets sent = 0 igb0: Queue(2) tdh = 0, tdt = 0 igb0: no descriptors avail event = 0 igb0: TX(2) MSIX IRQ Handled = 0 igb0: TX(2) Packets sent = 0 igb0: Queue(3) tdh = 0, tdt = 0 igb0: no descriptors avail event = 0 igb0: TX(3) MSIX IRQ Handled = 0 igb0: TX(3) Packets sent = 0 igb0: Queue(0) rdh = 186, rdt = 185 igb0: RX(0) Packets received = 18340736258 igb0: RX(0) Split Packets = 0 igb0: RX(0) Byte count = 12477314664354 igb0: RX(0) MSIX IRQ Handled = 11343806385 igb0: RX(0) LRO Queued= 0 igb0: RX(0) LRO Flushed= 0 igb0: Queue(1) rdh = 150, rdt = 149 igb0: RX(1) Packets received = 18360157016 igb0: RX(1) Split Packets = 0 igb0: RX(1) Byte count = 12474408387147 igb0: RX(1) MSIX IRQ Handled = 11362784395 igb0: RX(1) LRO Queued= 0 igb0: RX(1) LRO Flushed= 0 igb0: Queue(2) rdh = 142, rdt = 140 igb0: RX(2) Packets received = 18122235444 igb0: RX(2) Split Packets = 0 igb0: RX(2) Byte count = 12331139081653 igb0: RX(2) MSIX IRQ Handled = 11268633644 igb0: RX(2) LRO Queued= 0 igb0: RX(2) LRO Flushed= 0 igb0: Queue(3) rdh = 95, rdt = 94 igb0: RX(3) Packets received = 18379433903 igb0: RX(3) Split Packets = 0 igb0: RX(3) Byte count = 12407828772109 igb0: RX(3) MSIX IRQ Handled = 11387935869 igb0: RX(3) LRO Queued= 0 igb0: RX(3) LRO Flushed= 0 igb0: LINK MSIX IRQ Handled = 5 igb0: Mbuf defrag failed = 0 igb0: Std mbuf header failed = 0 igb0: Std mbuf packet failed = 0 igb0: Driver dropped packets = 0 igb0: Driver tx dma failure in xmit = 0 border1 igb1: Adapter hardware address = 0xffffff000451a528 igb1: CTRL = 0x18c00241 RCTL = 0x801a igb1: Packet buffer = Tx=0k Rx=0k igb1: Flow control watermarks high = 63488 low = 61988 igb1: Queue(0) tdh = 150, tdt = 151 igb1: no descriptors avail event = 1 igb1: TX(0) MSIX IRQ Handled = 9053729033 igb1: TX(0) Packets sent = 15117561092 igb1: Queue(1) tdh = 0, tdt = 0 igb1: no descriptors avail event = 0 igb1: TX(1) MSIX IRQ Handled = 0 igb1: TX(1) Packets sent = 0 igb1: Queue(2) tdh = 0, tdt = 0 igb1: no descriptors avail event = 0 igb1: TX(2) MSIX IRQ Handled = 0 igb1: TX(2) Packets sent = 0 igb1: Queue(3) tdh = 0, tdt = 0 igb1: no descriptors avail event = 0 igb1: TX(3) MSIX IRQ Handled = 0 igb1: TX(3) Packets sent = 0 igb1: Queue(0) rdh = 253, rdt = 251 igb1: RX(0) Packets received = 3887801598 igb1: RX(0) Split Packets = 0 igb1: RX(0) Byte count = 3419307814950 igb1: RX(0) MSIX IRQ Handled = 2880414195 igb1: RX(0) LRO Queued= 0 igb1: RX(0) LRO Flushed= 0 igb1: Queue(1) rdh = 119, rdt = 118 igb1: RX(1) Packets received = 3908297847 igb1: RX(1) Split Packets = 0 igb1: RX(1) Byte count = 3460826092933 igb1: RX(1) MSIX IRQ Handled = 2888114773 igb1: RX(1) LRO Queued= 0 igb1: RX(1) LRO Flushed= 0 igb1: Queue(2) rdh = 152, rdt = 151 igb1: RX(2) Packets received = 3922202264 igb1: RX(2) Split Packets = 0 igb1: RX(2) Byte count = 3503326896463 igb1: RX(2) MSIX IRQ Handled = 2886529770 igb1: RX(2) LRO Queued= 0 igb1: RX(2) LRO Flushed= 0 igb1: Queue(3) rdh = 92, rdt = 91 igb1: RX(3) Packets received = 3887004765 igb1: RX(3) Split Packets = 0 igb1: RX(3) Byte count = 3441233186193 igb1: RX(3) MSIX IRQ Handled = 2879712428 igb1: RX(3) LRO Queued= 0 igb1: RX(3) LRO Flushed= 0 igb1: LINK MSIX IRQ Handled = 10 igb1: Mbuf defrag failed = 0 igb1: Std mbuf header failed = 0 igb1: Std mbuf packet failed = 0 igb1: Driver dropped packets = 0 igb1: Driver tx dma failure in xmit = 0 Border2: igb2: Adapter hardware address = 0xffffff0004575528 igb2: CTRL = 0x18c00241 RCTL = 0x801a igb2: Packet buffer = Tx=0k Rx=0k igb2: Flow control watermarks high = 63488 low = 61988 igb2: Queue(0) tdh = 74, tdt = 74 igb2: no descriptors avail event = 3117 igb2: TX(0) MSIX IRQ Handled = 22305812305 igb2: TX(0) Packets sent = 57738555533 igb2: Queue(1) tdh = 0, tdt = 0 igb2: no descriptors avail event = 0 igb2: TX(1) MSIX IRQ Handled = 0 igb2: TX(1) Packets sent = 0 igb2: Queue(2) tdh = 0, tdt = 0 igb2: no descriptors avail event = 0 igb2: TX(2) MSIX IRQ Handled = 0 igb2: TX(2) Packets sent = 0 igb2: Queue(3) tdh = 0, tdt = 0 igb2: no descriptors avail event = 0 igb2: TX(3) MSIX IRQ Handled = 0 igb2: TX(3) Packets sent = 0 igb2: Queue(0) rdh = 246, rdt = 245 igb2: RX(0) Packets received = 13727912438 igb2: RX(0) Split Packets = 0 igb2: RX(0) Byte count = 10677902707967 igb2: RX(0) MSIX IRQ Handled = 8730210616 igb2: RX(0) LRO Queued= 0 igb2: RX(0) LRO Flushed= 0 igb2: Queue(1) rdh = 51, rdt = 50 igb2: RX(1) Packets received = 13676394291 igb2: RX(1) Split Packets = 0 igb2: RX(1) Byte count = 10622417591526 igb2: RX(1) MSIX IRQ Handled = 8706454649 igb2: RX(1) LRO Queued= 0 igb2: RX(1) LRO Flushed= 0 igb2: Queue(2) rdh = 194, rdt = 193 igb2: RX(2) Packets received = 13687518658 igb2: RX(2) Split Packets = 0 igb2: RX(2) Byte count = 10608204213317 igb2: RX(2) MSIX IRQ Handled = 8716002420 igb2: RX(2) LRO Queued= 0 igb2: RX(2) LRO Flushed= 0 igb2: Queue(3) rdh = 131, rdt = 130 igb2: RX(3) Packets received = 13660077699 igb2: RX(3) Split Packets = 0 igb2: RX(3) Byte count = 10588060857368 igb2: RX(3) MSIX IRQ Handled = 8692141405 igb2: RX(3) LRO Queued= 0 igb2: RX(3) LRO Flushed= 0 igb2: LINK MSIX IRQ Handled = 32 igb2: Mbuf defrag failed = 0 igb2: Std mbuf header failed = 0 igb2: Std mbuf packet failed = 0 igb2: Driver dropped packets = 0 We have changed the cable between bridge and borfer1...but haven`t got any results