From owner-freebsd-net@FreeBSD.ORG Thu Jul 11 18:00:31 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id C94D410E for ; Thu, 11 Jul 2013 18:00:31 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-qc0-x235.google.com (mail-qc0-x235.google.com [IPv6:2607:f8b0:400d:c01::235]) by mx1.freebsd.org (Postfix) with ESMTP id 909A11D13 for ; Thu, 11 Jul 2013 18:00:31 +0000 (UTC) Received: by mail-qc0-f181.google.com with SMTP id u12so4451090qcx.26 for ; Thu, 11 Jul 2013 11:00:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=hTyoVqIJU78DxwIT24HDLeei1wRlu3Pe/0lgMeUHp5Q=; b=rqGDvTzhekfww0tDkH404r8BsVaMf76PwAT7VlJeSz8y/jDT7vUb3+sjzA6kY55RP5 tbku8lD4BBLZkX1LnvZgzyYTrNJm5XVxTkjL85AzQ1wY7GsLdVsQz1/5asqD6O9Q6I1g opbkXJSzHFVrEQH6vl56r6HDSIwFvjPh6tbMOmKRdK8JkmPUSLSwwk9AX75aVeYA9AaK cUcoEQR67Lj8syvuF4320jdvEGWgcBtpPshcyPUXCnZihhKYE4BH8q1MV+VM0Xbd3F66 B3cMYR44mWFpBn7igYnuEMUsEvyfd4queDtBV088bLJ6Qi0gY1J0LEptk2lo8OgVzszt O9Rg== MIME-Version: 1.0 X-Received: by 10.224.30.74 with SMTP id t10mr25640793qac.2.1373565631116; Thu, 11 Jul 2013 11:00:31 -0700 (PDT) Sender: asomers@gmail.com Received: by 10.49.37.226 with HTTP; Thu, 11 Jul 2013 11:00:31 -0700 (PDT) In-Reply-To: <16706.1373563943.8842093075575209984@ffe17.ukr.net> References: <16706.1373563943.8842093075575209984@ffe17.ukr.net> Date: Thu, 11 Jul 2013 12:00:31 -0600 X-Google-Sender-Auth: OoKzNi79l6rwJxkxINFwhDu0vok Message-ID: Subject: Re: FreeBSD router problems From: Alan Somers To: isp Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Jul 2013 18:00:31 -0000 How are you benchmarking it? Each TCP connection only uses one member of a lagg port. So if you want to see > 1 Gbps, you'll need to benchmark with multiple TCP connections. You may also need multiple systems; I don't know the full details of LACP. On Thu, Jul 11, 2013 at 11:32 AM, isp wrote: > > > > Hi! I have a problem with my FreeBSD router, I can't get more than 1 Gbps > throught it, but I have 2 Gbps LAGG on it. There are only 27 IPFW rules > (NAT+Shaping). IPoE only. > lagg0 (VLAN's + shaping) - two 'igb' adapters > lagg1 (NAT, tso if off) - two 'em' adapters > > I tried to switch off dummynet, but it doesn't helps. > > # uname -a > [code]FreeBSD router 9.1-RELEASE-p3 FreeBSD 9.1-RELEASE-p3 #0: Tue Apr 30 > 20:02:00 EEST 2013 root@south:/usr/obj/usr/src/sys/ROUTER amd64 > > # top -aSPHI > last pid: 91712; load averages: 2.18, 2.06, > 1.97 > up 20+22:28:36 17:40:22 > 120 processes: 7 running, 87 sleeping, 26 waiting > CPU 0: 0.0% user, 0.0% nice, 1.6% system, 38.6% interrupt, 59.8% idle > CPU 1: 0.0% user, 0.0% nice, 7.1% system, 37.0% interrupt, 55.9% idle > CPU 2: 0.0% user, 0.0% nice, 3.9% system, 38.6% interrupt, 57.5% idle > CPU 3: 0.0% user, 0.0% nice, 15.7% system, 26.8% interrupt, 57.5% idle > Mem: 59M Active, 1102M Inact, 942M Wired, 800M Buf, 5529M Free > Swap: 16G Total, 16G Free > > PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 12 root -72 - 0K 448K RUN 1 153:39 72.22% [intr{swi1: > netisr 0}] > 11 root 155 ki31 0K 64K RUN 1 494.2H 65.19% [idle{idle: > cpu1}] > 11 root 155 ki31 0K 64K CPU2 2 494.3H 64.65% [idle{idle: > cpu2}] > 11 root 155 ki31 0K 64K RUN 0 493.3H 63.38% [idle{idle: > cpu0}] > 11 root 155 ki31 0K 64K CPU3 3 496.4H 62.55% [idle{idle: > cpu3}] > 12 root -92 - 0K 448K WAIT 2 58:49 9.38% [intr{irq266: > igb0:que}] > 12 root -92 - 0K 448K WAIT 2 59:32 9.03% [intr{irq271: > igb1:que}] > 12 root -92 - 0K 448K CPU1 1 59:09 8.94% [intr{irq265: > igb0:que}] > 12 root -92 - 0K 448K WAIT 3 57:52 8.01% [intr{irq272: > igb1:que}] > 12 root -92 - 0K 448K WAIT 1 59:32 7.96% [intr{irq270: > igb1:que}] > 12 root -92 - 0K 448K WAIT 3 55:47 7.81% [intr{irq267: > igb0:que}] > 12 root -92 - 0K 448K WAIT 0 55:24 7.23% [intr{irq264: > igb0:que}] > 12 root -92 - 0K 448K WAIT 0 56:57 6.69% [intr{irq269: > igb1:que}] > 12 root -92 - 0K 448K WAIT 3 203:34 4.74% [intr{irq275: > em1:rx 0}] > 0 root -92 0 0K 336K - 2 427:03 2.64% > [kernel{dummynet}] > 0 root -92 0 0K 336K - 3 206:57 2.54% [kernel{em0 > que}] > 86278 root 20 0 33348K 8588K select 0 8:35 0.54% > /usr/local/sbin/snmpd -p /var/run/net_snmpd.pid -r > 12 root -92 - 0K 448K WAIT 2 7:56 0.20% [intr{irq276: > em1:tx 0}] > > # cat /etc/sysctl.conf > dev.igb.0.rx_processing_limit=3D4096 > dev.igb.1.rx_processing_limit=3D4096 > dev.em.0.rx_int_delay=3D200 > dev.em.0.tx_int_delay=3D200 > dev.em.0.rx_abs_int_delay=3D4000 > dev.em.0.tx_abs_int_delay=3D4000 > dev.em.0.rx_processing_limit=3D4096 > dev.em.1.rx_int_delay=3D200 > dev.em.1.tx_int_delay=3D200 > dev.em.1.rx_abs_int_delay=3D4000 > dev.em.1.tx_abs_int_delay=3D4000 > dev.em.1.rx_processing_limit=3D4096 > net.inet.ip.forwarding=3D1 > net.inet.ip.fastforwarding=3D1 > net.inet.tcp.blackhole=3D2 > net.inet.udp.blackhole=3D0 > net.inet.ip.redirect=3D0 > net.inet.tcp.delayed_ack=3D0 > net.inet.tcp.recvbuf_max=3D4194304 > net.inet.tcp.sendbuf_max=3D4194304 > net.inet.tcp.sack.enable=3D0 > net.inet.tcp.drop_synfin=3D1 > net.inet.tcp.nolocaltimewait=3D1 > net.inet.ip.ttl=3D255 > net.inet.ip.sourceroute=3D0 > net.inet.ip.accept_sourceroute=3D0 > net.inet.udp.recvspace=3D64080 > net.inet.ip.rtmaxcache=3D1024 > net.inet.ip.intr_queue_maxlen=3D5120 > kern.ipc.nmbclusters=3D824288 > kern.ipc.maxsockbuf=3D83886080 > kern.ipc.maxsockets=3D102400 > net.inet.tcp.recvspace=3D95536 > net.inet.tcp.sendspace=3D95536 > net.local.stream.recvspace=3D32768 > net.local.stream.sendspace=3D32768 > kern.ipc.somaxconn=3D32768 > net.inet.tcp.maxtcptw=3D65535 > net.inet.ip.fw.one_pass=3D1 > net.inet.ip.fw.dyn_max=3D65535 > net.inet.ip.fw.dyn_buckets=3D2048 > net.inet.ip.fw.dyn_syn_lifetime=3D10 > net.inet.ip.fw.dyn_ack_lifetime=3D120 > net.inet.ip.fw.verbose=3D0 > net.inet.ip.dummynet.io_fast=3D1 > net.inet.ip.dummynet.hash_size=3D65536 > net.inet.ip.dummynet.pipe_slot_limit=3D1000 > net.inet.icmp.icmplim=3D3000 > net.inet.icmp.drop_redirect=3D1 > net.inet.icmp.log_redirect=3D0 > net.inet.icmp.bmcastecho=3D0 > net.inet.icmp.maskrepl=3D0 > kern.random.sys.harvest.ethernet=3D0 > kern.random.sys.harvest.point_to_point=3D0 > kern.random.sys.harvest.interrupt=3D0 > net.inet.raw.maxdgram=3D16384 > net.inet.raw.recvspace=3D16384 > net.route.netisr_maxqlen=3D8192 > net.inet.ip.intr_queue_maxlen=3D10240 > net.isr.dispatch=3Ddeferred > > # cat /boot/loader.conf > loader_logo=3D"beastie" > autoboot_delay=3D3 > geom_mirror_load=3D"YES" > hw.igb.rxd=3D4096 > hw.igb.txd=3D4096 > hw.igb.rx_process_limit=3D4096 > hw.igb.max_interrupt_rate=3D32000 > hw.igb.num_queues=3D4 > hw.igb.fc_setting=3D0 > hw.igb.lro=3D0 > hw.em.rxd=3D4096 > hw.em.txd=3D4096 > hw.em.rx_process_limit=3D4096 > hw.em.fc_setting=3D0 > dev.em.0.rx_int_delay=3D200 > dev.em.0.tx_int_delay=3D200 > dev.em.0.rx_abs_int_delay=3D4000 > dev.em.0.tx_abs_int_delay=3D4000 > dev.em.1.rx_int_delay=3D200 > dev.em.1.tx_int_delay=3D200 > dev.em.1.rx_abs_int_delay=3D4000 > dev.em.1.tx_abs_int_delay=3D4000 > net.isr.maxthreads=3D4 > net.isr.bindthreads=3D0 > net.inet.tcp.tcbhashsize=3D32000 > net.link.ifqmaxlen=3D10240 > net.isr.defaultqlimit=3D8192 > > # vmstat -i > interrupt total rate > irq20: ehci1 4171628 2 > irq21: atapci0 1561194 0 > irq22: ehci0+ 2713150 1 > cpu0:timer 14622957598 8082 > irq264: igb0:que 0 515616328 284 > irq265: igb0:que 1 738456087 408 > irq266: igb0:que 2 711371660 393 > irq267: igb0:que 3 462738813 255 > irq268: igb0:link 3 0 > irq269: igb1:que 0 656044816 362 > irq270: igb1:que 1 546931002 302 > irq271: igb1:que 2 617173223 341 > irq272: igb1:que 3 644295672 356 > irq273: igb1:link 4 0 > irq274: em0 557400132 308 > irq275: em1:rx 0 424252744 234 > irq276: em1:tx 0 708469817 391 > irq277: em1:link 2 0 > cpu3:timer 678408141 374 > cpu1:timer 674674076 372 > cpu2:timer 621495291 343 > Total 23188731381 12816 > > # netstat -w1 > input (Total) output > packets errs idrops bytes packets errs bytes colls > 442k 0 0 304M 457k 0 393M 0 > 449k 0 0 308M 463k 0 395M 0 > 445k 0 0 304M 461k 0 393M 0 > 439k 0 0 303M 456k 0 393M 0 > 434k 0 0 297M 450k 0 387M 0 > 440k 0 0 301M 456k 0 392M 0 > 438k 0 0 300M 455k 0 391M 0 > > # ifconfig lagg0 (internal, 500 VLAN's) > lagg0: flags=3D8843 metric 0 mtu > 1500 > options=3D401bb > ether a0:36:9f:16:d0:9c > media: Ethernet autoselect > status: active > laggproto lacp lagghash l2,l3,l4 > laggport: igb1 flags=3D1c > laggport: igb0 flags=3D1c > > # ifconfig lagg1 - (external, NAT) > lagg1: flags=3D8843 metric 0 mtu > 1500 > options=3D4209b > ether 00:1e:67:59:ea:89 > inet =D0=A5=D0=A5=D0=A5.=D0=A5=D0=A5=D0=A5.=D0=A5=D0=A5=D0=A5.14 netmask = 0xffffffe0 broadcast > =D0=A5=D0=A5=D0=A5.=D0=A5=D0=A5=D0=A5.=D0=A5=D0=A5=D0=A5.31 > inet =D0=A5=D0=A5=D0=A5.=D0=A5=D0=A5=D0=A5.=D0=A5=D0=A5=D0=A5.70 netmask = 0xffffffff broadcast > =D0=A5=D0=A5=D0=A5.=D0=A5=D0=A5=D0=A5.=D0=A5=D0=A5=D0=A5.70 > inet =D0=A5=D0=A5=D0=A5.=D0=A5=D0=A5=D0=A5.=D0=A5=D0=A5=D0=A5.71 netmask = 0xffffffff broadcast > =D0=A5=D0=A5=D0=A5.=D0=A5=D0=A5=D0=A5.=D0=A5=D0=A5=D0=A5.71 > inet =D0=A5=D0=A5=D0=A5.=D0=A5=D0=A5=D0=A5.=D0=A5=D0=A5=D0=A5.72 netmask = 0xffffffff broadcast > =D0=A5=D0=A5=D0=A5.=D0=A5=D0=A5=D0=A5.=D0=A5=D0=A5=D0=A5.72 > inet =D0=A5=D0=A5=D0=A5.=D0=A5=D0=A5=D0=A5.=D0=A5=D0=A5=D0=A5.73 netmask = 0xffffffff broadcast > =D0=A5=D0=A5=D0=A5.=D0=A5=D0=A5=D0=A5.=D0=A5=D0=A5=D0=A5.73 > inet =D0=A5=D0=A5=D0=A5.=D0=A5=D0=A5=D0=A5.=D0=A5=D0=A5=D0=A5.74 netmask = 0xffffffff broadcast > =D0=A5=D0=A5=D0=A5.=D0=A5=D0=A5=D0=A5.=D0=A5=D0=A5=D0=A5.74 > inet =D0=A5=D0=A5=D0=A5.=D0=A5=D0=A5=D0=A5.=D0=A5=D0=A5=D0=A5.75 netmask = 0xffffffff broadcast > =D0=A5=D0=A5=D0=A5.=D0=A5=D0=A5=D0=A5.=D0=A5=D0=A5=D0=A5.75 > inet =D0=A5=D0=A5=D0=A5.=D0=A5=D0=A5=D0=A5.=D0=A5=D0=A5=D0=A5.76 netmask = 0xffffffff broadcast > =D0=A5=D0=A5=D0=A5.=D0=A5=D0=A5=D0=A5.=D0=A5=D0=A5=D0=A5.76 > inet =D0=A5=D0=A5=D0=A5.=D0=A5=D0=A5=D0=A5.=D0=A5=D0=A5=D0=A5.77 netmask = 0xffffffff broadcast > =D0=A5=D0=A5=D0=A5.=D0=A5=D0=A5=D0=A5.=D0=A5=D0=A5=D0=A5.77 > inet =D0=A5=D0=A5=D0=A5.=D0=A5=D0=A5=D0=A5.=D0=A5=D0=A5=D0=A5.78 netmask = 0xffffffff broadcast > =D0=A5=D0=A5=D0=A5.=D0=A5=D0=A5=D0=A5.=D0=A5=D0=A5=D0=A5.78 > inet =D0=A5=D0=A5=D0=A5.=D0=A5=D0=A5=D0=A5.=D0=A5=D0=A5=D0=A5.79 netmask = 0xffffffff broadcast > =D0=A5=D0=A5=D0=A5.=D0=A5=D0=A5=D0=A5.=D0=A5=D0=A5=D0=A5.79 > inet =D0=A5=D0=A5=D0=A5.=D0=A5=D0=A5=D0=A5.=D0=A5=D0=A5=D0=A5.33 netmask = 0xfffffff0 broadcast > =D0=A5=D0=A5=D0=A5.=D0=A5=D0=A5=D0=A5.=D0=A5=D0=A5=D0=A5.47 > media: Ethernet autoselect > status: active > laggproto lacp lagghash l2,l3,l4 > laggport: em1 flags=3D1c > laggport: em0 flags=3D1c > > # netstat -w1 -I em0 > input (em0) output > packets errs idrops bytes packets errs bytes colls > 101k 0 0 111M 36k 0 13M 0 > 101k 0 0 112M 36k 0 13M 0 > 100k 0 0 112M 37k 0 13M 0 > > # netstat -w1 -I em1 > [code] input (em1) output > packets errs idrops bytes packets errs bytes colls > 100k 0 0 111M 37k 0 9.1M 0 > 102k 0 0 113M 39k 0 10M 0 > 91k 0 0 101M 38k 0 9.7M 0 > > # netstat -w1 -I igb0 > input (igb0) output > packets errs idrops bytes packets errs bytes colls > 39k 0 0 9.1M 51k 0 57M 0 > 38k 0 0 9.1M 49k 0 54M 0 > 39k 0 0 9.4M 51k 0 56M 0 > > # netstat -w1 -I igb1 > input (igb1) output > packets errs idrops bytes packets errs bytes colls > 36k 0 0 14M 48k 0 56M 0 > 35k 0 0 14M 50k 0 59M 0 > 34k 0 0 13M 48k 0 57M 0 > > # netstat -w1 -I lagg0 > input (lagg0) output > packets errs idrops bytes packets errs bytes colls > 75k 0 0 23M 98k 0 113M 0 > 73k 0 0 21M 98k 0 113M 0 > 73k 0 0 23M 98k 0 112M 0 > > # netstat -w1 -I lagg1 > input (lagg1) output > packets errs idrops bytes packets errs bytes colls > 100k 0 0 112M 74k 0 24M 0 > 101k 0 0 113M 73k 0 24M 0 > 102k 0 0 114M 74k 0 24M 0 > >> > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"