From owner-freebsd-net@FreeBSD.ORG Fri Dec 27 11:16:45 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3B99D351 for ; Fri, 27 Dec 2013 11:16:45 +0000 (UTC) Received: from mail.tcm.by (mail.tcm.by [84.201.224.251]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 4A2ED16B5 for ; Fri, 27 Dec 2013 11:16:43 +0000 (UTC) Received: from skipped_antispam (localhost [127.0.0.1]) by localhost (Postfix) with ESMTP id 22F46FDB8 for ; Fri, 27 Dec 2013 14:16:41 +0300 (FET) Received: from mailhub (localhost [127.0.0.1]) by localhost (Postfix) with ESMTP id 5EB7DFDAF for ; Fri, 27 Dec 2013 14:16:40 +0300 (FET) Received: from dialup-dynamic-pool1-45.tcm.by (unknown [84.201.225.45]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.tcm.by (Postfix) with ESMTP id B8CE7FDA8; Fri, 27 Dec 2013 14:16:39 +0300 (FET) Date: Fri, 27 Dec 2013 14:16:38 +0300 From: "Denis V. Klimkov" Organization: Telecom Media Systems JLLC X-Priority: 3 (Normal) Message-ID: <27299961.20131227141638@tcm.by> To: "Alexander V. Chernikov" Subject: Re: ipfw verrevpath performance broken in 9.2 In-Reply-To: <52BD5598.9020100@FreeBSD.org> References: <21356442.20131227093416@tcm.by> <52BD5598.9020100@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Dec 2013 11:16:45 -0000 Hello Alexander, Friday, December 27, 2013, 1:25:28 PM, you wrote: >> Recently upgraded router system from 9.0-RELEASE to 9.2-STABLE and >> got 100% CPU utilisation on all cores with interrupts under the same >> load that had about 25-30% CPU utilisation before. Of course that lead AVC> Looks interesting. AVC> Are you sure all other configs/data load are the same? Yes, everything was the same. Later changed NIC from 4 igbs to 1 ix. AVC> I'm particularly interested in changes in: number of NIC queues, their AVC> bindings and firewall ruleset. igb0: port 0x3020-0x303f mem 0xc6b20000-0xc6b3ffff,0xc6b44000-0xc6b47fff irq 40 at device 0.0 on pci1 igb0: Using MSIX interrupts with 5 vectors igb0: Ethernet address: 00:15:17:b9:ef:dc igb0: Bound queue 0 to cpu 0 igb0: Bound queue 1 to cpu 1 igb0: Bound queue 2 to cpu 2 igb0: Bound queue 3 to cpu 3 igb1: port 0x3000-0x301f mem 0xc6b00000-0xc6b1ffff,0xc6b40000-0xc6b43fff irq 28 at device 0.1 on pci1 igb1: Using MSIX interrupts with 5 vectors igb1: Ethernet address: 00:15:17:b9:ef:dd igb1: Bound queue 0 to cpu 4 igb1: Bound queue 1 to cpu 5 igb1: Bound queue 2 to cpu 6 igb1: Bound queue 3 to cpu 7 pcib2: irq 24 at device 3.0 on pci0 pci2: on pcib2 pcib3: irq 26 at device 5.0 on pci0 pci3: on pcib3 igb2: port 0x2020-0x203f mem 0xc6420000-0xc643ffff,0xc6000000-0xc63fffff,0xc64c4000-0xc64c7fff irq 26 at device 0.0 on pci 3 igb2: Using MSIX interrupts with 5 vectors igb2: Ethernet address: 00:1b:21:4a:69:78 igb2: Bound queue 0 to cpu 8 igb2: Bound queue 1 to cpu 9 igb2: Bound queue 2 to cpu 10 igb2: Bound queue 3 to cpu 11 igb3: port 0x2000-0x201f mem 0xc6400000-0xc641ffff,0xc5c00000-0xc5ffffff,0xc64c0000-0xc64c3fff irq 25 at device 0.1 on pci 3 igb3: Using MSIX interrupts with 5 vectors igb3: Ethernet address: 00:1b:21:4a:69:79 igb3: Bound queue 0 to cpu 12 igb3: Bound queue 1 to cpu 13 igb3: Bound queue 2 to cpu 14 igb3: Bound queue 3 to cpu 15 09000 546827 20995102 deny ip from any to 224.0.0.0/8 09900 251418446 34849277439 fwd 127.0.0.1,3333 tcp from table(100) to not table(9) dst-port 80 09901 251226827 74150859375 allow tcp from any 80 to table(100) out 09999 324676485 22931487657 deny ip from not table(9) to table(100) 09999 93075888 5276322115 deny ip from table(100) to not table(9) 10000 234714177213 241730704799083 allow ip from table(5) to any 10005 245356169 18235355072 deny ip from any to any dst-port 135,137-139,445 out 10006 2929342953 182985124889 deny ip from table(104) to any 10020 688240709 620932403164 divert 8668 ip from any to 1.1.1.1 10400 682416642 620798165276 allow ip from any to any diverted 10770 73183544 9041870946 deny ip from table(2) to any out via vlan18 10772 11698 802274 deny ip from table(3) to any out via vlan4 10773 8807403 463870927 deny ip from any to table(2) out iptos reliability 10774 4923414 300617694 deny ip from any to table(3) out iptos reliability 10775 99485 4397077 deny ip from any to table(3) out iptos throughput 11010 3659429 430047150 deny ip from any to any not verrevpath in via vlan6 11020 719931 58619220 deny ip from any to any not verrevpath in via vlan7 11025 68141 5144481 deny ip from any to any not verrevpath in via vlan8 11030 202144 6785732 deny ip from any to any not verrevpath in via vlan9 11040 171291 56196945 deny ip from any to any not verrevpath in via vlan10 11045 291914032 39427773226 deny ip from any to any not verrevpath in via vlan11 11060 6102962 441745213 deny ip from any to any not verrevpath in via vlan15 11070 4832442 1259880158 deny ip from any to any not verrevpath in via vlan16 11080 814769 95745079 deny ip from any to any not verrevpath in via vlan17 11101 2901098 628552748 deny ip from any to any not verrevpath in via vlan26 11102 1264750 146468688 deny ip from any to any not verrevpath in via vlan27 11110 902441 294155831 deny ip from any to any not verrevpath in via vlan21 11120 628324 31060933 deny ip from any to any not verrevpath in via vlan23 11130 1381 83245 deny ip from any to any not verrevpath in via vlan24 11138 4258607 3389925416 deny ip from any to any not verrevpath in via vlan31 11150 56 2792 deny ip from any to any not verrevpath in via vlan40 15000 3363576 188412499 deny ip from not table(30) to table(31) out 19950 64832991 3461330324 deny tcp from table(25) to not table(8) dst-port 25 out 19960 693595 34424883 deny ip from table(101) to table(103) out 19970 466690 57539243 deny ip from not table(30) to me dst-port 161,162,21,3306 20000 35523656903 32569055261754 pipe tablearg ip from any to table(1) out iptos reliability 20010 36208900912 9635678183009 pipe tablearg ip from table(6) to any out via vlan18 20020 6963415930 5823875049163 pipe tablearg ip from any to table(10) out 20030 5370808609 1175572076679 pipe tablearg ip from table(11) to any out 60005 3749710 1625777707 deny udp from any to 2.2.2.100 dst-port 5060 60005 7940451 2910219814 deny udp from any to 2.2.2.1 dst-port 5060 60020 578206 71125954 divert 8668 ip from 192.168.0.0/16 to any out via vlan4 60020 120740 17363073 divert 8668 ip from 192.168.0.0/16 to any out via vlan5 60020 6485285 2421107818 divert 8668 ip from 192.168.0.0/16 to any out via vlan18 60020 22096 1876197 divert 8668 ip from 192.168.0.0/16 to any out via vlan11 60600 529456103 183816441399 allow ip from any to any diverted 62110 2482047796 207871928397 deny ip from not table(32) to any out via vlan18 62120 34184526 40243097237 allow ip from 3.3.3.0/24 to 3.3.3.0/24 via vlan4 62130 19323045 1282467423 deny ip from not table(32) to any out via vlan4 62140 21168902 1790816969 deny ip from any to not table(32) in via vlan4 64000 8160465887601 5338926261446363 allow ip from any to any 65000 1165747 214509370 allow ip from any to any 65535 5625 3645710 deny ip from any to any AVC> Can you share your traffic rate (e.g. netstat -i -w1), cpu info and NIC AVC> info? Now it's: # netstat -i -w1 input (Total) output packets errs idrops bytes packets errs bytes colls 312136 0 0 216478043 312375 0 216359751 0 311760 0 0 217559784 311654 0 217792531 0 295196 0 0 203318550 295319 0 211926680 0 300204 0 0 206880841 300219 0 206348483 0 297019 0 0 203171215 296930 0 207103301 0 308142 0 0 211553806 308294 0 207969407 0 320911 0 0 221584256 320955 0 218811245 0 CPU: Intel(R) Xeon(R) CPU E5520 @ 2.27GHz (2261.30-MHz 686-class CPU) AVC> What does system load (without verrevpath) looks like in comparison with AVC> 9.0 (in terms of CPU _and_ packets/sec) ? Sorry, cannot compare it. Old graphs are lost. AFAIR it was up to 30 LA in peak times when there was about 400+ kpss in and same out. I can try to add some rules with verrevpath now in 9.2 system. Without verrevpath rules top ISHP shows: last pid: 58440; load averages: 2.52, 2.52, 2.51 up 1+06:25:38 14:05:02 268 processes: 17 running, 177 sleeping, 74 waiting CPU 0: 0.0% user, 0.0% nice, 0.0% system, 28.2% interrupt, 71.8% idle CPU 1: 0.0% user, 0.0% nice, 0.0% system, 38.0% interrupt, 62.0% idle CPU 2: 0.4% user, 0.0% nice, 0.8% system, 29.8% interrupt, 69.0% idle CPU 3: 0.0% user, 0.0% nice, 0.4% system, 26.7% interrupt, 72.9% idle CPU 4: 0.0% user, 0.0% nice, 0.8% system, 32.5% interrupt, 66.7% idle CPU 5: 0.0% user, 0.0% nice, 0.8% system, 31.4% interrupt, 67.8% idle CPU 6: 0.0% user, 0.0% nice, 0.0% system, 30.2% interrupt, 69.8% idle CPU 7: 0.0% user, 0.0% nice, 0.0% system, 32.2% interrupt, 67.8% idle CPU 8: 0.0% user, 0.0% nice, 0.8% system, 0.0% interrupt, 99.2% idle CPU 9: 0.8% user, 0.0% nice, 0.0% system, 0.0% interrupt, 99.2% idle CPU 10: 0.4% user, 0.0% nice, 1.2% system, 0.0% interrupt, 98.4% idle CPU 11: 0.0% user, 0.0% nice, 0.0% system, 0.8% interrupt, 99.2% idle CPU 12: 0.4% user, 0.0% nice, 0.0% system, 0.8% interrupt, 98.8% idle CPU 13: 0.0% user, 0.0% nice, 0.4% system, 0.0% interrupt, 99.6% idle CPU 14: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 15: 0.0% user, 0.0% nice, 0.8% system, 0.0% interrupt, 99.2% idle netstat -iw 1 input (Total) output packets errs idrops bytes packets errs bytes colls 322k 0 0 219M 322k 0 220M 0 324k 0 0 224M 324k 0 222M 0 325k 0 0 227M 325k 0 227M 0 352k 0 0 247M 352k 0 242M 0 After adding verrevpath rules: last pid: 58471; load averages: 3.19, 2.82, 2.64 up 1+06:30:04 14:09:28 270 processes: 21 running, 179 sleeping, 70 waiting CPU 0: 0.0% user, 0.0% nice, 0.4% system, 51.4% interrupt, 48.2% idle CPU 1: 0.0% user, 0.0% nice, 0.4% system, 44.7% interrupt, 54.9% idle CPU 2: 0.0% user, 0.0% nice, 0.8% system, 37.6% interrupt, 61.6% idle CPU 3: 0.0% user, 0.0% nice, 0.0% system, 38.8% interrupt, 61.2% idle CPU 4: 0.4% user, 0.0% nice, 0.0% system, 38.8% interrupt, 60.8% idle CPU 5: 0.0% user, 0.0% nice, 0.4% system, 41.2% interrupt, 58.4% idle CPU 6: 0.4% user, 0.0% nice, 0.4% system, 43.9% interrupt, 55.3% idle CPU 7: 0.0% user, 0.0% nice, 0.0% system, 41.6% interrupt, 58.4% idle Looks like now this rules does not affect load such a way it was before. But now NICs configuration differs. There were ifconfig_lagg0="laggproto loadbalance laggport igb0 laggport igb1 laggport igb2 laggport igb3" and all vlans over lagg0. Now it is one ix0 without lagg and all vlans are over ix0. --- Denis V. Klimkov