From owner-freebsd-net@FreeBSD.ORG Wed Dec 31 10:25:00 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4753D54C for ; Wed, 31 Dec 2014 10:25:00 +0000 (UTC) Received: from mx.aknet.kg (mx.aknet.kg [212.112.96.8]) by mx1.freebsd.org (Postfix) with ESMTP id 9882B674E4 for ; Wed, 31 Dec 2014 10:24:59 +0000 (UTC) Received: from mx.aknet.kg (localhost.aknet.kg [127.0.0.1]) by mx.aknet.kg (Postfix) with ESMTP id B392C1CDF7; Wed, 31 Dec 2014 16:24:52 +0600 (KGT) Received: (from nobody@localhost) by mx.aknet.kg (8.13.8/8.13.8/Submit) id sBVAOqMF005995; Wed, 31 Dec 2014 16:24:52 +0600 (KGT) (envelope-from info@aknet.kg) X-Authentication-Warning: mx.aknet.kg: nobody set sender to info@aknet.kg using -f To: Subject: Netmap-Ipfw: eats 90-100% of CPU, is it normal behaviour =?UTF-8?Q?=3F?= X-PHP-Originating-Script: 501:main.inc MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Date: Wed, 31 Dec 2014 16:24:51 +0600 From: info@aknet.kg Message-ID: X-Sender: info@aknet.kg User-Agent: Roundcube Webmail/0.7.2 Cc: rizzo@iet.unipi.it X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 31 Dec 2014 10:25:00 -0000 Hello, All ! We tried to use netmap-ipfw in production (as filtering bridge) for traffic sanity and bandwidth limitation. And meet a problem. Will be explaned below. CPU: i5-4690 CPU @ 3.50GHz RAM: 8GB x 1800Mhz NET: Intel DA 520 (2 x 10Gbps) kipfw starts as: /usr/local/netmap-ipfw/kipfw netmap:ix0 netmap:ix1 ruleset: 00100 allow ip from 192.168.254.0/24 to 192.168.254.0/24 00200 allow ip from any to 192.168.0.0/16 - incoming (for customers) traffic goes without touching 00400 pipe 665 udp from 192.168.0.0/16 to any dst-port 6881 00500 pipe 666 tcp from 192.168.0.0/16 to any tcpflags syn 00600 deny tcp from table(25) to any dst-port 25 00700 deny tcp from 192.168.0.0/16 to table(26) dst-port 25 00750 allow ip from 192.168.0.0/16 to any - this rule we have to use (explaned below) 00800 pipe 10 ip from 192.168.0.0/16 to any - main rule for this bridge 65535 allow ip from any to any pipes: # BW for packets with SYN flag and UDP-6881 ${fw} pipe 665 config mask src-ip 0xffffffff bw 384Kbit/s ${fw} pipe 666 config mask src-ip 0xffffffff bw 64Kbit/s # Outgoing BW for each IP ${fw} pipe 10 config mask src-ip 0xffffffff bw 5120Kbit/s table 25 has about 100 IP's table 26 has about 15 sub-networks this bridge serves about 25K subscribers with IP's from network: 192.168.0.0/16 current traffic: netstat -bdh -w1 -I ix1 input ix1 output packets errs idrops bytes packets errs bytes colls drops 607K 0 0 753M 452K 0 88M 0 0 603K 0 0 750M 449K 0 87M 0 0 604K 0 0 751M 448K 0 88M 0 0 604K 0 0 747M 452K 0 92M 0 0 all traffic: netstat -bdh -w1 input (Total) output packets errs idrops bytes packets errs bytes colls drops 2M 0 0 1.6G 2M 0 1.6G 0 0 2M 0 0 1.6G 2M 0 1.6G 0 0 current CPU: CPU 0: 31.1% user, 0.0% nice, 56.1% system, 5.1% interrupt, 7.7% idle CPU 1: 0.0% user, 0.0% nice, 0.5% system, 8.2% interrupt, 91.3% idle CPU 2: 0.0% user, 0.0% nice, 0.0% system, 4.6% interrupt, 95.4% idle CPU 3: 0.0% user, 0.0% nice, 0.5% system, 7.1% interrupt, 92.3% idle THE Question: is it normal for kipfw to eat so much resoures ? 660 root 99 0 873M 325M CPU0 0 272:03 91.46% kipfw Also, the rule #750 I have to place into ruleset, cos without it kipfw begins to use all 100% 00750 allow ip from 192.168.0.0/16 to any 00800 pipe 10 ip from 192.168.0.0/16 to any - this rule is the main for using of this bridge, it assigns the same outgoing bandwidth for each of IP addresses - 5120Kbit/s (5Mbps) # BW for packets with SYN flag and UDP-6881 ${fw} pipe 665 config mask src-ip 0xffffffff bw 384Kbit/s ${fw} pipe 666 config mask src-ip 0xffffffff bw 64Kbit/s # Outgoing BW for each IP ${fw} pipe 10 config mask src-ip 0xffffffff bw 5120Kbit/s With working rule #800 after 30-50 mins kipfw begins to use 100% in top -PHS and incoming (for users) traffic downs from 750Mbytes/s (about 6Gbit/s) to 330Mbytes/s (2.6Gbit/s), delay increases from 65ms to 250ms and high percentage of drops. Is it real limit of using netmap-ipfw ? We can give any additional info if it will be usefull to expand limits of kipfw. With regards and happy New Year ! Azamat B. Umurzakov AkNet ISP