From owner-freebsd-net@freebsd.org Tue Aug 27 18:50:11 2019 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 77933DC487 for ; Tue, 27 Aug 2019 18:50:11 +0000 (UTC) (envelope-from vit@otcnet.ru) Received: from mail.otcnet.ru (mail.otcnet.ru [194.190.78.3]) by mx1.freebsd.org (Postfix) with ESMTP id 46Hyc23yLMz4C6b for ; Tue, 27 Aug 2019 18:50:09 +0000 (UTC) (envelope-from vit@otcnet.ru) Received: from Victors-MacBook-Air-2.local (unknown [195.91.148.145]) by mail.otcnet.ru (Postfix) with ESMTPSA id 2961289DB5 for ; Tue, 27 Aug 2019 21:50:09 +0300 (MSK) Subject: Re: finding optimal ipfw strategy To: freebsd-net@freebsd.org References: <4ff39c8f-341c-5d72-1b26-6558c57bff8d@grosbein.net> From: Victor Gamov Organization: OTCnet Message-ID: Date: Tue, 27 Aug 2019 21:50:08 +0300 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:60.0) Gecko/20100101 Thunderbird/60.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 46Hyc23yLMz4C6b X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of vit@otcnet.ru designates 194.190.78.3 as permitted sender) smtp.mailfrom=vit@otcnet.ru X-Spamd-Result: default: False [-2.58 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-0.98)[-0.985,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+a:mail.otcnet.ru]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; HAS_ORG_HEADER(0.00)[]; DMARC_NA(0.00)[otcnet.ru]; NEURAL_HAM_LONG(-1.00)[-0.996,0]; NEURAL_HAM_SHORT(-0.41)[-0.406,0]; IP_SCORE(0.00)[country: RU(0.01)]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:50822, ipnet:194.190.78.0/24, country:RU]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Aug 2019 18:50:11 -0000 On 27/08/2019 21:03, Andrey V. Elsukov wrote: > On 26.08.2019 19:25, Victor Gamov wrote: >> More general question about my current config.  I have about 200Mbit >> input multicasts which bridged and filtered later (about 380 Mbit >> bridged if trafshow does not lie me :-) ) >> >> My FreeBSD box (12.0-STABLE r348449 GENERIC  amd64)  has one "Intel(R) >> Xeon(R) CPU E31270 @ 3.40GHz"  and 4-ports  "Intel(R) PRO/1000 >> PCI-Express Network Driver".  HT disabled and traffic mainly income via >> igb0 and out both via igb0 and igb2.  About 30 VLANs now active some at >> igb0 and some at igb2. >> >> >> And I have following `top` stat: >> ===== >> CPU 0:  0.0% user,  0.0% nice, 80.5% system,  0.0% interrupt, 19.5% idle >> CPU 1:  0.0% user,  0.0% nice, 34.1% system,  0.0% interrupt, 65.9% idle >> CPU 2:  0.0% user,  0.0% nice, 17.1% system,  0.0% interrupt, 82.9% idle >> CPU 3:  0.0% user,  0.0% nice, 46.3% system,  0.0% interrupt, 53.7% idle >> ===== > > This doesn't look like heavy ipfw load. Andrew, I have 0.0% interrupt but 80.5% system load. As this box hasn't any processes running (besides kernel + ntp + bsnmp) so I think this load produced by ipfw. Also I think this load source may be packets processing by bridge: get one packet, bridge it (copy/malloc?) into many interfaces, drop packets on unnecessary ifaces (free?) > E.g. this is top output from slightly loaded firewall (300Mbytes/s > ~500kpps): Yes, it's not a problem for 28 cores :-) I have 4 cores only and about 700Mbit in/out via bridge > last pid: 58184; load averages: 9.07, 8.98, 8.83 > up > 72+07:45:55 21:01:36 > 821 processes: 36 running, 680 sleeping, 105 waiting > CPU 0: 0.0% user, 0.0% nice, 0.0% system, 28.1% interrupt, 71.9% idle > CPU 27: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle > > # pmcstat -S instructions -Tw1 > PMC: [INSTR_RETIRED_ANY] Samples: 443074 (100.0%) , 0 unresolved > Key: q => exiting... > %SAMP IMAGE FUNCTION CALLERS > 39.2 kernel sched_idletd fork_exit > 10.9 ipfw.ko ipfw_chk ipfw_check_packet > 3.6 kernel cpu_search_lowest cpu_search_lowest > 2.8 kernel lock_delay _mtx_lock_spin_cookie > 2.5 kernel _rm_rlock in6_localip:1.3 pfil_run_hooks:0.6 > 2.2 kernel rn_match ta_lookup_radix:1.5 > fib6_lookup_nh_basic:0.6 > > As you can see, when ipfw produces high load, interrupt column is more > than system. -- С уважением, Гамов Виктор