From owner-freebsd-pf@freebsd.org Sun Dec 11 16:22:40 2016 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 88B14C7106C for ; Sun, 11 Dec 2016 16:22:40 +0000 (UTC) (envelope-from cgodspd@gmail.com) Received: from mail-wm0-x22f.google.com (mail-wm0-x22f.google.com [IPv6:2a00:1450:400c:c09::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 198B1B63 for ; Sun, 11 Dec 2016 16:22:40 +0000 (UTC) (envelope-from cgodspd@gmail.com) Received: by mail-wm0-x22f.google.com with SMTP id u144so7104701wmu.1 for ; Sun, 11 Dec 2016 08:22:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to; bh=2VZUrsUTJOgD2OtJPngdV7ip8g1Z6mKkWzt5c6KjAFw=; b=bRmra0Uoj7WnPtZ+tHYoQTqfPtpfOUzn6xZbArlQfekUjLBg6l4aGwZ7FhuSyncObM UfNDDWIQtLiApR3bMuNVKx2/Gc+tbCXs5EFePp9gzMVT5TYlyDqcZ70f3y7i6CrFWAvu WKGSl7yU0sDVAd26kicQToQZfp1A4Lyx7Y4/eXDEdLkMYH4xAjBByp3TpKW0kysb2Xq6 dvNTWPX4N3EI7/KQ4xvz3vbhZqYyl+iU+AQYBf2hX0DiZrynVPobFHPAlFeHVqPPvlEc etaGMMBTFI3Tw1wEmIUFL52jaiKPhscRONs5/OwpcYp9ci8v+KfqGRhE1vEgWWVzxE/L vMzw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=2VZUrsUTJOgD2OtJPngdV7ip8g1Z6mKkWzt5c6KjAFw=; b=jTr/Csrpo29KGf8fkG3AEswWze/K7XmumntOl4gvcDKmWJJ4QjPqz6+xvFIYc3n61J RxRci/3ih4ssTLBfH3A1DtmKDmlFwn/r2fIOw9oZaR/XQv34JZbBR2C7BfAunzpdTZKH 92XK6dUZjRfQvsRN5BN+PpQacWCY6CgN5YcXoc9veM8BRNPYibBLOIBhohOl4FzoKur8 e86HRyJYqGm1kIm1HO0PntPtr9itf72lT8om4+ynFJoNCB6r8XUFWSCdMS29QSZTFGKx nuPfI8DjPha/kzFZL/AAa74J11GTSbQPFGcsWfKQJhEGfIHPs3deCxxAjFk7ZCrRf83g pRtw== X-Gm-Message-State: AKaTC015Q1DWrFINJXrIobvXZoLRz5lALPBTl82xC3oDM3xFKu6XI2K/1FqS0xTE9/Cw2lfXqmsLfMmV/WZepA== X-Received: by 10.28.97.139 with SMTP id v133mr5632828wmb.117.1481473357953; Sun, 11 Dec 2016 08:22:37 -0800 (PST) MIME-Version: 1.0 Received: by 10.80.154.65 with HTTP; Sun, 11 Dec 2016 08:22:37 -0800 (PST) From: chris g Date: Sun, 11 Dec 2016 17:22:37 +0100 Message-ID: Subject: Poor PF performance with 2.5k rdr's To: freebsd-pf@freebsd.org Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Dec 2016 16:22:40 -0000 Hello, I've decided to write here, as we had no luck troubleshooting PF's poor performance on 1GE interface. Network scheme, given as simplest as possible is: ISP <-> BGP ROUTER <-> PF ROUTER with many rdr rules <-> LAN Problem is reproducible on any PF ROUTER's connection - to LAN and to BGP ROUTER Both BGP and PF routers' OS versions and tunables, hardware: Hardware: E3-1230 V2 with HT on, 8GB RAM, ASUS P8B-E, NICs: Intel I350 on PCIe FreeBSD versions tested: 9.2-RELEASE amd64 with Custom kernel, 10.3-STABLE(compiled 4th Dec 2016) amd64 with Generic kernel. Basic tunables (for 9.2-RELEASE): net.inet.ip.forwarding=1 net.inet.ip.fastforwarding=1 kern.ipc.somaxconn=65535 net.inet.tcp.sendspace=65536 net.inet.tcp.recvspace=65536 net.inet.udp.recvspace=65536 kern.random.sys.harvest.ethernet=0 kern.random.sys.harvest.point_to_point=0 kern.random.sys.harvest.interrupt=0 kern.polling.idle_poll=1 BGP router doesn't have any firewall. PF options of PF router are: set state-policy floating set limit { states 2048000, frags 2000, src-nodes 384000 } set optimization normal Problem description: We are experiencing low throughput when PF is enabled with all the rdr's. If 'skip' is set on benchmarked interface or the rdr rules are commented (not present) - the bandwidth is flawless. In PF, there is no scrubbing done, most of roughly 2500 rdr rules look like this, please note that no interface is specified and it's intentional: rdr pass inet proto tcp from any to 1.2.3.4 port 1235 -> 192.168.0.100 port 1235 All measurements were taken using iperf 2.0.5 with options "-c " or "-c -m -t 60 -P 8" on client side and "-s" on server side. We changed directions too. Please note that this is a production environment and there was some other traffic on bencharked interfaces (let's say 20-100Mbps) during both tests, thus iperf won't show full Gigabit. There is no networking eqipment between 'client' and 'server' - just 2 NICs independly connected with Cat6 cable. Without further ado, here are benchmark results: server's PF enabled with fw rules but without rdr rules: root@client:~ # iperf -c server ------------------------------------------------------------ Client connecting to server, TCP port 5001 TCP window size: 65.0 KByte (default) ------------------------------------------------------------ [ 3] local clients_ip port 51361 connected with server port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 1.09 GBytes 936 Mbits/sec server's PF enabled with fw rules and around 2500 redirects present: root@client:~ # iperf -c seerver ------------------------------------------------------------ Client connecting to server, TCP port 5001 TCP window size: 65.0 KByte (default) ------------------------------------------------------------ [ 3] local clients_ip port 45671 connected with server port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 402 MBytes 337 Mbits/sec That much of a difference is 100% reproducible on production env. Performance depends on hours of day&night, the result is 160-400Mbps with RDR rules present and always above 900Mbps with RDR rules disabled. Some additional information: # pfctl -s info Status: Enabled for 267 days 10:25:22 Debug: Urgent State Table Total Rate current entries 132810 searches 5863318875 253.8/s inserts 140051669 6.1/s removals 139918859 6.1/s Counters match 1777051606 76.9/s bad-offset 0 0.0/s fragment 191 0.0/s short 518 0.0/s normalize 0 0.0/s memory 0 0.0/s bad-timestamp 0 0.0/s congestion 0 0.0/s ip-option 4383 0.0/s proto-cksum 0 0.0/s state-mismatch 52574 0.0/s state-insert 172 0.0/s state-limit 0 0.0/s src-limit 0 0.0/s synproxy 0 0.0/s # pfctl -s states | wc -l 113705 # pfctl -s memory states hard limit 2048000 src-nodes hard limit 384000 frags hard limit 2000 tables hard limit 1000 table-entries hard limit 200000 # pfctl -s Interfaces|wc -l 75 # pfctl -s rules | wc -l 1226 In our opinion hardware is not too weak as we have only 10-30% of CPU usage and during the benchmark it doesn't go to 100%. Even any one vcore isn't filled up to 100% of CPU usage. I would be really grateful if someone could point me at the right direction. Thank you, Chris From owner-freebsd-pf@freebsd.org Sun Dec 11 21:00:57 2016 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7FB75C727A2 for ; Sun, 11 Dec 2016 21:00:57 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5D54F1D6 for ; Sun, 11 Dec 2016 21:00:57 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id uBBL015h056066 for ; Sun, 11 Dec 2016 21:00:57 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <201612112100.uBBL015h056066@kenobi.freebsd.org> From: bugzilla-noreply@FreeBSD.org To: freebsd-pf@FreeBSD.org Subject: Problem reports for freebsd-pf@FreeBSD.org that need special attention Date: Sun, 11 Dec 2016 21:00:57 +0000 X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Dec 2016 21:00:57 -0000 To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- Open | 203735 | Transparent interception of ipv6 with squid and p 1 problems total for which you should take action. From owner-freebsd-pf@freebsd.org Sun Dec 11 21:21:08 2016 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 447FFC732E0 for ; Sun, 11 Dec 2016 21:21:08 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 347B4104C for ; Sun, 11 Dec 2016 21:21:08 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id uBBLL7nQ073785 for ; Sun, 11 Dec 2016 21:21:08 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-pf@FreeBSD.org Subject: [Bug 215041] [pf] Handshake to certain (fixed) hosts is dropped Date: Sun, 11 Dec 2016 21:21:07 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: kp@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-pf@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Dec 2016 21:21:08 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D215041 Kristof Provost changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |kp@freebsd.org --- Comment #1 from Kristof Provost --- Just to confirm, your host (running pf) is bridging, not routing, right? --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-pf@freebsd.org Sun Dec 11 21:45:24 2016 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5468FC737CE for ; Sun, 11 Dec 2016 21:45:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3C0901B71 for ; Sun, 11 Dec 2016 21:45:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id uBBLjNl1030281 for ; Sun, 11 Dec 2016 21:45:24 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-pf@FreeBSD.org Subject: [Bug 215041] [pf] Handshake to certain (fixed) hosts is dropped Date: Sun, 11 Dec 2016 21:45:24 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: bsd@ddh.de1.cc X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-pf@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Dec 2016 21:45:24 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D215041 --- Comment #2 from bsd@ddh.de1.cc --- (In reply to Kristof Provost from comment #1) Correct, it's just working as a MAC-level bridge. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-pf@freebsd.org Mon Dec 12 12:23:55 2016 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AA397C735E9 for ; Mon, 12 Dec 2016 12:23:55 +0000 (UTC) (envelope-from crest@rlwinm.de) Received: from smtp.rlwinm.de (smtp.rlwinm.de [IPv6:2a01:4f8:201:31ef::e]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 550DDE0F for ; Mon, 12 Dec 2016 12:23:55 +0000 (UTC) (envelope-from crest@rlwinm.de) Received: from crest.local (unknown [87.253.189.132]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.rlwinm.de (Postfix) with ESMTPSA id AB1875925 for ; Mon, 12 Dec 2016 13:23:52 +0100 (CET) Subject: Re: Poor PF performance with 2.5k rdr's To: freebsd-pf@freebsd.org References: From: Jan Bramkamp Message-ID: <5c14294a-66e8-1f50-0ef0-a5d77a9b656b@rlwinm.de> Date: Mon, 12 Dec 2016 13:23:52 +0100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:45.0) Gecko/20100101 Thunderbird/45.5.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Dec 2016 12:23:55 -0000 On 11/12/2016 17:22, chris g wrote: > Hello, > > I've decided to write here, as we had no luck troubleshooting PF's > poor performance on 1GE interface. > > Network scheme, given as simplest as possible is: > > ISP <-> BGP ROUTER <-> PF ROUTER with many rdr rules <-> LAN > > Problem is reproducible on any PF ROUTER's connection - to LAN and to BGP ROUTER > > > Both BGP and PF routers' OS versions and tunables, hardware: > > Hardware: E3-1230 V2 with HT on, 8GB RAM, ASUS P8B-E, NICs: Intel I350 on PCIe > > FreeBSD versions tested: 9.2-RELEASE amd64 with Custom kernel, > 10.3-STABLE(compiled 4th Dec 2016) amd64 with Generic kernel. > > Basic tunables (for 9.2-RELEASE): > net.inet.ip.forwarding=1 > net.inet.ip.fastforwarding=1 > kern.ipc.somaxconn=65535 > net.inet.tcp.sendspace=65536 > net.inet.tcp.recvspace=65536 > net.inet.udp.recvspace=65536 > kern.random.sys.harvest.ethernet=0 > kern.random.sys.harvest.point_to_point=0 > kern.random.sys.harvest.interrupt=0 > kern.polling.idle_poll=1 > > BGP router doesn't have any firewall. > > PF options of PF router are: > set state-policy floating > set limit { states 2048000, frags 2000, src-nodes 384000 } > set optimization normal > > > Problem description: > We are experiencing low throughput when PF is enabled with all the > rdr's. If 'skip' is set on benchmarked interface or the rdr rules are > commented (not present) - the bandwidth is flawless. In PF, there is > no scrubbing done, most of roughly 2500 rdr rules look like this, > please note that no interface is specified and it's intentional: > > rdr pass inet proto tcp from any to 1.2.3.4 port 1235 -> 192.168.0.100 port 1235 > > All measurements were taken using iperf 2.0.5 with options "-c " > or "-c -m -t 60 -P 8" on client side and "-s" on server side. We > changed directions too. > Please note that this is a production environment and there was some > other traffic on bencharked interfaces (let's say 20-100Mbps) during > both tests, thus iperf won't show full Gigabit. There is no networking > eqipment between 'client' and 'server' - just 2 NICs independly > connected with Cat6 cable. > > Without further ado, here are benchmark results: > > server's PF enabled with fw rules but without rdr rules: > root@client:~ # iperf -c server > ------------------------------------------------------------ > Client connecting to server, TCP port 5001 > TCP window size: 65.0 KByte (default) > ------------------------------------------------------------ > [ 3] local clients_ip port 51361 connected with server port 5001 > [ ID] Interval Transfer Bandwidth > [ 3] 0.0-10.0 sec 1.09 GBytes 936 Mbits/sec > > > > server's PF enabled with fw rules and around 2500 redirects present: > root@client:~ # iperf -c seerver > ------------------------------------------------------------ > Client connecting to server, TCP port 5001 > TCP window size: 65.0 KByte (default) > ------------------------------------------------------------ > [ 3] local clients_ip port 45671 connected with server port 5001 > [ ID] Interval Transfer Bandwidth > [ 3] 0.0-10.0 sec 402 MBytes 337 Mbits/sec > > > That much of a difference is 100% reproducible on production env. > > Performance depends on hours of day&night, the result is 160-400Mbps > with RDR rules present and always above 900Mbps with RDR rules > disabled. > > > Some additional information: > > # pfctl -s info > Status: Enabled for 267 days 10:25:22 Debug: Urgent > > State Table Total Rate > current entries 132810 > searches 5863318875 253.8/s > inserts 140051669 6.1/s > removals 139918859 6.1/s > Counters > match 1777051606 76.9/s > bad-offset 0 0.0/s > fragment 191 0.0/s > short 518 0.0/s > normalize 0 0.0/s > memory 0 0.0/s > bad-timestamp 0 0.0/s > congestion 0 0.0/s > ip-option 4383 0.0/s > proto-cksum 0 0.0/s > state-mismatch 52574 0.0/s > state-insert 172 0.0/s > state-limit 0 0.0/s > src-limit 0 0.0/s > synproxy 0 0.0/s > > # pfctl -s states | wc -l > 113705 > > # pfctl -s memory > states hard limit 2048000 > src-nodes hard limit 384000 > frags hard limit 2000 > tables hard limit 1000 > table-entries hard limit 200000 > > # pfctl -s Interfaces|wc -l > 75 > > # pfctl -s rules | wc -l > 1226 > > > In our opinion hardware is not too weak as we have only 10-30% of CPU > usage and during the benchmark it doesn't go to 100%. Even any one > vcore isn't filled up to 100% of CPU usage. > > > I would be really grateful if someone could point me at the right direction. PF uses a linear search (with some optimizations to skip over rules which can't match) to establish new flows. If your PF config is really that simple give IPFW a try. While PF has a lot nicer syntax IPFW supports more powerful tables. IPFW tables are key value maps and the value can be used as argument to most actions. It may reduce your 2500 lookups to one table lookup. If you can afford to loose the source IP and port you could use a userspace TCP proxy. From owner-freebsd-pf@freebsd.org Tue Dec 13 15:18:15 2016 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7AA01C75BFB for ; Tue, 13 Dec 2016 15:18:15 +0000 (UTC) (envelope-from ian.freislich@capeaugusta.com) Received: from mail-yb0-x235.google.com (mail-yb0-x235.google.com [IPv6:2607:f8b0:4002:c09::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4025CB91 for ; Tue, 13 Dec 2016 15:18:14 +0000 (UTC) (envelope-from ian.freislich@capeaugusta.com) Received: by mail-yb0-x235.google.com with SMTP id d59so16758443ybi.1 for ; Tue, 13 Dec 2016 07:18:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=capeaugusta-com.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to; bh=T9ih0UrVQulskWnxk3ju1wCqbZhOAUAfOYG4NYFfWig=; b=OSJTkyhp/PsFLxfqBjwhv5A4QtbbetjmiW5OuoYYZHBNItfs+FRMU833qRHk7m+ULu Fq4mVwRUHwBnDYEShimsgOi/3wjmm+93S//ebev9YG+HMZr0gAHJFoq76tyUK/oPQ4aA 4EDYQDkdLr8ZQ2uk/jJqXOQgvunbyokT6BTTFZvHF+trKu6Ut/rDK4D2FTU39ToJ5jiB DG7YpkVj5+qkXi55Fq8RWd2RALroEtgsnunJdUaqmVmKGbqI+iw7fyWGKX97Ml6BC2AI QnxXdcDw9/1gT1MJ5fmXT1n2dGIx9vtspNASFEFznT1nD0j1QKzfoyDBuIJiNB025kJq OsPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to; bh=T9ih0UrVQulskWnxk3ju1wCqbZhOAUAfOYG4NYFfWig=; b=nD+71YoOF4eCOgugESZFJqxWOM13rPmlwpz52R3heHWE3id/Jef48kc4b64ig4Zjoh wXcII5B7cPqjRBs6YN/bs6batqpfbEUznpwCvXkMAxSEO+Zuh7a43swwUqKOy2/P9eq3 OgyCVijG8le5oNvd2Su990s5Abp30fh56uNWB4Bs5L2W3HNRgRTkgixBaeUPaJcMW0jI foVfhemJ2HQH3JqGtNrV4i4U3Bjkr9Grlcmv7dRPtzILn9FG4Zzy5Ja9/8FwjtwFAzsN W7OMQRn41Kb/nLWh4/xy2kUoeNkdkRFoK2E2kbxCV/n7Uqa30W0FORYxIpAYM8EUTu4f X1YQ== X-Gm-Message-State: AKaTC00zX/ugRsfL+k1V1TIttuMvayaeZbapuXOVgjzhJOI8dgad4zGjQC4TshcWrZquiB48FQCJR+qK8B0GsCKXVn7cTS8yEiL6M9aQ6Swd0d9+Xt0VqshGW9EgkdrU05xNigWp4aPA6yCueQqIB2a7/Le5ESzLSBeI/T2eoykoXYk72PfXnI9SPbfgb6mWMt6OdA== X-Received: by 10.129.55.13 with SMTP id e13mr106252250ywa.344.1481642290317; Tue, 13 Dec 2016 07:18:10 -0800 (PST) Received: from zen.clue.co.za ([64.53.114.237]) by smtp.gmail.com with ESMTPSA id q186sm19751114ywb.2.2016.12.13.07.18.09 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 13 Dec 2016 07:18:10 -0800 (PST) Subject: Re: Poor PF performance with 2.5k rdr's To: freebsd-pf@freebsd.org References: From: Ian FREISLICH Message-ID: <5d8f9f65-bb3a-ef25-0fbe-bfc28b9025df@capeaugusta.com> Date: Tue, 13 Dec 2016 10:18:09 -0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.5.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252 X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2016 15:18:15 -0000 Chris, It's been a fairly long time since I ran a FreeBSD router in a production environment, 10-CURRENT at the time. tcp.sendspace/recvspace will have no effect on forwarding performance. Digging around in my email I've found some of my config: --- /etc/pf.conf # Options # ~~~~~~~ set timeout { \ adaptive.start 900000, \ adaptive.end 1800000 \ } set block-policy return set state-policy if-bound set optimization normal set ruleset-optimization basic set limit states 1500000 set limit frags 40000 set limit src-nodes 150000 --- --- /etc/sysctl.conf --- net.inet.ip.fastforwarding=1 net.inet.tcp.blackhole=2 net.inet.udp.blackhole=1 net.inet.ip.fastforwarding=1 net.inet.carp.preempt=1 net.inet.icmp.icmplim_output=0 net.inet.icmp.icmplim=0 kern.random.sys.harvest.interrupt=0 kern.random.sys.harvest.ethernet=0 kern.random.sys.harvest.point_to_point=0 net.route.netisr_maxqlen=8192 --- --- /boot/loader.conf console="comconsole" net.isr.maxthreads="8" net.isr.defaultqlimit="4096" net.isr.maxqlimit="81920" net.isr.direct="0" net.isr.direct_force="0" kern.ipc.nmbclusters="262144" kern.maxusers="1024" hw.bce.rx_pages="8" hw.bce.tx_pages="8" --- Our pfctl -s inf at the time: State Table Total Rate current entries 330022 searches 516720212 91910.4/s inserts 24545254 4365.9/s removals 24215232 4307.2/s Counters match 66166232 11769.2/s We were using a different NIC to you and eventually moved to ixgb(4) and bxe(4) NICs to handle the traffic, but the principle is the same: tune the queues. We didn't have as many rdr rules as you do, but the rule set is only linearly searched when there is no matching state in the state table. This means the rules are linearly searched for the first packet in each flow. In my testing, the other large contributor to forwarding rate is L1 cache size. Intel CPUs have traditionally had very small L1 cache sizes ranging from 12K to 32K and they're almost never quoted in marketing or comparison material. Your CPU has 32K of L1 data and 32K of L1 instruction cache per core. You may want to try disabling HT if that's possible these days to reduce L1 contention with the HT instance on each core. I may be talking total rubbish regarding HT and cache architecture but I think it's worth a try. Ian -- Ian Freislich On 12/11/16 11:22, chris g wrote: > Hello, > > I've decided to write here, as we had no luck troubleshooting PF's > poor performance on 1GE interface. > > Network scheme, given as simplest as possible is: > > ISP <-> BGP ROUTER <-> PF ROUTER with many rdr rules <-> LAN > > Problem is reproducible on any PF ROUTER's connection - to LAN and to BGP ROUTER > > > Both BGP and PF routers' OS versions and tunables, hardware: > > Hardware: E3-1230 V2 with HT on, 8GB RAM, ASUS P8B-E, NICs: Intel I350 on PCIe > > FreeBSD versions tested: 9.2-RELEASE amd64 with Custom kernel, > 10.3-STABLE(compiled 4th Dec 2016) amd64 with Generic kernel. > > Basic tunables (for 9.2-RELEASE): > net.inet.ip.forwarding=1 > net.inet.ip.fastforwarding=1 > kern.ipc.somaxconn=65535 > net.inet.tcp.sendspace=65536 > net.inet.tcp.recvspace=65536 > net.inet.udp.recvspace=65536 > kern.random.sys.harvest.ethernet=0 > kern.random.sys.harvest.point_to_point=0 > kern.random.sys.harvest.interrupt=0 > kern.polling.idle_poll=1 > > BGP router doesn't have any firewall. > > PF options of PF router are: > set state-policy floating > set limit { states 2048000, frags 2000, src-nodes 384000 } > set optimization normal > > > Problem description: > We are experiencing low throughput when PF is enabled with all the > rdr's. If 'skip' is set on benchmarked interface or the rdr rules are > commented (not present) - the bandwidth is flawless. In PF, there is > no scrubbing done, most of roughly 2500 rdr rules look like this, > please note that no interface is specified and it's intentional: > > rdr pass inet proto tcp from any to 1.2.3.4 port 1235 -> 192.168.0.100 port 1235 > > All measurements were taken using iperf 2.0.5 with options "-c " > or "-c -m -t 60 -P 8" on client side and "-s" on server side. We > changed directions too. > Please note that this is a production environment and there was some > other traffic on bencharked interfaces (let's say 20-100Mbps) during > both tests, thus iperf won't show full Gigabit. There is no networking > eqipment between 'client' and 'server' - just 2 NICs independly > connected with Cat6 cable. > > Without further ado, here are benchmark results: > > server's PF enabled with fw rules but without rdr rules: > root@client:~ # iperf -c server > ------------------------------------------------------------ > Client connecting to server, TCP port 5001 > TCP window size: 65.0 KByte (default) > ------------------------------------------------------------ > [ 3] local clients_ip port 51361 connected with server port 5001 > [ ID] Interval Transfer Bandwidth > [ 3] 0.0-10.0 sec 1.09 GBytes 936 Mbits/sec > > > > server's PF enabled with fw rules and around 2500 redirects present: > root@client:~ # iperf -c seerver > ------------------------------------------------------------ > Client connecting to server, TCP port 5001 > TCP window size: 65.0 KByte (default) > ------------------------------------------------------------ > [ 3] local clients_ip port 45671 connected with server port 5001 > [ ID] Interval Transfer Bandwidth > [ 3] 0.0-10.0 sec 402 MBytes 337 Mbits/sec > > > That much of a difference is 100% reproducible on production env. > > Performance depends on hours of day&night, the result is 160-400Mbps > with RDR rules present and always above 900Mbps with RDR rules > disabled. > > > Some additional information: > > # pfctl -s info > Status: Enabled for 267 days 10:25:22 Debug: Urgent > > State Table Total Rate > current entries 132810 > searches 5863318875 253.8/s > inserts 140051669 6.1/s > removals 139918859 6.1/s > Counters > match 1777051606 76.9/s > bad-offset 0 0.0/s > fragment 191 0.0/s > short 518 0.0/s > normalize 0 0.0/s > memory 0 0.0/s > bad-timestamp 0 0.0/s > congestion 0 0.0/s > ip-option 4383 0.0/s > proto-cksum 0 0.0/s > state-mismatch 52574 0.0/s > state-insert 172 0.0/s > state-limit 0 0.0/s > src-limit 0 0.0/s > synproxy 0 0.0/s > > # pfctl -s states | wc -l > 113705 > > # pfctl -s memory > states hard limit 2048000 > src-nodes hard limit 384000 > frags hard limit 2000 > tables hard limit 1000 > table-entries hard limit 200000 > > # pfctl -s Interfaces|wc -l > 75 > > # pfctl -s rules | wc -l > 1226 > > > In our opinion hardware is not too weak as we have only 10-30% of CPU > usage and during the benchmark it doesn't go to 100%. Even any one > vcore isn't filled up to 100% of CPU usage. > > > I would be really grateful if someone could point me at the right direction. > > > Thank you, > Chris > _______________________________________________ > freebsd-pf@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-pf > To unsubscribe, send any mail to "freebsd-pf-unsubscribe@freebsd.org" -- Cape Augusta Digital Properties, LLC a Cape Augusta Company *Breach of confidentiality & accidental breach of confidentiality * This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. From owner-freebsd-pf@freebsd.org Tue Dec 13 23:42:05 2016 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ED8E0C764B9 for ; Tue, 13 Dec 2016 23:42:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DD5FC19F for ; Tue, 13 Dec 2016 23:42:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id uBDNg53h053982 for ; Tue, 13 Dec 2016 23:42:05 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-pf@FreeBSD.org Subject: [Bug 211730] pf uses 32bit value for bandwith with altq Date: Tue, 13 Dec 2016 23:42:06 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.3-RELEASE X-Bugzilla-Keywords: patch X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: ncrogers@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-pf@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2016 23:42:06 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D211730 --- Comment #12 from ncrogers@gmail.com --- I ran into this same problem myself today on a 10G ixgbe(4) interface runni= ng 10.1-RELEASE... Did anyone manage to get a working patch going? Looks like activity towards a resolution has died off. Even though ALTQ is unlikely to support speeds higher than 4Gb/s on most hardware, it would be nice to know that we can configure a 10G link speed. Currently if you set an interface a= ltq to 10G it confusingly becomes around 1.4Gb without warning due to the 32bit unsigned integer. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-pf@freebsd.org Thu Dec 15 09:03:05 2016 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 27230C8019E for ; Thu, 15 Dec 2016 09:03:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 170EDB90 for ; Thu, 15 Dec 2016 09:03:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id uBF934Fr005674 for ; Thu, 15 Dec 2016 09:03:04 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-pf@FreeBSD.org Subject: [Bug 215041] [pf] Handshake to certain (fixed) hosts is dropped Date: Thu, 15 Dec 2016 09:03:05 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: bsd@ddh.de1.cc X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-pf@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2016 09:03:05 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D215041 --- Comment #3 from bsd@ddh.de1.cc --- Update: The problem seems to center on the line "pass [log] all". When I comment out the line and do "pfctl -F all -f configfile", the handshake to 185.60.115.40:443 works. Comment it in again, flush/reload, and the handsha= kes disappear again. Same story with a slightly bigger config: int_if=3D"em0" ext_if=3D"re0" rdr on $int_if inet proto tcp from any to any port www -> 127.0.0.1 port 31= 28 pass in quick on $int_if route-to lo0 inet proto tcp from any to 127.0.0.1 = port 3128 keep state pass all -> Handshakes get dropped. Remove the "pass all", handshakes work. Is this some intricacy of the rule syntax I'm missing or a legit bug? PS: Sorry for not testing this earlier, a "pass all" ruleset seemed too min= imal to have any effect... --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-pf@freebsd.org Fri Dec 16 12:16:21 2016 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4ADBDC82E5E for ; Fri, 16 Dec 2016 12:16:21 +0000 (UTC) (envelope-from admin@x158.save85off.com) Received: from x158.save85off.com (x158.save85off.com [43.240.238.158]) by mx1.freebsd.org (Postfix) with ESMTP id 174C81D65 for ; Fri, 16 Dec 2016 12:16:20 +0000 (UTC) (envelope-from admin@x158.save85off.com) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; s=save85off; d=x158.save85off.com; h=MIME-Version:From:To:Date:Subject:Content-Type:Content-Transfer-Encoding; i=admin@x158.save85off.com; bh=oKQhuRg9B+HVAAsGe/q/CA3HOh4=; b=NklDpNd6DjznGIutOZ2iCBRlMQMlkjTmOGv9Pdrl6G6Q0JU29wa99SVfoojhSLBQewmWudLyDyvf Ej55v721ro9ZsUtNTtkqYWePa7ODuyQcCQThkcI8rMU684vVIAWfyqtE38k5wd8DDbFEEDkq8XHJ nPZ25R/qeK0JiTuV788= DomainKey-Signature: a=rsa-sha1; c=nofws; q=dns; s=save85off; d=x158.save85off.com; b=kM8Y2sVQ2Y4QrhNLYOJd1uCS94IlCVJTjyPih5O6WDujCvk7QWDS0quJBxJIa4WoUFk67BzZYtPL 9ESQRRnTb/ZLqfHgWk9hKyvGAlEcks2yDW8pk0zA/cG7fkIZXcUyDeTgsuZtGKrpyojHwQjxqs0r Xh1KzpdSXp79kXMKmdc=; From: "UGG Big Deals" To: freebsd-pf@freebsd.org Date: 16 Dec 2016 20:04:19 +0800 Subject: You'll be thankful for these savings on Christmas! win 133$ MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Dec 2016 12:16:21 -0000