From owner-freebsd-ipfw@FreeBSD.ORG Sun Sep 14 10:37:17 2014 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2DC74DC3 for ; Sun, 14 Sep 2014 10:37:17 +0000 (UTC) Received: from smtp.digiware.nl (unknown [IPv6:2001:4cb8:90:ffff::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DDC4E67D for ; Sun, 14 Sep 2014 10:37:16 +0000 (UTC) Received: from rack1.digiware.nl (unknown [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id 657BC153448; Sun, 14 Sep 2014 12:37:05 +0200 (CEST) X-Virus-Scanned: amavisd-new at digiware.nl Received: from smtp.digiware.nl ([127.0.0.1]) by rack1.digiware.nl (rack1.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id MTiL1bMN8ftB; Sun, 14 Sep 2014 12:36:42 +0200 (CEST) Received: from [192.168.10.9] (vaio [192.168.10.9]) (using TLSv1 with cipher ECDHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by smtp.digiware.nl (Postfix) with ESMTPSA id E6134153A8C; Sun, 14 Sep 2014 12:36:42 +0200 (CEST) Message-ID: <54156FBB.1030907@digiware.nl> Date: Sun, 14 Sep 2014 12:36:43 +0200 From: Willem Jan Withagen User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: Freddie Cash , bycn82 Subject: Re: IPFW rule sets and automatic rule numbering References: <541469D4.6070107@gmail.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-ipfw@freebsd.org X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Sep 2014 10:37:17 -0000 On 13-9-2014 21:51, Freddie Cash wrote: > You can replicate it using 3 rules, loaded into two sets: > > ipfw set disable 1 > ipfw add allow ip from any to any > ipfw add 65524 allow ip from any to any > ipfw add allow ip from any to any > ipfw set swap 1 0 > > Run that two or 3 times. Every rule will be numbered 65534 after the 2nd or > 3rd run. > > I expected it to be numbered 10, 65524, 65534 after every run. > > However, after reading the man page a few more times and thinking about it > a little more, it makes sense that the numbering is global across all sets, > as you can have multiple sets enabled simultaneously. > > It just doesn't mesh with my desire to use auto numbering. I'm in the midst > of manually numbering all my rules now. :) > _______________________________________________ > freebsd-ipfw@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-ipfw > To unsubscribe, send any mail to "freebsd-ipfw-unsubscribe@freebsd.org" > This is easily circumvented by making shure that the first rule is ipfw add 10 ..... like: ipfw add count ip4 from any to any via vlan126 (vlan126 is my outside connection) And then you are home free. I actually use this to also separate diffent types of block by injecting: ipfw add count ip from any to any like: 03000 713812041 425643462848 count ip from any to any 03010 0 0 deny ip6 from fc00::/7 to any via vlan126 And the 3000 block contains all antispoofing and likes. --WjW From owner-freebsd-ipfw@FreeBSD.ORG Sun Sep 14 11:44:25 2014 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D918EA11; Sun, 14 Sep 2014 11:44:25 +0000 (UTC) Received: from sola.nimnet.asn.au (paqi.nimnet.asn.au [115.70.110.159]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2F17BCA2; Sun, 14 Sep 2014 11:44:24 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by sola.nimnet.asn.au (8.14.2/8.14.2) with ESMTP id s8EBi5LW078052; Sun, 14 Sep 2014 21:44:05 +1000 (EST) (envelope-from smithi@nimnet.asn.au) Date: Sun, 14 Sep 2014 21:44:05 +1000 (EST) From: Ian Smith To: Willem Jan Withagen Subject: Re: IPFW rule sets and automatic rule numbering In-Reply-To: <54156FBB.1030907@digiware.nl> Message-ID: <20140914204055.K61666@sola.nimnet.asn.au> References: <541469D4.6070107@gmail.com> <54156FBB.1030907@digiware.nl> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Cc: "Alexander V. Chernikov" , Freddie Cash , bycn82 , freebsd-ipfw@freebsd.org X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Sep 2014 11:44:25 -0000 On Sun, 14 Sep 2014 12:36:43 +0200, Willem Jan Withagen wrote: > On 13-9-2014 21:51, Freddie Cash wrote: > > You can replicate it using 3 rules, loaded into two sets: > > > > ipfw set disable 1 > > ipfw add allow ip from any to any > > ipfw add 65524 allow ip from any to any > > ipfw add allow ip from any to any > > ipfw set swap 1 0 > > > > Run that two or 3 times. Every rule will be numbered 65534 after the 2nd or > > 3rd run. > > > > > I expected it to be numbered 10, 65524, 65534 after every run. > > > > However, after reading the man page a few more times and thinking about it > > a little more, it makes sense that the numbering is global across all sets, > > as you can have multiple sets enabled simultaneously. > > > > It just doesn't mesh with my desire to use auto numbering. I'm in the midst > > of manually numbering all my rules now. :) > This is easily circumvented by making shure that the first rule is > > ipfw add 10 ..... > like: > ipfw add count ip4 from any to any via vlan126 > (vlan126 is my outside connection) > And then you are home free. > > I actually use this to also separate diffent types of block by injecting: > ipfw add count ip from any to any > > like: > 03000 713812041 425643462848 count ip from any to any > 03010 0 0 deny ip6 from fc00::/7 to any via vlan126 > > And the 3000 block contains all antispoofing and likes. I'd almost replied along the same lines - also tending to number first rules in a block and then add unnumbered rules until other sections - before realising that once Freddie had added his rule 65524, maybe in another set, it's game over; every unnumbered rule added after that is going to be >= 65524 and <= 65534. And even as you have it above, if you rerun your script again without flushing the entire ruleset, apart from specifically numbered rule/s, everything will get readded after the highest rule previously used. Alexander said: > I think we can consider implementing sysctl which permits per-set > auto-numbering. Perhaps rather than a separate high water mark per set that would be reset on set N flush - which I think is what Alexander means? - if one could set something like maybe 'net.inet.ip.fw.autoinc_last' which would default, as now, to the highest rule added so far, but could be reset to something else before adding more auto_inc rules? Might get tricky, and of course it has to not break older rule scripts - some VERY old :) cheers, Ian From owner-freebsd-ipfw@FreeBSD.ORG Sun Sep 14 12:47:28 2014 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2E485B0A; Sun, 14 Sep 2014 12:47:28 +0000 (UTC) Received: from smtp.digiware.nl (unknown [IPv6:2001:4cb8:90:ffff::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B3861227; Sun, 14 Sep 2014 12:47:27 +0000 (UTC) Received: from rack1.digiware.nl (unknown [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id E80B91534EC; Sun, 14 Sep 2014 14:47:24 +0200 (CEST) X-Virus-Scanned: amavisd-new at digiware.nl Received: from smtp.digiware.nl ([127.0.0.1]) by rack1.digiware.nl (rack1.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id wlLW4Ph4l5qn; Sun, 14 Sep 2014 14:47:05 +0200 (CEST) Received: from [192.168.10.9] (vaio [192.168.10.9]) (using TLSv1 with cipher ECDHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by smtp.digiware.nl (Postfix) with ESMTPSA id 852E1153448; Sun, 14 Sep 2014 14:47:05 +0200 (CEST) Message-ID: <54158E49.7090102@digiware.nl> Date: Sun, 14 Sep 2014 14:47:05 +0200 From: Willem Jan Withagen User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: Ian Smith Subject: Re: IPFW rule sets and automatic rule numbering References: <541469D4.6070107@gmail.com> <54156FBB.1030907@digiware.nl> <20140914204055.K61666@sola.nimnet.asn.au> In-Reply-To: <20140914204055.K61666@sola.nimnet.asn.au> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: "Alexander V. Chernikov" , Freddie Cash , bycn82 , freebsd-ipfw@freebsd.org X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Sep 2014 12:47:28 -0000 On 14-9-2014 13:44, Ian Smith wrote: > On Sun, 14 Sep 2014 12:36:43 +0200, Willem Jan Withagen wrote: > > On 13-9-2014 21:51, Freddie Cash wrote: > > > You can replicate it using 3 rules, loaded into two sets: > > > > > > ipfw set disable 1 > > > ipfw add allow ip from any to any > > > ipfw add 65524 allow ip from any to any > > > ipfw add allow ip from any to any > > > ipfw set swap 1 0 > > > > > > Run that two or 3 times. Every rule will be numbered 65534 after the 2nd or > > > 3rd run. > > > > > > > > I expected it to be numbered 10, 65524, 65534 after every run. > > > > > > However, after reading the man page a few more times and thinking about it > > > a little more, it makes sense that the numbering is global across all sets, > > > as you can have multiple sets enabled simultaneously. > > > > > > It just doesn't mesh with my desire to use auto numbering. I'm in the midst > > > of manually numbering all my rules now. :) > > > This is easily circumvented by making shure that the first rule is > > > > ipfw add 10 ..... > > like: > > ipfw add count ip4 from any to any via vlan126 > > (vlan126 is my outside connection) > > And then you are home free. > > > > I actually use this to also separate diffent types of block by injecting: > > ipfw add count ip from any to any > > > > like: > > 03000 713812041 425643462848 count ip from any to any > > 03010 0 0 deny ip6 from fc00::/7 to any via vlan126 > > > > And the 3000 block contains all antispoofing and likes. > > I'd almost replied along the same lines - also tending to number first > rules in a block and then add unnumbered rules until other sections - > before realising that once Freddie had added his rule 65524, maybe in > another set, it's game over; every unnumbered rule added after that is > going to be >= 65524 and <= 65534. > > And even as you have it above, if you rerun your script again without > flushing the entire ruleset, apart from specifically numbered rule/s, > everything will get readded after the highest rule previously used. > > Alexander said: > > I think we can consider implementing sysctl which permits per-set > > auto-numbering. > > Perhaps rather than a separate high water mark per set that would be > reset on set N flush - which I think is what Alexander means? - if one > could set something like maybe 'net.inet.ip.fw.autoinc_last' which would > default, as now, to the highest rule added so far, but could be reset to > something else before adding more auto_inc rules? Might get tricky, and > of course it has to not break older rule scripts - some VERY old :) Could very well be. I was for the same reason going to implement Freddy's strategy and swap sets... So could be that I'd run into the same problem in near future. It is hard to imagine that people would depend on the fact that the last rule used would survive a set swap, and actually build on it. Being able to actual set the last number used, aka the next number to be assigned, would be equivalent to my trick using 'ipfw add count....' to preset a certain startingpoint. It could even be made into either a flag on ipfw like -k keep the last index for future increment or an ipfw instruction ipfw setautoincr And that would not interfere with old scripts. tinkering it through sysctl would be possible too, but I'd prefer things like this integrated in the command-interface. --WjW From owner-freebsd-ipfw@FreeBSD.ORG Sun Sep 14 15:39:47 2014 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3EB985FB; Sun, 14 Sep 2014 15:39:47 +0000 (UTC) Received: from mail-pa0-x229.google.com (mail-pa0-x229.google.com [IPv6:2607:f8b0:400e:c03::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 07466300; Sun, 14 Sep 2014 15:39:47 +0000 (UTC) Received: by mail-pa0-f41.google.com with SMTP id bj1so4761236pad.0 for ; Sun, 14 Sep 2014 08:39:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type; bh=SwET4Y6I7HB4cEjZNIdsj31CmHdEudwUZalcd4F46T0=; b=RvO7dP4+lqB1uf6Sa6Mhzr1b99QnrG8PFZAgJO+6DRAzXr/5ICNIeUvCHUCMd8Scrc Nd3J1v+2W6CAtpu6aDdMS5a+iZfNobYXsvraZjrleg0RTBlWGUQZ8LsLG5GKPNkSUL1V UVsPrZi1nMxRLtpFudhaX+dgL5WBrj1Z9HLnaCgekYR8a+pah4KdpHiOxTW6vR2/ZM5a FNEbuE77JMevqajusa6F8Yk4eRqgWk/KshvG8nmqCNfZodKxQllLuEXaI6CwddOxsKGA HaJz3nxwWQ9gWoo/5c9OaTt8oT+2gU0Mx10QvajZr6/0Ht/8gA3WTkV2aawJVu5PbPHN nYGA== X-Received: by 10.68.102.132 with SMTP id fo4mr31223929pbb.96.1410709186201; Sun, 14 Sep 2014 08:39:46 -0700 (PDT) Received: from [192.168.1.99] ([203.117.37.212]) by mx.google.com with ESMTPSA id j5sm9123545pdp.9.2014.09.14.08.39.41 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 14 Sep 2014 08:39:45 -0700 (PDT) Message-ID: <5415B6BC.4090604@gmail.com> Date: Sun, 14 Sep 2014 23:39:40 +0800 From: bycn82 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.1.0 MIME-Version: 1.0 To: Willem Jan Withagen , Ian Smith Subject: Re: IPFW rule sets and automatic rule numbering References: <541469D4.6070107@gmail.com> <54156FBB.1030907@digiware.nl> <20140914204055.K61666@sola.nimnet.asn.au> <54158E49.7090102@digiware.nl> In-Reply-To: <54158E49.7090102@digiware.nl> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "Alexander V. Chernikov" , Freddie Cash , freebsd-ipfw@freebsd.org X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Sep 2014 15:39:47 -0000 On 9/14/14 20:47, Willem Jan Withagen wrote: > On 14-9-2014 13:44, Ian Smith wrote: >> On Sun, 14 Sep 2014 12:36:43 +0200, Willem Jan Withagen wrote: >> > On 13-9-2014 21:51, Freddie Cash wrote: >> > > You can replicate it using 3 rules, loaded into two sets: >> > > >> > > ipfw set disable 1 >> > > ipfw add allow ip from any to any >> > > ipfw add 65524 allow ip from any to any >> > > ipfw add allow ip from any to any >> > > ipfw set swap 1 0 >> > > >> > > Run that two or 3 times. Every rule will be numbered 65534 after the 2nd or >> > > 3rd run. >> > >> > > >> > > I expected it to be numbered 10, 65524, 65534 after every run. >> > > >> > > However, after reading the man page a few more times and thinking about it >> > > a little more, it makes sense that the numbering is global across all sets, >> > > as you can have multiple sets enabled simultaneously. >> > > >> > > It just doesn't mesh with my desire to use auto numbering. I'm in the midst >> > > of manually numbering all my rules now. :) >> >> > This is easily circumvented by making shure that the first rule is >> > >> > ipfw add 10 ..... >> > like: >> > ipfw add count ip4 from any to any via vlan126 >> > (vlan126 is my outside connection) >> > And then you are home free. >> > >> > I actually use this to also separate diffent types of block by injecting: >> > ipfw add count ip from any to any >> > >> > like: >> > 03000 713812041 425643462848 count ip from any to any >> > 03010 0 0 deny ip6 from fc00::/7 to any via vlan126 >> > >> > And the 3000 block contains all antispoofing and likes. >> >> I'd almost replied along the same lines - also tending to number first >> rules in a block and then add unnumbered rules until other sections - >> before realising that once Freddie had added his rule 65524, maybe in >> another set, it's game over; every unnumbered rule added after that is >> going to be >= 65524 and <= 65534. >> >> And even as you have it above, if you rerun your script again without >> flushing the entire ruleset, apart from specifically numbered rule/s, >> everything will get readded after the highest rule previously used. >> >> Alexander said: >> > I think we can consider implementing sysctl which permits per-set >> > auto-numbering. >> >> Perhaps rather than a separate high water mark per set that would be >> reset on set N flush - which I think is what Alexander means? - if one >> could set something like maybe 'net.inet.ip.fw.autoinc_last' which would >> default, as now, to the highest rule added so far, but could be reset to >> something else before adding more auto_inc rules? Might get tricky, and >> of course it has to not break older rule scripts - some VERY old :) > Could very well be. > I was for the same reason going to implement Freddy's strategy and swap > sets... So could be that I'd run into the same problem in near future. > > It is hard to imagine that people would depend on the fact that the last > rule used would survive a set swap, and actually build on it. > > Being able to actual set the last number used, aka the next number to be > assigned, would be equivalent to my trick using > 'ipfw add count....' > to preset a certain startingpoint. > It could even be made into either a flag on ipfw like > -k keep the last index for future increment > or an ipfw instruction > ipfw setautoincr > And that would not interfere with old scripts. > > tinkering it through sysctl would be possible too, but I'd prefer things > like this integrated in the command-interface. > > --WjW > > I think the per-set auto-numbering is a good idea. But we are there is because of the "set", the "set" in IPFW is a trouble maker. I stared to learn the IPFW in Apr this year, and I was disappoint on some features, and the "set" is one of them. I created a fix patch about it but not relevant to this topic. Regards, Bill Yuan From owner-freebsd-ipfw@FreeBSD.ORG Sun Sep 14 16:16:06 2014 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 66B97333 for ; Sun, 14 Sep 2014 16:16:06 +0000 (UTC) Received: from mail-oa0-x22a.google.com (mail-oa0-x22a.google.com [IPv6:2607:f8b0:4003:c02::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2B2418A9 for ; Sun, 14 Sep 2014 16:16:06 +0000 (UTC) Received: by mail-oa0-f42.google.com with SMTP id g18so1916590oah.29 for ; Sun, 14 Sep 2014 09:16:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=S1u/6pY3Vy/FFUiy8G2AXPENZM6ziI+7vtLfpFqJEUA=; b=T5I9NNhQ217N5e1ZdbPQdKZvgFPnFZYOgMrvQuGHhlWtunN0OTsMWKiFnrZkDpRHPb MBQNe+/20OG+0w5tSg+6UFyf8kLXN2TaTR7rLUncANKlIzZDM14erD+DqCCKZfgCT3Q6 XJSYqfj08qvhENwotGBf/o/hSB+t1V9EWIajkXWk8bjB4UiuKK6nkLhaOTR1IWghgJPY QIzLbLR1qku98Y7bBnJ/cNxUwaTwU6ABEh+AjNmRhzCgRUTllsoKlhFx2aJOELcdyfQ2 l5q1m8fO3XZuZJeFGbR3I/D+6RoiLKV+3uahCEfT0ShBerxO01M4gp+r2Oz+Mutv1vGE rhmg== MIME-Version: 1.0 X-Received: by 10.182.153.68 with SMTP id ve4mr2709294obb.60.1410711365223; Sun, 14 Sep 2014 09:16:05 -0700 (PDT) Received: by 10.202.199.11 with HTTP; Sun, 14 Sep 2014 09:16:05 -0700 (PDT) Received: by 10.202.199.11 with HTTP; Sun, 14 Sep 2014 09:16:05 -0700 (PDT) In-Reply-To: <54156FBB.1030907@digiware.nl> References: <541469D4.6070107@gmail.com> <54156FBB.1030907@digiware.nl> Date: Sun, 14 Sep 2014 09:16:05 -0700 Message-ID: Subject: Re: IPFW rule sets and automatic rule numbering From: Freddie Cash To: Willem Jan Withagen Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: freebsd-ipfw@freebsd.org, bycn82 X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Sep 2014 16:16:06 -0000 On Sep 14, 2014 3:37 AM, "Willem Jan Withagen" wrote: > > On 13-9-2014 21:51, Freddie Cash wrote: > > You can replicate it using 3 rules, loaded into two sets: > > > > ipfw set disable 1 > > ipfw add allow ip from any to any > > ipfw add 65524 allow ip from any to any > > ipfw add allow ip from any to any > > ipfw set swap 1 0 > > > > Run that two or 3 times. Every rule will be numbered 65534 after the 2nd or > > 3rd run. > > > > > I expected it to be numbered 10, 65524, 65534 after every run. > > > > However, after reading the man page a few more times and thinking about it > > a little more, it makes sense that the numbering is global across all sets, > > as you can have multiple sets enabled simultaneously. > > > > It just doesn't mesh with my desire to use auto numbering. I'm in the midst > > of manually numbering all my rules now. :) > > _______________________________________________ > > freebsd-ipfw@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-ipfw > > To unsubscribe, send any mail to "freebsd-ipfw-unsubscribe@freebsd.org" > > > > This is easily circumvented by making shure that the first rule is Nope. It doesn't matter what the first number is. What matters is the _last_ number used. The auto-increment feature tracks the last rule number used, then adds the auto-increment number (as set by sysctl, I believe the default is 10) to any rules without explicit numbers. My rules start at 2, are specifically numbered up to 20, then I group things together by starting the server- specific scripts on even 100s or 1000s, and auto-increment through that file. Which works great if all you do is: - clear all rules (which resets the "last number" to 0) - load all rules That's what I used to do, and what I do on all school firewall boxes. Works beautifully. Has for years. No sets involved. The only downside is that it breaks all traffic for X seconds while the new rules load. Fine for a school, as it's usually under 2 seconds, and nobody notices. At the main school board office, it takes almost 30 seconds to load the rules, so I was looking into rule sets to allow for almost instantaneous loading (swap) of the rules, minimising downtime. Unfortunately, that's when I hit the =E2=80=9Drule numbers are global acros= s sets =E2=80=9D issue. It took me by surprise. It makes sense to me now, and I can work with it that way. It just want what I expected. Originally, I had hoped to just use sets without changing my scripts like so: - disable and clear set 1 - load into set 1 - swap sets 1 and 0 I can still do that, I just need to manually number all my rules first. And probably try the following: - disable and clear set 1 - load into set 1 - enable set 1 - swap sets 1 and 0 - disable set 1 That way, there shouldn't be any downtime at all, and all connections should continue during the reload. From owner-freebsd-ipfw@FreeBSD.ORG Wed Sep 17 14:58:48 2014 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 51FEA5B9 for ; Wed, 17 Sep 2014 14:58:48 +0000 (UTC) Received: from mail-oa0-x22d.google.com (mail-oa0-x22d.google.com [IPv6:2607:f8b0:4003:c02::22d]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1719498D for ; Wed, 17 Sep 2014 14:58:48 +0000 (UTC) Received: by mail-oa0-f45.google.com with SMTP id i7so1228170oag.18 for ; Wed, 17 Sep 2014 07:58:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=ko07nLc8w4xqH9h9oYOikLzswn04n315NoA00NfbQYA=; b=dmxCNPTjheWy9AGLahBoquFrX4MsBspmZAFe4YjlUg0Auf3JsN0BrjBGZdgDpzhiFz HAO14801Q0fTq+uaethjuBZar7vIzq9aDRcq2bfunWsggBgKhcNR9wjsun6Ui4Z66oP+ 0Pk1a42nYncZxJM1XMbyhTaxIze7u2a/Wmg/iO/+6Gq9RTU+E8fd/4AiuDxFBFPZpZLE 3tKzb6sH47bXQ2x/ZvSYQnLmw7Frt55+8T+AMK+YC1q5ByptTSo7cBmvlCrO9E4oYrRN ZMkgFfWtGFC4i4AWT8AULurAfBkcGv/0IkLcdjLTKhRatdvtoEtBHjl2hsUrklyers2V 04LA== MIME-Version: 1.0 X-Received: by 10.182.27.40 with SMTP id q8mr3171373obg.86.1410965927443; Wed, 17 Sep 2014 07:58:47 -0700 (PDT) Received: by 10.202.199.11 with HTTP; Wed, 17 Sep 2014 07:58:47 -0700 (PDT) In-Reply-To: References: <541469D4.6070107@gmail.com> <54156FBB.1030907@digiware.nl> Date: Wed, 17 Sep 2014 07:58:47 -0700 Message-ID: Subject: Re: IPFW rule sets and automatic rule numbering From: Freddie Cash To: Willem Jan Withagen Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "freebsd-ipfw@freebsd.org" , bycn82 X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Sep 2014 14:58:48 -0000 Just to summarise everything: 1. Automatic rule numbering works beautifully if you only ever use the default rule set (set 0). Meaning, if you don't use any set commands at all. 2. If you manually number every rule, then using rule sets works beautifully. 3. Doing a little set manipulation allows you to load updated rules without disconnecting anyone or dropping any packets: disable set 1 load rules into set 1 enable set 1 swap set 1 0 disable set 1 I understand how everything works a little bit better now. Thanks for all the help and pointers and discussion. -- Freddie Cash fjwcash@gmail.com From owner-freebsd-ipfw@FreeBSD.ORG Thu Sep 18 00:34:46 2014 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E989D331 for ; Thu, 18 Sep 2014 00:34:46 +0000 (UTC) Received: from mail-pd0-x230.google.com (mail-pd0-x230.google.com [IPv6:2607:f8b0:400e:c02::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B7E7EEB6 for ; Thu, 18 Sep 2014 00:34:46 +0000 (UTC) Received: by mail-pd0-f176.google.com with SMTP id g10so200601pdj.7 for ; Wed, 17 Sep 2014 17:34:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type; bh=2gIJyd7N8qLfOeLpNDe2pQASzqsZyk501n5zvcUh534=; b=Y06ijJ1Rj30Pxb/yQPQbopvfFBCy1HFnvNZFRKE9to+DJ/012DTmxHzXWxG9gqRnyo kfVqRl2Dt60Q38NrfWhbPkgr1f2W88D7cSBCBeRr0dv1GGBnQ8OTyNmdIfjIoMKlSDYD kKweSHTeCHNehZkmZrhnq+0S3Mssj60yJPnKsuznWx0jdh2Qu4i3+26Hl6Su6uxxG4hN qZch6mUUE9RfkSe68uKhPOvZUR/ERrLYMb/Fzb9ghv8Lt4doOTo+gy/FgHcLuqCbtfnj NOpPC22KdDXVgZokaOL/lLjlQWKq6CNFEHQVJgGOkfpUUhwnE3W3oODpXpWBxoRywyHd ULYg== X-Received: by 10.68.69.33 with SMTP id b1mr1129055pbu.59.1411000485931; Wed, 17 Sep 2014 17:34:45 -0700 (PDT) Received: from [192.168.1.99] ([183.90.37.7]) by mx.google.com with ESMTPSA id ti8sm9153607pac.20.2014.09.17.17.34.42 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 17 Sep 2014 17:34:45 -0700 (PDT) Message-ID: <541A28A3.2090300@gmail.com> Date: Thu, 18 Sep 2014 08:34:43 +0800 From: bycn82 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.1.0 MIME-Version: 1.0 To: Freddie Cash , Willem Jan Withagen Subject: Re: IPFW rule sets and automatic rule numbering References: <541469D4.6070107@gmail.com> <54156FBB.1030907@digiware.nl> In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "freebsd-ipfw@freebsd.org" X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Sep 2014 00:34:47 -0000 On 9/17/14 22:58, Freddie Cash wrote: > Just to summarise everything: > > 1. Automatic rule numbering works beautifully if you only ever use > the default rule set (set 0). Meaning, if you don't use any set > commands at all. > > 2. If you manually number every rule, then using rule sets works > beautifully. > > 3. Doing a little set manipulation allows you to load updated rules > without disconnecting anyone or dropping any packets: > disable set 1 > load rules into set 1 > enable set 1 you dont need below steps. > swap set 1 0 > disable set 1 > > I understand how everything works a little bit better now. Thanks for > all the help and pointers and discussion. > > -- > Freddie Cash > fjwcash@gmail.com From owner-freebsd-ipfw@FreeBSD.ORG Thu Sep 18 16:26:10 2014 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B4901E71 for ; Thu, 18 Sep 2014 16:26:10 +0000 (UTC) Received: from mail-ie0-x230.google.com (mail-ie0-x230.google.com [IPv6:2607:f8b0:4001:c03::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 83CED10B for ; Thu, 18 Sep 2014 16:26:10 +0000 (UTC) Received: by mail-ie0-f176.google.com with SMTP id ar1so1559491iec.21 for ; Thu, 18 Sep 2014 09:26:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=BT0PdWFAk3K+zaoZ8LSi1aO9Zmj5mnB3sU2VUKmESb0=; b=riHWd5ke3eNcbjYC/fGZ9j4KHIvZ3BVQhfHIgyU8OzlM7eJzccySebklPRQNhKSoXc nCWaoVcJ6uiQhavBZJ3bf3h1cQwecvPekmUKI/3CfVrbPCnZWxsA6DieoEJbme7l1l/f 6wF6PDEhp3xD/szFVISSr2e4wnf1rVQKBfv/VtZ30v2Zn3bJJeEM4s7UEqwOyR0+bA2O CaMXg3sFJqT/c30OP2qdYWP22kduTLS77bRzNpjCUsrJY4Thpadrm8wJh3dlqqG8a/NX BUInHa53ILRq4O+zT1McMfUNDSH6EszK3s9ds8b2f6ho71bl2q0t/2mx7P82bJDYQIIj XrKA== MIME-Version: 1.0 X-Received: by 10.50.33.73 with SMTP id p9mr47895076igi.24.1411057569882; Thu, 18 Sep 2014 09:26:09 -0700 (PDT) Received: by 10.202.104.89 with HTTP; Thu, 18 Sep 2014 09:26:09 -0700 (PDT) Date: Thu, 18 Sep 2014 09:26:09 -0700 Message-ID: Subject: High intr CPU % and slow throughput From: Freddie Cash To: "freebsd-ipfw@freebsd.org" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Sep 2014 16:26:10 -0000 [Not sure if this is more appropriate for the -ipfw or -stable mailing lists.] 64-bit FreeBSD 10.0-p7 Dual-core AMD Opteron 1218 CPU @ 2.6 GHz =E2=80=8B2 GB of DDR2 RAM Intel i350-T4 quad-port gigabit NIC using igb(4) Each of the gigabit NIC ports are connected to gigabit links (we have a gigabit fibre link to our ISP, which has dual 10 Gbps links to the public Internet). Using the following simple ruleset (there are more rules, but these are the ones that match when we test transfers to/from the Internet): ipfw nat 8668 config ip 142.24. =E2=80=8Bx.y=E2=80=8B same_ports 10 allow ip from any to any via lo0 12 allow carp from any to any 20 reject log logamount 10000 ip from 10.0.0.0/8 to any in recv igb0 22 reject log logamount 10000 ip from 127.0.0.0/8 to any in recv igb0 =E2=80=8B2=E2=80=8B 4 reject log logamount 10000 ip from 172.16.0.0/20 to any in recv igb0 26 reject log logamount 10000 ip from 192.168.0.0/16 to any in recv igb0 50 skipto 65000 ip from 192.168.0.0/24 to not 142.24. =E2=80=8Bx.z /25 in recv igb2 52 skipto 65000 ip from not 142.24.13.128/25 to 142.24. =E2=80=8Bx.y in recv igb0 65000 allow ip from 192.168.0.0/24 to any in recv igb2 65002 nat 8668 ip from 192.168.0.0/24 to any out xmit igb0 65004 allow ip from 142.24. =E2=80=8Bx.y=E2=80=8B to any out xmit igb0 65006 nat 8668 ip from any to 142.24. =E2=80=8Bx.y=E2=80=8B in recv igb0 65008 allow ip from any to 192.168.0.0/24 in recv igb0 65010 allow ip from any to 192.168.0.0/24 out xmit igb2 When we start a large download or file transfer from the Internet (a single file from a single server), CPU usage for the [intr{irq256: igb0:que}] kernel thread jumps to over 90% (one CPU core) and causes all traffic through the firewall (even traffic that doesn't go through igb0) to grind to a standstill. Some TCP connections through other interfaces are even dropped.=E2=80=8B During this time, the other CPU core is under 50% usage. IIUIC, the [intr{irq256: igb0:que}] isn't showing actual CPU usage for processing hardware interrupts, but is showing the CPU usage used to process the packets going through IPFW. Correct? "vmstat -i" shows only 10-15 interrupts per second for each of the igb interfaces. The really depressing part is that throughput (as shown by "iftop -i igb0" and snmp graphing) never goes above 40 Mbps. :( What can I do to try and track down exactly why this is occurring? Is there anything I can do to reduce or mitigate this CPU usage? Or, is this simply a case of the CPU being too old? /boot/loader.conf currently has the following (been playing with most of these lately, without much change in CPU usage): ## Tune the igb(4) interfaces a little hw.igb.enable_aim=3D"1" hw.igb.enable_msix=3D"1" hw.igb.header_split=3D"0" hw.igb.max_interrupt_rate=3D"16000" hw.igb.num_queues=3D"0" hw.igb.rx_process_limit=3D"1000" hw.igb.rxd=3D"4096" hw.igb.txd=3D"4096" ## Configure kernel kern.hz=3D"4000" ## Configure IPFW net.inet.ip.fw.default_to_accept=3D"1" net.inet.ip.fw.verbose=3D"1" ## Configure network threads net.isr.bindthreads=3D"1" net.isr.direct=3D"1" net.isr.maxthreads=3D"2" =E2=80=8B/etc/sysctl.conf has the following (haven't changed these in a lon= g time): =E2=80=8B# IPFW options net.inet.ip.fw.autoinc_step=3D2 net.inet.ip.fw.enable=3D1 net.inet.ip.fw.one_pass=3D1 net.inet.ip.fw.verbose=3D1 net.inet.ip.fw.verbose_limit=3D10000 At lunch today, we'll be failing-over to the other firewall, which will be running without any /boot/loader.conf or /etc/sysctl.conf entries to see if my "optimisations" are actually "pessimisations". --=20 Freddie Cash fjwcash@gmail.com From owner-freebsd-ipfw@FreeBSD.ORG Thu Sep 18 16:42:57 2014 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BD080B33 for ; Thu, 18 Sep 2014 16:42:57 +0000 (UTC) Received: from mail.ipfw.ru (mail.ipfw.ru [IPv6:2a01:4f8:120:6141::2]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5753D32D for ; Thu, 18 Sep 2014 16:42:57 +0000 (UTC) Received: from secured.by.ipfw.ru ([95.143.220.47] helo=ws.su29.net) by mail.ipfw.ru with esmtpsa (TLSv1:DHE-RSA-AES128-SHA:128) (Exim 4.82 (FreeBSD)) (envelope-from ) id 1XUaoV-000KcY-Dq; Thu, 18 Sep 2014 16:27:55 +0400 Message-ID: <541B0B42.6050403@FreeBSD.org> Date: Thu, 18 Sep 2014 20:41:38 +0400 From: "Alexander V. Chernikov" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: Freddie Cash , "freebsd-ipfw@freebsd.org" Subject: Re: High intr CPU % and slow throughput References: In-Reply-To: X-Enigmail-Version: 1.6 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Sep 2014 16:42:57 -0000 On 18.09.2014 20:26, Freddie Cash wrote: > [Not sure if this is more appropriate for the -ipfw or -stable mailing > lists.] > > > 64-bit FreeBSD 10.0-p7 > > Dual-core AMD Opteron 1218 CPU @ 2.6 GHz > ​2 GB of DDR2 RAM > Intel i350-T4 quad-port gigabit NIC using igb(4) > > Each of the gigabit NIC ports are connected to gigabit links (we have a > gigabit fibre link to our ISP, which has dual 10 Gbps links to the public > Internet). > > Using the following simple ruleset (there are more rules, but these are the > ones that match when we test transfers to/from the Internet): Please show all the ruleset with counters. > > ipfw nat 8668 config ip 142.24. > ​x.y​ > same_ports > > 10 allow ip from any to any via lo0 > 12 allow carp from any to any > > 20 reject log logamount 10000 ip from 10.0.0.0/8 to any in recv igb0 > 22 reject log logamount 10000 ip from 127.0.0.0/8 to any in recv igb0 > ​2​ > 4 reject log logamount 10000 ip from 172.16.0.0/20 to any in recv igb0 > 26 reject log logamount 10000 ip from 192.168.0.0/16 to any in recv igb0 > > 50 skipto 65000 ip from 192.168.0.0/24 to not 142.24. > ​x.z > /25 in recv igb2 > 52 skipto 65000 ip from not 142.24.13.128/25 to 142.24. > ​x.y > in recv igb0 > > 65000 allow ip from 192.168.0.0/24 to any in recv igb2 > 65002 nat 8668 ip from 192.168.0.0/24 to any out xmit igb0 > 65004 allow ip from 142.24. > ​x.y​ > to any out xmit igb0 > > 65006 nat 8668 ip from any to 142.24. > ​x.y​ > in recv igb0 > 65008 allow ip from any to 192.168.0.0/24 in recv igb0 > 65010 allow ip from any to 192.168.0.0/24 out xmit igb2 > > When we start a large download or file transfer from the Internet (a single > file from a single server), CPU usage for the [intr{irq256: igb0:que}] > kernel thread jumps to over 90% (one CPU core) and causes all traffic > through the firewall (even traffic that doesn't go through igb0) to grind > to a standstill. Some TCP connections through other interfaces are even > dropped.​ During this time, the other CPU core is under 50% usage. can you do the following: kldload hwpmc sudo pmcstat -TS instructions -w 1 and show its output when the problem is observed? > > IIUIC, the [intr{irq256: igb0:que}] isn't showing actual CPU usage for > processing hardware interrupts, but is showing the CPU usage used to > process the packets going through IPFW. Correct? "vmstat -i" shows only > 10-15 interrupts per second for each of the igb interfaces. > > The really depressing part is that throughput (as shown by "iftop -i igb0" > and snmp graphing) never goes above 40 Mbps. :( > > What can I do to try and track down exactly why this is occurring? > > Is there anything I can do to reduce or mitigate this CPU usage? > > Or, is this simply a case of the CPU being too old? > > /boot/loader.conf currently has the following (been playing with most of > these lately, without much change in CPU usage): > > ## Tune the igb(4) interfaces a little > hw.igb.enable_aim="1" > hw.igb.enable_msix="1" > hw.igb.header_split="0" > hw.igb.max_interrupt_rate="16000" > hw.igb.num_queues="0" > hw.igb.rx_process_limit="1000" > hw.igb.rxd="4096" > hw.igb.txd="4096" > > ## Configure kernel > kern.hz="4000" > > ## Configure IPFW > net.inet.ip.fw.default_to_accept="1" > net.inet.ip.fw.verbose="1" > > ## Configure network threads > net.isr.bindthreads="1" > net.isr.direct="1" > net.isr.maxthreads="2" > > > ​/etc/sysctl.conf has the following (haven't changed these in a long time): > > ​# IPFW options > net.inet.ip.fw.autoinc_step=2 > net.inet.ip.fw.enable=1 > net.inet.ip.fw.one_pass=1 > net.inet.ip.fw.verbose=1 > net.inet.ip.fw.verbose_limit=10000 > > > At lunch today, we'll be failing-over to the other firewall, which will be > running without any /boot/loader.conf or /etc/sysctl.conf entries to see if > my "optimisations" are actually "pessimisations". > > From owner-freebsd-ipfw@FreeBSD.ORG Thu Sep 18 19:30:08 2014 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 348A0AA2 for ; Thu, 18 Sep 2014 19:30:08 +0000 (UTC) Received: from mail-oa0-x236.google.com (mail-oa0-x236.google.com [IPv6:2607:f8b0:4003:c02::236]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id F156C8CC for ; Thu, 18 Sep 2014 19:30:07 +0000 (UTC) Received: by mail-oa0-f54.google.com with SMTP id m19so930136oag.27 for ; Thu, 18 Sep 2014 12:30:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=0+Htn0l40EYLQrVo4lyZfL6jxAvPRZmnyldAWg8XMIA=; b=diLsgjOuvn0EjA2u41pEnWGRiSQREmwCyjSAYIpRQZFAxin+3wAXbNiEXOxrofcakO bp4brADxOIFGsiZ5EYMXmCkrM+Cdwb7HmXUuWS9sAnCLZcIHI0dfBzWW8v0BCC57Wgvw vp83P4Zrhozhnb69sjrL1ZneomC/ngCXEjnY/8tItquoCnI4Rve2L0bYyzGUwI0+mep9 +fGimMCa7mpc7lefXyGNZFYWttJn5mpBx0v+vsXy7AEK8CuuBEZq+xD9AHW1eNLo54ya ETVKrweqtQRLqo4nKE/q/dkDsjtCaN8sMCvD7PjSFszSbvTj6FHTUvBPyDcR0i/VFV0h LnBQ== MIME-Version: 1.0 X-Received: by 10.182.245.135 with SMTP id xo7mr888397obc.23.1411068607184; Thu, 18 Sep 2014 12:30:07 -0700 (PDT) Received: by 10.202.104.89 with HTTP; Thu, 18 Sep 2014 12:30:07 -0700 (PDT) In-Reply-To: References: Date: Thu, 18 Sep 2014 12:30:07 -0700 Message-ID: Subject: Re: High intr CPU % and slow throughput From: Freddie Cash To: "freebsd-ipfw@freebsd.org" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Sep 2014 19:30:08 -0000 =E2=80=8BAha! I believe I've found the cause of our current issue. In an effort to allow reloading of the firewall rules during the day without disconnecting anyone (dropping TCP connections), I started playing with rule sets. And everything appeared to be working wonderfully, in that I could restart the rules multiple times without dropping any packets or disconnecting anyone. But, CPU usage skyrocketed on large downloads and =E2=80=8B =E2=80=8Bwe were capped at a little less than 40 Mbps. :( It seems that if you do the following (at least twice, to make sure rules are in both sets), your CPU will melt: - clear set 1 - disable set 1 - load 4000 rules into set 1 - enable set 1 - swap sets 1 and 0 - disable set 1 =E2=80=8BI thought that would leave only the rules in set 0 active, which w= ould be the equivalent of only having loaded rules into set 0. However, it seems that ipfw still checks rules in disabled sets! Or does some kind of processing with disabled sets. pmcstat was showing lots (200-2000) of unresolved samples and ipfw.ko sitting at 80-90% in the list, even when CPU usage was around 30%. I did the above, but added "ipfw -f set 1 flush" as the last step, and everything is back to normal. pmstat is now empty (0 unresolved). We can now push 75 Mbps through the firewall with CPU usage under 80%. More importantly, though, other traffic is not impacted by large downloads and speedtests and streaming video! And, CPU usage is sitting at under 10% for "normal" traffic. =E2=80=8BYes, I know 4000 rules is =E2=80=8Ba lot (doing NAT for 66 systems= and 2 local subnets). Until now I was focusing on getting things working (migrating from FreeBSD 7 using IPFW+natd with lots of private IP to private IP rules; to FreeBSD 10 using IPFW + in-kernel NAT and proper double-NAT across networks using public IPs only). Optimisation work is just now beginning. :) --=20 Freddie Cash fjwcash@gmail.com From owner-freebsd-ipfw@FreeBSD.ORG Fri Sep 19 08:34:31 2014 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 738BCCC4 for ; Fri, 19 Sep 2014 08:34:31 +0000 (UTC) Received: from mail.ipfw.ru (mail.ipfw.ru [IPv6:2a01:4f8:120:6141::2]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 34C63C4E for ; Fri, 19 Sep 2014 08:34:31 +0000 (UTC) Received: from secured.by.ipfw.ru ([95.143.220.47] helo=ws.su29.net) by mail.ipfw.ru with esmtpsa (TLSv1:DHE-RSA-AES128-SHA:128) (Exim 4.82 (FreeBSD)) (envelope-from ) id 1XUpfL-0006Fu-Il; Fri, 19 Sep 2014 08:19:27 +0400 Message-ID: <541BEA45.6060001@FreeBSD.org> Date: Fri, 19 Sep 2014 12:33:09 +0400 From: "Alexander V. Chernikov" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: Freddie Cash , "freebsd-ipfw@freebsd.org" Subject: Re: High intr CPU % and slow throughput References: In-Reply-To: X-Enigmail-Version: 1.6 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Sep 2014 08:34:31 -0000 On 18.09.2014 23:30, Freddie Cash wrote: > ​Aha! I believe I've found the cause of our current issue. > > In an effort to allow reloading of the firewall rules during the day > without disconnecting anyone (dropping TCP connections), I started playing > with rule sets. And everything appeared to be working wonderfully, in that > I could restart the rules multiple times without dropping any packets or > disconnecting anyone. > > But, CPU usage skyrocketed on large downloads and ​ > > ​we were capped at a little less than 40 Mbps. :( > > It seems that if you do the following (at least twice, to make sure rules > are in both sets), your CPU will melt: > - clear set 1 > - disable set 1 > - load 4000 rules into set 1 > - enable set 1 > - swap sets 1 and 0 > - disable set 1 > > ​I thought that would leave only the rules in set 0 active, which would be > the equivalent of only having loaded rules into set 0. However, it seems > that ipfw still checks rules in disabled sets! Or does some kind of > processing with disabled sets. Yes. _All_ rules in all sets are referenced inside single array. ipfw does not process disabled rules itself, but 1) it has to load given rule to cpu cache 2) check if disabled set mask matches > > pmcstat was showing lots (200-2000) of unresolved samples and ipfw.ko > sitting at 80-90% in the list, even when CPU usage was around 30%. > > I did the above, but added "ipfw -f set 1 flush" as the last step, and > everything is back to normal. pmstat is now empty (0 unresolved). > > We can now push 75 Mbps through the firewall with CPU usage under 80%. > More importantly, though, other traffic is not impacted by large downloads > and speedtests and streaming video! And, CPU usage is sitting at under 10% > for "normal" traffic. > > ​Yes, I know 4000 rules is ​a lot (doing NAT for 66 systems and 2 local > subnets). Until now I was focusing on getting things working (migrating > from FreeBSD 7 using IPFW+natd with lots of private IP to private IP rules; > to FreeBSD 10 using IPFW + in-kernel NAT and proper double-NAT across > networks using public IPs only). Optimisation work is just now beginning. > :) > From owner-freebsd-ipfw@FreeBSD.ORG Fri Sep 19 17:35:12 2014 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 85358630 for ; Fri, 19 Sep 2014 17:35:12 +0000 (UTC) Received: from BAY004-OMC1S6.hotmail.com (bay004-omc1s6.hotmail.com [65.54.190.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "*.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 692E11E0 for ; Fri, 19 Sep 2014 17:35:12 +0000 (UTC) Received: from BAY182-W13 ([65.54.190.61]) by BAY004-OMC1S6.hotmail.com with Microsoft SMTPSVC(7.5.7601.22724); Fri, 19 Sep 2014 10:35:07 -0700 X-TMN: [2lcKQOBtURfgkXAz3k4922625i6R+9yX] X-Originating-Email: [vdiernieshugobeturt@outlook.com] Message-ID: From: Shubert Vernie To: "freebsd-ipfw@freebsd.org" Subject: Ipfw. it's u from kik chat? :) Date: Fri, 19 Sep 2014 17:35:06 +0000 Importance: Normal MIME-Version: 1.0 X-OriginalArrivalTime: 19 Sep 2014 17:35:07.0509 (UTC) FILETIME=[0DFAC250:01CFD430] Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Sep 2014 17:35:12 -0000 Hey ya.=2C sexy =3B) Come back=2C I miss u. check my profile I'm on now lets have fun again :) http://Chantal.terrilyntgilley.tumblr.com/webcam07 Waiting 4 U=2C=2C Joel Cincinnati manufacturer continues acquisition spree. Manufacturer Plans Boo= ne County Plant. Vampiric David Byrne Joins Arcade Fire for Suicide Cover. = China asserts paternal rights over Hong Kong in democracy clash. Crowds tur= n out to support aerodrome as vintage planes take to the sky. Some senior C= hinese cadres oppose universal suffrage for Hong Kong=2C says . GITAM Univ = Inks MoU with BARC for Research on Gamma Radiation. Virat Kohli Needs to Im= prove Poor Technique=2C Says Geoffrey Boycott. Nikon D750=2C a midrange ful= l frame revised (pictures). Tamron SP 15-30mm F/2.8 Di VC USD. Telltale Dro= ps Second Teaser For 'The Walking Dead Season 2: No Going Back . Brebires: = saison de transition pour l'Association sportive. US threatened to fine Yah= oo over PRISM. Buttocks Worth 60000: Transgender Woman Gets 16lb Silicon Pu= mped into Her . The Walking Dead Season Two Finale Trailer Released From Te= lltale Games. FIRST LOOK: Olympus M.Zuiko Digital 40-150mm f/2.8 PRO Lens. = San Diego s'illumine grce des lampes LED intelligentes. Rose Byrne goes ma= ke-up free as she arrives for performance of new play. Sophie Monk left sur= prised after seeing Kim Kardashian's famous derriere as . Walking or cyclin= g to work 'improves wellbeing'. Buttocks Worth 60000: Transgender Woman Get= s 16lb Silicon Pumped into Her . Snowden never expressed his concern regard= ing 'Operation Prism': NSA. The 'wow factor' of the Scorpion jet. Tales fro= m the Borderlands - Episode 1. Deputies Say $20000 Worth Of Crack Cocaine F= ound Hidden In Man's Shoes. Transmisin pelea Floyd Mayweather vs Chino Maid= ana en vivo Canal 7 online . West Coast Happenings. Julie Gayet : Charmeuse= et espigle Venise=2C l'actrice illumine la Mostra. Celebration=2C Serenit= y=2C And Mourning Part II (Moed Katan). Airam: El ao pasado no nos reponamo= s de una situacin adversa. India face arduous task to level the series at t= he Oval. Transmisin en vivo de la segunda pelea Floyd Mayweather vs. Marcos= Maidana. Court: Basking Ridge couple must pay Allstate $800000 after house= fire. Dont'a Hightower: Patriots' Run Defense Lacked Technique=2C Not Powe= r. Cleaveland: CPR still a go-to technique. + Mazel tov to Klauber and Manc= ini. HK ready for suffrage' as Leung slams veto threat. Les aurores borales= ont illumin le ciel aprs une violente tempte solaire. Filipe Lus finds sat= isfaction at conclusion of arduous journey. NFL Calendario y Transmisin En = Lnea Thrusday Night Football: Prediccin . Premier Austrian window and door= manufacturer picks Jacksonville for its U.S. . New Virtual Try-On Feature = Makes Buying Eyeglasses Online Fun and Convenient. Transelec presenta las m= ejores ofertas en licitacin de transmisin troncal para . Barcelona vs. Athl= etic Club=2C transmisin en vivo por DirecTV. US Gov't Threatened Yahoo If I= t Didn't Use PRISM. Pulling The Hard Stance: An Interview With Prostitutes.= Transelec se adjudica obras de transmisin troncal por US$100 millones. Tel= ltale Tweets Game of Thrones Teaser. Bomb Squad Detonates Suspicious Packag= e Found at Downtown Bus Station. Telltale Games releases new details about = 'Game of Thrones' video game series. First Bahamian ICAO Aerodrome Inspecto= rs Graduate. Fishing: An arduous occupation. CY warns pan-dems will 'have a= case to answer' if they block universal . Le mythique chanteur Cedric Myto= n illumine la butte de Plan-les-Ouates. Celebration=2C Serenity=2C And Mour= ning (Moed Katan 19a). Nikon D750 Officially Announced: A 24MP Full Frame D= SLR Designed for Photo . You're Storing Your Produce All Wrong. Le mythique= chanteur Cedric Myton illumine la butte de Plan-les-Ouates. Mazel Tov on y= our wedding Angelina Jolie and Brad Pitt. Bevestigingsartikelen in inches. = Comcast is fighting CenturyLink's Prism TV service=2C the telecom tells FCC= . Jetpack Manufacturer Martin Aircraft Raises $6.5 Million=2C Could List On= ASX . Il Sudtirol piega il Monza. Pistoiese in extremis. First Bahamians g= raduate from ICAO Aerodrome Inspectors Training Programme. Anggota DPRD Sum= ut Periode 2014-2019 Resmi Dilantik. US backs Hong Kong activists' push for= 'universal suffrage'. PaleyFest: O'Brien Chose to Go Public With CBS' 'Sc= orpion' to Help Others. Six of Orange County's biggest fires and what lesso= ns were learned. Comcast is fighting CenturyLink's Prism TV service=2C the = telecom tells FCC. Zenit vence in extremis antes da visita Luz. (CRONICA) = JOR. 4 Marbella FC 1-1 Cdiz CF Airam rescata a los amarillos del . Yahoo ca= ved to US PRISM threat (because money). YWCA joins 'Purple Purse Challenge'= . Valdagno=2C rischiata strage: appartamento saturo di gas=2C salvati in ex= tremis. FIRST LOOK: Olympus M.Zuiko Digital 40-150mm f/2.8 PRO Lens. Women'= s suffrage activist honored at Capitol. YWCA joins 'Purple Purse Challenge'= . The arduous task of proving your 'independence'. Family creates ruckus at= Aerodrome police. Lawmakers discuss HK electoral reform. Calciomercato Vic= enza=2C in extremis si cerca di chiudere Moretti. Basic Law stressed in HK = universal suffrage. Quantitative Gamma-H2AX Assay. Alonso: 'It was arduous'= . New Outbreak Of Anti-Semitic Graffiti Hits Miami Area. Family creates ruc= kus at Aerodrome police. Allstate looks to timber=2C real estate to prop up= ailing annuity division. Transmisin Pumas vs Tigres por Canal de las Estre= llas de Televisa en vivo . Researchers part water: 'Electric prism' separat= es water's nuclear spin states. Prism Plastics co-owner Jerry Williams dies= at 51. Women's suffrage activist to be honored at Capitol. It's a 100-poun= d girl Mazal tov to Israeli Safari's White Rhino mom. Springfield Township = addresses concerns about bus depot lighting. Barcelona vs. Athletic Club=2C= transmisin en vivo por DirecTV. 1-1: El Zaragoza pierde 'in extremis' ante= el Sabadell. Hong Kong Protesters March as Leader Urges 'Good Sense'. Woma= n suffrage lecture available free from UM professor for 100th anniversary. = match du soir Paul Pierce illumine New York en playoffs (2011). New Virtual= Try-On Feature Makes Buying Eyeglasses Online Fun and Convenient. Jennifer= Lopez steals spotlight at Fashion Rocks=2C exposes famous derriere in . Pa= leyFest: O'Brien Chose to Go Public With CBS' 'Scorpion' to Help Others. I= t's an early harvest for Mid-Columbia's Canoe Ridge Estate. Garth Brooks fe= els love in 1st show back at Rosemont's Allstate Arena. NJ Transit plans to= replace trains=2C buses and add seats. New intercalation technique may pav= e way for industrial-scale graphene . Kindness to wildlife pays off in the = garden. Transmisin Leones Negros vs Amrica en vivo online por ESPN 2 Liga M= X en . Transelec se adjudica obras de transmisin troncal por US$100 millone= s. Horario Arsenal vs River Plate. Transmisin TV Pblica en vivo. Byrne clai= ms two medals at Invictus Games. Trisha Yearwood Lights Up Allstate Arena o= n First Night of World Tour. Sharni Vinson pays tribute to Australian celeb= rity manager Mark Byrne. Court: Basking Ridge couple must pay Allstate $800= 000 after house fire. Season Two of 'The Walking Dead' video game lives up = to predecessor's . Refurbished/Replacement TCDs are available for legacy Va= rian GCs.. Some senior Chinese cadres oppose universal suffrage for Hong Ko= ng=2C says . Qu televisin ofrece la transmisin en vivo Real Madrid vs Atlti= co de Madrid . Chinese Officials Explain HK Universal Suffrage. Garth Brook= s launches comeback at Allstate Arena. No Moses Needed: Researchers Part Wa= ter With Electric Prism. The Walking Dead Season Two Finale Trailer Release= d From Telltale Games. Real Madrid vs. Atltico de Madrid=2C transmisin en v= ivo por ESPN. Are Brad Pitt and Angelina Jolie traitors to the gay communit= y?. Despite higher threshold=2C universal suffrage is an improvement: Starr= y Lee. CBS Drama 'Scorpion' Takes Stories From Computer Genius's Life=2C Wo= rk. Jennifer Lopez and Nicki Minaj put derrieres on display for Fashion Roc= ks. It's an early harvest for Mid-Columbia's Canoe Ridge Estate. Jennifer L= opez and Nicki Minaj put derrieres on display for Fashion Rocks. The Revolu= tionary Technique That Quietly Changed Machine Vision Forever. Une structur= e qui s'illumine grce aux dchets. Ogier wins Rally Australia. Making the mo= st out of metal. Pedestrian crossings get Kate Sheppard makeover. New Tyson= s Circulator bus routes get mixed reviews. Le mythique chanteur Cedric Myto= n illumine la butte de Plan-les-Ouates. Jennifer Lopez and Nicki Minaj put = derrieres on display for Fashion Rocks. Snowden never expressed his concern= regarding 'Operation Prism': NSA. L.A. Party Bus Pulls up to LAPD Station = After Double Shooting of Revelers. Anggota DPRD Sumut Periode 2014-2019 Res= mi Dilantik. Yahoo Fought the NSA's PRISM Program in Court. HK legislators = urged to promote universal suffrage. Stock Update: The Allstate Corporation= (NYSE:ALL) Federal Appeals Court . Palermo vs. Gallardo: River=2C el punt= ero=2C visita a Arsenal. Peenya Bus Stand Awaits Commuters. Dnde emiten en = vivo la final EEUU vs Serbia. Transmite ESPN. Paul Hurst accentuating posit= ives from arduous opening week for Grimsby Town. The Congress Challenges Bu= t Thrills. CBS's 'Scorpion' Cast and Producers Discuss the Agony of Genius.= Opinion: Approaching Yom Tov=2C we recall media bias. Dancing Skeletons=2C= Boys in High Heels and David Byrne: The Arcade Fire's . iPhone 6 Plus vs. = Samsung Galaxy Note 4: Key Features and Specifications . Jennifer Lopez fla= unts her derriere in unforgiving yoga pants during lunch . Apple Aperture: = So bertragen Sie Ihre Fotos in Adobe Lightroom. Lansing manufacturer Camero= n Tool plans expansion. Gen. Adekunle=2C 'The Black Scorpion'=2C Is Dead. U= niversal suffrage for Hong Kong. PaleyFest: O'Brien Chose to Go Public Wit= h CBS' 'Scorpion' to Help Others. Transmisin Arsenal vs River Plate en vivo= online por TV Pblica y ESPN+. Tamron SP 15-30mm F/2.8 Di VC USD. Phi Gamma= Delta Fraternity banned from University of Arizona until April 2019. World= War I aviation still alive at aerodrome in New York. City Manufacturing Ta= kes a Hit as Brooklyn Firm Plans Move to New Jersey. Parking Lot And Door T= imes Info For Garth Brooks Allstate Arena Shows. Are Brad Pitt and Angelina= Jolie traitors to the gay community?. iPhone 6 Plus vs. Samsung Galaxy Not= e 4: Key Features and Specifications . The Monkey in the Middle: Gamma Chis= Ready for Rush Next Week. Comcast is fighting CenturyLink's Prism TV servi= ce=2C the telecom tells FCC. Jonathan Bennett peeks at Jenna Johnson's derr= iere before DWTS rehearsals. Telltale Games Walking Dead Season 2 Episode 5= 'No Going Back' Release . Jane Byrne doesn't deserve this. Football / Vict= oire in-extremis de Nice face Metz - 13/09. Rare bird survives arduous 300= 0-mile journey from Africa to Britain only to . Officials offer $1.2 milli= on in incentives to Florida battery manufacturer. Schlegel: Mazel Tov to An= gie and Brad. Stillwater's Glasson finds Oak Tree National arduous. Scooter= Braun's Wife Yael Cohen Pregnant With First Child -- Congrats. Circle Inte= rchange to be renamed to honor Jane Byrne. Mazel Tov on your wedding Angeli= na Jolie and Brad Pitt. Transmisin en vivo de la etapa 20 de la Vuelta a Es= paa 2014. Allstate grant to help teen drivers in the Ozarks. Got bad memori= es? Scientists can use light technique to overwrite them. East Bay high sch= ool/community college scoreboard for Saturday=2C Aug. 30. Springfield Towns= hip addresses concerns about bus depot lighting. George Washington Bridge B= us Station Closes for Long-Delayed Upgrade. Pantech Opens New Corporate Hea= d Office In Pasir Gudang=2C Johor. Research and Markets: Global and Chinese= Gamma-Hexyl-gamma . Congressman Bradley Byrne backs Obama on ISIS=2C hopes= 'actions match his . onOne Perfect Photo Suite upgrade will compete with L= ightroom and Aperture. When Cara Delevingne bit Jourdan Dunn's derriere. Sa= fety barrier manufacturer moving US operations to Fuquay-Varina. Cleaveland= : CPR still a go-to technique. Transmisin Pumas vs Tigres por Canal de las = Estrellas de Televisa en vivo . 1-0: El APOEL gana 'in extremis' antes de v= isitar el Camp Nou. Man recovering after being stabbed near South Austin bu= s stop. Premier Austrian window and door manufacturer picks Jacksonville fo= r its U.S. . Yahoo resisted NSA Prism requests -- US government threatened = $250000 daily . Zenit vence in extremis antes da visita Luz. Garth Brooks = comeback tour kickoff at Allstate Arena in Rosemont. Letnick: Physical acti= vity does not have to be arduous. Immigrants Work in More Arduous Jobs than= U.S. Natives=2C New Study Shows. Pantech Opens New Corporate Head Office I= n Pasir Gudang=2C Johor. Scooter Braun's Wife Yael Cohen Pregnant With Firs= t Child -- Congrats. Dont'a Hightower: Patriots' Run Defense Lacked Techniq= ue=2C Not Power. Mazal Tov Scarlett Has Baby Girl. Chicago Wolves to extend= Allstate Arena contract for five more years. Allstate grant to help teen d= rivers in the Ozarks. Peach Days ripen for Celebration of our Heritage in H= urricane. Yahoo resisted NSA Prism requests -- US government threatened $25= 0000 daily . Australian celebrity manager Mark Byrne's sudden death rocks t= he industry. Polisi siaga amankan pelantikan DPRD Sumut. Maubourguet s'illu= mine pour une novillada nocturne. Semana desastrosa para imagen de la NFL. = Une traductrice sexy illumine la prsentation de Marin. Delta Kappa Gamma ho= lds gumbo competition. Smart Woman: New Technique Corrects Newborn Heart Pr= oblems. Telltale Games' 'The Walking Dead' season finale and #GamerGate: Ho= w . Pide OPS a pases de Amrica estar alerta a temporada de dengue. Calciome= rcato Vicenza=2C in extremis si cerca di chiudere Moretti. Occupy Central c= ooks up 'banquet' action plan of civil disobedience. (CRONICA) JOR. 4 Marbe= lla FC 1-1 Cdiz CF Airam rescata a los amarillos del . REVEALED: Yahoo foug= ht NSA PRISM program in court. Dick Newman: A boy in the 40s: Junior high f= ull of challenges. Technique Talk: Josh Barnett's pushing and pulling for c= atch wrestling's respect. Prism Plastics co-owner Jerry Williams dies at 51= . Lega Pro: Lecce=2C pari col Matera. Pistoiese=2C gol nel finale. Transmis= in en vivo del partido Arsenal vs. Manchester City=2C Liga Premier 2014. Th= e Congress Challenges But Thrills. New baldies protest at Beijing's unkinde= st cut. Tamron aims for high end with ultrawide zoom lens. Howes Percival = Redhill Aerodrome v Secretary of State for Communities and . Yahoo strikes = back at PRISM. Did others?. Old Rhinebeck Aerodrome hustles to raise funds = for Visitor Center. Teenager killed after car crashes into chemist at Kogar= ah in Sydney's south. The Walking Dead Season Two Finale Trailer Released F= rom Telltale Games. Comcast is fighting CenturyLink's Prism TV service=2C t= he telecom tells FCC. Lega Pro: Lecce=2C pari col Matera. Pistoiese=2C gol = nel finale. Nikon D750 Officially Announced: A 24MP Full Frame DSLR Designe= d for Photo . Airam: El ao pasado no nos reponamos de una situacin adversa.= Western Australia's grain harvest set to begin a full fortnight earlier th= an . Rita Ora flashes her bare derriere in tiny corset dress at NYFW Fashio= n Rocks . Jennifer Lopez flaunts her derriere in unforgiving yoga pants dur= ing lunch . Rare bird survives arduous 3000-mile journey from Africa to Bri= tain only to . Alonso: 'It was arduous'. Il Sudtirol piega il Monza. Pisto= iese in extremis. HTC One M8 vs Samsung Galaxy S5 Camera Comparison The Ca= meras Face . L.A. Party Bus Pulls up to LAPD Station After Double Shooting = of Revelers. Radioactive Cobalt Detected In A Supernova Explosion. Textron = Aviation CEO gives update on company at WIBA gathering. Bahrain Internation= al Airport secures aerodrome certification. Buttocks Worth 60000: Transgend= er Woman Gets 16lb Silicon Pumped into Her . US threatened massive fine to = force Yahoo to release data. Allstate Athlete of the Week: Craig Dawkins. P= antech Opens New Corporate Head Office In Pasir Gudang=2C Johor. Le mapping= vido=2C le phnomne qui illumine les villes. India face arduous task to lev= el the series at the Oval. Hong Kong divided over Beijing's proposal for de= mocracy. Beijing spells it out on 2017. Gamma radiation threat to Andhra=2C= Telangana. Silver Line bus stop near alternative high school key for stude= nts. Le mythique chanteur Cedric Myton illumine la butte de Plan-les-Ouates= . iProspect hires Mark Byrne for new general manager role. The Allstate Fou= ndation Invites Public to Fill Up Purple Purses: Online . El apoyo alemn sa= lva in extremis a Rajoy y da la comisara de Energa a Arias . 2014 Rally Aus= tralia - Press Conference. Mazel Tov on the Brangelina Wedding. Better year= ahead for Texans? Check telltale signs. Feds Threatened to Fine Yahoo $250= K Daily for Not Complying With PRISM. 1-0: El APOEL gana 'in extremis' ante= s de visitar el Camp Nou. Nikon D750 Officially Announced: A 24MP Full Fram= e DSLR Designed for Photo . Radiant Engineers Unveils New Heat Exchanger Pl= ant. REVEALED: Yahoo fought NSA PRISM program in court. Heavy rains bring s= corpions: Treating their stings=2C keeping them out. Basic Law stressed in = HK universal suffrage. Nikon D750=2C a midrange full frame revised (picture= s). Byrne claims two medals at Invictus Games. Bahrain Airport Company Obta= ins Aerodrome Certification for Bahrain . FIRST LOOK: Olympus M.Zuiko Digit= al 40-150mm f/2.8 PRO Lens. Garth Brooks launches comeback at Allstate Aren= a. Olympus adds 40-150mm f/2.8 zoom to PRO range.=20 =