Skip site navigation (1)Skip section navigation (2)
Date:      Sun, 31 Jul 2016 17:17:44 -0300
From:      "Dr. Rolf Jansen" <>
Cc:        Ian Smith <>
Subject:   Re: ipfw divert filter for IPv4 geo-blocking
Message-ID:  <>
In-Reply-To: <>
References:  <> <> <> <> <> <> <> <> <> <> <> <> <> <> <> <>

Next in thread | Previous in thread | Raw E-Mail | Index | Archive | Help
> Am 31.07.2016 um 15:38 schrieb Ian Smith <>:
> On Sat, 30 Jul 2016 11:17:13 -0300, Dr. Rolf Jansen wrote:
>> I finished the work on CIDR conformity of the IP ranges tables=20
>> generated by the tool geoip. The main constraint is that the start=20
>> and end address of an IP block given by the delegation files MUST BE=20=

>> PRESERVED during the transformation to a set of CIDR records. This=20
>> target is achieved by:
>> 1. Finding the largest common netmask boundary of the start address =
>>    int(log2(addr_count)); then iteration like Euclid's algorithm in =
>>    a GCD.
>> 2. Output the CIDR with the given start address and the masklen =
>>    to the found netmask.
>> 3. If the CIDR does not match the whole original IP range then set =
the start
>>    address of the next CIDR block to the next boundary of the common =
>>    and loop over starting at 1. until the original range has been =
>> I carefully tested the algorithm and a table that I pipe by the new=20=

>> geoip tool into ipfw is 100 % identical to the output of the ipfw=20
>> command 'table N list'.
> Great.  I suppose that caters for some of the odd delegations one =
> such as perhaps a /16 then a /15 (ie 3/4 of a /14) followed by maybe a=20=

> /12, maybw with another /15 tacked on the end .. but I'm unsure if =
> applies to country allocations as much as it does within countries.
>> It is worth to note, that already the original RIR delegation files=20=

>> contain 457 non CIDR conforming IPv4 ranges in a total of 165815=20
>> original records. I guess that this number will increase in the=20
>> future because the RIR's ran empty on new IPv4 ranges and are urged=20=

>> to subdivide returned old ranges for new delegations. The above=20
>> algorithm is ready for this.
> Yes, and just as well.  I'm surprised it's as few as 457 .. I looked=20=

> into it a bit back when 115.70/17 was first allocated to my ISP, after=20=

> previously having been, as I recall, in China .. so of course we fell=20=

> foul of a number of (probably perennially) out-of-date geoip blockers,=20=

> for months in some cases .. malevolent beasts if not kept well fed :)
>> Generally, CIDR conforming tables are more than twice as large as=20
>> optimized (joined adjacencies) IP range tables. All said changes have=20=

>> been pushed to GitHup already.
> So how many table entries does 'the world' come to, around 400,000?

No, it is not that bad. The total number of original entries in the =
delegation statistics files of all 5 RIR's is about 166000. The ipdb =
tool which compiles these ranges into a consolidated sorted binary =
table, that is suitable for loading it directly into a binary search =
tree, reduces the number of entries to a bit more than one half, namely =
ca. 83500.

Consolidation primarily means, resolving of overlaps, because these =
could not be handled in a meaningful way by a binary search tree. Only =
as an additional benefit in the same go, that routine combines =
adjacencies with the same country code, although, skipping the =
combination is technically not a show stopper for the BST, this is only =
to increase the performance.

The geoip tool which generates the tables of CIDR ranges per country =
code out of the consolidated tables would output a count of 167500 =
entries for all countries. That is a little bit more than the original =
count, however this table is still optimized, because original ranges =
that when combined form a new valid CIDR are not broken down again, but =
the combined CIDR is passed.

>> I am still a little bit amazed how ipfw come to accept incorrect CIDR=20=

>> ranges and arbitrarily moves the start/end addresses in order to=20
>> achieve CIDR conformity, and that without any further notice, and=20
>> that given that ipfw can be considered as being quite relevant to=20
>> system security. Or, may I assume that ipfw knows always better than=20=

>> the user what should be allowed or denied. Otherwise, perhaps I am=20
>> the only one ever who input incorrect CIDR ranges for processing by=20=

>> ipfw.
> You've lost me here, Rolf.  Do you mean that ipfw adds incorrect table=20=

> entries for a given IPv4 address and mask length?  Or that it c/should=20=

> itself accept IP ranges and generate the needed CIDR entries to match?

Perhaps an example may explain it better. Remember that the first =
incarnation of geoip passed the incorrect range to ipfw. =
This is an incorrect CIDR because the start address does not match a =
mask boundary defined by the given masklen. The point now is that this =
error is caused by EITHER the masklen is incorrect OR the start address =
is incorrect. ipfw can determine only that the CIDR is incorrect, and =
does rectify it for further processing:

  # ipfw table 1 add
  # ipfw table 1 list
  --> 0

So actually ipfw happily takes an incorrect CIDR and transforms this =
into a correct one under the arbitrary assumption that the masklen is =
the correct part and the start address is sort of variable. Technically =
the addition by ipfw is a correct CIDR, but this is not necessarily the =
range the user wanted to add.

> If the former, how to reproduce for a bug report?  If the latter, =
> you contemplate adding that functionality to ipfw

Well, in the meantime, I saw that this kind of automatic rectification =
of incorrect CIDR entries is within ipfw throughout.

  # ipfw add 50000 allow ip from to any
  -->  50000 allow ip from to any

At least in this case the user is informed directly about the CIDR that =
has been actually utilized, anyway, it is not exactly the range the user =
has asked for, and the tiny difference can easily be overseen.

Given that ipfw can't know what the user actually intended - did he =
mistype the start address or obtained somehow a wrong masklen -, I =
advocate that ipfw should accept the wrong CIDR and transform it to a =
correct one exactly as above, but should output a warning, either to the =
shell or to syslog in case no shell is connected. Ideally this warning =
would give a useful explanation why the input CIDR is wrong and how it =
could be made correct by either adjusting the masklen or the start =

> - or is ipfw better=20
> off being driven to generate tables from the output of such as geoip?

Of course, tools like geoip HAVE TO produce by 100 % valid CIDR, =
otherwise the tool is buggy and the bug must be fixed as in the cse of =

Of course, ipfw must not receive any so called "bug fix" that =
immediately may break lots of firewall installations around the world, =
given that it is so easy to inform invalid CIDR, and that ipfw happily =
accepted these up to now.

I don't want to look more catholic than the pope. My main concern is not =
that ipfw does a choice for correcting invalid CIDR, my concern is that =
the choice is sort of arbitrary. I advocate for logging a suitable =
warning, but keep on processing invalid CIDR's by transforming it to =
valid ones exactly as before.

Best regards


Want to link to this message? Use this URL: <>