From owner-freebsd-ipfw@freebsd.org Mon Jul 25 14:29:06 2016 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C405BBA4D7B for ; Mon, 25 Jul 2016 14:29:06 +0000 (UTC) (envelope-from rj@cyclaero.com) Received: from mo6-p00-ob.smtp.rzone.de (mo6-p00-ob.smtp.rzone.de [IPv6:2a01:238:20a:202:5300::9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.smtp.rzone.de", Issuer "TeleSec ServerPass DE-2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6968D164E for ; Mon, 25 Jul 2016 14:29:06 +0000 (UTC) (envelope-from rj@cyclaero.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; t=1469456942; l=967; s=domk; d=cyclaero.com; h=Mime-Version:To:Date:Subject:Content-Transfer-Encoding:Content-Type: From; bh=/wH+f/obFQzLKYzujk2Xn/TKYJ+a0z1fW/0lj6elFo8=; b=Hnd7fbfgIZaiakUkfsBKI36U/fvaarQxfvxRBTDSbhahFmL8u38Uq2GJy0jI/BFXtWs wzb0x99rS0jJn/7inn3/JgB3NrynjlLoPmUF7IGZJxgKGgKdqV1Wqfro49E5xvMbCpw2z Jd+f8ER4FT+8dsqDWNCt6fftja1a+uAjDgU= X-RZG-AUTH: :PmYkdlmrd/5oFO+v3d3wrWNKA6HwNiCiTtCELX2wAfChnSFe7Pbi4CZ28Nw= X-RZG-CLASS-ID: mo00 Received: from rolf.projectworld.net (bb02aae1.virtua.com.br [187.2.170.225]) by smtp.strato.de (RZmta 38.13 DYNA|AUTH) with ESMTPSA id j0b8ees6PET0LYJ (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (curve secp521r1 with 521 ECDH bits, eq. 15360 bits RSA)) (Client did not present a certificate) for ; Mon, 25 Jul 2016 16:29:00 +0200 (CEST) From: "Dr. Rolf Jansen" Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Subject: ipfw divert filter for IPv4 geo-blocking Message-Id: <61DFB3E2-6E34-4EEA-8AC6-70094CEACA72@cyclaero.com> Date: Mon, 25 Jul 2016 11:28:58 -0300 To: freebsd-ipfw@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Jul 2016 14:29:06 -0000 I have written a ipfw divert filter daemon for IPv4 geo-blocking. It is = working flawlessly on two server installations since a week. Anyway, I am still in doubt whether I do the blocking in the correct = way. Once the filter receives a packet from the respective divert socket = it looks up the country code of the source IP in the IP-Ranges database, = and if the country code shall be allowed then it returns the unaltered = packet via said socket, otherwise, the filter does no further = processing, so the packet is effectively gone, lost, dropped, discarded, = or whatever would be the correct terminology. Is this the really the = correct way of denying a packet, or is it necessary to inform ipfw = somehow about the circumstances, so it can run a proper dropping = procedure? I uploaded the filter + accompanying tools to GitHub https://github.com/cyclaero/ipdb Many thnaks for any advices in advance. Best regards Rolf =20 From owner-freebsd-ipfw@freebsd.org Mon Jul 25 15:47:29 2016 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F31FBBA42ED for ; Mon, 25 Jul 2016 15:47:29 +0000 (UTC) (envelope-from kudzu@tenebras.com) Received: from mail-qt0-x236.google.com (mail-qt0-x236.google.com [IPv6:2607:f8b0:400d:c0d::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C33171E7B for ; Mon, 25 Jul 2016 15:47:29 +0000 (UTC) (envelope-from kudzu@tenebras.com) Received: by mail-qt0-x236.google.com with SMTP id x25so99398623qtx.2 for ; Mon, 25 Jul 2016 08:47:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tenebras-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc; bh=XZg3LCLdBme7zbeiY/ZEQKpK+nwSmKNuo5LPOCe5/iw=; b=tz6de0MrvQ9KqUjGI5MoI+HOMLBJAAorsl/0zMtD7S763ikPO21fPz5qVkE69Z8Pbp IN2N0gfv6eNn9movMW6On1Np38yA8JXw85qvjxBaf46Ib0wADaEOK+9sDVxsqmaml2QX 0pWuyLbBcM8qq5z11Tmq5gr1iliQgvWurNt+idWok8HiCnK0fJj628EfwJ8KMGzaRNGx MSluIsfbwX62fEx20J2ObLMNlQyRZadiVhSUe8+p00WfT7rjX5p2r/BjhY8jz/rEqGhE c2j1+EePFHsAqpM1mEIOThKCylP5FbkYnj9jePEXjoc5I2Inlvpi4fSlawRaCgJWtLkL QJeQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc; bh=XZg3LCLdBme7zbeiY/ZEQKpK+nwSmKNuo5LPOCe5/iw=; b=ijcaocoUPOZqGLyYj074PN6HQzUEF3h/8POoWKQQF14iTHfWVELgMu577KZ/7JZGtt qOPjkQnOIDk9iN/qQ5X/pB9dhjB0sLiU2QsOfX3uNGoYOVSIDPh7Ie7SOnOvm4U5pU75 LixaJey9HwqKMqvu1Tdu85KlOWXu9i9ODU4NgDLDHWmRDD1n9BDCk0Kd+SL2QZTHOY21 UN/wA9gQhPQRR7iaX/7VBvA7e9Ogxx8KNfaC9nao+Vpv/NN/fCcb2z4RMlqrro3+Km4e XpUZ2IBSraUYd8xcNAo5FQ7lNijean4AjotBcMP6wC46e+YmQGSWbnp0cmQtgWeKw5QH RrJQ== X-Gm-Message-State: AEkoouui1IYrDaMMP+whHt3PQOOsCz0JOG+h1FAuOu2Lt7idPD81VHXHN7KHGDqTSkuiRElNzA5Z0zGKKHik6lzE MIME-Version: 1.0 X-Received: by 10.237.37.99 with SMTP id w32mr29745862qtc.59.1469461648756; Mon, 25 Jul 2016 08:47:28 -0700 (PDT) Received: by 10.200.39.249 with HTTP; Mon, 25 Jul 2016 08:47:28 -0700 (PDT) Received: by 10.200.39.249 with HTTP; Mon, 25 Jul 2016 08:47:28 -0700 (PDT) In-Reply-To: <61DFB3E2-6E34-4EEA-8AC6-70094CEACA72@cyclaero.com> References: <61DFB3E2-6E34-4EEA-8AC6-70094CEACA72@cyclaero.com> Date: Mon, 25 Jul 2016 08:47:28 -0700 Message-ID: Subject: Re: ipfw divert filter for IPv4 geo-blocking From: Michael Sierchio To: "Dr. Rolf Jansen" Cc: freebsd-ipfw@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Jul 2016 15:47:30 -0000 Writing a divert daemon is a praiseworthy project, but I think you could do this without sending packets to user land. You could use tables - in fact, a single table of consolidated nets by country, in which the table entry is a CIDR block and the table arg is a country code - and you match on table arg. You could simply put nets you want to block in a table, and dispense with table args. That is how I do it. In order to do changes atomically, you need a pair of tables and a pair of rulesets, and you can swap rulesets when you have built the new table. On Jul 25, 2016 07:29, "Dr. Rolf Jansen" wrote: > I have written a ipfw divert filter daemon for IPv4 geo-blocking. It is > working flawlessly on two server installations since a week. > > Anyway, I am still in doubt whether I do the blocking in the correct way. > Once the filter receives a packet from the respective divert socket it > looks up the country code of the source IP in the IP-Ranges database, and > if the country code shall be allowed then it returns the unaltered packet > via said socket, otherwise, the filter does no further processing, so the > packet is effectively gone, lost, dropped, discarded, or whatever would be > the correct terminology. Is this the really the correct way of denying a > packet, or is it necessary to inform ipfw somehow about the circumstances, > so it can run a proper dropping procedure? > > I uploaded the filter + accompanying tools to GitHub > > https://github.com/cyclaero/ipdb > > Many thnaks for any advices in advance. > > Best regards > > Rolf > > > _______________________________________________ > freebsd-ipfw@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw > To unsubscribe, send any mail to "freebsd-ipfw-unsubscribe@freebsd.org" > From owner-freebsd-ipfw@freebsd.org Mon Jul 25 17:02:01 2016 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4AE85BA4CDB for ; Mon, 25 Jul 2016 17:02:01 +0000 (UTC) (envelope-from crest@rlwinm.de) Received: from smtp.rlwinm.de (smtp.rlwinm.de [IPv6:2a01:4f8:201:31ef::e]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1183A166B for ; Mon, 25 Jul 2016 17:02:01 +0000 (UTC) (envelope-from crest@rlwinm.de) Received: from vader9.bultmann.eu (unknown [87.253.189.132]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.rlwinm.de (Postfix) with ESMTPSA id CDEFF79E2 for ; Mon, 25 Jul 2016 19:01:57 +0200 (CEST) Subject: Re: ipfw divert filter for IPv4 geo-blocking To: freebsd-ipfw@freebsd.org References: <61DFB3E2-6E34-4EEA-8AC6-70094CEACA72@cyclaero.com> From: Jan Bramkamp Message-ID: <9d0a3ad8-a66a-c527-3906-3290b8d58476@rlwinm.de> Date: Mon, 25 Jul 2016 19:01:57 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 In-Reply-To: <61DFB3E2-6E34-4EEA-8AC6-70094CEACA72@cyclaero.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Jul 2016 17:02:01 -0000 On 25/07/16 16:28, Dr. Rolf Jansen wrote: > I have written a ipfw divert filter daemon for IPv4 geo-blocking. It is working flawlessly on two server installations since a week. > > Anyway, I am still in doubt whether I do the blocking in the correct way. Once the filter receives a packet from the respective divert socket it looks up the country code of the source IP in the IP-Ranges database, and if the country code shall be allowed then it returns the unaltered packet via said socket, otherwise, the filter does no further processing, so the packet is effectively gone, lost, dropped, discarded, or whatever would be the correct terminology. Is this the really the correct way of denying a packet, or is it necessary to inform ipfw somehow about the circumstances, so it can run a proper dropping procedure? > > I uploaded the filter + accompanying tools to GitHub > > https://github.com/cyclaero/ipdb > > Many thnaks for any advices in advance. I would use a set of IPFW tables with skipto/call tablearg rules instead. Use the daemon to maintain the IPFW tables. I assume your database is a list of of (CIDR, country code) pairs. In that case the daemon config should probably map from sets of country codes to table values e.g. table 1 { DE, NL } -> 10000, { US, UK } -> 10100 table 2 { CN, KO, TR } -> 20000 Next the daemon would calculate the minimal set of table entries to match these policies exactly and patch the kernel table contents if the database changes. This design avoids the userspace<->kernel copies without losing flexibility. From owner-freebsd-ipfw@freebsd.org Mon Jul 25 17:41:10 2016 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ED1B4BA3E08 for ; Mon, 25 Jul 2016 17:41:10 +0000 (UTC) (envelope-from rj@obsigna.com) Received: from mo6-p00-ob.smtp.rzone.de (mo6-p00-ob.smtp.rzone.de [IPv6:2a01:238:20a:202:5300::4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.smtp.rzone.de", Issuer "TeleSec ServerPass DE-2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8D5511197 for ; Mon, 25 Jul 2016 17:41:10 +0000 (UTC) (envelope-from rj@obsigna.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; t=1469468467; l=1628; s=domk; d=obsigna.com; h=To:References:Content-Transfer-Encoding:Cc:Date:In-Reply-To:From: Subject:Mime-Version:Content-Type; bh=H3clgvAmKCbwVdqv/uooJasJ4iOAQGEItXcMJPcYeHo=; b=OPsQRTgEwmHiCBjOpCdF1QWtULc/uJRWDctusBBis8Pudp9iT88MTwQrGImTABrP8LN 1pUm41gHrX1BaM0vsL7pdeXcII50LCRmLfL9eQG1tbso47UMOlPz7KrXbG8EfAZOxY6XT v71tv1cbsnMBfZZP9w9RRu8f7D7UyoqFUeM= X-RZG-AUTH: :O2kGeEG7b/pS1EK7WHa0hxqKZr4lnx6UhToX1IWHkW4X7v2ImaU2BqdKi+qzhv/Yf+zarg== X-RZG-CLASS-ID: mo00 Received: from mail.obsigna.com (bb02aae1.virtua.com.br [187.2.170.225]) by smtp.strato.de (RZmta 38.13 DYNA|AUTH) with ESMTPSA id L0a6f2s6PHf5M45 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (curve secp521r1 with 521 ECDH bits, eq. 15360 bits RSA)) (Client did not present a certificate); Mon, 25 Jul 2016 19:41:05 +0200 (CEST) Received: from rolf.projectworld.net (rolf.projectworld.net [192.168.222.25]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.obsigna.com (Postfix) with ESMTPSA id 4197F229861E; Mon, 25 Jul 2016 14:41:01 -0300 (BRT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: ipfw divert filter for IPv4 geo-blocking From: "Dr. Rolf Jansen" In-Reply-To: Date: Mon, 25 Jul 2016 14:41:00 -0300 Cc: Michael Sierchio , Jan Bramkamp Content-Transfer-Encoding: quoted-printable Message-Id: <4D047727-F7D0-4BEE-BD42-2501F44C9550@obsigna.com> References: <61DFB3E2-6E34-4EEA-8AC6-70094CEACA72@cyclaero.com> To: freebsd-ipfw@freebsd.org X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Jul 2016 17:41:11 -0000 > Am 25.07.2016 um 12:47 schrieb Michael Sierchio : >=20 > Writing a divert daemon is a praiseworthy project, but I think you = could do > this without sending packets to user land. >=20 > You could use tables - =E2=80=A6 > Am 25.07.2016 um 14:01 schrieb Jan Bramkamp : >=20 > I would use a set of IPFW tables with skipto/call tablearg rules = instead =E2=80=A6 Michael and Jan, many thanks for your suggestions. As everybody knows, 'Many roads lead to Rome.', and I am already there. = I don't feel alike going all the way back only for the sake of trying = out other routes. Once a week, the IP ranges are compiled from original sources into a = binary sorted table, containing as of today 83162 consolidated range/cc = pairs. On starting-up, the divert daemon reads the binary file in one = block and stores the ranges into a totally balanced binary search tree. = Looking-up a country code for a given IPv4 address in the BST takes on = average 20 nanoseconds on an AWS-EC2 micro instance. I don't know the = overhead of diverting, though. I guess this may be one or two orders of = magnitudes higher. Even though, I won't see any performance issues. Independent from the actual usage case (geo-blocking), let's talk about = divert filtering in general. The original question which is still = unanswered can be generalized to, whether "dropping/denying" a package = simply means 'forget about it' or whether the divert filter is required = to do something more involved, e.g. communicate the situation somehow to = ipfw. Best regards Rolf= From owner-freebsd-ipfw@freebsd.org Tue Jul 26 16:23:31 2016 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 906FEBA4918 for ; Tue, 26 Jul 2016 16:23:31 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vps1.elischer.org", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 629AA1271 for ; Tue, 26 Jul 2016 16:23:31 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from Julian-MBP3.local (ppp121-45-233-115.lns20.per1.internode.on.net [121.45.233.115]) (authenticated bits=0) by vps1.elischer.org (8.15.2/8.15.2) with ESMTPSA id u6QGNC0W067590 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Tue, 26 Jul 2016 09:23:20 -0700 (PDT) (envelope-from julian@freebsd.org) Subject: Re: ipfw divert filter for IPv4 geo-blocking To: "Dr. Rolf Jansen" , freebsd-ipfw@freebsd.org References: <61DFB3E2-6E34-4EEA-8AC6-70094CEACA72@cyclaero.com> <4D047727-F7D0-4BEE-BD42-2501F44C9550@obsigna.com> Cc: Michael Sierchio , Jan Bramkamp From: Julian Elischer Message-ID: Date: Wed, 27 Jul 2016 00:23:06 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 In-Reply-To: <4D047727-F7D0-4BEE-BD42-2501F44C9550@obsigna.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Jul 2016 16:23:31 -0000 On 26/07/2016 1:41 AM, Dr. Rolf Jansen wrote: >> Am 25.07.2016 um 12:47 schrieb Michael Sierchio : >> >> Writing a divert daemon is a praiseworthy project, but I think you could do >> this without sending packets to user land. >> >> You could use tables - … > >> Am 25.07.2016 um 14:01 schrieb Jan Bramkamp : >> >> I would use a set of IPFW tables with skipto/call tablearg rules instead … > Michael and Jan, many thanks for your suggestions. > > As everybody knows, 'Many roads lead to Rome.', and I am already there. I don't feel alike going all the way back only for the sake of trying out other routes. and I personally am responsible for at least parts of several of them ;-) (parts of ipdivert, netgraph, and various ipfw bits). > > Once a week, the IP ranges are compiled from original sources into a binary sorted table, containing as of today 83162 consolidated range/cc pairs. On starting-up, the divert daemon reads the binary file in one block and stores the ranges into a totally balanced binary search tree. Looking-up a country code for a given IPv4 address in the BST takes on average 20 nanoseconds on an AWS-EC2 micro instance. I don't know the overhead of diverting, though. I guess this may be one or two orders of magnitudes higher. Even though, I won't see any performance issues. yes the diversion to user space is not a fast operation. When we wrote it, fast was 10Mbits/sec. The firewall tables use a radix tree (*) and might be slower than what you have, but possibly it might be made up for by not having to do the divert logic. it's not entorely clear from your description why you look up a country rather than just a pass/block result, but maybe different sources can access different countries?. I did similar once using ipfw tables but couldn't find a reliable source of data. > > Independent from the actual usage case (geo-blocking), let's talk about divert filtering in general. The original question which is still unanswered can be generalized to, whether "dropping/denying" a package simply means 'forget about it' or whether the divert filter is required to do something more involved, e.g. communicate the situation somehow to ipfw. there is no residual information about the packet in the kernel once it has been passed to the user process. so just "forgetting to hand it back" is sufficient to drop it. > > Best regards > > Rolf > _______________________________________________ > freebsd-ipfw@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw > To unsubscribe, send any mail to "freebsd-ipfw-unsubscribe@freebsd.org" > > From owner-freebsd-ipfw@freebsd.org Tue Jul 26 16:26:51 2016 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 52657BA498D for ; Tue, 26 Jul 2016 16:26:51 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vps1.elischer.org", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 30FA71315 for ; Tue, 26 Jul 2016 16:26:50 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from Julian-MBP3.local (ppp121-45-233-115.lns20.per1.internode.on.net [121.45.233.115]) (authenticated bits=0) by vps1.elischer.org (8.15.2/8.15.2) with ESMTPSA id u6QGQhnL067602 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO) for ; Tue, 26 Jul 2016 09:26:48 -0700 (PDT) (envelope-from julian@freebsd.org) Subject: Re: ipfw divert filter for IPv4 geo-blocking To: freebsd-ipfw@freebsd.org References: <61DFB3E2-6E34-4EEA-8AC6-70094CEACA72@cyclaero.com> <9d0a3ad8-a66a-c527-3906-3290b8d58476@rlwinm.de> From: Julian Elischer Message-ID: Date: Wed, 27 Jul 2016 00:26:37 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 In-Reply-To: <9d0a3ad8-a66a-c527-3906-3290b8d58476@rlwinm.de> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Jul 2016 16:26:51 -0000 On 26/07/2016 1:01 AM, Jan Bramkamp wrote: > > > On 25/07/16 16:28, Dr. Rolf Jansen wrote: >> I have written a ipfw divert filter daemon for IPv4 geo-blocking. >> It is working flawlessly on two server installations since a week. >> >> Anyway, I am still in doubt whether I do the blocking in the >> correct way. Once the filter receives a packet from the respective >> divert socket it looks up the country code of the source IP in the >> IP-Ranges database, and if the country code shall be allowed then >> it returns the unaltered packet via said socket, otherwise, the >> filter does no further processing, so the packet is effectively >> gone, lost, dropped, discarded, or whatever would be the correct >> terminology. Is this the really the correct way of denying a >> packet, or is it necessary to inform ipfw somehow about the >> circumstances, so it can run a proper dropping procedure? >> >> I uploaded the filter + accompanying tools to GitHub >> >> https://github.com/cyclaero/ipdb >> >> Many thnaks for any advices in advance. > > I would use a set of IPFW tables with skipto/call tablearg rules > instead. > Use the daemon to maintain the IPFW tables. I assume your database > is a list of of (CIDR, country code) pairs. In that case the daemon > config should probably map from sets of country codes to table > values e.g. > > table 1 { DE, NL } -> 10000, > { US, UK } -> 10100 > table 2 { CN, KO, TR } -> 20000 why multiple tables? if you load the table at once you can assign a country code as the tablearg for every run of addresses. all in one table. > > Next the daemon would calculate the minimal set of table entries to > match these policies exactly and patch the kernel table contents if > the database changes. > > This design avoids the userspace<->kernel copies without losing > flexibility. > _______________________________________________ > freebsd-ipfw@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw > To unsubscribe, send any mail to "freebsd-ipfw-unsubscribe@freebsd.org" > From owner-freebsd-ipfw@freebsd.org Tue Jul 26 17:40:31 2016 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 38A16BA583E for ; Tue, 26 Jul 2016 17:40:31 +0000 (UTC) (envelope-from kudzu@tenebras.com) Received: from mail-qt0-x233.google.com (mail-qt0-x233.google.com [IPv6:2607:f8b0:400d:c0d::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id ECD51122C for ; Tue, 26 Jul 2016 17:40:30 +0000 (UTC) (envelope-from kudzu@tenebras.com) Received: by mail-qt0-x233.google.com with SMTP id 52so14131004qtq.3 for ; Tue, 26 Jul 2016 10:40:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tenebras-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=itFycwFEvkMSPZ/1stvu/MF19zioTOm8valqQkOhUto=; b=AXru0WjW5H5YmKwgQ3FCHA+XtZS1lR3XCPZIl46klpU0/f+DD0SCkkkXvI7oMdH25s TFRvqs42mvb+PIoIzoKfMr1SV77ZkB+zkQWnTlNvvebV4etrhOs2rvrphUZoj8TPGJk3 BvfGQkAkyunURCAqHoNjgLIHiGeM/nvrQ23SLAbMfOtlK5GwmdHnmQ3Gd9NR2lNPE1g7 nrV7HTE/RY1onIFAiB3kXjr3BolMm8Lf0ZQUHrUJK8ge0j4W2+GmI1abvbCrpr5yvizN YpY/WPY+w/EiJSKjeW/3f3JqiJCQfR0OWfeMKd0w0Q9WUbcLedh5pTZB5M3HZD/woAzq k07w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=itFycwFEvkMSPZ/1stvu/MF19zioTOm8valqQkOhUto=; b=B8NLsrj9enL+XFLA6jtExmWFOkpkLaYY0ZALlHQv2/U5k15hajBCTGm1zfPoIZrfKr 5QLIE2F4qkDIONZVayDA0y66lYCKs/5Tqk0tdxCvb51VhhTfdefX4Sr78RQAvIee38CJ XaHsXOBIG98+dR7HlYrI8V84B7ayLz108dIFvLg53z5V6KzOAegSlYIL7hKhy0cCNO6s GaLFVqATBpp78eEMhbSsPO4dcigrw6A7LvBt+c81Jou2E4TlAHclK4UE9C5bwfX6PNXd 4ARZkTw4DzsQ5XUj/A1kQzda9tNael9YU2H6VgYUqwIaHwqyTzaEaAuPsof8mGDJb8zR Bbuw== X-Gm-Message-State: AEkoouuVEHFCtUKRun1pV5YkqwdAelH/rpLRC9aN6rfH11SV0XFhpTYTJkMTFxjrLNfqSLPyGXBirT++59sO6NiY X-Received: by 10.237.46.166 with SMTP id k35mr41743900qtd.93.1469554829749; Tue, 26 Jul 2016 10:40:29 -0700 (PDT) MIME-Version: 1.0 Received: by 10.200.39.249 with HTTP; Tue, 26 Jul 2016 10:40:29 -0700 (PDT) In-Reply-To: References: <61DFB3E2-6E34-4EEA-8AC6-70094CEACA72@cyclaero.com> <9d0a3ad8-a66a-c527-3906-3290b8d58476@rlwinm.de> From: Michael Sierchio Date: Tue, 26 Jul 2016 10:40:29 -0700 Message-ID: Subject: Re: ipfw divert filter for IPv4 geo-blocking To: "freebsd-ipfw@freebsd.org" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Jul 2016 17:40:31 -0000 On Tue, Jul 26, 2016 at 9:26 AM, Julian Elischer wrote= : table 1 { DE, NL } -> 10000, >> { US, UK } -> 10100 >> table 2 { CN, KO, TR } -> 20000 >> > why multiple tables? > if you load the table at once you can assign a country code as the > tablearg for every run of addresses. all in one table. I mentioned that in my earlier response - but if the point is to block entire countries (or any collection of CIDR blocks, for that matter), it's sufficient to have a whitelist table and a blacklist table. The table arg could also be a skipto rule number, right? And you can do policy-based routing, with the table arg as a FIB number. Passing the packet to userland via divert sockets was a brilliant idea in 2003. natd was pretty much the first NAT mechanism to properly handle ICMP error responses, too. --=20 "Well," Brahma said, "even after ten thousand explanations, a fool is no wiser, but an intelligent man requires only two thousand five hundred." - The Mah=C4=81bh=C4=81rata From owner-freebsd-ipfw@freebsd.org Tue Jul 26 19:05:46 2016 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8C3A6BA5105 for ; Tue, 26 Jul 2016 19:05:46 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vps1.elischer.org", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 699781805 for ; Tue, 26 Jul 2016 19:05:46 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from Julian-MBP3.local (ppp121-45-233-115.lns20.per1.internode.on.net [121.45.233.115]) (authenticated bits=0) by vps1.elischer.org (8.15.2/8.15.2) with ESMTPSA id u6QJ5eJU068078 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Tue, 26 Jul 2016 12:05:43 -0700 (PDT) (envelope-from julian@freebsd.org) Subject: Re: ipfw divert filter for IPv4 geo-blocking To: Michael Sierchio , "freebsd-ipfw@freebsd.org" References: <61DFB3E2-6E34-4EEA-8AC6-70094CEACA72@cyclaero.com> <9d0a3ad8-a66a-c527-3906-3290b8d58476@rlwinm.de> From: Julian Elischer Message-ID: <59d70c14-1524-8279-7b91-1620e2f688a7@freebsd.org> Date: Wed, 27 Jul 2016 03:05:35 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Jul 2016 19:05:46 -0000 On 27/07/2016 1:40 AM, Michael Sierchio wrote: > On Tue, Jul 26, 2016 at 9:26 AM, Julian Elischer wrote: > > table 1 { DE, NL } -> 10000, >>> { US, UK } -> 10100 >>> table 2 { CN, KO, TR } -> 20000 >>> >> why multiple tables? >> if you load the table at once you can assign a country code as the >> tablearg for every run of addresses. all in one table. > > I mentioned that in my earlier response - but if the point is to block > entire countries (or any collection of CIDR blocks, for that matter), it's > sufficient to have a whitelist table and a blacklist table. The table arg > could also be a skipto rule number, right? And you can do policy-based > routing, with the table arg as a FIB number. > > Passing the packet to userland via divert sockets was a brilliant idea in > 2003. natd was pretty much the first NAT mechanism to properly handle ICMP > error responses, too. 2003? nahh we wrote it and divert in 96 :-) > From owner-freebsd-ipfw@freebsd.org Tue Jul 26 19:06:25 2016 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 79FECBA519E for ; Tue, 26 Jul 2016 19:06:25 +0000 (UTC) (envelope-from rj@obsigna.com) Received: from mo6-p00-ob.smtp.rzone.de (mo6-p00-ob.smtp.rzone.de [IPv6:2a01:238:20a:202:5300::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.smtp.rzone.de", Issuer "TeleSec ServerPass DE-2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E7EE61860; Tue, 26 Jul 2016 19:06:24 +0000 (UTC) (envelope-from rj@obsigna.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; t=1469559980; l=9980; s=domk; d=obsigna.com; h=To:References:Cc:Date:In-Reply-To:From:Subject:Mime-Version: Content-Type; bh=MopXyfQZAiw7O+HFoWB+zO6uOE6C6jUWt3AnpgcngaM=; b=hY6HEqEs3B+SdqKNo7a1NWf6ZUR8Wtf/0sKE+vETL63LgMnrhfigL8YhujgFeKmr8uT iFlHnAspUyF47T25zL7RG8ePtQ8ldMl8hm+EX6MPshojqDAnp9SK8/x0F0B4Gu6vitcJy mcdT2pv0v9pRRJIvUmOFadCstsqhFYAZLMk= X-RZG-AUTH: :O2kGeEG7b/pS1EK7WHa0hxqKZr4lnx6UhToX1IWHkW4X7v2ImaU2BqdKi+qzh/nYybNiUg== X-RZG-CLASS-ID: mo00 Received: from mail.obsigna.com (bb02a6dc.virtua.com.br [187.2.166.220]) by smtp.strato.de (RZmta 38.13 DYNA|AUTH) with ESMTPSA id L01561s6QJ6KVG1 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (curve secp521r1 with 521 ECDH bits, eq. 15360 bits RSA)) (Client did not present a certificate); Tue, 26 Jul 2016 21:06:20 +0200 (CEST) Received: from rolf.projectworld.net (rolf.projectworld.net [192.168.222.25]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.obsigna.com (Postfix) with ESMTPSA id 4FDE82298622; Tue, 26 Jul 2016 16:06:16 -0300 (BRT) Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: ipfw divert filter for IPv4 geo-blocking From: "Dr. Rolf Jansen" In-Reply-To: Date: Tue, 26 Jul 2016 16:06:14 -0300 Cc: Julian Elischer Message-Id: <9641D08A-0501-4AA2-9DF6-D5AFE6CB2975@obsigna.com> References: <61DFB3E2-6E34-4EEA-8AC6-70094CEACA72@cyclaero.com> <4D047727-F7D0-4BEE-BD42-2501F44C9550@obsigna.com> To: freebsd-ipfw@freebsd.org X-Mailer: Apple Mail (2.3124) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Jul 2016 19:06:25 -0000 > Am 26.07.2016 um 13:23 schrieb Julian Elischer : > On 26/07/2016 1:41 AM, Dr. Rolf Jansen wrote: >> Once a week, the IP ranges are compiled from original sources into a = binary sorted table, containing as of today 83162 consolidated range/cc = pairs. On starting-up, the divert daemon reads the binary file in one = block and stores the ranges into a totally balanced binary search tree. = Looking-up a country code for a given IPv4 address in the BST takes on = average 20 nanoseconds on an AWS-EC2 micro instance. I don't know the = overhead of diverting, though. I guess this may be one or two orders of = magnitudes higher. Even though, I won't see any performance issues. >=20 > yes the diversion to user space is not a fast operation. When we wrote = it, fast was 10Mbits/sec. > The firewall tables use a radix tree (*) and might be slower than what = you have, but possibly it might be made up for by not having to do the = divert logic. it's not entorely clear from your description why you look = up a country rather than just a pass/block result, but maybe different = sources can access different countries?. The basic idea was to develop a facility for ipfw for filtering IPv4 = packets by country code - see: https://github.com/cyclaero/ipdb I simply put into /etc/rc.conf: geod_enable=3D"YES" geod_flags=3D"-a DE:BR:US" The -a flag tells, that source IP addresses only from these countries = are allowed (i.e. passed through the filter). I added also a -d flag, = which means deny (i.e. drop packets) from the given list of countries. With that in place, I need to add a respective divert rule to the ipfw = ruleset (the divert port of the geod daemon is 8669, remembering that = 8668 is the port of the natd daemon): ipfw -q add 70 divert 8669 tcp from any to any 80,443 in recv WAN_if = setup > I did similar once using ipfw tables but couldn't find a reliable = source of data. The IP/CC database is compiled from downloads of the daily published = delegation statistics files of the 5 RIR's. I consider the RIR's being = the authoritative source. Anyway, on my systems the IP/CC-database is = updated only weekly, although, daily updating would be possible. I wrote = a shell script for this, that can be executed by a cron job. https://github.com/cyclaero/ipdb/blob/master/ipdb-update.sh = There is another tool called geoip , that I uploaded to GitHub, and that = I use for looking up country codes by IP addresses on the command line. https://github.com/cyclaero/ipdb/blob/master/geoip.c This one could easily be extended to produce sorted IP ranges per CC = that could be fed into tables of ipfw. I am thinking of adding a command = line option for specifying CC's for which the IP ranges should be = exported, something like: geoip -e DE:BR:US:IT:FR:ES=20 And this could print sorted IP-Ranges belonging to the listed countries. = For this purpose, what would be the ideal format for directly feeding = the produced output into ipfw tables? >> Independent from the actual usage case (geo-blocking), let's talk = about divert filtering in general. The original question which is still = unanswered can be generalized to, whether "dropping/denying" a package = simply means 'forget about it' or whether the divert filter is required = to do something more involved, e.g. communicate the situation somehow to = ipfw. >=20 > there is no residual information about the packet in the kernel once = it has been passed to the user process. > so just "forgetting to hand it back" is sufficient to drop it. OK, many thanks, that just answers my original doubt. At least = technically, my daemon handles package dropping correctly, although, = more elegant ways can be imagined to do the same thing. Best regards Rolf=20= From owner-freebsd-ipfw@freebsd.org Wed Jul 27 02:03:13 2016 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 95256BA6783 for ; Wed, 27 Jul 2016 02:03:13 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vps1.elischer.org", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 73FCE14A5 for ; Wed, 27 Jul 2016 02:03:13 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from Julian-MBP3.local (ppp121-45-233-115.lns20.per1.internode.on.net [121.45.233.115]) (authenticated bits=0) by vps1.elischer.org (8.15.2/8.15.2) with ESMTPSA id u6R237ti069254 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Tue, 26 Jul 2016 19:03:10 -0700 (PDT) (envelope-from julian@freebsd.org) Subject: Re: ipfw divert filter for IPv4 geo-blocking To: "Dr. Rolf Jansen" , freebsd-ipfw@freebsd.org References: <61DFB3E2-6E34-4EEA-8AC6-70094CEACA72@cyclaero.com> <4D047727-F7D0-4BEE-BD42-2501F44C9550@obsigna.com> <9641D08A-0501-4AA2-9DF6-D5AFE6CB2975@obsigna.com> From: Julian Elischer Message-ID: <4d76a492-17ae-cbff-f92f-5bbbb1339aad@freebsd.org> Date: Wed, 27 Jul 2016 10:03:01 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 In-Reply-To: <9641D08A-0501-4AA2-9DF6-D5AFE6CB2975@obsigna.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Jul 2016 02:03:13 -0000 On 27/07/2016 3:06 AM, Dr. Rolf Jansen wrote: >> Am 26.07.2016 um 13:23 schrieb Julian Elischer : >> On 26/07/2016 1:41 AM, Dr. Rolf Jansen wrote: >>> Once a week, the IP ranges are compiled from original sources into a binary sorted table, containing as of today 83162 consolidated range/cc pairs. On starting-up, the divert daemon reads the binary file in one block and stores the ranges into a totally balanced binary search tree. Looking-up a country code for a given IPv4 address in the BST takes on average 20 nanoseconds on an AWS-EC2 micro instance. I don't know the overhead of diverting, though. I guess this may be one or two orders of magnitudes higher. Even though, I won't see any performance issues. >> yes the diversion to user space is not a fast operation. When we wrote it, fast was 10Mbits/sec. >> The firewall tables use a radix tree (*) and might be slower than what you have, but possibly it might be made up for by not having to do the divert logic. it's not entorely clear from your description why you look up a country rather than just a pass/block result, but maybe different sources can access different countries?. > The basic idea was to develop a facility for ipfw for filtering IPv4 packets by country code - see: https://github.com/cyclaero/ipdb > > I simply put into /etc/rc.conf: > > geod_enable="YES" > geod_flags="-a DE:BR:US" > > The -a flag tells, that source IP addresses only from these countries are allowed (i.e. passed through the filter). I added also a -d flag, which means deny (i.e. drop packets) from the given list of countries. > > With that in place, I need to add a respective divert rule to the ipfw ruleset (the divert port of the geod daemon is 8669, remembering that 8668 is the port of the natd daemon): > > ipfw -q add 70 divert 8669 tcp from any to any 80,443 in recv WAN_if setup > >> I did similar once using ipfw tables but couldn't find a reliable source of data. > The IP/CC database is compiled from downloads of the daily published delegation statistics files of the 5 RIR's. I consider the RIR's being the authoritative source. Anyway, on my systems the IP/CC-database is updated only weekly, although, daily updating would be possible. I wrote a shell script for this, that can be executed by a cron job. > > https://github.com/cyclaero/ipdb/blob/master/ipdb-update.sh > > There is another tool called geoip , that I uploaded to GitHub, and that I use for looking up country codes by IP addresses on the command line. > > https://github.com/cyclaero/ipdb/blob/master/geoip.c > > This one could easily be extended to produce sorted IP ranges per CC that could be fed into tables of ipfw. I am thinking of adding a command line option for specifying CC's for which the IP ranges should be exported, something like: > > geoip -e DE:BR:US:IT:FR:ES > > And this could print sorted IP-Ranges belonging to the listed countries. For this purpose, what would be the ideal format for directly feeding the produced output into ipfw tables? The format for using tables directly is the same as that used for routing tables. so imagine that you had to generate a routing table that sent packets to two different routers depending on their source. here's a simple rule set that filters web traffic by such a 'routing table' except it's routing to two different rules. It also sorts OUTGOING web traffic to the same rules. ipfw -q /dev/stdin <<-DONE # we hate this guy table 5 add 1.1.1.0/32 1000 # but all ow our people to visit everyone else in that subnet table 5 add 1.1.0.0/24 2000 # we block 1.1.2.0 through 1.1.3.255 table 5 add 1.1.2.0/23 1000 # but we allow 1.1.4.0 through to 1.1.7.255 table 5 add 1.1.4.0/22 2000 # etc table 5 add 1.1.8.0/21 1000 table 5 add 1.2.0.0/16 1000 table 5 add 0.0.0.0/0 2000 # default check-state # If we already decided what to do, do it # select out only external traffic, into direction specific rules. add 400 skipto 500 ip from any to any in recv WAN_if add 410 skipto 700 ip from any to any out xmit WAN_If add 320 skipto 10000 # Incoming packets add 500 skipto tablearg tcp from table(5) to any 80,443 setup keep-state # sort tcp setup packets between rules 1000 and 2000 add 600 skipto 10000 # outgoing packets add 700 skipto tablearg tcp from any to table(5) 80,443 setup keep-state # sort tcp setup packets between rules 100 and 2000 add 800 skipto 10000 add 1000 drop ip from any to any add 2000 allow ip from any to any # further processing add 10000 .. # further processing for non tcp DONE for full configurability you could have a rule for each country, and a number for it in the table: table 5 add 150.101.0.0/16 10610 # Australia [...] add 10610 block tcp from any to any 445 # only allow non encrypted web to those Aussie scum. add 10611 allow ip from any to any then by changing the rules at that location you could change the policy for a country without changing everything else. (the downside is that dynamic skipto's are not very efficient as they do a linear search of the rules, where static skiptos cache the location of the rule to skip to. it's not a terrible cost but it needs to be kept in mind. (but faster than a divert socket) your application becomes an application for configuring the firewall. (which you do by feeding commands down a pipe to ipfw, which is started as 'ipfw -q /dev/stdin') >>> Independent from the actual usage case (geo-blocking), let's talk about divert filtering in general. The original question which is still unanswered can be generalized to, whether "dropping/denying" a package simply means 'forget about it' or whether the divert filter is required to do something more involved, e.g. communicate the situation somehow to ipfw. >> there is no residual information about the packet in the kernel once it has been passed to the user process. >> so just "forgetting to hand it back" is sufficient to drop it. > OK, many thanks, that just answers my original doubt. At least technically, my daemon handles package dropping correctly, although, more elegant ways can be imagined to do the same thing. > > Best regards > > Rolf > _______________________________________________ > freebsd-ipfw@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw > To unsubscribe, send any mail to "freebsd-ipfw-unsubscribe@freebsd.org" > From owner-freebsd-ipfw@freebsd.org Wed Jul 27 13:36:16 2016 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 37DBDBA6F89 for ; Wed, 27 Jul 2016 13:36:16 +0000 (UTC) (envelope-from rj@obsigna.com) Received: from mo6-p00-ob.smtp.rzone.de (mo6-p00-ob.smtp.rzone.de [IPv6:2a01:238:20a:202:5300::8]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.smtp.rzone.de", Issuer "TeleSec ServerPass DE-2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A30361F0D; Wed, 27 Jul 2016 13:36:15 +0000 (UTC) (envelope-from rj@obsigna.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; t=1469626572; l=3070; s=domk; d=obsigna.com; h=To:References:Content-Transfer-Encoding:Cc:Date:In-Reply-To:From: Subject:Mime-Version:Content-Type; bh=urt9oqwpDIsM0HG00nwyKiit5mEaYR/3vXXqiqq6MIw=; b=hl1YH8oD0MpZ9BA4R2oq+gj0me7vyntdoYfaTAY/QUTnTUtoCZB7Q8JTAgNpWhEWw+4 cKKKTqV37/i6jlqUW9wvA/gBv52islCRLPHtNGQ05+tAMYYcBnU3wwRkdmnddPwmIf3Om gWhmlnQGVBuPxu+TbnfIiN0c6BS7rBgTFMY= X-RZG-AUTH: :O2kGeEG7b/pS1EK7WHa0hxqKZr4lnx6UhToX1IWHkW4X7v2ImaU2BqdKi+qzhvjYXRln X-RZG-CLASS-ID: mo00 Received: from mail.obsigna.com (bb02b159.virtua.com.br [187.2.177.89]) by smtp.strato.de (RZmta 38.13 DYNA|AUTH) with ESMTPSA id 60b96es6RDaAdL3 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (curve secp521r1 with 521 ECDH bits, eq. 15360 bits RSA)) (Client did not present a certificate); Wed, 27 Jul 2016 15:36:10 +0200 (CEST) Received: from rolf.projectworld.net (rolf.projectworld.net [192.168.222.25]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.obsigna.com (Postfix) with ESMTPSA id D08CA229861E; Wed, 27 Jul 2016 10:36:06 -0300 (BRT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: ipfw divert filter for IPv4 geo-blocking From: "Dr. Rolf Jansen" In-Reply-To: <4d76a492-17ae-cbff-f92f-5bbbb1339aad@freebsd.org> Date: Wed, 27 Jul 2016 10:36:05 -0300 Cc: Julian Elischer Content-Transfer-Encoding: quoted-printable Message-Id: References: <61DFB3E2-6E34-4EEA-8AC6-70094CEACA72@cyclaero.com> <4D047727-F7D0-4BEE-BD42-2501F44C9550@obsigna.com> <9641D08A-0501-4AA2-9DF6-D5AFE6CB2975@obsigna.com> <4d76a492-17ae-cbff-f92f-5bbbb1339aad@freebsd.org> To: freebsd-ipfw@freebsd.org X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Jul 2016 13:36:16 -0000 > Am 26.07.2016 um 23:03 schrieb Julian Elischer : > On 27/07/2016 3:06 AM, Dr. Rolf Jansen wrote: >> There is another tool called geoip , that I uploaded to GitHub, and = that I use for looking up country codes by IP addresses on the command = line. >>=20 >> https://github.com/cyclaero/ipdb/blob/master/geoip.c >>=20 >> This one could easily be extended to produce sorted IP ranges per CC = that could be fed into tables of ipfw. I am thinking of adding a command = line option for specifying CC's for which the IP ranges should be = exported, something like: >>=20 >> geoip -e DE:BR:US:IT:FR:ES >>=20 >> And this could print sorted IP-Ranges belonging to the listed = countries. For this purpose, what would be the ideal format for directly = feeding the produced output into ipfw tables? > The format for using tables directly is the same as that used for = routing tables. > =E2=80=A6 > table 5 add 1.1.1.0/32 1000 > =E2=80=A6 > your application becomes an application for configuring the firewall. > (which you do by feeding commands down a pipe to ipfw, which is = started as 'ipfw -q /dev/stdin') I finished adding a second usage form for the geoip tool, namely = generation of ipfw table construction directives filtered by country = codes. ______________ $ geoip -h geoip v1.0.1 (16), Copyright =C2=A9 2016 Dr. Rolf Jansen Usage: 1) look-up the country code belonging to an IPv4 address given by the = last command line argument: geoip [-r bstfile] [-h] a dotted IPv4 address to be looked-up. 2) generate a sorted list of IPv4 address/masklen pairs per country = code, formatted as ipfw table construction directives: geoip -t [CC:DD:EE:..] [-n table number] [-v table value] [-r = bstfile] [-h] -t [CC:DD:EE:..] output all IPv4 address/masklen pairs belonging = to the listed countries, given by 2 letter capital country codes, separated by colon. An = empty CC list means any country code. -n table number the ipfw table number between 0 and 65534 = [default: 0]. -v table value the 32-bit unsigned value of the ipfw table = entry [default: 0]. valid arguments in both usage forms: -r bstfile the path to the binary file with the = consolidated IP ranges that has been. generated by the 'ipdb' tool [default: = /usr/local/etc/ipdb/IPRanges/ipcc.bst]. -h show these usage instructions. ______________ With that, the ipfw configuration script may contain something alike: =E2=80=A6 # allow only web access from DE, BR, US: /usr/local/bin/geoip -t DE:BR:US -n 7 | /sbin/ipfw -q /dev/stdin /sbin/ipfw -q add 70 deny tcp from not table\(7\) to any 80,443 in = recv WAN_if setup =E2=80=A6 OR, the other way around: =E2=80=A6 # deny web access from certain disgraceful regions: /usr/local/bin/geoip -t KO:TR:SA:RU:GB -n 66 | /sbin/ipfw -q = /dev/stdin /sbin/ipfw -q add 70 allow tcp from not table\(66\) to any 80,443 in = recv WAN_if setup =E2=80=A6 ____________ Best regards Rolf From owner-freebsd-ipfw@freebsd.org Wed Jul 27 15:31:16 2016 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ADA4BBA2C1B for ; Wed, 27 Jul 2016 15:31:16 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vps1.elischer.org", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 933DA1EA1 for ; Wed, 27 Jul 2016 15:31:16 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from Julian-MBP3.local (ppp121-45-233-115.lns20.per1.internode.on.net [121.45.233.115]) (authenticated bits=0) by vps1.elischer.org (8.15.2/8.15.2) with ESMTPSA id u6RFVAAq072130 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Wed, 27 Jul 2016 08:31:13 -0700 (PDT) (envelope-from julian@freebsd.org) Subject: Re: ipfw divert filter for IPv4 geo-blocking To: "Dr. Rolf Jansen" , freebsd-ipfw@freebsd.org References: <61DFB3E2-6E34-4EEA-8AC6-70094CEACA72@cyclaero.com> <4D047727-F7D0-4BEE-BD42-2501F44C9550@obsigna.com> <9641D08A-0501-4AA2-9DF6-D5AFE6CB2975@obsigna.com> <4d76a492-17ae-cbff-f92f-5bbbb1339aad@freebsd.org> From: Julian Elischer Message-ID: <677900fb-c717-743f-fcfe-86b603466e33@freebsd.org> Date: Wed, 27 Jul 2016 23:31:04 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Jul 2016 15:31:16 -0000 On 27/07/2016 9:36 PM, Dr. Rolf Jansen wrote: >> Am 26.07.2016 um 23:03 schrieb Julian Elischer : >> On 27/07/2016 3:06 AM, Dr. Rolf Jansen wrote: >>> There is another tool called geoip , that I uploaded to GitHub, and that I use for looking up country codes by IP addresses on the command line. >>> >>> https://github.com/cyclaero/ipdb/blob/master/geoip.c >>> >>> This one could easily be extended to produce sorted IP ranges per CC that could be fed into tables of ipfw. I am thinking of adding a command line option for specifying CC's for which the IP ranges should be exported, something like: >>> >>> geoip -e DE:BR:US:IT:FR:ES >>> >>> And this could print sorted IP-Ranges belonging to the listed countries. For this purpose, what would be the ideal format for directly feeding the produced output into ipfw tables? >> The format for using tables directly is the same as that used for routing tables. >> … >> table 5 add 1.1.1.0/32 1000 >> … >> your application becomes an application for configuring the firewall. >> (which you do by feeding commands down a pipe to ipfw, which is started as 'ipfw -q /dev/stdin') > I finished adding a second usage form for the geoip tool, namely generation of ipfw table construction directives filtered by country codes. wow, wonderful! with that tool, and ipfw tables we have a fully functional geo blocking/munging solution in about 4 lines of shell script. > ______________ > $ geoip -h > geoip v1.0.1 (16), Copyright © 2016 Dr. Rolf Jansen > > Usage: > > 1) look-up the country code belonging to an IPv4 address given by the last command line argument: > > geoip [-r bstfile] [-h] > a dotted IPv4 address to be looked-up. > > 2) generate a sorted list of IPv4 address/masklen pairs per country code, formatted as ipfw table construction directives: > > geoip -t [CC:DD:EE:..] [-n table number] [-v table value] [-r bstfile] [-h] > > -t [CC:DD:EE:..] output all IPv4 address/masklen pairs belonging to the listed countries, given by 2 letter > capital country codes, separated by colon. An empty CC list means any country code. > -n table number the ipfw table number between 0 and 65534 [default: 0]. > -v table value the 32-bit unsigned value of the ipfw table entry [default: 0]. > > valid arguments in both usage forms: > > -r bstfile the path to the binary file with the consolidated IP ranges that has been. > generated by the 'ipdb' tool [default: /usr/local/etc/ipdb/IPRanges/ipcc.bst]. > -h show these usage instructions. > ______________ > > With that, the ipfw configuration script may contain something alike: > > … > # allow only web access from DE, BR, US: > /usr/local/bin/geoip -t DE:BR:US -n 7 | /sbin/ipfw -q /dev/stdin > /sbin/ipfw -q add 70 deny tcp from not table\(7\) to any 80,443 in recv WAN_if setup > … > > OR, the other way around: > … > # deny web access from certain disgraceful regions: > /usr/local/bin/geoip -t KO:TR:SA:RU:GB -n 66 | /sbin/ipfw -q /dev/stdin > /sbin/ipfw -q add 70 allow tcp from not table\(66\) to any 80,443 in recv WAN_if setup > … > ____________ > > > Best regards > > Rolf > > > > From owner-freebsd-ipfw@freebsd.org Wed Jul 27 16:05:50 2016 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E14C0BA67ED for ; Wed, 27 Jul 2016 16:05:50 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from sola.nimnet.asn.au (paqi.nimnet.asn.au [115.70.110.159]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EEAA018D3; Wed, 27 Jul 2016 16:05:48 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from localhost (localhost [127.0.0.1]) by sola.nimnet.asn.au (8.14.2/8.14.2) with ESMTP id u6RFpOs4097836; Thu, 28 Jul 2016 01:51:25 +1000 (EST) (envelope-from smithi@nimnet.asn.au) Date: Thu, 28 Jul 2016 01:51:24 +1000 (EST) From: Ian Smith To: Julian Elischer cc: "Dr. Rolf Jansen" , Mike Makonnen , freebsd-ipfw@freebsd.org Subject: Re: ipfw divert filter for IPv4 geo-blocking In-Reply-To: <4d76a492-17ae-cbff-f92f-5bbbb1339aad@freebsd.org> Message-ID: <20160728004622.T29054@sola.nimnet.asn.au> References: <61DFB3E2-6E34-4EEA-8AC6-70094CEACA72@cyclaero.com> <4D047727-F7D0-4BEE-BD42-2501F44C9550@obsigna.com> <9641D08A-0501-4AA2-9DF6-D5AFE6CB2975@obsigna.com> <4d76a492-17ae-cbff-f92f-5bbbb1339aad@freebsd.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Jul 2016 16:05:51 -0000 On Wed, 27 Jul 2016 10:03:01 +0800, Julian Elischer wrote: > On 27/07/2016 3:06 AM, Dr. Rolf Jansen wrote: > > > Am 26.07.2016 um 13:23 schrieb Julian Elischer : > > > On 26/07/2016 1:41 AM, Dr. Rolf Jansen wrote: > > > > Once a week, the IP ranges are compiled from original sources into a > > > > binary sorted table, containing as of today 83162 consolidated range/cc > > > > pairs. On starting-up, the divert daemon reads the binary file in one > > > > block and stores the ranges into a totally balanced binary search tree. > > > > Looking-up a country code for a given IPv4 address in the BST takes on > > > > average 20 nanoseconds on an AWS-EC2 micro instance. I don't know the > > > > overhead of diverting, though. I guess this may be one or two orders of > > > > magnitudes higher. Even though, I won't see any performance issues. > > > yes the diversion to user space is not a fast operation. When we wrote > > > it, fast was 10Mbits/sec. > > > The firewall tables use a radix tree (*) and might be slower than what > > > you have, but possibly it might be made up for by not having to do the > > > divert logic. it's not entorely clear from your description why you look > > > up a country rather than just a pass/block result, but maybe different > > > sources can access different countries?. > > The basic idea was to develop a facility for ipfw for filtering IPv4 > > packets by country code - see: https://github.com/cyclaero/ipdb > > > > I simply put into /etc/rc.conf: > > > > geod_enable="YES" > > geod_flags="-a DE:BR:US" > > > > The -a flag tells, that source IP addresses only from these countries are > > allowed (i.e. passed through the filter). I added also a -d flag, which > > means deny (i.e. drop packets) from the given list of countries. > > > > With that in place, I need to add a respective divert rule to the ipfw > > ruleset (the divert port of the geod daemon is 8669, remembering that 8668 > > is the port of the natd daemon): > > > > ipfw -q add 70 divert 8669 tcp from any to any 80,443 in recv WAN_if > > setup > > > > > I did similar once using ipfw tables but couldn't find a reliable source > > > of data. > > The IP/CC database is compiled from downloads of the daily published > > delegation statistics files of the 5 RIR's. I consider the RIR's being the > > authoritative source. Anyway, on my systems the IP/CC-database is updated > > only weekly, although, daily updating would be possible. I wrote a shell > > script for this, that can be executed by a cron job. > > > > https://github.com/cyclaero/ipdb/blob/master/ipdb-update.sh > > > > > > There is another tool called geoip , that I uploaded to GitHub, and that I > > use for looking up country codes by IP addresses on the command line. > > > > https://github.com/cyclaero/ipdb/blob/master/geoip.c > > > > This one could easily be extended to produce sorted IP ranges per CC that > > could be fed into tables of ipfw. I am thinking of adding a command line > > option for specifying CC's for which the IP ranges should be exported, > > something like: > > > > geoip -e DE:BR:US:IT:FR:ES > > > > And this could print sorted IP-Ranges belonging to the listed countries. > > For this purpose, what would be the ideal format for directly feeding the > > produced output into ipfw tables? > The format for using tables directly is the same as that used for routing > tables. > so imagine that you had to generate a routing table that sent packets to two > different routers depending on their source. > > here's a simple rule set that filters web traffic by such a 'routing table' > except it's routing to two different rules. It also sorts OUTGOING web > traffic to the same rules. > > ipfw -q /dev/stdin <<-DONE > # we hate this guy > table 5 add 1.1.1.0/32 1000 Yeah, I've had trouble with him too .. > # but all ow our people to visit everyone else in that subnet > table 5 add 1.1.0.0/24 2000 > # we block 1.1.2.0 through 1.1.3.255 > table 5 add 1.1.2.0/23 1000 > # but we allow 1.1.4.0 through to 1.1.7.255 > table 5 add 1.1.4.0/22 2000 > # etc > table 5 add 1.1.8.0/21 1000 > table 5 add 1.2.0.0/16 1000 > table 5 add 0.0.0.0/0 2000 # default Now this was news to me. It's obvious, especially knowing it uses the same radix table method as does routing, but never occurred to me. Ta! > check-state # If we already decided what to do, do it > # select out only external traffic, into direction specific rules. > add 400 skipto 500 ip from any to any in recv WAN_if > add 410 skipto 700 ip from any to any out xmit WAN_If > add 320 skipto 10000 # a 420 moment? > # Incoming packets > add 500 skipto tablearg tcp from table(5) to any 80,443 setup keep-state # > sort tcp setup packets between rules 1000 and 2000 > add 600 skipto 10000 > # outgoing packets > add 700 skipto tablearg tcp from any to table(5) 80,443 setup keep-state # > sort tcp setup packets between rules 100 and 2000 > add 800 skipto 10000 > add 1000 drop ip from any to any > add 2000 allow ip from any to any > # further processing > add 10000 .. # further processing for non tcp > DONE > > for full configurability you could have a rule for each country, and a number > for it in the table: > table 5 add 150.101.0.0/16 10610 # Australia > [...] > add 10610 block tcp from any to any 445 # only allow non encrypted web to > those Aussie scum. Indeed :) > add 10611 allow ip from any to any > > then by changing the rules at that location you could change the policy for a > country without changing everything else. > (the downside is that dynamic skipto's are not very efficient as they do a > linear search of the rules, where static skiptos cache the location of the > rule to skip to. it's not a terrible cost but it needs to be kept in mind. > (but faster than a divert socket) I forget .. is that linear search from the beginning, or from the position of the rule querying the table? Just thnking about grouping skipto target rules to minimise traversal. These targets in turn could use static skiptos that will be cached. > your application becomes an application for configuring the firewall. > (which you do by feeding commands down a pipe to ipfw, which is started as > 'ipfw -q /dev/stdin') I went looking though ports for ipfw-classifyd, which attracted my interest in 2008, but seems never to have made it to ports. Written by Mike Makonnen (cc'd), it uses divert sockets with the linux- based 'l7' filters for detecting traffic from a wide array of UDP and TCP protocols, with the primary intent then of detecting various P2P traffic and shunting it through dummynet pipes for bandwidth limiting. It used a technique of modifying the (passed) ipfw rule number so that when returning packets to ipfw, depending on classification, it would (re)start at specific rule numbers which were the values assigned to particular protocols of interest .. which is kind of analagous to using tablearg values as skipto targets. This might be doubly slow, both from the diversion process and linear ruleset scan on return, but I thought was a neat solution for such classification, needing to run real-time in userland and so not amenable to table construction (as this one is ..) > > > > Independent from the actual usage case (geo-blocking), let's talk about > > > > divert filtering in general. The original question which is still > > > > unanswered can be generalized to, whether "dropping/denying" a package > > > > simply means 'forget about it' or whether the divert filter is required > > > > to do something more involved, e.g. communicate the situation somehow > > > > to ipfw. > > > there is no residual information about the packet in the kernel once it > > > has been passed to the user process. > > > so just "forgetting to hand it back" is sufficient to drop it. Checking up on that was the reason I looked at the ipfw-classifyd code again; some irrelevant packets there were just dropped, indeed. > > OK, many thanks, that just answers my original doubt. At least technically, > > my daemon handles package dropping correctly, although, more elegant ways > > can be imagined to do the same thing. > > > > Best regards > > > > Rolf Interesting discussion, and thanks for info on geoip tables etc. cheers, Ian From owner-freebsd-ipfw@freebsd.org Wed Jul 27 16:54:52 2016 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AA2ABBA656D for ; Wed, 27 Jul 2016 16:54:52 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vps1.elischer.org", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 85F721A22; Wed, 27 Jul 2016 16:54:52 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from Julian-MBP3.local (ppp121-45-233-115.lns20.per1.internode.on.net [121.45.233.115]) (authenticated bits=0) by vps1.elischer.org (8.15.2/8.15.2) with ESMTPSA id u6RGsbpP072456 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Wed, 27 Jul 2016 09:54:42 -0700 (PDT) (envelope-from julian@freebsd.org) Subject: Re: ipfw divert filter for IPv4 geo-blocking To: Ian Smith References: <61DFB3E2-6E34-4EEA-8AC6-70094CEACA72@cyclaero.com> <4D047727-F7D0-4BEE-BD42-2501F44C9550@obsigna.com> <9641D08A-0501-4AA2-9DF6-D5AFE6CB2975@obsigna.com> <4d76a492-17ae-cbff-f92f-5bbbb1339aad@freebsd.org> <20160728004622.T29054@sola.nimnet.asn.au> Cc: "Dr. Rolf Jansen" , Mike Makonnen , freebsd-ipfw@freebsd.org From: Julian Elischer Message-ID: <64148a94-ff8b-102f-992f-ca2d707ac61a@freebsd.org> Date: Thu, 28 Jul 2016 00:54:31 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 In-Reply-To: <20160728004622.T29054@sola.nimnet.asn.au> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Jul 2016 16:54:52 -0000 trimming.... On 27/07/2016 11:51 PM, Ian Smith wrote: > On Wed, 27 Jul 2016 10:03:01 +0800, Julian Elischer wrote: > [...] > > > country without changing everything else. > > (the downside is that dynamic skipto's are not very efficient as they do a > > linear search of the rules, where static skiptos cache the location of the > > rule to skip to. it's not a terrible cost but it needs to be kept in mind. > > (but faster than a divert socket) > > I forget .. is that linear search from the beginning, or from the > position of the rule querying the table? Just thnking about grouping > skipto target rules to minimise traversal. These targets in turn could > use static skiptos that will be cached. it starts searching forwards from the current location, to stop loops. (though it turns out you CAN make loops using some arcane sequences that I will not make public). However divert reinjection searches from the start to get to the place you want to restart processing. (but it's a very small loop) so put the diverts near the front if you can. > > > your application becomes an application for configuring the firewall. > > (which you do by feeding commands down a pipe to ipfw, which is started as > > 'ipfw -q /dev/stdin') > > I went looking though ports for ipfw-classifyd, which attracted my > interest in 2008, but seems never to have made it to ports. Written by > Mike Makonnen (cc'd), it uses divert sockets with the > linux- based 'l7' filters for detecting traffic from a wide array of UDP > and TCP protocols, with the primary intent then of detecting various P2P > traffic and shunting it through dummynet pipes for bandwidth limiting. I vaguely remember it. > > Interesting discussion, and thanks for info on geoip tables etc. > > cheers, Ian > From owner-freebsd-ipfw@freebsd.org Wed Jul 27 20:08:56 2016 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0AB58BA6B7F for ; Wed, 27 Jul 2016 20:08:56 +0000 (UTC) (envelope-from ohauer@gmx.de) Received: from mout.gmx.net (mout.gmx.net [212.227.17.22]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass DE-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6332B1551 for ; Wed, 27 Jul 2016 20:08:54 +0000 (UTC) (envelope-from ohauer@gmx.de) Received: from [192.168.100.100] ([87.139.233.65]) by mail.gmx.com (mrgmx102) with ESMTPSA (Nemesis) id 0MhNk6-1bgItU2Ijq-00McsL; Wed, 27 Jul 2016 22:08:42 +0200 Subject: Re: ipfw divert filter for IPv4 geo-blocking To: freebsd-ipfw@freebsd.org References: <61DFB3E2-6E34-4EEA-8AC6-70094CEACA72@cyclaero.com> <4D047727-F7D0-4BEE-BD42-2501F44C9550@obsigna.com> <9641D08A-0501-4AA2-9DF6-D5AFE6CB2975@obsigna.com> <4d76a492-17ae-cbff-f92f-5bbbb1339aad@freebsd.org> From: olli hauer Cc: "Dr. Rolf Jansen" Reply-To: freebsd-ipfw@freebsd.org Message-ID: Date: Wed, 27 Jul 2016 22:08:41 +0200 User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Provags-ID: V03:K0:l3cqATW2hgzwRl/T+fH1dsgiWyouY3Mmfq3ZENdGUqBHD5ticXv ysR+02JPqzOxBNtSPR5OipGeujQ/2pYZVlr6LgzQ86sb2AUcuOh7bUxLtAEYWkVfYPoXXEV NZlGFO5yjNo5915bckTiylupd5D3oGURXkNSnETSJX4QKgoLy1TfFcg2ff1IFIjGAnJmOpQ 82vcN25oieJTEeEQn3+zA== X-UI-Out-Filterresults: notjunk:1;V01:K0:r51phnoxP18=:YB+5BG4y6tkvIuLEIRUYA4 +4uppE/4miLpeOgmmzJG5UaBAupRMBeXmsPrCU+F2ExW3vwpUSf95slPv7kdbpXxrjd8weybT BesXSsGHGuiOpoAFPp9yMD4VuQv3LhzAYCjE2h41uMugLSdHT7VNKTxA09DCqTAbIja34BPmJ R+PIM/3xTPOUJSeJZwrpV1x8jVOmI2d9vLsCUQWPnQ28evZR47sjklcaSZ4AI9RT0i1Zi/K/r qbMRnatvyQyPhTD9H4jDjaSX0gO+hxvf5nKJAr51UQB1zPBiqqWDTs1aq9t1NkLB4DMXylHZp mzkHuwfct8VYr8oUUYMjQVnZW0AT8QRrsiHBBcwPc7oin9xX3zg6pxlj3QcxD57EPy0gjNUZK POyQz6SiXUHPS7TeaUZSBptvGHUl2c8wVkwcE5ZozB1z71zKPMfzgcnwXUtXgxcj/H9KailPc vtG5x7O1eah1Uh96IN/a+2eD/amQQqC3MBa+XUMw1MIm7o08RVDlPlTFQYUVxcD3mH3YHptAo /lh7CB/jNYx9/lpFjQ5rZTD4giewpP/XCYPymMCotohnakNBCLYHsd5yXEVQG8lzcClRCZVRV hdBych/7NIpLfJrJOpIF2xEy98jTtgCD+daQjoPl7dUxHEukEFEoMrgsnRoU+Exhlz047YZZn UD7/t7e7jJ3HoxB2SnCcNNqSzYfC9NX/PvRfEVYHct7+YmqSyxbK+oymJs2+50+KUIdTGBZPT oCU33eWa/hZa8SvpVZoRnYe0/tFOiOw4OCzF8cIBGxyf/JaOnWPE1Jp/rTmr5Ls25/WkbAXwH ggAq0vk X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Jul 2016 20:08:56 -0000 On 2016-07-27 15:36, Dr. Rolf Jansen wrote: >> Am 26.07.2016 um 23:03 schrieb Julian Elischer : >> On 27/07/2016 3:06 AM, Dr. Rolf Jansen wrote: >>> There is another tool called geoip , that I uploaded to GitHub, and that I use for looking up country codes by IP addresses on the command line. >>> >>> https://github.com/cyclaero/ipdb/blob/master/geoip.c >>> >>> This one could easily be extended to produce sorted IP ranges per CC that could be fed into tables of ipfw. I am thinking of adding a command line option for specifying CC's for which the IP ranges should be exported, something like: >>> >>> geoip -e DE:BR:US:IT:FR:ES >>> >>> And this could print sorted IP-Ranges belonging to the listed countries. For this purpose, what would be the ideal format for directly feeding the produced output into ipfw tables? >> The format for using tables directly is the same as that used for routing tables. >> … >> table 5 add 1.1.1.0/32 1000 >> … >> your application becomes an application for configuring the firewall. >> (which you do by feeding commands down a pipe to ipfw, which is started as 'ipfw -q /dev/stdin') > > I finished adding a second usage form for the geoip tool, namely generation of ipfw table construction directives filtered by country codes. > > ______________ > $ geoip -h > geoip v1.0.1 (16), Copyright © 2016 Dr. Rolf Jansen > > Usage: > > 1) look-up the country code belonging to an IPv4 address given by the last command line argument: > > geoip [-r bstfile] [-h] > a dotted IPv4 address to be looked-up. > > 2) generate a sorted list of IPv4 address/masklen pairs per country code, formatted as ipfw table construction directives: > > geoip -t [CC:DD:EE:..] [-n table number] [-v table value] [-r bstfile] [-h] > > -t [CC:DD:EE:..] output all IPv4 address/masklen pairs belonging to the listed countries, given by 2 letter > capital country codes, separated by colon. An empty CC list means any country code. > -n table number the ipfw table number between 0 and 65534 [default: 0]. > -v table value the 32-bit unsigned value of the ipfw table entry [default: 0]. > > valid arguments in both usage forms: > > -r bstfile the path to the binary file with the consolidated IP ranges that has been. > generated by the 'ipdb' tool [default: /usr/local/etc/ipdb/IPRanges/ipcc.bst]. > -h show these usage instructions. > ______________ > > With that, the ipfw configuration script may contain something alike: > > … > # allow only web access from DE, BR, US: > /usr/local/bin/geoip -t DE:BR:US -n 7 | /sbin/ipfw -q /dev/stdin > /sbin/ipfw -q add 70 deny tcp from not table\(7\) to any 80,443 in recv WAN_if setup > … > > OR, the other way around: > … > # deny web access from certain disgraceful regions: > /usr/local/bin/geoip -t KO:TR:SA:RU:GB -n 66 | /sbin/ipfw -q /dev/stdin > /sbin/ipfw -q add 70 allow tcp from not table\(66\) to any 80,443 in recv WAN_if setup > … > ____________ > > > Best regards > > Rolf Nice work :) Now it is also possible to use geoip to create files usable for pf. (just pipe the output through sed -e 's/table 0 add //') Perhaps the following diff for Makefile is useful. - use PREFIX instead hard coded path - use "install -s" instead "strip -x -o" - use "install -m" instead "cp ; chmod" --- Makefile.orig 2016-07-27 21:44:25.617707000 +0200 +++ Makefile 2016-07-27 21:47:18.341092000 +0200 @@ -43,6 +43,7 @@ CC = clang CFLAGS = $(CDEFS) -DSVNREV=\"$(REVNUM)\" -std=c11 -g0 -Ofast -mssse3 -Wno-parentheses -Wno-empty-body LDFLAGS = -lm +PREFIX ?= /usr/local HEADER = store.h SOURCES = store.c ipdb.c geoip.c geod.c @@ -71,9 +72,8 @@ update: clean all install: ipdb geoip geod - strip -x -o /usr/local/bin/ipdb ipdb - strip -x -o /usr/local/bin/geoip geoip - strip -x -o /usr/local/bin/geod geod - cp geod.rc /usr/local/etc/rc.d/geod - chmod 555 /usr/local/etc/rc.d/geod - cp ipdb-update.sh /usr/local/bin/ipdb-update.sh + install -s -m 555 ipdb ${PREFIX}/bin/ipdb + install -s -m 555 geoip ${PREFIX}/bin/geoip + install -s -m 555 geod ${PREFIX}/bin/geod + install -m 555 geod.rc ${PREFIX}/etc/rc.d/geod + install -m 555 ipdb-update.sh ${PREFIX}/bin/ipdb-update.sh -- olli From owner-freebsd-ipfw@freebsd.org Wed Jul 27 21:15:42 2016 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3227FBA60AD for ; Wed, 27 Jul 2016 21:15:42 +0000 (UTC) (envelope-from rj@obsigna.com) Received: from mo6-p00-ob.smtp.rzone.de (mo6-p00-ob.smtp.rzone.de [IPv6:2a01:238:20a:202:5300::9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.smtp.rzone.de", Issuer "TeleSec ServerPass DE-2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B79851792 for ; Wed, 27 Jul 2016 21:15:41 +0000 (UTC) (envelope-from rj@obsigna.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; t=1469654138; l=2810; s=domk; d=obsigna.com; h=To:References:Content-Transfer-Encoding:Cc:Date:In-Reply-To:From: Subject:Mime-Version:Content-Type; bh=zwm7xOuROwOhHvUaFF0eGTTOQ/hT1FoOgAgxsKbSDrY=; b=ZX8Xw5GGi1+Nf2FC74D4u0IGOeaYnVAcSLJSFrwbWTh26llJydQOuPjFwTIraDYh9h+ YDj1Nhhg+jCwTF/HxxJ2xG/NfEh6bjZjyt2b+9dNSQIvwANpKMR/tfOk/Cecv9tvsti6S 2gWUEQEEkWncwjuD5FJR79OAcCKwofamNqo= X-RZG-AUTH: :O2kGeEG7b/pS1EK7WHa0hxqKZr4lnx6UhToX1IWHkW4X7v2ImaU2BqdKi+qzhvjYXRln X-RZG-CLASS-ID: mo00 Received: from mail.obsigna.com (bb02b159.virtua.com.br [187.2.177.89]) by smtp.strato.de (RZmta 38.13 DYNA|AUTH) with ESMTPSA id 30a584s6RLFbfV9 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (curve secp521r1 with 521 ECDH bits, eq. 15360 bits RSA)) (Client did not present a certificate); Wed, 27 Jul 2016 23:15:37 +0200 (CEST) Received: from rolf.projectworld.net (rolf.projectworld.net [192.168.222.25]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.obsigna.com (Postfix) with ESMTPSA id 40336229861E; Wed, 27 Jul 2016 18:15:34 -0300 (BRT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: ipfw divert filter for IPv4 geo-blocking From: "Dr. Rolf Jansen" In-Reply-To: Date: Wed, 27 Jul 2016 18:15:33 -0300 Cc: olli hauer Content-Transfer-Encoding: quoted-printable Message-Id: References: <61DFB3E2-6E34-4EEA-8AC6-70094CEACA72@cyclaero.com> <4D047727-F7D0-4BEE-BD42-2501F44C9550@obsigna.com> <9641D08A-0501-4AA2-9DF6-D5AFE6CB2975@obsigna.com> <4d76a492-17ae-cbff-f92f-5bbbb1339aad@freebsd.org> To: freebsd-ipfw@freebsd.org X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Jul 2016 21:15:42 -0000 > Am 27.07.2016 um 17:08 schrieb olli hauer : > On 2016-07-27 15:36, Dr. Rolf Jansen wrote: >>=20 >> I finished adding a second usage form for the geoip tool, namely = generation of ipfw table construction directives filtered by country = codes. >>=20 >> ______________ >> $ geoip -h >> geoip v1.0.1 (16), Copyright =C2=A9 2016 Dr. Rolf Jansen >>=20 >> Usage: >>=20 >> 1) look-up the country code belonging to an IPv4 address given by the = last command line argument: >>=20 >> geoip [-r bstfile] [-h] >> a dotted IPv4 address to be looked-up. >>=20 >> 2) generate a sorted list of IPv4 address/masklen pairs per country = code, formatted as ipfw table construction directives: >>=20 >> geoip -t [CC:DD:EE:..] [-n table number] [-v table value] [-r = bstfile] [-h] >>=20 >> -t [CC:DD:EE:..] output all IPv4 address/masklen pairs = belonging to the listed countries, given by 2 letter >> capital country codes, separated by colon. An = empty CC list means any country code. >> -n table number the ipfw table number between 0 and 65534 = [default: 0]. >> -v table value the 32-bit unsigned value of the ipfw table = entry [default: 0]. >>=20 >> valid arguments in both usage forms: >>=20 >> -r bstfile the path to the binary file with the = consolidated IP ranges that has been. >> generated by the 'ipdb' tool [default: = /usr/local/etc/ipdb/IPRanges/ipcc.bst]. >> -h show these usage instructions. >> ______________ >>=20 >> With that, the ipfw configuration script may contain something alike: >>=20 >> =E2=80=A6 >> # allow only web access from DE, BR, US: >> /usr/local/bin/geoip -t DE:BR:US -n 7 | /sbin/ipfw -q /dev/stdin >> /sbin/ipfw -q add 70 deny tcp from not table\(7\) to any 80,443 in = recv WAN_if setup >> =E2=80=A6 >>=20 >> OR, the other way around: >> =E2=80=A6 >> # deny web access from certain disgraceful regions: >> /usr/local/bin/geoip -t KO:TR:SA:RU:GB -n 66 | /sbin/ipfw -q = /dev/stdin >> /sbin/ipfw -q add 70 allow tcp from not table\(66\) to any 80,443 = in recv WAN_if setup >> =E2=80=A6 >> ____________ >=20 > Nice work :) >=20 > Now it is also possible to use geoip to create files usable for pf. > (just pipe the output through sed -e 's/table 0 add //') >=20 > Perhaps the following diff for Makefile is useful. > - use PREFIX instead hard coded path > - use "install -s" instead "strip -x -o" > - use "install -m" instead "cp ; chmod" I changed the Makefile according to your suggestions, and I added = another command line option to the geoip tool: =E2=80=A6 -p plain IP table generation, i.e. without ipfw construction = directives, -n and -v flags are ignored. =E2=80=A6 The changes are already uploaded to GitHub. Best regards Rolf From owner-freebsd-ipfw@freebsd.org Thu Jul 28 03:20:38 2016 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4E58DBA6F53 for ; Thu, 28 Jul 2016 03:20:38 +0000 (UTC) (envelope-from ohauer@gmx.de) Received: from mout.gmx.net (mout.gmx.net [212.227.17.21]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass DE-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B8EFC1269 for ; Thu, 28 Jul 2016 03:20:36 +0000 (UTC) (envelope-from ohauer@gmx.de) Received: from [192.168.100.100] ([87.139.233.65]) by mail.gmx.com (mrgmx101) with ESMTPSA (Nemesis) id 0MLB89-1bSKkp0HeD-000Ozt; Thu, 28 Jul 2016 05:20:22 +0200 Subject: Re: ipfw divert filter for IPv4 geo-blocking To: freebsd-ipfw@freebsd.org References: <61DFB3E2-6E34-4EEA-8AC6-70094CEACA72@cyclaero.com> <4D047727-F7D0-4BEE-BD42-2501F44C9550@obsigna.com> <9641D08A-0501-4AA2-9DF6-D5AFE6CB2975@obsigna.com> <4d76a492-17ae-cbff-f92f-5bbbb1339aad@freebsd.org> Cc: "Dr. Rolf Jansen" Reply-To: freebsd-ipfw@freebsd.org From: olli hauer Message-ID: <49c5b646-e524-a1d2-1745-a03c43610819@gmx.de> Date: Thu, 28 Jul 2016 05:20:21 +0200 User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Provags-ID: V03:K0:7bwFGnglpGKnnNGHGBTYFvn+x+u5Zsh1C2fsOifqeX2t+Ad6fkA kBa6iWutv/xJO6RD8BaxkVKMM9gmE676avtxKIQOvzCdcrMkeB6grvFpP/mQsnXXXLdzqo6 8RNnP6V+sR90S7m8mrC9LRXozxFzLLQGMf/vivn2ZxuWHzmQKbx50R5PSGSBWwM1/4TV6u6 e/S8K5bv8djbmEGRo7IMA== X-UI-Out-Filterresults: notjunk:1;V01:K0:XlibDPsilDo=:1iFeJr6dTjZ+iFwNjxYycw A98hY3Eaj4P2DCHK0pyS3xzak9KGauB9g8+2dZwgVFe+3/sOrf+8eLwceKXkvHHrnCHzoqVmC XafXDA0Syom7tjfJ/Oeb4cFnGpbBIdxGqoB743LBCvM4f2DdW+b80+WzoxxNNPGI9vbQK5ykU Gt28RO4eRCnoMXE4RHz59yuP+m7YJrlUl+mB49T1mKS+Y2VMxbv3wOQQBb1kufOLXGNfDbQuF hhygmMGmwcXKQ4kjtcXJkWgmH/8GYxI0ODLuIDWQizE6KtrP3NsQ58elOILawGHiiVfFX3Mpn PSlAeJZEwCSN/U4qrH8ApIS5/u5PQw415EcSTy650kKaKMdsEBVnPmU/MEQtKWyUXfieL+Hnz UgtUM7Fm2DWvuDoDtSMOMOLIrznLlD+nS+2MGl7DCBaUuQMqXTfKnyVLBB39txEs8eE8EmrHI zZI+0Dibvr3LOYudQzrTi7gXHAdYtEEJaGov3AcjoaV1UpzQTRIwU+5iUaOwcuJrEoz9b580D TDWezwMDeenMx2Rv6cDTX0noYfT44k3QESSBlugZmgcr+Aey/VAWzkQtjCE8LPD+3zS1GdVm9 2B0udF7noS8yJvTrW7iwFJSJgJHZkoNSt5DeFUpWE3BCCgzyXIsC+EWguXih9sdYVjnW19GbA WJkrqfphwowDa5OejZUClqVtpNrWEBlA3h4VjGTXZlVIVgPw0KXNUQ/GCQ0Hgoz+dN1rBqcFE AnX1EanDL1X/a9mwOdh9PFntJ0jfdSgm2FVdeMUQOZvJ7ghb8zMKKIvyJglZJarx/v441oXlC knPkaes X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Jul 2016 03:20:38 -0000 On 2016-07-27 23:15, Dr. Rolf Jansen wrote: >> Am 27.07.2016 um 17:08 schrieb olli hauer : >> On 2016-07-27 15:36, Dr. Rolf Jansen wrote: >>> >>> I finished adding a second usage form for the geoip tool, namely generation of ipfw table construction directives filtered by country codes. >>> >>> ______________ >>> $ geoip -h >>> geoip v1.0.1 (16), Copyright © 2016 Dr. Rolf Jansen >>> >>> Usage: >>> >>> 1) look-up the country code belonging to an IPv4 address given by the last command line argument: >>> >>> geoip [-r bstfile] [-h] >>> a dotted IPv4 address to be looked-up. >>> >>> 2) generate a sorted list of IPv4 address/masklen pairs per country code, formatted as ipfw table construction directives: >>> >>> geoip -t [CC:DD:EE:..] [-n table number] [-v table value] [-r bstfile] [-h] >>> >>> -t [CC:DD:EE:..] output all IPv4 address/masklen pairs belonging to the listed countries, given by 2 letter >>> capital country codes, separated by colon. An empty CC list means any country code. >>> -n table number the ipfw table number between 0 and 65534 [default: 0]. >>> -v table value the 32-bit unsigned value of the ipfw table entry [default: 0]. >>> >>> valid arguments in both usage forms: >>> >>> -r bstfile the path to the binary file with the consolidated IP ranges that has been. >>> generated by the 'ipdb' tool [default: /usr/local/etc/ipdb/IPRanges/ipcc.bst]. >>> -h show these usage instructions. >>> ______________ >>> >>> With that, the ipfw configuration script may contain something alike: >>> >>> … >>> # allow only web access from DE, BR, US: >>> /usr/local/bin/geoip -t DE:BR:US -n 7 | /sbin/ipfw -q /dev/stdin >>> /sbin/ipfw -q add 70 deny tcp from not table\(7\) to any 80,443 in recv WAN_if setup >>> … >>> >>> OR, the other way around: >>> … >>> # deny web access from certain disgraceful regions: >>> /usr/local/bin/geoip -t KO:TR:SA:RU:GB -n 66 | /sbin/ipfw -q /dev/stdin >>> /sbin/ipfw -q add 70 allow tcp from not table\(66\) to any 80,443 in recv WAN_if setup >>> … >>> ____________ >> >> Nice work :) >> >> Now it is also possible to use geoip to create files usable for pf. >> (just pipe the output through sed -e 's/table 0 add //') >> >> Perhaps the following diff for Makefile is useful. >> - use PREFIX instead hard coded path >> - use "install -s" instead "strip -x -o" >> - use "install -m" instead "cp ; chmod" > > I changed the Makefile according to your suggestions, and I added another command line option to the geoip tool: > > … > -p plain IP table generation, i.e. without ipfw construction directives, -n and -v flags are ignored. > … > > The changes are already uploaded to GitHub. Thank you :) -- Regards, olli From owner-freebsd-ipfw@freebsd.org Fri Jul 29 02:21:12 2016 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B2CA6BA6E7A for ; Fri, 29 Jul 2016 02:21:12 +0000 (UTC) (envelope-from rj@obsigna.com) Received: from mo6-p00-ob.smtp.rzone.de (mo6-p00-ob.smtp.rzone.de [IPv6:2a01:238:20a:202:5300::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.smtp.rzone.de", Issuer "TeleSec ServerPass DE-2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2C6951BC2; Fri, 29 Jul 2016 02:21:11 +0000 (UTC) (envelope-from rj@obsigna.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; t=1469758868; l=2377; s=domk; d=obsigna.com; h=To:References:Content-Transfer-Encoding:Cc:Date:In-Reply-To:From: Subject:Mime-Version:Content-Type; bh=iLdAmQmHGRqG71qGeIyhytGU3qn8J8IXXD9QgL4bpz4=; b=jiljdxL5lPFgoYnn3JsPBIbXroi9/B96C4hG3S67jFKWFrkolNRM0fszbj6oUKEY9fE 35bUbQogWvO+UAUO4dDXvcmpn9D5V4WlHeqVewHYOSIS+H6+ZtNhBupCCkE7O2V46dnRU LB0NkWfujMDPtoLay++Uvdt1KlB5us0xEtU= X-RZG-AUTH: :O2kGeEG7b/pS1EK7WHa0hxqKZr4lnx6UhToX1IWHkW4X7v2ImaU2BqdKi+qzhvjYXRln X-RZG-CLASS-ID: mo00 Received: from mail.obsigna.com (bb02b159.virtua.com.br [187.2.177.89]) by smtp.strato.de (RZmta 38.13 DYNA|AUTH) with ESMTPSA id k071c4s6T2L6r5q (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (curve secp521r1 with 521 ECDH bits, eq. 15360 bits RSA)) (Client did not present a certificate); Fri, 29 Jul 2016 04:21:06 +0200 (CEST) Received: from rolf.projectworld.net (rolf.projectworld.net [192.168.222.25]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.obsigna.com (Postfix) with ESMTPSA id 0ED20229861E; Thu, 28 Jul 2016 23:21:02 -0300 (BRT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: ipfw divert filter for IPv4 geo-blocking From: "Dr. Rolf Jansen" In-Reply-To: <677900fb-c717-743f-fcfe-86b603466e33@freebsd.org> Date: Thu, 28 Jul 2016 23:21:01 -0300 Cc: Julian Elischer Content-Transfer-Encoding: quoted-printable Message-Id: <0D3C9016-7A4A-46BA-B35F-3844D07562A8@obsigna.com> References: <61DFB3E2-6E34-4EEA-8AC6-70094CEACA72@cyclaero.com> <4D047727-F7D0-4BEE-BD42-2501F44C9550@obsigna.com> <9641D08A-0501-4AA2-9DF6-D5AFE6CB2975@obsigna.com> <4d76a492-17ae-cbff-f92f-5bbbb1339aad@freebsd.org> <677900fb-c717-743f-fcfe-86b603466e33@freebsd.org> To: freebsd-ipfw@freebsd.org X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Jul 2016 02:21:12 -0000 > Am 27.07.2016 um 12:31 schrieb Julian Elischer : > On 27/07/2016 9:36 PM, Dr. Rolf Jansen wrote: >>> Am 26.07.2016 um 23:03 schrieb Julian Elischer : >>> On 27/07/2016 3:06 AM, Dr. Rolf Jansen wrote: >>>> There is another tool called geoip , that I uploaded to GitHub, and = that I use for looking up country codes by IP addresses on the command = line. >>>>=20 >>>> https://github.com/cyclaero/ipdb/blob/master/geoip.c >>>>=20 >>>> This one could easily be extended to produce sorted IP ranges per = CC that could be fed into tables of ipfw. I am thinking of adding a = command line option for specifying CC's for which the IP ranges should = be exported, something like: >>>>=20 >>>> geoip -e DE:BR:US:IT:FR:ES >>>>=20 >>>> And this could print sorted IP-Ranges belonging to the listed = countries. For this purpose, what would be the ideal format for directly = feeding the produced output into ipfw tables? >>> The format for using tables directly is the same as that used for = routing tables. >>> =E2=80=A6 >>> table 5 add 1.1.1.0/32 1000 >>> =E2=80=A6 >>> your application becomes an application for configuring the = firewall. >>> (which you do by feeding commands down a pipe to ipfw, which is = started as 'ipfw -q /dev/stdin') >> I finished adding a second usage form for the geoip tool, namely = generation of ipfw table construction directives filtered by country = codes. > wow, wonderful! >=20 > with that tool, and ipfw tables we have a fully functional geo = blocking/munging solution in about 4 lines of shell script. Unfortunately, I finally discovered that ipfw tables as they are, are = unsuitable for the given purpose, because for some reason ipfw mangles = about 20 % of the passed IP address/masklen pairs. For example: # ipfw table 1 add 201.222.20.0/20 # ipfw table 1 list --> 201.222.16.0/20 0 $ geoip 201.222.20.1 --> 201.222.20.1 in 201.222.20.0-201.222.31.255 in BR $ geoip 201.222.16.1 --> 201.222.16.1 in 201.222.16.0-201.222.19.255 in AR Effectively, I asked ipfw to add an IP-range of Brazil to table 1, but = it actually added another one which belongs to Argentina. This doesn't = make too much sense, does it? For the time being I switched my servers back to geo-blocking with the = divert filter daemon. Best regards Rolf From owner-freebsd-ipfw@freebsd.org Fri Jul 29 02:48:30 2016 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B72DABA756B for ; Fri, 29 Jul 2016 02:48:30 +0000 (UTC) (envelope-from leeb@ratnaling.org) Received: from mail-ua0-x22b.google.com (mail-ua0-x22b.google.com [IPv6:2607:f8b0:400c:c08::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 72F6815F6 for ; Fri, 29 Jul 2016 02:48:30 +0000 (UTC) (envelope-from leeb@ratnaling.org) Received: by mail-ua0-x22b.google.com with SMTP id j59so53011455uaj.3 for ; Thu, 28 Jul 2016 19:48:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ratnaling-org.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=fCLwJaz3xg6NsBmHJhhHEzZ0jMv2hmHD669ZrqAod6c=; b=OiOaiwCtazA9frJVptuntOQ6zSHDhBCo4d7/1v+T72Ml6sEp+htkoQYhXY8d4i24sa mnxD6I6cs04EAyTsaqpdZuhN8KASSrmqb/Ap0mE/chBYQzYpFp+uXXrIRtxN0zHVtQmV 28w6wIJrt5e8oOGnvDSkoEz1eEp+ifVDDkXQVnfHbTdBfDoNEXliy/Gsscq16KxS8k3m xVGdR2tx6zrFr+BOjVF4TLhEOwgIldHEirIxFlDDLtYo0aSwupN0gtO16VnV98y+ZO68 U6FlwYrQpnBbCbLncrEm/KjMFyxRpTqeNUmAiZ9ICg8pTJxJh1YLoFArXyMXMs0B3D67 BsaQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=fCLwJaz3xg6NsBmHJhhHEzZ0jMv2hmHD669ZrqAod6c=; b=k1aEMwdEQuLZElzUL7ai+Vlp6/lw77tIUcBks9VGHMLtoBkcllnCSzMnz3GKWWQIuL Y8LyN9+YX+K3C5RSVKmUkVisiFP6VTDfzQfEMK6eOImfNV9vQbiqR81WuSDwpYN3njwd 9XikHz6aFw5dvqkcWRvlOuRqL3ayNUF3e6qcTok5ONZjN0oM8Fqtfc/eqzPsxmETJdOQ dyFL56Ndkqv4CRqXp//jjr3kQZwUtYXqWbD2DGiBnSbqIopvBtj6aUU3yYjDhfTUTZWT YO4TpMLbrD36yEc8tJCfhG3qudQyWyPPM03Up7rAMS0xMsdKatqx1hIXidyE1AEaM6mF 0zKA== X-Gm-Message-State: AEkoouvGGc3kdkRYlspJ2qgLlISEZYBak3HIZ4ZvhVr3PMVOGnPexvcdZi0yyAfN/5Obo3i99w7q7NpKARUCyQ== X-Received: by 10.159.41.69 with SMTP id t63mr17883893uat.66.1469760509169; Thu, 28 Jul 2016 19:48:29 -0700 (PDT) MIME-Version: 1.0 Received: by 10.176.4.72 with HTTP; Thu, 28 Jul 2016 19:48:28 -0700 (PDT) In-Reply-To: <0D3C9016-7A4A-46BA-B35F-3844D07562A8@obsigna.com> References: <61DFB3E2-6E34-4EEA-8AC6-70094CEACA72@cyclaero.com> <4D047727-F7D0-4BEE-BD42-2501F44C9550@obsigna.com> <9641D08A-0501-4AA2-9DF6-D5AFE6CB2975@obsigna.com> <4d76a492-17ae-cbff-f92f-5bbbb1339aad@freebsd.org> <677900fb-c717-743f-fcfe-86b603466e33@freebsd.org> <0D3C9016-7A4A-46BA-B35F-3844D07562A8@obsigna.com> From: Lee Brown Date: Thu, 28 Jul 2016 19:48:28 -0700 Message-ID: Subject: Re: ipfw divert filter for IPv4 geo-blocking To: freebsd-ipfw@freebsd.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Jul 2016 02:48:30 -0000 That makes sense to me. Your /20 range encompasses 201.222.16.0 - 201.222.31.255. If you want 201.222.20.0-201.222.31.255, you'll need 3 ranges: 201.222.20.0/22 (201.222.20.0-201.222.23.255) 201.222.24.0/22 (201.222.24.0-201.222.27.255) 201.222.28.0/22 (201.222.28.0-201.222.31.255) this helps :) On Thu, Jul 28, 2016 at 7:21 PM, Dr. Rolf Jansen wrote: > > > Am 27.07.2016 um 12:31 schrieb Julian Elischer : > > On 27/07/2016 9:36 PM, Dr. Rolf Jansen wrote: > >>> Am 26.07.2016 um 23:03 schrieb Julian Elischer : > >>> On 27/07/2016 3:06 AM, Dr. Rolf Jansen wrote: > >>>> There is another tool called geoip , that I uploaded to GitHub, and > that I use for looking up country codes by IP addresses on the command li= ne. > >>>> > >>>> https://github.com/cyclaero/ipdb/blob/master/geoip.c > >>>> > >>>> This one could easily be extended to produce sorted IP ranges per CC > that could be fed into tables of ipfw. I am thinking of adding a command > line option for specifying CC's for which the IP ranges should be exporte= d, > something like: > >>>> > >>>> geoip -e DE:BR:US:IT:FR:ES > >>>> > >>>> And this could print sorted IP-Ranges belonging to the listed > countries. For this purpose, what would be the ideal format for directly > feeding the produced output into ipfw tables? > >>> The format for using tables directly is the same as that used for > routing tables. > >>> =E2=80=A6 > >>> table 5 add 1.1.1.0/32 1000 > >>> =E2=80=A6 > >>> your application becomes an application for configuring the firewall. > >>> (which you do by feeding commands down a pipe to ipfw, which is > started as 'ipfw -q /dev/stdin') > >> I finished adding a second usage form for the geoip tool, namely > generation of ipfw table construction directives filtered by country code= s. > > wow, wonderful! > > > > with that tool, and ipfw tables we have a fully functional geo > blocking/munging solution in about 4 lines of shell script. > > Unfortunately, I finally discovered that ipfw tables as they are, are > unsuitable for the given purpose, because for some reason ipfw mangles > about 20 % of the passed IP address/masklen pairs. > > For example: > > # ipfw table 1 add 201.222.20.0/20 > # ipfw table 1 list > --> 201.222.16.0/20 0 > > $ geoip 201.222.20.1 > --> 201.222.20.1 in 201.222.20.0-201.222.31.255 in BR > > $ geoip 201.222.16.1 > --> 201.222.16.1 in 201.222.16.0-201.222.19.255 in AR > > Effectively, I asked ipfw to add an IP-range of Brazil to table 1, but it > actually added another one which belongs to Argentina. This doesn't make > too much sense, does it? > > For the time being I switched my servers back to geo-blocking with the > divert filter daemon. > > Best regards > > Rolf > > _______________________________________________ > freebsd-ipfw@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw > To unsubscribe, send any mail to "freebsd-ipfw-unsubscribe@freebsd.org" From owner-freebsd-ipfw@freebsd.org Fri Jul 29 04:57:00 2016 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6FB2FBA6F2B for ; Fri, 29 Jul 2016 04:57:00 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vps1.elischer.org", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 43249100F for ; Fri, 29 Jul 2016 04:56:59 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from Julian-MBP3.local (ppp121-45-233-115.lns20.per1.internode.on.net [121.45.233.115]) (authenticated bits=0) by vps1.elischer.org (8.15.2/8.15.2) with ESMTPSA id u6T4ula8079317 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Thu, 28 Jul 2016 21:56:51 -0700 (PDT) (envelope-from julian@freebsd.org) Subject: Re: ipfw divert filter for IPv4 geo-blocking To: Lee Brown , freebsd-ipfw@freebsd.org, "Dr. Rolf Jansen" References: <61DFB3E2-6E34-4EEA-8AC6-70094CEACA72@cyclaero.com> <4D047727-F7D0-4BEE-BD42-2501F44C9550@obsigna.com> <9641D08A-0501-4AA2-9DF6-D5AFE6CB2975@obsigna.com> <4d76a492-17ae-cbff-f92f-5bbbb1339aad@freebsd.org> <677900fb-c717-743f-fcfe-86b603466e33@freebsd.org> <0D3C9016-7A4A-46BA-B35F-3844D07562A8@obsigna.com> From: Julian Elischer Message-ID: <089bea1b-dd31-aec1-c0e4-dba4ac6066aa@freebsd.org> Date: Fri, 29 Jul 2016 12:56:41 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Jul 2016 04:57:00 -0000 On 29/07/2016 10:48 AM, Lee Brown wrote: > That makes sense to me. Your /20 range encompasses 201.222.16.0 - > 201.222.31.255. > If you want 201.222.20.0-201.222.31.255, you'll need 3 ranges: whether it makes sense depends on whether you add the other ranges as well with the default result. Your daemon has the entire set loaded so it can exclude other internal ranges, however it looks like you have only provided ipfw with partial information. you'd need to add: 201.222.20.0/20 1 201.222.16.0/22 0 or more correctly what Lee showed above either the imput data is incorrect, or your cidr aggregation code has a bug. because 201.222.20.0/20 (well more correctly "misleading") cidr description). It implies 201.222.16.0/20 because if you mask 201.222.20.0 with 255.255.240.0 (/20) that's what you get. observe: soekris# route add 201.222.20.0/20 127.0.0.1 add net 201.222.20.0: gateway 127.0.0.1 soekris# netstat -rn Routing tables Internet: Destination Gateway Flags Refs Use Netif Expire default 192.168.0.1 UGS 0 9082198 vr2 [...] 201.222.16.0/20 127.0.0.1 US 0 0 lo0 [...] to make this work correctly you'd either need lee's answer, or you need to give it more information: 201.222.16.0/20 {result for brazil} 201.222.16.0/22 {default result, to get subtracted from the above. your program give s the correct result because it has all the information, including the AR information but you didn't tell ipfw about that. I suggest you need either input data sanitization, or to fix our output code to give the correct subranges. becasue 201.222.16.0/20 is definately bad input to ipfw. > > 201.222.20.0/22 (201.222.20.0-201.222.23.255) > 201.222.24.0/22 (201.222.24.0-201.222.27.255) > 201.222.28.0/22 (201.222.28.0-201.222.31.255) > > this helps :) > > On Thu, Jul 28, 2016 at 7:21 PM, Dr. Rolf Jansen wrote: > >>> Am 27.07.2016 um 12:31 schrieb Julian Elischer : >>> On 27/07/2016 9:36 PM, Dr. Rolf Jansen wrote: >>>>> Am 26.07.2016 um 23:03 schrieb Julian Elischer : >>>>> On 27/07/2016 3:06 AM, Dr. Rolf Jansen wrote: >>>>>> There is another tool called geoip , that I uploaded to GitHub, and >> that I use for looking up country codes by IP addresses on the command line. >>>>>> https://github.com/cyclaero/ipdb/blob/master/geoip.c >>>>>> >>>>>> This one could easily be extended to produce sorted IP ranges per CC >> that could be fed into tables of ipfw. I am thinking of adding a command >> line option for specifying CC's for which the IP ranges should be exported, >> something like: >>>>>> geoip -e DE:BR:US:IT:FR:ES >>>>>> >>>>>> And this could print sorted IP-Ranges belonging to the listed >> countries. For this purpose, what would be the ideal format for directly >> feeding the produced output into ipfw tables? >>>>> The format for using tables directly is the same as that used for >> routing tables. >>>>> … >>>>> table 5 add 1.1.1.0/32 1000 >>>>> … >>>>> your application becomes an application for configuring the firewall. >>>>> (which you do by feeding commands down a pipe to ipfw, which is >> started as 'ipfw -q /dev/stdin') >>>> I finished adding a second usage form for the geoip tool, namely >> generation of ipfw table construction directives filtered by country codes. >>> wow, wonderful! >>> >>> with that tool, and ipfw tables we have a fully functional geo >> blocking/munging solution in about 4 lines of shell script. >> >> Unfortunately, I finally discovered that ipfw tables as they are, are >> unsuitable for the given purpose, because for some reason ipfw mangles >> about 20 % of the passed IP address/masklen pairs. >> >> For example: >> >> # ipfw table 1 add 201.222.20.0/20 >> # ipfw table 1 list >> --> 201.222.16.0/20 0 >> >> $ geoip 201.222.20.1 >> --> 201.222.20.1 in 201.222.20.0-201.222.31.255 in BR >> >> $ geoip 201.222.16.1 >> --> 201.222.16.1 in 201.222.16.0-201.222.19.255 in AR >> >> Effectively, I asked ipfw to add an IP-range of Brazil to table 1, but it >> actually added another one which belongs to Argentina. This doesn't make >> too much sense, does it? >> >> For the time being I switched my servers back to geo-blocking with the >> divert filter daemon. >> >> Best regards >> >> Rolf >> >> _______________________________________________ >> freebsd-ipfw@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw >> To unsubscribe, send any mail to "freebsd-ipfw-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-ipfw@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw > To unsubscribe, send any mail to "freebsd-ipfw-unsubscribe@freebsd.org" > > From owner-freebsd-ipfw@freebsd.org Fri Jul 29 06:00:37 2016 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 13DDABA7E7D for ; Fri, 29 Jul 2016 06:00:37 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from sola.nimnet.asn.au (paqi.nimnet.asn.au [115.70.110.159]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 63C2B1177; Fri, 29 Jul 2016 06:00:35 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from localhost (localhost [127.0.0.1]) by sola.nimnet.asn.au (8.14.2/8.14.2) with ESMTP id u6T60O2Y075521; Fri, 29 Jul 2016 16:00:24 +1000 (EST) (envelope-from smithi@nimnet.asn.au) Date: Fri, 29 Jul 2016 16:00:24 +1000 (EST) From: Ian Smith To: "Dr. Rolf Jansen" cc: freebsd-ipfw@freebsd.org, Julian Elischer Subject: Re: ipfw divert filter for IPv4 geo-blocking In-Reply-To: <0D3C9016-7A4A-46BA-B35F-3844D07562A8@obsigna.com> Message-ID: <20160729151802.X29054@sola.nimnet.asn.au> References: <61DFB3E2-6E34-4EEA-8AC6-70094CEACA72@cyclaero.com> <4D047727-F7D0-4BEE-BD42-2501F44C9550@obsigna.com> <9641D08A-0501-4AA2-9DF6-D5AFE6CB2975@obsigna.com> <4d76a492-17ae-cbff-f92f-5bbbb1339aad@freebsd.org> <677900fb-c717-743f-fcfe-86b603466e33@freebsd.org> <0D3C9016-7A4A-46BA-B35F-3844D07562A8@obsigna.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Jul 2016 06:00:37 -0000 On Thu, 28 Jul 2016 23:21:01 -0300, Dr. Rolf Jansen wrote: > Am 27.07.2016 um 12:31 schrieb Julian Elischer : [..] >> wow, wonderful! >> with that tool, and ipfw tables we have a fully functional geo >> blocking/munging solution in about 4 lines of shell script. > Unfortunately, I finally discovered that ipfw tables as they are, are > unsuitable for the given purpose, because for some reason ipfw > mangles about 20 % of the passed IP address/masklen pairs. > For example: > # ipfw table 1 add 201.222.20.0/20 > # ipfw table 1 list > --> 201.222.16.0/20 0 > $ geoip 201.222.20.1 > --> 201.222.20.1 in 201.222.20.0-201.222.31.255 in BR > $ geoip 201.222.16.1 > --> 201.222.16.1 in 201.222.16.0-201.222.19.255 in AR Just to add to what Julian and Lee observed, testing IPs at (sourced from LACNIC thence whois.registro.br) inetnum: 201.222.20/22 aut-num: AS61902 abuse-c: CSJ45 owner: Bahialink - Technology ownerid: 004.724.687/0001-69 country: BR So the geoip result for 201.222.20.1 is just wrong - it should return 201.222.20.0 - 201.222.23.255 (ie, /22) and not 201.222.16.0 - 201.222.31.255 (ie, /20) While the range for 201.222.16.1 is in fact a /22: [..] inetnum: 201.222.16/22 status: allocated aut-num: N/A owner: G2KHosting S.A. ownerid: AR-GKSA-LACNIC responsible: Mauro Ferraro address: Maipu, 33, address: 2900 - San Nicolas de los Arroyos - BA country: AR > Effectively, I asked ipfw to add an IP-range of Brazil to table 1, > but it actually added another one which belongs to Argentina. This > doesn't make too much sense, does it? Not if geoip is returning the wrong address range for 201.222.20.1, no. > For the time being I switched my servers back to geo-blocking with > the divert filter daemon. I don't know what's wrong or where, just that it is .. How are you getting from geoip's IP range to /maskbits? cheers, Ian From owner-freebsd-ipfw@freebsd.org Fri Jul 29 08:54:08 2016 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8C0F7BA7345 for ; Fri, 29 Jul 2016 08:54:08 +0000 (UTC) (envelope-from rj@obsigna.com) Received: from mo6-p00-ob.smtp.rzone.de (mo6-p00-ob.smtp.rzone.de [IPv6:2a01:238:20a:202:5300::6]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.smtp.rzone.de", Issuer "TeleSec ServerPass DE-2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2170019A2 for ; Fri, 29 Jul 2016 08:54:07 +0000 (UTC) (envelope-from rj@obsigna.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; t=1469782444; l=2394; s=domk; d=obsigna.com; h=To:References:Content-Transfer-Encoding:Date:In-Reply-To:From: Subject:Mime-Version:Content-Type; bh=dEiTR4QavjrH88BBHMK/iw2Wo1O2Ur0JxcsTIJrCMd0=; b=BVFvJ3N2cDRUxsPVSZ6ByWfoavEyp0iOVncNcIV5jds616pf80rnzkB4dj+WGwNUUA/ GmbtSnT53ldPNDZJ+0opUCaX4B7oIREKVXSpdPv2+FWpwXIZrwdprNBht1ozJpudqvsRu ljYezY4XcYWci+hcXl+iL3J6O2nlthztV9E= X-RZG-AUTH: :O2kGeEG7b/pS1EK7WHa0hxqKZr4lnx6UhToX1IWHkW4X7v2ImaU2BqlKi/2sgPrOHjSDsnQ= X-RZG-CLASS-ID: mo00 Received: from mail.obsigna.com (bd1d9e9f.virtua.com.br [189.29.158.159]) by smtp.strato.de (RZmta 38.13 DYNA|AUTH) with ESMTPSA id 601786s6T8s2sB0 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (curve secp521r1 with 521 ECDH bits, eq. 15360 bits RSA)) (Client did not present a certificate) for ; Fri, 29 Jul 2016 10:54:02 +0200 (CEST) Received: from rolf.projectworld.net (rolf.projectworld.net [192.168.222.25]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.obsigna.com (Postfix) with ESMTPSA id B8737229861E for ; Fri, 29 Jul 2016 05:53:59 -0300 (BRT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: ipfw divert filter for IPv4 geo-blocking From: "Dr. Rolf Jansen" In-Reply-To: Date: Fri, 29 Jul 2016 05:53:58 -0300 Content-Transfer-Encoding: quoted-printable Message-Id: <1E1DB7E0-D354-4D7A-B657-0ECF94C12CE0@obsigna.com> References: <61DFB3E2-6E34-4EEA-8AC6-70094CEACA72@cyclaero.com> <4D047727-F7D0-4BEE-BD42-2501F44C9550@obsigna.com> <9641D08A-0501-4AA2-9DF6-D5AFE6CB2975@obsigna.com> <4d76a492-17ae-cbff-f92f-5bbbb1339aad@freebsd.org> <677900fb-c717-743f-fcfe-86b603466e33@freebsd.org> <0D3C9016-7A4A-46BA-B35F-3844D07562A8@obsigna.com> To: freebsd-ipfw@freebsd.org X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Jul 2016 08:54:08 -0000 > Am 28.07.2016 um 23:48 schrieb Lee Brown : >=20 > That makes sense to me. Your /20 range encompasses 201.222.16.0 - > 201.222.31.255. > If you want 201.222.20.0-201.222.31.255, you'll need 3 ranges: >=20 > 201.222.20.0/22 (201.222.20.0-201.222.23.255) > 201.222.24.0/22 (201.222.24.0-201.222.27.255) > 201.222.28.0/22 (201.222.28.0-201.222.31.255) Ian, Julian and Lee, Thank you vary much for your responses. In order not bloat the thread, I = answer only to one message. I found the problem. As a matter of fact, the respective IP ranges in = the LACNIC delegation statistics file are 3 adjacent blocks with 1024 = addresses, i.e. those that you listed in your message above: $grep 201.222.2 /usr/local/etc/ipdb/IPRanges/lacnic.dat lacnic|BR|ipv4|201.222.20.0|1024|20140710|allocated|164725 lacnic|BR|ipv4|201.222.24.0|1024|20140630|allocated|138376 lacnic|BR|ipv4|201.222.28.0|1024|20140701|allocated|129095 However, my database compilation combines adjacent blocks with the same = country code, and the ranges above turn into one block of 3072 = addresses, which obviously doesn't have a valid netmask - log(3072) =3D = 11,5849625. My divert filter daemon is agnostic about this, because the = exact ranges are stored in the database and for the purpose of country = code lookup the address lookup routine doesn't need to work with = netmasks. I choose it this way, because I read some RIR documentation = stating that delegated address blocks are not necessarily complete CIDR = ranges. Even if the above ranges conform to CIDR, future delegations may = be different, and I wanted to stay on the safe side. So far so good. Now, the actual problem with ipfw tables in the given = context is that explicit IP ranges are not accepted but ranges must be = given in CIDR notation, and I simply forgot about the tiny detail that = CIDR is inherently granular, and that this may of course conflict with = my CC/IP database optimization strategy. Without combining adjacencies, = the complete database has 165815 instead of 83274 records. Perhaps, I = add an option to the db tool for CIDR conformity. In this respect, it is = not sufficient to forget about optimization but I need to check also = whether, the delegation files contain already some non-CIDR ranges, = which needs to be broken down. Best regards Rolf From owner-freebsd-ipfw@freebsd.org Fri Jul 29 09:22:38 2016 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 099A1BA7DB5 for ; Fri, 29 Jul 2016 09:22:38 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vps1.elischer.org", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id DE4EB1A3A for ; Fri, 29 Jul 2016 09:22:37 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from Julian-MBP3.local (ppp121-45-233-115.lns20.per1.internode.on.net [121.45.233.115]) (authenticated bits=0) by vps1.elischer.org (8.15.2/8.15.2) with ESMTPSA id u6T9MWlY080284 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Fri, 29 Jul 2016 02:22:35 -0700 (PDT) (envelope-from julian@freebsd.org) Subject: Re: ipfw divert filter for IPv4 geo-blocking To: "Dr. Rolf Jansen" , freebsd-ipfw@freebsd.org References: <61DFB3E2-6E34-4EEA-8AC6-70094CEACA72@cyclaero.com> <4D047727-F7D0-4BEE-BD42-2501F44C9550@obsigna.com> <9641D08A-0501-4AA2-9DF6-D5AFE6CB2975@obsigna.com> <4d76a492-17ae-cbff-f92f-5bbbb1339aad@freebsd.org> <677900fb-c717-743f-fcfe-86b603466e33@freebsd.org> <0D3C9016-7A4A-46BA-B35F-3844D07562A8@obsigna.com> <1E1DB7E0-D354-4D7A-B657-0ECF94C12CE0@obsigna.com> From: Julian Elischer Message-ID: <50d405a4-3f8f-a706-9cac-d1162925e56a@freebsd.org> Date: Fri, 29 Jul 2016 17:22:26 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 In-Reply-To: <1E1DB7E0-D354-4D7A-B657-0ECF94C12CE0@obsigna.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Jul 2016 09:22:38 -0000 On 29/07/2016 4:53 PM, Dr. Rolf Jansen wrote: >> Am 28.07.2016 um 23:48 schrieb Lee Brown : >> >> That makes sense to me. Your /20 range encompasses 201.222.16.0 - >> 201.222.31.255. >> If you want 201.222.20.0-201.222.31.255, you'll need 3 ranges: >> >> 201.222.20.0/22 (201.222.20.0-201.222.23.255) >> 201.222.24.0/22 (201.222.24.0-201.222.27.255) >> 201.222.28.0/22 (201.222.28.0-201.222.31.255) > > Ian, Julian and Lee, > > Thank you vary much for your responses. In order not bloat the thread, I answer only to one message. > > I found the problem. As a matter of fact, the respective IP ranges in the LACNIC delegation statistics file are 3 adjacent blocks with 1024 addresses, i.e. those that you listed in your message above: > > $grep 201.222.2 /usr/local/etc/ipdb/IPRanges/lacnic.dat > lacnic|BR|ipv4|201.222.20.0|1024|20140710|allocated|164725 > lacnic|BR|ipv4|201.222.24.0|1024|20140630|allocated|138376 > lacnic|BR|ipv4|201.222.28.0|1024|20140701|allocated|129095 > > However, my database compilation combines adjacent blocks with the same country code, and the ranges above turn into one block of 3072 addresses, which obviously doesn't have a valid netmask - log(3072) = 11,5849625. My divert filter daemon is agnostic about this, because the exact ranges are stored in the database and for the purpose of country code lookup the address lookup routine doesn't need to work with netmasks. I choose it this way, because I read some RIR documentation stating that delegated address blocks are not necessarily complete CIDR ranges. Even if the above ranges conform to CIDR, future delegations may be different, and I wanted to stay on the safe side. > > So far so good. Now, the actual problem with ipfw tables in the given context is that explicit IP ranges are not accepted but ranges must be given in CIDR notation, and I simply forgot about the tiny detail that CIDR is inherently granular, and that this may of course conflict with my CC/IP database optimization strategy. Without combining adjacencies, the complete database has 165815 instead of 83274 records. Perhaps, I add an option to the db tool for CIDR conformity. In this respect, it is not sufficient to forget about optimization but I need to check also whether, the delegation files contain already some non-CIDR ranges, which needs to be broken down. there is code to take ranges and produce cidr sets. We used to have exactly that code in the appletalk code before we took it out. Appletalk uses ranges. https://svnweb.freebsd.org/base/release/3.2.0/sys/netatalk/at_control.c?view=annotate#l703 maybe you can find other versions on the net. however if you fully populate the table, you will get the correct result because more specific entries will override less specific entries. To do that you would have to have a way to describe to your program what value each table entry should output. If you did what you do now, then you would specify the value for the required countries, and give a default falue for "all others". aggregation of adjacent ranges with same value would be an optimisation. > > Best regards > > Rolf > > _______________________________________________ > freebsd-ipfw@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw > To unsubscribe, send any mail to "freebsd-ipfw-unsubscribe@freebsd.org" > From owner-freebsd-ipfw@freebsd.org Fri Jul 29 09:50:30 2016 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7CC3FBA6130 for ; Fri, 29 Jul 2016 09:50:30 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vps1.elischer.org", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 5F6321278 for ; Fri, 29 Jul 2016 09:50:29 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from Julian-MBP3.local (ppp121-45-233-115.lns20.per1.internode.on.net [121.45.233.115]) (authenticated bits=0) by vps1.elischer.org (8.15.2/8.15.2) with ESMTPSA id u6T9oOWq080360 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Fri, 29 Jul 2016 02:50:27 -0700 (PDT) (envelope-from julian@freebsd.org) Subject: Re: ipfw divert filter for IPv4 geo-blocking To: "Dr. Rolf Jansen" , freebsd-ipfw@freebsd.org References: <61DFB3E2-6E34-4EEA-8AC6-70094CEACA72@cyclaero.com> <4D047727-F7D0-4BEE-BD42-2501F44C9550@obsigna.com> <9641D08A-0501-4AA2-9DF6-D5AFE6CB2975@obsigna.com> <4d76a492-17ae-cbff-f92f-5bbbb1339aad@freebsd.org> <677900fb-c717-743f-fcfe-86b603466e33@freebsd.org> <0D3C9016-7A4A-46BA-B35F-3844D07562A8@obsigna.com> <1E1DB7E0-D354-4D7A-B657-0ECF94C12CE0@obsigna.com> <50d405a4-3f8f-a706-9cac-d1162925e56a@freebsd.org> From: Julian Elischer Message-ID: Date: Fri, 29 Jul 2016 17:50:19 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 In-Reply-To: <50d405a4-3f8f-a706-9cac-d1162925e56a@freebsd.org> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Jul 2016 09:50:30 -0000 On 29/07/2016 5:22 PM, Julian Elischer wrote: > On 29/07/2016 4:53 PM, Dr. Rolf Jansen wrote: >>> Am 28.07.2016 um 23:48 schrieb Lee Brown : >>> >>> That makes sense to me. Your /20 range encompasses 201.222.16.0 - >>> 201.222.31.255. >>> If you want 201.222.20.0-201.222.31.255, you'll need 3 ranges: >>> >>> 201.222.20.0/22 (201.222.20.0-201.222.23.255) >>> 201.222.24.0/22 (201.222.24.0-201.222.27.255) >>> 201.222.28.0/22 (201.222.28.0-201.222.31.255) >> >> Ian, Julian and Lee, >> >> Thank you vary much for your responses. In order not bloat the >> thread, I answer only to one message. >> >> I found the problem. As a matter of fact, the respective IP ranges >> in the LACNIC delegation statistics file are 3 adjacent blocks with >> 1024 addresses, i.e. those that you listed in your message above: >> >> $grep 201.222.2 /usr/local/etc/ipdb/IPRanges/lacnic.dat >> lacnic|BR|ipv4|201.222.20.0|1024|20140710|allocated|164725 >> lacnic|BR|ipv4|201.222.24.0|1024|20140630|allocated|138376 >> lacnic|BR|ipv4|201.222.28.0|1024|20140701|allocated|129095 >> >> However, my database compilation combines adjacent blocks with the >> same country code, and the ranges above turn into one block of 3072 >> addresses, which obviously doesn't have a valid netmask - log(3072) >> = 11,5849625. My divert filter daemon is agnostic about this, >> because the exact ranges are stored in the database and for the >> purpose of country code lookup the address lookup routine doesn't >> need to work with netmasks. I choose it this way, because I read >> some RIR documentation stating that delegated address blocks are >> not necessarily complete CIDR ranges. Even if the above ranges >> conform to CIDR, future delegations may be different, and I wanted >> to stay on the safe side. >> >> So far so good. Now, the actual problem with ipfw tables in the >> given context is that explicit IP ranges are not accepted but >> ranges must be given in CIDR notation, and I simply forgot about >> the tiny detail that CIDR is inherently granular, and that this may >> of course conflict with my CC/IP database optimization strategy. >> Without combining adjacencies, the complete database has 165815 >> instead of 83274 records. Perhaps, I add an option to the db tool >> for CIDR conformity. In this respect, it is not sufficient to >> forget about optimization but I need to check also whether, the >> delegation files contain already some non-CIDR ranges, which needs >> to be broken down. > there is code to take ranges and produce cidr sets. > > We used to have exactly that code in the appletalk code before we > took it out. Appletalk uses ranges. > https://svnweb.freebsd.org/base/release/3.2.0/sys/netatalk/at_control.c?view=annotate#l703 > though htat uassumes input in the form af an appletak sockaddr.. there is also this python module https://pythonhosted.org/netaddr/tutorial_01.html#support-for-non-standard-address-ranges > > maybe you can find other versions on the net. > however if you fully populate the table, you will get the correct > result because more specific entries will > override less specific entries. To do that you would have to have a > way to describe to your program what > value each table entry should output. > If you did what you do now, then you would specify the value for the > required countries, and give a default falue for "all others". > aggregation of adjacent ranges with same value would be an > optimisation. > > > >> >> Best regards >> >> Rolf >> >> _______________________________________________ >> freebsd-ipfw@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw >> To unsubscribe, send any mail to >> "freebsd-ipfw-unsubscribe@freebsd.org" >> > > _______________________________________________ > freebsd-ipfw@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw > To unsubscribe, send any mail to "freebsd-ipfw-unsubscribe@freebsd.org" > From owner-freebsd-ipfw@freebsd.org Fri Jul 29 13:23:44 2016 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 86F7ABA7EB2 for ; Fri, 29 Jul 2016 13:23:44 +0000 (UTC) (envelope-from rj@obsigna.com) Received: from mo6-p00-ob.smtp.rzone.de (mo6-p00-ob.smtp.rzone.de [IPv6:2a01:238:20a:202:5300::4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.smtp.rzone.de", Issuer "TeleSec ServerPass DE-2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id F2D8D1712; Fri, 29 Jul 2016 13:23:43 +0000 (UTC) (envelope-from rj@obsigna.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; t=1469798621; l=3695; s=domk; d=obsigna.com; h=To:References:Content-Transfer-Encoding:Cc:Date:In-Reply-To:From: Subject:Mime-Version:Content-Type; bh=3oxNXZ4GHD7G2qWCXMTzoX7lXBcpCGEQUVu8+zUVsWo=; b=OqTZSqnzAlozc5fmCO0JjW0lxOZCKTi6zLNmDUq9vb9U3rwHHoGSAZXNWv/7KaxCTKV qp1pLegaZ1O/1pi7uCHbvw0l9fbDazRvSGPcR+7R7k7lpkSWpknAJOj8tMGYV/JZoRITv s9612hMtKg7N4cfF0PCZSOdM6ulod4WmSxs= X-RZG-AUTH: :O2kGeEG7b/pS1EK7WHa0hxqKZr4lnx6UhToX1IWHkW4X7v2ImaU2BqlKi/2sgPrOHjSDsnQ= X-RZG-CLASS-ID: mo00 Received: from mail.obsigna.com (bd1d9e9f.virtua.com.br [189.29.158.159]) by smtp.strato.de (RZmta 38.13 DYNA|AUTH) with ESMTPSA id f057f3s6TDNdxHv (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (curve secp521r1 with 521 ECDH bits, eq. 15360 bits RSA)) (Client did not present a certificate); Fri, 29 Jul 2016 15:23:39 +0200 (CEST) Received: from rolf.projectworld.net (rolf.projectworld.net [192.168.222.25]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.obsigna.com (Postfix) with ESMTPSA id 52AB7229861E; Fri, 29 Jul 2016 10:23:36 -0300 (BRT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: ipfw divert filter for IPv4 geo-blocking From: "Dr. Rolf Jansen" In-Reply-To: Date: Fri, 29 Jul 2016 10:23:35 -0300 Cc: Julian Elischer Content-Transfer-Encoding: quoted-printable Message-Id: <9222BB10-C700-4DE7-83A3-BE7A38A11713@obsigna.com> References: <61DFB3E2-6E34-4EEA-8AC6-70094CEACA72@cyclaero.com> <4D047727-F7D0-4BEE-BD42-2501F44C9550@obsigna.com> <9641D08A-0501-4AA2-9DF6-D5AFE6CB2975@obsigna.com> <4d76a492-17ae-cbff-f92f-5bbbb1339aad@freebsd.org> <677900fb-c717-743f-fcfe-86b603466e33@freebsd.org> <0D3C9016-7A4A-46BA-B35F-3844D07562A8@obsigna.com> <1E1DB7E0-D354-4D7A-B657-0ECF94C12CE0@obsigna.com> <50d405a4-3f8f-a706-9cac-d1162925e56a@freebsd.org> To: freebsd-ipfw@freebsd.org X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Jul 2016 13:23:44 -0000 > Am 29.07.2016 um 06:50 schrieb Julian Elischer : > On 29/07/2016 5:22 PM, Julian Elischer wrote: >> On 29/07/2016 4:53 PM, Dr. Rolf Jansen wrote: >>>> Am 28.07.2016 um 23:48 schrieb Lee Brown : >>>>=20 >>>> That makes sense to me. Your /20 range encompasses 201.222.16.0 - >>>> 201.222.31.255. >>>> If you want 201.222.20.0-201.222.31.255, you'll need 3 ranges: >>>>=20 >>>> 201.222.20.0/22 (201.222.20.0-201.222.23.255) >>>> 201.222.24.0/22 (201.222.24.0-201.222.27.255) >>>> 201.222.28.0/22 (201.222.28.0-201.222.31.255) >>>=20 >>> Ian, Julian and Lee, >>>=20 >>> Thank you vary much for your responses. In order not bloat the = thread, I answer only to one message. >>>=20 >>> I found the problem. As a matter of fact, the respective IP ranges = in the LACNIC delegation statistics file are 3 adjacent blocks with 1024 = addresses, i.e. those that you listed in your message above: >>>=20 >>> $grep 201.222.2 /usr/local/etc/ipdb/IPRanges/lacnic.dat >>> lacnic|BR|ipv4|201.222.20.0|1024|20140710|allocated|164725 >>> lacnic|BR|ipv4|201.222.24.0|1024|20140630|allocated|138376 >>> lacnic|BR|ipv4|201.222.28.0|1024|20140701|allocated|129095 >>>=20 >>> However, my database compilation combines adjacent blocks with the = same country code, and the ranges above turn into one block of 3072 = addresses, which obviously doesn't have a valid netmask - log(3072) =3D = 11,5849625. >>> ... >>> ..., it is not sufficient to forget about optimization but I need to = check also whether, the delegation files contain already some non-CIDR = ranges, which need to be broken down. >>=20 >> there is code to take ranges and produce cidr sets. >>=20 >> We used to have exactly that code in the appletalk code before we = took it out. Appletalk uses ranges. >> = https://svnweb.freebsd.org/base/release/3.2.0/sys/netatalk/at_control.c?vi= ew=3Dannotate#l703=20 >=20 > though htat uassumes input in the form af an appletak sockaddr.. > there is also this python module > = https://pythonhosted.org/netaddr/tutorial_01.html#support-for-non-standard= -address-ranges >=20 >> maybe you can find other versions on the net. >> however if you fully populate the table, you will get the correct = result because more specific entries will >> override less specific entries. To do that you would have to have a = way to describe to your program what >> value each table entry should output. >> If you did what you do now, then you would specify the value for the = required countries, and give a default falue for "all others". >> aggregation of adjacent ranges with same value would be an = optimisation. Don't worry, breaking down an arbitrary IP-range into a CIDR conforming = set of ranges, doesn't seem too difficult. I quickly hacked into the = for() loop for table generation in geoip.c a nested do while() for if = necessary breaking down the given range: for (i =3D 0; i < n; i++) { if (!*ccList || findCC(CCTable, sortedIPSets[i][2])) { ipv4_lo =3D sortedIPSets[i][0]; do { m =3D intlb(sortedIPSets[i][1] - ipv4_lo + 1); while (ipv4_lo % (k =3D (int)lround(exp2(m)))) m--; printRange(ipv4_lo, 32 - m, table_number, table_value); = =20 } while ((ipv4_lo +=3D k) < sortedIPSets[i][1]); } } This seems to work as expected, however, I need to check this more = carefully (in the course of the weekend) until pushing it to GitHub. At = the first glance, the tables become quite large. For example, the table = for Brazil of 824 joined entries is bloated to 6621 CIDR records. Once I came to a conclusion, I will post it to this mailing list. Best regards Rolf From owner-freebsd-ipfw@freebsd.org Sat Jul 30 14:17:24 2016 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A8414BA83DB for ; Sat, 30 Jul 2016 14:17:24 +0000 (UTC) (envelope-from rj@obsigna.com) Received: from mo6-p00-ob.smtp.rzone.de (mo6-p00-ob.smtp.rzone.de [IPv6:2a01:238:20a:202:5300::10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.smtp.rzone.de", Issuer "TeleSec ServerPass DE-2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4335F159A for ; Sat, 30 Jul 2016 14:17:23 +0000 (UTC) (envelope-from rj@obsigna.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; t=1469888240; l=5068; s=domk; d=obsigna.com; h=To:References:Content-Transfer-Encoding:Date:In-Reply-To:From: Subject:Mime-Version:Content-Type; bh=9XGldtdeuDaUVC9//kKag858f/wox4J+OkYSZmVXIT0=; b=A9NpB7sYPmJONs4KRkwAnPrFrYRv9Ukmhu3HuXM82PTK9cch0QGeFWKgy4ZAcib+17k oRSc+P4k4OPlvy7Ns/KYJknNq6nw9tXsjDan9430etsObiJB2+Il6LDgDotMQfeSUv16E yFvGLwYRZQM+7lZUkvEbwhH+6JhhYsfyhUc= X-RZG-AUTH: :O2kGeEG7b/pS1EK7WHa0hxqKZr4lnx6UhToX1IWHkW4X7v2ImaU2B+3KSGnPFnK/TBWBAh4A X-RZG-CLASS-ID: mo00 Received: from mail.obsigna.com (bfb6bfbf.virtua.com.br [191.182.191.191]) by smtp.strato.de (RZmta 38.13 DYNA|AUTH) with ESMTPSA id D0b9bcs6UEHJ08N (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (curve secp521r1 with 521 ECDH bits, eq. 15360 bits RSA)) (Client did not present a certificate) for ; Sat, 30 Jul 2016 16:17:19 +0200 (CEST) Received: from rolf.projectworld.net (rolf.projectworld.net [192.168.222.25]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.obsigna.com (Postfix) with ESMTPSA id CAE18229861E for ; Sat, 30 Jul 2016 11:17:15 -0300 (BRT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: ipfw divert filter for IPv4 geo-blocking From: "Dr. Rolf Jansen" In-Reply-To: <9222BB10-C700-4DE7-83A3-BE7A38A11713@obsigna.com> Date: Sat, 30 Jul 2016 11:17:13 -0300 Content-Transfer-Encoding: quoted-printable Message-Id: <1B36CAD7-A139-436B-B7EC-0FFF232F9C6A@obsigna.com> References: <61DFB3E2-6E34-4EEA-8AC6-70094CEACA72@cyclaero.com> <4D047727-F7D0-4BEE-BD42-2501F44C9550@obsigna.com> <9641D08A-0501-4AA2-9DF6-D5AFE6CB2975@obsigna.com> <4d76a492-17ae-cbff-f92f-5bbbb1339aad@freebsd.org> <677900fb-c717-743f-fcfe-86b603466e33@freebsd.org> <0D3C9016-7A4A-46BA-B35F-3844D07562A8@obsigna.com> <1E1DB7E0-D354-4D7A-B657-0ECF94C12CE0@obsigna.com> <50d405a4-3f8f-a706-9cac-d1162925e56a@freebsd.org> <9222BB10-C700-4DE7-83A3-BE7A38A11713@obsigna.com> To: freebsd-ipfw@freebsd.org X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 30 Jul 2016 14:17:24 -0000 > Am 29.07.2016 um 10:23 schrieb Dr. Rolf Jansen : >> Am 29.07.2016 um 06:50 schrieb Julian Elischer : >> On 29/07/2016 5:22 PM, Julian Elischer wrote: >>> On 29/07/2016 4:53 PM, Dr. Rolf Jansen wrote: >>>>> Am 28.07.2016 um 23:48 schrieb Lee Brown : >>>>>=20 >>>>> That makes sense to me. Your /20 range encompasses 201.222.16.0 - >>>>> 201.222.31.255. >>>>> If you want 201.222.20.0-201.222.31.255, you'll need 3 ranges: >>>>>=20 >>>>> 201.222.20.0/22 (201.222.20.0-201.222.23.255) >>>>> 201.222.24.0/22 (201.222.24.0-201.222.27.255) >>>>> 201.222.28.0/22 (201.222.28.0-201.222.31.255) >>>>=20 >>>> Ian, Julian and Lee, >>>>=20 >>>> Thank you vary much for your responses. In order not bloat the = thread, I answer only to one message. >>>>=20 >>>> I found the problem. As a matter of fact, the respective IP ranges = in the LACNIC delegation statistics file are 3 adjacent blocks with 1024 = addresses, i.e. those that you listed in your message above: >>>>=20 >>>> $grep 201.222.2 /usr/local/etc/ipdb/IPRanges/lacnic.dat >>>> lacnic|BR|ipv4|201.222.20.0|1024|20140710|allocated|164725 >>>> lacnic|BR|ipv4|201.222.24.0|1024|20140630|allocated|138376 >>>> lacnic|BR|ipv4|201.222.28.0|1024|20140701|allocated|129095 >>>>=20 >>>> However, my database compilation combines adjacent blocks with the = same country code, and the ranges above turn into one block of 3072 = addresses, which obviously doesn't have a valid netmask - log(3072) =3D = 11,5849625. >>>> ... >>>> ..., it is not sufficient to forget about optimization but I need = to check also whether, the delegation files contain already some = non-CIDR ranges, which need to be broken down. >>>=20 >>> there is code to take ranges and produce cidr sets. >>>=20 >>> We used to have exactly that code in the appletalk code before we = took it out. Appletalk uses ranges. >>> = https://svnweb.freebsd.org/base/release/3.2.0/sys/netatalk/at_control.c?vi= ew=3Dannotate#l703=20 >>=20 >> though htat uassumes input in the form af an appletak sockaddr.. >> there is also this python module >> = https://pythonhosted.org/netaddr/tutorial_01.html#support-for-non-standard= -address-ranges >>=20 >>> maybe you can find other versions on the net. >>> however if you fully populate the table, you will get the correct = result because more specific entries will >>> override less specific entries. To do that you would have to have a = way to describe to your program what >>> value each table entry should output. >>> If you did what you do now, then you would specify the value for the = required countries, and give a default falue for "all others". >>> aggregation of adjacent ranges with same value would be an = optimisation. >=20 > Don't worry, breaking down an arbitrary IP-range into a CIDR = conforming set of ranges, doesn't seem too difficult. ... > ... > Once I come to a conclusion, I will post it to this mailing list. I finished the work on CIDR conformity of the IP ranges tables generated = by the tool geoip. The main constraint is that the start and end address = of an IP block given by the delegation files MUST BE PRESERVED during = the transformation to a set of CIDR records. This target is achieved by: 1. Finding the largest common netmask boundary of the start address = utilizing int(log2(addr_count)); then iteration like Euclid's algorithm in = computing a GCD. 2. Output the CIDR with the given start address and the masklen = belonging to the found netmask. 3. If the CIDR does not match the whole original IP range then set the = start address of the next CIDR block to the next boundary of the common = netmask, and loop over starting at 1. until the original range has been = satisfied. I carefully tested the algorithm and a table that I pipe by the new = geoip tool into ipfw is 100 % identical to the output of the ipfw = command 'table N list'. It is worth to note, that already the original RIR delegation files = contain 457 non CIDR conforming IPv4 ranges in a total of 165815 = original records. I guess that this number will increase in the future = because the RIR's ran empty on new IPv4 ranges and are urged to = subdivide returned old ranges for new delegations. The above algorithm = is ready for this. Generally, CIDR conforming tables are more than twice as large as = optimized (joined adjacencies) IP range tables. All said changes have = been pushed to GitHup already. I am still a little bit amazed how ipfw come to accept incorrect CIDR = ranges and arbitrarily moves the start/end addresses in order to achieve = CIDR conformity, and that without any further notice, and that given = that ipfw can be considered as being quite relevant to system security. = Or, may I assume that ipfw knows always better than the user what should = be allowed or denied. Otherwise, perhaps I am the only one ever who = input incorrect CIDR ranges for processing by ipfw. Best regards Rolf