From owner-freebsd-pf@freebsd.org Sun Jun 28 08:06:21 2015 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0729998EC14 for ; Sun, 28 Jun 2015 08:06:21 +0000 (UTC) (envelope-from freebsd-pf@dino.sk) Received: from mailhost.netlabit.sk (mailhost.netlabit.sk [84.245.65.72]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 89EEF19C5 for ; Sun, 28 Jun 2015 08:06:19 +0000 (UTC) (envelope-from freebsd-pf@dino.sk) Received: from zeta.dino.sk (fw1.dino.sk [84.245.95.252]) (AUTH: LOGIN milan) by mailhost.netlabit.sk with ESMTPA; Sun, 28 Jun 2015 10:06:09 +0200 id 00EB08B3.558FAAF1.0000750B Date: Sun, 28 Jun 2015 10:06:09 +0200 From: Milan Obuch To: Ian FREISLICH Cc: freebsd-pf@freebsd.org Subject: Re: Large scale NAT with PF - some weird problem Message-ID: <20150628100609.635544e0@zeta.dino.sk> In-Reply-To: <20150623112331.668395d1@zeta.dino.sk> References: <20150623101225.4bc7f2d0@zeta.dino.sk> <20150623073856.334ebd61@zeta.dino.sk> <20150621133236.75a4d86d@zeta.dino.sk> <20150620182432.62797ec5@zeta.dino.sk> <20150619091857.304b707b@zeta.dino.sk> <14e119e8fa8.2755.abfb21602af57f30a7457738c46ad3ae@capeaugusta.com> <20150621195753.7b162633@zeta.dino.sk> <20150623112331.668395d1@zeta.dino.sk> X-Mailer: Claws Mail 3.11.1 (GTK+ 2.24.27; i386-portbld-freebsd10.1) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 28 Jun 2015 08:06:21 -0000 On Tue, 23 Jun 2015 11:23:31 +0200 Milan Obuch wrote: > On Tue, 23 Jun 2015 10:57:44 +0200 > Ian FREISLICH wrote: > [ snip ] > > So, I think that the problem is with 9-STABLE. I hate "upgrade to > > solve your problems" answers because they may not. I do know that > > 10 has seen a lot of work and none of that work will make it back > > into 9 because of the PF rewrite. Maybe someone else in this group > > will chime in. > > > > That's OK. I am a bit conservative on upgrades here because with > hundreds - thousands users you need a bit of stability too, but > upgrade to 10-STABLE is currently being prepared. That being written, > it will not occur today. > So, now I am at 10.2-PRERELEASE, r284884, and the issue is still here. It is totally weird, just change of IP the device is being natted to makes the issue disappear for this particular customer, but as soon as this exact IP is used again, the issue is here again. Could anybody help me to debug this better? It looks like I really REALLY need some help :( Hate to write anything like this, but it is urgent for me and I am out of ideas... Regards, Milan From owner-freebsd-pf@freebsd.org Mon Jun 29 07:04:02 2015 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3D21398EA01 for ; Mon, 29 Jun 2015 07:04:02 +0000 (UTC) (envelope-from dhartmei@insomnia.benzedrine.ch) Received: from insomnia.benzedrine.ch (106.30.3.213.static.wline.lns.sme.cust.swisscom.ch [213.3.30.106]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "insomnia.benzedrine.ch", Issuer "StartCom Class 1 Primary Intermediate Server CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id C42121BC7 for ; Mon, 29 Jun 2015 07:04:01 +0000 (UTC) (envelope-from dhartmei@insomnia.benzedrine.ch) Received: from insomnia.benzedrine.ch (localhost [127.0.0.1]) by insomnia.benzedrine.ch (8.14.6/8.14.6) with ESMTP id t5T6wdgI013480 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 29 Jun 2015 08:58:39 +0200 (MEST) Received: (from dhartmei@localhost) by insomnia.benzedrine.ch (8.14.6/8.14.5/Submit) id t5T6wcfN001194; Mon, 29 Jun 2015 08:58:38 +0200 (MEST) Date: Mon, 29 Jun 2015 08:58:38 +0200 From: Daniel Hartmeier To: Milan Obuch Cc: Ian FREISLICH , freebsd-pf@freebsd.org Subject: Re: Large scale NAT with PF - some weird problem Message-ID: <20150629065838.GA13722@insomnia.benzedrine.ch> References: <20150620182432.62797ec5@zeta.dino.sk> <20150619091857.304b707b@zeta.dino.sk> <14e119e8fa8.2755.abfb21602af57f30a7457738c46ad3ae@capeaugusta.com> <20150621195753.7b162633@zeta.dino.sk> <20150623112331.668395d1@zeta.dino.sk> <20150628100609.635544e0@zeta.dino.sk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20150628100609.635544e0@zeta.dino.sk> User-Agent: Mutt/1.5.21 (2010-09-15) X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Jun 2015 07:04:02 -0000 On Sun, Jun 28, 2015 at 10:06:09AM +0200, Milan Obuch wrote: > So, now I am at 10.2-PRERELEASE, r284884, and the issue is still here. > It is totally weird, just change of IP the device is being natted to > makes the issue disappear for this particular customer, but as soon as > this exact IP is used again, the issue is here again. I'd go over the entire network config (pf.conf, pfctl -sa, rc.conf, netstat -anr, ifconfig, arp -an) and look for any mistake, like a typo or a netmask which isn't what you thought it is (like on an alias), or for any weirdness related to that IP address. Daniel From owner-freebsd-pf@freebsd.org Mon Jun 29 07:43:22 2015 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D429898F4CB for ; Mon, 29 Jun 2015 07:43:22 +0000 (UTC) (envelope-from freebsd-pf@dino.sk) Received: from mailhost.netlabit.sk (mailhost.netlabit.sk [84.245.65.72]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 50637157B for ; Mon, 29 Jun 2015 07:43:21 +0000 (UTC) (envelope-from freebsd-pf@dino.sk) Received: from zeta.dino.sk (fw1.dino.sk [84.245.95.252]) (AUTH: LOGIN milan) by mailhost.netlabit.sk with ESMTPA; Mon, 29 Jun 2015 09:43:18 +0200 id 000F1815.5590F716.00002349 Date: Mon, 29 Jun 2015 09:43:17 +0200 From: Milan Obuch To: Daniel Hartmeier Cc: Ian FREISLICH , freebsd-pf@freebsd.org Subject: Re: Large scale NAT with PF - some weird problem Message-ID: <20150629094317.5a0cd61a@zeta.dino.sk> In-Reply-To: <20150629065838.GA13722@insomnia.benzedrine.ch> References: <20150620182432.62797ec5@zeta.dino.sk> <20150619091857.304b707b@zeta.dino.sk> <14e119e8fa8.2755.abfb21602af57f30a7457738c46ad3ae@capeaugusta.com> <20150621195753.7b162633@zeta.dino.sk> <20150623112331.668395d1@zeta.dino.sk> <20150628100609.635544e0@zeta.dino.sk> <20150629065838.GA13722@insomnia.benzedrine.ch> X-Mailer: Claws Mail 3.11.1 (GTK+ 2.24.27; i386-portbld-freebsd10.1) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Jun 2015 07:43:23 -0000 On Mon, 29 Jun 2015 08:58:38 +0200 Daniel Hartmeier wrote: > On Sun, Jun 28, 2015 at 10:06:09AM +0200, Milan Obuch wrote: > > > So, now I am at 10.2-PRERELEASE, r284884, and the issue is still > > here. It is totally weird, just change of IP the device is being > > natted to makes the issue disappear for this particular customer, > > but as soon as this exact IP is used again, the issue is here again. > > I'd go over the entire network config (pf.conf, pfctl -sa, rc.conf, > netstat -anr, ifconfig, arp -an) and look for any mistake, like a > typo or a netmask which isn't what you thought it is (like on an > alias), or for any weirdness related to that IP address. > > Daniel Thanks for hint, there is some logic in there, however grep /etc/* yields nothing, it is never mentioned in any config, just as part of pool in pf.conf statement nat on $if_ext from to any -> $pool_ext round-robin sticky-address It is not mentioned in 'pfctl -sa' output, 'arp -an' output, 'netstat -anr' output... nowhere. I did not mention this box runs quagga for configuring network, mainly routing via OSPF, but I do not think it is relevant to the problem I see as this is basically userland process communicating with forwarding path in kernel to configure routing, nothing else, and, naturally, it does not work with this particular IP either. I should have seen it otherwise in some of above mentioned commands output, I think. Just to repeat myself a bit, when this problematic state occurs, some intenal IP is translated to this one offending public IP, and communication is broken in such a way I see no returning packets from outside world on uplink interface in tcpdump even if I know they are there because I can ping some other box outside where I can verify that and they are there... I just found some other strange, to me, thing - in 'pfctl -sa' output, section SOURCE TRACKING NODES, almost all entries are in form -> ( states ..., connections ..., rate ... ) but there are some of them with first IP being public, second one 0.0.0.0 - where they could come from? Also, there are only couple of them, but in one is something even a bit more weird - in parens is 'states 4294967295', which seems a bit absurd to me, also, worth to mention, it is 0xffffffff in hexadecimal, and this looks like some underflow issue in the code. Maybe this deserves some closer pf developer's attention, I just don't know who that could be... smells like a bug. Regards, Milan From owner-freebsd-pf@freebsd.org Mon Jun 29 07:57:12 2015 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 44CAF98F81C for ; Mon, 29 Jun 2015 07:57:12 +0000 (UTC) (envelope-from ian.freislich@capeaugusta.com) Received: from mail-wi0-f170.google.com (mail-wi0-f170.google.com [209.85.212.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CA35D1D6D for ; Mon, 29 Jun 2015 07:57:11 +0000 (UTC) (envelope-from ian.freislich@capeaugusta.com) Received: by wiwl6 with SMTP id l6so91364896wiw.0 for ; Mon, 29 Jun 2015 00:57:04 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:in-reply-to:references :mime-version:content-type:content-id:content-transfer-encoding:date :message-id; bh=EUO3mJedPBLIDrjDdg1rHCN9ql2k3pwiwDWxQS42tkM=; b=f4xGAxd1ozGGFDC7pgUPdHS3Q3Nre/CrupIdbtB9KJsVP09lkQ6/f5mia7tPa43e4I nhIRvQEX0TcIpfnMI982wJoh/uWLrbP7/wWJpwLhbvyd6SVDcjHS5pGK+z2pEOWGZwbo O8UUh/nxczZZHMS9YN17QWzBak/uWjAgZQQsgIUG6S2PRkRCe78DOnA7ADeL5Zwpq5/+ kHvjikP4Z1HiZVjFTROrRzackxUPA4q9rxlB5bBloH/423uHkXttD5xtIeXSOKsEW1uc ayBngYuplkvkY3msv7LPCzoh2e/+GfWYihB9+0CpYopI6sH94bclrjrnUJ/1ZB+fq/RI cKSA== X-Gm-Message-State: ALoCoQnyuYoyMtwjXd2XwXalyyP5kfSAOrl9LDJcOGLOGLOrAW7XmLtHr7ItJrzF4Q/QYawEMRaN X-Received: by 10.180.36.4 with SMTP id m4mr19852888wij.34.1435564154267; Mon, 29 Jun 2015 00:49:14 -0700 (PDT) Received: from clue.co.za ([197.89.34.55]) by mx.google.com with ESMTPSA id q4sm62644571wju.14.2015.06.29.00.49.13 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 29 Jun 2015 00:49:13 -0700 (PDT) From: Ian FREISLICH X-Google-Original-From: Ian FREISLICH Received: from localhost ([127.0.0.1] helo=zen) by clue.co.za with esmtp (Exim 4.85 (FreeBSD)) (envelope-from ) id 1Z9ToU-000Ph3-KT; Mon, 29 Jun 2015 09:49:10 +0200 To: Milan Obuch cc: Daniel Hartmeier , freebsd-pf@freebsd.org Subject: Re: Large scale NAT with PF - some weird problem In-Reply-To: <20150629094317.5a0cd61a@zeta.dino.sk> References: <20150629094317.5a0cd61a@zeta.dino.sk> <20150620182432.62797ec5@zeta.dino.sk> <20150619091857.304b707b@zeta.dino.sk> <14e119e8fa8.2755.abfb21602af57f30a7457738c46ad3ae@capeaugusta.com> <20150621195753.7b162633@zeta.dino.sk> <20150623112331.668395d1@zeta.dino.sk> <20150628100609.635544e0@zeta.dino.sk> <20150629065838.GA13722@insomnia.benzedrine.ch> X-Attribution: BOFH MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <98767.1435564150.1@zen> Content-Transfer-Encoding: quoted-printable Date: Mon, 29 Jun 2015 09:49:10 +0200 Message-Id: X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Jun 2015 07:57:12 -0000 Milan Obuch wrote: > On Mon, 29 Jun 2015 08:58:38 +0200 > Daniel Hartmeier wrote: > = > > On Sun, Jun 28, 2015 at 10:06:09AM +0200, Milan Obuch wrote: > > = > > > So, now I am at 10.2-PRERELEASE, r284884, and the issue is still > > > here. It is totally weird, just change of IP the device is being > > > natted to makes the issue disappear for this particular customer, > > > but as soon as this exact IP is used again, the issue is here again. > > = > > I'd go over the entire network config (pf.conf, pfctl -sa, rc.conf, > > netstat -anr, ifconfig, arp -an) and look for any mistake, like a > > typo or a netmask which isn't what you thought it is (like on an > > alias), or for any weirdness related to that IP address. > > = > > Daniel > = > Thanks for hint, there is some logic in there, however > = > grep /etc/* > = > yields nothing, it is never mentioned in any config, just as part of > pool in pf.conf statement > = > nat on $if_ext from to any -> $pool_ext round-robin sticky-add= ress > = > It is not mentioned in 'pfctl -sa' output, 'arp -an' output, > 'netstat -anr' output... nowhere. > = > I did not mention this box runs quagga for configuring network, mainly > routing via OSPF, but I do not think it is relevant to the problem I > see as this is basically userland process communicating with forwarding > path in kernel to configure routing, nothing else, and, naturally, it > does not work with this particular IP either. I should have seen it > otherwise in some of above mentioned commands output, I think. > = > Just to repeat myself a bit, when this problematic state occurs, some > intenal IP is translated to this one offending public IP, and > communication is broken in such a way I see no returning packets from > outside world on uplink interface in tcpdump even if I know they are > there because I can ping some other box outside where I can verify that > and they are there... > = > I just found some other strange, to me, thing - in 'pfctl -sa' output, > section SOURCE TRACKING NODES, almost all entries are in form > = > -> ( st= ates = ..., connections ..., rate ... ) > = > but there are some of them with first IP being public, second one > 0.0.0.0 - where they could come from? Also, there are only couple of > them, but in one is something even a bit more weird - in parens is > 'states 4294967295', which seems a bit absurd to me, also, worth to > mention, it is 0xffffffff in hexadecimal, and this looks like some > underflow issue in the code. Try making your pool smaller. With the number of NAT states you mentioned earlier, you should easily manage with a /24. Ian -- = Ian Freislich From owner-freebsd-pf@freebsd.org Mon Jun 29 08:00:45 2015 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F241E98FB6B for ; Mon, 29 Jun 2015 08:00:44 +0000 (UTC) (envelope-from freebsd-pf@dino.sk) Received: from mailhost.netlabit.sk (mailhost.netlabit.sk [84.245.65.72]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6FA8D1E7C for ; Mon, 29 Jun 2015 08:00:43 +0000 (UTC) (envelope-from freebsd-pf@dino.sk) Received: from zeta.dino.sk (fw1.dino.sk [84.245.95.252]) (AUTH: LOGIN milan) by mailhost.netlabit.sk with ESMTPA; Mon, 29 Jun 2015 10:00:41 +0200 id 000F190F.5590FB29.0000256C Date: Mon, 29 Jun 2015 10:00:40 +0200 From: Milan Obuch To: Ian FREISLICH Cc: Daniel Hartmeier , freebsd-pf@freebsd.org Subject: Re: Large scale NAT with PF - some weird problem Message-ID: <20150629100040.218c23a2@zeta.dino.sk> In-Reply-To: References: <20150629094317.5a0cd61a@zeta.dino.sk> <20150620182432.62797ec5@zeta.dino.sk> <20150619091857.304b707b@zeta.dino.sk> <14e119e8fa8.2755.abfb21602af57f30a7457738c46ad3ae@capeaugusta.com> <20150621195753.7b162633@zeta.dino.sk> <20150623112331.668395d1@zeta.dino.sk> <20150628100609.635544e0@zeta.dino.sk> <20150629065838.GA13722@insomnia.benzedrine.ch> X-Mailer: Claws Mail 3.11.1 (GTK+ 2.24.27; i386-portbld-freebsd10.1) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Jun 2015 08:00:45 -0000 On Mon, 29 Jun 2015 09:49:10 +0200 Ian FREISLICH wrote: > Milan Obuch wrote: > > On Mon, 29 Jun 2015 08:58:38 +0200 > > Daniel Hartmeier wrote: > > > > > On Sun, Jun 28, 2015 at 10:06:09AM +0200, Milan Obuch wrote: > > > > > > > So, now I am at 10.2-PRERELEASE, r284884, and the issue is still > > > > here. It is totally weird, just change of IP the device is being > > > > natted to makes the issue disappear for this particular > > > > customer, but as soon as this exact IP is used again, the issue > > > > is here again. > > > > > > I'd go over the entire network config (pf.conf, pfctl -sa, > > > rc.conf, netstat -anr, ifconfig, arp -an) and look for any > > > mistake, like a typo or a netmask which isn't what you thought it > > > is (like on an alias), or for any weirdness related to that IP > > > address. > > > > > > Daniel > > > > Thanks for hint, there is some logic in there, however > > > > grep /etc/* > > > > yields nothing, it is never mentioned in any config, just as part of > > pool in pf.conf statement > > > > nat on $if_ext from to any -> $pool_ext round-robin > > sticky-address > > > > It is not mentioned in 'pfctl -sa' output, 'arp -an' output, > > 'netstat -anr' output... nowhere. > > > > I did not mention this box runs quagga for configuring network, > > mainly routing via OSPF, but I do not think it is relevant to the > > problem I see as this is basically userland process communicating > > with forwarding path in kernel to configure routing, nothing else, > > and, naturally, it does not work with this particular IP either. I > > should have seen it otherwise in some of above mentioned commands > > output, I think. > > > > Just to repeat myself a bit, when this problematic state occurs, > > some intenal IP is translated to this one offending public IP, and > > communication is broken in such a way I see no returning packets > > from outside world on uplink interface in tcpdump even if I know > > they are there because I can ping some other box outside where I > > can verify that and they are there... > > > > I just found some other strange, to me, thing - in 'pfctl -sa' > > output, section SOURCE TRACKING NODES, almost all entries are in > > form > > > > -> > > ( states > ..., connections ..., rate ... ) > > > > but there are some of them with first IP being public, second one > > 0.0.0.0 - where they could come from? Also, there are only couple of > > them, but in one is something even a bit more weird - in parens is > > 'states 4294967295', which seems a bit absurd to me, also, worth to > > mention, it is 0xffffffff in hexadecimal, and this looks like some > > underflow issue in the code. > > Try making your pool smaller. With the number of NAT states you > mentioned earlier, you should easily manage with a /24. > OK, I changed pool_ext size from /23 to /24... just would like to know, why should this have desired effect, please... So I am going to observe how it works, but I am sure I had this pool defined for some maybe years and did not receive any complaint until just recently. Regards, Milan From owner-freebsd-pf@freebsd.org Mon Jun 29 08:27:02 2015 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 48D2798FEAD for ; Mon, 29 Jun 2015 08:27:02 +0000 (UTC) (envelope-from dhartmei@insomnia.benzedrine.ch) Received: from insomnia.benzedrine.ch (106.30.3.213.static.wline.lns.sme.cust.swisscom.ch [213.3.30.106]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "insomnia.benzedrine.ch", Issuer "StartCom Class 1 Primary Intermediate Server CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id CFE4A287C for ; Mon, 29 Jun 2015 08:27:01 +0000 (UTC) (envelope-from dhartmei@insomnia.benzedrine.ch) Received: from insomnia.benzedrine.ch (localhost [127.0.0.1]) by insomnia.benzedrine.ch (8.14.6/8.14.6) with ESMTP id t5T8QtLt023440 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 29 Jun 2015 10:26:55 +0200 (MEST) Received: (from dhartmei@localhost) by insomnia.benzedrine.ch (8.14.6/8.14.5/Submit) id t5T8Qs28004575; Mon, 29 Jun 2015 10:26:55 +0200 (MEST) Date: Mon, 29 Jun 2015 10:26:54 +0200 From: Daniel Hartmeier To: Milan Obuch Cc: Ian FREISLICH , freebsd-pf@freebsd.org Subject: Re: Large scale NAT with PF - some weird problem Message-ID: <20150629082654.GA22693@insomnia.benzedrine.ch> References: <20150620182432.62797ec5@zeta.dino.sk> <20150619091857.304b707b@zeta.dino.sk> <14e119e8fa8.2755.abfb21602af57f30a7457738c46ad3ae@capeaugusta.com> <20150621195753.7b162633@zeta.dino.sk> <20150623112331.668395d1@zeta.dino.sk> <20150628100609.635544e0@zeta.dino.sk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20150628100609.635544e0@zeta.dino.sk> User-Agent: Mutt/1.5.21 (2010-09-15) X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Jun 2015 08:27:02 -0000 On Sun, Jun 28, 2015 at 10:06:09AM +0200, Milan Obuch wrote: > So, now I am at 10.2-PRERELEASE, r284884, and the issue is still here. > It is totally weird, just change of IP the device is being natted to > makes the issue disappear for this particular customer, but as soon as > this exact IP is used again, the issue is here again. Do you have access to the upstream router? Can you check its ARP table? It could have a static ARP entry for this specific IP address, or there could be an address conflict for that IP address... Can't you tell us the network, netmask and the IP address? Not even with the first octet redacted? Daniel From owner-freebsd-pf@freebsd.org Mon Jun 29 08:33:57 2015 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6DAC298D077 for ; Mon, 29 Jun 2015 08:33:57 +0000 (UTC) (envelope-from dave@horsfall.org) Received: from teparybean2.exetel.com.au (teparybean2.exetel.com.au [220.233.0.94]) by mx1.freebsd.org (Postfix) with ESMTP id 2E6362D52 for ; Mon, 29 Jun 2015 08:33:56 +0000 (UTC) (envelope-from dave@horsfall.org) Received: from phasia.kd.net.au ([220.233.10.47]) by teparybean2.exetel.com.au with esmtp (Exim 4.84) (envelope-from ) id 1Z9UGq-0005KH-JA for freebsd-pf@freebsd.org; Mon, 29 Jun 2015 18:18:28 +1000 Received: from aneurin.horsfall.org (unknown [120.146.8.15]) by dermis.kd (Postfix) with ESMTP id 32279CDA8 for ; Mon, 29 Jun 2015 18:18:28 +1000 (EST) Received: from aneurin.horsfall.org (localhost [127.0.0.1]) by aneurin.horsfall.org (8.14.9/8.14.9) with ESMTP id t5T8IQ5l002422 for ; Mon, 29 Jun 2015 18:18:27 +1000 (EST) (envelope-from dave@horsfall.org) Received: from localhost (dave@localhost) by aneurin.horsfall.org (8.14.9/8.14.9/Submit) with ESMTP id t5T8IQwC002419 for ; Mon, 29 Jun 2015 18:18:26 +1000 (EST) (envelope-from dave@horsfall.org) X-Authentication-Warning: aneurin.horsfall.org: dave owned process doing -bs Date: Mon, 29 Jun 2015 18:18:26 +1000 (EST) From: Dave Horsfall To: FreeBSD PF List Subject: Re: Large scale NAT with PF - some weird problem In-Reply-To: <20150629094317.5a0cd61a@zeta.dino.sk> Message-ID: References: <20150620182432.62797ec5@zeta.dino.sk> <20150619091857.304b707b@zeta.dino.sk> <14e119e8fa8.2755.abfb21602af57f30a7457738c46ad3ae@capeaugusta.com> <20150621195753.7b162633@zeta.dino.sk> <20150623112331.668395d1@zeta.dino.sk> <20150628100609.635544e0@zeta.dino.sk> <20150629065838.GA13722@insomnia.benzedrine.ch> <20150629094317.5a0cd61a@zeta.dino.sk> User-Agent: Alpine 2.11 (BSF 23 2013-08-11) X-Witty-Saying: "chmod 666 the_mode_of_the_beast" MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Jun 2015 08:33:57 -0000 On Mon, 29 Jun 2015, Milan Obuch wrote: > Thanks for hint, there is some logic in there, however > > grep /etc/* > > yields nothing, it is never mentioned in any config, just as part of > pool in pf.conf statement What about "grep -r"? My ACLs are under /etc/mail, for example. -- Dave Horsfall (VK2KFU) "Those who don't understand security will suffer" http://www.horsfall.org/ It's just a silly little web site, that's all... From owner-freebsd-pf@freebsd.org Mon Jun 29 08:52:12 2015 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BB75098D550 for ; Mon, 29 Jun 2015 08:52:12 +0000 (UTC) (envelope-from freebsd-pf@dino.sk) Received: from mailhost.netlabit.sk (mailhost.netlabit.sk [84.245.65.72]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 51D671BA8 for ; Mon, 29 Jun 2015 08:52:11 +0000 (UTC) (envelope-from freebsd-pf@dino.sk) Received: from zeta.dino.sk (fw1.dino.sk [84.245.95.252]) (AUTH: LOGIN milan) by mailhost.netlabit.sk with ESMTPA; Mon, 29 Jun 2015 10:52:02 +0200 id 000F19B0.55910732.00002ABA Date: Mon, 29 Jun 2015 10:52:01 +0200 From: Milan Obuch To: Daniel Hartmeier Cc: Ian FREISLICH , freebsd-pf@freebsd.org Subject: Re: Large scale NAT with PF - some weird problem Message-ID: <20150629105201.7ee24e38@zeta.dino.sk> In-Reply-To: <20150629082654.GA22693@insomnia.benzedrine.ch> References: <20150620182432.62797ec5@zeta.dino.sk> <20150619091857.304b707b@zeta.dino.sk> <14e119e8fa8.2755.abfb21602af57f30a7457738c46ad3ae@capeaugusta.com> <20150621195753.7b162633@zeta.dino.sk> <20150623112331.668395d1@zeta.dino.sk> <20150628100609.635544e0@zeta.dino.sk> <20150629082654.GA22693@insomnia.benzedrine.ch> X-Mailer: Claws Mail 3.11.1 (GTK+ 2.24.27; i386-portbld-freebsd10.1) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Jun 2015 08:52:12 -0000 On Mon, 29 Jun 2015 10:26:54 +0200 Daniel Hartmeier wrote: > On Sun, Jun 28, 2015 at 10:06:09AM +0200, Milan Obuch wrote: >=20 > > So, now I am at 10.2-PRERELEASE, r284884, and the issue is still > > here. It is totally weird, just change of IP the device is being > > natted to makes the issue disappear for this particular customer, > > but as soon as this exact IP is used again, the issue is here again. >=20 > Do you have access to the upstream router? > Can you check its ARP table? No, I do not have access here, I can't get info from there directly. I could get some info from some admin, but this would take some time, and I do not think it could really help me... > It could have a static ARP entry for this specific IP address, or > there could be an address conflict for that IP address... Well, no reason for that, some more background below. > Can't you tell us the network, netmask and the IP address? > Not even with the first octet redacted? Well, I do not like to give full details in public, but partially redacted - all public address are from one /16 block, lets call it x.y.0.0/16. On my side, uplink interface is em0 with IP x.y.3.19/29, on upstream router, there is x.y.3.17/29, used as default gateway for me. On upstream router, there is statically routed network x.y.24.0/22 to x.y.3.19, my IP. Other IPs on uplink segment are not used currently. =46rom this x.y.24.0/22 address block, some smaller segments are directly connected to my box, such as public servers (DNS, www, mail...) or some customers with dedicated public IP. For this purpose, x.y.24.0/24 address block is used, divided into smaller segments. Next block, x.y.25.0/24, is used mainly for binat'ed IPs, in pf.conf one will see handfull of binat on $if_ext from 172.a.b.c to any -> x.y.25.z statements, and the rest, x.y.26.0/23, is used as $pool_ext, assigned dynamically to all customers. Per Ian's advice, I am currently testing my setup with just x.y.26.0/24 being used for NAT pool. As for question about ARP - I think there is not anythink like static arp on upstream router. I could ping the offending address from outside and see them arriving on uplink interface, em0, with tcpdump. No replies are being generated, however, but I considered this as good evidence there is nothing blocking me on upstream router. Does this answerred your question fully or something more would be usefull? Regards, Milan From owner-freebsd-pf@freebsd.org Mon Jun 29 08:54:11 2015 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3451198D5F4 for ; Mon, 29 Jun 2015 08:54:11 +0000 (UTC) (envelope-from freebsd-pf@dino.sk) Received: from mailhost.netlabit.sk (mailhost.netlabit.sk [84.245.65.72]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A4EE31C31 for ; Mon, 29 Jun 2015 08:54:09 +0000 (UTC) (envelope-from freebsd-pf@dino.sk) Received: from zeta.dino.sk (fw1.dino.sk [84.245.95.252]) (AUTH: LOGIN milan) by mailhost.netlabit.sk with ESMTPA; Mon, 29 Jun 2015 10:54:07 +0200 id 000F180E.559107AF.00002B03 Date: Mon, 29 Jun 2015 10:54:07 +0200 From: Milan Obuch To: freebsd-pf@freebsd.org Subject: Re: Large scale NAT with PF - some weird problem Message-ID: <20150629105407.450167de@zeta.dino.sk> In-Reply-To: References: <20150620182432.62797ec5@zeta.dino.sk> <20150619091857.304b707b@zeta.dino.sk> <14e119e8fa8.2755.abfb21602af57f30a7457738c46ad3ae@capeaugusta.com> <20150621195753.7b162633@zeta.dino.sk> <20150623112331.668395d1@zeta.dino.sk> <20150628100609.635544e0@zeta.dino.sk> <20150629065838.GA13722@insomnia.benzedrine.ch> <20150629094317.5a0cd61a@zeta.dino.sk> X-Mailer: Claws Mail 3.11.1 (GTK+ 2.24.27; i386-portbld-freebsd10.1) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Jun 2015 08:54:11 -0000 On Mon, 29 Jun 2015 18:18:26 +1000 (EST) Dave Horsfall wrote: > On Mon, 29 Jun 2015, Milan Obuch wrote: > > > Thanks for hint, there is some logic in there, however > > > > grep /etc/* > > > > yields nothing, it is never mentioned in any config, just as part of > > pool in pf.conf statement > > What about "grep -r"? My ACLs are under /etc/mail, for example. > Exactly the same, nothing, just for info. Milan From owner-freebsd-pf@freebsd.org Mon Jun 29 09:04:53 2015 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 818AB98C61F for ; Mon, 29 Jun 2015 09:04:53 +0000 (UTC) (envelope-from dhartmei@insomnia.benzedrine.ch) Received: from insomnia.benzedrine.ch (106.30.3.213.static.wline.lns.sme.cust.swisscom.ch [213.3.30.106]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "insomnia.benzedrine.ch", Issuer "StartCom Class 1 Primary Intermediate Server CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 143F012CD for ; Mon, 29 Jun 2015 09:04:52 +0000 (UTC) (envelope-from dhartmei@insomnia.benzedrine.ch) Received: from insomnia.benzedrine.ch (localhost [127.0.0.1]) by insomnia.benzedrine.ch (8.14.6/8.14.6) with ESMTP id t5T94m5V027488 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 29 Jun 2015 11:04:48 +0200 (MEST) Received: (from dhartmei@localhost) by insomnia.benzedrine.ch (8.14.6/8.14.5/Submit) id t5T94mSK008282; Mon, 29 Jun 2015 11:04:48 +0200 (MEST) Date: Mon, 29 Jun 2015 11:04:48 +0200 From: Daniel Hartmeier To: Milan Obuch Cc: Ian FREISLICH , freebsd-pf@freebsd.org Subject: Re: Large scale NAT with PF - some weird problem Message-ID: <20150629090448.GB22693@insomnia.benzedrine.ch> References: <14e119e8fa8.2755.abfb21602af57f30a7457738c46ad3ae@capeaugusta.com> <20150621195753.7b162633@zeta.dino.sk> <20150623112331.668395d1@zeta.dino.sk> <20150628100609.635544e0@zeta.dino.sk> <20150629082654.GA22693@insomnia.benzedrine.ch> <20150629105201.7ee24e38@zeta.dino.sk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20150629105201.7ee24e38@zeta.dino.sk> User-Agent: Mutt/1.5.21 (2010-09-15) X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Jun 2015 09:04:53 -0000 On Mon, Jun 29, 2015 at 10:52:01AM +0200, Milan Obuch wrote: > Does this answerred your question fully or something more would be > usefull? Which one is the magical IP address, i.e. the one that causes trouble once it's being used (I guess from x.y.26.0/27)? It's always the same one, even across reloads and reboots? Daniel From owner-freebsd-pf@freebsd.org Mon Jun 29 09:12:49 2015 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2DAA098C7E2 for ; Mon, 29 Jun 2015 09:12:49 +0000 (UTC) (envelope-from freebsd-pf@dino.sk) Received: from mailhost.netlabit.sk (mailhost.netlabit.sk [84.245.65.72]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9FDD61AC2 for ; Mon, 29 Jun 2015 09:12:48 +0000 (UTC) (envelope-from freebsd-pf@dino.sk) Received: from zeta.dino.sk (fw1.dino.sk [84.245.95.252]) (AUTH: LOGIN milan) by mailhost.netlabit.sk with ESMTPA; Mon, 29 Jun 2015 11:12:46 +0200 id 000F19B0.55910C0E.00002D83 Date: Mon, 29 Jun 2015 11:12:45 +0200 From: Milan Obuch To: Daniel Hartmeier Cc: Ian FREISLICH , freebsd-pf@freebsd.org Subject: Re: Large scale NAT with PF - some weird problem Message-ID: <20150629111245.2fab747c@zeta.dino.sk> In-Reply-To: <20150629090448.GB22693@insomnia.benzedrine.ch> References: <14e119e8fa8.2755.abfb21602af57f30a7457738c46ad3ae@capeaugusta.com> <20150621195753.7b162633@zeta.dino.sk> <20150623112331.668395d1@zeta.dino.sk> <20150628100609.635544e0@zeta.dino.sk> <20150629082654.GA22693@insomnia.benzedrine.ch> <20150629105201.7ee24e38@zeta.dino.sk> <20150629090448.GB22693@insomnia.benzedrine.ch> X-Mailer: Claws Mail 3.11.1 (GTK+ 2.24.27; i386-portbld-freebsd10.1) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Jun 2015 09:12:49 -0000 On Mon, 29 Jun 2015 11:04:48 +0200 Daniel Hartmeier wrote: > On Mon, Jun 29, 2015 at 10:52:01AM +0200, Milan Obuch wrote: > > > Does this answerred your question fully or something more would be > > usefull? > > Which one is the magical IP address, i.e. the one that causes trouble > once it's being used (I guess from x.y.26.0/27)? > > It's always the same one, even across reloads and reboots? > > Daniel x.y.27.152, it was always the same, it kept reappear randomly. Now, with pool reduced to x.y.26.0/24 it can't happen, maybe for some time this workaround will be enough. If, however, anybody would like to test something, it can be changed for test again. I did not observed this behaviour with any other IP after I upgrade to 10.2-PRERELASE, but I remember seeing this issue with x.y.27.153 and x.y.27.154, just less frequently. Most of the time I just looked for x.y.27.152, nothing more was necessary. Milan From owner-freebsd-pf@freebsd.org Mon Jun 29 09:29:38 2015 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4DCC898CAD9 for ; Mon, 29 Jun 2015 09:29:38 +0000 (UTC) (envelope-from dhartmei@insomnia.benzedrine.ch) Received: from insomnia.benzedrine.ch (106.30.3.213.static.wline.lns.sme.cust.swisscom.ch [213.3.30.106]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "insomnia.benzedrine.ch", Issuer "StartCom Class 1 Primary Intermediate Server CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id D4E6C2261 for ; Mon, 29 Jun 2015 09:29:37 +0000 (UTC) (envelope-from dhartmei@insomnia.benzedrine.ch) Received: from insomnia.benzedrine.ch (localhost [127.0.0.1]) by insomnia.benzedrine.ch (8.14.6/8.14.6) with ESMTP id t5T9TX6Y028629 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 29 Jun 2015 11:29:33 +0200 (MEST) Received: (from dhartmei@localhost) by insomnia.benzedrine.ch (8.14.6/8.14.5/Submit) id t5T9TWoB016389; Mon, 29 Jun 2015 11:29:32 +0200 (MEST) Date: Mon, 29 Jun 2015 11:29:32 +0200 From: Daniel Hartmeier To: Milan Obuch Cc: Ian FREISLICH , freebsd-pf@freebsd.org Subject: Re: Large scale NAT with PF - some weird problem Message-ID: <20150629092932.GC22693@insomnia.benzedrine.ch> References: <14e119e8fa8.2755.abfb21602af57f30a7457738c46ad3ae@capeaugusta.com> <20150621195753.7b162633@zeta.dino.sk> <20150623112331.668395d1@zeta.dino.sk> <20150628100609.635544e0@zeta.dino.sk> <20150629082654.GA22693@insomnia.benzedrine.ch> <20150629105201.7ee24e38@zeta.dino.sk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20150629105201.7ee24e38@zeta.dino.sk> User-Agent: Mutt/1.5.21 (2010-09-15) X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Jun 2015 09:29:38 -0000 On Mon, Jun 29, 2015 at 10:52:01AM +0200, Milan Obuch wrote: > Does this answerred your question fully or something more would be > usefull? How are you doing ARP? You're not assigning every address on x.y.26.0/23 as an alias, are you? So who answers ARP requests of the upstream router? Daniel From owner-freebsd-pf@freebsd.org Mon Jun 29 09:45:09 2015 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F041298CE78 for ; Mon, 29 Jun 2015 09:45:09 +0000 (UTC) (envelope-from freebsd-pf@dino.sk) Received: from mailhost.netlabit.sk (mailhost.netlabit.sk [84.245.65.72]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6D7F72A23 for ; Mon, 29 Jun 2015 09:45:08 +0000 (UTC) (envelope-from freebsd-pf@dino.sk) Received: from zeta.dino.sk (fw1.dino.sk [84.245.95.252]) (AUTH: LOGIN milan) by mailhost.netlabit.sk with ESMTPA; Mon, 29 Jun 2015 11:45:06 +0200 id 000F180F.559113A2.000031F3 Date: Mon, 29 Jun 2015 11:45:06 +0200 From: Milan Obuch To: Daniel Hartmeier Cc: Ian FREISLICH , freebsd-pf@freebsd.org Subject: Re: Large scale NAT with PF - some weird problem Message-ID: <20150629114506.1cfd6f1b@zeta.dino.sk> In-Reply-To: <20150629092932.GC22693@insomnia.benzedrine.ch> References: <14e119e8fa8.2755.abfb21602af57f30a7457738c46ad3ae@capeaugusta.com> <20150621195753.7b162633@zeta.dino.sk> <20150623112331.668395d1@zeta.dino.sk> <20150628100609.635544e0@zeta.dino.sk> <20150629082654.GA22693@insomnia.benzedrine.ch> <20150629105201.7ee24e38@zeta.dino.sk> <20150629092932.GC22693@insomnia.benzedrine.ch> X-Mailer: Claws Mail 3.11.1 (GTK+ 2.24.27; i386-portbld-freebsd10.1) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Jun 2015 09:45:10 -0000 On Mon, 29 Jun 2015 11:29:32 +0200 Daniel Hartmeier wrote: > On Mon, Jun 29, 2015 at 10:52:01AM +0200, Milan Obuch wrote: > > > Does this answerred your question fully or something more would be > > usefull? > > How are you doing ARP? > > You're not assigning every address on x.y.26.0/23 as an alias, are > you? > > So who answers ARP requests of the upstream router? There is no ARP on routed address block. In cisco speak, there is just ip route x.y.24.0 255.255.252.0 x.y.3.19 statement and that's it. Nothing more. Whole address range from x.y.24.0 to x.y.27.254 is routed here as it should be. For something like this ARP would be really evil solution. Milan From owner-freebsd-pf@freebsd.org Mon Jun 29 10:42:35 2015 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 457E498F2A1 for ; Mon, 29 Jun 2015 10:42:35 +0000 (UTC) (envelope-from ian.freislich@capeaugusta.com) Received: from mail-wi0-f180.google.com (mail-wi0-f180.google.com [209.85.212.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D4EA11CFD for ; Mon, 29 Jun 2015 10:42:34 +0000 (UTC) (envelope-from ian.freislich@capeaugusta.com) Received: by wiga1 with SMTP id a1so67412604wig.0 for ; Mon, 29 Jun 2015 03:42:27 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:in-reply-to:references :mime-version:content-type:content-id:date:message-id; bh=u93+E1ipx8NApmtTssaCb/dkow4R+m3sWi89De6zHH0=; b=IyxjcDGy8jouz2kHsB6UbEf7XjrZFWazrCsuqyoj04bLsL5Uu4CXhah39hi+agv1Dp mnF3IoRmBszi/iRmJMg0qwSiQDZHAqSEqetto5+mTH/MFIajQgD31g5M+sfe2WfrB4vn dWU8mXDWc0Ewwj/czgeIJGkkstCp7FIAfWBgRMmH1i1y3a/xIvWjt0rpzo9eQGu/WQw7 xIrVDuHWkR/qvOgwMa+VacOfQu7MZSKayKQ4HvQsWOL4kIs1uu1n9/b961RorSymqY2q KJfC9HqffffoqyrnLvDJzUmmE1DhcR8V0Urp6F8NOF6Vv9hbZyFNyo7nxTYqZmei1HgR V+dA== X-Gm-Message-State: ALoCoQnlVItan25qFncx0D+ZB5/U32XPnUYVD37HPtnfa1ATv1BlFnaC6Y/71LLMcN7oEsAFCFx9 X-Received: by 10.194.187.51 with SMTP id fp19mr25974229wjc.67.1435574546970; Mon, 29 Jun 2015 03:42:26 -0700 (PDT) Received: from clue.co.za ([197.89.34.55]) by mx.google.com with ESMTPSA id pd7sm63398018wjb.27.2015.06.29.03.42.25 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 29 Jun 2015 03:42:25 -0700 (PDT) From: Ian FREISLICH X-Google-Original-From: Ian FREISLICH Received: from localhost ([127.0.0.1] helo=zen) by clue.co.za with esmtp (Exim 4.85 (FreeBSD)) (envelope-from ) id 1Z9WW6-000PzF-PO; Mon, 29 Jun 2015 12:42:22 +0200 To: Milan Obuch cc: Daniel Hartmeier , freebsd-pf@freebsd.org Subject: Re: Large scale NAT with PF - some weird problem In-Reply-To: <20150629114506.1cfd6f1b@zeta.dino.sk> References: <20150629114506.1cfd6f1b@zeta.dino.sk> <14e119e8fa8.2755.abfb21602af57f30a7457738c46ad3ae@capeaugusta.com> <20150621195753.7b162633@zeta.dino.sk> <20150623112331.668395d1@zeta.dino.sk> <20150628100609.635544e0@zeta.dino.sk> <20150629082654.GA22693@insomnia.benzedrine.ch> <20150629105201.7ee24e38@zeta.dino.sk> <20150629092932.GC22693@insomnia.benzedrine.ch> X-Attribution: BOFH MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <99895.1435574542.1@zen> Date: Mon, 29 Jun 2015 12:42:22 +0200 Message-Id: X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Jun 2015 10:42:35 -0000 Milan Obuch wrote: > On Mon, 29 Jun 2015 11:29:32 +0200 > Daniel Hartmeier wrote: > > > On Mon, Jun 29, 2015 at 10:52:01AM +0200, Milan Obuch wrote: > > > > > Does this answerred your question fully or something more would be > > > usefull? > > > > How are you doing ARP? > > > > You're not assigning every address on x.y.26.0/23 as an alias, are > > you? > > > > So who answers ARP requests of the upstream router? > > There is no ARP on routed address block. > > In cisco speak, there is just > > ip route x.y.24.0 255.255.252.0 x.y.3.19 > > statement and that's it. Nothing more. Whole address range from > x.y.24.0 to x.y.27.254 is routed here as it should be. For something > like this ARP would be really evil solution. That's OK, as long as the NAT network is routed to your PF box it will work. The situation you mentioned in a previous message where you see lots and lots of NAT states for a single public IP address is what I suspected was happening. When you require more NAT states per IP than ephemeral ports you will run into issues because you will run out of NAT space. If the round-robin works with a smaller pool, then I suspect Glebius will be interested. Ian -- Ian Freislich From owner-freebsd-pf@freebsd.org Mon Jun 29 10:46:22 2015 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2A11D98F3F2 for ; Mon, 29 Jun 2015 10:46:22 +0000 (UTC) (envelope-from dhartmei@insomnia.benzedrine.ch) Received: from insomnia.benzedrine.ch (106.30.3.213.static.wline.lns.sme.cust.swisscom.ch [213.3.30.106]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "insomnia.benzedrine.ch", Issuer "StartCom Class 1 Primary Intermediate Server CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id B305F1DB0 for ; Mon, 29 Jun 2015 10:46:20 +0000 (UTC) (envelope-from dhartmei@insomnia.benzedrine.ch) Received: from insomnia.benzedrine.ch (localhost [127.0.0.1]) by insomnia.benzedrine.ch (8.14.6/8.14.6) with ESMTP id t5TAkE5J021579 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 29 Jun 2015 12:46:14 +0200 (MEST) Received: (from dhartmei@localhost) by insomnia.benzedrine.ch (8.14.6/8.14.5/Submit) id t5TAkET9015385; Mon, 29 Jun 2015 12:46:14 +0200 (MEST) Date: Mon, 29 Jun 2015 12:46:14 +0200 From: Daniel Hartmeier To: Milan Obuch Cc: Ian FREISLICH , freebsd-pf@freebsd.org Subject: Re: Large scale NAT with PF - some weird problem Message-ID: <20150629104614.GD22693@insomnia.benzedrine.ch> References: <20150620182432.62797ec5@zeta.dino.sk> <20150619091857.304b707b@zeta.dino.sk> <14e119e8fa8.2755.abfb21602af57f30a7457738c46ad3ae@capeaugusta.com> <20150621133236.75a4d86d@zeta.dino.sk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20150621133236.75a4d86d@zeta.dino.sk> User-Agent: Mutt/1.5.21 (2010-09-15) X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Jun 2015 10:46:22 -0000 On Sun, Jun 21, 2015 at 01:32:36PM +0200, Milan Obuch wrote: > One observation, on pfctl -vs info output - when src-limit counters > rises to 30 or so, I am getting first messages someone has problem. Is > it only coincidence or is there really some relation to my problem? This might be a clue. That counter shouldn't increase. It means something triggered a PFRES_SRCLIMIT. Are you using source tracking for anything else besides the NAT sticky address feature? If not, the only explanation for a PFRES_SRCLIMIT in a translation rule is a failure of pf.c pf_insert_src_node(), which could only be an allocation failure with uma_zalloc(). Do you see any allocation failures? Log entries about uma, "source nodes limit reached"? How about vmstat -m? Daniel From owner-freebsd-pf@freebsd.org Mon Jun 29 10:54:37 2015 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5342498F556 for ; Mon, 29 Jun 2015 10:54:37 +0000 (UTC) (envelope-from freebsd-pf@dino.sk) Received: from mailhost.netlabit.sk (mailhost.netlabit.sk [84.245.65.72]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D2B7910C3 for ; Mon, 29 Jun 2015 10:54:36 +0000 (UTC) (envelope-from freebsd-pf@dino.sk) Received: from zeta.dino.sk (fw1.dino.sk [84.245.95.252]) (AUTH: LOGIN milan) by mailhost.netlabit.sk with ESMTPA; Mon, 29 Jun 2015 12:54:33 +0200 id 000F19B1.559123E9.000038EC Date: Mon, 29 Jun 2015 12:54:32 +0200 From: Milan Obuch To: Ian FREISLICH Cc: Daniel Hartmeier , freebsd-pf@freebsd.org Subject: Re: Large scale NAT with PF - some weird problem Message-ID: <20150629125432.7aff9e66@zeta.dino.sk> In-Reply-To: References: <20150629114506.1cfd6f1b@zeta.dino.sk> <14e119e8fa8.2755.abfb21602af57f30a7457738c46ad3ae@capeaugusta.com> <20150621195753.7b162633@zeta.dino.sk> <20150623112331.668395d1@zeta.dino.sk> <20150628100609.635544e0@zeta.dino.sk> <20150629082654.GA22693@insomnia.benzedrine.ch> <20150629105201.7ee24e38@zeta.dino.sk> <20150629092932.GC22693@insomnia.benzedrine.ch> X-Mailer: Claws Mail 3.11.1 (GTK+ 2.24.27; i386-portbld-freebsd10.1) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Jun 2015 10:54:37 -0000 On Mon, 29 Jun 2015 12:42:22 +0200 Ian FREISLICH wrote: > Milan Obuch wrote: > > On Mon, 29 Jun 2015 11:29:32 +0200 > > Daniel Hartmeier wrote: > > > > > On Mon, Jun 29, 2015 at 10:52:01AM +0200, Milan Obuch wrote: > > > > > > > Does this answerred your question fully or something more would > > > > be usefull? > > > > > > How are you doing ARP? > > > > > > You're not assigning every address on x.y.26.0/23 as an alias, are > > > you? > > > > > > So who answers ARP requests of the upstream router? > > > > There is no ARP on routed address block. > > > > In cisco speak, there is just > > > > ip route x.y.24.0 255.255.252.0 x.y.3.19 > > > > statement and that's it. Nothing more. Whole address range from > > x.y.24.0 to x.y.27.254 is routed here as it should be. For something > > like this ARP would be really evil solution. > > That's OK, as long as the NAT network is routed to your PF box it > will work. > This was just an explanation, I am sure this is OK, as I have some network experience already for... well, a ong time. > The situation you mentioned in a previous message where you see > lots and lots of NAT states for a single public IP address is what > I suspected was happening. When you require more NAT states per > IP than ephemeral ports you will run into issues because you will > run out of NAT space. > No, there were not much states per problematic IP, maybe just tens of them for one or couple internal IPs. That's weird. > If the round-robin works with a smaller pool, then I suspect Glebius > will be interested. > Well, if he chimes in, I would only welcome that. Currently I am waiting for any signs of troubles with shrinked pool, if there will be any. Milan From owner-freebsd-pf@freebsd.org Mon Jun 29 10:58:37 2015 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9E9FD98F626 for ; Mon, 29 Jun 2015 10:58:37 +0000 (UTC) (envelope-from ian.freislich@capeaugusta.com) Received: from mail-wi0-f178.google.com (mail-wi0-f178.google.com [209.85.212.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 569131180 for ; Mon, 29 Jun 2015 10:58:37 +0000 (UTC) (envelope-from ian.freislich@capeaugusta.com) Received: by wiwl6 with SMTP id l6so95708808wiw.0 for ; Mon, 29 Jun 2015 03:58:35 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:in-reply-to:references :mime-version:content-type:content-id:date:message-id; bh=9TkfxpxB5K/cbCucygS8m3TAtorCgsXzOb8FL/nvcHI=; b=SXwlAUl3CT6Q4mByqfl1I1DR477+GIJoAIFcM00EuNWKx8pVUWMyc7rA19KYMuw/vL //Y4OmyP1/Di9SMAoQ5LnlEbsuxzbt1MIknKembqnygvG4f5xud37OrlKc6rVTiYG/mm qUiNiSB8QGJRtDP56o2HlwUBjfUs1p3Hl5o8Qz8iKSNtjccIsXvNxHlgtKAOe6cmYsTX t5lX2mQowQORmjdnV5TD8h2M4V983b+yjjNS6fWwQ7pX3SAEcLulRiEB4PG2SIgZepI0 QRQJgU2JT8uNxzH6/EVEt7LvnCjw4r9GPjwtXlzG+g1Q/FWJxGS7iXzpeRHzg4XtkzNa TQ+Q== X-Gm-Message-State: ALoCoQm3Ewo5rZfIP6FSgk24aXb6uYEmvIdhlyvcBxdCRjL663bctXJAUa9cP59F1tE7EvM4bUnU X-Received: by 10.180.105.227 with SMTP id gp3mr21692311wib.56.1435575515718; Mon, 29 Jun 2015 03:58:35 -0700 (PDT) Received: from clue.co.za ([197.89.34.55]) by mx.google.com with ESMTPSA id gw7sm11595024wib.15.2015.06.29.03.58.34 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 29 Jun 2015 03:58:34 -0700 (PDT) From: Ian FREISLICH X-Google-Original-From: Ian FREISLICH Received: from localhost ([127.0.0.1] helo=zen) by clue.co.za with esmtp (Exim 4.85 (FreeBSD)) (envelope-from ) id 1Z9Wlk-00003g-0l; Mon, 29 Jun 2015 12:58:32 +0200 To: Milan Obuch cc: Daniel Hartmeier , freebsd-pf@freebsd.org Subject: Re: Large scale NAT with PF - some weird problem In-Reply-To: <20150629125432.7aff9e66@zeta.dino.sk> References: <20150629125432.7aff9e66@zeta.dino.sk> <20150629114506.1cfd6f1b@zeta.dino.sk> <14e119e8fa8.2755.abfb21602af57f30a7457738c46ad3ae@capeaugusta.com> <20150621195753.7b162633@zeta.dino.sk> <20150623112331.668395d1@zeta.dino.sk> <20150628100609.635544e0@zeta.dino.sk> <20150629082654.GA22693@insomnia.benzedrine.ch> <20150629105201.7ee24e38@zeta.dino.sk> <20150629092932.GC22693@insomnia.benzedrine.ch> X-Attribution: BOFH MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <226.1435575512.1@zen> Date: Mon, 29 Jun 2015 12:58:32 +0200 Message-Id: X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Jun 2015 10:58:37 -0000 Milan Obuch wrote: > > No, there were not much states per problematic IP, maybe just tens of > them for one or couple internal IPs. That's weird. What's the output of 'pfctl -sa' (without the states). Ian -- Ian Freislich From owner-freebsd-pf@freebsd.org Mon Jun 29 11:05:23 2015 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5769198F7AD for ; Mon, 29 Jun 2015 11:05:23 +0000 (UTC) (envelope-from freebsd-pf@dino.sk) Received: from mailhost.netlabit.sk (mailhost.netlabit.sk [84.245.65.72]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E44F81595 for ; Mon, 29 Jun 2015 11:05:22 +0000 (UTC) (envelope-from freebsd-pf@dino.sk) Received: from zeta.dino.sk (fw1.dino.sk [84.245.95.252]) (AUTH: LOGIN milan) by mailhost.netlabit.sk with ESMTPA; Mon, 29 Jun 2015 13:05:19 +0200 id 000F19B1.5591266F.00003A35 Date: Mon, 29 Jun 2015 13:05:19 +0200 From: Milan Obuch To: Daniel Hartmeier Cc: Ian FREISLICH , freebsd-pf@freebsd.org Subject: Re: Large scale NAT with PF - some weird problem Message-ID: <20150629130519.168f0efc@zeta.dino.sk> In-Reply-To: <20150629104614.GD22693@insomnia.benzedrine.ch> References: <20150620182432.62797ec5@zeta.dino.sk> <20150619091857.304b707b@zeta.dino.sk> <14e119e8fa8.2755.abfb21602af57f30a7457738c46ad3ae@capeaugusta.com> <20150621133236.75a4d86d@zeta.dino.sk> <20150629104614.GD22693@insomnia.benzedrine.ch> X-Mailer: Claws Mail 3.11.1 (GTK+ 2.24.27; i386-portbld-freebsd10.1) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Jun 2015 11:05:23 -0000 On Mon, 29 Jun 2015 12:46:14 +0200 Daniel Hartmeier wrote: > On Sun, Jun 21, 2015 at 01:32:36PM +0200, Milan Obuch wrote: > > > One observation, on pfctl -vs info output - when src-limit counters > > rises to 30 or so, I am getting first messages someone has problem. > > Is it only coincidence or is there really some relation to my > > problem? > > This might be a clue. That counter shouldn't increase. It means > something triggered a PFRES_SRCLIMIT. > OK, I will keep an eye on this for some time too. I do not have much knowledge regarding pf internals, so my observations may or may not be relevant, just as my questions. > Are you using source tracking for anything else besides the NAT sticky > address feature? > I reviewed recently some pfctl output and I think this mechanism is used in other scenarios as well, namely following one for ssh protection: block in quick on $if_ext inet proto tcp from to any port 22 pass in on $if_ext proto tcp to x.y.24.0/22 port ssh flags S/SA keep state (max-src-conn 10, max-src-conn-rate 5/5, overload flush) (somewhat mail-mangled, but I am sure you know this one) > If not, the only explanation for a PFRES_SRCLIMIT in a translation > rule is a failure of pf.c pf_insert_src_node(), which could only be an > allocation failure with uma_zalloc(). > > Do you see any allocation failures? Log entries about uma, "source > nodes limit reached"? How about vmstat -m? > Where should these failures come? I see nothing in /var/log/messages. As for 'vmstat -m', I think following lines could be of some interest: Type InUse MemUse HighUse Requests Size(s) pf_hash 3 1728K - 3 pf_temp 0 0K - 955 32,64 pf_ifnet 21 7K - 282 128,256,2048 pf_osfp 1130 102K - 6780 32,128 pf_rule 222 129K - 468 128,1024 pf_table 9 18K - 35 2048 but no idea how to interpret this. Regards, Milan From owner-freebsd-pf@freebsd.org Mon Jun 29 11:09:07 2015 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 05F7798F849 for ; Mon, 29 Jun 2015 11:09:07 +0000 (UTC) (envelope-from freebsd-pf@dino.sk) Received: from mailhost.netlabit.sk (mailhost.netlabit.sk [84.245.65.72]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 714C51646 for ; Mon, 29 Jun 2015 11:09:05 +0000 (UTC) (envelope-from freebsd-pf@dino.sk) Received: from zeta.dino.sk (fw1.dino.sk [84.245.95.252]) (AUTH: LOGIN milan) by mailhost.netlabit.sk with ESMTPA; Mon, 29 Jun 2015 13:09:03 +0200 id 000F19B1.5591274F.00003A9B Date: Mon, 29 Jun 2015 13:09:02 +0200 From: Milan Obuch To: Ian FREISLICH Cc: Daniel Hartmeier , freebsd-pf@freebsd.org Subject: Re: Large scale NAT with PF - some weird problem Message-ID: <20150629130902.56ce3a8a@zeta.dino.sk> In-Reply-To: References: <20150629125432.7aff9e66@zeta.dino.sk> <20150629114506.1cfd6f1b@zeta.dino.sk> <14e119e8fa8.2755.abfb21602af57f30a7457738c46ad3ae@capeaugusta.com> <20150621195753.7b162633@zeta.dino.sk> <20150623112331.668395d1@zeta.dino.sk> <20150628100609.635544e0@zeta.dino.sk> <20150629082654.GA22693@insomnia.benzedrine.ch> <20150629105201.7ee24e38@zeta.dino.sk> <20150629092932.GC22693@insomnia.benzedrine.ch> X-Mailer: Claws Mail 3.11.1 (GTK+ 2.24.27; i386-portbld-freebsd10.1) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Jun 2015 11:09:07 -0000 On Mon, 29 Jun 2015 12:58:32 +0200 Ian FREISLICH wrote: > Milan Obuch wrote: > > > > No, there were not much states per problematic IP, maybe just tens > > of them for one or couple internal IPs. That's weird. > > What's the output of 'pfctl -sa' (without the states). > > Ian > Well, it has some ~ 50k lines, and I see no issue currently. I will try to catch it as soon as this issue reappears. Will keep you informed. Milan From owner-freebsd-pf@freebsd.org Mon Jun 29 18:45:47 2015 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4EC5398F5BD for ; Mon, 29 Jun 2015 18:45:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3BCB81A4A for ; Mon, 29 Jun 2015 18:45:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t5TIjlVM014048 for ; Mon, 29 Jun 2015 18:45:47 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-pf@FreeBSD.org Subject: [Bug 193620] Problem with igb multiqueue together with pf Date: Mon, 29 Jun 2015 18:45:47 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.0-RELEASE X-Bugzilla-Keywords: IntelNetworking X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: sbruno@FreeBSD.org X-Bugzilla-Status: In Progress X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_status Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Jun 2015 18:45:47 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=193620 Sean Bruno changed: What |Removed |Added ---------------------------------------------------------------------------- Status|New |In Progress --- Comment #1 from Sean Bruno --- hw.igb.enable_msix=0 disables MSIX features which *does* disable multiqueue. You can however, set hw.igb.num_queues=1 to use MSIX and only 1 queue. Can you test this with 10.1 release and 10.2 release beta when available? -- You are receiving this mail because: You are on the CC list for the bug.