From owner-freebsd-ipfw@FreeBSD.ORG Mon Jul 19 12:08:57 2010 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7C08C106566B for ; Mon, 19 Jul 2010 12:08:57 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from sola.nimnet.asn.au (paqi.nimnet.asn.au [115.70.110.159]) by mx1.freebsd.org (Postfix) with ESMTP id BE3C28FC0C for ; Mon, 19 Jul 2010 12:08:56 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by sola.nimnet.asn.au (8.14.2/8.14.2) with ESMTP id o6JC8sG4015425; Mon, 19 Jul 2010 22:08:54 +1000 (EST) (envelope-from smithi@nimnet.asn.au) Date: Mon, 19 Jul 2010 22:08:53 +1000 (EST) From: Ian Smith To: Mamontov Roman In-Reply-To: <1207784719.20100719143104@gmail.com> Message-ID: <20100719203632.G86988@sola.nimnet.asn.au> References: <1931583025.20100715114512@gmail.com> <20100715183743.S86988@sola.nimnet.asn.au> <893037983.20100719092644@gmail.com> <20100719181208.A86988@sola.nimnet.asn.au> <1207784719.20100719143104@gmail.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Cc: freebsd-ipfw@freebsd.org Subject: Re: Problem with ipfw nat and packet to local services X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Jul 2010 12:08:57 -0000 On Mon, 19 Jul 2010, Mamontov Roman wrote: > > What's the value of sysctl net.inet.ip.fw.one_pass ? It needs to be 0 > > so that packets will re-enter the firewall after NAT processing. > > > Otherwise, it might help to > > > a) run 'ipfw zero' before any tests .. I'm wondering about all those > > packets hitting rule 65535; were they from before adding rule 65000? > > > b) add some count rules before and after nat, to show all packets > > that may be eligible for NAT translation, maybe something like: > > > 00020 count log ip from any to any in recv ${ext_if} > > 00022 count log ip from any to any out xmit ${ext_if} > > 00024 count log ip from any to any out recv ${int_if} xmit ${ext_if} > > > 00035 nat ... > > > 00040 count log ip from any to any in recv ${ext_if} > > 00042 count log ip from any to any out xmit ${ext_if} > > 00044 count log ip from any to any out recv ${int_if} xmit ${ext_if} > > > So you actually get to see the flow of packets before and after nat, > > both to/from the local box and packets mapped to/rom inside addresses. > > Again, an 'ipfw zero' before tests will make packet counts clearer. > > > Of course something like '# tcpdump -pn -i ext_if' will also show all > > packets via ext_if in some detail. Be more specific if just looking for > > some particular flows, like maybe appending 'udp port NNNNN' to that. > > > That is, try to follow packets you'd expect to be coming in for services > > on the local box so if they are disappearing, you'll know where or why. > > 'netstat -finet -an' will show all those services that are listening. > > > If that doesn't help, we'll need more information. > > > cheers, Ian > > # sysctl net.inet.ip.fw.one_pass > net.inet.ip.fw.one_pass: 0 > > # ipfw show 20-49 > 00020 40 2016 count log ip from any to me dst-port 22 in recv ext_if1 > 00021 0 0 count log ip from me 22 to any out xmit ext_if1 > 00035 13192 9028716 nat 1 ip from any to any via ext_if1 > 00040 0 0 count log ip from any to me dst-port 22 in recv ext_if1 > 00041 0 0 count log ip from me 22 to any out xmit ext_if1 Ouch. It does look like nat is just swallowing the ssh packets on the inbound pass .. these surely should appear again after NAT (unchanged). > # ipfw nat show config > ipfw nat 1 config ip xxx.xxx.xxx.xxx > > # tcpdump -pn -i ext_if1 'host yyy.yyy.yyy.yyy' > 14:12:48.885011 IP yyy.yyy.yyy.yyy.2777 > xxx.xxx.xxx.xxx.22: Flags [S], seq 2880611174, win 65535, options [mss 1460,nop,wscale 1,nop,nop,TS[|tcp]> > 14:12:51.888823 IP yyy.yyy.yyy.yyy.2777 > xxx.xxx.xxx.xxx.22: Flags [S], seq 2880611174, win 65535, options [mss 1460,nop,wscale 1,nop,nop,TS[|tcp]> > 14:12:54.884966 IP yyy.yyy.yyy.yyy.2777 > xxx.xxx.xxx.xxx.22: Flags [S], seq 2880611174, win 65535, options [mss 1460,nop,wscale 1,nop,nop,TS[|tcp]> > 14:12:57.884090 IP yyy.yyy.yyy.yyy.2777 > xxx.xxx.xxx.xxx.22: Flags [S], seq 2880611174, win 65535, options [mss 1460], length 0 > 14:13:00.885131 IP yyy.yyy.yyy.yyy.2777 > xxx.xxx.xxx.xxx.22: Flags [S], seq 2880611174, win 65535, options [mss 1460], length 0 > 14:13:03.887094 IP yyy.yyy.yyy.yyy.2777 > xxx.xxx.xxx.xxx.22: Flags [S], seq 2880611174, win 65535, options [mss 1460], length 0 Ah, if that 'TS' means TSO - tcpdump(1) doesn't mention either - then from the last section of ipfw(8): Due to the architecture of libalias(3), ipfw nat is not compatible with the TCP segmentation offloading (TSO). Thus, to reliably nat your net- work traffic, please disable TSO on your NICs using ifconfig(8). ifconfig ext_if1 ? > Output > # netstat -finet -an | grep yyy.yyy.yyy.yyy > is blank. > > Without rule 35 nat 1 ip from any to any via ext_if1 inbound packet to ssh (for example) > pass correctly. > > # ipfw delete 35 > tcpdump -pn -i ext_if 'host yyy.yyy.yyy.yyy' > 14:21:45.467233 IP yyy.yyy.yyy.yyy.2790 > xxx.xxx.xxx.xxx.22: Flags [S], seq 376101413, win 65535, options [mss 1460,nop,wscale 1,nop,nop,TS[|tcp]> > 14:21:45.467670 IP xxx.xxx.xxx.xxx.22 > xxx.xxx.xxx.xxx.2790: Flags [S.], seq 3270699616, ack 376101414, win 65535, options [mss 1460,nop,wscale 3,nop,nop,TS[|tcp]> I guess that dest is just mis-obscured, ie yyy.yyy.yyy.yyy:2790 > 14:21:45.468960 IP yyy.yyy.yyy.yyy.2790 > xxx.xxx.xxx.xxx.22: Flags [.], ack 1, win 33304, options [nop,nop,TS val 40088404 ecr 1166915706], length 0 > 14:21:45.527438 IP xxx.xxx.xxx.xxx.22 > yyy.yyy.yyy.yyy.2790: Flags [P.], ack 1, win 8326, options [nop,nop,TS val 1166915766 ecr 40088404], length 40 > > # netstat -finet -an | grep yyy.yyy.yyy.yyy > tcp4 0 0 xxx.xxx.xxx.xxx.22 yyy.yyy.yyy.yyy.2790 FIN_WAIT_2 > > 00020 8 1403 count log ip from any to me dst-port 22 in recv ext_if1 > 00021 6 2280 count log ip from me 22 to any out xmit ext_if1 > 00040 8 1403 count log ip from any to me dst-port 22 in recv ext_if1 > 00041 6 2280 count log ip from me 22 to any out xmit ext_if1 > > Any ideas? [later: if you had TSO enabled on the interface, ignore the below ..] I see you're not logging nat packets now, and wonder if there'd be any more clues in the log, but then, does nat log packets it doesn't touch? Unless I'm completely missing something (wouldn't be the first time :) this looks like a bug, and unless someone else has a better clue, you should file a PR with what you've demonstrated to date. cheers, Ian (only vaguely useful with usage, and not at all with code)