Date: Fri, 22 Feb 2019 17:09:46 +0100 (CET) From: Giacomo Olgeni <olgeni@olgeni.com> To: freebsd-net@FreeBSD.org Subject: IPFW NAT in VNET jail Message-ID: <alpine.OSX.2.21.1902221706270.55983@macbook.local>
next in thread | raw e-mail | index | archive | help
Hi,
I have been playing around with IPFW and in-kernel NAT in a VNET jail, in
order to implement NAT before IPsec. The IPsec part is provided by a
different VNET jail that seems to be working fine.
Problem is, I could not get NAT working in any way so far. And just to be
sure, I already disabled anything related to checksum offloading from all
the physical network cards (txcsum, rxcsum, tso, etc.).
My routing is from epair1b (192.168.3.0/24) to epair0b (10.64.23.34, using
10.64.23.33 as gw).
I can see that NAT is kind of doing what is expected, but then something
weird happens.
I run "telnet 10.101.214.155 80" (to reach the remote IPsec subnet)
from 192.168.3.9; connection goes through 192.168.3.223 (the gateway
jail, never mind my temporary addresses), and from there it goes out
through NAT to the IPsec jail.
tcpdump on the NAT jail (outbound interface epair0b) shows this:
17:00:04.330364 IP (tos 0x10, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 60)
10.64.23.34.56647 > 10.101.214.155.http: Flags [S], cksum 0x4112 (correct), seq 3633527062, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 1310639916 ecr 0], length 0
17:00:04.349254 IP (tos 0x0, ttl 59, id 0, offset 0, flags [DF], proto TCP (6), length 56)
10.101.214.155.http > 10.64.23.34.56647: Flags [S.], cksum 0xede4 (correct), seq 619471197, ack 3633527063, win 4344, options [mss 1360,sackOK,TS val 91081438 ecr 1310639916], length 0
17:00:04.349464 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
10.64.23.34.58666 > 10.101.214.155.http: Flags [R], cksum 0xb658 (correct), seq 3633527063, win 0, length 0
So, I'm get the right SYN/ACK to the right port, but then a RST is sent
from an unrelated port, and the connection starts retransmitting the SYN.
I have "nat 1 config log if epair0b", and "ipfw show" lists the following rules:
00100 0 0 nat 1 ip from any to any recv epair0b
00200 74 4080 nat 1 ip from any to any xmit epair0b
00300 0 0 check-state :default
00400 6 360 allow tcp from any to any out xmit epair0b setup keep-state :default
00500 0 0 allow udp from any to any out xmit epair0b setup
65535 113149 11125118 allow ip from any to any
I'm running with one_pass=1. If I set one_pass=0 the only change is that
rule 400 is hit by packets going out from NAT, but the result doesn't
change.
I added rule 100 to see if I had any match from the incoming SYN/ACK, but
the counter is always 0.
Rule 300 has 0 packets too (even with one_pass=0), so it's not clear
to me whether the SYN/ACK is actually being seen by the firewall or if
the RST is coming from somewhere else.
I tried to run dwatch from the jail host and got this:
# sudo dwatch -X tcp | grep 10.64.23.34
INFO Sourcing tcp profile [found in /usr/libexec/dwatch]
INFO Watching 'tcp:::accept-established, [...]'
2019 Feb 22 17:00:04 0.0 intr[12]: 10.64.23.34:56647 <- 10.101.214.155:80 34 bytes
2019 Feb 22 17:00:04 0.0 intr[12]: 10.64.23.34:56647 X- 10.101.214.155:80
2019 Feb 22 17:00:04 0.0 intr[12]: 10.64.23.34:56647 -> 10.101.214.155:80 19 bytes
The events look the same but there's no port 58666 anywhere here, which is
where the RST comes from.
My naive interpretation would be that NAT has nowhere to map the SYN/ACK
for some reason, possibly because rule 100 is never hit? But that wouldn't
explain the odd port number.
I'm a bit out of ideas short of not using a VNET jail and trying to do
everything from a bhyve VM, so any suggestion/interpretation is welcome :)
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?alpine.OSX.2.21.1902221706270.55983>