Date: Thu, 7 Apr 2016 17:08:38 +0100 From: Dr Josef Karthauser <joe@truespeed.com> To: FreeBSD Stable <stable@freebsd.org> Cc: freebsd-net@freebsd.org Subject: IPFW with NAT : Problems with duplicate packets on FreeBSD 10.3-RC3 Message-ID: <A03E136A-7599-4992-9F9E-13E7350F972B@truespeed.com>
next in thread | raw e-mail | index | archive | help
I=E2=80=99m scratching my head with an IPFW / NAT configuration; could = someone please throw me a bone? I=E2=80=99ve got a jail, and I=E2=80=99m NATing using IPFW to connect it = to the outside world. In particular I=E2=80=99m forwarding port 8080 from the host=E2=80=99s = public address to the jail=E2=80=99s private address. When I pull an HTTP connection from port publicip:8080 I get the first = packet of the TCP stream twice, and then the HTTP connection fails. That ought not to happen :(. The firewall rule is very simple nat 1 config if vlan10 reset redirect_port tcp 10.17.0.16:8080 8080 // = NAT for jails - forward to portal on 8080 nat 1 ip from any to any via vlan10 in nat 1 ip from any to any via vlan10 out add allow ip from any to any If I tcpdump on the host: # tcpdump -i vlan10 port 8080 tcpdump: verbose output suppressed, use -v or -vv for full protocol = decode listening on vlan10, link-type EN10MB (Ethernet), capture size 65535 = bytes 17:02:02.478760 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [S], seq = 3088565770, win 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val = 672977930 ecr 0,sackOK,eol], length 0 17:02:02.478797 IP X.X.X.216.8080 > X.X.X.211.63289: Flags [S.], seq = 425576427, ack 3088565771, win 65535, options [mss 1460,nop,wscale = 6,sackOK,TS val 1035319863 ecr 672977930], length 0 17:02:02.480137 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [.], ack 1, = win 4117, options [nop,nop,TS val 672977931 ecr 1035319863], length 0 17:02:02.480393 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [P.], seq = 1:86, ack 1, win 4117, options [nop,nop,TS val 672977931 ecr = 1035319863], length 85 17:02:02.714225 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [P.], seq = 1:86, ack 1, win 4117, options [nop,nop,TS val 672978161 ecr = 1035319863], length 85 17:02:02.975220 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [P.], seq = 1:86, ack 1, win 4117, options [nop,nop,TS val 672978421 ecr = 1035319863], length 85 17:02:02.975239 IP X.X.X.216.8080 > X.X.X.211.63289: Flags [.], seq = 1:1449, ack 86, win 1040, options [nop,nop,TS val 1035320360 ecr = 672977931], length 1448 17:02:03.079324 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [.], ack = 1449, win 4096, options [nop,nop,TS val 672978522 ecr 1035320360], = length 0 17:02:03.079336 IP X.X.X.216.8080 > X.X.X.211.63289: Flags [.], seq = 1449:4345, ack 86, win 1040, options [nop,nop,TS val 1035320464 ecr = 672978522], length 2896 17:02:03.080931 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [.], ack = 4345, win 4050, options [nop,nop,TS val 672978523 ecr 1035320464], = length 0 17:02:03.578732 IP X.X.X.216.8080 > X.X.X.211.63289: Flags [.], seq = 4345:5793, ack 86, win 1040, options [nop,nop,TS val 1035320963 ecr = 672978523], length 1448 17:02:03.725858 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [.], ack = 5793, win 4096, options [nop,nop,TS val 672979158 ecr 1035320963], = length 0 17:02:03.725888 IP X.X.X.216.8080 > X.X.X.211.63289: Flags [.], seq = 5793:8689, ack 86, win 1040, options [nop,nop,TS val 1035321110 ecr = 672979158], length 2896 17:02:03.727352 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [.], ack = 8689, win 4050, options [nop,nop,TS val 672979159 ecr 1035321110], = length 0 17:02:04.260416 IP X.X.X.216.8080 > X.X.X.211.63289: Flags [.], seq = 8689:10137, ack 86, win 1040, options [nop,nop,TS val 1035321645 ecr = 672979159], length 1448 17:02:04.340844 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [.], ack = 10137, win 4096, options [nop,nop,TS val 672979770 ecr 1035321645], = length 0 17:02:04.340855 IP X.X.X.216.8080 > X.X.X.211.63289: Flags [.], seq = 10137:13033, ack 86, win 1040, options [nop,nop,TS val 1035321725 ecr = 672979770], length 2896 17:02:04.342775 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [F.], seq 86, = ack 11585, win 4096, options [nop,nop,TS val 672979771 ecr 1035321725], = length 0 17:02:04.342803 IP X.X.X.216.8080 > X.X.X.211.63289: Flags [.], seq = 13033:15929, ack 87, win 1040, options [nop,nop,TS val 1035321727 ecr = 672979771], length 2896 17:02:04.343154 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [R], seq = 3088565856, win 0, length 0 17:02:04.344440 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [R], seq = 3088565857, win 0, length 0 17:02:04.344740 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [R], seq = 3088565857, win 0, length 0 And the client doing the http request gets: phoenix:~ joe$ curl -v http://X.X.X.216:8080/ * Trying 31.210.26.216... * Connected to X.X.X.216 port 8080 (#0) > GET / HTTP/1.1 > Host: x.x.com:8080 > User-Agent: curl/7.43.0 > Accept: */* >=20 < HTTP/1.1 200 OK < Server: Apache-Coyote/1.1 < Content-Type: text/html;charset=3DISO-8859-1 < Transfer-Encoding: chunked < Date: Thu, 07 Apr 2016 16:02:02 GMT <=20 <!DOCTYPE html> <html lang=3D"en"> <head> <title>Apache Tomcat/7.0.68</title> <link href=3D"favicon.ico" rel=3D"icon" type=3D"image/x-icon" /> <link href=3D"favicon.ico" rel=3D"shortcut icon" = type=3D"image/x-icon" /> <link href=3D"tomcat.css" rel=3D"stylesheet" type=3D"text/css" = /> </head> <body> <div id=3D"wrapper"> <div id=3D"navigation" class=3D"curved container"> <span id=3D"nav-home"><a = href=3D"http://tomcat.apache.org/">Home</a></span>; <span id=3D"nav-hosts"><a = href=3D"/docs/">Documentation</a></span> <span id=3D"nav-config"><a = href=3D"/docs/config/">Configuration</a></span> <span id=3D"nav-examples"><a = href=3D"/examples/">Examples</a></span> <span id=3D"nav-wiki"><a = href=3D"http://wiki.apache.org/tomcat/FrontPage">Wiki</a></span>; [CUT] <div class=3D"col20"> <div class=3D"container"> <h4>Other Documentation</h4> <ul> <li><a = href=3D"http://tomcat.apache.org/connectors-doc/">Tomcat = Connectors</a></li> <li><a = href=3D"http://tomcat.apache.org/connectors-doc/">mod_jk = Documentation</a></li> HTTP/1.1 200 OK Server: Apache-Coyote/1.1 Content-Type: text/html;charset=3DISO-8859-1 Transfer-Encoding: chunked Date: Thu, 07 Apr 2016 16:02:02 GMT 2000 <!DOCTYPE html> <html lang=3D"en"> <head> <title>Apache Tomcat/7.0.68</title> <link href=3D"favicon.ico" rel=3D"icon" type=3D"image/x-icon" /> <link href=3D"favicon.ico" rel=3D"shortcut icon" = type=3D"image/x-icon" /> <link href=3D"tomcat.css" rel=3D"stylesheet" type=3D"text/css" = /> </head> <body> <div id=3D"wrapper"> <div id=3D"navigation" class=3D"curved container"> [CUT] </div> </div> <div id=3D"actions"> <div class=3D"button"> <a class=3D"container shadow" = href=3D"/manager/status"><span>Server Status</span></a> * Malformed encoding found in chunked-encoding * Closing connection 0 curl: (56) Malformed encoding found in chunked-encoding phoenix:~ joe$=20 Looks like the first packet is being retransmitted, which means that the = nat is probably misconfigured and the TCP connection is broken in some = strange way. Does anyone have a clue as to where to look? The ipfw rules are simple = enough - what have I missed? Thanks, Joe p.s. I also have one_pass disabled: # sysctl net.inet.ip.fw.one_pass net.inet.ip.fw.one_pass: 0 =20 =E2=80=94=20 Dr Josef Karthauser Chief Technical Officer (01225) 300371 / (07703) 596893 www.truespeed.com <http://www.truespeed.com/>; / theTRUESPEED <http://www.facebook.com/theTRUESPEED>=20 @theTRUESPEED <https://twitter.com/thetruespeed>; =20 This email contains TrueSpeed information, which may be privileged or = confidential. It's meant only for the individual(s) or entity named = above. If you're not the intended recipient, note that disclosing, = copying, distributing or using this information is prohibited. If you've = received this email in error, please let me know immediately on the = email address above. Thank you. We monitor our email system, and may record your emails.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?A03E136A-7599-4992-9F9E-13E7350F972B>