Date: Tue, 19 May 2015 00:41:22 -0700 From: Wojciech Wojtyniak <wow@0x89.net> To: freebsd-net@freebsd.org Subject: Problem with ipfw, in-kernel NAT and port redirection to jails Message-ID: <A88A7FED-B5DD-4B1E-96A4-AE1F3EAB8E30@0x89.net>
next in thread | raw e-mail | index | archive | help
Hello, I have a vps on vultr.com <http://vultr.com/>; running FreeBSD 10.1-p9 = and a generic kernel: % uname -a FreeBSD tzar 10.1-RELEASE-p9 FreeBSD 10.1-RELEASE-p9 #0: Tue Apr 7 = 01:09:46 UTC 2015 = root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 My goal is to run multiple services in jails (hopefully using ezjail or = other convenient manager) and make them accessible from the Internet = only on arbitrary ports (like 80 for http(s) server). So far my approach = is as follows: I clone the lo0 interface and assign IP from 127.0.0.0/8 = space to the jail and redirect port in ipfw nat definition to given = address (example in configs below, I tried also with other addresses on = vtnet0, which is my base network interface, with similar issues). = Unfortunately this configuration doesn=E2=80=99t work for me. I tested this for znc (IRC bouncer) and nginx. If I run them on main = host (without NAT in front of them) everything works fine. However, if I = run them in jail, behind NAT and send a HTTP(S) request to get some = file, connections get dropped (znc has a web admin module, which is = broken because of that). It works fine for small files but breaks for = larger (I haven=E2=80=99t check the threshold but can do this if this is = necessary). For example given curl command and znc service: % curl http://$my_ip:6697/pub/jquery-1.11.2.min.js > /dev/null # (stats cut out) curl: (18) transfer closed with 58648 bytes remaining to read if I tcpdump connection, transfer looks fine for some time and then ends = with a following sequence (run on main host, jail master): % sudo tcpdump port 6697 tcpdump: verbose output suppressed, use -v or -vv for full protocol = decode listening on vtnet0, link-type EN10MB (Ethernet), capture size 65535 = bytes 23:37:28.621409 IP $my_home_host.51256 > $my_vultr_host.vultr.com.6697: = Flags [S], seq 3967654146, win 65535, options [mss 1460,nop,wscale = 5,nop,nop,TS val 601353780 ecr 0,sackOK,eol], length 0 23:37:28.621468 IP $my_vultr_host.vultr.com.6697 > $my_home_host.51256: = Flags [S.], seq 517055725, ack 3967654147, win 65535, options [mss = 1460,nop,wscale 6,sackOK,TS val 2553669008 ecr 601353780], length 0 23:37:28.635788 IP $my_home_host.51256 > $my_vultr_host.vultr.com.6697: = Flags [.], ack 1, win 4117, options [nop,nop,TS val 601353791 ecr = 2553669008], length 0 23:37:28.635865 IP $my_home_host.51256 > $my_vultr_host.vultr.com.6697: = Flags [P.], seq 1:109, ack 1, win 4117, options [nop,nop,TS val = 601353791 ecr 2553669008], length 108 23:37:28.636122 IP $my_vultr_host.vultr.com.6697 > $my_home_host.51256: = Flags [P.], seq 1:18, ack 109, win 1040, options [nop,nop,TS val = 2553669022 ecr 601353791], length 17 23:37:28.650153 IP $my_home_host.51256 > $my_vultr_host.vultr.com.6697: = Flags [.], ack 18, win 4117, options [nop,nop,TS val 601353805 ecr = 2553669022], length 0 23:37:29.123244 IP $my_vultr_host.vultr.com.6697 > $my_home_host.51256: = Flags [.], seq 18:1466, ack 109, win 1040, options [nop,nop,TS val = 2553669510 ecr 601353805], length 1448 (transfer goes normally) 23:37:35.519163 IP $my_vultr_host.vultr.com.6697 > $my_home_host.51256: = Flags [F.], seq 37666, ack 109, win 1040, options [nop,nop,TS val = 2553675906 ecr 601360615], length 0 23:37:35.531004 IP $my_home_host.51256 > $my_vultr_host.vultr.com.6697: = Flags [.], ack 33322, win 4096, options [nop,nop,TS val 601360640 ecr = 2553675880], length 0 23:37:36.165352 IP $my_vultr_host.vultr.com.6697 > $my_home_host.51256: = Flags [.], seq 33322:34770, ack 109, win 1040, options [nop,nop,TS val = 2553676552 ecr 601360640], length 1448 23:37:36.184582 IP $my_home_host.51256 > $my_vultr_host.vultr.com.6697: = Flags [.], ack 34770, win 4050, options [nop,nop,TS val 601361283 ecr = 2553676552], length 0 23:37:36.801437 IP $my_vultr_host.vultr.com.6697 > $my_home_host.51256: = Flags [.], seq 34770:36218, ack 109, win 1040, options [nop,nop,TS val = 2553677188 ecr 601361283], length 1448 23:37:36.910742 IP $my_home_host.51256 > $my_vultr_host.vultr.com.6697: = Flags [.], ack 36218, win 4050, options [nop,nop,TS val 601362012 ecr = 2553677188], length 0 23:37:36.910796 IP $my_vultr_host.vultr.com.6697 > $my_home_host.51256: = Flags [FP.], seq 36218:37666, ack 109, win 1040, options [nop,nop,TS val = 2553677297 ecr 601362012], length 1448 23:37:36.922685 IP $my_home_host.51256 > $my_vultr_host.vultr.com.6697: = Flags [F.], seq 109, ack 37667, win 4096, options [nop,nop,TS val = 601362025 ecr 2553677297], length 0 23:37:36.922742 IP $my_vultr_host.vultr.com.6697 > $my_home_host.51256: = Flags [.], ack 110, win 1040, options [nop,nop,TS val 2553677309 ecr = 601362025], length 0 My ipfw log doesn=E2=80=99t show any rejected packages in this case. For = comparison when I run service on the main host (without NAT and port = redirection) sending transfer is longer and ending sequence looks like = follows: % sudo tcpdump port 6696 tcpdump: verbose output suppressed, use -v or -vv for full protocol = decode listening on vtnet0, link-type EN10MB (Ethernet), capture size 65535 = bytes 23:44:54.151840 IP $my_home_host.51368 > $my_vultr_host.vultr.com.6696: = Flags [S], seq 3928283896, win 65535, options [mss 1460,nop,wscale = 5,nop,nop,TS val 601797509 ecr 0,sackOK,eol], length 0 23:44:54.151879 IP $my_vultr_host.vultr.com.6696 > $my_home_host.51368: = Flags [S.], seq 3837375255, ack 3928283897, win 65535, options [mss = 1460,nop,wscale 6,sackOK,TS val 1729228577 ecr 601797509], length 0 23:44:54.167015 IP $my_home_host.51368 > $my_vultr_host.vultr.com.6696: = Flags [.], ack 1, win 4117, options [nop,nop,TS val 601797522 ecr = 1729228577], length 0 23:44:54.167052 IP $my_home_host.51368 > $my_vultr_host.vultr.com.6696: = Flags [P.], seq 1:109, ack 1, win 4117, options [nop,nop,TS val = 601797522 ecr 1729228577], length 108 23:44:54.167508 IP $my_vultr_host.vultr.com.6696 > $my_home_host.51368: = Flags [P.], seq 1:18, ack 109, win 1040, options [nop,nop,TS val = 1729228593 ecr 601797522], length 17 23:44:54.167573 IP $my_vultr_host.vultr.com.6696 > $my_home_host.51368: = Flags [P.], seq 18:4479, ack 109, win 1040, options [nop,nop,TS val = 1729228593 ecr 601797522], length 4461 23:44:54.167585 IP $my_vultr_host.vultr.com.6696 > $my_home_host.51368: = Flags [P.], seq 4479:8575, ack 109, win 1040, options [nop,nop,TS val = 1729228593 ecr 601797522], length 4096 23:44:54.167599 IP $my_vultr_host.vultr.com.6696 > $my_home_host.51368: = Flags [P.], seq 8575:12671, ack 109, win 1040, options [nop,nop,TS val = 1729228593 ecr 601797522], length 4096 23:44:54.167611 IP $my_vultr_host.vultr.com.6696 > $my_home_host.51368: = Flags [.], seq 12671:14481, ack 109, win 1040, options [nop,nop,TS val = 1729228593 ecr 601797522], length 1810 23:44:54.178628 IP $my_home_host.51368 > $my_vultr_host.vultr.com.6696: = Flags [.], ack 14481, win 4039, options [nop,nop,TS val 601797540 ecr = 1729228593], length 0 (transfer goes normally) 23:44:54.257686 IP $my_vultr_host.vultr.com.6696 > $my_home_host.51368: = Flags [P.], seq 85433:96314, ack 109, win 1040, options [nop,nop,TS val = 1729228683 ecr 601797608], length 10881 23:44:54.271581 IP $my_home_host.51368 > $my_vultr_host.vultr.com.6696: = Flags [.], ack 86881, win 4096, options [nop,nop,TS val 601797622 ecr = 1729228683], length 0 23:44:54.271614 IP $my_vultr_host.vultr.com.6696 > $my_home_host.51368: = Flags [F.], seq 96314, ack 109, win 1040, options [nop,nop,TS val = 1729228697 ecr 601797622], length 0 23:44:54.276925 IP $my_home_host.51368 > $my_vultr_host.vultr.com.6696: = Flags [F.], seq 109, ack 96314, win 4096, options [nop,nop,TS val = 601797624 ecr 1729228683], length 0 23:44:54.276939 IP $my_vultr_host.vultr.com.6696 > $my_home_host.51368: = Flags [F.], seq 96314, ack 110, win 1040, options [nop,nop,TS val = 1729228703 ecr 601797624], length 0 23:44:54.286060 IP $my_home_host.51368 > $my_vultr_host.vultr.com.6696: = Flags [F.], seq 109, ack 96315, win 4096, options [nop,nop,TS val = 601797633 ecr 1729228697], length 0 23:44:54.286085 IP $my_vultr_host.vultr.com.6696 > $my_home_host.51368: = Flags [.], ack 110, win 1040, options [nop,nop,TS val 1729228712 ecr = 601797624], length 0 23:44:54.291528 IP $my_home_host.51368 > $my_vultr_host.vultr.com.6696: = Flags [.], ack 96315, win 4096, options [nop,nop,TS val 601797637 ecr = 1729228697], length 0 23:44:54.720336 IP $my_vultr_host.vultr.com.6696 > $my_home_host.51368: = Flags [.], ack 109, win 0, length 0 23:44:54.974955 IP $my_home_host.51368 > $my_vultr_host.vultr.com.6696: = Flags [.], ack 96315, win 4096, options [nop,nop,TS val 601798310 ecr = 1729228712], length 0 Of course file transfer in the latter case works just fine. Connections = initiated from the jail seem to work just fine (I am able to download = larger files, like distfiles to build ports).=20 I=E2=80=99m grateful for every help. I=E2=80=99m happy to fix my configs = or even change approach as long as I can achieve my goals. My configs = are included below. % cat /etc/rc.conf hostname=3D=E2=80=9Cfoobar" ifconfig_vtnet0=3D"dhcp" sshd_enable=3D"YES" static_routes=3Dlinklocal route_linklocal=3D"-net 169.254.0.0/16 -interface vtnet0" crypto_load=3DYES cryptodev_load=3DYES aesni_load=3DYES virtio_random_load=3DYES ifconfig_vtnet0_ipv6=3D"inet6 2001:aaaa:bbbb:cccc::64 prefixlen 64" rtsold_enable=3DYES ipv6_activate_all_interfaces=3DYES rtsold_flags=3D"-aF" # Firewall configuration firewall_enable=3D"Yes" firewall_script=3D"/etc/ipfw.rules" firewall_logging=3D"YES" firewall_nat_enable=3D"YES" firewall_nat_interface=3D"vtnet0" # Internal network configuration cloned_interfaces=3D"${cloned_interfaces} lo1" gateway_enable=3D"YES" # enables the gateway # (ezjail is not here as I run it with: service ezjail onestart command) % cat /etc/sysctl.conf net.inet.ip.fw.one_pass=3D0 # Allow multiple passes for packet, = necessary for fw+nat to work net.inet.ip.fw.verbose_limit=3D25 % cat /etc/ipfw.rules #!/bin/sh ipfw -q -f flush # Set rules command prefix cmd=3D"ipfw add" pif=3D"vtnet0" # interface name of NIC attached to Internet skip=3D"skipto 10000" mip=3D=E2=80=9C111.222.111.222" # public ipv4 address ipfw nat 100 config ip $mip log reset same_ports \ redirect_port tcp 127.0.66.1:6697 6697 \ redirect_port tcp 127.0.66.1:80 80=20 ipfw add 1 allow log ip from any to any via lo0 $cmd 0003 allow tcp from $my_home_ip/32 to me 22 in setup keep-state=20 $cmd 5 allow icmp6 from any to any via vtnet0 # NAT for incoming packets $cmd 100 nat 100 log ip from any to $mip in $cmd 101 check-state log $cmd 500 $skip log tcp from 127.0.0.0/8 to any out setup keep-state $cmd 501 allow log tcp from me to any out setup keep-state $cmd 502 $skip log udp from 127.0.0.0/8 to any out keep-state $cmd 503 allow log udp from me to any out keep-state $cmd 504 $skip log icmp from 127.0.0.0/8 to any out keep-state $cmd 505 allow log icmp from me to any out keep-state $cmd 506 allow log icmp6 from me to any out keep-state $cmd 610 $skip log tcp from any to me 6697 in setup keep-state $cmd 620 $skip log tcp from any to me 80 in setup keep-state $cmd 630 allow log tcp from any to me 6696 in setup keep-state # Deny rest $cmd 9998 deny log all from any to any in via $pif $cmd 9999 deny log all from any to any out via $pif $cmd 10000 nat 100 log ip4 from 127.0.0.0/8 to any out $cmd 10001 allow log ip from any to any $cmd 10002 deny log ip from any to any % cat /usr/local/etc/ezjail/znc export jail_znc_hostname=3D"znc" export jail_znc_ip=3D"lo1|127.0.66.1" # export = jail_znc_ip=3D"lo1|127.0.66.1,lo1|::6697,vtnet0|10.0.0.66,vtnet0|2001:aaaa= :bbbb:cccc::6697" #export = jail_znc_ip=3D"vtnet0|10.0.0.66/0xffffff00,vtnet0|10.0.0.66,vtnet0|2001:aa= aa:bbbb:cccc::6697" export jail_znc_rootdir=3D"/usr/jails/znc" export jail_znc_exec_start=3D"/bin/sh /etc/rc" export jail_znc_exec_stop=3D"" export jail_znc_mount_enable=3D"YES" export jail_znc_devfs_enable=3D"YES" export jail_znc_devfs_ruleset=3D"devfsrules_jail" export jail_znc_procfs_enable=3D"YES" export jail_znc_fdescfs_enable=3D"YES" export jail_znc_image=3D"" export jail_znc_imagetype=3D"" export jail_znc_attachparams=3D"" export jail_znc_attachblocking=3D"" export jail_znc_forceblocking=3D"" export jail_znc_zfs_datasets=3D"" export jail_znc_cpuset=3D"" export jail_znc_fib=3D"" export jail_znc_parentzfs=3D"" export jail_znc_parameters=3D"" export jail_znc_post_start_script=3D"" export jail_znc_retention_policy=3D"" ### Only for debugging export jail_znc_parameters=3D"allow.raw_sockets=3D1=E2=80=9D If you need any more configs or tests, please tell me. Thanks for your = help! --=20 BRs, Wojciech Wojtyniak
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?A88A7FED-B5DD-4B1E-96A4-AE1F3EAB8E30>