From owner-freebsd-net@FreeBSD.ORG Mon Feb 3 09:57:39 2014 Return-Path: Delivered-To: freebsd-net@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BE7183E2 for ; Mon, 3 Feb 2014 09:57:39 +0000 (UTC) Received: from mail.made4.biz (mail.made4.biz [IPv6:2001:41d0:2:c018::1:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 79D6613B1 for ; Mon, 3 Feb 2014 09:57:39 +0000 (UTC) Received: from [2001:1b48:10b:cafe:225:64ff:febe:589f] (helo=viking.yzserv.com) by mail.made4.biz with esmtpsa (TLSv1:DHE-RSA-CAMELLIA256-SHA:256) (Exim 4.82 (FreeBSD)) (envelope-from ) id 1WAGHX-000GMa-Rh; Mon, 03 Feb 2014 10:57:37 +0100 Message-ID: <52EF67EF.1000803@FreeBSD.org> Date: Mon, 03 Feb 2014 10:57:03 +0100 From: =?ISO-8859-1?Q?Jean-S=E9bastien_P=E9dron?= User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 MIME-Version: 1.0 To: freebsd-net@FreeBSD.org Subject: Loosing TCP/IPv4 connections with jails+pf on 10.0-RELEASE X-Enigmail-Version: 1.6 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="wMeVt8JFm3k9gCEdwxul8KBTVPOt7GbIe" Cc: Christopher Faulet X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Feb 2014 09:57:39 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --wMeVt8JFm3k9gCEdwxul8KBTVPOt7GbIe Content-Type: multipart/mixed; boundary="------------050604020808060504070202" This is a multi-part message in MIME format. --------------050604020808060504070202 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hello! We have one server with multiple jails, each jail runs a service (mail, web, etc.). sysutils/ezjail is used to setup and start the jails. Beside the public IP address, IPv4 and IPv6 aliases are added to the main NIC (em0); one per jail. The server has a second NIC (em1) which is unused. As we only have one public IPv4 address, pf is used to o redirect connections to jails o NAT connections from jails With 8.3-RELEASE on another server, this setup was working without problem. Now that we switched to a new server and 10.0-RELEASE (we skipped 9.x), we see that TCP connections to jails over IPv4 are having troubles: o After around 10 days of uptime, connections from an IRC client on the host (not a jail) connected to an IRC server on a jail are getting dropped during the night (maybe because of no activity on the IRC channel). It seems that packets from the host (or a remote computer) to the jail are fine. However, packets from the jail never reach the peer. This was tested with nc(1) on both sides, so the uptime of the IRC client or server isn't related. o As the time passes, connections are dropped faster and faster: even during the day, when there's activity on the IRC channel. o At some point, connections only live for a few seconds and this affects short-lived connections to the SMTP/IMAP and web jails. A reboot solves the problem, until it comes back a week or more later. Troubles start to appear again since this week-end. IPv6 connections are NOT affected: they work perfectly. This is stock FreeBSD 10.0-RELEASE amd64 with GENERIC kernel. You'll find attached the output of ifconfig(8), our pf rules and one jail configuration in ezjail (other jails have a similar setup). Note that the pf rules we used on FreeBSD 8.3 are commented out at the end of pf.conf; we simplified them by using ports lists. Do you see something wrong with this setup? PS: I'm not subscribed to the list, please CC me. --=20 Jean-S=E9bastien P=E9dron --------------050604020808060504070202 Content-Type: text/plain; charset=UTF-8; name="ifconfig.txt" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="ifconfig.txt" em0: flags=3D8843 metric 0 mtu 15= 00 options=3D4219b ether 00:25:90:21:04:2c inet $PUBLIC_IP netmask 0xffffff00 broadcast $PUBLIC_BROADCAST=20 inet6 fe80::225:90ff:fe21:42c%em0 prefixlen 64 scopeid 0x1=20 inet6 $PUBLIC_IPV6::1 prefixlen 56=20 inet 10.0.0.1 netmask 0xffffffff broadcast 10.0.0.1=20 inet 10.0.0.3 netmask 0xffffffff broadcast 10.0.0.3=20 inet6 $PUBLIC_IPV6::1:3 prefixlen 64=20 inet 10.0.0.4 netmask 0xffffffff broadcast 10.0.0.4=20 inet6 $PUBLIC_IPV6::1:4 prefixlen 64=20 inet 10.0.0.2 netmask 0xffffffff broadcast 10.0.0.2=20 inet6 $PUBLIC_IPV6::1:2 prefixlen 64=20 nd6 options=3D21 media: Ethernet autoselect (1000baseT ) status: active em1: flags=3D8c02 metric 0 mtu 1500 options=3D4219b ether 00:25:90:21:04:2d nd6 options=3D29 media: Ethernet autoselect status: no carrier lo0: flags=3D8049 metric 0 mtu 16384 options=3D600003 inet6 ::1 prefixlen 128=20 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3=20 inet 127.0.0.1 netmask 0xff000000=20 nd6 options=3D21 --------------050604020808060504070202 Content-Type: text/plain; charset=UTF-8; name="pf.conf" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="pf.conf" # Interface declarations ext_if=3D"em0" int_if=3D"lo0" all_if=3D"{em0, lo0}" # Internal network subnet jail_net=3D"10.0.0.0/24" # Name and IP of our webserver MYSQL=3D"10.0.0.1" HTTPD=3D"10.0.0.2" VEXIM=3D"10.0.0.3" IRCD=3D"10.0.0.4" PUBLIC_IP=3D"..." #scrub in all nat pass on $ext_if inet from $jail_net to any -> $PUBLIC_IP rdr pass on $all_if inet proto tcp from any to $ext_if port {6667,6668,70= 00} -> $IRCD rdr pass on $all_if inet proto tcp from any to $ext_if port {80,443,8140}= -> $HTTPD rdr pass on $all_if inet proto tcp from any to $ext_if port {25,143,465,9= 93,995} -> $VEXIM rdr pass on $int_if inet proto tcp from any to $int_if port 25 -> $VEXIM ### OLD RULES (FreeBSD 8.3) ### #rdr on $all_if inet proto tcp from any to $ext_if port 80 -> $HTTPD port= 80 #rdr on $all_if inet proto tcp from any to $ext_if port 443 -> $HTTPD por= t 443 #rdr on $all_if inet proto tcp from any to $ext_if port 8140 -> $HTTPD po= rt 8140 #rdr on $all_if inet proto tcp from any to $ext_if port 995 -> $VEXIM por= t 995 #rdr on $all_if inet proto tcp from any to $ext_if port 993 -> $VEXIM por= t 993 #rdr on $all_if inet proto tcp from any to $ext_if port 143 -> $VEXIM por= t 143 #rdr on $all_if inet proto tcp from any to $ext_if port 25 -> $VEXIM port= 25 #rdr on $all_if inet proto tcp from any to $ext_if port 465 -> $VEXIM por= t 465 #rdr on $all_if inet proto tcp from any to $int_if port 25 -> $VEXIM port= 25 #rdr on $all_if inet proto tcp from any to $ext_if port 7000 -> $IRCD por= t 7000 #rdr on $all_if inet proto tcp from any to $ext_if port 6667 -> $IRCD por= t 6667 #rdr on $all_if inet proto tcp from any to $ext_if port 6668 -> $IRCD por= t 6668 #nat on $ext_if inet from $MYSQL to any -> $PUBLIC_IP #nat on $ext_if inet from $HTTPD to any -> $PUBLIC_IP #nat on $ext_if inet from $VEXIM to any -> $PUBLIC_IP #nat on $ext_if inet from $IRCD to any -> $PUBLIC_IP --------------050604020808060504070202 Content-Type: text/plain; charset=UTF-8; name="ezjail.conf" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="ezjail.conf" export jail_ircd_hostname=3D"ircd" export jail_ircd_ip=3D"em0|10.0.0.4,em0|$PUBLIC_IPV6::1:4" =2E.. export jail_ircd_parameters=3D"allow.raw_sockets=3D1" --------------050604020808060504070202-- --wMeVt8JFm3k9gCEdwxul8KBTVPOt7GbIe Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (FreeBSD) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlLvaA8ACgkQa+xGJsFYOlMmAQCZARoq/RVaaJz7owyaUap6rf89 Zb0Anjuo1uSG9dJ8RSny+gC9J1DFYwQ2 =aAk+ -----END PGP SIGNATURE----- --wMeVt8JFm3k9gCEdwxul8KBTVPOt7GbIe--