Date: Mon, 03 Feb 2014 10:57:03 +0100 From: =?ISO-8859-1?Q?Jean-S=E9bastien_P=E9dron?= <dumbbell@FreeBSD.org> To: freebsd-net@FreeBSD.org Cc: Christopher Faulet <christopher.faulet@capflam.org> Subject: Loosing TCP/IPv4 connections with jails+pf on 10.0-RELEASE Message-ID: <52EF67EF.1000803@FreeBSD.org>
next in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--wMeVt8JFm3k9gCEdwxul8KBTVPOt7GbIe
Content-Type: multipart/mixed;
boundary="------------050604020808060504070202"
This is a multi-part message in MIME format.
--------------050604020808060504070202
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Hello!
We have one server with multiple jails, each jail runs a service (mail,
web, etc.). sysutils/ezjail is used to setup and start the jails.
Beside the public IP address, IPv4 and IPv6 aliases are added to the
main NIC (em0); one per jail. The server has a second NIC (em1) which is
unused.
As we only have one public IPv4 address, pf is used to
o redirect connections to jails
o NAT connections from jails
With 8.3-RELEASE on another server, this setup was working without
problem. Now that we switched to a new server and 10.0-RELEASE (we
skipped 9.x), we see that TCP connections to jails over IPv4 are having
troubles:
o After around 10 days of uptime, connections from an IRC client
on the host (not a jail) connected to an IRC server on a jail
are getting dropped during the night (maybe because of no
activity on the IRC channel). It seems that packets from the
host (or a remote computer) to the jail are fine. However,
packets from the jail never reach the peer. This was tested with
nc(1) on both sides, so the uptime of the IRC client or server
isn't related.
o As the time passes, connections are dropped faster and faster:
even during the day, when there's activity on the IRC channel.
o At some point, connections only live for a few seconds and this
affects short-lived connections to the SMTP/IMAP and web jails.
A reboot solves the problem, until it comes back a week or more later.
Troubles start to appear again since this week-end.
IPv6 connections are NOT affected: they work perfectly.
This is stock FreeBSD 10.0-RELEASE amd64 with GENERIC kernel.
You'll find attached the output of ifconfig(8), our pf rules and one
jail configuration in ezjail (other jails have a similar setup). Note
that the pf rules we used on FreeBSD 8.3 are commented out at the end of
pf.conf; we simplified them by using ports lists.
Do you see something wrong with this setup?
PS: I'm not subscribed to the list, please CC me.
--=20
Jean-S=E9bastien P=E9dron
--------------050604020808060504070202
Content-Type: text/plain; charset=UTF-8;
name="ifconfig.txt"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: attachment;
filename="ifconfig.txt"
em0: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 15=
00
options=3D4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM=
,TSO4,WOL_MAGIC,VLAN_HWTSO>
ether 00:25:90:21:04:2c
inet $PUBLIC_IP netmask 0xffffff00 broadcast $PUBLIC_BROADCAST=20
inet6 fe80::225:90ff:fe21:42c%em0 prefixlen 64 scopeid 0x1=20
inet6 $PUBLIC_IPV6::1 prefixlen 56=20
inet 10.0.0.1 netmask 0xffffffff broadcast 10.0.0.1=20
inet 10.0.0.3 netmask 0xffffffff broadcast 10.0.0.3=20
inet6 $PUBLIC_IPV6::1:3 prefixlen 64=20
inet 10.0.0.4 netmask 0xffffffff broadcast 10.0.0.4=20
inet6 $PUBLIC_IPV6::1:4 prefixlen 64=20
inet 10.0.0.2 netmask 0xffffffff broadcast 10.0.0.2=20
inet6 $PUBLIC_IPV6::1:2 prefixlen 64=20
nd6 options=3D21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
em1: flags=3D8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=3D4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM=
,TSO4,WOL_MAGIC,VLAN_HWTSO>
ether 00:25:90:21:04:2d
nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: no carrier
lo0: flags=3D8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=3D600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128=20
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3=20
inet 127.0.0.1 netmask 0xff000000=20
nd6 options=3D21<PERFORMNUD,AUTO_LINKLOCAL>
--------------050604020808060504070202
Content-Type: text/plain; charset=UTF-8;
name="pf.conf"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: attachment;
filename="pf.conf"
# Interface declarations
ext_if=3D"em0"
int_if=3D"lo0"
all_if=3D"{em0, lo0}"
# Internal network subnet
jail_net=3D"10.0.0.0/24"
# Name and IP of our webserver
MYSQL=3D"10.0.0.1"
HTTPD=3D"10.0.0.2"
VEXIM=3D"10.0.0.3"
IRCD=3D"10.0.0.4"
PUBLIC_IP=3D"..."
#scrub in all
nat pass on $ext_if inet from $jail_net to any -> $PUBLIC_IP
rdr pass on $all_if inet proto tcp from any to $ext_if port {6667,6668,70=
00} -> $IRCD
rdr pass on $all_if inet proto tcp from any to $ext_if port {80,443,8140}=
-> $HTTPD
rdr pass on $all_if inet proto tcp from any to $ext_if port {25,143,465,9=
93,995} -> $VEXIM
rdr pass on $int_if inet proto tcp from any to $int_if port 25 -> $VEXIM
### OLD RULES (FreeBSD 8.3) ###
#rdr on $all_if inet proto tcp from any to $ext_if port 80 -> $HTTPD port=
80
#rdr on $all_if inet proto tcp from any to $ext_if port 443 -> $HTTPD por=
t 443
#rdr on $all_if inet proto tcp from any to $ext_if port 8140 -> $HTTPD po=
rt 8140
#rdr on $all_if inet proto tcp from any to $ext_if port 995 -> $VEXIM por=
t 995
#rdr on $all_if inet proto tcp from any to $ext_if port 993 -> $VEXIM por=
t 993
#rdr on $all_if inet proto tcp from any to $ext_if port 143 -> $VEXIM por=
t 143
#rdr on $all_if inet proto tcp from any to $ext_if port 25 -> $VEXIM port=
25
#rdr on $all_if inet proto tcp from any to $ext_if port 465 -> $VEXIM por=
t 465
#rdr on $all_if inet proto tcp from any to $int_if port 25 -> $VEXIM port=
25
#rdr on $all_if inet proto tcp from any to $ext_if port 7000 -> $IRCD por=
t 7000
#rdr on $all_if inet proto tcp from any to $ext_if port 6667 -> $IRCD por=
t 6667
#rdr on $all_if inet proto tcp from any to $ext_if port 6668 -> $IRCD por=
t 6668
#nat on $ext_if inet from $MYSQL to any -> $PUBLIC_IP
#nat on $ext_if inet from $HTTPD to any -> $PUBLIC_IP
#nat on $ext_if inet from $VEXIM to any -> $PUBLIC_IP
#nat on $ext_if inet from $IRCD to any -> $PUBLIC_IP
--------------050604020808060504070202
Content-Type: text/plain; charset=UTF-8;
name="ezjail.conf"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: attachment;
filename="ezjail.conf"
export jail_ircd_hostname=3D"ircd"
export jail_ircd_ip=3D"em0|10.0.0.4,em0|$PUBLIC_IPV6::1:4"
=2E..
export jail_ircd_parameters=3D"allow.raw_sockets=3D1"
--------------050604020808060504070202--
--wMeVt8JFm3k9gCEdwxul8KBTVPOt7GbIe
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (FreeBSD)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEARECAAYFAlLvaA8ACgkQa+xGJsFYOlMmAQCZARoq/RVaaJz7owyaUap6rf89
Zb0Anjuo1uSG9dJ8RSny+gC9J1DFYwQ2
=aAk+
-----END PGP SIGNATURE-----
--wMeVt8JFm3k9gCEdwxul8KBTVPOt7GbIe--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?52EF67EF.1000803>