Date: Thu, 26 Jul 2007 11:16:00 +0200 From: Gergely CZUCZY <phoemix@harmless.hu> To: freebsd-pf@freebsd.org, max@love2party.net Subject: connect: not permitted by pf state lookup failures on heavier load Message-ID: <20070726091600.GA79956@harmless.hu>
next in thread | raw e-mail | index | archive | help
--6TrnltStXW4iwmi0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
Hello,
Recently I've been playing around with a carp+pfsync+pound applevel proxy.
On a high connection rate I've noticed some failed connections and the
applevel proxy rendered the backend web servers DEAD, that means unreachabl=
e.
Pound sets on the gateway, accepts connections from the outside world and m=
akes
connections to the backend servers.
The state table grew up to 32K states in total. On a very hight rate when
pound tried to reach a backend server with connect(2) it recieved an
"operation permitted" response, that was quite strange. Sometimes there
are "borken pipes", sometimes also around those state lookup failures.
On farther digging i've set pf's loglevel to misc and I've noticed state
table lookup failures before pound's connect(2) error messages.
It looks like this:
Jul 26 10:46:54 lvs1 kernel: pf: BAD state: TCP 192.168.4.55:80 192.168.4.5=
5:80 192.168.4.251:42688 [lo=3D3773866253 high=3D3773932711 win=3D2003 modu=
lator=3D155307840 wscale=3D5] [lo=3D9137549 high=3D9201645 win=3D33304 modu=
lator=3D2788154389 wscale=3D1] 9:9 S seq=3D3822349776 ack=3D9137549 len=3D0=
ackskew=3D0 pkts=3D35:42 dir=3Din,fwd
Jul 26 10:46:54 lvs1 kernel: pf: State failure on: 1 | 5
Also there are lots of operation timeouts and connection reset by peers.
=2E When I disable pf there's a lot less of them.
The pf.conf is the following:
--- BEGIN pf.conf ---
if_ext=3D"em0"
if_vvv=3D"fxp0"
if_sync=3D"em1"
ip_pub=3D"192.168.4.55"
ip_vvv=3D"10.0.0.254"
ip_vvv1=3D"10.0.0.1"
ip_vvv2=3D"10.0.0.2"
ip_vvv3=3D"10.0.0.3"
table <vvv> {$ip_vvv1, $ip_vvv2, $ip_vvv3}
# Options: tune the behavior of pf, default values are given.
set timeout { interval 5, frag 30 }
#set timeout { tcp.first 120, tcp.opening 30, tcp.established 86400 }
set timeout { tcp.closing 900, tcp.finwait 30, tcp.closed 60 }
#set timeout { udp.first 60, udp.single 30, udp.multiple 60 }
#set timeout { icmp.first 20, icmp.error 10 }
#set timeout { other.first 60, other.single 30, other.multiple 60 }
set timeout { adaptive.start 30000, adaptive.end 90000 }
set limit { states 100000, frags 2000 }
#set loginterface none
set block-policy return
set require-order yes
set fingerprints "/etc/pf.os"
set debug misc
set skip on lo0
#scrub in all
rdr on $if_ext proto tcp from any to $ip_pub port 10001 -> $ip_vvv1 port 22
rdr on $if_ext proto tcp from any to $ip_pub port 10002 -> $ip_vvv2 port 22
rdr on $if_ext proto tcp from any to $ip_pub port 10003 -> $ip_vvv3 port 22
block in log on $if_ext all
pass in quick on {$if_ext,$if_vvv} proto vrrp
pass out quick on {$if_ext,$if_vvv} proto vrrp
pass out quick on $if_ext proto udp from any to 192.168.4.200 port 123 keep=
state
pass in quick on $if_ext proto tcp from any to $if_ext:0 port 22 flags S/SA=
synproxy state (no-sync)
pass in quick on $if_ext proto tcp from any to $ip_pub port 80 flags S/SA m=
odulate state (no-sync)
pass out quick on $if_ext proto udp from $if_ext:0 to port 53 keep state (n=
o-sync)
pass out quick on $if_ext proto udp from any to port 53 keep state
pass out quick on $if_ext proto tcp from $if_ext:0 to port 80 flags S/SA ke=
ep state (no-sync)
pass out quick on $if_ext proto tcp from any to port 80 flags S/SA keep sta=
te
pass in quick on $if_ext proto tcp from any to <vvv> port 22 flags S/SA syn=
proxy state
#pass out quick on $if_vvv proto tcp from ($if_vvv) to <vvv> port 80 flags =
S/SA keep state (no-sync)
pass out quick on $if_vvv proto tcp from ($if_vvv) to {$ip_vvv1,$ip_vvv2,$i=
p_vvv3} port 80 flags S/SA keep state (no-sync)
--- END pf.conf ---
Here, i've player around with the tcp timeouts, scrubbing, adaptive settings
and swapped the last two rules (table VS individual rules), but they lead to
nowhere, nothing changed.
I'm testing this proxy with around 10-15 ab's (apache benchmark, part of th=
e port),
with 8 or 16 connections per instance and 500 requests/instance in an infin=
ite loop.
Here's an hour's messages log. pf wasn't enabled for the whole hour, but ac=
cordingly
more then half an hour:
http://phoemix.harmless.hu/messages-pffail.0.bz2
The question is, what can cause this high rate of connection failures?
What have I done wrong? What's happening, I've never seen such a thing
=66rom pf?
How could it be repaired to make pf behave stable on a heavier load?
Sincerely,
Gergely Czuczy
mailto: gergely.czuczy@harmless.hu
--=20
Weenies test. Geniuses solve problems that arise.
--6TrnltStXW4iwmi0
Content-Type: application/pgp-signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (FreeBSD)
owGlWM2rW8cVdxOyGSjFu0I3w4vBdizp3Q9JT1KjOI6f67iFxM0z9SKEMLqaKw26
unMzc69k2TUtFEopWZTSRaCLdlFKFy100T+gm0AXpV0Wuit0Uyi0C/8B/Z2ZK72r
9x61k/iBuXNm5nz8zpnzoZ98+eVLL13+y+//8P6Nj3/68y/97vLfJ68tq7LMZ+2l
MCuVt8MgCNtHw2Gv347bSdI/SoKgf9QXQgbxN36T3r+t81LmZfvBppAjXspH5WGR
CZV/nSdzYawsx1WZtgdse+5Y2UJbVSqdj7jKM5XL3d4DI3KbStO+kyd6qvLZiH9U
6VJO24VReSkmmWTsbZllusXYezLBnWzD711dST6RMucQvMEtLoyu8ilfq3LOBU+E
KW4Uqd3kyY3CbYiiyORKZrww+tGmw97NcWyuZnOe6DyXCSnHjSil553rUiVyyq1e
Sp4KleH79KDlAizLuWT7bLmR+VQa6fb4RCQLSTrJCbfSrKSx/PjOreMWdkXJlxKW
8yo3UgC2STZmssPYfactMLQcChGbGZRai02LiySRBehNPVKjl+6UrkqrppKvtcmm
Tr3lmImFtKx5vNR7mtVaQewDUG1J5jvI+czINa8KuhBH3/JbFr4DoRRZhzv4cHfj
MCw9cuu5zJmHuzSKUNDcWYez+yK9n2rNrkXXuQILmShAScqzA11IsCSf4GOpSgTE
AU4gjnIrawDXwiJWVEmaI4pmssNP4K5SLaEqzDRwj5H8YKLNgiJFFdIetJxL/RmR
Wb0NnHKu7RaCTOsFbCe3V5AJeGBuKgzx5FM1m1HAKQoT+IkX6VWLKzMfBzB5qWzi
PLAfScSaeXTPCEAkpxqaOuiu2iYu0hhtECrWipnT5F7pbkOiWsBZc2VH7JtVxqM+
D4NRtz/qdXm2siFfSJPLbAT1RvytW8de/og/uH2fh8OoE/YHnW6n1xsNguZ6zM5Q
ol446kb9wYC/n+lxfBwfHcWDfj/qxc7zNWUYR0dhCK/mIERBEPOlnlZjlolSG5DC
Xi8OjgbdgK9tIjIJUu+DmuMwjI963eGW3TAKwn63V/OK4zjonmEWHQ0GYa8bD4an
3MIP+HA05CfwyEd0bRBFcXd4dNTHq1k0hGSSuAZjRnS7kGta8WJRWrrVg6lwMAlR
eStdT58P7YmLmNqTnBJcyP2/7/IeY7coxFwscorFTNO7TvlpeFMk0tt1AdNMRZJi
a7JB/LsXOo7u8Id4X/weFLQuiorUc0bECOIM46zjDuqyftNF2gHTlCv3IniqkUjX
lGVZu93mb925e++d3RlQmEo/RDqH/QdyGRzQcrVa0TJ9VPg15VS/Hx4wpooPi2pC
62ZQHRC9vhgGHfqLel1/HOSwQQ+3Z6Pm4S0xbhBj3PfP53XsvMGfXKm5tXj9Fe2+
4qeMvcrfLVzWQ4mqcukTn5yLlcKLAkpF2uJTmYoqK/lKZBUlBLhoplYy7zBCv/YN
f4LEVyJviYz3Wki5YsbjgD9lr+4fKpOikypjSx5GQcst4eeckkVcr6UlC5SdIyEM
+t2AuJxnkmSolrg1DOprqcrXAilyy4YOgEP/AiWq6VaJPg7TilgBtLheLmGvQtG6
8LZKltvrZIJb+hQUXnBaU/g1pPl1Q54n7EvcZyGmAk5ayQ6AMWRgQDbvqFQxhkSr
L2YKtQDX6noUBv48ucRC42CnJPKx81kqEkrBaDjcc8p0smgXOlMJ1eqyMrmjG4k6
YmRbG1RvvkHRJCpQn0nj2hDLDw5lmRziqWh74HanclLNXK5nbm0XqqCSnekAsWcT
U02oXoosY8xMDW1d8a+LegUqxEnhy7fIN1Q2rvi3hCoAIMiykLff2AZ06MlR9Dl5
RQ1e0RfkFTd4xTtezIFLJgP7JleHQCGsax+AMw5h90m93bric8zTWvrKmMKfpgB5
seMXnN83CXG/Z1KjvCFkvGFRjLwuC1QGX6nPq/w8mNzuKNgiwtOMovLk8OQWMd3k
vkX0Lca1XLcpk17/7GIa3kClPhVCDR/qZCZKeU7IiyN0agaoTkjPA7PjCSlN1T8D
7OfZPV+xnf3nFdu3fiHRQBcXw/si/Js6nuXs+Y7Z5wgLX6vOhwQFxJg1QgJJ42Jd
weAs72s1/foZCTu9x2xP8xdA5HlSTqvtrtheUTDAV9v/B9uecOo67rxzvNdzYLwz
6OldQ03jHFJwc5qjuk06bTsldPCUXCfIzqeVgjrxEhTLqI+ya8xl9RSWCVSnco1J
pEJ/xK/5FuI7J/DhVK3UtEJVd1vXW3wCVHBng05K0PTCcr2eO+XQxc+pJGNSQ1GY
UiN+FYMX6jlRqQ2vJ0A/gHr1w6Ad9riYoEG7JgrMQdSA5MkcY/aixQsqeK5bQ+A6
AK+3mLs+4FRw+3uDHhpGaAw4cxQ0srGHzEV1CyrYw90OVZwc/6N2jZnys0zR8RBT
n5jzua4MvrYjBWXrDrWSmKbyqyWXOeEzRadoHHzruQZcdMnDI5IxJkpDk3q2YUsa
W0pqS+ciS7fsR2xelsXo8LCYa7lUjzpzYZbUnHbm1eFWcLtIqWtGYzd5HPlm1RlD
za+Ck9c04iXgmIjK+knHz+tu1ARujWZ5O0jdZA/pFno8SX0yyj5fG53PbnKiw+o5
xUXuIscPZ5KGUUs/JNjKzanOzWzc79MLKNKb7G29hqgKQzU6j4kE5oVQxg+3SwzY
hJ1rK13WpdjS7qcFiUYTvDMtpjcZO1HwjpHZpsXYXWlm+OK3H1fJY2AI5Us94jNP
7iSO/GYDM4ZnM44C9hB6KpptAVOH38UCyFgMtBm9HKMhfGn9bCyMsrLDfnTz5Vcu
0U822597Lr/06JNLv3jzKx//4MfPPv327a9+7Yd//PWv/vbXf/7300u//P4rn/z5
t68/+9nT/zy5/+9/vPfsT73v/et/
=I3PG
-----END PGP SIGNATURE-----
--6TrnltStXW4iwmi0--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20070726091600.GA79956>