From owner-freebsd-pf@FreeBSD.ORG Thu Jul 26 09:16:18 2007 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7A63C16A419 for ; Thu, 26 Jul 2007 09:16:18 +0000 (UTC) (envelope-from gergely.czuczy@harmless.hu) Received: from marvin.harmless.hu (marvin.harmless.hu [195.56.55.204]) by mx1.freebsd.org (Postfix) with ESMTP id CF70913C428 for ; Thu, 26 Jul 2007 09:16:17 +0000 (UTC) (envelope-from gergely.czuczy@harmless.hu) Received: from localhost (marvin-mail [192.168.0.2]) by marvin.harmless.hu (Postfix) with ESMTP id 42AFD7C0ACF; Thu, 26 Jul 2007 11:16:16 +0200 (CEST) X-Virus-Scanned: by amavisd-new-2.4.2 (20060627) (Debian) at harmless.hu Received: from marvin.harmless.hu ([192.168.0.2]) by localhost (marvin.harmless.hu [192.168.0.2]) (amavisd-new, port 10024) with ESMTP id dQz5-BRt8bbh; Thu, 26 Jul 2007 11:16:15 +0200 (CEST) Received: from marvin.harmless.hu (localhost [127.0.0.1]) by marvin.harmless.hu (Postfix) with ESMTP id 99B0A7C0AC9; Thu, 26 Jul 2007 11:16:00 +0200 (CEST) Date: Thu, 26 Jul 2007 11:16:00 +0200 From: Gergely CZUCZY To: freebsd-pf@freebsd.org, max@love2party.net Message-ID: <20070726091600.GA79956@harmless.hu> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=x-unknown; protocol="application/pgp-signature"; boundary="6TrnltStXW4iwmi0" Content-Disposition: inline User-Agent: mutt-ng/devel-r804 (FreeBSD) Cc: Subject: connect: not permitted by pf state lookup failures on heavier load X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Jul 2007 09:16:18 -0000 --6TrnltStXW4iwmi0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hello, Recently I've been playing around with a carp+pfsync+pound applevel proxy. On a high connection rate I've noticed some failed connections and the applevel proxy rendered the backend web servers DEAD, that means unreachabl= e. Pound sets on the gateway, accepts connections from the outside world and m= akes connections to the backend servers. The state table grew up to 32K states in total. On a very hight rate when pound tried to reach a backend server with connect(2) it recieved an "operation permitted" response, that was quite strange. Sometimes there are "borken pipes", sometimes also around those state lookup failures. On farther digging i've set pf's loglevel to misc and I've noticed state table lookup failures before pound's connect(2) error messages. It looks like this: Jul 26 10:46:54 lvs1 kernel: pf: BAD state: TCP 192.168.4.55:80 192.168.4.5= 5:80 192.168.4.251:42688 [lo=3D3773866253 high=3D3773932711 win=3D2003 modu= lator=3D155307840 wscale=3D5] [lo=3D9137549 high=3D9201645 win=3D33304 modu= lator=3D2788154389 wscale=3D1] 9:9 S seq=3D3822349776 ack=3D9137549 len=3D0= ackskew=3D0 pkts=3D35:42 dir=3Din,fwd Jul 26 10:46:54 lvs1 kernel: pf: State failure on: 1 | 5 Also there are lots of operation timeouts and connection reset by peers. =2E When I disable pf there's a lot less of them. The pf.conf is the following: --- BEGIN pf.conf --- if_ext=3D"em0" if_vvv=3D"fxp0" if_sync=3D"em1" ip_pub=3D"192.168.4.55" ip_vvv=3D"10.0.0.254" ip_vvv1=3D"10.0.0.1" ip_vvv2=3D"10.0.0.2" ip_vvv3=3D"10.0.0.3" table {$ip_vvv1, $ip_vvv2, $ip_vvv3} # Options: tune the behavior of pf, default values are given. set timeout { interval 5, frag 30 } #set timeout { tcp.first 120, tcp.opening 30, tcp.established 86400 } set timeout { tcp.closing 900, tcp.finwait 30, tcp.closed 60 } #set timeout { udp.first 60, udp.single 30, udp.multiple 60 } #set timeout { icmp.first 20, icmp.error 10 } #set timeout { other.first 60, other.single 30, other.multiple 60 } set timeout { adaptive.start 30000, adaptive.end 90000 } set limit { states 100000, frags 2000 } #set loginterface none set block-policy return set require-order yes set fingerprints "/etc/pf.os" set debug misc set skip on lo0 #scrub in all rdr on $if_ext proto tcp from any to $ip_pub port 10001 -> $ip_vvv1 port 22 rdr on $if_ext proto tcp from any to $ip_pub port 10002 -> $ip_vvv2 port 22 rdr on $if_ext proto tcp from any to $ip_pub port 10003 -> $ip_vvv3 port 22 block in log on $if_ext all pass in quick on {$if_ext,$if_vvv} proto vrrp pass out quick on {$if_ext,$if_vvv} proto vrrp pass out quick on $if_ext proto udp from any to 192.168.4.200 port 123 keep= state pass in quick on $if_ext proto tcp from any to $if_ext:0 port 22 flags S/SA= synproxy state (no-sync) pass in quick on $if_ext proto tcp from any to $ip_pub port 80 flags S/SA m= odulate state (no-sync) pass out quick on $if_ext proto udp from $if_ext:0 to port 53 keep state (n= o-sync) pass out quick on $if_ext proto udp from any to port 53 keep state pass out quick on $if_ext proto tcp from $if_ext:0 to port 80 flags S/SA ke= ep state (no-sync) pass out quick on $if_ext proto tcp from any to port 80 flags S/SA keep sta= te pass in quick on $if_ext proto tcp from any to port 22 flags S/SA syn= proxy state #pass out quick on $if_vvv proto tcp from ($if_vvv) to port 80 flags = S/SA keep state (no-sync) pass out quick on $if_vvv proto tcp from ($if_vvv) to {$ip_vvv1,$ip_vvv2,$i= p_vvv3} port 80 flags S/SA keep state (no-sync) --- END pf.conf --- Here, i've player around with the tcp timeouts, scrubbing, adaptive settings and swapped the last two rules (table VS individual rules), but they lead to nowhere, nothing changed. I'm testing this proxy with around 10-15 ab's (apache benchmark, part of th= e port), with 8 or 16 connections per instance and 500 requests/instance in an infin= ite loop. Here's an hour's messages log. pf wasn't enabled for the whole hour, but ac= cordingly more then half an hour: http://phoemix.harmless.hu/messages-pffail.0.bz2 The question is, what can cause this high rate of connection failures? What have I done wrong? What's happening, I've never seen such a thing =66rom pf? How could it be repaired to make pf behave stable on a heavier load? Sincerely, Gergely Czuczy mailto: gergely.czuczy@harmless.hu --=20 Weenies test. Geniuses solve problems that arise. --6TrnltStXW4iwmi0 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (FreeBSD) owGlWM2rW8cVdxOyGSjFu0I3w4vBdizp3Q9JT1KjOI6f67iFxM0z9SKEMLqaKw26 unMzc69k2TUtFEopWZTSRaCLdlFKFy100T+gm0AXpV0Wuit0Uyi0C/8B/Z2ZK72r 9x61k/iBuXNm5nz8zpnzoZ98+eVLL13+y+//8P6Nj3/68y/97vLfJ68tq7LMZ+2l MCuVt8MgCNtHw2Gv347bSdI/SoKgf9QXQgbxN36T3r+t81LmZfvBppAjXspH5WGR CZV/nSdzYawsx1WZtgdse+5Y2UJbVSqdj7jKM5XL3d4DI3KbStO+kyd6qvLZiH9U 6VJO24VReSkmmWTsbZllusXYezLBnWzD711dST6RMucQvMEtLoyu8ilfq3LOBU+E KW4Uqd3kyY3CbYiiyORKZrww+tGmw97NcWyuZnOe6DyXCSnHjSil553rUiVyyq1e Sp4KleH79KDlAizLuWT7bLmR+VQa6fb4RCQLSTrJCbfSrKSx/PjOreMWdkXJlxKW 8yo3UgC2STZmssPYfactMLQcChGbGZRai02LiySRBehNPVKjl+6UrkqrppKvtcmm Tr3lmImFtKx5vNR7mtVaQewDUG1J5jvI+czINa8KuhBH3/JbFr4DoRRZhzv4cHfj MCw9cuu5zJmHuzSKUNDcWYez+yK9n2rNrkXXuQILmShAScqzA11IsCSf4GOpSgTE AU4gjnIrawDXwiJWVEmaI4pmssNP4K5SLaEqzDRwj5H8YKLNgiJFFdIetJxL/RmR Wb0NnHKu7RaCTOsFbCe3V5AJeGBuKgzx5FM1m1HAKQoT+IkX6VWLKzMfBzB5qWzi PLAfScSaeXTPCEAkpxqaOuiu2iYu0hhtECrWipnT5F7pbkOiWsBZc2VH7JtVxqM+ D4NRtz/qdXm2siFfSJPLbAT1RvytW8de/og/uH2fh8OoE/YHnW6n1xsNguZ6zM5Q ol446kb9wYC/n+lxfBwfHcWDfj/qxc7zNWUYR0dhCK/mIERBEPOlnlZjlolSG5DC Xi8OjgbdgK9tIjIJUu+DmuMwjI963eGW3TAKwn63V/OK4zjonmEWHQ0GYa8bD4an 3MIP+HA05CfwyEd0bRBFcXd4dNTHq1k0hGSSuAZjRnS7kGta8WJRWrrVg6lwMAlR eStdT58P7YmLmNqTnBJcyP2/7/IeY7coxFwscorFTNO7TvlpeFMk0tt1AdNMRZJi a7JB/LsXOo7u8Id4X/weFLQuiorUc0bECOIM46zjDuqyftNF2gHTlCv3IniqkUjX lGVZu93mb925e++d3RlQmEo/RDqH/QdyGRzQcrVa0TJ9VPg15VS/Hx4wpooPi2pC 62ZQHRC9vhgGHfqLel1/HOSwQQ+3Z6Pm4S0xbhBj3PfP53XsvMGfXKm5tXj9Fe2+ 4qeMvcrfLVzWQ4mqcukTn5yLlcKLAkpF2uJTmYoqK/lKZBUlBLhoplYy7zBCv/YN f4LEVyJviYz3Wki5YsbjgD9lr+4fKpOikypjSx5GQcst4eeckkVcr6UlC5SdIyEM +t2AuJxnkmSolrg1DOprqcrXAilyy4YOgEP/AiWq6VaJPg7TilgBtLheLmGvQtG6 8LZKltvrZIJb+hQUXnBaU/g1pPl1Q54n7EvcZyGmAk5ayQ6AMWRgQDbvqFQxhkSr L2YKtQDX6noUBv48ucRC42CnJPKx81kqEkrBaDjcc8p0smgXOlMJ1eqyMrmjG4k6 YmRbG1RvvkHRJCpQn0nj2hDLDw5lmRziqWh74HanclLNXK5nbm0XqqCSnekAsWcT U02oXoosY8xMDW1d8a+LegUqxEnhy7fIN1Q2rvi3hCoAIMiykLff2AZ06MlR9Dl5 RQ1e0RfkFTd4xTtezIFLJgP7JleHQCGsax+AMw5h90m93bric8zTWvrKmMKfpgB5 seMXnN83CXG/Z1KjvCFkvGFRjLwuC1QGX6nPq/w8mNzuKNgiwtOMovLk8OQWMd3k vkX0Lca1XLcpk17/7GIa3kClPhVCDR/qZCZKeU7IiyN0agaoTkjPA7PjCSlN1T8D 7OfZPV+xnf3nFdu3fiHRQBcXw/si/Js6nuXs+Y7Z5wgLX6vOhwQFxJg1QgJJ42Jd weAs72s1/foZCTu9x2xP8xdA5HlSTqvtrtheUTDAV9v/B9uecOo67rxzvNdzYLwz 6OldQ03jHFJwc5qjuk06bTsldPCUXCfIzqeVgjrxEhTLqI+ya8xl9RSWCVSnco1J pEJ/xK/5FuI7J/DhVK3UtEJVd1vXW3wCVHBng05K0PTCcr2eO+XQxc+pJGNSQ1GY UiN+FYMX6jlRqQ2vJ0A/gHr1w6Ad9riYoEG7JgrMQdSA5MkcY/aixQsqeK5bQ+A6 AK+3mLs+4FRw+3uDHhpGaAw4cxQ0srGHzEV1CyrYw90OVZwc/6N2jZnys0zR8RBT n5jzua4MvrYjBWXrDrWSmKbyqyWXOeEzRadoHHzruQZcdMnDI5IxJkpDk3q2YUsa W0pqS+ciS7fsR2xelsXo8LCYa7lUjzpzYZbUnHbm1eFWcLtIqWtGYzd5HPlm1RlD za+Ck9c04iXgmIjK+knHz+tu1ARujWZ5O0jdZA/pFno8SX0yyj5fG53PbnKiw+o5 xUXuIscPZ5KGUUs/JNjKzanOzWzc79MLKNKb7G29hqgKQzU6j4kE5oVQxg+3SwzY hJ1rK13WpdjS7qcFiUYTvDMtpjcZO1HwjpHZpsXYXWlm+OK3H1fJY2AI5Us94jNP 7iSO/GYDM4ZnM44C9hB6KpptAVOH38UCyFgMtBm9HKMhfGn9bCyMsrLDfnTz5Vcu 0U822597Lr/06JNLv3jzKx//4MfPPv327a9+7Yd//PWv/vbXf/7300u//P4rn/z5 t68/+9nT/zy5/+9/vPfsT73v/et/ =I3PG -----END PGP SIGNATURE----- --6TrnltStXW4iwmi0--