From owner-freebsd-pf@FreeBSD.ORG Fri Sep 7 18:15:56 2012 Return-Path: Delivered-To: pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6F6F3106564A; Fri, 7 Sep 2012 18:15:56 +0000 (UTC) (envelope-from ermal.luci@gmail.com) Received: from mail-ee0-f54.google.com (mail-ee0-f54.google.com [74.125.83.54]) by mx1.freebsd.org (Postfix) with ESMTP id C03768FC0A; Fri, 7 Sep 2012 18:15:55 +0000 (UTC) Received: by eeke52 with SMTP id e52so1514548eek.13 for ; Fri, 07 Sep 2012 11:15:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=FAKiOZce5SrraQfG34RrYnhVHo5WRGjfLg0ZcFGCpcQ=; b=tsLbQoFyFQicWyB2N1YhjhUdPvsZDseN7Wp3sThGvdB2aKoBrP5J5oAhMbEdB8+LSX cCCJtBgWuUoM9C8egdqYAzJsOtQxhoiKxIYq21Qe9s/g5FT1cEKsy31BXMvb5cQSxM5r ZxLhi2B6Cd3gO9rTUXvrZwG/rZG6rl1MGFaIwSAl2Vi0s14cOFfPPxosDDb5jGWERjUW bGY7TiVwlTMIhoZRUPLbB/DLUeHjmRLoAHKZu3AkKA06g63oW9+DzbI6BSsoxcryvtkn V30bkfMXcmbFPv+o8ZmyYhHIGNnNgjSwgt9a00b7Pb2syDxyFuzmAY2MvLmKN4kNBOez ZPTg== MIME-Version: 1.0 Received: by 10.205.139.6 with SMTP id iu6mr3086132bkc.20.1347041754430; Fri, 07 Sep 2012 11:15:54 -0700 (PDT) Sender: ermal.luci@gmail.com Received: by 10.204.48.194 with HTTP; Fri, 7 Sep 2012 11:15:54 -0700 (PDT) In-Reply-To: References: <20120905115140.GF15915@FreeBSD.org> <50476187.8000303@gibfest.dk> <20120905183607.GI15915@glebius.int.ru> <20120906064640.GL15915@glebius.int.ru> Date: Fri, 7 Sep 2012 20:15:54 +0200 X-Google-Sender-Auth: Wx3FYmsIHLWp0RLK8tmNO4UMg7k Message-ID: From: =?ISO-8859-1?Q?Ermal_Lu=E7i?= To: Ian FREISLICH Content-Type: text/plain; charset=ISO-8859-1 Cc: pf@freebsd.org Subject: Re: [HEADS UP] merging projects/pf into head X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Sep 2012 18:15:56 -0000 On Fri, Sep 7, 2012 at 2:05 PM, Ian FREISLICH wrote: > =?ISO-8859-1?Q?Ermal_Lu=E7i?= wrote: >> > - the "pf: state key linking mismatch" which affects pf as far back >> > as we've been prepared to test (FreeBSD-8.0). Although it only >> > became visible in the logs in -CURRENT before 9-RELEASE with the >> > pf import then. It manifests as connections stalling randomly. >> > >> This has been an issue since new pf(4) import. > > My contention is that this issue is also present in earlier pf. > It's just not logged verbosely: > > [firewall1.jnb1] ~ # uname -a > FreeBSD firewall1.jnb1.gp-online.net 8.1-RELEASE FreeBSD 8.1-RELEASE #23: Tue Aug 7 20:21:54 SAST 2012 ianf@firewall1.jnb1.gp-online.net:/usr/obj/usr/src/sys/FIREWALL amd64 > [firewall1.jnb1] ~ # pfctl -s inf > Status: Enabled for 30 days 16:27:26 Debug: Urgent > > State Table Total Rate > current entries 377102 > searches 126189706387 47596.4/s > inserts 6358571792 2398.3/s > removals 6358194690 2398.2/s > Counters > match 23798723897 8976.4/s > bad-offset 0 0.0/s > fragment 29807 0.0/s > short 76362 0.0/s > normalize 234 0.0/s > memory 0 0.0/s > bad-timestamp 0 0.0/s > congestion 0 0.0/s > ip-option 78290 0.0/s > proto-cksum 11023818 4.2/s > state-mismatch 4799367 1.8/s > state-insert 75295 0.0/s > state-limit 22 0.0/s > src-limit 0 0.0/s > synproxy 0 0.0/s > > Every time the state-mismatch counter increments, the connection > stalls. This manifests as as web pages needing to be reloaded > sometimes in order to complete downloading, or ssh connections being > reset. While 4799367 is a small fraction of the total searches, > the chance of your flow being bitten is multiplied by each hop > through a FreeBSD router running pf. While composing this email, > the state-mismatch counter increased by 11589. > This is not enough information to debug anything. - Please post your ruleset - A dump of your state table at the time - Describe your environment to allow understanding - Any kind of routing related - Tcpdump would be helpful as well Normally this issue, should exist in Gleb repo even though you are not facing it loudly. Nothing has changed in Gleb's repo related to this behaviour apart not having the linked state functionality(right?), which as you say does not seem the source of this since happens even before 9.0 anyway. I have not seen this reported in pfSense side of things either. If you can try a quick test with pfSense, either just copying the kernel and pfctl binary, and see if you have same behavior would be helpful. > We don't see this issue at all with Gleb's patches applied and > forwarding performance is greatly improved. > That's a good thing in general and is good to have improvements just i am a bit sceptic about its changes in some areas. > Whatever happens I'd like a way forward to be found because pf > deployed at the scale we're using it is unuseable post 2011-06-28 > (and not ideal before). > >> > There's not been a fix since it was first reported. We're seeing >> > 0.08% of our connections dropped on the floor or about 4 per second. >> > As a result, we've been seriously considering replacing our FreeBSD >> > routers. >> >> I have missed the report of this, can you point to details? > > http://www.freebsd.org/cgi/query-pr.cgi?pr=163208 > > Comes to mind. I'm sure there were some earlier reports, but I > can't find them in a hurry. I'm also pretty sure there have been > reports on current@. > > I posted to current@ > http://www.freebsd.org/cgi/getmsg.cgi?fetch=164206+169604+/usr/local/www/db/text/2012/freebsd-current/20120812.freebsd-current > > Which is how I came to this list on mail from Gleb. > > I can tell you that this is not peculiar to 9 and later. pf pre-9 > was just silent about dropping the flows although the problem occurs > less frequently. > > Ian > > -- > Ian Freislich -- Ermal