From owner-freebsd-pf@FreeBSD.ORG Mon May 2 11:07:04 2011 Return-Path: Delivered-To: freebsd-pf@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7B6AE106566B for ; Mon, 2 May 2011 11:07:04 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 5FA048FC21 for ; Mon, 2 May 2011 11:07:04 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p42B74Ee064165 for ; Mon, 2 May 2011 11:07:04 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p42B73xU064163 for freebsd-pf@FreeBSD.org; Mon, 2 May 2011 11:07:03 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 2 May 2011 11:07:03 GMT Message-Id: <201105021107.p42B73xU064163@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-pf@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-pf@FreeBSD.org X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2011 11:07:04 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/155736 pf [pf] [altq] borrow from parent queue does not work wit o kern/153307 pf [pf] Bug with PF firewall o kern/148290 pf [pf] "sticky-address" option of Packet Filter (PF) blo o kern/148260 pf [pf] [patch] pf rdr incompatible with dummynet o kern/147789 pf [pf] Firewall PF no longer drops connections by sendin o kern/146832 pf [pf] "(self)" not always matching all local IPv6 addre o kern/143543 pf [pf] [panic] PF route-to causes kernel panic o bin/143504 pf [patch] outgoing states are not killed by authpf(8) o conf/142961 pf [pf] No way to adjust pidfile in pflogd o conf/142817 pf [patch] etc/rc.d/pf: silence pfctl o kern/141905 pf [pf] [panic] pf kernel panic on 7.2-RELEASE with empty o kern/140697 pf [pf] pf behaviour changes - must be documented o kern/137982 pf [pf] when pf can hit state limits, random IP failures o kern/136781 pf [pf] Packets appear to drop with pf scrub and if_bridg o kern/135948 pf [pf] [gre] pf not natting gre protocol o kern/135162 pf [pfsync] pfsync(4) not usable with GENERIC kernel o kern/134996 pf [pf] Anchor tables not included when pfctl(8) is run w o kern/133732 pf [pf] max-src-conn issue o kern/132769 pf [pf] [lor] 2 LOR's with pf task mtx / ifnet and rtent f kern/132176 pf [pf] pf stalls connection when using route-to [regress o conf/130381 pf [rc.d] [pf] [ip6] ipv6 not fully configured when pf st o kern/129861 pf [pf] [patch] Argument names reversed in pf_table.c:_co o kern/127920 pf [pf] ipv6 and synproxy don't play well together o conf/127814 pf [pf] The flush in pf_reload in /etc/rc.d/pf does not w o kern/127439 pf [pf] deadlock in pf f kern/127345 pf [pf] Problem with PF on FreeBSD7.0 [regression] o kern/127121 pf [pf] [patch] pf incorrect log priority o kern/127042 pf [pf] [patch] pf recursion panic if interface group is o kern/125467 pf [pf] pf keep state bug while handling sessions between s kern/124933 pf [pf] [ip6] pf does not support (drops) IPv6 fragmented o kern/124364 pf [pf] [panic] Kernel panic with pf + bridge o kern/122773 pf [pf] pf doesn't log uid or pid when configured to o kern/122014 pf [pf] [panic] FreeBSD 6.2 panic in pf o kern/120281 pf [pf] [request] lost returning packets to PF for a rdr o kern/120057 pf [pf] [patch] Allow proper settings of ALTQ_HFSC. The c o bin/118355 pf [pf] [patch] pfctl(8) help message options order false o kern/114567 pf [pf] [lor] pf_ioctl.c + if.c o kern/114095 pf [carp] carp+pf delay with high state limit s conf/110838 pf [pf] tagged parameter on nat not working on FreeBSD 5. o kern/103283 pf pfsync fails to sucessfully transfer some sessions o kern/103281 pf pfsync reports bulk update failures o kern/93825 pf [pf] pf reply-to doesn't work o sparc/93530 pf [pf] Incorrect checksums when using pf's route-to on s o kern/92949 pf [pf] PF + ALTQ problems with latency o bin/86635 pf [patch] pfctl(8): allow new page character (^L) in pf. o kern/82271 pf [pf] cbq scheduler cause bad latency 46 problems total. From owner-freebsd-pf@FreeBSD.ORG Mon May 2 14:58:33 2011 Return-Path: Delivered-To: freebsd-pf@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 162E21065672 for ; Mon, 2 May 2011 14:58:33 +0000 (UTC) (envelope-from zhushazang@yahoo.com.br) Received: from wmail.liec.ufscar.br (wmail.liec.ufscar.br [200.136.226.171]) by mx1.freebsd.org (Postfix) with ESMTP id C00928FC08 for ; Mon, 2 May 2011 14:58:32 +0000 (UTC) Received: from localhost (localhost.localdomain [127.0.0.1]) by wmail.liec.ufscar.br (Postfix) with ESMTP id B2719565D33 for ; Mon, 2 May 2011 11:41:25 -0300 (BRT) X-Virus-Scanned: amavisd-new at liec.ufscar.br Received: from wmail.liec.ufscar.br ([127.0.0.1]) by localhost (wmail.liec.ufscar.br [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ERaHCzRyt30N for ; Mon, 2 May 2011 11:41:25 -0300 (BRT) Received: from [200.136.226.178] (asgard.liec.ufscar.br [200.136.226.178]) by wmail.liec.ufscar.br (Postfix) with ESMTPSA id 06C91560E9B for ; Mon, 2 May 2011 11:41:25 -0300 (BRT) Message-ID: <4DBEC293.1010607@yahoo.com.br> Date: Mon, 02 May 2011 11:41:23 -0300 From: Zhu Sha Zang User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.17) Gecko/20110430 Lightning/1.0b3pre Thunderbird/3.1.10 ThunderBrowse/3.3.5 MIME-Version: 1.0 To: freebsd-pf@FreeBSD.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Subject: blocking facebook X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2011 14:58:33 -0000 I'm trying to block facebook access only using PF in FreeBSD 8.2. But putting the name or the ip returned with the command host www.facebook.com i can't deny any user to connect facebook. Some trick to do that? Thanks for now. From owner-freebsd-pf@FreeBSD.ORG Mon May 2 15:30:37 2011 Return-Path: Delivered-To: freebsd-pf@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 552EA1065676 for ; Mon, 2 May 2011 15:30:37 +0000 (UTC) (envelope-from db@bsdsystems.de) Received: from fop.bsdsystems.de (mx.bsdsystems.de [88.198.57.43]) by mx1.freebsd.org (Postfix) with ESMTP id C88B48FC0C for ; Mon, 2 May 2011 15:30:36 +0000 (UTC) Received: from wuerschtl.global.intra.guj.com (fwhide.guj.de [193.7.250.35]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by fop.bsdsystems.de (Postfix) with ESMTP id 10D81363E5; Mon, 2 May 2011 17:02:59 +0200 (CEST) Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: dennis berger In-Reply-To: <4DBEC293.1010607@yahoo.com.br> Date: Mon, 2 May 2011 17:02:58 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: References: <4DBEC293.1010607@yahoo.com.br> To: Zhu Sha Zang X-Mailer: Apple Mail (2.1084) Cc: freebsd-pf@FreeBSD.org Subject: Re: blocking facebook X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2011 15:30:37 -0000 I've blocked myspace and facebook using those networks. blockedremotehosts=3D" = {69.63.176.0/20,63.135.80.0/20,216.178.32.0/20,194.97.156.0/24} " best, -dennis Am 02.05.2011 um 16:41 schrieb Zhu Sha Zang: > I'm trying to block facebook access only using PF in FreeBSD 8.2. >=20 > But putting the name or the ip returned with the command host = www.facebook.com i can't deny any user to connect facebook. >=20 > Some trick to do that? >=20 > Thanks for now. > _______________________________________________ > freebsd-pf@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-pf > To unsubscribe, send any mail to "freebsd-pf-unsubscribe@freebsd.org" Dennis Berger= From owner-freebsd-pf@FreeBSD.ORG Mon May 2 15:40:13 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4B9DD106566C for ; Mon, 2 May 2011 15:40:13 +0000 (UTC) (envelope-from kevin.wilcox@gmail.com) Received: from mail-iy0-f182.google.com (mail-iy0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id 158448FC1B for ; Mon, 2 May 2011 15:40:12 +0000 (UTC) Received: by iyj12 with SMTP id 12so6686219iyj.13 for ; Mon, 02 May 2011 08:40:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=Ya6v2D3P4Y1BZyJcXDAcgyGC5px/AIEux9DbKnXP6qI=; b=s7joKj9w0kMRmgeocNiP8hTvXdfdu2PxMutcl9gE8MJC0d5qVHVxBv34xXKTJdnEpS ukTXq0hkV0+wwpaGEV8L/Duy2/9JgXJfUHvTxgYBC6WsXj4jJuwfdxeEjCoMT/ToPrZQ lZwRrFSbGCjznv54DioeursOBPySziQwwPk+Y= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=AusYMeDohkTnE5mGvsWrntGg4UiAXVVW/TkT63Rc7h1YyBww5bHToGWSp3SNeA4pzT oWjsyRG5DVADu0wqWeWB7DzwhN5QKHvKADhf8cTnHhJ0q0eABKIMHXffehKAeflELTmh UoA4G4QWinNgDV1sYVfF0x6MbYThh/UVllRag= MIME-Version: 1.0 Received: by 10.231.179.197 with SMTP id br5mr4821104ibb.146.1304348989176; Mon, 02 May 2011 08:09:49 -0700 (PDT) Received: by 10.231.36.195 with HTTP; Mon, 2 May 2011 08:09:49 -0700 (PDT) In-Reply-To: <4DBEC293.1010607@yahoo.com.br> References: <4DBEC293.1010607@yahoo.com.br> Date: Mon, 2 May 2011 11:09:49 -0400 Message-ID: From: Kevin Wilcox To: Zhu Sha Zang Content-Type: text/plain; charset=UTF-8 Cc: freebsd-pf@freebsd.org Subject: Re: blocking facebook X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2011 15:40:13 -0000 On Mon, May 2, 2011 at 10:41, Zhu Sha Zang wrote: > I'm trying to block facebook access only using PF in FreeBSD 8.2. > > But putting the name or the ip returned with the command host > www.facebook.com i can't deny any user to connect facebook. > > Some trick to do that? > > Thanks for now. Short version: you can't block via domain in pf. Long version: when pf starts, it reads its config file. If you have a domain name listed, and you can't reach your DNS server because DHCP or other networking scripts aren't loaded, it can't resolve the domain. If it CAN resolve the domain, it will use the IP address it received from a DNS lookup. For domains backed by a single IP, no problem. For domains that span multiple IPs, and multiple networks, that's a pretty big problem. Additionally, pf does NOT do deep packet inspection. It's extremely taxing and it's not what pf was designed to do. If you want to block facebook I would suggest a multi-faceted approach (though it's not foolproof, it just keeps MOST people from going there). 1) Control DNS. You can have lookups for *.facebook.com (and associated CDN addresses) go to whatever. 2) Control the browser - if you can blacklist *.facebook.com (and associated CDN addresses) you can limit a lot of it. 3) Force your users through squid or another web proxy. This is probably the best method as you can think block anything going to a facebook.com address, or with certain strings in the URL, by redirecting them to a page saying, "I'm sorry, that is not allowed on this network." This scales remarkably well on commodity hardware up to several thousand users assuming you aren't doing 10Gb. Good luck. kmw From owner-freebsd-pf@FreeBSD.ORG Mon May 2 15:48:50 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A6EFC106566C for ; Mon, 2 May 2011 15:48:50 +0000 (UTC) (envelope-from britneyfreek@googlemail.com) Received: from mail-ww0-f50.google.com (mail-ww0-f50.google.com [74.125.82.50]) by mx1.freebsd.org (Postfix) with ESMTP id 36B428FC0A for ; Mon, 2 May 2011 15:48:49 +0000 (UTC) Received: by wwc33 with SMTP id 33so5856191wwc.31 for ; Mon, 02 May 2011 08:48:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=gamma; h=domainkey-signature:date:from:to:cc:message-id:in-reply-to :references:subject:x-mailer:mime-version:content-type :content-transfer-encoding:content-disposition; bh=W9ipeCZ9gyo13LDGmYeBo2GcYx74puvI/lF4IxXTBCI=; b=PLxTrH4N2LiCGNssXCCyfPrBdjKpEc6jNlqZp2ZJk+o/s18/ysbj4srr2MDfH6Eu09 2yXaLNM2YSbFW3KYqAgZNscdJHLaqM6M5LSI9HYH3XGnoQ1DCmfk67mPq1lu9242N24I zhGnHQB5rLvdHfSx8Ksa+IeuX5tglC8ZnRaGw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=googlemail.com; s=gamma; h=date:from:to:cc:message-id:in-reply-to:references:subject:x-mailer :mime-version:content-type:content-transfer-encoding :content-disposition; b=QgUbE13CpwcSRynNc70IKLXDWqI+is3e4FTagHqct5T6HFp9qwhFV0TuqTSC4XLp9B yIcoWi4No4lZOgK1xyj5FzdoxAnlNxYqJemGN2dCFXpDs+J5IyF+3gFu+NzlaCN1ZovB KwDnFWInED7p8vS49K6lrClGdz8OuXvJlRQDY= Received: by 10.227.32.84 with SMTP id b20mr2659189wbd.105.1304349643823; Mon, 02 May 2011 08:20:43 -0700 (PDT) Received: from mmbp.local (port-92-206-28-38.dynamic.qsc.de [92.206.28.38]) by mx.google.com with ESMTPS id w12sm3449534wby.41.2011.05.02.08.20.40 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 02 May 2011 08:20:41 -0700 (PDT) Date: Mon, 2 May 2011 17:20:38 +0200 From: bfreek To: Zhu Sha Zang Message-ID: In-Reply-To: <4DBEC293.1010607@yahoo.com.br> References: <4DBEC293.1010607@yahoo.com.br> X-Mailer: sparrow 1.1.2 (build 688.7) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit Content-Disposition: inline Cc: freebsd-pf@freebsd.org Subject: Re: blocking facebook X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2011 15:48:50 -0000 could you provide us with the rules involved? -- Sent with Sparrow On Montag, 2. Mai 2011 at 16:41, Zhu Sha Zang wrote: > I'm trying to block facebook access only using PF in FreeBSD 8.2. > > But putting the name or the ip returned with the command host > www.facebook.com i can't deny any user to connect facebook. > > Some trick to do that? > > Thanks for now. > _______________________________________________ > freebsd-pf@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-pf > To unsubscribe, send any mail to "freebsd-pf-unsubscribe@freebsd.org" > From owner-freebsd-pf@FreeBSD.ORG Mon May 2 15:54:48 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8AC041065702 for ; Mon, 2 May 2011 15:54:48 +0000 (UTC) (envelope-from dudu@dudu.ro) Received: from mail-vx0-f182.google.com (mail-vx0-f182.google.com [209.85.220.182]) by mx1.freebsd.org (Postfix) with ESMTP id 4FC918FC21 for ; Mon, 2 May 2011 15:54:48 +0000 (UTC) Received: by vxc34 with SMTP id 34so5782743vxc.13 for ; Mon, 02 May 2011 08:54:47 -0700 (PDT) Received: by 10.52.175.99 with SMTP id bz3mr4952130vdc.117.1304350328063; Mon, 02 May 2011 08:32:08 -0700 (PDT) MIME-Version: 1.0 Received: by 10.52.112.199 with HTTP; Mon, 2 May 2011 08:31:28 -0700 (PDT) In-Reply-To: <4DBEC293.1010607@yahoo.com.br> References: <4DBEC293.1010607@yahoo.com.br> From: Vlad Galu Date: Mon, 2 May 2011 17:31:28 +0200 Message-ID: To: Zhu Sha Zang Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-pf@freebsd.org Subject: Re: blocking facebook X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2011 15:54:48 -0000 On Mon, May 2, 2011 at 4:41 PM, Zhu Sha Zang wrote: > I'm trying to block facebook access only using PF in FreeBSD 8.2. > > But putting the name or the ip returned with the command host > www.facebook.com i can't deny any user to connect facebook. > > Some trick to do that? > > You could simply block all their prefixes: http://www.robtex.com/as/as32934.html -- Good, fast & cheap. Pick any two. From owner-freebsd-pf@FreeBSD.ORG Mon May 2 16:37:58 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6A41E106566C for ; Mon, 2 May 2011 16:37:58 +0000 (UTC) (envelope-from edwinlculp@gmail.com) Received: from mail-vx0-f182.google.com (mail-vx0-f182.google.com [209.85.220.182]) by mx1.freebsd.org (Postfix) with ESMTP id 1F2988FC13 for ; Mon, 2 May 2011 16:37:57 +0000 (UTC) Received: by vxc34 with SMTP id 34so5827444vxc.13 for ; Mon, 02 May 2011 09:37:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=R8VAhkz7FddgqH8J5pqHuPyC3hdVrZbICXynyYpLkys=; b=CQ1C20my62qMwMqEqK8Ipes/hgu48qHqkcKWWIWk25/phRYtu87BQj1x6yGMrEfBFB CZOQ+Vp8zzieqm3YOVUL47NviqxLZkxWQ0hRWrSEJh1Fs3gosP7aCH3QBswcFkbLhnMU UMMnmDGt4QjWvdaHl/tCXdLcifJU425IDS4o0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=hvTxeBAC7XQG16KWqXjQMpBbRvA3xJV/GMSL2fLLY3C1vxKjX+f2CdB1hzI8hLUY9M /Vvreu9QpF70s0LasHm+Lts0wZAZtE3KNtBx9A0T5SMMNiygzsFmyR6udVJ+OKztnNyj iCxj25pn3XLI2QeJlgaJKpD9j5U9hqXySM1D8= MIME-Version: 1.0 Received: by 10.52.68.168 with SMTP id x8mr1581915vdt.77.1304352393731; Mon, 02 May 2011 09:06:33 -0700 (PDT) Received: by 10.52.107.5 with HTTP; Mon, 2 May 2011 09:06:33 -0700 (PDT) In-Reply-To: <4DBEC293.1010607@yahoo.com.br> References: <4DBEC293.1010607@yahoo.com.br> Date: Mon, 2 May 2011 11:06:33 -0500 Message-ID: From: "Edwin L. Culp W." To: Zhu Sha Zang Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-pf@freebsd.org Subject: Re: blocking facebook X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2011 16:37:58 -0000 On Mon, May 2, 2011 at 9:41 AM, Zhu Sha Zang wrote: > I'm trying to block facebook access only using PF in FreeBSD 8.2. > > But putting the name or the ip returned with the command host > www.facebook.com i can't deny any user to connect facebook. I found a way to block it with pf but didn't have the control that I wanted so I started using Squid and am super happy. I even set it by time spans, days, etc. etc. I have a file that has facebook in the /usr/local/log/squid/ directory /usr/local/etc/squid # cat squid-block.acl .facebook.com .fbcdn.net In my squid.conf file i added. # This is a special "public" machine that on ocassion needs facebook accss.. acl myclients src 172.16.0.5/32 http_access allow myclients # This should be clear with times and weekdays specified and it is just under the allow for 172.16.0.5 acl bad url_regex -i "/usr/local/etc/squid/squid-block.acl" acl lunchtime time MTWHF 14:00-16:15 acl night time MTWHF 18:45-23:59 acl morning time MTWHF 00:00-10:30 http_access deny bad !lunchtime !morning !night I find it works fine and prefer it be in squid than PF I use the following in PF and it seems to work but IMMHO I still prefer squid and find it much safer. I have only used pf to block my LAN and and haven't taken time to find a way to allow some ip's and delete the rest plus I don't see it as practical. My pf.conf is confusing enough without adding lan user stuff. You might wan to look at http://wiki.squid-cache.org/SquidFaq/SquidAcl#Access_Lists Hope this helps, ed > > Some trick to do that? > > Thanks for now. > _______________________________________________ > freebsd-pf@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-pf > To unsubscribe, send any mail to "freebsd-pf-unsubscribe@freebsd.org" > From owner-freebsd-pf@FreeBSD.ORG Tue May 3 02:12:14 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1B2D5106564A for ; Tue, 3 May 2011 02:12:14 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta15.westchester.pa.mail.comcast.net (qmta15.westchester.pa.mail.comcast.net [76.96.59.228]) by mx1.freebsd.org (Postfix) with ESMTP id B8BE28FC13 for ; Tue, 3 May 2011 02:12:13 +0000 (UTC) Received: from omta21.westchester.pa.mail.comcast.net ([76.96.62.72]) by qmta15.westchester.pa.mail.comcast.net with comcast id ept61g0061ZXKqc5Fpyyge; Tue, 03 May 2011 01:58:58 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta21.westchester.pa.mail.comcast.net with comcast id epyw1g00d1t3BNj3hpyxTl; Tue, 03 May 2011 01:58:58 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id E58B49B418; Mon, 2 May 2011 18:58:54 -0700 (PDT) Date: Mon, 2 May 2011 18:58:54 -0700 From: Jeremy Chadwick To: freebsd-pf@freebsd.org Message-ID: <20110503015854.GA31444@icarus.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-stable@freebsd.org Subject: RELENG_8 pf stack issue (state count spiraling out of control) X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 02:12:14 -0000 (Please keep me CC'd as I'm not subscribed to freebsd-pf. And apologies for cross-posting, but the issue is severe enough that I wanted to make it known on -stable) The below issue I'm describing is from a machine running 8.2-PRERELEASE (RELENG_8) using src dated Tue Feb 15 05:46:02 PST 2011. Please read the story in full, as I have taken the time to describe everything I did, plus log output, as well as induce a panic via "call doadump" from ddb so I have a capture of the system at the time. I also have a theory as to what caused the problem, but how to trigger it is unknown; it may be a rare race condition. This morning I woke up to find a report from one of our users that he could not connect to a specific TCP port (not SSH) on one of our servers. I also found that I couldn't SSH into the same box. Serial console was working fine, and the serial console log showed no sign of any problems. I started to debug the issue of me not being able to SSH into the machine and within a few minutes became immediately concerned: pfctl indicated we had reached the maximum number permitted state table entries (10,000). ============================================================ # pfctl -s info Status: Enabled for 76 days 06:49:10 Debug: Urgent Interface Stats for em0 IPv4 IPv6 Bytes In 8969748840 0 Bytes Out 8296135477 0 Packets In Passed 128211763 0 Blocked 621379 0 Packets Out Passed 138483868 0 Blocked 2579 0 State Table Total Rate current entries 10000 searches 267316807 40.6/s inserts 4440553 0.7/s removals 4430553 0.7/s Counters match 5067474 0.8/s bad-offset 0 0.0/s fragment 324 0.0/s short 0 0.0/s normalize 32 0.0/s memory 336946 0.1/s bad-timestamp 0 0.0/s congestion 0 0.0/s ip-option 0 0.0/s proto-cksum 1611 0.0/s state-mismatch 509 0.0/s state-insert 0 0.0/s state-limit 0 0.0/s src-limit 0 0.0/s synproxy 0 0.0/s # pfctl -s memory states hard limit 10000 src-nodes hard limit 10000 frags hard limit 5000 tables hard limit 1000 table-entries hard limit 100000 ============================================================ The above is mainly for em0 (our WAN interface); our LAN interface (em1) was not impacted because we use "set skip on em1". And it's a good thing too: we have lots of LAN-based services that this machine provides that would have been impacted. We also use "set skip on lo0". I immediately went to look at our monitoring graphs, which monitor pf state (specifically state table entries), polled via bsnmpd(8). This data is even more frightening: http://jdc.parodius.com/freebsd/pf-issue/pf_states-day.png http://jdc.parodius.com/freebsd/pf-issue/pf_states-week.png Literally something was spiraling out of control, starting at approx. 2011/05/01 (Sun) at 12:30 PDT. The situation became dire at approx. 19:45 PDT the same day, but I wasn't aware of it until said user brought an issue to my attention. You can see from the network I/O graphs (taken from SNMP polling our switch, NOT from the host/box itself) that there was no DoS attack or anything like that occurring -- this was something within FreeBSD itself. More evidence of that will become apparent. http://jdc.parodius.com/freebsd/pf-issue/port_03-day.png http://jdc.parodius.com/freebsd/pf-issue/port_03-week.png The first thing I did was "/etc/rc.d/pf reload". This command hung. Any attempt to send Ctrl-C/SIGINT did nothing. I was able to Ctrl-Z/SIGSTOP it, then use kill %1, but the actual reload process did not truly die (despite csh stating "Terminated"). The only way to kill it was to kill -9. Attempts to shut down any daemons which utilised the network -- including things that only used em1 -- would not shut down. This included things like postfix, mysqld, and some inet-based services. I was forced to kill -9 them. Things like bsnmpd, however, did shut down. Equally as uncomfortable, "shutdown -r now" did not reboot the system. That is to say, wall(1)'s announcement was shown, but the actual stopping of services did not begin. The next thing I tried was "/etc/rc.d/pf stop", which worked. Then I did "/etc/rc.d/pf start", which also worked. However, what I saw next surely indicated a bug in the pf layer somewhere -- "pfctl -s states" and "pfctl -s info" disagreed on the state count: ============================================================ # pfctl -s info Status: Enabled for 0 days 00:00:16 Debug: Urgent Interface Stats for em0 IPv4 IPv6 Bytes In 3459 0 Bytes Out 0 0 Packets In Passed 0 0 Blocked 29 0 Packets Out Passed 0 0 Blocked 0 0 State Table Total Rate current entries 10000 searches 29 1.8/s inserts 0 0.0/s removals 0 0.0/s Counters match 29 1.8/s bad-offset 0 0.0/s fragment 0 0.0/s short 0 0.0/s normalize 0 0.0/s memory 18 1.1/s bad-timestamp 0 0.0/s congestion 0 0.0/s ip-option 0 0.0/s proto-cksum 0 0.0/s state-mismatch 0 0.0/s state-insert 0 0.0/s state-limit 0 0.0/s src-limit 0 0.0/s synproxy 0 0.0/s # pfctl -s state | wc -l 0 ============================================================ The "pf uptime" shown above, by the way, matches system uptime. I then attempted "pfctl -F state", but nothing changed (looked the same as above). Since I could not reboot the box, I was forced to drop to ddb via serial console. I did some commands like "ps" and the like, and then "call doadump" to induce a kernel panic, and then "reboot" (which worked). Once the machine came back up, savecore(8) ran, wrote the data out, and everything has been fine since. /var/crash/core.txt.0 is ~68KBytes and I do not feel comfortable sharing its content publicly, but will be happy to hand it to developer(s) who are interested. Relevant tidbits I can discern: ------------------------------------------------------------------------ ps -axl UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 0 422 0 0 -16 0 0 0 pftm DL ?? 1362773081:04.00 [pfpurge] ------------------------------------------------------------------------ ------------------------------------------------------------------------ vmstat -z ITEM SIZE LIMIT USED FREE REQUESTS FAILURES pfsrctrpl: 152, 10000, 0, 0, 0, 0 pfrulepl: 912, 0, 40, 88, 806, 0 pfstatepl: 392, 10000, 10000, 0, 4440553, 341638 pfaltqpl: 240, 0, 0, 0, 0, 0 pfpooladdrpl: 88, 0, 0, 0, 0, 0 pfrktable: 1296, 1002, 4, 20, 112, 0 pfrkentry: 216, 100008, 603, 891, 15384, 0 pfrkentry2: 216, 0, 0, 0, 0, 0 pffrent: 32, 5050, 0, 303, 1620, 0 pffrag: 80, 0, 0, 135, 807, 0 pffrcache: 80, 10035, 0, 0, 0, 0 pffrcent: 24, 50022, 0, 0, 0, 0 pfstatescrub: 40, 0, 0, 0, 0, 0 pfiaddrpl: 120, 0, 0, 0, 0, 0 pfospfen: 112, 0, 696, 30, 18096, 0 pfosfp: 40, 0, 407, 97, 10582, 0 ------------------------------------------------------------------------ You can see evidence of processes not exiting/doing what they should do here: ------------------------------------------------------------------------ fstat USER CMD PID FD MOUNT INUM MODE SZ|DV R/W root shutdown 91155 root / 2 drwxr-xr-x 512 r root shutdown 91155 wd / 2 drwxr-xr-x 512 r root shutdown 91155 text / 47195 -r-sr-x--- 15912 r root shutdown 91155 0 /dev 38 crw------- ttyu0 rw root shutdown 91155 1 /dev 38 crw------- ttyu0 rw root shutdown 91155 2 /dev 38 crw------- ttyu0 rw root sh 91129 root / 2 drwxr-xr-x 512 r root sh 91129 wd / 2 drwxr-xr-x 512 r root sh 91129 text / 44 -r-xr-xr-x 134848 r root sh 91129 0 /dev 38 crw------- ttyu0 rw root sh 91129 1* pipe ffffff01e78fc9e0 <-> ffffff01e78fc888 0 rw root sh 91129 2 /dev 20 crw-rw-rw- null w root shutdown 91115 root / 2 drwxr-xr-x 512 r root shutdown 91115 wd /storage 5 drwx------ 37 r root shutdown 91115 text / 47195 -r-sr-x--- 15912 r root shutdown 91115 0 /dev 38 crw------- ttyu0 rw root shutdown 91115 1 /dev 38 crw------- ttyu0 rw root shutdown 91115 2 /dev 38 crw------- ttyu0 rw root shutdown 91115 3* local dgram ffffff008ff92960 root sh 90818 root / 2 drwxr-xr-x 512 r root sh 90818 wd / 70659 drwxr-xr-x 512 r root sh 90818 text / 44 -r-xr-xr-x 134848 r root sh 90818 0 /dev 38 crw------- ttyu0 rw root sh 90818 1* pipe ffffff0043f1ecb8 <-> ffffff0043f1eb60 0 rw root sh 90818 2 /dev 20 crw-rw-rw- null w root csh 90802 root / 2 drwxr-xr-x 512 r root csh 90802 wd / 2 drwxr-xr-x 512 r root csh 90802 text / 51 -r-xr-xr-x 358752 r root csh 90802 15 /dev 38 crw------- ttyu0 rw root csh 90802 16 /dev 38 crw------- ttyu0 rw root csh 90802 17 /dev 38 crw------- ttyu0 rw root csh 90802 18 /dev 38 crw------- ttyu0 rw root csh 90802 19 /dev 38 crw------- ttyu0 rw ------------------------------------------------------------------------ No indication of mbuf exhaustion, putting further focus on the pf stack: ------------------------------------------------------------------------ netstat -m 2054/1786/3840 mbufs in use (current/cache/total) 2048/1414/3462/25600 mbuf clusters in use (current/cache/total/max) 2048/896 mbuf+clusters out of packet secondary zone in use (current/cache) 0/320/320/12800 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/19200 9k jumbo clusters in use (current/cache/total/max) 0/0/0/12800 16k jumbo clusters in use (current/cache/total/max) 4609K/4554K/9164K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 0 calls to protocol drain routines ------------------------------------------------------------------------ Here's one piece of core.0.txt which makes no sense to me -- the "rate" column. I have a very hard time believing that was the interrupt rate of all the relevant devices at the time (way too high). Maybe this data becomes wrong only during a coredump? The total column I could believe. ------------------------------------------------------------------------ vmstat -i interrupt total rate irq4: uart0 54768 912 irq6: fdc0 1 0 irq17: uhci1+ 172 2 irq23: uhci3 ehci1+ 2367 39 cpu0: timer 13183882632 219731377 irq256: em0 260491055 4341517 irq257: em1 127555036 2125917 irq258: ahci0 225923164 3765386 cpu2: timer 13183881837 219731363 cpu1: timer 13002196469 216703274 cpu3: timer 13183881783 219731363 Total 53167869284 886131154 ------------------------------------------------------------------------ Here's what a normal "vmstat -i" shows from the command-line: # vmstat -i interrupt total rate irq4: uart0 518 0 irq6: fdc0 1 0 irq23: uhci3 ehci1+ 145 0 cpu0: timer 19041199 1999 irq256: em0 614280 64 irq257: em1 168529 17 irq258: ahci0 355536 37 cpu2: timer 19040462 1999 cpu1: timer 19040458 1999 cpu3: timer 19040454 1999 Total 77301582 8119 We graph many aspects of this box, including CPU load, memory/swap usage, etc. and none show any sign that the interrupt rate on all of those devices was even remotely out of control. (I would expect to see CPU through the roof given the above data) I have since rebuilt/reinstalled world/kernel on the machine with the latest RELENG_8 code (box is now 8.2-STABLE #0: Mon May 2 14:44:18 PDT 2011), hoping whatever this was may have been fixed. As for what I think may have triggered it, but I have no hard evidence of such: on April 29th, I changed our pf.conf and did "/etc/rc.d/pf reload". The pf.conf change was a single line: Old: scrub on em0 all New: scrub in on em0 all Why it took the problem approximately 3 days to start is unknown. It's the only change we've made to the system (truly/honestly), and it was a change to pf.conf. If anyone has advice (or has seen the above problem), or is interested in debugging it -- as I said, I have a vmcore -- I'm happy to assist in any way I can. I would hate for someone else to get bit by this, and really am hoping its something that has been fixed between February and now. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-pf@FreeBSD.ORG Tue May 3 05:06:41 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 811A4106564A; Tue, 3 May 2011 05:06:41 +0000 (UTC) (envelope-from jhellenthal@gmail.com) Received: from mail-iy0-f182.google.com (mail-iy0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id 282878FC08; Tue, 3 May 2011 05:06:40 +0000 (UTC) Received: by iyj12 with SMTP id 12so7359715iyj.13 for ; Mon, 02 May 2011 22:06:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:sender:date:from:to:cc:subject:message-id :references:mime-version:content-type:content-disposition :in-reply-to:x-openpgp-key-id:x-openpgp-key-fingerprint :x-openpgp-key-url; bh=neU4BmS9TB85aasLW7PY2eL/gr8SmoXox+uWqdd219o=; b=UFMfL3ZwWcOm4C4KIVQMmCYL9E24Mtg9xm+1mroVu3yHONU6Gh376zyAa7NkT8Av3I VQ2cosaeD/tBMgwZYFdzKQUc70dPf+4Wi5MkwHFr8avLJ9hco9L+qwMyy3kd22ZCEnI1 ndl9tfQHAuU9wvQQwPYnO+F7wH+BFOZ2PeeVU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:x-openpgp-key-id :x-openpgp-key-fingerprint:x-openpgp-key-url; b=KoeOo1hQYNSzv1kFi6NZFI/gtphI38eqsxvpfxXprI03NSYf8norXJmq2SoY6U8Spw hbneoKdZpilfCQ6GidEFKHZFY27RGwN5HiLccoUToxHQFlMjoz/+9ODRep0ZgXNfu+ji /cyEKbJ7d94ZdLhZOx4JIXGXSRLdSwJ8SnQWI= Received: by 10.42.115.138 with SMTP id k10mr1900703icq.81.1304399200506; Mon, 02 May 2011 22:06:40 -0700 (PDT) Received: from DataIX.net ([99.190.84.116]) by mx.google.com with ESMTPS id vr5sm2395488icb.12.2011.05.02.22.06.37 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 02 May 2011 22:06:38 -0700 (PDT) Sender: "J. Hellenthal" Received: from DataIX.net (localhost [127.0.0.1]) by DataIX.net (8.14.4/8.14.4) with ESMTP id p4356YJY018800 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 3 May 2011 01:06:35 -0400 (EDT) (envelope-from jhell@DataIX.net) Received: (from jhell@localhost) by DataIX.net (8.14.4/8.14.4/Submit) id p4356YPf018799; Tue, 3 May 2011 01:06:34 -0400 (EDT) (envelope-from jhell@DataIX.net) Date: Tue, 3 May 2011 01:06:34 -0400 From: Jason Hellenthal To: Jeremy Chadwick Message-ID: <20110503050634.GB53414@DataIX.net> References: <20110503015854.GA31444@icarus.home.lan> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="DBIVS5p969aUjpLe" Content-Disposition: inline In-Reply-To: <20110503015854.GA31444@icarus.home.lan> X-OpenPGP-Key-Id: 0x89D8547E X-OpenPGP-Key-Fingerprint: 85EF E26B 07BB 3777 76BE B12A 9057 8789 89D8 547E X-OpenPGP-Key-URL: http://bit.ly/0x89D8547E Cc: freebsd-stable@freebsd.org, freebsd-pf@freebsd.org Subject: Re: RELENG_8 pf stack issue (state count spiraling out of control) X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 05:06:41 -0000 --DBIVS5p969aUjpLe Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Jeremy, On Mon, May 02, 2011 at 06:58:54PM -0700, Jeremy Chadwick wrote: >(Please keep me CC'd as I'm not subscribed to freebsd-pf. And apologies >for cross-posting, but the issue is severe enough that I wanted to make >it known on -stable) > >The below issue I'm describing is from a machine running 8.2-PRERELEASE >(RELENG_8) using src dated Tue Feb 15 05:46:02 PST 2011. > >Please read the story in full, as I have taken the time to describe >everything I did, plus log output, as well as induce a panic via "call >doadump" from ddb so I have a capture of the system at the time. I also >have a theory as to what caused the problem, but how to trigger it is >unknown; it may be a rare race condition. > > >This morning I woke up to find a report from one of our users that he >could not connect to a specific TCP port (not SSH) on one of our >servers. I also found that I couldn't SSH into the same box. Serial >console was working fine, and the serial console log showed no sign of >any problems. > >I started to debug the issue of me not being able to SSH into the >machine and within a few minutes became immediately concerned: pfctl >indicated we had reached the maximum number permitted state table >entries (10,000). > >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D ># pfctl -s info >Status: Enabled for 76 days 06:49:10 Debug: Urgent > >Interface Stats for em0 IPv4 IPv6 > Bytes In 8969748840 0 > Bytes Out 8296135477 0 > Packets In > Passed 128211763 0 > Blocked 621379 0 > Packets Out > Passed 138483868 0 > Blocked 2579 0 > >State Table Total Rate > current entries 10000 > searches 267316807 40.6/s > inserts 4440553 0.7/s > removals 4430553 0.7/s >Counters > match 5067474 0.8/s > bad-offset 0 0.0/s > fragment 324 0.0/s > short 0 0.0/s > normalize 32 0.0/s > memory 336946 0.1/s > bad-timestamp 0 0.0/s > congestion 0 0.0/s > ip-option 0 0.0/s > proto-cksum 1611 0.0/s > state-mismatch 509 0.0/s > state-insert 0 0.0/s > state-limit 0 0.0/s > src-limit 0 0.0/s > synproxy 0 0.0/s > ># pfctl -s memory >states hard limit 10000 >src-nodes hard limit 10000 >frags hard limit 5000 >tables hard limit 1000 >table-entries hard limit 100000 >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > >The above is mainly for em0 (our WAN interface); our LAN interface (em1) >was not impacted because we use "set skip on em1". And it's a good >thing too: we have lots of LAN-based services that this machine provides >that would have been impacted. We also use "set skip on lo0". > >I immediately went to look at our monitoring graphs, which monitor pf >state (specifically state table entries), polled via bsnmpd(8). This >data is even more frightening: > >http://jdc.parodius.com/freebsd/pf-issue/pf_states-day.png >http://jdc.parodius.com/freebsd/pf-issue/pf_states-week.png > >Literally something was spiraling out of control, starting at approx. >2011/05/01 (Sun) at 12:30 PDT. The situation became dire at approx. >19:45 PDT the same day, but I wasn't aware of it until said user brought >an issue to my attention. > >You can see from the network I/O graphs (taken from SNMP polling our >switch, NOT from the host/box itself) that there was no DoS attack or >anything like that occurring -- this was something within FreeBSD >itself. More evidence of that will become apparent. > >http://jdc.parodius.com/freebsd/pf-issue/port_03-day.png >http://jdc.parodius.com/freebsd/pf-issue/port_03-week.png > >The first thing I did was "/etc/rc.d/pf reload". This command hung. >Any attempt to send Ctrl-C/SIGINT did nothing. I was able to >Ctrl-Z/SIGSTOP it, then use kill %1, but the actual reload process did >not truly die (despite csh stating "Terminated"). The only way to kill >it was to kill -9. > >Attempts to shut down any daemons which utilised the network -- >including things that only used em1 -- would not shut down. This >included things like postfix, mysqld, and some inet-based services. I >was forced to kill -9 them. Things like bsnmpd, however, did shut down. > >Equally as uncomfortable, "shutdown -r now" did not reboot the system. >That is to say, wall(1)'s announcement was shown, but the actual >stopping of services did not begin. > >The next thing I tried was "/etc/rc.d/pf stop", which worked. Then I >did "/etc/rc.d/pf start", which also worked. However, what I saw next >surely indicated a bug in the pf layer somewhere -- "pfctl -s states" >and "pfctl -s info" disagreed on the state count: > >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D ># pfctl -s info >Status: Enabled for 0 days 00:00:16 Debug: Urgent > >Interface Stats for em0 IPv4 IPv6 > Bytes In 3459 0 > Bytes Out 0 0 > Packets In > Passed 0 0 > Blocked 29 0 > Packets Out > Passed 0 0 > Blocked 0 0 > >State Table Total Rate > current entries 10000 > searches 29 1.8/s > inserts 0 0.0/s > removals 0 0.0/s >Counters > match 29 1.8/s > bad-offset 0 0.0/s > fragment 0 0.0/s > short 0 0.0/s > normalize 0 0.0/s > memory 18 1.1/s > bad-timestamp 0 0.0/s > congestion 0 0.0/s > ip-option 0 0.0/s > proto-cksum 0 0.0/s > state-mismatch 0 0.0/s > state-insert 0 0.0/s > state-limit 0 0.0/s > src-limit 0 0.0/s > synproxy 0 0.0/s > ># pfctl -s state | wc -l > 0 >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > >The "pf uptime" shown above, by the way, matches system uptime. > >I then attempted "pfctl -F state", but nothing changed (looked the same >as above). > >Since I could not reboot the box, I was forced to drop to ddb via serial >console. I did some commands like "ps" and the like, and then "call >doadump" to induce a kernel panic, and then "reboot" (which worked). > >Once the machine came back up, savecore(8) ran, wrote the data out, and >everything has been fine since. /var/crash/core.txt.0 is ~68KBytes and >I do not feel comfortable sharing its content publicly, but will be >happy to hand it to developer(s) who are interested. Relevant tidbits I >can discern: > >------------------------------------------------------------------------ >ps -axl > > UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND > 0 422 0 0 -16 0 0 0 pftm DL ?? 1362773081:04.00= [pfpurge] >------------------------------------------------------------------------ > >------------------------------------------------------------------------ >vmstat -z > >ITEM SIZE LIMIT USED FREE REQUESTS FAI= LURES >pfsrctrpl: 152, 10000, 0, 0, 0, = 0 >pfrulepl: 912, 0, 40, 88, 806, = 0 >pfstatepl: 392, 10000, 10000, 0, 4440553, 3= 41638 >pfaltqpl: 240, 0, 0, 0, 0, = 0 >pfpooladdrpl: 88, 0, 0, 0, 0, = 0 >pfrktable: 1296, 1002, 4, 20, 112, = 0 >pfrkentry: 216, 100008, 603, 891, 15384, = 0 >pfrkentry2: 216, 0, 0, 0, 0, = 0 >pffrent: 32, 5050, 0, 303, 1620, = 0 >pffrag: 80, 0, 0, 135, 807, = 0 >pffrcache: 80, 10035, 0, 0, 0, = 0 >pffrcent: 24, 50022, 0, 0, 0, = 0 >pfstatescrub: 40, 0, 0, 0, 0, = 0 >pfiaddrpl: 120, 0, 0, 0, 0, = 0 >pfospfen: 112, 0, 696, 30, 18096, = 0 >pfosfp: 40, 0, 407, 97, 10582, = 0 >------------------------------------------------------------------------ > >You can see evidence of processes not exiting/doing what they should do >here: > >------------------------------------------------------------------------ >fstat > >USER CMD PID FD MOUNT INUM MODE SZ|DV R/W >root shutdown 91155 root / 2 drwxr-xr-x 512 r >root shutdown 91155 wd / 2 drwxr-xr-x 512 r >root shutdown 91155 text / 47195 -r-sr-x--- 15912 r >root shutdown 91155 0 /dev 38 crw------- ttyu0 rw >root shutdown 91155 1 /dev 38 crw------- ttyu0 rw >root shutdown 91155 2 /dev 38 crw------- ttyu0 rw >root sh 91129 root / 2 drwxr-xr-x 512 r >root sh 91129 wd / 2 drwxr-xr-x 512 r >root sh 91129 text / 44 -r-xr-xr-x 134848 r >root sh 91129 0 /dev 38 crw------- ttyu0 rw >root sh 91129 1* pipe ffffff01e78fc9e0 <-> ffffff01e78fc888= 0 rw >root sh 91129 2 /dev 20 crw-rw-rw- null w >root shutdown 91115 root / 2 drwxr-xr-x 512 r >root shutdown 91115 wd /storage 5 drwx------ 37 r >root shutdown 91115 text / 47195 -r-sr-x--- 15912 r >root shutdown 91115 0 /dev 38 crw------- ttyu0 rw >root shutdown 91115 1 /dev 38 crw------- ttyu0 rw >root shutdown 91115 2 /dev 38 crw------- ttyu0 rw >root shutdown 91115 3* local dgram ffffff008ff92960 >root sh 90818 root / 2 drwxr-xr-x 512 r >root sh 90818 wd / 70659 drwxr-xr-x 512 r >root sh 90818 text / 44 -r-xr-xr-x 134848 r >root sh 90818 0 /dev 38 crw------- ttyu0 rw >root sh 90818 1* pipe ffffff0043f1ecb8 <-> ffffff0043f1eb60= 0 rw >root sh 90818 2 /dev 20 crw-rw-rw- null w >root csh 90802 root / 2 drwxr-xr-x 512 r >root csh 90802 wd / 2 drwxr-xr-x 512 r >root csh 90802 text / 51 -r-xr-xr-x 358752 r >root csh 90802 15 /dev 38 crw------- ttyu0 rw >root csh 90802 16 /dev 38 crw------- ttyu0 rw >root csh 90802 17 /dev 38 crw------- ttyu0 rw >root csh 90802 18 /dev 38 crw------- ttyu0 rw >root csh 90802 19 /dev 38 crw------- ttyu0 rw >------------------------------------------------------------------------ > >No indication of mbuf exhaustion, putting further focus on the pf stack: > >------------------------------------------------------------------------ >netstat -m > >2054/1786/3840 mbufs in use (current/cache/total) >2048/1414/3462/25600 mbuf clusters in use (current/cache/total/max) >2048/896 mbuf+clusters out of packet secondary zone in use (current/cache) >0/320/320/12800 4k (page size) jumbo clusters in use (current/cache/total/= max) >0/0/0/19200 9k jumbo clusters in use (current/cache/total/max) >0/0/0/12800 16k jumbo clusters in use (current/cache/total/max) >4609K/4554K/9164K bytes allocated to network (current/cache/total) >0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) >0/0/0 requests for jumbo clusters denied (4k/9k/16k) >0 requests for sfbufs denied >0 requests for sfbufs delayed >0 requests for I/O initiated by sendfile >0 calls to protocol drain routines >------------------------------------------------------------------------ > >Here's one piece of core.0.txt which makes no sense to me -- the "rate" >column. I have a very hard time believing that was the interrupt rate >of all the relevant devices at the time (way too high). Maybe this data >becomes wrong only during a coredump? The total column I could believe. > >------------------------------------------------------------------------ >vmstat -i > >interrupt total rate >irq4: uart0 54768 912 >irq6: fdc0 1 0 >irq17: uhci1+ 172 2 >irq23: uhci3 ehci1+ 2367 39 >cpu0: timer 13183882632 219731377 >irq256: em0 260491055 4341517 >irq257: em1 127555036 2125917 >irq258: ahci0 225923164 3765386 >cpu2: timer 13183881837 219731363 >cpu1: timer 13002196469 216703274 >cpu3: timer 13183881783 219731363 >Total 53167869284 886131154 >------------------------------------------------------------------------ > >Here's what a normal "vmstat -i" shows from the command-line: > ># vmstat -i >interrupt total rate >irq4: uart0 518 0 >irq6: fdc0 1 0 >irq23: uhci3 ehci1+ 145 0 >cpu0: timer 19041199 1999 >irq256: em0 614280 64 >irq257: em1 168529 17 >irq258: ahci0 355536 37 >cpu2: timer 19040462 1999 >cpu1: timer 19040458 1999 >cpu3: timer 19040454 1999 >Total 77301582 8119 > >We graph many aspects of this box, including CPU load, memory/swap >usage, etc. and none show any sign that the interrupt rate on all of >those devices was even remotely out of control. (I would expect to see >CPU through the roof given the above data) > >I have since rebuilt/reinstalled world/kernel on the machine with the >latest RELENG_8 code (box is now 8.2-STABLE #0: Mon May 2 14:44:18 PDT >2011), hoping whatever this was may have been fixed. > >As for what I think may have triggered it, but I have no hard evidence >of such: on April 29th, I changed our pf.conf and did "/etc/rc.d/pf >reload". The pf.conf change was a single line: > >Old: scrub on em0 all >New: scrub in on em0 all > >Why it took the problem approximately 3 days to start is unknown. It's >the only change we've made to the system (truly/honestly), and it was a >change to pf.conf. > >If anyone has advice (or has seen the above problem), or is interested >in debugging it -- as I said, I have a vmcore -- I'm happy to assist in >any way I can. I would hate for someone else to get bit by this, and >really am hoping its something that has been fixed between February and >now. > That's quite the deduction there. I've noticed recently that you were also= =20 experimenting with the new NFS server recompiling kernel etc etc. Seeing=20 as weird things can happen with DNS, NFS and mountpoint's, is this the same= =20 machine that you were doing that on ? If so can you check to see how many requests for NFS operations were done= =20 to/from that box ? as well the names that would be being resolved and if=20 that machine can resolve them ? Also I would believe your using tables in your pf.conf, if so do any of=20 those tables contain a FQDN that cannot be resolved from that machine ? I think you probably see what I am getting at here as it could be some=20 sort of concurrent recursive DNS failure that can only be seen from the=20 machine caused by possibly the new NFS backend or a change in one of the=20 tables that pf would use. --=20 Regards, (jhell) Jason Hellenthal --DBIVS5p969aUjpLe Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.17 (FreeBSD) Comment: http://bit.ly/0x89D8547E iQEcBAEBAgAGBQJNv41ZAAoJEJBXh4mJ2FR+kv0H/jLwWK5wBbRHO0t4PEeUTITN pKljwFvrnTSTDC1WIeYAe6AEL5zERfHaYsl2zGG2XiYCiVoSOaVRRYP/4lkztG3h X3K68z0q8WgTksr9L1iIR/oj2YRLGzvTI9ChV2lQh+lORF2mLW+doYlipPa0sAGq PJrruYlQk5uaY/9rEQn+/5QxvFM1A73yBVIoQKI4UAEdlv8tGlFAcyrqkOGnJ8ju ji6VOQfOndbwI6B+0rezkvtFGlSjPQTCVsl8+IAkA2rgcUPkGgOm9LxGFHsVzbc+ Kkj9zRKCSDc92qVKyw7+Asaucnm6lnhF/IHDYgvVPIGcG/eLi/PvVnbh003svmg= =xLPk -----END PGP SIGNATURE----- --DBIVS5p969aUjpLe-- From owner-freebsd-pf@FreeBSD.ORG Tue May 3 05:22:51 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A4E4F106566C for ; Tue, 3 May 2011 05:22:51 +0000 (UTC) (envelope-from dudu@dudu.ro) Received: from mail-qw0-f54.google.com (mail-qw0-f54.google.com [209.85.216.54]) by mx1.freebsd.org (Postfix) with ESMTP id 34D2A8FC16 for ; Tue, 3 May 2011 05:22:50 +0000 (UTC) Received: by qwc9 with SMTP id 9so3730732qwc.13 for ; Mon, 02 May 2011 22:22:50 -0700 (PDT) Received: by 10.229.114.77 with SMTP id d13mr6803064qcq.219.1304400170184; Mon, 02 May 2011 22:22:50 -0700 (PDT) MIME-Version: 1.0 Received: by 10.229.105.18 with HTTP; Mon, 2 May 2011 22:22:10 -0700 (PDT) In-Reply-To: <20110503015854.GA31444@icarus.home.lan> References: <20110503015854.GA31444@icarus.home.lan> From: Vlad Galu Date: Tue, 3 May 2011 07:22:10 +0200 Message-ID: To: Jeremy Chadwick Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-stable@freebsd.org, freebsd-pf@freebsd.org Subject: Re: RELENG_8 pf stack issue (state count spiraling out of control) X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 05:22:51 -0000 On Tue, May 3, 2011 at 3:58 AM, Jeremy Chadwick wrote: > (Please keep me CC'd as I'm not subscribed to freebsd-pf. And apologies > for cross-posting, but the issue is severe enough that I wanted to make > it known on -stable) > > The below issue I'm describing is from a machine running 8.2-PRERELEASE > (RELENG_8) using src dated Tue Feb 15 05:46:02 PST 2011. > > Please read the story in full, as I have taken the time to describe > everything I did, plus log output, as well as induce a panic via "call > doadump" from ddb so I have a capture of the system at the time. I also > have a theory as to what caused the problem, but how to trigger it is > unknown; it may be a rare race condition. > > > This morning I woke up to find a report from one of our users that he > could not connect to a specific TCP port (not SSH) on one of our > servers. I also found that I couldn't SSH into the same box. Serial > console was working fine, and the serial console log showed no sign of > any problems. > > I started to debug the issue of me not being able to SSH into the > machine and within a few minutes became immediately concerned: pfctl > indicated we had reached the maximum number permitted state table > entries (10,000). > > ============================================================ > # pfctl -s info > Status: Enabled for 76 days 06:49:10 Debug: Urgent > > Interface Stats for em0 IPv4 IPv6 > Bytes In 8969748840 0 > Bytes Out 8296135477 0 > Packets In > Passed 128211763 0 > Blocked 621379 0 > Packets Out > Passed 138483868 0 > Blocked 2579 0 > > State Table Total Rate > current entries 10000 > searches 267316807 40.6/s > inserts 4440553 0.7/s > removals 4430553 0.7/s > Counters > match 5067474 0.8/s > bad-offset 0 0.0/s > fragment 324 0.0/s > short 0 0.0/s > normalize 32 0.0/s > memory 336946 0.1/s > bad-timestamp 0 0.0/s > congestion 0 0.0/s > ip-option 0 0.0/s > proto-cksum 1611 0.0/s > state-mismatch 509 0.0/s > state-insert 0 0.0/s > state-limit 0 0.0/s > src-limit 0 0.0/s > synproxy 0 0.0/s > > # pfctl -s memory > states hard limit 10000 > src-nodes hard limit 10000 > frags hard limit 5000 > tables hard limit 1000 > table-entries hard limit 100000 > ============================================================ > > The above is mainly for em0 (our WAN interface); our LAN interface (em1) > was not impacted because we use "set skip on em1". And it's a good > thing too: we have lots of LAN-based services that this machine provides > that would have been impacted. We also use "set skip on lo0". > > I immediately went to look at our monitoring graphs, which monitor pf > state (specifically state table entries), polled via bsnmpd(8). This > data is even more frightening: > > http://jdc.parodius.com/freebsd/pf-issue/pf_states-day.png > http://jdc.parodius.com/freebsd/pf-issue/pf_states-week.png > > Literally something was spiraling out of control, starting at approx. > 2011/05/01 (Sun) at 12:30 PDT. The situation became dire at approx. > 19:45 PDT the same day, but I wasn't aware of it until said user brought > an issue to my attention. > > You can see from the network I/O graphs (taken from SNMP polling our > switch, NOT from the host/box itself) that there was no DoS attack or > anything like that occurring -- this was something within FreeBSD > itself. More evidence of that will become apparent. > > http://jdc.parodius.com/freebsd/pf-issue/port_03-day.png > http://jdc.parodius.com/freebsd/pf-issue/port_03-week.png > > The first thing I did was "/etc/rc.d/pf reload". This command hung. > Any attempt to send Ctrl-C/SIGINT did nothing. I was able to > Ctrl-Z/SIGSTOP it, then use kill %1, but the actual reload process did > not truly die (despite csh stating "Terminated"). The only way to kill > it was to kill -9. > > Attempts to shut down any daemons which utilised the network -- > including things that only used em1 -- would not shut down. This > included things like postfix, mysqld, and some inet-based services. I > was forced to kill -9 them. Things like bsnmpd, however, did shut down. > > Equally as uncomfortable, "shutdown -r now" did not reboot the system. > That is to say, wall(1)'s announcement was shown, but the actual > stopping of services did not begin. > > The next thing I tried was "/etc/rc.d/pf stop", which worked. Then I > did "/etc/rc.d/pf start", which also worked. However, what I saw next > surely indicated a bug in the pf layer somewhere -- "pfctl -s states" > and "pfctl -s info" disagreed on the state count: > > ============================================================ > # pfctl -s info > Status: Enabled for 0 days 00:00:16 Debug: Urgent > > Interface Stats for em0 IPv4 IPv6 > Bytes In 3459 0 > Bytes Out 0 0 > Packets In > Passed 0 0 > Blocked 29 0 > Packets Out > Passed 0 0 > Blocked 0 0 > > State Table Total Rate > current entries 10000 > searches 29 1.8/s > inserts 0 0.0/s > removals 0 0.0/s > Counters > match 29 1.8/s > bad-offset 0 0.0/s > fragment 0 0.0/s > short 0 0.0/s > normalize 0 0.0/s > memory 18 1.1/s > bad-timestamp 0 0.0/s > congestion 0 0.0/s > ip-option 0 0.0/s > proto-cksum 0 0.0/s > state-mismatch 0 0.0/s > state-insert 0 0.0/s > state-limit 0 0.0/s > src-limit 0 0.0/s > synproxy 0 0.0/s > > # pfctl -s state | wc -l > 0 > ============================================================ > > The "pf uptime" shown above, by the way, matches system uptime. > > I then attempted "pfctl -F state", but nothing changed (looked the same > as above). > > Since I could not reboot the box, I was forced to drop to ddb via serial > console. I did some commands like "ps" and the like, and then "call > doadump" to induce a kernel panic, and then "reboot" (which worked). > > Once the machine came back up, savecore(8) ran, wrote the data out, and > everything has been fine since. /var/crash/core.txt.0 is ~68KBytes and > I do not feel comfortable sharing its content publicly, but will be > happy to hand it to developer(s) who are interested. Relevant tidbits I > can discern: > > ------------------------------------------------------------------------ > ps -axl > > UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND > 0 422 0 0 -16 0 0 0 pftm DL ?? 1362773081:04.00 > [pfpurge] > ------------------------------------------------------------------------ > > ------------------------------------------------------------------------ > vmstat -z > > ITEM SIZE LIMIT USED FREE REQUESTS > FAILURES > pfsrctrpl: 152, 10000, 0, 0, 0, > 0 > pfrulepl: 912, 0, 40, 88, 806, > 0 > pfstatepl: 392, 10000, 10000, 0, 4440553, > 341638 > pfaltqpl: 240, 0, 0, 0, 0, > 0 > pfpooladdrpl: 88, 0, 0, 0, 0, > 0 > pfrktable: 1296, 1002, 4, 20, 112, > 0 > pfrkentry: 216, 100008, 603, 891, 15384, > 0 > pfrkentry2: 216, 0, 0, 0, 0, > 0 > pffrent: 32, 5050, 0, 303, 1620, > 0 > pffrag: 80, 0, 0, 135, 807, > 0 > pffrcache: 80, 10035, 0, 0, 0, > 0 > pffrcent: 24, 50022, 0, 0, 0, > 0 > pfstatescrub: 40, 0, 0, 0, 0, > 0 > pfiaddrpl: 120, 0, 0, 0, 0, > 0 > pfospfen: 112, 0, 696, 30, 18096, > 0 > pfosfp: 40, 0, 407, 97, 10582, > 0 > ------------------------------------------------------------------------ > > You can see evidence of processes not exiting/doing what they should do > here: > > ------------------------------------------------------------------------ > fstat > > USER CMD PID FD MOUNT INUM MODE SZ|DV R/W > root shutdown 91155 root / 2 drwxr-xr-x 512 r > root shutdown 91155 wd / 2 drwxr-xr-x 512 r > root shutdown 91155 text / 47195 -r-sr-x--- 15912 r > root shutdown 91155 0 /dev 38 crw------- ttyu0 rw > root shutdown 91155 1 /dev 38 crw------- ttyu0 rw > root shutdown 91155 2 /dev 38 crw------- ttyu0 rw > root sh 91129 root / 2 drwxr-xr-x 512 r > root sh 91129 wd / 2 drwxr-xr-x 512 r > root sh 91129 text / 44 -r-xr-xr-x 134848 r > root sh 91129 0 /dev 38 crw------- ttyu0 rw > root sh 91129 1* pipe ffffff01e78fc9e0 <-> ffffff01e78fc888 > 0 rw > root sh 91129 2 /dev 20 crw-rw-rw- null w > root shutdown 91115 root / 2 drwxr-xr-x 512 r > root shutdown 91115 wd /storage 5 drwx------ 37 r > root shutdown 91115 text / 47195 -r-sr-x--- 15912 r > root shutdown 91115 0 /dev 38 crw------- ttyu0 rw > root shutdown 91115 1 /dev 38 crw------- ttyu0 rw > root shutdown 91115 2 /dev 38 crw------- ttyu0 rw > root shutdown 91115 3* local dgram ffffff008ff92960 > root sh 90818 root / 2 drwxr-xr-x 512 r > root sh 90818 wd / 70659 drwxr-xr-x 512 r > root sh 90818 text / 44 -r-xr-xr-x 134848 r > root sh 90818 0 /dev 38 crw------- ttyu0 rw > root sh 90818 1* pipe ffffff0043f1ecb8 <-> ffffff0043f1eb60 > 0 rw > root sh 90818 2 /dev 20 crw-rw-rw- null w > root csh 90802 root / 2 drwxr-xr-x 512 r > root csh 90802 wd / 2 drwxr-xr-x 512 r > root csh 90802 text / 51 -r-xr-xr-x 358752 r > root csh 90802 15 /dev 38 crw------- ttyu0 rw > root csh 90802 16 /dev 38 crw------- ttyu0 rw > root csh 90802 17 /dev 38 crw------- ttyu0 rw > root csh 90802 18 /dev 38 crw------- ttyu0 rw > root csh 90802 19 /dev 38 crw------- ttyu0 rw > ------------------------------------------------------------------------ > > No indication of mbuf exhaustion, putting further focus on the pf stack: > > ------------------------------------------------------------------------ > netstat -m > > 2054/1786/3840 mbufs in use (current/cache/total) > 2048/1414/3462/25600 mbuf clusters in use (current/cache/total/max) > 2048/896 mbuf+clusters out of packet secondary zone in use (current/cache) > 0/320/320/12800 4k (page size) jumbo clusters in use > (current/cache/total/max) > 0/0/0/19200 9k jumbo clusters in use (current/cache/total/max) > 0/0/0/12800 16k jumbo clusters in use (current/cache/total/max) > 4609K/4554K/9164K bytes allocated to network (current/cache/total) > 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > 0/0/0 requests for jumbo clusters denied (4k/9k/16k) > 0 requests for sfbufs denied > 0 requests for sfbufs delayed > 0 requests for I/O initiated by sendfile > 0 calls to protocol drain routines > ------------------------------------------------------------------------ > > Here's one piece of core.0.txt which makes no sense to me -- the "rate" > column. I have a very hard time believing that was the interrupt rate > of all the relevant devices at the time (way too high). Maybe this data > becomes wrong only during a coredump? The total column I could believe. > > ------------------------------------------------------------------------ > vmstat -i > > interrupt total rate > irq4: uart0 54768 912 > irq6: fdc0 1 0 > irq17: uhci1+ 172 2 > irq23: uhci3 ehci1+ 2367 39 > cpu0: timer 13183882632 219731377 > irq256: em0 260491055 4341517 > irq257: em1 127555036 2125917 > irq258: ahci0 225923164 3765386 > cpu2: timer 13183881837 219731363 > cpu1: timer 13002196469 216703274 > cpu3: timer 13183881783 219731363 > Total 53167869284 886131154 > ------------------------------------------------------------------------ > > Here's what a normal "vmstat -i" shows from the command-line: > > # vmstat -i > interrupt total rate > irq4: uart0 518 0 > irq6: fdc0 1 0 > irq23: uhci3 ehci1+ 145 0 > cpu0: timer 19041199 1999 > irq256: em0 614280 64 > irq257: em1 168529 17 > irq258: ahci0 355536 37 > cpu2: timer 19040462 1999 > cpu1: timer 19040458 1999 > cpu3: timer 19040454 1999 > Total 77301582 8119 > > We graph many aspects of this box, including CPU load, memory/swap > usage, etc. and none show any sign that the interrupt rate on all of > those devices was even remotely out of control. (I would expect to see > CPU through the roof given the above data) > > I have since rebuilt/reinstalled world/kernel on the machine with the > latest RELENG_8 code (box is now 8.2-STABLE #0: Mon May 2 14:44:18 PDT > 2011), hoping whatever this was may have been fixed. > > As for what I think may have triggered it, but I have no hard evidence > of such: on April 29th, I changed our pf.conf and did "/etc/rc.d/pf > reload". The pf.conf change was a single line: > > Old: scrub on em0 all > New: scrub in on em0 all > > Why it took the problem approximately 3 days to start is unknown. It's > the only change we've made to the system (truly/honestly), and it was a > change to pf.conf. > > If anyone has advice (or has seen the above problem), or is interested > in debugging it -- as I said, I have a vmcore -- I'm happy to assist in > any way I can. I would hate for someone else to get bit by this, and > really am hoping its something that has been fixed between February and > now. > > I'm seeing this as well. You could change your scrub rules so that you specifically avoid TCP reassembly (that creates states). -- Good, fast & cheap. Pick any two. From owner-freebsd-pf@FreeBSD.ORG Tue May 3 05:58:42 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2486C106564A for ; Tue, 3 May 2011 05:58:42 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta09.emeryville.ca.mail.comcast.net (qmta09.emeryville.ca.mail.comcast.net [76.96.30.96]) by mx1.freebsd.org (Postfix) with ESMTP id 0B19E8FC08 for ; Tue, 3 May 2011 05:58:42 +0000 (UTC) Received: from omta08.emeryville.ca.mail.comcast.net ([76.96.30.12]) by qmta09.emeryville.ca.mail.comcast.net with comcast id etje1g0070FhH24A9tlXSX; Tue, 03 May 2011 05:45:31 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta08.emeryville.ca.mail.comcast.net with comcast id etlW1g00D1t3BNj8UtlWNj; Tue, 03 May 2011 05:45:31 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id EAD8F9B418; Mon, 2 May 2011 22:45:29 -0700 (PDT) Date: Mon, 2 May 2011 22:45:29 -0700 From: Jeremy Chadwick To: Jason Hellenthal Message-ID: <20110503054529.GA35688@icarus.home.lan> References: <20110503015854.GA31444@icarus.home.lan> <20110503050634.GB53414@DataIX.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20110503050634.GB53414@DataIX.net> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-stable@freebsd.org, freebsd-pf@freebsd.org Subject: Re: RELENG_8 pf stack issue (state count spiraling out of control) X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 05:58:42 -0000 On Tue, May 03, 2011 at 01:06:34AM -0400, Jason Hellenthal wrote: > On Mon, May 02, 2011 at 06:58:54PM -0700, Jeremy Chadwick wrote: > >(Please keep me CC'd as I'm not subscribed to freebsd-pf. And apologies > >for cross-posting, but the issue is severe enough that I wanted to make > >it known on -stable) > > > >The below issue I'm describing is from a machine running 8.2-PRERELEASE > >(RELENG_8) using src dated Tue Feb 15 05:46:02 PST 2011. > > > >Please read the story in full, as I have taken the time to describe > >everything I did, plus log output, as well as induce a panic via "call > >doadump" from ddb so I have a capture of the system at the time. I also > >have a theory as to what caused the problem, but how to trigger it is > >unknown; it may be a rare race condition. > > > > > >This morning I woke up to find a report from one of our users that he > >could not connect to a specific TCP port (not SSH) on one of our > >servers. I also found that I couldn't SSH into the same box. Serial > >console was working fine, and the serial console log showed no sign of > >any problems. > > > >I started to debug the issue of me not being able to SSH into the > >machine and within a few minutes became immediately concerned: pfctl > >indicated we had reached the maximum number permitted state table > >entries (10,000). > > > >============================================================ > ># pfctl -s info > >Status: Enabled for 76 days 06:49:10 Debug: Urgent > > > >Interface Stats for em0 IPv4 IPv6 > > Bytes In 8969748840 0 > > Bytes Out 8296135477 0 > > Packets In > > Passed 128211763 0 > > Blocked 621379 0 > > Packets Out > > Passed 138483868 0 > > Blocked 2579 0 > > > >State Table Total Rate > > current entries 10000 > > searches 267316807 40.6/s > > inserts 4440553 0.7/s > > removals 4430553 0.7/s > >Counters > > match 5067474 0.8/s > > bad-offset 0 0.0/s > > fragment 324 0.0/s > > short 0 0.0/s > > normalize 32 0.0/s > > memory 336946 0.1/s > > bad-timestamp 0 0.0/s > > congestion 0 0.0/s > > ip-option 0 0.0/s > > proto-cksum 1611 0.0/s > > state-mismatch 509 0.0/s > > state-insert 0 0.0/s > > state-limit 0 0.0/s > > src-limit 0 0.0/s > > synproxy 0 0.0/s > > > ># pfctl -s memory > >states hard limit 10000 > >src-nodes hard limit 10000 > >frags hard limit 5000 > >tables hard limit 1000 > >table-entries hard limit 100000 > >============================================================ > > > >The above is mainly for em0 (our WAN interface); our LAN interface (em1) > >was not impacted because we use "set skip on em1". And it's a good > >thing too: we have lots of LAN-based services that this machine provides > >that would have been impacted. We also use "set skip on lo0". > > > >I immediately went to look at our monitoring graphs, which monitor pf > >state (specifically state table entries), polled via bsnmpd(8). This > >data is even more frightening: > > > >http://jdc.parodius.com/freebsd/pf-issue/pf_states-day.png > >http://jdc.parodius.com/freebsd/pf-issue/pf_states-week.png > > > >Literally something was spiraling out of control, starting at approx. > >2011/05/01 (Sun) at 12:30 PDT. The situation became dire at approx. > >19:45 PDT the same day, but I wasn't aware of it until said user brought > >an issue to my attention. > > > >You can see from the network I/O graphs (taken from SNMP polling our > >switch, NOT from the host/box itself) that there was no DoS attack or > >anything like that occurring -- this was something within FreeBSD > >itself. More evidence of that will become apparent. > > > >http://jdc.parodius.com/freebsd/pf-issue/port_03-day.png > >http://jdc.parodius.com/freebsd/pf-issue/port_03-week.png > > > >The first thing I did was "/etc/rc.d/pf reload". This command hung. > >Any attempt to send Ctrl-C/SIGINT did nothing. I was able to > >Ctrl-Z/SIGSTOP it, then use kill %1, but the actual reload process did > >not truly die (despite csh stating "Terminated"). The only way to kill > >it was to kill -9. > > > >Attempts to shut down any daemons which utilised the network -- > >including things that only used em1 -- would not shut down. This > >included things like postfix, mysqld, and some inet-based services. I > >was forced to kill -9 them. Things like bsnmpd, however, did shut down. > > > >Equally as uncomfortable, "shutdown -r now" did not reboot the system. > >That is to say, wall(1)'s announcement was shown, but the actual > >stopping of services did not begin. > > > >The next thing I tried was "/etc/rc.d/pf stop", which worked. Then I > >did "/etc/rc.d/pf start", which also worked. However, what I saw next > >surely indicated a bug in the pf layer somewhere -- "pfctl -s states" > >and "pfctl -s info" disagreed on the state count: > > > >============================================================ > ># pfctl -s info > >Status: Enabled for 0 days 00:00:16 Debug: Urgent > > > >Interface Stats for em0 IPv4 IPv6 > > Bytes In 3459 0 > > Bytes Out 0 0 > > Packets In > > Passed 0 0 > > Blocked 29 0 > > Packets Out > > Passed 0 0 > > Blocked 0 0 > > > >State Table Total Rate > > current entries 10000 > > searches 29 1.8/s > > inserts 0 0.0/s > > removals 0 0.0/s > >Counters > > match 29 1.8/s > > bad-offset 0 0.0/s > > fragment 0 0.0/s > > short 0 0.0/s > > normalize 0 0.0/s > > memory 18 1.1/s > > bad-timestamp 0 0.0/s > > congestion 0 0.0/s > > ip-option 0 0.0/s > > proto-cksum 0 0.0/s > > state-mismatch 0 0.0/s > > state-insert 0 0.0/s > > state-limit 0 0.0/s > > src-limit 0 0.0/s > > synproxy 0 0.0/s > > > ># pfctl -s state | wc -l > > 0 > >============================================================ > > > >The "pf uptime" shown above, by the way, matches system uptime. > > > >I then attempted "pfctl -F state", but nothing changed (looked the same > >as above). > > > >Since I could not reboot the box, I was forced to drop to ddb via serial > >console. I did some commands like "ps" and the like, and then "call > >doadump" to induce a kernel panic, and then "reboot" (which worked). > > > >Once the machine came back up, savecore(8) ran, wrote the data out, and > >everything has been fine since. /var/crash/core.txt.0 is ~68KBytes and > >I do not feel comfortable sharing its content publicly, but will be > >happy to hand it to developer(s) who are interested. Relevant tidbits I > >can discern: > > > >------------------------------------------------------------------------ > >ps -axl > > > > UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND > > 0 422 0 0 -16 0 0 0 pftm DL ?? 1362773081:04.00 [pfpurge] > >------------------------------------------------------------------------ > > > >------------------------------------------------------------------------ > >vmstat -z > > > >ITEM SIZE LIMIT USED FREE REQUESTS FAILURES > >pfsrctrpl: 152, 10000, 0, 0, 0, 0 > >pfrulepl: 912, 0, 40, 88, 806, 0 > >pfstatepl: 392, 10000, 10000, 0, 4440553, 341638 > >pfaltqpl: 240, 0, 0, 0, 0, 0 > >pfpooladdrpl: 88, 0, 0, 0, 0, 0 > >pfrktable: 1296, 1002, 4, 20, 112, 0 > >pfrkentry: 216, 100008, 603, 891, 15384, 0 > >pfrkentry2: 216, 0, 0, 0, 0, 0 > >pffrent: 32, 5050, 0, 303, 1620, 0 > >pffrag: 80, 0, 0, 135, 807, 0 > >pffrcache: 80, 10035, 0, 0, 0, 0 > >pffrcent: 24, 50022, 0, 0, 0, 0 > >pfstatescrub: 40, 0, 0, 0, 0, 0 > >pfiaddrpl: 120, 0, 0, 0, 0, 0 > >pfospfen: 112, 0, 696, 30, 18096, 0 > >pfosfp: 40, 0, 407, 97, 10582, 0 > >------------------------------------------------------------------------ > > > >You can see evidence of processes not exiting/doing what they should do > >here: > > > >------------------------------------------------------------------------ > >fstat > > > >USER CMD PID FD MOUNT INUM MODE SZ|DV R/W > >root shutdown 91155 root / 2 drwxr-xr-x 512 r > >root shutdown 91155 wd / 2 drwxr-xr-x 512 r > >root shutdown 91155 text / 47195 -r-sr-x--- 15912 r > >root shutdown 91155 0 /dev 38 crw------- ttyu0 rw > >root shutdown 91155 1 /dev 38 crw------- ttyu0 rw > >root shutdown 91155 2 /dev 38 crw------- ttyu0 rw > >root sh 91129 root / 2 drwxr-xr-x 512 r > >root sh 91129 wd / 2 drwxr-xr-x 512 r > >root sh 91129 text / 44 -r-xr-xr-x 134848 r > >root sh 91129 0 /dev 38 crw------- ttyu0 rw > >root sh 91129 1* pipe ffffff01e78fc9e0 <-> ffffff01e78fc888 0 rw > >root sh 91129 2 /dev 20 crw-rw-rw- null w > >root shutdown 91115 root / 2 drwxr-xr-x 512 r > >root shutdown 91115 wd /storage 5 drwx------ 37 r > >root shutdown 91115 text / 47195 -r-sr-x--- 15912 r > >root shutdown 91115 0 /dev 38 crw------- ttyu0 rw > >root shutdown 91115 1 /dev 38 crw------- ttyu0 rw > >root shutdown 91115 2 /dev 38 crw------- ttyu0 rw > >root shutdown 91115 3* local dgram ffffff008ff92960 > >root sh 90818 root / 2 drwxr-xr-x 512 r > >root sh 90818 wd / 70659 drwxr-xr-x 512 r > >root sh 90818 text / 44 -r-xr-xr-x 134848 r > >root sh 90818 0 /dev 38 crw------- ttyu0 rw > >root sh 90818 1* pipe ffffff0043f1ecb8 <-> ffffff0043f1eb60 0 rw > >root sh 90818 2 /dev 20 crw-rw-rw- null w > >root csh 90802 root / 2 drwxr-xr-x 512 r > >root csh 90802 wd / 2 drwxr-xr-x 512 r > >root csh 90802 text / 51 -r-xr-xr-x 358752 r > >root csh 90802 15 /dev 38 crw------- ttyu0 rw > >root csh 90802 16 /dev 38 crw------- ttyu0 rw > >root csh 90802 17 /dev 38 crw------- ttyu0 rw > >root csh 90802 18 /dev 38 crw------- ttyu0 rw > >root csh 90802 19 /dev 38 crw------- ttyu0 rw > >------------------------------------------------------------------------ > > > >No indication of mbuf exhaustion, putting further focus on the pf stack: > > > >------------------------------------------------------------------------ > >netstat -m > > > >2054/1786/3840 mbufs in use (current/cache/total) > >2048/1414/3462/25600 mbuf clusters in use (current/cache/total/max) > >2048/896 mbuf+clusters out of packet secondary zone in use (current/cache) > >0/320/320/12800 4k (page size) jumbo clusters in use (current/cache/total/max) > >0/0/0/19200 9k jumbo clusters in use (current/cache/total/max) > >0/0/0/12800 16k jumbo clusters in use (current/cache/total/max) > >4609K/4554K/9164K bytes allocated to network (current/cache/total) > >0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > >0/0/0 requests for jumbo clusters denied (4k/9k/16k) > >0 requests for sfbufs denied > >0 requests for sfbufs delayed > >0 requests for I/O initiated by sendfile > >0 calls to protocol drain routines > >------------------------------------------------------------------------ > > > >Here's one piece of core.0.txt which makes no sense to me -- the "rate" > >column. I have a very hard time believing that was the interrupt rate > >of all the relevant devices at the time (way too high). Maybe this data > >becomes wrong only during a coredump? The total column I could believe. > > > >------------------------------------------------------------------------ > >vmstat -i > > > >interrupt total rate > >irq4: uart0 54768 912 > >irq6: fdc0 1 0 > >irq17: uhci1+ 172 2 > >irq23: uhci3 ehci1+ 2367 39 > >cpu0: timer 13183882632 219731377 > >irq256: em0 260491055 4341517 > >irq257: em1 127555036 2125917 > >irq258: ahci0 225923164 3765386 > >cpu2: timer 13183881837 219731363 > >cpu1: timer 13002196469 216703274 > >cpu3: timer 13183881783 219731363 > >Total 53167869284 886131154 > >------------------------------------------------------------------------ > > > >Here's what a normal "vmstat -i" shows from the command-line: > > > ># vmstat -i > >interrupt total rate > >irq4: uart0 518 0 > >irq6: fdc0 1 0 > >irq23: uhci3 ehci1+ 145 0 > >cpu0: timer 19041199 1999 > >irq256: em0 614280 64 > >irq257: em1 168529 17 > >irq258: ahci0 355536 37 > >cpu2: timer 19040462 1999 > >cpu1: timer 19040458 1999 > >cpu3: timer 19040454 1999 > >Total 77301582 8119 > > > >We graph many aspects of this box, including CPU load, memory/swap > >usage, etc. and none show any sign that the interrupt rate on all of > >those devices was even remotely out of control. (I would expect to see > >CPU through the roof given the above data) > > > >I have since rebuilt/reinstalled world/kernel on the machine with the > >latest RELENG_8 code (box is now 8.2-STABLE #0: Mon May 2 14:44:18 PDT > >2011), hoping whatever this was may have been fixed. > > > >As for what I think may have triggered it, but I have no hard evidence > >of such: on April 29th, I changed our pf.conf and did "/etc/rc.d/pf > >reload". The pf.conf change was a single line: > > > >Old: scrub on em0 all > >New: scrub in on em0 all > > > >Why it took the problem approximately 3 days to start is unknown. It's > >the only change we've made to the system (truly/honestly), and it was a > >change to pf.conf. > > > >If anyone has advice (or has seen the above problem), or is interested > >in debugging it -- as I said, I have a vmcore -- I'm happy to assist in > >any way I can. I would hate for someone else to get bit by this, and > >really am hoping its something that has been fixed between February and > >now. > > > > That's quite the deduction there. I've noticed recently that you were also > experimenting with the new NFS server recompiling kernel etc etc. Seeing > as weird things can happen with DNS, NFS and mountpoint's, is this the same > machine that you were doing that on ? Completely different machine. The machine which exhibited the problem does mount a single NFS filesystem from the other box you're referring to, but it does not use NFSv4. Most importantly: the NFS mounts are also done across the em1 interface exclusively, which as mentioned falls under "set skip". I realise my analysis results in a very ballsy claim, but the graphs are quite damning, as is pfctl output and machine behaviour. I wish I knew why the machine couldn't be rebooted cleanly / why processes were "sort of" wedging. That seems to indicate something weird going on in the kernel or the pf stack; not sure which. > If so can you check to see how many requests for NFS operations were done > to/from that box ? as well the names that would be being resolved and if > that machine can resolve them ? Each machine on our network runs named. Each named instance slaves a copy of three zones from the masters: reverse in-addr.arpa for the LAN, reverse in-addr.arpa for the WAN, and the forward zone associated with the domain name (parodius.com). So, literally every machine on the network avoids having to go out and ask the Internet for lookups under our domain, or our IP blocks (public or private). Our resolv.conf is tuned appropriately for this too ("search parodius.com private.network" and a nameserver of 127.0.0.1). The slaving does happen over the public interface for a lot of reasons I'd rather not get into here. Regarding NFS statistics -- no NFS operations were occurring during the 24 hour window where pf's state count went through the roof, except for at approximately 04:00 PDT when our automated backups ran. The issue did not start then as indicated by my analysis. Regardless, below are the NFS statistics which were provided by "call doadump" and are in core.0.txt: ------------------------------------------------------------------------ nfsstat Client Info: Rpc Counts: Getattr Setattr Lookup Readlink Read Write Create Remove 172 0 33 0 0 0 0 0 Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access 0 0 0 0 0 3 0 3 Mknod Fsstat Fsinfo PathConf Commit 0 56 1 0 0 Rpc Info: TimedOut Invalid X Replies Retries Requests 0 0 0 0 268 Cache Info: Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW Hits Misses 95 172 26 33 0 0 0 0 BioRLHits Misses BioD Hits Misses DirE Hits Misses 0 0 3 3 3 0 Server Info: Getattr Setattr Lookup Readlink Read Write Create Remove 0 0 0 0 0 0 0 0 Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access 0 0 0 0 0 0 0 0 Mknod Fsstat Fsinfo PathConf Commit 0 0 0 0 0 Server Ret-Failed 0 Server Faults 0 Server Cache Stats: Inprog Idem Non-idem Misses 0 0 0 0 Server Write Gathering: WriteOps WriteRPC Opsaved 0 0 0 ------------------------------------------------------------------------ These numbers should act as confirmation that the problem is almost certainly not NFS-related; these numbers are very small. For fun, here's the "netstat -s" output of the core. ------------------------------------------------------------------------ netstat -s tcp: 70963757 packets sent 56716136 data packets (93106798714 bytes) 20493 data packets (14888440 bytes) retransmitted 3258 data packets unnecessarily retransmitted 0 resends initiated by MTU discovery 7640376 ack-only packets (433685 delayed) 0 URG only packets 0 window probe packets 3629755 window update packets 2956596 control packets 109589225 packets received 81346171 acks (for 93113064503 bytes) 2970456 duplicate acks 0 acks for unsent data 56067087 packets (8649172006 bytes) received in-sequence 5950 completely duplicate packets (5254577 bytes) 26 old duplicate packets 152 packets with some dup. data (35886 bytes duped) 7412 out-of-order packets (10192893 bytes) 0 packets (0 bytes) of data after window 0 window probes 10001480 window update packets 16 packets received after close 1561 discarded for bad checksums 0 discarded for bad header offset fields 0 discarded because packet too short 6 discarded due to memory problems 2159 connection requests 2954080 connection accepts 0 bad connection attempts 0 listen queue overflows 491 ignored RSTs in the windows 2955571 connections established (including accepts) 3066783 connections closed (including 1401 drops) 2952708 connections updated cached RTT on close 2952890 connections updated cached RTT variance on close 2928379 connections updated cached ssthresh on close 220 embryonic connections dropped 81149655 segments updated rtt (of 58223063 attempts) 17684 retransmit timeouts 169 connections dropped by rexmit timeout 0 persist timeouts 0 connections dropped by persist timeout 0 Connections (fin_wait_2) dropped because of timeout 76 keepalive timeouts 76 keepalive probes sent 0 connections dropped by keepalive 222190 correct ACK header predictions 6584196 correct data packet header predictions 2955062 syncache entries added 603 retransmitted 188 dupsyn 0 dropped 2954080 completed 0 bucket overflow 0 cache overflow 869 reset 113 stale 0 aborted 0 badack 0 unreach 0 zone failures 2955062 cookies sent 0 cookies received 126 SACK recovery episodes 78 segment rexmits in SACK recovery episodes 78254 byte rexmits in SACK recovery episodes 2962 SACK options (SACK blocks) received 8870 SACK options (SACK blocks) sent 0 SACK scoreboard overflow 0 packets with ECN CE bit set 0 packets with ECN ECT(0) bit set 0 packets with ECN ECT(1) bit set 0 successful ECN handshakes 0 times ECN reduced the congestion window udp: 6218432 datagrams received 0 with incomplete header 0 with bad data length field 29 with bad checksum 15493 with no checksum 4896 dropped due to no socket 0 broadcast/multicast datagrams undelivered 0 dropped due to full socket buffers 0 not for hashed pcb 6213507 delivered 6405573 datagrams output 0 times multicast source filter matched ip: 236156523 total packets received 0 bad header checksums 0 with size smaller than minimum 0 with data size < data length 0 with ip length > max ip packet size 0 with header length < data size 0 with data length < header length 0 with bad options 0 with incorrect version number 0 fragments received 0 fragments dropped (dup or out of space) 0 fragments dropped after timeout 0 packets reassembled ok 235463206 packets for this host 66789 packets for unknown/unsupported protocol 0 packets forwarded (0 packets fast forwarded) 0 packets not forwardable 0 packets received for unknown multicast group 0 redirects sent 81178132 packets sent from this host 126684324 packets sent with fabricated ip header 0 output packets dropped due to no bufs, etc. 0 output packets discarded due to no route 0 output datagrams fragmented 0 fragments created 0 datagrams that can't be fragmented 0 tunneling packets that can't find gif 0 datagrams with bad address in header icmp: 4896 calls to icmp_error 0 errors not generated in response to an icmp message Output histogram: echo reply: 3801761 destination unreachable: 4896 0 messages with bad code fields 0 messages less than the minimum length 21 messages with bad checksum 0 messages with bad length 0 multicast echo requests ignored 0 multicast timestamp requests ignored Input histogram: echo reply: 7552810 destination unreachable: 58965 source quench: 10 routing redirect: 75 echo: 3801761 time exceeded: 108309026 3801761 message responses generated 0 invalid return addresses 0 no return routes igmp: 0 messages received 0 messages received with too few bytes 0 messages received with wrong TTL 0 messages received with bad checksum 0 V1/V2 membership queries received 0 V3 membership queries received 0 membership queries received with invalid field(s) 0 general queries received 0 group queries received 0 group-source queries received 0 group-source queries dropped 0 membership reports received 0 membership reports received with invalid field(s) 0 membership reports received for groups to which we belong 0 V3 reports received without Router Alert 0 membership reports sent arp: 5537 ARP requests sent 26179 ARP replies sent 1794911 ARP requests received 5531 ARP replies received 1800442 ARP packets received 18 total packets dropped due to no ARP entry 9834 ARP entrys timed out 0 Duplicate IPs seen ------------------------------------------------------------------------ > Also I would believe your using tables in your pf.conf, if so do any of > those tables contain a FQDN that cannot be resolved from that machine ? > > I think you probably see what I am getting at here as it could be some > sort of concurrent recursive DNS failure that can only be seen from the > machine caused by possibly the new NFS backend or a change in one of the > tables that pf would use. I absolutely see where you're going with this, but as I understand it, DNS resolution on pf.conf rules happens during pf load-time (though I have no idea how TTL would be honoured). However/regardless: There are no FQDNs or hostnames used in the pf.conf on the machine which exhibited the above problem; everything uses hard-coded IP addresses. The same goes for the contents of the (extremely few; only 4) tables referenced in pf.conf. The only non-numeric data used for resolution of any kind are a couple service names (e.g. "ssh" instead of 22). The pf.conf on the machine which witnessed the issue is extremely small and very simple. It's also the 2nd-to-lowest machine on our network when it comes to amount of network traffic received/sent. How can I get a dump of the contents of the pf state table -- specifically referring to whatever it thinks still contained 10,000 state entries despite "pfctl -s state" showing 0? There is obviously a mismatch there, so I'm wondering how the 10,000 counter could stay at 10,000 (especially after doing a full stop/start of the pf stack) when "pfctl -s state" shows 0 (ditto with "pfctl -F state"). I really, *really* want to debug this. It's to the point where I'm happily willing to pay for a kernel developer's time, hourly, to figure it out (if at all possible). It might be easier to figure it out in real-time rather than post-mortem, but post-mortem is what I've got right now. I need to get off my butt and get an actual dev/testbed box in our production co-lo where I can bang on this stuff for days and see if I can reproduce it. All I can conclude on my own is that "something bad" can happen when reloading pf rules + changing "scrub" from both directions to just incoming. I don't even know if the scrub change is what triggered it, maybe just the reload did. Maybe the problem can only happen when a packet of a certain type is inbound or outbound and in a certain internal state. Again, it's speculation, but there's some pretty strong evidence that something very bad happened, almost like an infinite loop that eventually exhausted all stateful memory/entries. A DoS across em0 is what it sounds like, but the network graphs on the switch confirm that isn't the case. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-pf@FreeBSD.ORG Tue May 3 06:01:09 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id ED01D1065672 for ; Tue, 3 May 2011 06:01:09 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta12.emeryville.ca.mail.comcast.net (qmta12.emeryville.ca.mail.comcast.net [76.96.27.227]) by mx1.freebsd.org (Postfix) with ESMTP id D38F88FC0A for ; Tue, 3 May 2011 06:01:09 +0000 (UTC) Received: from omta20.emeryville.ca.mail.comcast.net ([76.96.30.87]) by qmta12.emeryville.ca.mail.comcast.net with comcast id ettN1g0031smiN4ACu19E1; Tue, 03 May 2011 06:01:09 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta20.emeryville.ca.mail.comcast.net with comcast id eu161g0071t3BNj8gu18Qm; Tue, 03 May 2011 06:01:08 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 6662D9B418; Mon, 2 May 2011 23:01:06 -0700 (PDT) Date: Mon, 2 May 2011 23:01:06 -0700 From: Jeremy Chadwick To: Vlad Galu Message-ID: <20110503060106.GA36331@icarus.home.lan> References: <20110503015854.GA31444@icarus.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-stable@freebsd.org, freebsd-pf@freebsd.org Subject: Re: RELENG_8 pf stack issue (state count spiraling out of control) X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 06:01:10 -0000 On Tue, May 03, 2011 at 07:22:10AM +0200, Vlad Galu wrote: > On Tue, May 3, 2011 at 3:58 AM, Jeremy Chadwick wrote: > > > (Please keep me CC'd as I'm not subscribed to freebsd-pf. And apologies > > for cross-posting, but the issue is severe enough that I wanted to make > > it known on -stable) > > > > The below issue I'm describing is from a machine running 8.2-PRERELEASE > > (RELENG_8) using src dated Tue Feb 15 05:46:02 PST 2011. > > > > Please read the story in full, as I have taken the time to describe > > everything I did, plus log output, as well as induce a panic via "call > > doadump" from ddb so I have a capture of the system at the time. I also > > have a theory as to what caused the problem, but how to trigger it is > > unknown; it may be a rare race condition. > > > > > > This morning I woke up to find a report from one of our users that he > > could not connect to a specific TCP port (not SSH) on one of our > > servers. I also found that I couldn't SSH into the same box. Serial > > console was working fine, and the serial console log showed no sign of > > any problems. > > > > I started to debug the issue of me not being able to SSH into the > > machine and within a few minutes became immediately concerned: pfctl > > indicated we had reached the maximum number permitted state table > > entries (10,000). > > > > ============================================================ > > # pfctl -s info > > Status: Enabled for 76 days 06:49:10 Debug: Urgent > > > > Interface Stats for em0 IPv4 IPv6 > > Bytes In 8969748840 0 > > Bytes Out 8296135477 0 > > Packets In > > Passed 128211763 0 > > Blocked 621379 0 > > Packets Out > > Passed 138483868 0 > > Blocked 2579 0 > > > > State Table Total Rate > > current entries 10000 > > searches 267316807 40.6/s > > inserts 4440553 0.7/s > > removals 4430553 0.7/s > > Counters > > match 5067474 0.8/s > > bad-offset 0 0.0/s > > fragment 324 0.0/s > > short 0 0.0/s > > normalize 32 0.0/s > > memory 336946 0.1/s > > bad-timestamp 0 0.0/s > > congestion 0 0.0/s > > ip-option 0 0.0/s > > proto-cksum 1611 0.0/s > > state-mismatch 509 0.0/s > > state-insert 0 0.0/s > > state-limit 0 0.0/s > > src-limit 0 0.0/s > > synproxy 0 0.0/s > > > > # pfctl -s memory > > states hard limit 10000 > > src-nodes hard limit 10000 > > frags hard limit 5000 > > tables hard limit 1000 > > table-entries hard limit 100000 > > ============================================================ > > > > The above is mainly for em0 (our WAN interface); our LAN interface (em1) > > was not impacted because we use "set skip on em1". And it's a good > > thing too: we have lots of LAN-based services that this machine provides > > that would have been impacted. We also use "set skip on lo0". > > > > I immediately went to look at our monitoring graphs, which monitor pf > > state (specifically state table entries), polled via bsnmpd(8). This > > data is even more frightening: > > > > http://jdc.parodius.com/freebsd/pf-issue/pf_states-day.png > > http://jdc.parodius.com/freebsd/pf-issue/pf_states-week.png > > > > Literally something was spiraling out of control, starting at approx. > > 2011/05/01 (Sun) at 12:30 PDT. The situation became dire at approx. > > 19:45 PDT the same day, but I wasn't aware of it until said user brought > > an issue to my attention. > > > > You can see from the network I/O graphs (taken from SNMP polling our > > switch, NOT from the host/box itself) that there was no DoS attack or > > anything like that occurring -- this was something within FreeBSD > > itself. More evidence of that will become apparent. > > > > http://jdc.parodius.com/freebsd/pf-issue/port_03-day.png > > http://jdc.parodius.com/freebsd/pf-issue/port_03-week.png > > > > The first thing I did was "/etc/rc.d/pf reload". This command hung. > > Any attempt to send Ctrl-C/SIGINT did nothing. I was able to > > Ctrl-Z/SIGSTOP it, then use kill %1, but the actual reload process did > > not truly die (despite csh stating "Terminated"). The only way to kill > > it was to kill -9. > > > > Attempts to shut down any daemons which utilised the network -- > > including things that only used em1 -- would not shut down. This > > included things like postfix, mysqld, and some inet-based services. I > > was forced to kill -9 them. Things like bsnmpd, however, did shut down. > > > > Equally as uncomfortable, "shutdown -r now" did not reboot the system. > > That is to say, wall(1)'s announcement was shown, but the actual > > stopping of services did not begin. > > > > The next thing I tried was "/etc/rc.d/pf stop", which worked. Then I > > did "/etc/rc.d/pf start", which also worked. However, what I saw next > > surely indicated a bug in the pf layer somewhere -- "pfctl -s states" > > and "pfctl -s info" disagreed on the state count: > > > > ============================================================ > > # pfctl -s info > > Status: Enabled for 0 days 00:00:16 Debug: Urgent > > > > Interface Stats for em0 IPv4 IPv6 > > Bytes In 3459 0 > > Bytes Out 0 0 > > Packets In > > Passed 0 0 > > Blocked 29 0 > > Packets Out > > Passed 0 0 > > Blocked 0 0 > > > > State Table Total Rate > > current entries 10000 > > searches 29 1.8/s > > inserts 0 0.0/s > > removals 0 0.0/s > > Counters > > match 29 1.8/s > > bad-offset 0 0.0/s > > fragment 0 0.0/s > > short 0 0.0/s > > normalize 0 0.0/s > > memory 18 1.1/s > > bad-timestamp 0 0.0/s > > congestion 0 0.0/s > > ip-option 0 0.0/s > > proto-cksum 0 0.0/s > > state-mismatch 0 0.0/s > > state-insert 0 0.0/s > > state-limit 0 0.0/s > > src-limit 0 0.0/s > > synproxy 0 0.0/s > > > > # pfctl -s state | wc -l > > 0 > > ============================================================ > > > > The "pf uptime" shown above, by the way, matches system uptime. > > > > I then attempted "pfctl -F state", but nothing changed (looked the same > > as above). > > > > Since I could not reboot the box, I was forced to drop to ddb via serial > > console. I did some commands like "ps" and the like, and then "call > > doadump" to induce a kernel panic, and then "reboot" (which worked). > > > > Once the machine came back up, savecore(8) ran, wrote the data out, and > > everything has been fine since. /var/crash/core.txt.0 is ~68KBytes and > > I do not feel comfortable sharing its content publicly, but will be > > happy to hand it to developer(s) who are interested. Relevant tidbits I > > can discern: > > > > ------------------------------------------------------------------------ > > ps -axl > > > > UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND > > 0 422 0 0 -16 0 0 0 pftm DL ?? 1362773081:04.00 > > [pfpurge] > > ------------------------------------------------------------------------ > > > > ------------------------------------------------------------------------ > > vmstat -z > > > > ITEM SIZE LIMIT USED FREE REQUESTS > > FAILURES > > pfsrctrpl: 152, 10000, 0, 0, 0, > > 0 > > pfrulepl: 912, 0, 40, 88, 806, > > 0 > > pfstatepl: 392, 10000, 10000, 0, 4440553, > > 341638 > > pfaltqpl: 240, 0, 0, 0, 0, > > 0 > > pfpooladdrpl: 88, 0, 0, 0, 0, > > 0 > > pfrktable: 1296, 1002, 4, 20, 112, > > 0 > > pfrkentry: 216, 100008, 603, 891, 15384, > > 0 > > pfrkentry2: 216, 0, 0, 0, 0, > > 0 > > pffrent: 32, 5050, 0, 303, 1620, > > 0 > > pffrag: 80, 0, 0, 135, 807, > > 0 > > pffrcache: 80, 10035, 0, 0, 0, > > 0 > > pffrcent: 24, 50022, 0, 0, 0, > > 0 > > pfstatescrub: 40, 0, 0, 0, 0, > > 0 > > pfiaddrpl: 120, 0, 0, 0, 0, > > 0 > > pfospfen: 112, 0, 696, 30, 18096, > > 0 > > pfosfp: 40, 0, 407, 97, 10582, > > 0 > > ------------------------------------------------------------------------ > > > > You can see evidence of processes not exiting/doing what they should do > > here: > > > > ------------------------------------------------------------------------ > > fstat > > > > USER CMD PID FD MOUNT INUM MODE SZ|DV R/W > > root shutdown 91155 root / 2 drwxr-xr-x 512 r > > root shutdown 91155 wd / 2 drwxr-xr-x 512 r > > root shutdown 91155 text / 47195 -r-sr-x--- 15912 r > > root shutdown 91155 0 /dev 38 crw------- ttyu0 rw > > root shutdown 91155 1 /dev 38 crw------- ttyu0 rw > > root shutdown 91155 2 /dev 38 crw------- ttyu0 rw > > root sh 91129 root / 2 drwxr-xr-x 512 r > > root sh 91129 wd / 2 drwxr-xr-x 512 r > > root sh 91129 text / 44 -r-xr-xr-x 134848 r > > root sh 91129 0 /dev 38 crw------- ttyu0 rw > > root sh 91129 1* pipe ffffff01e78fc9e0 <-> ffffff01e78fc888 > > 0 rw > > root sh 91129 2 /dev 20 crw-rw-rw- null w > > root shutdown 91115 root / 2 drwxr-xr-x 512 r > > root shutdown 91115 wd /storage 5 drwx------ 37 r > > root shutdown 91115 text / 47195 -r-sr-x--- 15912 r > > root shutdown 91115 0 /dev 38 crw------- ttyu0 rw > > root shutdown 91115 1 /dev 38 crw------- ttyu0 rw > > root shutdown 91115 2 /dev 38 crw------- ttyu0 rw > > root shutdown 91115 3* local dgram ffffff008ff92960 > > root sh 90818 root / 2 drwxr-xr-x 512 r > > root sh 90818 wd / 70659 drwxr-xr-x 512 r > > root sh 90818 text / 44 -r-xr-xr-x 134848 r > > root sh 90818 0 /dev 38 crw------- ttyu0 rw > > root sh 90818 1* pipe ffffff0043f1ecb8 <-> ffffff0043f1eb60 > > 0 rw > > root sh 90818 2 /dev 20 crw-rw-rw- null w > > root csh 90802 root / 2 drwxr-xr-x 512 r > > root csh 90802 wd / 2 drwxr-xr-x 512 r > > root csh 90802 text / 51 -r-xr-xr-x 358752 r > > root csh 90802 15 /dev 38 crw------- ttyu0 rw > > root csh 90802 16 /dev 38 crw------- ttyu0 rw > > root csh 90802 17 /dev 38 crw------- ttyu0 rw > > root csh 90802 18 /dev 38 crw------- ttyu0 rw > > root csh 90802 19 /dev 38 crw------- ttyu0 rw > > ------------------------------------------------------------------------ > > > > No indication of mbuf exhaustion, putting further focus on the pf stack: > > > > ------------------------------------------------------------------------ > > netstat -m > > > > 2054/1786/3840 mbufs in use (current/cache/total) > > 2048/1414/3462/25600 mbuf clusters in use (current/cache/total/max) > > 2048/896 mbuf+clusters out of packet secondary zone in use (current/cache) > > 0/320/320/12800 4k (page size) jumbo clusters in use > > (current/cache/total/max) > > 0/0/0/19200 9k jumbo clusters in use (current/cache/total/max) > > 0/0/0/12800 16k jumbo clusters in use (current/cache/total/max) > > 4609K/4554K/9164K bytes allocated to network (current/cache/total) > > 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > > 0/0/0 requests for jumbo clusters denied (4k/9k/16k) > > 0 requests for sfbufs denied > > 0 requests for sfbufs delayed > > 0 requests for I/O initiated by sendfile > > 0 calls to protocol drain routines > > ------------------------------------------------------------------------ > > > > Here's one piece of core.0.txt which makes no sense to me -- the "rate" > > column. I have a very hard time believing that was the interrupt rate > > of all the relevant devices at the time (way too high). Maybe this data > > becomes wrong only during a coredump? The total column I could believe. > > > > ------------------------------------------------------------------------ > > vmstat -i > > > > interrupt total rate > > irq4: uart0 54768 912 > > irq6: fdc0 1 0 > > irq17: uhci1+ 172 2 > > irq23: uhci3 ehci1+ 2367 39 > > cpu0: timer 13183882632 219731377 > > irq256: em0 260491055 4341517 > > irq257: em1 127555036 2125917 > > irq258: ahci0 225923164 3765386 > > cpu2: timer 13183881837 219731363 > > cpu1: timer 13002196469 216703274 > > cpu3: timer 13183881783 219731363 > > Total 53167869284 886131154 > > ------------------------------------------------------------------------ > > > > Here's what a normal "vmstat -i" shows from the command-line: > > > > # vmstat -i > > interrupt total rate > > irq4: uart0 518 0 > > irq6: fdc0 1 0 > > irq23: uhci3 ehci1+ 145 0 > > cpu0: timer 19041199 1999 > > irq256: em0 614280 64 > > irq257: em1 168529 17 > > irq258: ahci0 355536 37 > > cpu2: timer 19040462 1999 > > cpu1: timer 19040458 1999 > > cpu3: timer 19040454 1999 > > Total 77301582 8119 > > > > We graph many aspects of this box, including CPU load, memory/swap > > usage, etc. and none show any sign that the interrupt rate on all of > > those devices was even remotely out of control. (I would expect to see > > CPU through the roof given the above data) > > > > I have since rebuilt/reinstalled world/kernel on the machine with the > > latest RELENG_8 code (box is now 8.2-STABLE #0: Mon May 2 14:44:18 PDT > > 2011), hoping whatever this was may have been fixed. > > > > As for what I think may have triggered it, but I have no hard evidence > > of such: on April 29th, I changed our pf.conf and did "/etc/rc.d/pf > > reload". The pf.conf change was a single line: > > > > Old: scrub on em0 all > > New: scrub in on em0 all > > > > Why it took the problem approximately 3 days to start is unknown. It's > > the only change we've made to the system (truly/honestly), and it was a > > change to pf.conf. > > > > If anyone has advice (or has seen the above problem), or is interested > > in debugging it -- as I said, I have a vmcore -- I'm happy to assist in > > any way I can. I would hate for someone else to get bit by this, and > > really am hoping its something that has been fixed between February and > > now. > > > > > I'm seeing this as well. You could change your scrub rules so that you > specifically avoid TCP reassembly (that creates states). Thank you very much. This helps -- and I'm also glad someone else has seen this behaviour. That confirms it's not specific to my equipment, which is good. Regarding scrubbing and TCP reassembly (option "reassemble tcp" according to pf.conf): I wasn't under the impression this option was enabled by default. This got me wondering what the defaults actually are (pf.conf(5) is somewhat vague in this regard, but it does state that "fragment reassemble" is enabled by default). The rule I use: scrub in on em0 all Appears to get evaluated into this (per "pfctl -s rules -v"): scrub in on em0 all fragment reassemble Did you mean to tell me to disable the "fragment reassemble" option (which is different from "reassemble tcp")? If so, how do I do that? It looks like I could use a "no scrub" rule, except I can't find any examples of this on the web or in the docs. Or is it just better to remove use of scrub entirely until whatever this is gets fixed? -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-pf@FreeBSD.ORG Tue May 3 06:10:02 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 85BF5106566B; Tue, 3 May 2011 06:10:02 +0000 (UTC) (envelope-from dudu@dudu.ro) Received: from mail-qw0-f54.google.com (mail-qw0-f54.google.com [209.85.216.54]) by mx1.freebsd.org (Postfix) with ESMTP id F24458FC15; Tue, 3 May 2011 06:10:01 +0000 (UTC) Received: by qwc9 with SMTP id 9so3744649qwc.13 for ; Mon, 02 May 2011 23:10:01 -0700 (PDT) Received: by 10.229.43.209 with SMTP id x17mr6770395qce.257.1304403001141; Mon, 02 May 2011 23:10:01 -0700 (PDT) MIME-Version: 1.0 Received: by 10.229.105.18 with HTTP; Mon, 2 May 2011 23:09:21 -0700 (PDT) In-Reply-To: <20110503060106.GA36331@icarus.home.lan> References: <20110503015854.GA31444@icarus.home.lan> <20110503060106.GA36331@icarus.home.lan> From: Vlad Galu Date: Tue, 3 May 2011 08:09:21 +0200 Message-ID: To: Jeremy Chadwick Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-stable@freebsd.org, freebsd-pf@freebsd.org Subject: Re: RELENG_8 pf stack issue (state count spiraling out of control) X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 06:10:02 -0000 On Tue, May 3, 2011 at 8:01 AM, Jeremy Chadwick wrote: > On Tue, May 03, 2011 at 07:22:10AM +0200, Vlad Galu wrote: > > On Tue, May 3, 2011 at 3:58 AM, Jeremy Chadwick < > freebsd@jdc.parodius.com>wrote: > > > > > (Please keep me CC'd as I'm not subscribed to freebsd-pf. And > apologies > > > for cross-posting, but the issue is severe enough that I wanted to make > > > it known on -stable) > > > > > > The below issue I'm describing is from a machine running 8.2-PRERELEASE > > > (RELENG_8) using src dated Tue Feb 15 05:46:02 PST 2011. > > > > > > Please read the story in full, as I have taken the time to describe > > > everything I did, plus log output, as well as induce a panic via "call > > > doadump" from ddb so I have a capture of the system at the time. I > also > > > have a theory as to what caused the problem, but how to trigger it is > > > unknown; it may be a rare race condition. > > > > > > > > > This morning I woke up to find a report from one of our users that he > > > could not connect to a specific TCP port (not SSH) on one of our > > > servers. I also found that I couldn't SSH into the same box. Serial > > > console was working fine, and the serial console log showed no sign of > > > any problems. > > > > > > I started to debug the issue of me not being able to SSH into the > > > machine and within a few minutes became immediately concerned: pfctl > > > indicated we had reached the maximum number permitted state table > > > entries (10,000). > > > > > > ============================================================ > > > # pfctl -s info > > > Status: Enabled for 76 days 06:49:10 Debug: Urgent > > > > > > Interface Stats for em0 IPv4 IPv6 > > > Bytes In 8969748840 0 > > > Bytes Out 8296135477 0 > > > Packets In > > > Passed 128211763 0 > > > Blocked 621379 0 > > > Packets Out > > > Passed 138483868 0 > > > Blocked 2579 0 > > > > > > State Table Total Rate > > > current entries 10000 > > > searches 267316807 40.6/s > > > inserts 4440553 0.7/s > > > removals 4430553 0.7/s > > > Counters > > > match 5067474 0.8/s > > > bad-offset 0 0.0/s > > > fragment 324 0.0/s > > > short 0 0.0/s > > > normalize 32 0.0/s > > > memory 336946 0.1/s > > > bad-timestamp 0 0.0/s > > > congestion 0 0.0/s > > > ip-option 0 0.0/s > > > proto-cksum 1611 0.0/s > > > state-mismatch 509 0.0/s > > > state-insert 0 0.0/s > > > state-limit 0 0.0/s > > > src-limit 0 0.0/s > > > synproxy 0 0.0/s > > > > > > # pfctl -s memory > > > states hard limit 10000 > > > src-nodes hard limit 10000 > > > frags hard limit 5000 > > > tables hard limit 1000 > > > table-entries hard limit 100000 > > > ============================================================ > > > > > > The above is mainly for em0 (our WAN interface); our LAN interface > (em1) > > > was not impacted because we use "set skip on em1". And it's a good > > > thing too: we have lots of LAN-based services that this machine > provides > > > that would have been impacted. We also use "set skip on lo0". > > > > > > I immediately went to look at our monitoring graphs, which monitor pf > > > state (specifically state table entries), polled via bsnmpd(8). This > > > data is even more frightening: > > > > > > http://jdc.parodius.com/freebsd/pf-issue/pf_states-day.png > > > http://jdc.parodius.com/freebsd/pf-issue/pf_states-week.png > > > > > > Literally something was spiraling out of control, starting at approx. > > > 2011/05/01 (Sun) at 12:30 PDT. The situation became dire at approx. > > > 19:45 PDT the same day, but I wasn't aware of it until said user > brought > > > an issue to my attention. > > > > > > You can see from the network I/O graphs (taken from SNMP polling our > > > switch, NOT from the host/box itself) that there was no DoS attack or > > > anything like that occurring -- this was something within FreeBSD > > > itself. More evidence of that will become apparent. > > > > > > http://jdc.parodius.com/freebsd/pf-issue/port_03-day.png > > > http://jdc.parodius.com/freebsd/pf-issue/port_03-week.png > > > > > > The first thing I did was "/etc/rc.d/pf reload". This command hung. > > > Any attempt to send Ctrl-C/SIGINT did nothing. I was able to > > > Ctrl-Z/SIGSTOP it, then use kill %1, but the actual reload process did > > > not truly die (despite csh stating "Terminated"). The only way to kill > > > it was to kill -9. > > > > > > Attempts to shut down any daemons which utilised the network -- > > > including things that only used em1 -- would not shut down. This > > > included things like postfix, mysqld, and some inet-based services. I > > > was forced to kill -9 them. Things like bsnmpd, however, did shut > down. > > > > > > Equally as uncomfortable, "shutdown -r now" did not reboot the system. > > > That is to say, wall(1)'s announcement was shown, but the actual > > > stopping of services did not begin. > > > > > > The next thing I tried was "/etc/rc.d/pf stop", which worked. Then I > > > did "/etc/rc.d/pf start", which also worked. However, what I saw next > > > surely indicated a bug in the pf layer somewhere -- "pfctl -s states" > > > and "pfctl -s info" disagreed on the state count: > > > > > > ============================================================ > > > # pfctl -s info > > > Status: Enabled for 0 days 00:00:16 Debug: Urgent > > > > > > Interface Stats for em0 IPv4 IPv6 > > > Bytes In 3459 0 > > > Bytes Out 0 0 > > > Packets In > > > Passed 0 0 > > > Blocked 29 0 > > > Packets Out > > > Passed 0 0 > > > Blocked 0 0 > > > > > > State Table Total Rate > > > current entries 10000 > > > searches 29 1.8/s > > > inserts 0 0.0/s > > > removals 0 0.0/s > > > Counters > > > match 29 1.8/s > > > bad-offset 0 0.0/s > > > fragment 0 0.0/s > > > short 0 0.0/s > > > normalize 0 0.0/s > > > memory 18 1.1/s > > > bad-timestamp 0 0.0/s > > > congestion 0 0.0/s > > > ip-option 0 0.0/s > > > proto-cksum 0 0.0/s > > > state-mismatch 0 0.0/s > > > state-insert 0 0.0/s > > > state-limit 0 0.0/s > > > src-limit 0 0.0/s > > > synproxy 0 0.0/s > > > > > > # pfctl -s state | wc -l > > > 0 > > > ============================================================ > > > > > > The "pf uptime" shown above, by the way, matches system uptime. > > > > > > I then attempted "pfctl -F state", but nothing changed (looked the same > > > as above). > > > > > > Since I could not reboot the box, I was forced to drop to ddb via > serial > > > console. I did some commands like "ps" and the like, and then "call > > > doadump" to induce a kernel panic, and then "reboot" (which worked). > > > > > > Once the machine came back up, savecore(8) ran, wrote the data out, and > > > everything has been fine since. /var/crash/core.txt.0 is ~68KBytes and > > > I do not feel comfortable sharing its content publicly, but will be > > > happy to hand it to developer(s) who are interested. Relevant tidbits > I > > > can discern: > > > > > > > ------------------------------------------------------------------------ > > > ps -axl > > > > > > UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME > COMMAND > > > 0 422 0 0 -16 0 0 0 pftm DL ?? > 1362773081:04.00 > > > [pfpurge] > > > > ------------------------------------------------------------------------ > > > > > > > ------------------------------------------------------------------------ > > > vmstat -z > > > > > > ITEM SIZE LIMIT USED FREE REQUESTS > > > FAILURES > > > pfsrctrpl: 152, 10000, 0, 0, 0, > > > 0 > > > pfrulepl: 912, 0, 40, 88, 806, > > > 0 > > > pfstatepl: 392, 10000, 10000, 0, 4440553, > > > 341638 > > > pfaltqpl: 240, 0, 0, 0, 0, > > > 0 > > > pfpooladdrpl: 88, 0, 0, 0, 0, > > > 0 > > > pfrktable: 1296, 1002, 4, 20, 112, > > > 0 > > > pfrkentry: 216, 100008, 603, 891, 15384, > > > 0 > > > pfrkentry2: 216, 0, 0, 0, 0, > > > 0 > > > pffrent: 32, 5050, 0, 303, 1620, > > > 0 > > > pffrag: 80, 0, 0, 135, 807, > > > 0 > > > pffrcache: 80, 10035, 0, 0, 0, > > > 0 > > > pffrcent: 24, 50022, 0, 0, 0, > > > 0 > > > pfstatescrub: 40, 0, 0, 0, 0, > > > 0 > > > pfiaddrpl: 120, 0, 0, 0, 0, > > > 0 > > > pfospfen: 112, 0, 696, 30, 18096, > > > 0 > > > pfosfp: 40, 0, 407, 97, 10582, > > > 0 > > > > ------------------------------------------------------------------------ > > > > > > You can see evidence of processes not exiting/doing what they should do > > > here: > > > > > > > ------------------------------------------------------------------------ > > > fstat > > > > > > USER CMD PID FD MOUNT INUM MODE SZ|DV R/W > > > root shutdown 91155 root / 2 drwxr-xr-x 512 r > > > root shutdown 91155 wd / 2 drwxr-xr-x 512 r > > > root shutdown 91155 text / 47195 -r-sr-x--- 15912 r > > > root shutdown 91155 0 /dev 38 crw------- ttyu0 rw > > > root shutdown 91155 1 /dev 38 crw------- ttyu0 rw > > > root shutdown 91155 2 /dev 38 crw------- ttyu0 rw > > > root sh 91129 root / 2 drwxr-xr-x 512 r > > > root sh 91129 wd / 2 drwxr-xr-x 512 r > > > root sh 91129 text / 44 -r-xr-xr-x 134848 r > > > root sh 91129 0 /dev 38 crw------- ttyu0 rw > > > root sh 91129 1* pipe ffffff01e78fc9e0 <-> > ffffff01e78fc888 > > > 0 rw > > > root sh 91129 2 /dev 20 crw-rw-rw- null w > > > root shutdown 91115 root / 2 drwxr-xr-x 512 r > > > root shutdown 91115 wd /storage 5 drwx------ 37 r > > > root shutdown 91115 text / 47195 -r-sr-x--- 15912 r > > > root shutdown 91115 0 /dev 38 crw------- ttyu0 rw > > > root shutdown 91115 1 /dev 38 crw------- ttyu0 rw > > > root shutdown 91115 2 /dev 38 crw------- ttyu0 rw > > > root shutdown 91115 3* local dgram ffffff008ff92960 > > > root sh 90818 root / 2 drwxr-xr-x 512 r > > > root sh 90818 wd / 70659 drwxr-xr-x 512 r > > > root sh 90818 text / 44 -r-xr-xr-x 134848 r > > > root sh 90818 0 /dev 38 crw------- ttyu0 rw > > > root sh 90818 1* pipe ffffff0043f1ecb8 <-> > ffffff0043f1eb60 > > > 0 rw > > > root sh 90818 2 /dev 20 crw-rw-rw- null w > > > root csh 90802 root / 2 drwxr-xr-x 512 r > > > root csh 90802 wd / 2 drwxr-xr-x 512 r > > > root csh 90802 text / 51 -r-xr-xr-x 358752 r > > > root csh 90802 15 /dev 38 crw------- ttyu0 rw > > > root csh 90802 16 /dev 38 crw------- ttyu0 rw > > > root csh 90802 17 /dev 38 crw------- ttyu0 rw > > > root csh 90802 18 /dev 38 crw------- ttyu0 rw > > > root csh 90802 19 /dev 38 crw------- ttyu0 rw > > > > ------------------------------------------------------------------------ > > > > > > No indication of mbuf exhaustion, putting further focus on the pf > stack: > > > > > > > ------------------------------------------------------------------------ > > > netstat -m > > > > > > 2054/1786/3840 mbufs in use (current/cache/total) > > > 2048/1414/3462/25600 mbuf clusters in use (current/cache/total/max) > > > 2048/896 mbuf+clusters out of packet secondary zone in use > (current/cache) > > > 0/320/320/12800 4k (page size) jumbo clusters in use > > > (current/cache/total/max) > > > 0/0/0/19200 9k jumbo clusters in use (current/cache/total/max) > > > 0/0/0/12800 16k jumbo clusters in use (current/cache/total/max) > > > 4609K/4554K/9164K bytes allocated to network (current/cache/total) > > > 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > > > 0/0/0 requests for jumbo clusters denied (4k/9k/16k) > > > 0 requests for sfbufs denied > > > 0 requests for sfbufs delayed > > > 0 requests for I/O initiated by sendfile > > > 0 calls to protocol drain routines > > > > ------------------------------------------------------------------------ > > > > > > Here's one piece of core.0.txt which makes no sense to me -- the "rate" > > > column. I have a very hard time believing that was the interrupt rate > > > of all the relevant devices at the time (way too high). Maybe this > data > > > becomes wrong only during a coredump? The total column I could > believe. > > > > > > > ------------------------------------------------------------------------ > > > vmstat -i > > > > > > interrupt total rate > > > irq4: uart0 54768 912 > > > irq6: fdc0 1 0 > > > irq17: uhci1+ 172 2 > > > irq23: uhci3 ehci1+ 2367 39 > > > cpu0: timer 13183882632 219731377 > > > irq256: em0 260491055 4341517 > > > irq257: em1 127555036 2125917 > > > irq258: ahci0 225923164 3765386 > > > cpu2: timer 13183881837 219731363 > > > cpu1: timer 13002196469 216703274 > > > cpu3: timer 13183881783 219731363 > > > Total 53167869284 886131154 > > > > ------------------------------------------------------------------------ > > > > > > Here's what a normal "vmstat -i" shows from the command-line: > > > > > > # vmstat -i > > > interrupt total rate > > > irq4: uart0 518 0 > > > irq6: fdc0 1 0 > > > irq23: uhci3 ehci1+ 145 0 > > > cpu0: timer 19041199 1999 > > > irq256: em0 614280 64 > > > irq257: em1 168529 17 > > > irq258: ahci0 355536 37 > > > cpu2: timer 19040462 1999 > > > cpu1: timer 19040458 1999 > > > cpu3: timer 19040454 1999 > > > Total 77301582 8119 > > > > > > We graph many aspects of this box, including CPU load, memory/swap > > > usage, etc. and none show any sign that the interrupt rate on all of > > > those devices was even remotely out of control. (I would expect to see > > > CPU through the roof given the above data) > > > > > > I have since rebuilt/reinstalled world/kernel on the machine with the > > > latest RELENG_8 code (box is now 8.2-STABLE #0: Mon May 2 14:44:18 PDT > > > 2011), hoping whatever this was may have been fixed. > > > > > > As for what I think may have triggered it, but I have no hard evidence > > > of such: on April 29th, I changed our pf.conf and did "/etc/rc.d/pf > > > reload". The pf.conf change was a single line: > > > > > > Old: scrub on em0 all > > > New: scrub in on em0 all > > > > > > Why it took the problem approximately 3 days to start is unknown. It's > > > the only change we've made to the system (truly/honestly), and it was a > > > change to pf.conf. > > > > > > If anyone has advice (or has seen the above problem), or is interested > > > in debugging it -- as I said, I have a vmcore -- I'm happy to assist in > > > any way I can. I would hate for someone else to get bit by this, and > > > really am hoping its something that has been fixed between February and > > > now. > > > > > > > > I'm seeing this as well. You could change your scrub rules so that you > > specifically avoid TCP reassembly (that creates states). > > Thank you very much. This helps -- and I'm also glad someone else has > seen this behaviour. That confirms it's not specific to my equipment, > which is good. > > Regarding scrubbing and TCP reassembly (option "reassemble tcp" > according to pf.conf): I wasn't under the impression this option was > enabled by default. This got me wondering what the defaults actually > are (pf.conf(5) is somewhat vague in this regard, but it does state > that "fragment reassemble" is enabled by default). The rule I use: > > scrub in on em0 all > > Appears to get evaluated into this (per "pfctl -s rules -v"): > > scrub in on em0 all fragment reassemble > Ah, ok, my bad then. I used to have "reassemble tcp" enabled and disabled it this morning. However, if your symptoms are there without it, that's bad news. I disabled tcp reassembly to make sure there are no other states than the ones I allow via explicit keep/modulate/synproxy state rules. Disabling scrubbing altogether seems like a good next step. > Did you mean to tell me to disable the "fragment reassemble" option > (which is different from "reassemble tcp")? If so, how do I do that? > It looks like I could use a "no scrub" rule, except I can't find any > examples of this on the web or in the docs. Or is it just better to > remove use of scrub entirely until whatever this is gets fixed? > > -- > | Jeremy Chadwick jdc@parodius.com | > | Parodius Networking http://www.parodius.com/ | > | UNIX Systems Administrator Mountain View, CA, USA | > | Making life hard for others since 1977. PGP 4BD6C0CB | > > -- Good, fast & cheap. Pick any two. From owner-freebsd-pf@FreeBSD.ORG Tue May 3 06:29:33 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4DA58106566C; Tue, 3 May 2011 06:29:33 +0000 (UTC) (envelope-from snabb@epipe.com) Received: from tiktik.epipe.com (tiktik.epipe.com [IPv6:2001:1828:0:3::2]) by mx1.freebsd.org (Postfix) with ESMTP id 0AB538FC19; Tue, 3 May 2011 06:29:32 +0000 (UTC) Received: from tiktik.epipe.com (tiktik.epipe.com [IPv6:2001:1828:0:3::2]) by tiktik.epipe.com (8.14.4/8.14.4) with ESMTP id p436TT0K022213 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 3 May 2011 06:29:30 GMT (envelope-from snabb@epipe.com) X-DKIM: Sendmail DKIM Filter v2.8.3 tiktik.epipe.com p436TT0K022213 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=epipe.com; s=default; t=1304404170; x=1305008970; bh=uz5U/CRUnb5x4xoCMzRhNwlsTr1SkMKxm+/BIfct7rc=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=aQM90aRv08ZtDD1p7KCqj7HcAUj05Wtta4CVhMs7bp8S3hIQRtGgIFH4mUz+uoBpy bHpK7w1kKyY5iugl/WyTg/8WDw6WkF8bA8S5+D0I2OSXWxG/OiYQFL9IzT18kHfQgT drmK1HQUxCitPlV0o/MNDBFTOeBcxp6kSfuRroVU= Date: Tue, 3 May 2011 06:29:29 +0000 (UTC) From: Janne Snabb To: Vlad Galu In-Reply-To: Message-ID: References: <20110503015854.GA31444@icarus.home.lan> <20110503060106.GA36331@icarus.home.lan> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.2.7 (tiktik.epipe.com [IPv6:2001:1828:0:3::2]); Tue, 03 May 2011 06:29:30 +0000 (UTC) Cc: freebsd-stable@freebsd.org, Jeremy Chadwick , freebsd-pf@freebsd.org Subject: Re: RELENG_8 pf stack issue (state count spiraling out of control) X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 06:29:33 -0000 On Tue, 3 May 2011, Vlad Galu wrote: > Disabling scrubbing altogether seems like a good next step. I used to get all kinds of strange problems when I tried scrubbing on FreeBSD 8.1. Especially with IPv6 traffic. After I disabled scrubbing altogether I have had zero problems. The IP & TCP stacks behind this particular pf are good ones anyway, so scrubbing was useless anyway. My belief is that scrubbing is just broken, but I do not have any hard facts about it. I did not bother wasting my time trying to debug it after I noticed that the pf code has not been updated from the upstream for quite a while. The first thing would be to get on the same level with the upstream in case the problem is fixed there. However, I do not want to touch OpenBSD code for personal reasons. -- Janne Snabb / EPIPE Communications snabb@epipe.com - http://epipe.com/ From owner-freebsd-pf@FreeBSD.ORG Tue May 3 07:00:45 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9F5C6106564A for ; Tue, 3 May 2011 07:00:45 +0000 (UTC) (envelope-from dhartmei@insomnia.benzedrine.cx) Received: from insomnia.benzedrine.cx (106-30.3-213.fix.bluewin.ch [213.3.30.106]) by mx1.freebsd.org (Postfix) with ESMTP id A8BD48FC20 for ; Tue, 3 May 2011 07:00:44 +0000 (UTC) Received: from insomnia.benzedrine.cx (localhost.benzedrine.cx [127.0.0.1]) by insomnia.benzedrine.cx (8.14.1/8.13.4) with ESMTP id p4370gxU015502 (version=TLSv1/SSLv3 cipher=DHE-DSS-AES256-SHA bits=256 verify=NO); Tue, 3 May 2011 09:00:42 +0200 (MEST) Received: (from dhartmei@localhost) by insomnia.benzedrine.cx (8.14.1/8.12.10/Submit) id p4370gYQ032745; Tue, 3 May 2011 09:00:42 +0200 (MEST) Date: Tue, 3 May 2011 09:00:42 +0200 From: Daniel Hartmeier To: Jeremy Chadwick Message-ID: <20110503070042.GA9657@insomnia.benzedrine.cx> References: <20110503015854.GA31444@icarus.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20110503015854.GA31444@icarus.home.lan> User-Agent: Mutt/1.5.12-2006-07-14 Cc: freebsd-stable@freebsd.org, freebsd-pf@freebsd.org Subject: Re: RELENG_8 pf stack issue (state count spiraling out of control) X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 07:00:45 -0000 I read those graphs differently: the problem doesn't arise slowly, but rather seems to start suddenly at 13:00. Right after 13:00, traffic on em0 drops, i.e. the firewall seems to stop forwarding packets completely. Yet, at the same time, the states start to increase, almost linearly at about one state every two seconds, until the limit of 10,000 is reached. Reaching the limit seems to be only a side-effect of a problem that started at 13:00. > Here's one piece of core.0.txt which makes no sense to me -- the "rate" > column. I have a very hard time believing that was the interrupt rate > of all the relevant devices at the time (way too high). Maybe this data > becomes wrong only during a coredump? The total column I could believe. > > ------------------------------------------------------------------------ > vmstat -i > > interrupt total rate > irq4: uart0 54768 912 > irq6: fdc0 1 0 > irq17: uhci1+ 172 2 > irq23: uhci3 ehci1+ 2367 39 > cpu0: timer 13183882632 219731377 > irq256: em0 260491055 4341517 > irq257: em1 127555036 2125917 > irq258: ahci0 225923164 3765386 > cpu2: timer 13183881837 219731363 > cpu1: timer 13002196469 216703274 > cpu3: timer 13183881783 219731363 > Total 53167869284 886131154 > ------------------------------------------------------------------------ I find this suspect as well, but I don't have an explanation yet. Are you using anything non-GENERIC related to timers, like change HZ or enable polling? Are you sure the problem didn't start right at 13:00, and cause complete packet loss for the entire period, and that it grew gradually worse instead? Daniel From owner-freebsd-pf@FreeBSD.ORG Tue May 3 07:32:12 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 06597106566C for ; Tue, 3 May 2011 07:32:12 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta04.emeryville.ca.mail.comcast.net (qmta04.emeryville.ca.mail.comcast.net [76.96.30.40]) by mx1.freebsd.org (Postfix) with ESMTP id E0EEC8FC15 for ; Tue, 3 May 2011 07:32:11 +0000 (UTC) Received: from omta03.emeryville.ca.mail.comcast.net ([76.96.30.27]) by qmta04.emeryville.ca.mail.comcast.net with comcast id evXl1g0010b6N64A4vYBGr; Tue, 03 May 2011 07:32:11 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta03.emeryville.ca.mail.comcast.net with comcast id evY91g0041t3BNj8PvY9Nh; Tue, 03 May 2011 07:32:11 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 7088E9B418; Tue, 3 May 2011 00:32:09 -0700 (PDT) Date: Tue, 3 May 2011 00:32:09 -0700 From: Jeremy Chadwick To: Daniel Hartmeier Message-ID: <20110503073209.GA37676@icarus.home.lan> References: <20110503015854.GA31444@icarus.home.lan> <20110503070042.GA9657@insomnia.benzedrine.cx> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20110503070042.GA9657@insomnia.benzedrine.cx> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-stable@freebsd.org, freebsd-pf@freebsd.org Subject: Re: RELENG_8 pf stack issue (state count spiraling out of control) X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 07:32:12 -0000 On Tue, May 03, 2011 at 09:00:42AM +0200, Daniel Hartmeier wrote: > I read those graphs differently: the problem doesn't arise slowly, > but rather seems to start suddenly at 13:00. > > Right after 13:00, traffic on em0 drops, i.e. the firewall seems > to stop forwarding packets completely. > > Yet, at the same time, the states start to increase, almost linearly > at about one state every two seconds, until the limit of 10,000 is > reached. Reaching the limit seems to be only a side-effect of a > problem that started at 13:00. > > > Here's one piece of core.0.txt which makes no sense to me -- the "rate" > > column. I have a very hard time believing that was the interrupt rate > > of all the relevant devices at the time (way too high). Maybe this data > > becomes wrong only during a coredump? The total column I could believe. > > > > ------------------------------------------------------------------------ > > vmstat -i > > > > interrupt total rate > > irq4: uart0 54768 912 > > irq6: fdc0 1 0 > > irq17: uhci1+ 172 2 > > irq23: uhci3 ehci1+ 2367 39 > > cpu0: timer 13183882632 219731377 > > irq256: em0 260491055 4341517 > > irq257: em1 127555036 2125917 > > irq258: ahci0 225923164 3765386 > > cpu2: timer 13183881837 219731363 > > cpu1: timer 13002196469 216703274 > > cpu3: timer 13183881783 219731363 > > Total 53167869284 886131154 > > ------------------------------------------------------------------------ > > I find this suspect as well, but I don't have an explanation yet. > > Are you using anything non-GENERIC related to timers, like change > HZ or enable polling? HZ is standard (1000 is the default I believe), and I do not use polling. > Are you sure the problem didn't start right at 13:00, and cause complete > packet loss for the entire period, and that it grew gradually worse > instead? It's hard to discern from the graphs, but I can tell you exactly what I saw TCP-wise since I did have some already-existing/established TCP connections to the box (e.g. connections which already had ESTABLISHED states according to pfctl -s state) when it began exhibiting issues. Any packets which already had existing state entries in pf's state table continued to work, and bidirectionally. New inbound connections to the box via em0 would result in no response/timeout (and as indicated per pfctl, such packets were being dropped due to the state limit being reached). Outbound connections from the box via em0 to the outside world also resulted in no response/timeout. I will show you evidence of the latter. The first indication of a problem in syslog is the following message from named -- this is the first in my entire life I've ever seen this message, but seems to indicate some kind of internal watchdog was fired within named itself. The log I'm looking at, by the way, is /var/log/all.log -- yes, I do turn that on (for reasons exactly like this). This box is a secondary nameserver (public), so keep that in mind too. Anyway: May 1 12:50:14 isis named[728]: *** POKED TIMER *** Seconds later, I see unexpected RCODE messages, lame server messages, etc.. -- all which indicate packets to some degree are working ("the usual" badly-configured nameservers on the Internet). A few minutes later: May 1 12:53:15 isis named[728]: *** POKED TIMER *** May 1 12:53:54 isis named[728]: *** POKED TIMER *** With more of the usual unexpected RCODE/SERVFAIL messages after that. The next message: May 1 13:28:55 isis named[728]: *** POKED TIMER *** May 1 13:29:13 isis named[728]: *** POKED TIMER *** May 1 13:30:11 isis last message repeated 3 times Then more RCODE/SERVFAIL and something called "FORMERR" but that could be normal as well. Remember, all from named. This "cycle" of behaviour continued, with the number of POKED TIMER messages gradually increasing more and more as time went on. By 16:07 on May 1st, these messages were arriving usually in "bursts" of 5 or 6. Things finally "exploded", from named's perspective, here (with slaved zones X'd out): May 1 19:23:21 isis named[728]: *** POKED TIMER *** May 1 19:28:59 isis named[728]: zone XXXXXXXX/IN: refresh: failure trying master x.x.x.x#53 (source x.x.x.x#0): operation canceled May 1 19:35:32 isis named[728]: host unreachable resolving 'dns2.djaweb.dz/A/IN': 213.179.160.66#53 May 1 19:35:32 isis named[728]: host unreachable resolving 'dns2.djaweb.dz/A/IN': 193.0.12.4#53 May 1 19:35:32 isis named[728]: host unreachable resolving 'dns2.djaweb.dz/A/IN': 193.194.64.242#53 May 1 19:35:32 isis named[728]: host unreachable resolving 'dns2.djaweb.dz/A/IN': 192.134.0.49#53 And many other slaved zones reporting the exact same error. The hostnames which couldn't be resolved have no relevancy (honestly there is no consistency in any of them, and given what our service does, the randomness is expected. (Read: I just picked linear log lines above; don't focus on dns2.djaweb.dz like it's somehow responsible for this)) To me, this indicates a serious IP stack, firewall, or network exhaustion situation. The "host unreachable" messages became more and more regular, along with the occasional POKED TIMER. So what I'm saying is that the "most chatty" of all the daemons on the machine during this time was named, which does not surprise me given the machine's role. I finally logged in on May 2nd at 13:41 and tried to put an end to the madness. Other non-syslog-based daemons on the machine doing logging show no signs of issues, I/O errors (network or otherwise), etc.. Again, this doesn't surprise me, because nobody with existing connections to the machine was complaining (and hadn't). Only when I was told by a user that *new* connections were failing did I begin to investigate. So, it seems likely that network I/O mostly stopped because the IP stack stopped handling new I/O due to whatever was going on with pf. While at the same time, pf's state counter started gradually incrementing for reasons unknown -- an indicator that something bad was happening, almost certainly within pf itself, or somewhere within the kernel. I'm inclined to believe pf, because existing/ESTABLISHED stateful entries continued to get processed okay. Therefore, to me, the graphs are indicators that the problem started at around 12:50 PDT (meaning impact began at 12:50 PDT), and something very bad continued to happen within the pf stack, until I managed to intervene. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-pf@FreeBSD.ORG Tue May 3 08:48:02 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 17146106566B; Tue, 3 May 2011 08:48:02 +0000 (UTC) (envelope-from dhartmei@insomnia.benzedrine.cx) Received: from insomnia.benzedrine.cx (106-30.3-213.fix.bluewin.ch [213.3.30.106]) by mx1.freebsd.org (Postfix) with ESMTP id 731378FC12; Tue, 3 May 2011 08:48:01 +0000 (UTC) Received: from insomnia.benzedrine.cx (localhost.benzedrine.cx [127.0.0.1]) by insomnia.benzedrine.cx (8.14.1/8.13.4) with ESMTP id p438m1oY009095 (version=TLSv1/SSLv3 cipher=DHE-DSS-AES256-SHA bits=256 verify=NO); Tue, 3 May 2011 10:48:01 +0200 (MEST) Received: (from dhartmei@localhost) by insomnia.benzedrine.cx (8.14.1/8.12.10/Submit) id p438m0eU029544; Tue, 3 May 2011 10:48:00 +0200 (MEST) Date: Tue, 3 May 2011 10:48:00 +0200 From: Daniel Hartmeier To: Jeremy Chadwick Message-ID: <20110503084800.GB9657@insomnia.benzedrine.cx> References: <20110503015854.GA31444@icarus.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20110503015854.GA31444@icarus.home.lan> User-Agent: Mutt/1.5.12-2006-07-14 Cc: freebsd-stable@freebsd.org, freebsd-pf@freebsd.org Subject: Re: RELENG_8 pf stack issue (state count spiraling out of control) X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 08:48:02 -0000 On Mon, May 02, 2011 at 06:58:54PM -0700, Jeremy Chadwick wrote: > Status: Enabled for 76 days 06:49:10 Debug: Urgent > The "pf uptime" shown above, by the way, matches system uptime. > ps -axl > > UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND > 0 422 0 0 -16 0 0 0 pftm DL ?? 1362773081:04.00 [pfpurge] This looks weird, too. 1362773081 minutes would be >2500 years. Usually, you should see [idle] with almost uptime in minutes, and [pfpurge] with much less, like in # uptime 10:22AM up 87 days, 19:36, 1 user, load averages: 0.00, 0.03, 0.05 # echo "((87*24)+19)*60+36" | bc 126456 # ps -axl UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 0 7 0 0 44 0 0 8 pftm DL ?? 0:13.16 [pfpurge] 0 11 0 0 171 0 0 8 - RL ?? 124311:23.04 [idle] How is time handled on your machine? ntpdate on boot and then ntpd? Any manual time changes since the last boot? Daniel From owner-freebsd-pf@FreeBSD.ORG Tue May 3 09:16:23 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0FA39106566C for ; Tue, 3 May 2011 09:16:23 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from QMTA11.westchester.pa.mail.comcast.net (qmta11.westchester.pa.mail.comcast.net [76.96.59.211]) by mx1.freebsd.org (Postfix) with ESMTP id AB5D38FC20 for ; Tue, 3 May 2011 09:16:22 +0000 (UTC) Received: from omta03.westchester.pa.mail.comcast.net ([76.96.62.27]) by QMTA11.westchester.pa.mail.comcast.net with comcast id exFN1g0010bG4ec5BxGNsW; Tue, 03 May 2011 09:16:22 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta03.westchester.pa.mail.comcast.net with comcast id exGL1g00A1t3BNj3PxGMoV; Tue, 03 May 2011 09:16:21 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 486319B418; Tue, 3 May 2011 02:16:19 -0700 (PDT) Date: Tue, 3 May 2011 02:16:19 -0700 From: Jeremy Chadwick To: Daniel Hartmeier Message-ID: <20110503091619.GA39329@icarus.home.lan> References: <20110503015854.GA31444@icarus.home.lan> <20110503084800.GB9657@insomnia.benzedrine.cx> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20110503084800.GB9657@insomnia.benzedrine.cx> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-stable@freebsd.org, freebsd-pf@freebsd.org Subject: Re: RELENG_8 pf stack issue (state count spiraling out of control) X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 09:16:23 -0000 On Tue, May 03, 2011 at 10:48:00AM +0200, Daniel Hartmeier wrote: > On Mon, May 02, 2011 at 06:58:54PM -0700, Jeremy Chadwick wrote: > > > Status: Enabled for 76 days 06:49:10 Debug: Urgent > > > The "pf uptime" shown above, by the way, matches system uptime. > > > ps -axl > > > > UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND > > 0 422 0 0 -16 0 0 0 pftm DL ?? 1362773081:04.00 [pfpurge] > > This looks weird, too. 1362773081 minutes would be >2500 years. > > Usually, you should see [idle] with almost uptime in minutes, and > [pfpurge] with much less, like in > > # uptime > 10:22AM up 87 days, 19:36, 1 user, load averages: 0.00, 0.03, 0.05 > # echo "((87*24)+19)*60+36" | bc > 126456 > > # ps -axl > UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND > 0 7 0 0 44 0 0 8 pftm DL ?? 0:13.16 [pfpurge] > 0 11 0 0 171 0 0 8 - RL ?? 124311:23.04 [idle] Agreed -- and that's exactly how things look on the same box right now: $ ps -axl | egrep 'UID|pfpurge|idle' UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 0 11 0 0 171 0 0 64 - RL ?? 2375:15.91 [idle] 0 422 0 0 -16 0 0 16 pftm DL ?? 0:00.28 [pfpurge] The ps -axl output I provided earlier came from /var/crash/core.0.txt. So it's interesting that ps -axl as well as vmstat -i both showed something off-the-wall. I wonder if this can happen when within ddb? Unsure. I do have the core from "call doadump", so I should be able to go back and re-examine it with kgdb. I just wish I knew what to poke around looking for in there. Sadly I don't see a way with bsnmpd(8) to monitor things like interrupt usage, etc. otherwise I'd be graphing that. The more monitoring the better; at least then I could say "wow, interrupts really did shoot through the roof -- the box went crazy!" and RMA the thing. :-) > How is time handled on your machine? ntpdate on boot and then ntpd? Yep, you got it: ntpdate_enable="yes" ntpdate_config="/conf/ME/ntp.conf" ntpd_enable="yes" ntpd_config="/conf/ME/ntp.conf" I don't use ntpd_sync_on_start because I've never had reason to. I always set the system/BIOS clock to UTC time when building a system. I use ntpd's complaint about excessive offset as an indicator that something bad happened. /conf/ME/ntp.conf on this machine syncs from another on the private network (em1) only, and that machine syncs from a series of geographically-diverse stratum 2 servers and one stratum 1 server. I've never seen high delays, offsets, or jitter using "ntpq -c peers" on any box we have. Actual timecounters (not time itself) are handled by ACPI-safe or ACPI-fast (varies per boot; I've talked to jhb@ about this before and it's normal). powerd is in use on all our systems, and on this box use of processor sleep states (lowest state = C2; physical CPU only supports C0-C2 and I wouldn't go any lower than that anyway :-) ). Appropriate /boot/loader.conf entries that pertain to it: # Enable use of P-state CPU frequency throttling. # http://wiki.freebsd.org/TuningPowerConsumption hint.p4tcc.0.disabled="1" hint.acpi_throttle.0.disabled="1" There are numerous other systems exactly like this one (literally same model of hardware, RAM amount, CPU model, BIOS version and settings, and system configuration, including pf) that have much higher load and fire many more interrupts (particularly the NFS server!) that haven't exhibited any problems. This box had an uptime of 72 days, and prior to that around 100 (before being taken down for world/kernel upgrades). All machines have ECC RAM too, and MCA/MCE is in use. You don't know how bad I'd love to blame this on a hardware issue (it's always possible in some way or another), but the way this manifest itself was extremely specific. The problem could be super rare and something triggered it that hasn't been seen before by developers. So far there's only 1 other user who has seen this behaviour but his was attributed to use of "reassemble tcp" which I wasn't using; so the true problem could still be out there. I feel better knowing I'm not the only one who's seen this oddity. Since his post, I've removed all scrub rules from all of our machines as a precaution. If it ever happens again we'll have one more thing to safely rule out. We have other machines (different hardware, running RELENG_7 i386) which have had 1+ year uptimes also using pf, so the possibility of just some "crazy fluke" is plausible to me. > Any manual time changes since the last boot? None unless adjkerntz did something during the PST->PDT switchover, but that would manifest itself as a +1 hour offset difference. Since the machine rebooted the system synced its time without issue and well within acceptable delta (1.075993 sec). I did not power-cycle the box during any of this; pure soft reboots. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-pf@FreeBSD.ORG Tue May 3 09:22:59 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 639CE106567A; Tue, 3 May 2011 09:22:59 +0000 (UTC) (envelope-from dhartmei@insomnia.benzedrine.cx) Received: from insomnia.benzedrine.cx (106-30.3-213.fix.bluewin.ch [213.3.30.106]) by mx1.freebsd.org (Postfix) with ESMTP id 8EE768FC12; Tue, 3 May 2011 09:22:57 +0000 (UTC) Received: from insomnia.benzedrine.cx (localhost.benzedrine.cx [127.0.0.1]) by insomnia.benzedrine.cx (8.14.1/8.13.4) with ESMTP id p439MvwH015703 (version=TLSv1/SSLv3 cipher=DHE-DSS-AES256-SHA bits=256 verify=NO); Tue, 3 May 2011 11:22:57 +0200 (MEST) Received: (from dhartmei@localhost) by insomnia.benzedrine.cx (8.14.1/8.12.10/Submit) id p439Mvm4009848; Tue, 3 May 2011 11:22:57 +0200 (MEST) Date: Tue, 3 May 2011 11:22:57 +0200 From: Daniel Hartmeier To: Jeremy Chadwick Message-ID: <20110503092257.GC9657@insomnia.benzedrine.cx> References: <20110503015854.GA31444@icarus.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20110503015854.GA31444@icarus.home.lan> User-Agent: Mutt/1.5.12-2006-07-14 Cc: freebsd-stable@freebsd.org, freebsd-pf@freebsd.org Subject: Re: RELENG_8 pf stack issue (state count spiraling out of control) X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 09:22:59 -0000 On Mon, May 02, 2011 at 06:58:54PM -0700, Jeremy Chadwick wrote: > Here's one piece of core.0.txt which makes no sense to me -- the "rate" > column. I have a very hard time believing that was the interrupt rate > of all the relevant devices at the time (way too high). Maybe this data > becomes wrong only during a coredump? The total column I could believe. > > ------------------------------------------------------------------------ > vmstat -i > > interrupt total rate > irq4: uart0 54768 912 > irq6: fdc0 1 0 > irq17: uhci1+ 172 2 > irq23: uhci3 ehci1+ 2367 39 > cpu0: timer 13183882632 219731377 > irq256: em0 260491055 4341517 > irq257: em1 127555036 2125917 > irq258: ahci0 225923164 3765386 > cpu2: timer 13183881837 219731363 > cpu1: timer 13002196469 216703274 > cpu3: timer 13183881783 219731363 > Total 53167869284 886131154 > ------------------------------------------------------------------------ > > Here's what a normal "vmstat -i" shows from the command-line: > > # vmstat -i > interrupt total rate > irq4: uart0 518 0 > irq6: fdc0 1 0 > irq23: uhci3 ehci1+ 145 0 > cpu0: timer 19041199 1999 > irq256: em0 614280 64 > irq257: em1 168529 17 > irq258: ahci0 355536 37 > cpu2: timer 19040462 1999 > cpu1: timer 19040458 1999 > cpu3: timer 19040454 1999 > Total 77301582 8119 The cpu0-3 timer totals seem consistent in the first output: 13183881783/1999/60/60/24 matches 76 days of uptime. The high rate in the first output comes from vmstat.c dointr()'s division of the total by the uptime: struct timespec sp; clock_gettime(CLOCK_MONOTONIC, &sp); uptime = sp.tv_sec; for (i = 0; i < nintr; i++) { printf("%-*s %20lu %10lu\n", istrnamlen, intrname, *intrcnt, *intrcnt / uptime); } >From this we can deduce that the value of uptime must have been 13183881783/219731363 = 60 (seconds). Since the uptime was 76 days (and not just 60 seconds), the CLOCK_MONOTONIC clock must have reset, wrapped, or been overwritten. I don't know how that's possible, but if this means that the kernel variable time_second was possibly going back, that could very well have messed up pf's state purging. Daniel From owner-freebsd-pf@FreeBSD.ORG Tue May 3 09:32:02 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2F11D106564A; Tue, 3 May 2011 09:32:02 +0000 (UTC) (envelope-from vince@unsane.co.uk) Received: from unsane.co.uk (unsane-pt.tunnel.tserv5.lon1.ipv6.he.net [IPv6:2001:470:1f08:110::2]) by mx1.freebsd.org (Postfix) with ESMTP id A75BA8FC18; Tue, 3 May 2011 09:32:01 +0000 (UTC) Received: from vhoffman.lon.namesco.net (lon.namesco.net [195.7.254.102]) (authenticated bits=0) by unsane.co.uk (8.14.4/8.14.4) with ESMTP id p439VvXc016393 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Tue, 3 May 2011 10:31:58 +0100 (BST) (envelope-from vince@unsane.co.uk) Message-ID: <4DBFCB8D.10105@unsane.co.uk> Date: Tue, 03 May 2011 10:31:57 +0100 From: Vincent Hoffman User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.15) Gecko/20110303 Thunderbird/3.1.9 MIME-Version: 1.0 To: Jeremy Chadwick References: <20110503015854.GA31444@icarus.home.lan> <20110503084800.GB9657@insomnia.benzedrine.cx> <20110503091619.GA39329@icarus.home.lan> In-Reply-To: <20110503091619.GA39329@icarus.home.lan> X-Enigmail-Version: 1.1.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-stable@freebsd.org, freebsd-pf@freebsd.org Subject: Re: RELENG_8 pf stack issue (state count spiraling out of control) X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 09:32:02 -0000 On 03/05/2011 10:16, Jeremy Chadwick wrote: > Sadly I don't see a way with bsnmpd(8) to monitor things like interrupt > usage, etc. otherwise I'd be graphing that. The more monitoring the > better; at least then I could say "wow, interrupts really did shoot > through the roof -- the box went crazy!" and RMA the thing. :-) > you could use net-mgmt/bsnmp-regex although I dont know what the overhead for that is like. Vince From owner-freebsd-pf@FreeBSD.ORG Tue May 3 10:13:36 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9FF081065672; Tue, 3 May 2011 10:13:36 +0000 (UTC) (envelope-from dudu@dudu.ro) Received: from mail-qy0-f182.google.com (mail-qy0-f182.google.com [209.85.216.182]) by mx1.freebsd.org (Postfix) with ESMTP id 444588FC13; Tue, 3 May 2011 10:13:35 +0000 (UTC) Received: by qyk27 with SMTP id 27so3979990qyk.13 for ; Tue, 03 May 2011 03:13:35 -0700 (PDT) Received: by 10.229.43.209 with SMTP id x17mr6945302qce.257.1304417615239; Tue, 03 May 2011 03:13:35 -0700 (PDT) MIME-Version: 1.0 Received: by 10.229.105.18 with HTTP; Tue, 3 May 2011 03:12:55 -0700 (PDT) In-Reply-To: <4DBFCB8D.10105@unsane.co.uk> References: <20110503015854.GA31444@icarus.home.lan> <20110503084800.GB9657@insomnia.benzedrine.cx> <20110503091619.GA39329@icarus.home.lan> <4DBFCB8D.10105@unsane.co.uk> From: Vlad Galu Date: Tue, 3 May 2011 12:12:55 +0200 Message-ID: To: Vincent Hoffman Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-stable@freebsd.org, Jeremy Chadwick , freebsd-pf@freebsd.org Subject: Re: RELENG_8 pf stack issue (state count spiraling out of control) X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 10:13:36 -0000 On Tue, May 3, 2011 at 11:31 AM, Vincent Hoffman wrote: > On 03/05/2011 10:16, Jeremy Chadwick wrote: > > > > Sadly I don't see a way with bsnmpd(8) to monitor things like interrupt > > usage, etc. otherwise I'd be graphing that. The more monitoring the > > better; at least then I could say "wow, interrupts really did shoot > > through the roof -- the box went crazy!" and RMA the thing. :-) > > > you could use net-mgmt/bsnmp-regex although I dont know what the > overhead for that is like. > I use munin for graphing, as it allows easy scripting without using SNMP. My case is a bit different from Jeremy's. Every once in a while there is a sudden traffic spike which impacts pf performance as well. However, the graphed figures are nowhere near what I'd consider alarming levels (this box has withstood more in the past). I was able to coincidentally log in after such a spike and noticed the pfpurge thread eating up about 30% of the CPU while using the normal optimization policy. In my case, it could be related to another issue I'm seeing on this box - mbuma allocation failures. Here are my graphs: http://dl.dropbox.com/u/14650083/PF/bge_bits_1-week.png http://dl.dropbox.com/u/14650083/PF/bge_packets_1-week.png http://dl.dropbox.com/u/14650083/PF/bge_stats_1-week.png http://dl.dropbox.com/u/14650083/PF/load-week.png http://dl.dropbox.com/u/14650083/PF/mbuf_errors-week.png http://dl.dropbox.com/u/14650083/PF/mbuf_usage-week.png http://dl.dropbox.com/u/14650083/PF/pf_inserts-week.png http://dl.dropbox.com/u/14650083/PF/pf_matches-week.png http://dl.dropbox.com/u/14650083/PF/pf_removals-week.png http://dl.dropbox.com/u/14650083/PF/pf_searches-week.png http://dl.dropbox.com/u/14650083/PF/pf_src_limit-week.png http://dl.dropbox.com/u/14650083/PF/pf_states-week.png http://dl.dropbox.com/u/14650083/PF/pf_synproxy-week.png I'll wait for the next time the symptom occurs to switch to a stateless configuration. -- Good, fast & cheap. Pick any two. From owner-freebsd-pf@FreeBSD.ORG Tue May 3 10:18:02 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B172B1065677; Tue, 3 May 2011 10:18:02 +0000 (UTC) (envelope-from dudu@dudu.ro) Received: from mail-qy0-f182.google.com (mail-qy0-f182.google.com [209.85.216.182]) by mx1.freebsd.org (Postfix) with ESMTP id 5322C8FC0A; Tue, 3 May 2011 10:18:01 +0000 (UTC) Received: by qyk27 with SMTP id 27so3982165qyk.13 for ; Tue, 03 May 2011 03:18:01 -0700 (PDT) Received: by 10.229.29.20 with SMTP id o20mr7031652qcc.181.1304417881210; Tue, 03 May 2011 03:18:01 -0700 (PDT) MIME-Version: 1.0 Received: by 10.229.105.18 with HTTP; Tue, 3 May 2011 03:17:21 -0700 (PDT) In-Reply-To: References: <20110503015854.GA31444@icarus.home.lan> <20110503084800.GB9657@insomnia.benzedrine.cx> <20110503091619.GA39329@icarus.home.lan> <4DBFCB8D.10105@unsane.co.uk> From: Vlad Galu Date: Tue, 3 May 2011 12:17:21 +0200 Message-ID: To: Vincent Hoffman Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-stable@freebsd.org, Jeremy Chadwick , freebsd-pf@freebsd.org Subject: Re: RELENG_8 pf stack issue (state count spiraling out of control) X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 10:18:02 -0000 On Tue, May 3, 2011 at 12:12 PM, Vlad Galu wrote: > > > On Tue, May 3, 2011 at 11:31 AM, Vincent Hoffman wrote: > >> On 03/05/2011 10:16, Jeremy Chadwick wrote: >> >> >> > Sadly I don't see a way with bsnmpd(8) to monitor things like interrupt >> > usage, etc. otherwise I'd be graphing that. The more monitoring the >> > better; at least then I could say "wow, interrupts really did shoot >> > through the roof -- the box went crazy!" and RMA the thing. :-) >> > >> you could use net-mgmt/bsnmp-regex although I dont know what the >> overhead for that is like. >> > > I use munin for graphing, as it allows easy scripting without using SNMP. > > My case is a bit different from Jeremy's. Every once in a while there is a > sudden traffic spike which impacts pf performance as well. However, the > graphed figures are nowhere near what I'd consider alarming levels (this box > has withstood more in the past). I was able to coincidentally log in after > such a spike and noticed the pfpurge thread eating up about 30% of the CPU > while using the normal optimization policy. In my case, it could be related > to another issue I'm seeing on this box - mbuma allocation failures. Here > are my graphs: > > http://dl.dropbox.com/u/14650083/PF/bge_bits_1-week.png > http://dl.dropbox.com/u/14650083/PF/bge_packets_1-week.png > http://dl.dropbox.com/u/14650083/PF/bge_stats_1-week.png > http://dl.dropbox.com/u/14650083/PF/load-week.png > http://dl.dropbox.com/u/14650083/PF/mbuf_errors-week.png > http://dl.dropbox.com/u/14650083/PF/mbuf_usage-week.png > http://dl.dropbox.com/u/14650083/PF/pf_inserts-week.png > http://dl.dropbox.com/u/14650083/PF/pf_matches-week.png > http://dl.dropbox.com/u/14650083/PF/pf_removals-week.png > http://dl.dropbox.com/u/14650083/PF/pf_searches-week.png > http://dl.dropbox.com/u/14650083/PF/pf_src_limit-week.png > http://dl.dropbox.com/u/14650083/PF/pf_states-week.png > http://dl.dropbox.com/u/14650083/PF/pf_synproxy-week.png > > I'll wait for the next time the symptom occurs to switch to a stateless > configuration. > > I forgot to mention this is a UP box using TSC for timekeeping and running ntpd. -- /boot/loader.conf -- hint.p4tcc.0.disabled="1" hint.acpi_throttle.0.disabled="1" debug.acpi.disabled="timer" -- /boot/loader.conf -- -- sysctl output -- kern.timecounter.choice: TSC(800) i8254(0) dummy(-1000000) kern.timecounter.hardware: TSC -- sysctl output -- -- Good, fast & cheap. Pick any two. From owner-freebsd-pf@FreeBSD.ORG Tue May 3 10:39:53 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 881D21065673; Tue, 3 May 2011 10:39:53 +0000 (UTC) (envelope-from dhartmei@insomnia.benzedrine.cx) Received: from insomnia.benzedrine.cx (106-30.3-213.fix.bluewin.ch [213.3.30.106]) by mx1.freebsd.org (Postfix) with ESMTP id E10438FC14; Tue, 3 May 2011 10:39:52 +0000 (UTC) Received: from insomnia.benzedrine.cx (localhost.benzedrine.cx [127.0.0.1]) by insomnia.benzedrine.cx (8.14.1/8.13.4) with ESMTP id p43AdqG0018522 (version=TLSv1/SSLv3 cipher=DHE-DSS-AES256-SHA bits=256 verify=NO); Tue, 3 May 2011 12:39:52 +0200 (MEST) Received: (from dhartmei@localhost) by insomnia.benzedrine.cx (8.14.1/8.12.10/Submit) id p43AdohQ030494; Tue, 3 May 2011 12:39:50 +0200 (MEST) Date: Tue, 3 May 2011 12:39:50 +0200 From: Daniel Hartmeier To: Jeremy Chadwick Message-ID: <20110503103950.GD9657@insomnia.benzedrine.cx> References: <20110503015854.GA31444@icarus.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20110503015854.GA31444@icarus.home.lan> User-Agent: Mutt/1.5.12-2006-07-14 Cc: freebsd-stable@freebsd.org, freebsd-pf@freebsd.org Subject: Re: RELENG_8 pf stack issue (state count spiraling out of control) X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 10:39:53 -0000 On Mon, May 02, 2011 at 06:58:54PM -0700, Jeremy Chadwick wrote: > The next thing I tried was "/etc/rc.d/pf stop", which worked. Then I > did "/etc/rc.d/pf start", which also worked. However, what I saw next > surely indicated a bug in the pf layer somewhere -- "pfctl -s states" > and "pfctl -s info" disagreed on the state count: This can be explained. Note that "/etc/rc.d/pf start" does first flush all states by calling pfctl -F all. This calls pf_unlink_state() for every state in the kernel, which marks each state with PFTM_UNLINKED, but doesn't free it yet. Such states do not show up in pfctl -s state output, but are still counted in pfctl -s info output. Normally, they are freed the next time the pfpurge thread runs (once per second). It looks like the pfpurge thread was either a) sleeping indefinitely, not returning once a second from tsleep(pf_purge_thread, PWAIT, "pftm", 1 * hz); or b) constantly failing to acquire a lock with if (!sx_try_upgrade(&pf_consistency_lock)) return (0); Maybe a) is possible when CLOCK_MONOTONIC is decreasing? And the "POKED TIMER" messages you get from BIND, too? Kind regards, Daniel From owner-freebsd-pf@FreeBSD.ORG Tue May 3 12:42:25 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6D9181065677 for ; Tue, 3 May 2011 12:42:25 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta07.emeryville.ca.mail.comcast.net (qmta07.emeryville.ca.mail.comcast.net [76.96.30.64]) by mx1.freebsd.org (Postfix) with ESMTP id 4A3728FC15 for ; Tue, 3 May 2011 12:42:24 +0000 (UTC) Received: from omta23.emeryville.ca.mail.comcast.net ([76.96.30.90]) by qmta07.emeryville.ca.mail.comcast.net with comcast id ezsm1g0031wfjNsA70iQZE; Tue, 03 May 2011 12:42:24 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta23.emeryville.ca.mail.comcast.net with comcast id f0iM1g00H1t3BNj8j0iMp6; Tue, 03 May 2011 12:42:24 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 4B220102C31; Tue, 3 May 2011 05:42:21 -0700 (PDT) Date: Tue, 3 May 2011 05:42:21 -0700 From: Jeremy Chadwick To: Vincent Hoffman Message-ID: <20110503124221.GB13811@icarus.home.lan> References: <20110503015854.GA31444@icarus.home.lan> <20110503084800.GB9657@insomnia.benzedrine.cx> <20110503091619.GA39329@icarus.home.lan> <4DBFCB8D.10105@unsane.co.uk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4DBFCB8D.10105@unsane.co.uk> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-stable@freebsd.org, freebsd-pf@freebsd.org Subject: Re: RELENG_8 pf stack issue (state count spiraling out of control) X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 12:42:25 -0000 On Tue, May 03, 2011 at 10:31:57AM +0100, Vincent Hoffman wrote: > On 03/05/2011 10:16, Jeremy Chadwick wrote: > > > > Sadly I don't see a way with bsnmpd(8) to monitor things like interrupt > > usage, etc. otherwise I'd be graphing that. The more monitoring the > > better; at least then I could say "wow, interrupts really did shoot > > through the roof -- the box went crazy!" and RMA the thing. :-) > > > you could use net-mgmt/bsnmp-regex although I dont know what the > overhead for that is like. Thanks for the tip. I've investigated that plugin before, and its implementation model seems like a very hackish way to accomplish something that should ultimately be done inside of bsnmpd(8) itself via native C. It's good for parsing a single log file via tail -F (not "tail -f" like the man page indicates), but it doesn't scale well. bsnmpd(8) just needs to be enhanced and fixed, and I know there's efforts underway by syrinx@ to do exactly that. I have chatted with her about some existing problems with bsnmpd(8) and its SNMP parser, and have chatted with philip@ about a pf-related bug with bsnmp(8) (but I can't remember what the details of that one is; I have a file with the info around here somewhere...) There was also a recent commit to net-mgmt/net-snmp that pertains to *properly* monitoring swap, which makes me wonder if net-mgmt/bsnmp-ucd (which a lot of people, myself included, rely on) also does the wrong thing. http://www.freebsd.org/cgi/query-pr.cgi?pr=153179 http://www.freebsd.org/cgi/cvsweb.cgi/ports/net-mgmt/net-snmp/files/patch-memory_freebsd.c Things like this make me question my graphs and my monitoring data pretty much every time I look at them. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-pf@FreeBSD.ORG Tue May 3 15:44:54 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BF4901065676 for ; Tue, 3 May 2011 15:44:54 +0000 (UTC) (envelope-from np.mapn@gmail.com) Received: from mail-iy0-f182.google.com (mail-iy0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id 872758FC08 for ; Tue, 3 May 2011 15:44:54 +0000 (UTC) Received: by iyj12 with SMTP id 12so239151iyj.13 for ; Tue, 03 May 2011 08:44:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:date:message-id:subject:from:to :content-type; bh=j2PWZvxmpkgBGdYyK1oX0roDelWD67zL6aGuctitgbY=; b=XJEYilE4qY7Sz80nbyK0FrpLuZVojYxKG8pFECXqMVsZlEHEB9HG0970aCa4VPj0sr 0yl5c/9u+QysX3aZcJGb9k+JfLM6WH+B7Rr8mmja96ZoDG0q2XPONDDErPgkzGAQayKu fn4wp2TYgIumCm2NbFxKrBlugqfMyA8OavxHM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=x51eF9Ifxe9OS/PqPfeVIxHFek9SjjZAYXhCtkn5eFykStz+HxqRo/B8JuqD7TyZ25 1/cJ9XQyii/c3eDA2MoFXayf49aubqUqrMcQCVvZOoczy0rybPgbX/Th+VBgVKOzWOjp LKYBbOFl95jmCq0vY9KWQcU8TdTCrc6xZRaB4= MIME-Version: 1.0 Received: by 10.42.46.80 with SMTP id j16mr9282695icf.393.1304435634209; Tue, 03 May 2011 08:13:54 -0700 (PDT) Received: by 10.231.35.3 with HTTP; Tue, 3 May 2011 08:13:54 -0700 (PDT) Date: Tue, 3 May 2011 18:13:54 +0300 Message-ID: From: np map To: freebsd-pf@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: pf(4) from OpenBSD 4.5 for FreeBSD 8.2 X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2011 15:44:54 -0000 Hello, I saw the patch that Mr. Ermal Luci posted to this list. I tried it on FreeBSD 8.2. I also copied a few files from the latest pf45 snapshot to try to get it to build. It didn't compile and threw a lot of errors even after the missing header was added and altq-red.c was replaced. May we have a pf4.5 patch for FreeBSD 8.2, please? I would like to test this on a test machine with VIMAGE. Thank you. From owner-freebsd-pf@FreeBSD.ORG Fri May 6 13:58:22 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6A5641065674 for ; Fri, 6 May 2011 13:58:22 +0000 (UTC) (envelope-from aftaha@cirp.usp.br) Received: from quartzo.cirp.usp.br (quartzo.cirp.usp.br [143.107.200.45]) by mx1.freebsd.org (Postfix) with ESMTP id 235698FC16 for ; Fri, 6 May 2011 13:58:21 +0000 (UTC) Received: from granito2.cirp.usp.br (granito2.cirp.usp.br [143.107.200.101]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by quartzo.cirp.usp.br (Postfix) with ESMTPSA id 0A347330840 for ; Fri, 6 May 2011 10:39:21 -0300 (BRT) Message-ID: <4DC3FA08.6020700@cirp.usp.br> Date: Fri, 06 May 2011 10:39:20 -0300 From: Ali Faiez Taha Organization: Centro de =?ISO-8859-1?Q?Inform=E1tica_-_USP_-_Ribe?= =?ISO-8859-1?Q?ir=E3o_Preto?= User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.9.1.9) Gecko/20100408 Thunderbird/3.0.4 MIME-Version: 1.0 To: freebsd-pf@freebsd.org X-Enigmail-Version: 1.0.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: update rules X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: aftaha@cirp.usp.br List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 May 2011 13:58:22 -0000 I am trying to set up a PF in OpenBSD 4.9 and using old rules from OpenBSD 3.9 There are some SYNTAX ERRORS : no rdr on $ext_if inet proto tcp from to any rdr on $ext_if inet proto tcp from to any port smtp -> 127.0.0.1 port spamd rdr on $ext_if inet proto tcp from ! to any port smtp -> 127.0.0.1 port spamd rdr on $ext_if inet proto tcp from any to any port 21 -> $ftp_server port 21 pass in on $ext_if route-to lo0 inet proto tcp from any to 127.0.0.1 port spamd I am new on PF and need some help to solve this problem. Some book to understand the changes in PF ? Thanks a lot. From owner-freebsd-pf@FreeBSD.ORG Fri May 6 15:05:49 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4DADF106566C for ; Fri, 6 May 2011 15:05:49 +0000 (UTC) (envelope-from jon@radel.com) Received: from wave.radel.com (wave.radel.com [216.143.151.4]) by mx1.freebsd.org (Postfix) with ESMTP id 0E8828FC16 for ; Fri, 6 May 2011 15:05:48 +0000 (UTC) Received: by wave.radel.com (CommuniGate Pro PIPE 4.1.6) with PIPE id 10162943; Fri, 06 May 2011 10:05:48 -0400 Received: from [192.168.43.232] (account jon@radel.com HELO gravenstein.local) by wave.radel.com (CommuniGate Pro SMTP 4.1.6) with ESMTP-TLS id 10162941 for freebsd-pf@freebsd.org; Fri, 06 May 2011 10:05:34 -0400 Message-ID: <4DC4002F.3080202@radel.com> Date: Fri, 06 May 2011 10:05:35 -0400 From: Jon Radel User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.15) Gecko/20110303 Thunderbird/3.1.9 MIME-Version: 1.0 To: freebsd-pf@freebsd.org References: <4DC3FA08.6020700@cirp.usp.br> In-Reply-To: <4DC3FA08.6020700@cirp.usp.br> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit X-Radel.com-MailScanner-Information: Please contact Jon for more information X-Radel.com-MailScanner: Found to be clean X-Mailer: CommuniGate Pro CLI mailer Subject: Re: update rules X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 May 2011 15:05:49 -0000 On 5/6/11 9:39 AM, Ali Faiez Taha wrote: > > > I am trying to set up a PF in OpenBSD 4.9 and using old rules from OpenBSD 3.9 > I am new on PF and need some help to solve this problem. > Some book to understand the changes in PF ? I'd suggest you check out the PF documentation on the OpenBSD website, in particular something like http://openbsd.org/faq/pf/index.html and then ask follow-up questions on the appropriate OpenBSD mailing list. --Jon Radel jon@radel.com From owner-freebsd-pf@FreeBSD.ORG Fri May 6 16:58:29 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5BC101065672 for ; Fri, 6 May 2011 16:58:29 +0000 (UTC) (envelope-from artemrts@ukr.net) Received: from ffe10.ukr.net (ffe10.ukr.net [195.214.192.60]) by mx1.freebsd.org (Postfix) with ESMTP id 10EA68FC08 for ; Fri, 6 May 2011 16:58:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=ukr.net; s=ffe; h=Date:Message-Id:From:To:References:In-Reply-To:Subject:Cc:Content-Type:Content-Transfer-Encoding:MIME-Version; bh=1fX51tW8iOqKxDKDGygzg3shhIIS6vqkbhKve7RTgxQ=; b=jQXyfn2UXq29OQc/CVNBaUqdgUXo8h3J//+gtD/orzqQx+WAh6cLD8c07clT5zLAYdEBDBNvH8r7aWY4v/8RoXE5QzDtbX1y7MHMk0utbe/wtZ056Tj9pvZFQN4cHfDJIZqoQhEwKodh2fwtK0oF9QczdgO7pb9xL4YPTp7kdWg=; Received: from mail by ffe10.ukr.net with local ID 1QIO5v-000LlY-Rs ; Fri, 06 May 2011 19:41:35 +0300 MIME-Version: 1.0 Content-Disposition: inline Content-Transfer-Encoding: binary Content-Type: text/plain; charset="windows-1251" In-Reply-To: <4DC3FA08.6020700@cirp.usp.br> References: <4DC3FA08.6020700@cirp.usp.br> To: "Ali Faiez Taha" From: =?WINDOWS-1251?B?wujy4Ovo6SDC6+Dk6Ozo8O7i6Pc=?= X-Mailer: freemail.ukr.net 4.0 X-Originating-Ip: [195.200.251.1] X-Browser: Mozilla/5.0 (Unix) Message-Id: Date: Fri, 06 May 2011 19:41:35 +0300 Cc: freebsd-pf@freebsd.org Subject: Re: update rules X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 May 2011 16:58:29 -0000 --- Original Message --- From: "Ali Faiez Taha" To: freebsd-pf@freebsd.org Date: 6 May 2011, 16:39:20 Subject: Re: update rules > I am trying to set up a PF in OpenBSD 4.9 and using old rules from OpenBSD 3.9 > > There are some SYNTAX ERRORS : > > no rdr on $ext_if inet proto tcp from to any > > rdr on $ext_if inet proto tcp from to any port smtp -> 127.0.0.1 port spamd > > rdr on $ext_if inet proto tcp from ! to any port smtp -> 127.0.0.1 port spamd > > rdr on $ext_if inet proto tcp from any to any port 21 -> $ftp_server port 21 > > pass in on $ext_if route-to lo0 inet proto tcp from any to 127.0.0.1 port spamd > > I am new on PF and need some help to solve this problem. > Some book to understand the changes in PF ? > > Thanks a lot. You should check new features in pf on the official OpenBSD website http://openbsd.org/39.html#new http://openbsd.org/40.html#new http://openbsd.org/41.html#new ...... From owner-freebsd-pf@FreeBSD.ORG Fri May 6 20:52:48 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 685CE1065672 for ; Fri, 6 May 2011 20:52:48 +0000 (UTC) (envelope-from mh.unet@gmail.com) Received: from mail-pw0-f54.google.com (mail-pw0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id 4090F8FC14 for ; Fri, 6 May 2011 20:52:48 +0000 (UTC) Received: by pwj8 with SMTP id 8so2093979pwj.13 for ; Fri, 06 May 2011 13:52:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:date:message-id:subject:from:to :content-type; bh=9mwffTvIeJfAHp67wRCZn0AamJFVh3sf0YlON/AuBv8=; b=EpmFjdc218FVztmiUPaMa1ofbsFRBVJYtBnDGw6kZkxGmL+9IWTzWZqKjxB+v0IHl4 CqxNh6QNEgYlv+5sOqGHgrL0oYMtAL3NQrFdU+XRi6J/p9WdQhRnhcaVtOycS4721jmy VUC6TOIpxsZxdoiyxZ8bFHrcOsM7JLZsx1MtQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=KsvYfumRA/Wkz4gMriv313c3Ke4FGGXJICQdwdZynAJi3i7vFVR7Ala4/r00DCy9Hd bRzLKvuJICLQCJdDEC0ockQAdINWj+ziwSkWaKzRDdjAtU2na0lzDFyHYGklwnSYszYS /oiSx5z549nzFPR1HEBNmbWw0E6o14SG95RL4= MIME-Version: 1.0 Received: by 10.142.98.13 with SMTP id v13mr2226357wfb.279.1304713626395; Fri, 06 May 2011 13:27:06 -0700 (PDT) Received: by 10.143.33.3 with HTTP; Fri, 6 May 2011 13:27:06 -0700 (PDT) Date: Fri, 6 May 2011 13:27:06 -0700 Message-ID: From: Mickey Harvey To: freebsd-pf@freebsd.org X-Mailman-Approved-At: Fri, 06 May 2011 20:56:29 +0000 Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Running pf or ipfw in a jail X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 May 2011 20:52:48 -0000 Is it possible to run pf or ipfw within a jail? I am running 8.2 and have vimage compiled in the kernel. From owner-freebsd-pf@FreeBSD.ORG Sat May 7 07:31:13 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3330D106566B for ; Sat, 7 May 2011 07:31:13 +0000 (UTC) (envelope-from jhellenthal@gmail.com) Received: from mail-iw0-f182.google.com (mail-iw0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id DD1988FC0A for ; Sat, 7 May 2011 07:31:12 +0000 (UTC) Received: by iwn33 with SMTP id 33so4496750iwn.13 for ; Sat, 07 May 2011 00:31:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:sender:date:from:to:cc:subject:message-id :references:mime-version:content-type:content-disposition :in-reply-to:x-openpgp-key-id:x-openpgp-key-fingerprint :x-openpgp-key-url; bh=N8escXC269kwubaoJOXkaWuiM3Cn0psFN7HPTqFILr8=; b=sBZ2sWvoNJzFi/kvKf7Ua94q2ELb8d0KD8SJaSIwliBpkMhO6WtFVKEpZFy8LmvwRo T0ThlIC06c/RXL3NF+DEJFtulrRFgZzBgmFieK2cacng3dhA0CyyVHS8daZpWoro3KjJ IjWwJ3SBc7x2EUprQ/I/Bg3jSPktN499ldniM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:x-openpgp-key-id :x-openpgp-key-fingerprint:x-openpgp-key-url; b=P1aO/eYGxiL6nYU23XXWGqpRdicc46tqQGyWFyn5JccxMTD9x9KC6YSajXed/krk61 MkY35lYz8uEBcKbkLRIaVxvHWUBpF5lW5dZJAzdwhAt0W9b2XtjWbCoeQXc9ox5oSp0O igVu8y67Zzn+MXGMwnqZkKfgFcM45YUWRinAM= Received: by 10.42.115.138 with SMTP id k10mr3872221icq.81.1304753472308; Sat, 07 May 2011 00:31:12 -0700 (PDT) Received: from DataIX.net (adsl-99-190-84-116.dsl.klmzmi.sbcglobal.net [99.190.84.116]) by mx.google.com with ESMTPS id un1sm1512037icb.21.2011.05.07.00.31.10 (version=TLSv1/SSLv3 cipher=OTHER); Sat, 07 May 2011 00:31:10 -0700 (PDT) Sender: "J. Hellenthal" Received: from DataIX.net (localhost [127.0.0.1]) by DataIX.net (8.14.4/8.14.4) with ESMTP id p477V7rl011944 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 7 May 2011 03:31:08 -0400 (EDT) (envelope-from jhell@DataIX.net) Received: (from jhell@localhost) by DataIX.net (8.14.4/8.14.4/Submit) id p477V6x2011941; Sat, 7 May 2011 03:31:06 -0400 (EDT) (envelope-from jhell@DataIX.net) Date: Sat, 7 May 2011 03:31:06 -0400 From: Jason Hellenthal To: Mickey Harvey Message-ID: <20110507073106.GA8776@DataIX.net> References: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="ibTvN161/egqYuK8" Content-Disposition: inline In-Reply-To: X-OpenPGP-Key-Id: 0x89D8547E X-OpenPGP-Key-Fingerprint: 85EF E26B 07BB 3777 76BE B12A 9057 8789 89D8 547E X-OpenPGP-Key-URL: http://bit.ly/0x89D8547E Cc: freebsd-pf@freebsd.org Subject: Re: Running pf or ipfw in a jail X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 07 May 2011 07:31:13 -0000 --ibTvN161/egqYuK8 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Mickey, On Fri, May 06, 2011 at 01:27:06PM -0700, Mickey Harvey wrote: >Is it possible to run pf or ipfw within a jail? I am running 8.2 and have >vimage compiled in the kernel. PF in a jail or on FreeBSD 8.2 does not work with a VIMAGE kernel. As for= =20 IPFW I couldnt say much about since everything that can be done from the=20 jail can also be done from the master host. --=20 Regards, (jhell) Jason Hellenthal --ibTvN161/egqYuK8 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.17 (FreeBSD) Comment: http://bit.ly/0x89D8547E iQEcBAEBAgAGBQJNxPU5AAoJEJBXh4mJ2FR+4LAH/20jvxycVegIV/GqUBViQaHT cdM0iLPyZDomSq4pbi+RmwZPDdWeUfAh0BTFvzNM9ApjX9ueEL8kYgilC6pJsb1j GU7ALC4MYwq5iWPHgeW/kBkWnZD0YvgpyDoPFo617WAFLB8h7LQXE2T6a9qJmkgk Z22fULrWaVydUC6qIUvlLPXmGYzEHhQafBvIoEFgfIrCbw26yVWLI4a6jGsiiYor 7U1EPdguAW8sZQIZfra3LFtestXVF1cDLBkLiCm9QxDlYmq2xJ0JJC/hOVVgCESr CBtFrPGQr/InN7pOLpHPrkG2iQQ4muxA9blMRid+WE3xdJI9ksmRQ/r9wYq/y8M= =XSU7 -----END PGP SIGNATURE----- --ibTvN161/egqYuK8-- From owner-freebsd-pf@FreeBSD.ORG Sat May 7 08:04:47 2011 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 110C0106566B for ; Sat, 7 May 2011 08:04:47 +0000 (UTC) (envelope-from riaank@gmail.com) Received: from mail-yi0-f54.google.com (mail-yi0-f54.google.com [209.85.218.54]) by mx1.freebsd.org (Postfix) with ESMTP id C1CE78FC14 for ; Sat, 7 May 2011 08:04:46 +0000 (UTC) Received: by yie12 with SMTP id 12so1774954yie.13 for ; Sat, 07 May 2011 01:04:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=cp0GummL+OHvFSU6NVLCuKndiTn3/esTrFHJjLIOjzg=; b=aDXHUpn9zK4SF1uD5oCJRSJhIwMxDa15fFIOsEmI/F03JrnQKrNmkuFBB4ouR28Wwo tJXRE3R0uQLvM/2JyqSTVvp+KzTc5OLK9OIXfgaDJ8Xlp99aMENHhJ2zcOS5nQJCQhZ2 tkJRvPQLKigBL7o6HUHpg1s/POCkltJqGbzr0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=DMihc4gt67OB76AZPxGUsEzq1TElSSOtX6P3KqITARre+ZwdA5eDALr4Irb3iO4og/ 2dD6TRke81SlgmzIu8b4vby82Mxiwqe84DIxTcLzLPFBSiEXOMYnV9O9Om1QyT1sPBIX 2Qa9ycqgZh0B8RcruLnZRFTQlKRnsWmirkTkg= MIME-Version: 1.0 Received: by 10.147.157.33 with SMTP id j33mr4017729yao.1.1304753709716; Sat, 07 May 2011 00:35:09 -0700 (PDT) Received: by 10.147.136.8 with HTTP; Sat, 7 May 2011 00:35:09 -0700 (PDT) In-Reply-To: References: Date: Sat, 7 May 2011 09:35:09 +0200 Message-ID: From: Riaan Kruger To: freebsd-pf@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Re: Running pf or ipfw in a jail X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 07 May 2011 08:04:47 -0000 On Fri, May 6, 2011 at 10:27 PM, Mickey Harvey wrote: > Is it possible to run pf or ipfw within a jail? I am running 8.2 and have > vimage compiled in the kernel. > _______________________________________________ > freebsd-pf@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-pf > To unsubscribe, send any mail to "freebsd-pf-unsubscribe@freebsd.org" > As far as I know ipfw could work, pf not. Maybe look here for more info. http://2010.eurobsdcon.org/fileadmin/fe_user/bz/ZDZZ8M.pdf Riaan