From owner-freebsd-ipfw@FreeBSD.ORG Mon Jan 24 01:08:55 2011 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CE1831065724 for ; Mon, 24 Jan 2011 01:08:55 +0000 (UTC) (envelope-from ptyll@nitronet.pl) Received: from mail.nitronet.pl (smtp.nitronet.pl [195.90.106.27]) by mx1.freebsd.org (Postfix) with ESMTP id 8641C8FC20 for ; Mon, 24 Jan 2011 01:08:55 +0000 (UTC) Received: from mailnull by mail.nitronet.pl with virscan (Exim 4.72 (FreeBSD)) (envelope-from ) id 1PhAvO-0000YC-Ny for freebsd-ipfw@freebsd.org; Mon, 24 Jan 2011 02:08:54 +0100 Date: Mon, 24 Jan 2011 02:08:48 +0100 From: Pawel Tyll X-Priority: 3 (Normal) Message-ID: <13247006.20110124020848@nitronet.pl> To: Brandon Gooch In-Reply-To: References: <201953550.20110106221821@nitronet.pl> <57312439.20110107171430@nitronet.pl> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-Virus-Scanned: Nitronet.pl X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: ptyll@nitronet.pl X-SA-Exim-Scanned: No (on mail.nitronet.pl); SAEximRunCond expanded to false Cc: freebsd-net@freebsd.org, Luigi Rizzo , Jack Vogel , freebsd-ipfw@freebsd.org Subject: Re: [Panic] Dummynet/IPFW related recurring crash. X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Jan 2011 01:08:55 -0000 On Fri, Jan 7, 2011, Brandon Gooch wrote: > It's likely that the mbuf handling problem (in em_refresh_mbufs()) is > triggered by the processing you're doing with ipfw (or elsewhere for > that matter), so, yes, I think it's a bug fixed in the revision > discussed. > When you update and test, please let us know. Also, don't forget to > submit a follow-up to your PR. Unfortunately bad news: Machine fell after 14 days, 22:31:42 for the same reason according to what was left of panic screen. It didn't do a dump, nor reboot as is customary since some time on S3420GP boards (and other Intel server boards, since colleague has dual-cpu board from same epoch). What can I try next? From owner-freebsd-ipfw@FreeBSD.ORG Mon Jan 24 11:07:03 2011 Return-Path: Delivered-To: freebsd-ipfw@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 283841065719 for ; Mon, 24 Jan 2011 11:07:03 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 0C04C8FC16 for ; Mon, 24 Jan 2011 11:07:03 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p0OB727Z077833 for ; Mon, 24 Jan 2011 11:07:02 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p0OB72j6077831 for freebsd-ipfw@FreeBSD.org; Mon, 24 Jan 2011 11:07:02 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 24 Jan 2011 11:07:02 GMT Message-Id: <201101241107.p0OB72j6077831@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-ipfw@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-ipfw@FreeBSD.org X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Jan 2011 11:07:03 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/153415 ipfw [ipfw] [patch] Port numbers always zero in dynamic IPF o bin/153252 ipfw [ipfw][patch] ipfw lockdown system in subsequent call o kern/153161 ipfw IPFIREWALL does not allow specify rules with ICMP code o kern/152887 ipfw [ipfw] Can not set more then 1024 buckets with buckets o kern/152113 ipfw [ipfw] page fault on 8.1-RELEASE caused by certain amo o kern/150798 ipfw [ipfw] ipfw2 fwd rule matches packets but does not do o kern/148827 ipfw [ipfw] divert broken with in-kernel ipfw o kern/148689 ipfw [ipfw] antispoof wrongly triggers on link local IPv6 a o kern/148430 ipfw [ipfw] IPFW schedule delete broken. o kern/148157 ipfw [ipfw] IPFW in kernel nat BUG found in FreeBSD 8.1-PRE o kern/148091 ipfw [ipfw] ipfw ipv6 handling broken. o kern/147720 ipfw [ipfw] ipfw dynamic rules and fwd o kern/145733 ipfw [ipfw] [patch] ipfw flaws with ipv6 fragments o kern/145305 ipfw [ipfw] ipfw problems, panics, data corruption, ipv6 so o kern/144269 ipfw [ipfw] problem with ipfw tables o kern/144187 ipfw [ipfw] deadlock using multiple ipfw nat and multiple l o kern/143973 ipfw [ipfw] [panic] ipfw forward option causes kernel reboo o kern/143653 ipfw [ipfw] [patch] ipfw nat redirect_port "buf is too smal o kern/143621 ipfw [ipfw] [dummynet] [patch] dummynet and vnet use result o kern/143474 ipfw [ipfw] ipfw table contains the same address f kern/142951 ipfw [dummynet] using pipes&queues gives OUCH! pipe should o kern/139581 ipfw [ipfw] "ipfw pipe" not limiting bandwidth o kern/139226 ipfw [ipfw] install_state: entry already present, done o kern/137346 ipfw [ipfw] ipfw nat redirect_proto is broken o kern/137232 ipfw [ipfw] parser troubles o kern/136695 ipfw [ipfw] [patch] fwd reached after skipto in dynamic rul o kern/135476 ipfw [ipfw] IPFW table breaks after adding a large number o o bin/134975 ipfw [patch] ipfw(8) can't work with set in rule file. o kern/131817 ipfw [ipfw] blocks layer2 packets that should not be blocke o kern/131601 ipfw [ipfw] [panic] 7-STABLE panic in nat_finalise (tcp=0) o kern/131558 ipfw [ipfw] Inconsistent "via" ipfw behavior o bin/130132 ipfw [patch] ipfw(8): no way to get mask from ipfw pipe sho o kern/129103 ipfw [ipfw] IPFW check state does not work =( o kern/129093 ipfw [ipfw] ipfw nat must not drop packets o kern/129036 ipfw [ipfw] 'ipfw fwd' does not change outgoing interface n o kern/128260 ipfw [ipfw] [patch] ipfw_divert damages IPv6 packets o kern/127230 ipfw [ipfw] [patch] Feature request to add UID and/or GID l o kern/127209 ipfw [ipfw] IPFW table become corrupted after many changes o bin/125370 ipfw [ipfw] [patch] increase a line buffer limit o conf/123119 ipfw [patch] rc script for ipfw does not handle IPv6 o kern/122963 ipfw [ipfw] tcpdump does not show packets redirected by 'ip o kern/122109 ipfw [ipfw] ipfw nat traceroute problem s kern/121807 ipfw [request] TCP and UDP port_table in ipfw o kern/121382 ipfw [dummynet] 6.3-RELEASE-p1 page fault in dummynet (corr o kern/121122 ipfw [ipfw] [patch] add support to ToS IP PRECEDENCE fields o kern/118993 ipfw [ipfw] page fault - probably it's a locking problem o bin/117214 ipfw ipfw(8) fwd with IPv6 treats input as IPv4 o kern/116009 ipfw [ipfw] [patch] Ignore errors when loading ruleset from o docs/113803 ipfw [patch] ipfw(8) - don't get bitten by the fwd rule o kern/112561 ipfw [ipfw] ipfw fwd does not work with some TCP packets o kern/105330 ipfw [ipfw] [patch] ipfw (dummynet) does not allow to set q o bin/104921 ipfw [patch] ipfw(8) sometimes treats ipv6 input as ipv4 (a o kern/104682 ipfw [ipfw] [patch] Some minor language consistency fixes a o kern/103454 ipfw [ipfw] [patch] [request] add a facility to modify DF b o kern/103328 ipfw [ipfw] [request] sugestions about ipfw table o kern/102471 ipfw [ipfw] [patch] add tos and dscp support o kern/98831 ipfw [ipfw] ipfw has UDP hickups o kern/97951 ipfw [ipfw] [patch] ipfw does not tie interface details to o kern/95084 ipfw [ipfw] [regression] [patch] IPFW2 ignores "recv/xmit/v o kern/93300 ipfw [ipfw] ipfw pipe lost packets o kern/91847 ipfw [ipfw] ipfw with vlanX as the device o kern/88659 ipfw [modules] ipfw and ip6fw do not work properly as modul o kern/87032 ipfw [ipfw] [patch] ipfw ioctl interface implementation o kern/86957 ipfw [ipfw] [patch] ipfw mac logging o bin/83046 ipfw [ipfw] ipfw2 error: "setup" is allowed for icmp, but s o kern/82724 ipfw [ipfw] [patch] [request] Add setnexthop and defaultrou o bin/78785 ipfw [patch] ipfw(8) verbosity locks machine if /etc/rc.fir o kern/74104 ipfw [ipfw] ipfw2/1 conflict not detected or reported, manp o kern/73910 ipfw [ipfw] serious bug on forwarding of packets after NAT o kern/72987 ipfw [ipfw] ipfw/dummynet pipe/queue 'queue [BYTES]KBytes ( o kern/71366 ipfw [ipfw] "ipfw fwd" sometimes rewrites destination mac a o kern/69963 ipfw [ipfw] install_state warning about already existing en o kern/60719 ipfw [ipfw] Headerless fragments generate cryptic error mes o kern/55984 ipfw [ipfw] [patch] time based firewalling support for ipfw o kern/51274 ipfw [ipfw] [patch] ipfw2 create dynamic rules with parent o kern/48172 ipfw [ipfw] [patch] ipfw does not log size and flags o kern/46159 ipfw [ipfw] [patch] [request] ipfw dynamic rules lifetime f a kern/26534 ipfw [ipfw] Add an option to ipfw to log gid/uid of who cau 78 problems total. From owner-freebsd-ipfw@FreeBSD.ORG Mon Jan 24 17:11:14 2011 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2FB3910656A4; Mon, 24 Jan 2011 17:11:14 +0000 (UTC) (envelope-from jamesbrandongooch@gmail.com) Received: from mail-ew0-f54.google.com (mail-ew0-f54.google.com [209.85.215.54]) by mx1.freebsd.org (Postfix) with ESMTP id 878B88FC32; Mon, 24 Jan 2011 17:11:13 +0000 (UTC) Received: by ewy24 with SMTP id 24so2044876ewy.13 for ; Mon, 24 Jan 2011 09:11:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=inkRhYBEHV5u2A9+dcK5fyEN3ADJlJwpl31Fql1EcL0=; b=YjC3iSPs8DVf3vUmkLUPURewLe5Gq609ibGWdOvj3oogZYXU8RI+h8yAHIPXs7KKJZ 9hrMDVKWb/0VVdX7CNEZj6ZJUsKI2l7uzo1YSY1O1hGE9urYirIdribdqrXfPEGW4O0M T6tCoopOrHyFOm9UgbwZfasFJeakd9+F0O6y8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=Q82li1J/UusT2v5qoM0Idponrx3kB5pRQNu4M4h47fX2QI8615I0T7zucOzmS+ei3l hTw3skX7PYLrmULg0RxpmNlVQ22tyuFDk+GW8lPewAvuXbiZj3RDDrfFdv10YM67qlS5 b1tdZuARao2zFIaj71D5pczypOdXJJedC+p+w= MIME-Version: 1.0 Received: by 10.216.55.145 with SMTP id k17mr2762962wec.48.1295889072086; Mon, 24 Jan 2011 09:11:12 -0800 (PST) Received: by 10.216.36.71 with HTTP; Mon, 24 Jan 2011 09:11:12 -0800 (PST) In-Reply-To: <13247006.20110124020848@nitronet.pl> References: <201953550.20110106221821@nitronet.pl> <57312439.20110107171430@nitronet.pl> <13247006.20110124020848@nitronet.pl> Date: Mon, 24 Jan 2011 11:11:12 -0600 Message-ID: From: Brandon Gooch To: Pawel Tyll Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-net@freebsd.org, Luigi Rizzo , Jack Vogel , freebsd-ipfw@freebsd.org Subject: Re: [Panic] Dummynet/IPFW related recurring crash. X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Jan 2011 17:11:14 -0000 On Sun, Jan 23, 2011 at 7:08 PM, Pawel Tyll wrote: > On Fri, Jan 7, 2011, Brandon Gooch wrote: >> It's likely that the mbuf handling problem (in em_refresh_mbufs()) is >> triggered by the processing you're doing with ipfw (or elsewhere for >> that matter), so, yes, I think it's a bug fixed in the revision >> discussed. > >> When you update and test, please let us know. Also, don't forget to >> submit a follow-up to your PR. > > Unfortunately bad news: > > Machine fell after 14 days, 22:31:42 for the same reason according to > what was left of panic screen. It didn't do a dump, nor reboot as is > customary since some time on S3420GP boards (and other Intel server > boards, since colleague has dual-cpu board from same epoch). > > What can I try next? I'm not sure. Perhaps there is another code path in the em(4) driver that leads to the same, bad-pointer state. I'll have to defer to Jack Vogel or Luigi Rizzo for potential insight into the issue. Please provide as much information as possible (even if it comes down to taking a photo of the console, or transcribing the backtrace). FYI, I haven't been able to test this on any of my setups yet... -Brandon From owner-freebsd-ipfw@FreeBSD.ORG Mon Jan 24 17:36:59 2011 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CD23C106566C for ; Mon, 24 Jan 2011 17:36:59 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-gy0-f182.google.com (mail-gy0-f182.google.com [209.85.160.182]) by mx1.freebsd.org (Postfix) with ESMTP id 82EAA8FC0C for ; Mon, 24 Jan 2011 17:36:59 +0000 (UTC) Received: by gyf3 with SMTP id 3so1444645gyf.13 for ; Mon, 24 Jan 2011 09:36:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=0YWGxfmvtr6UoD1zv6R2IStcj+kcb6EFPsotc3qNF1I=; b=dh+Md5xYQ7EbK3wI8PCNW9uSA3wbI6z31/TWVYZnS9ekquyGyM5ZvF2eY3Uxw64VhL mIFg3wIQyvHu5ZOXeLtqfRKh8hE7jKhdyCndAcs259ZoTQtFN6QW+3FUfOW0QVy9DNkl +h9EafZPS9AU+gnCaNSW8pVnSg3TU1x9A0f/E= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=K/T0SKuslLEwsMlrqTpRgszeAoA8fk3NvwG1MT9B4+0R2+BeeEKS1yzlawwu0dB8GE mTGerTJuUjVpUE716s4SWsZg9bivK8k013JiCq9MW0Q2TFs/FIshhJOmaVBNTA4SL69k bTWt3zfpxHZw5146fZcr+kd7AEfrok9OhD/TE= MIME-Version: 1.0 Received: by 10.151.13.9 with SMTP id q9mr4974927ybi.73.1295889273123; Mon, 24 Jan 2011 09:14:33 -0800 (PST) Received: by 10.147.182.20 with HTTP; Mon, 24 Jan 2011 09:14:33 -0800 (PST) In-Reply-To: References: <201953550.20110106221821@nitronet.pl> <57312439.20110107171430@nitronet.pl> <13247006.20110124020848@nitronet.pl> Date: Mon, 24 Jan 2011 09:14:33 -0800 Message-ID: From: Jack Vogel To: Brandon Gooch Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Pawel Tyll , Luigi Rizzo , freebsd-ipfw@freebsd.org, freebsd-net@freebsd.org Subject: Re: [Panic] Dummynet/IPFW related recurring crash. X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Jan 2011 17:36:59 -0000 Just replying so you know I'm seeing it, but something that takes 14 days to even happen is NOT going to be an easy one to find. As Brandon said, all the info you can provide please. Jack On Mon, Jan 24, 2011 at 9:11 AM, Brandon Gooch wrote: > On Sun, Jan 23, 2011 at 7:08 PM, Pawel Tyll wrote: > > On Fri, Jan 7, 2011, Brandon Gooch wrote: > >> It's likely that the mbuf handling problem (in em_refresh_mbufs()) is > >> triggered by the processing you're doing with ipfw (or elsewhere for > >> that matter), so, yes, I think it's a bug fixed in the revision > >> discussed. > > > >> When you update and test, please let us know. Also, don't forget to > >> submit a follow-up to your PR. > > > > Unfortunately bad news: > > > > Machine fell after 14 days, 22:31:42 for the same reason according to > > what was left of panic screen. It didn't do a dump, nor reboot as is > > customary since some time on S3420GP boards (and other Intel server > > boards, since colleague has dual-cpu board from same epoch). > > > > What can I try next? > > I'm not sure. Perhaps there is another code path in the em(4) driver > that leads to the same, bad-pointer state. I'll have to defer to Jack > Vogel or Luigi Rizzo for potential insight into the issue. > > Please provide as much information as possible (even if it comes down > to taking a photo of the console, or transcribing the backtrace). > > FYI, I haven't been able to test this on any of my setups yet... > > -Brandon > From owner-freebsd-ipfw@FreeBSD.ORG Mon Jan 24 19:14:25 2011 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EC5E91065674 for ; Mon, 24 Jan 2011 19:14:24 +0000 (UTC) (envelope-from ptyll@nitronet.pl) Received: from mail.nitronet.pl (smtp.nitronet.pl [195.90.106.27]) by mx1.freebsd.org (Postfix) with ESMTP id A74798FC17 for ; Mon, 24 Jan 2011 19:14:24 +0000 (UTC) Received: from mailnull by mail.nitronet.pl with virscan (Exim 4.72 (FreeBSD)) (envelope-from ) id 1PhRrr-000Au2-2m for freebsd-ipfw@freebsd.org; Mon, 24 Jan 2011 20:14:23 +0100 Date: Mon, 24 Jan 2011 20:14:14 +0100 From: Pawel Tyll X-Priority: 3 (Normal) Message-ID: <814029043.20110124201414@nitronet.pl> To: Jack Vogel In-Reply-To: References: <201953550.20110106221821@nitronet.pl> <57312439.20110107171430@nitronet.pl> <13247006.20110124020848@nitronet.pl> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-Virus-Scanned: Nitronet.pl X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: ptyll@nitronet.pl X-SA-Exim-Scanned: No (on mail.nitronet.pl); SAEximRunCond expanded to false Cc: Brandon Gooch , freebsd-ipfw@freebsd.org, Luigi Rizzo , freebsd-net@freebsd.org Subject: Re: [Panic] Dummynet/IPFW related recurring crash. X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Jan 2011 19:14:25 -0000 > Just replying so you know I'm seeing it, but something that takes 14 days= to > even happen > is NOT going to be an easy one to find. As Brandon said, all the info you > can provide please. Here's the dump in case you've not seen it before. Somehow 8.1-RELEASE managed to make a proper dump, which became impossible later on. http://www.freebsd.org/cgi/query-pr.cgi?pr=3D152360 I strongly feel that it's related to dummynet. Not only panic seems to always be pointing at it, but also this is the only one of four identical machines, that crashes (and also only one that uses dummynet). From it's neighbor: up 166 days, 18:45 (FreeBSD 8.1-RELEASE) There's also this problem with fail to reboot after panic, and failure to dump properly. I think I have one more spare box laying around somewhere, so I will look into it. I can trace all this panic business back to one thing I started doing: # ipfw pipe list | grep flows | wc -l 2318 # crontab -l (...) */1 * * * * /root/fw/pipestats.sh (...) # cat /root/fw/pipestats.sh /sbin/ipfw pipe list > pipestats-`date "+%Y%m%d-%H%M%S"` Maybe something overflows here? Don't know :( Ruleset itself seems to be as simple as it gets: 00010 1397 106004 deny ip from any to any not antispoof in 00020 70490095 5395475808 fwd [...] ip from table(60) to table(61) 00050 3741 173481 allow tcp from any to [...] dst-port 53 00051 1868319 195628380 allow udp from any to [...] 00059 19993 1043277 deny ip from any to [...] 00100 603380 33725224 deny ip from any to table(10) dst-port 131-13= 9,445 00102 21201 874326 fwd [...] tcp from table(1) to not table(5) d= st-port 80 00103 0 0 fwd [...] tcp from table(2) to not table(5) d= st-port 80 00104 31 2196 fwd [...] tcp from table(3) to not table(5) 00105 4577 296736 deny ip from table(3) to not table(5) 30000 299626026 144893738712 pipe tablearg ip from table(100) to any in 30001 349984632 312762616666 pipe tablearg ip from any to table(101) out 34900 6724440 1768229912 skipto 35001 ip from table(10) to table(10) 35000 344337771 135015696767 fwd [...] ip from 192.168.0.0/16 to not 192.1= 68.0.0/16 65534 1118791481 888359380351 allow ip from any to any 65535 0 0 allow ip from any to any Two weeks seems to be rather strange. It never happens much earlier, and I don't remember panics much later. Too bad I didn't install uptimed back then, would have at least 10 panics recorded now ;) There's something elusive somewhere here, and sad part is it doesn't happen to others. Machine itself isn't really heavy loaded, processing 40-60kpps. Thank you for your interest, I very much appreciate it.