From owner-freebsd-pf@FreeBSD.ORG Thu Mar 3 17:45:13 2005 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 82A6816A4CE for ; Thu, 3 Mar 2005 17:45:13 +0000 (GMT) Received: from zixvpm01.seton.org (zixvpm01.seton.org [207.193.126.161]) by mx1.FreeBSD.org (Postfix) with ESMTP id C03D543D5C for ; Thu, 3 Mar 2005 17:45:12 +0000 (GMT) (envelope-from mgrooms@seton.org) Received: from zixvpm01.seton.org (ZixVPM [127.0.0.1]) by Outbound.seton.org (Proprietary) with ESMTP id 56C9F360081 for ; Thu, 3 Mar 2005 11:45:12 -0600 (CST) Received: from smtp-out.seton.org (unknown [10.21.254.249]) by zixvpm01.seton.org (Proprietary) with ESMTP id 98A11330057; Thu, 3 Mar 2005 11:45:09 -0600 (CST) Received: from localhost (unknown [127.0.0.1]) by smtp-out.seton.org (Postfix) with ESMTP id 70CA18014E24; Thu, 3 Mar 2005 11:45:09 -0600 (CST) Received: from smtp-out.seton.org ([10.21.254.249]) by localhost (mail [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 21381-16; Thu, 3 Mar 2005 11:45:09 -0600 (CST) Received: from ausexfe01.seton.org (ausexfe01.seton.org [10.20.10.211]) by smtp-out.seton.org (Postfix) with ESMTP id 579018014E23; Thu, 3 Mar 2005 11:45:09 -0600 (CST) Received: from [10.20.160.190] ([10.20.160.190]) by ausexfe01.seton.org with Microsoft SMTPSVC(6.0.3790.211); Thu, 3 Mar 2005 11:45:09 -0600 Message-ID: <42274E08.7050404@seton.org> Date: Thu, 03 Mar 2005 11:48:56 -0600 From: Matthew Grooms Organization: Seton Healthcare Network User-Agent: Mozilla Thunderbird 1.0 (Windows/20041206) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Daniel Hartmeier References: <200502282232.17646.max@love2party.net> <4223931C.9000607@seton.org> <200502282326.41760.max@love2party.net> <4224B078.9020301@seton.org> <20050301185431.GA81982@cell.sick.ru> <4225174C.801@seton.org> <20050302081051.GB87159@cell.sick.ru> <422600A2.2080907@seton.org> <20050302191656.GA93112@cell.sick.ru> <42264A0A.1090301@seton.org> <20050303013807.GH25140@insomnia.benzedrine.cx> In-Reply-To: <20050303013807.GH25140@insomnia.benzedrine.cx> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-OriginalArrivalTime: 03 Mar 2005 17:45:09.0092 (UTC) FILETIME=[BE1ECA40:01C52018] X-Virus-Scanned: by amavisd-new at seton.org cc: Gleb Smirnoff cc: freebsd-pf@freebsd.org Subject: Re: Fwd: pf + pfsync + carp testing ... X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Technical discussion and general questions about packet filter (pf) List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Mar 2005 17:45:13 -0000 Daniel, Please let me know if this is not what you want. I will try to do what it takes to get you any data that you may require. The stalled connection is coming from 192.168.254.51 to 192.168.251.100:80. Sorry for not paring it down but I did not want to cut out something you may want to see due to ignorance on my part. I will prepare the other output you requested unless I hear back from you first. example 1 - fw1 - pfctl -vvss self tcp 192.168.254.2:22 <- 192.168.254.51:4461 ESTABLISHED:ESTABLISHED [895578000 + 63960] [3194607704 + 65483] age 00:02:33, expires in 24:00:00, 511:579 pkts, 49580:61016 bytes, rule 4 id: 4226ef910000001f creatorid: 5357f190 self tcp 192.168.254.3:22 -> 192.168.254.51:4462 ESTABLISHED:ESTABLISHED [1673568462 + 63104] [3196457500 + 65535] age 00:02:42, expires in 23:59:11, 0:0 pkts, 0:0 bytes id: 4226ef8800000018 creatorid: 5357f190 self tcp 192.168.251.100:80 <- 192.168.254.51:4469 TIME_WAIT:TIME_WAIT [32533272 + 272] [3248810405 + 65535] age 00:02:10, expires in 00:01:12, 85852:161987 pkts, 3434080:242936092 bytes id: 4226ef8800000019 creatorid: 5357f190 self tcp 192.168.251.100:80 <- 192.168.254.51:4470 ESTABLISHED:ESTABLISHED [2020265516 + 64512] [3277486902 + 65535] age 00:00:16, expires in 23:59:57, 22968:43193 pkts, 919669:64635019 bytes, rule 4 id: 4226ef9100000022 creatorid: 5357f190 self tcp 192.168.254.51:4469 -> 192.168.251.100:80 TIME_WAIT:TIME_WAIT [3248810405 + 65535] [32533272 + 272] age 00:02:10, expires in 00:01:12, 85852:161987 pkts, 3434080:242936092 bytes id: 4226ef880000001a creatorid: 5357f190 self tcp 192.168.254.51:4470 -> 192.168.251.100:80 ESTABLISHED:ESTABLISHED [3277486902 + 65535] [2020265516 + 64512] age 00:00:16, expires in 23:59:57, 22968:43193 pkts, 919669:64635019 bytes, rule 4 id: 4226ef9100000023 creatorid: 5357f190 self tcp 192.168.253.1:62481 <- 64.233.187.104:80 TIME_WAIT:TIME_WAIT [3223153423 + 8190] [2943726748 + 2] age 00:00:41, expires in 00:00:49, 1:1 pkts, 40:40 bytes, rule 4 id: 4226ef9100000021 creatorid: 5357f190 example 1 - fw2 - pfctl -vvss self tcp 192.168.254.2:22 <- 192.168.254.51:4461 ESTABLISHED:ESTABLISHED [895580236 + 63532] [3194608276 + 65535] age 00:02:35, expires in 23:59:58, 0:0 pkts, 0:0 bytes id: 4226ef910000001f creatorid: 5357f190 self tcp 192.168.254.3:22 -> 192.168.254.51:4462 ESTABLISHED:ESTABLISHED [1673568634 + 64408] [3196457656 + 65535] age 00:02:44, expires in 24:00:00, 227:206 pkts, 37788:12244 bytes, rule 4 id: 4226ef8800000018 creatorid: 5357f190 self tcp 192.168.251.100:80 <- 192.168.254.51:4469 TIME_WAIT:TIME_WAIT [32533272 + 272] [3248810405 + 65535] age 00:02:13, expires in 00:01:11, 155592:293304 pkts, 6224629:439872827 bytes, rule 4 id: 4226ef8800000019 creatorid: 5357f190 self tcp 192.168.251.100:80 <- 192.168.254.51:4470 ESTABLISHED:ESTABLISHED [2016193576 + 51372] [3277486902 + 65535] age 00:00:18, expires in 23:59:54, 1479:2789 pkts, 59160:4183500 bytes id: 4226ef9100000022 creatorid: 5357f190 self tcp 192.168.254.51:4469 -> 192.168.251.100:80 TIME_WAIT:TIME_WAIT [3248810405 + 65535] [32533272 + 272] age 00:02:13, expires in 00:01:11, 155592:293304 pkts, 6224629:439872827 bytes, rule 4 id: 4226ef880000001a creatorid: 5357f190 self tcp 192.168.254.51:4470 -> 192.168.251.100:80 ESTABLISHED:ESTABLISHED [3277486902 + 65535] [2016193576 + 51372] age 00:00:18, expires in 23:59:54, 1479:2789 pkts, 59160:4183500 bytes id: 4226ef9100000023 creatorid: 5357f190 self tcp 192.168.253.1:62481 <- 64.233.187.104:80 TIME_WAIT:TIME_WAIT [3223153423 + 8190] [2943726748 + 2] age 00:00:43, expires in 00:00:47, 0:0 pkts, 0:0 bytes id: 4226ef9100000021 creatorid: 5357f190 example 2 - fw1 - pfctl -vvss self tcp 192.168.254.2:22 <- 192.168.254.51:4461 ESTABLISHED:ESTABLISHED [895581492 + 63756] [3194610408 + 65483] age 00:05:55, expires in 24:00:00, 560:633 pkts, 54244:66668 bytes, rule 4 id: 4226ef910000001f creatorid: 5357f190 self tcp 192.168.254.3:22 -> 192.168.254.51:4462 ESTABLISHED:ESTABLISHED [1673570802 + 63856] [3196458072 + 65535] age 00:06:04, expires in 23:56:41, 0:0 pkts, 0:0 bytes id: 4226ef8800000018 creatorid: 5357f190 self tcp 192.168.251.100:80 <- 192.168.254.51:4470 FIN_WAIT_2:FIN_WAIT_2 [2629520121 + 64512] [3277486903 + 65535] age 00:03:38, expires in 00:01:04, 244235:460539 pkts, 9770349:690583463 bytes, rule 4 id: 4226ef9100000022 creatorid: 5357f190 self tcp 192.168.251.100:80 <- 192.168.254.51:4472 ESTABLISHED:ESTABLISHED [679995100 + 272] [3327911464 + 65535] age 00:00:16, expires in 23:59:57, 25897:49715 pkts, 1036349:74514782 bytes, rule 4 id: 4226ef9100000027 creatorid: 5357f190 self tcp 192.168.254.51:4470 -> 192.168.251.100:80 FIN_WAIT_2:FIN_WAIT_2 [3277486903 + 65535] [2629520121 + 64512] age 00:03:38, expires in 00:01:04, 244235:460539 pkts, 9770349:690583463 bytes, rule 4 id: 4226ef9100000023 creatorid: 5357f190 self tcp 192.168.254.51:4472 -> 192.168.251.100:80 ESTABLISHED:ESTABLISHED [3327911464 + 65535] [679995100 + 272] age 00:00:16, expires in 23:59:57, 25897:49715 pkts, 1036349:74514782 bytes, rule 4 id: 4226ef9100000028 creatorid: 5357f190 example 2 - fw2 - pfctl -vvss self tcp 192.168.254.2:22 <- 192.168.254.51:4461 ESTABLISHED:ESTABLISHED [895583468 + 63448] [3194610928 + 65535] age 00:05:57, expires in 23:59:59, 0:0 pkts, 0:0 bytes id: 4226ef910000001f creatorid: 5357f190 self tcp 192.168.254.3:22 -> 192.168.254.51:4462 ESTABLISHED:ESTABLISHED [1673570974 + 63684] [3196458384 + 65483] age 00:06:06, expires in 24:00:00, 244:219 pkts, 40808:13492 bytes, rule 4 id: 4226ef8800000018 creatorid: 5357f190 self tcp 192.168.251.100:80 <- 192.168.254.51:4470 FIN_WAIT_2:FIN_WAIT_2 [2629520121 + 64512] [3277486903 + 65535] age 00:03:40, expires in 00:01:03, 1479:2789 pkts, 59160:4183500 bytes id: 4226ef9100000022 creatorid: 5357f190 self tcp 192.168.251.100:80 <- 192.168.254.51:4472 ESTABLISHED:ESTABLISHED [673471820 + 35312] [3327911464 + 65535] age 00:00:18, expires in 23:59:54, 2370:4468 pkts, 94800:6702000 bytes id: 4226ef9100000027 creatorid: 5357f190 self tcp 192.168.254.51:4470 -> 192.168.251.100:80 FIN_WAIT_2:FIN_WAIT_2 [3277486903 + 65535] [2629520121 + 64512] age 00:03:40, expires in 00:01:03, 1479:2789 pkts, 59160:4183500 bytes id: 4226ef9100000023 creatorid: 5357f190 self tcp 192.168.254.51:4472 -> 192.168.251.100:80 ESTABLISHED:ESTABLISHED [3327911464 + 65535] [673471820 + 35312] age 00:00:18, expires in 23:59:54, 2370:4468 pkts, 94800:6702000 bytes id: 4226ef9100000028 creatorid: 5357f190 Matthew Grooms Network Engineer Seton Healthcare Network mgrooms@seton.org (512) 324 9913 Daniel Hartmeier wrote: > On Wed, Mar 02, 2005 at 05:19:38PM -0600, Matthew Grooms wrote: > > >> On a slightly more depressing note, I don't think that state via >>pfsync seems to be working right between the two firewalls. Sometimes ( >>maybe every 1 out of 4 tries ) when the interfaces fail over, the >>traffic flow stops. The reason why I believe it is a state sync issue is >>that new connections can always be opened even while the previously >>opened connections are stalled. This doesn't always happen when an >>interface is going down either. It happens just as often when an >>interface is coming back up and reclaims a MASTER state. Any ideas? > > > It would help isolate the problem if you can provide the output of pfctl > -vvss for one such stalling connection on both boxes, for comparison. > > The obvious requirement is that the state is actually present on the > secondary box. If it is present, maybe we spot an inconsistency between > the two state entries. If they look the same, maybe you can get a > tcpdump -vvvS for the stalled connection (which matches the state > entry). > > If the state is not present on the secondary, a tcpdump -nvvvei pfsync0 > over the time between when the state was created on the primary and when > it should have arrived at the secondary would help. > > Daniel