From owner-freebsd-ipfw@FreeBSD.ORG Mon Nov 27 20:35:45 2006 Return-Path: X-Original-To: freebsd-ipfw@freebsd.org Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A69EF16A509 for ; Mon, 27 Nov 2006 20:35:45 +0000 (UTC) (envelope-from donald.teed@gmail.com) Received: from nz-out-0102.google.com (nz-out-0102.google.com [64.233.162.193]) by mx1.FreeBSD.org (Postfix) with ESMTP id E74D444887 for ; Mon, 27 Nov 2006 20:02:19 +0000 (GMT) (envelope-from donald.teed@gmail.com) Received: by nz-out-0102.google.com with SMTP id i11so712546nzh for ; Mon, 27 Nov 2006 12:03:04 -0800 (PST) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:to:subject:mime-version:content-type; b=URSRhfgpeUsSyIDJWGP99BsRdNR797zj019Vqbw3B66MnpjdB/69sBoCrUX0XszUEIJ/D2DhBnJBq6jWohJmTQBtuwstdAQQq+KrSFshNdPSeQEz1GAc3Zfa+jHYU4rQHSsZ0eZpk3jzXGVCA32Qih7ZhteCqZb7ESXEakuznRA= Received: by 10.78.138.6 with SMTP id l6mr13584544hud.1164657401064; Mon, 27 Nov 2006 11:56:41 -0800 (PST) Received: by 10.78.159.6 with HTTP; Mon, 27 Nov 2006 11:56:41 -0800 (PST) Message-ID: Date: Mon, 27 Nov 2006 15:56:41 -0400 From: "D G Teed" To: freebsd-ipfw@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: how to go about diagnosing cause of packet loss X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Nov 2006 20:35:45 -0000 Howdy, Lately we have been seeing increased packet loss on our firewall. Running a ping plotter outside of the firewall shows the hops are running clean. >From on or behind the firewall, we have 20 to 50% packet loss to each hop, reaching several popular test destinations. e.g.: $ mtr -c 100 -r www.cnn.com HOST: Loss% Snt Last Avg Best Wrst StDev 1. vlan-136.acadiau.ca 0.0% 100 0.4 6.1 0.4 179.9 26.5 2. silverhorde.acadiau.ca 4.0% 100 0.6 0.9 0.3 7.8 1.0 3. wfvlnsauh05-fe-0-0.aliant.ne 17.0% 100 3.4 6.3 2.6 55.0 8.8 4. hlfxns01h29-ge-4-0.aliant.ne 27.0% 100 3.6 3.8 2.5 12.4 1.4 5. rtp629049rts 15.0% 100 4.2 4.0 2.6 9.1 1.2 6. core1-halifax_POS5-0.net.bel 22.0% 100 6.2 3.7 2.6 6.2 0.9 7. core3-montrealak_pos1-1.net. 4.0% 100 24.2 26.8 20.3 126.2 19.2 8. core1-newyork83_pos_5_0_0.ne 19.0% 100 26.1 26.9 26.0 34.1 1.2 9. bx4-newyork83_pos_2_0_0.net. 31.0% 100 27.7 28.1 27.1 30.1 0.8 10. pop1-nye-P8-1.atdn.net 9.0% 100 26.2 45.2 26.2 227.4 48.0 11. bb2-nye-P0-0.atdn.net 16.0% 100 29.0 31.1 26.3 178.2 19.4 12. bb2-vie-P12-0.atdn.net 14.0% 100 33.0 46.3 32.3 206.4 37.6 13. bb2-atm-P3-0.atdn.net 18.0% 100 42.9 44.9 42.5 106.6 9.7 14. ??? 100.0 100 0.0 0.0 0.0 0.0 0.0 We suspect something in FreeBSD or ipfw has a flaw, but cannot find it. Running mtr from the firewall itself produces slightly different packet loss points than one hop behind the firewall running mtr. A reboot initially cleared up the problem, but 10 minutes later we saw the packet loss again, so I wonder if we are seeing some sort of saturation. Does anyone have suggestions no how to troubleshoot/resolve this problem? --Donald