From owner-freebsd-stable@FreeBSD.ORG Sun Jun 1 00:12:57 2003 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id EACAD37B401 for ; Sun, 1 Jun 2003 00:12:57 -0700 (PDT) Received: from pixies.tirloni.org (pixies.tirloni.org [200.203.183.37]) by mx1.FreeBSD.org (Postfix) with ESMTP id BB16243F93 for ; Sun, 1 Jun 2003 00:12:56 -0700 (PDT) (envelope-from tirloni@tirloni.org) Received: by pixies.tirloni.org (Postfix, from userid 1000) id B7ED51E17D5; Sun, 1 Jun 2003 04:12:55 -0300 (BRT) Date: Sun, 1 Jun 2003 04:12:55 -0300 From: "Giovanni P. Tirloni" To: freebsd-stable@freebsd.org Message-ID: <20030601071255.GC123@pixies.tirloni.org> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline X-Info: http://www.tirloni.org User-Agent: Mutt/1.5.3i Subject: Very weird network behaviour with 4.7-RELEASE-p10 (large) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 01 Jun 2003 07:12:58 -0000 Hi, I have been doing some tests using hping2 and TCP SYN pings targeting local and remote hosts from two FreeBSD 4.7-RELEASE-p10 and one Linux 2.4.18 host. The three machines have the same hardware configuration and have been running for 6 months now. The average load isnt too high (usually 0.01 to 0.15) on the FreeBSD machines. Here is the output from hping2 (excuse me the line wrap), root:~# hping -S -p 80 -c 6 www.freebsd.org HPING www.freebsd.org (fxp0 216.136.204.117): S set, 40 headers + 0 data bytes len=46 ip=216.136.204.117 ttl=50 DF id=20997 sport=80 flags=SA seq=0 win=57344 rtt=212.6 ms len=46 ip=216.136.204.117 ttl=50 DF id=21140 sport=80 flags=SA seq=1 win=57344 rtt=212.3 ms len=46 ip=216.136.204.117 ttl=50 DF id=21301 sport=80 flags=SA seq=2 win=57344 rtt=212.3 ms DUP! len=46 ip=216.136.204.117 ttl=50 DF id=21494 sport=80 flags=SA seq=0 win=57344 rtt=3207.3 ms len=46 ip=216.136.204.117 ttl=50 DF id=21498 sport=80 flags=SA seq=3 win=57344 rtt=212.3 ms DUP! len=46 ip=216.136.204.117 ttl=50 DF id=21650 sport=80 flags=SA seq=1 win=57344 rtt=3209.4 ms --- www.freebsd.org hping statistic --- 5 packets tramitted, 6 packets received, -20% packet loss round-trip min/avg/max = 212.3/1211.0/3209.4 ms And tcpdump confirms this and shows more packets after hping2 was stopped, root@srv0-cta:~# tcpdump -n host 216.136.204.117 and port 80 tcpdump: listening on fxp0 03:55:07.989538 200.203.183.32.1288 > 216.136.204.117.80: S 852569665:852569665(0) win 512 03:55:08.202061 216.136.204.117.80 > 200.203.183.32.1288: S 3771704683:3771704683(0) ack 85256966603:55:11.229621 216.136.204.117.80 > 200.203.183.32.1291: S 2387314094:2387314094(0) ack 2741695 win 57344 (DF) 03:55:12.027408 200.203.183.32.1292 > 216.136.204.117.80: S 1532350261:1532350261(0) win 512 03:55:12.206721 216.136.204.117.80 > 200.203.183.32.1289: S 3305368856:3305368856(0) ack 1218353109 win 57344 (DF) 03:55:12.239377 216.136.204.117.80 > 200.203.183.32.1292: S 3378516967:3378516967(0) ack 1532350262 win 57344 (DF) 03:55:13.216677 216.136.204.117.80 > 200.203.183.32.1290: S 2652913389:2652913389(0) ack 422129860 win 57344 (DF) 03:55:14.226573 216.136.204.117.80 > 200.203.183.32.1291: S 2387314094:2387314094(0) ack 2741695 win 57344 (DF) 03:55:15.236648 216.136.204.117.80 > 200.203.183.32.1292: S 3378516967:3378516967(0) ack 1532350262 win 57344 (DF) 03:55:17.196452 216.136.204.117.80 > 200.203.183.32.1288: S 3771704683:3771704683(0) ack 852569666 win 57344 (DF) 03:55:18.206412 216.136.204.117.80 > 200.203.183.32.1289: S 3305368856:3305368856(0) ack 1218353109 win 57344 (DF) 03:55:19.216406 216.136.204.117.80 > 200.203.183.32.1290: S 2652913389:2652913389(0) ack 422129860 win 57344 (DF) 03:55:20.226331 216.136.204.117.80 > 200.203.183.32.1291: S 2387314094:2387314094(0) ack 2741695 win 57344 (DF) 03:55:21.236273 216.136.204.117.80 > 200.203.183.32.1292: S 3378516967:3378516967(0) ack 1532350262 win 57344 (DF) 03:55:29.195990 216.136.204.117.80 > 200.203.183.32.1288: S 3771704683:3771704683(0) ack 852569666 win 57344 (DF) 03:55:30.205987 216.136.204.117.80 > 200.203.183.32.1289: S 3305368856:3305368856(0) ack 1218353109 win 57344 (DF) 03:55:31.216041 216.136.204.117.80 > 200.203.183.32.1290: S 2652913389:2652913389(0) ack 422129860 win 57344 (DF) 03:55:32.225922 216.136.204.117.80 > 200.203.183.32.1291: S 2387314094:2387314094(0) ack 2741695 win 57344 (DF) 03:55:33.235869 216.136.204.117.80 > 200.203.183.32.1292: S 3378516967:3378516967(0) ack 1532350262 win 57344 (DF) Another different output from hping (strange rtt's), root@srv0-cta:~# hping -S -p 80 -c 8 www.freebsd.org HPING www.freebsd.org (fxp0 216.136.204.117): S set, 40 headers + 0 data bytes len=46 ip=216.136.204.117 ttl=50 DF id=38289 sport=80 flags=SA seq=0 win=57344 rtt=212.5 ms len=46 ip=216.136.204.117 ttl=50 DF id=38291 sport=80 flags=SA seq=0 win=57344 rtt=0.0 ms len=46 ip=216.136.204.117 ttl=50 DF id=38371 sport=80 flags=SA seq=1 win=57344 rtt=211.9 ms len=46 ip=216.136.204.117 ttl=50 DF id=38372 sport=80 flags=SA seq=0 win=57344 rtt=0.0 ms len=46 ip=216.136.204.117 ttl=50 DF id=38404 sport=80 flags=SA seq=2 win=57344 rtt=211.9 ms len=46 ip=216.136.204.117 ttl=50 DF id=38405 sport=80 flags=SA seq=0 win=57344 rtt=0.0 ms DUP! len=46 ip=216.136.204.117 ttl=50 DF id=38687 sport=80 flags=SA seq=0 win=57344 rtt=3207.3 ms len=46 ip=216.136.204.117 ttl=50 DF id=38702 sport=80 flags=SA seq=3 win=57344 rtt=212.0 ms --- www.freebsd.org hping statistic --- 4 packets tramitted, 8 packets received, -100% packet loss round-trip min/avg/max = 211.9/811.1/3207.3 ms And the tcpdump output, 04:00:57.419466 216.136.204.117.80 > 200.203.183.32.2381: S 1076154568:1076154568(0) ack 1439276851 win 57344 (DF) 04:00:58.201940 200.203.183.32.1995 > 216.136.204.117.80: S 264430648:264430648(0) win 512 04:00:58.414397 216.136.204.117.80 > 200.203.183.32.1995: S 2379854270:2379854270(0) ack 264430649 win 57344 (DF) 04:00:58.429518 216.136.204.117.80 > 200.203.183.32.2382: S 3830791941:3830791941(0) ack 451178042 win 57344 (DF) 04:00:59.206642 200.203.183.32.1996 > 216.136.204.117.80: S 1522574199:1522574199(0) win 512 04:00:59.418526 216.136.204.117.80 > 200.203.183.32.1996: S 3804689919:3804689919(0) ack 1522574200 win 57344 (DF) 04:00:59.439330 216.136.204.117.80 > 200.203.183.32.2383: S 1160193874:1160193874(0) ack 1028452909 win 57344 (DF) 04:01:00.216649 200.203.183.32.1997 > 216.136.204.117.80: S 442375938:442375938(0) win 512 04:01:00.428557 216.136.204.117.80 > 200.203.183.32.1997: S 3901380776:3901380776(0) ack 442375939 win 57344 (DF) 04:01:00.449324 216.136.204.117.80 > 200.203.183.32.2384: S 3552821723:3552821723(0) ack 1069527292 win 57344 (DF) 04:01:01.226660 200.203.183.32.1998 > 216.136.204.117.80: S 240094281:240094281(0) win 512 04:01:01.409234 216.136.204.117.80 > 200.203.183.32.1995: S 2379854270:2379854270(0) ack 264430649 win 57344 (DF) 04:01:01.438601 216.136.204.117.80 > 200.203.183.32.1998: S 4153685029:4153685029(0) ack 240094282 win 57344 (DF) 04:01:01.529193 216.136.204.117.80 > 200.203.183.32.2065: S 4017739092:4017739092(0) ack 1616120023 win 57344 (DF) 04:01:02.409213 216.136.204.117.80 > 200.203.183.32.1996: S 3804689919:3804689919(0) ack 1522574200 win 57344 (DF) 04:01:02.529165 216.136.204.117.80 > 200.203.183.32.2066: S 754183561:754183561(0) ack 1231654702 win 57344 (DF) 04:01:03.419286 216.136.204.117.80 > 200.203.183.32.1997: S 3901380776:3901380776(0) ack 442375939 win 57344 (DF) 04:01:03.539156 216.136.204.117.80 > 200.203.183.32.2067: S 1059652438:1059652438(0) ack 2093248177 win 57344 (DF) 04:01:04.429131 216.136.204.117.80 > 200.203.183.32.1998: S 4153685029:4153685029(0) ack 240094282 win 57344 (DF) 04:01:04.549163 216.136.204.117.80 > 200.203.183.32.2068: S 3136323399:3136323399(0) ack 723495504 win 57344 (DF) 04:01:07.409090 216.136.204.117.80 > 200.203.183.32.1995: S 2379854270:2379854270(0) ack 264430649 win 57344 (DF) 04:01:08.408993 216.136.204.117.80 > 200.203.183.32.1996: S 3804689919:3804689919(0) ack 1522574200 win 57344 (DF) 04:01:09.418980 216.136.204.117.80 > 200.203.183.32.1997: S 3901380776:3901380776(0) ack 442375939 win 57344 (DF) 04:01:10.429020 216.136.204.117.80 > 200.203.183.32.1998: S 4153685029:4153685029(0) ack 240094282 win 57344 (DF) 04:01:19.408629 216.136.204.117.80 > 200.203.183.32.1995: S 2379854270:2379854270(0) ack 264430649 win 57344 (DF) 04:01:20.408544 216.136.204.117.80 > 200.203.183.32.1996: S 3804689919:3804689919(0) ack 1522574200 win 57344 (DF) 04:01:21.418566 216.136.204.117.80 > 200.203.183.32.1997: S 3901380776:3901380776(0) ack 442375939 win 57344 (DF) 04:01:22.428427 216.136.204.117.80 > 200.203.183.32.1998: S 4153685029:4153685029(0) ack 240094282 win 57344 (DF) Some users have complaint about server timeouts since Monday (but I havent experienced the problems they seem to be having). At first I thought it was the switch but I don't see errors from the Linux machine. That happens when pinging (with hping -S -p 80), bsd1:fxp0 -> bsd2 bsd2:fxp0 -> bsd1 bsd1:lo0 -> bsd1:lo0 bsd2:lo0 -> bsd2:lo0 bsd1:fxp0 -> remote bsd2:fxp0 -> remote bsd1:fxp0 -> linux Any help is welcome and sorry if this email doesn't have all the information needed to solve this problem. I'll be happy to provide more details and do more tests if needed. Here is the output from ifconfig fxp0, fxp0: flags=8843 mtu 1500 inet 200.203.183.32 netmask 0xffffffc0 broadcast 200.203.183.63 inet 200.203.183.33 netmask 0xffffffff broadcast 200.203.183.33 inet 200.203.183.37 netmask 0xffffffff broadcast 200.203.183.37 ether 00:07:e9:ad:2a:ab media: Ethernet autoselect (100baseTX ) status: active Someone mentioned cvsup'ing to 4.8-STABLE but I'm afraid of doing so because those two boxes are very critical. -- Giovanni P. Tirloni http://www.tirloni.org