Date: Thu, 22 Jan 2004 10:50:41 -0800 (PST) From: Roselyn Lee <rosel@verniernetworks.com> To: freebsd-gnats-submit@FreeBSD.org Subject: misc/61744: TCP hangs onto mbufs with no tcp data unnecessily under certain error conditions Message-ID: <200401221850.i0MIofn2006578@www.freebsd.org> Resent-Message-ID: <200401221900.i0MJ0aWd011114@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 61744
>Category: misc
>Synopsis: TCP hangs onto mbufs with no tcp data unnecessily under certain error conditions
>Confidential: no
>Severity: non-critical
>Priority: low
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Thu Jan 22 11:00:35 PST 2004
>Closed-Date:
>Last-Modified:
>Originator: Roselyn Lee
>Release: 4.8a1
>Organization:
Vernier Networks
>Environment:
4.8-RELEASE FreeBSD 4.8-RELEASE #0
>Description:
In tcp_reass(), if we get a duplicate ACK with len = 0, we queue the mbuf. This is not necessary because there is no additional data in this mbuf that's useful for this tcp session. Under certain error conditions, we've seen VMware clients running freebsd send thousands of FIN/ACKs with duplicate sequence #'s and tcp len = 0. These result in mbufs being held on the system until the socket is closed. Sometimes this could result in mbuf exhaustion on the system.
>How-To-Repeat:
In our environment, we can reproduce the the problem as follows:
1. setup a freebsd pptp client (mpd) to connect to the freebsd server but have it use the wrong password so it will keep trying to talk.
2. Have the client 'disappear' for seconds to minutes and 'reappear' to the server. We do this by making the client roam away out of range.
3. Sometimes, this will result in thousands of FIN/ACKs with same seq#.
Here is a partial tcpdump summary when the error condition is reached. Starting at froamd 2740, the FIN/ACK exchange continues until frame 20877 - looks like the client roams out of range at this point and returns a few seconds later. The tcpdump was filtered for this client only and so there a some gaps in the seq. But 90% of the packets captured from frames 2740 to 20877 were for this client.
2562 13:31:34.3043 42.0.0.1 42.171.25.102 PPP CHAP PPP CHAP Failure 127 2562
2563 13:31:34.3078 42.0.0.1 42.171.25.102 PPP LCP PPP LCP Termination Request 60 2563
2564 13:31:34.3113 42.171.25.102 42.0.0.1 PPTP CLEAR-CALL-REQUEST 70 2564
2565 13:31:34.3140 42.0.0.1 42.171.25.102 PPTP DISCONNECT-NOTIFY 202 2565
2566 13:31:34.3156 42.0.0.1 42.171.25.102 PPTP STOP-CONTROL-REQUEST 70 2566
2567 13:31:34.3174 42.171.25.102 42.0.0.1 TCP 1194 > pptp [ACK] Seq=3599677643 Ack=1998493514 Win=58236 Len=0 60 2567
2568 13:31:34.3231 42.171.25.102 42.0.0.1 PPTP STOP-CONTROL-REQUEST 70 2568
2569 13:31:34.3266 42.171.25.102 42.0.0.1 PPTP STOP-CONTROL-REPLY 70 2569
2570 13:31:34.3268 42.0.0.1 42.171.25.102 TCP pptp > 1194 [ACK] Seq=1998493514 Ack=3599677675 Win=65535 Len=0 60 2570
2571 13:31:34.3272 42.0.0.1 42.171.25.102 PPTP STOP-CONTROL-REPLY 70 2571
2572 13:31:34.3274 42.0.0.1 42.171.25.102 TCP pptp > 1194 [FIN, ACK] Seq=1998493530 Ack=3599677675 Win=65535 Len=0 60 2572
2573 13:31:34.3286 42.171.25.102 42.0.0.1 TCP 1194 > pptp [ACK] Seq=3599677675 Ack=1998493531 Win=58384 Len=0 60 2573
2574 13:31:34.3304 42.171.25.102 42.0.0.1 TCP 1194 > pptp [FIN, ACK] Seq=3599677675 Ack=1998493531 Win=58400 Len=0 60 2574
2575 13:31:34.3305 42.0.0.1 42.171.25.102 TCP pptp > 1194 [ACK] Seq=1998493531 Ack=3599677676 Win=65535 Len=0 60 2575
2739 13:31:40.3816 42.0.0.1 42.171.25.102 TCP pptp > 1193 [ACK] Seq=1565198537 Ack=2686410455 Win=32942 Len=0 60 2739
2740 13:31:40.3835 42.171.25.102 42.0.0.1 TCP 1193 > pptp [FIN, ACK] Seq=2686410655 Ack=1565198538 Win=57920 Len=0 66 2740
2741 13:31:40.3836 42.0.0.1 42.171.25.102 TCP pptp > 1193 [FIN, ACK] Seq=1565198702 Ack=2686410455 Win=32942 Len=0 66 2741
2742 13:31:40.3862 42.171.25.102 42.0.0.1 TCP 1193 > pptp [FIN, ACK] Seq=2686410655 Ack=1565198538 Win=57920 Len=0 66 2742
2743 13:31:40.3862 42.0.0.1 42.171.25.102 TCP pptp > 1193 [FIN, ACK] Seq=1565198702 Ack=2686410455 Win=32942 Len=0 66 2743
2744 13:31:40.3885 42.171.25.102 42.0.0.1 TCP 1193 > pptp [FIN, ACK] Seq=2686410655 Ack=1565198538 Win=57920 Len=0 66 2744
2745 13:31:40.3885 42.0.0.1 42.171.25.102 TCP pptp > 1193 [FIN, ACK] Seq=1565198702 Ack=2686410455 Win=32942 Len=0 66 2745
....
20877 13:31:49.3741 42.171.25.102 42.0.0.1 TCP 1193 > pptp [FIN, ACK] Seq=2686410655 Ack=1565198538 Win=57920 Len=0 66 20877
20878 13:31:49.3741 42.0.0.1 42.171.25.102 TCP pptp > 1193 [FIN, ACK] Seq=1565198702 Ack=2686410455 Win=32942 Len=0 66 20878
21192 13:31:56.2532 42.171.25.102 42.0.0.1 TCP 1196 > pptp [SYN] Seq=926832057 Ack=0 Win=57344 Len=0 74 21192
21193 13:31:56.2534 42.0.0.1 42.171.25.102 TCP pptp > 1196 [SYN, ACK] Seq=1341705398 Ack=926832058 Win=65535 Len=0 74 21193
21194 13:31:56.2549 42.171.25.102 42.0.0.1 TCP 1196 > pptp [ACK] Seq=926832058 Ack=1341705399 Win=57920 Len=0 66 21194
21195 13:31:56.2550 42.0.0.1 42.171.25.102 TCP pptp > 1196 [ACK] Seq=1341705399 Ack=926832058 Win=32942 Len=0 66 21195
>Fix:
The following patch to tcp_input.c's tcp_reass() routine seems to do the trick.
--- freebsd/sys/netinet/tcp_input.c 5 Jun 2003 02:12:11 -0000 1.3
+++ freebsd/sys/netinet/tcp_input.c 22 Jan 2004 17:52:54 -0000 1.4
@@ -219,20 +219,29 @@
* Try to present any queued data
* at the left window edge to the user.
* This is needed after the 3-WHS
* completes.
*/
goto present; /* ??? */
}
m_adj(m, i);
*tlenp -= i;
th->th_seq += i;
+ } else if (i == 0 && p->tqe_len == 0 && *tlenp == 0) {
+ /*
+ * This means we're getting the same seq# and
+ * there is no new data. Just drop it.
+ */
+ tcpstat.tcps_rcvduppack++;
+ m_freem(m);
+ free(te, M_TSEGQ);
+ return (0);
}
}
tcpstat.tcps_rcvoopack++;
tcpstat.tcps_rcvoobyte += *tlenp;
/*
* While we overlap succeeding segments trim them or,
* if they are completely covered, dequeue them.
*/
while (q) {
>Release-Note:
>Audit-Trail:
>Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200401221850.i0MIofn2006578>