Date: Sun, 2 Aug 2009 20:06:45 +1000 From: Peter Jeremy <peterjeremy@optushome.com.au> To: freebsd-bugs@freebsd.org, freebsd-net@freebsd.org Subject: Re: kern/137317: [tcp] logs full of syncache problems Message-ID: <20090802100644.GA74485@server.vk2pj.dyndns.org> In-Reply-To: <200907312352.n6VNqEge015819@freefall.freebsd.org> References: <200907312352.n6VNqEge015819@freefall.freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--VbJkn9YxBvnuCH5J Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable I've run into what appears to be the same issue. Approaching it from the other direction: I have a pair on boxes running 7.2-RELEASE/i386 and have found that they are generating stray RST packets. As an example, ssh server echo hello will generate a sequence like the following (as seen on the server): 10:03:29.277427 IP client.55871 > server.22: S 1808736437:1808736437(0) win= 49640 <mss 1460,nop,wscale 0,nop,nop,sackOK> 10:03:29.277674 IP server.22 > client.55871: S 3948204651:3948204651(0) ack= 1808736438 win 65535 <mss 1460,nop,wscale 3,sackOK,eol> 10:03:29.278202 IP client.55871 > server.22: . ack 3948204652 win 49640 =2E.. 10:03:30.078227 IP client.55871 > server.22: . ack 3948207204 win 49640 10:03:30.078637 IP client.55871 > server.22: P 1808738140:1808738172(32) ac= k 3948207204 win 49640 10:03:30.078732 IP client.55871 > server.22: F 1808738172:1808738172(0) ack= 3948207204 win 49640 10:03:30.078751 IP server.22 > client.55871: . ack 1808738173 win 8212 10:03:30.079135 IP server.22 > client.55871: F 3948207204:3948207204(0) ack= 1808738173 win 8212 10:03:30.079528 IP client.55871 > server.22: . ack 3948207205 win 49640 10:03:32.798071 IP server.22 > client.55871: R 3948207204:3948207204(0) win= 0 10:03:32.798086 IP server.22 > client.55871: . ack 1808738173 win 0 10:03:32.798518 IP client.55871 > server.22: . ack 3948207205 win 49640 10:03:32.798608 IP server.22 > client.55871: R 3948207205:3948207205(0) win= 0 The delay between the FIN/ACK and subsequent RST is normally about a second but varies between about 150msec and 3 seconds (and occasionally it doesn't generate a RST at all). When there are two sessions close together, RSTs for both sessions may be generated together: 10:37:25.345622 IP client.58124 > server.22: S 2602521411:2602521411(0) win= 49640 <mss 1460,nop,wscale 0,nop,nop,sackOK> 10:37:25.345818 IP server.22 > client.58124: S 954205035:954205035(0) ack 2= 602521412 win 65535 <mss 1460,nop,wscale 3,sackOK,eol> 10:37:25.346400 IP client.58124 > server.22: . ack 954205036 win 49640 =2E.. 10:37:26.012584 IP client.58124 > server.22: . ack 954207508 win 49640 10:37:26.013295 IP client.58124 > server.22: P 2602523114:2602523146(32) ac= k 954207604 win 49640 10:37:26.013391 IP client.58124 > server.22: F 2602523146:2602523146(0) ack= 954207604 win 49640 10:37:26.013421 IP server.22 > client.58124: . ack 2602523147 win 8208 10:37:26.014119 IP server.22 > client.58124: F 954207604:954207604(0) ack 2= 602523147 win 8208 10:37:26.014493 IP client.58124 > server.22: . ack 954207605 win 49640 10:37:27.264149 IP client.58128 > server.22: S 2603683222:2603683222(0) win= 49640 <mss 1460,nop,wscale 0,nop,nop,sackOK> 10:37:27.264229 IP server.22 > client.58128: S 2795742829:2795742829(0) ack= 2603683223 win 65535 <mss 1460,nop,wscale 3,sackOK,eol> 10:37:27.264630 IP client.58128 > server.22: . ack 2795742830 win 49640 =2E.. 10:37:27.889755 IP client.58128 > server.22: . ack 2795745382 win 49640 10:37:27.890051 IP client.58128 > server.22: P 2603684941:2603684973(32) ac= k 2795745382 win 49640 10:37:27.890146 IP client.58128 > server.22: F 2603684973:2603684973(0) ack= 2795745382 win 49640 10:37:27.890203 IP server.22 > client.58128: . ack 2603684974 win 8212 10:37:27.890924 IP server.22 > client.58128: F 2795745382:2795745382(0) ack= 2603684974 win 8212 10:37:27.891353 IP client.58128 > server.22: . ack 2795745383 win 49640 10:37:28.038288 IP server.22 > client.58124: R 954207604:954207604(0) win 0 10:37:28.038353 IP server.22 > client.58124: . ack 2602523147 win 0 10:37:28.038543 IP server.22 > client.58128: R 2795745382:2795745382(0) win= 0 10:37:28.038556 IP server.22 > client.58128: . ack 2603684974 win 0 10:37:28.038754 IP client.58124 > server.22: . ack 954207605 win 49640 10:37:28.038869 IP server.22 > client.58124: R 954207605:954207605(0) win 0 10:37:28.038887 IP client.58128 > server.22: . ack 2795745383 win 49640 10:37:28.038984 IP server.22 > client.58128: R 2795745383:2795745383(0) win= 0 The systems are using ipfw and have dummynet in the kernel but it isn't used. The type of client doesn't seem to matter (same behaviour with FreeBSD or Solaris clients) and it's not limited to ssh (that was just where I first noticed it). In conjection with the stray RST packets, I am also seeing TCP error messages in syslog: Jul 29 10:03:32 aalp05 kernel: TCP: [client]:55871 to [server]:22 tcpflags = 0x10<ACK>; syncache_expand: Segment failed SYNCOOKIE authentication, segmen= t rejected (probably spoofed) Jul 29 10:03:32 aalp05 kernel: TCP: [client]:55871 to [server]:22 tcpflags = 0x10<ACK>; syncache_expand: Segment failed SYNCOOKIE authentication, segmen= t rejected (probably spoofed) Jul 29 10:37:28 aalp05 kernel: TCP: [client]:58124 to [server]:22 tcpflags = 0x10<ACK>; syncache_expand: Segment failed SYNCOOKIE authentication, segmen= t rejected (probably spoofed) Jul 29 10:37:28 aalp05 kernel: TCP: [client]:58128 to [server]:22 tcpflags = 0x10<ACK>; syncache_expand: Segment failed SYNCOOKIE authentication, segmen= t rejected (probably spoofed) Jul 29 10:37:28 aalp05 kernel: TCP: [client]:58124 to [server]:22 tcpflags = 0x10<ACK>; syncache_expand: Segment failed SYNCOOKIE authentication, segmen= t rejected (probably spoofed) Jul 29 10:37:28 aalp05 kernel: TCP: [client]:58128 to [server]:22 tcpflags = 0x10<ACK>; syncache_expand: Segment failed SYNCOOKIE authentication, segmen= t rejected (probably spoofed) Note that the syslog message implies there is an incoming packet but tcpdump doesn't show one. --=20 Peter Jeremy --VbJkn9YxBvnuCH5J Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.12 (FreeBSD) iEYEARECAAYFAkp1ZTQACgkQ/opHv/APuIe1wQCePoKc9wDchWOwgCx4RisjwYu5 W9sAoJEFs+oZlGmoyvMFjz3AbSnt92of =EoN7 -----END PGP SIGNATURE----- --VbJkn9YxBvnuCH5J--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20090802100644.GA74485>