Date: Tue, 10 Jul 2007 00:20:49 -0500 (CDT) From: Mike Silbersack <silby@silby.com> To: current@freebsd.org, net@freebsd.org Cc: Robert Watson <rwatson@freebsd.org>, Andre Oppermann <andre@freebsd.org> Subject: FreeBSD 7 TCP syncache fix: request for testers Message-ID: <20070709234401.S29353@odysseus.silby.com>
next in thread | raw e-mail | index | archive | help
This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. --0-1185318269-1184043945=:29353 Content-Type: TEXT/PLAIN; CHARSET=US-ASCII; FORMAT=flowed Content-ID: <20070710000720.U29353@odysseus.silby.com> I've found one of the causes of the network instability of FreeBSD 7; the tcp syncache fails to retransmit SYN-ACK packets. This causes interesting problems when packet loss is experienced during connection setup. The symptoms that I have witnessed are twofold: 1. If the third part of the 3WHS is lost, the client will believe that the connection is in the ESTABLISHED state, while the server will still have the connection in the syncache. 2. Subsequently, the above syncache entry will stay stuck in the syncache forever. If you attempt to re-use that same 4-tuple, the syncache will ack the new SYN with the old sequence number. Anyway, the attached patch simplifies the syncache structure a bit and makes it retransmit properly. I'd appreciate testing from anyone who has experienced TCP problems with FreeBSD 7, as well as anyone who is pushing significant traffic through FreeBSD 7. I'm not interested in FreeBSD 6 testers, since the FreeBSD 6 syncache has a different structure and is not affected by this bug. FWIW, here's how to prove the existence of the bug. Install nemesis from ports, then use it to send SYN packets at your FreeBSD 7 machine. As of now, you should see only one SYN-ACK reply, and you should also notice that the sysctl net.inet.tcp.syncache.count goes up, but does not come back down. Once you have applied the patch, you should see the behavior demonstrated below: >From your client machine: (nemesis will pick an IP to spoof, change that if you wish.) nemesis tcp -y 80 -D 10.1.1.6 TCP Packet Injected On your FreeBSD 7 machine: patrocles# tcpdump -n port 80 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on nve0, link-type EN10MB (Ethernet), capture size 96 bytes 23:49:02.075118 IP 133.120.85.92.48922 > 10.1.1.6.80: S 1519649939:1519649939(0) win 4096 23:49:02.075165 IP 10.1.1.6.80 > 133.120.85.92.48922: S 269601671:269601671(0) ack 1519649940 win 65535 <mss 1460> 23:49:05.164195 IP 10.1.1.6.80 > 133.120.85.92.48922: S 269601671:269601671(0) ack 1519649940 win 65535 <mss 1460> 23:49:11.264245 IP 10.1.1.6.80 > 133.120.85.92.48922: S 269601671:269601671(0) ack 1519649940 win 65535 <mss 1460> 23:49:23.364342 IP 10.1.1.6.80 > 133.120.85.92.48922: S 269601671:269601671(0) ack 1519649940 win 65535 <mss 1460> Thanks, Mike "Silby" Silbersack --0-1185318269-1184043945=:29353 Content-Type: TEXT/PLAIN; CHARSET=US-ASCII; NAME=tcp_syncache.c-timerfix.patch Content-Transfer-Encoding: BASE64 Content-ID: <20070710000545.B29353@odysseus.silby.com> Content-Description: Content-Disposition: ATTACHMENT; FILENAME=tcp_syncache.c-timerfix.patch LS0tIC91c3Ivc3JjL3N5cy5vbGQvbmV0aW5ldC90Y3Bfc3luY2FjaGUuYwky MDA3LTA2LTI0IDIwOjE3OjMxLjAwMDAwMDAwMCAtMDUwMA0KKysrIC91c3Iv c3JjL3N5cy9uZXRpbmV0L3RjcF9zeW5jYWNoZS5jCTIwMDctMDctMDkgMDA6 NDY6MTguMDAwMDAwMDAwIC0wNTAwDQpAQCAtMTQ5LDcgKzE1MCw2IEBADQog CXN0cnVjdCBtdHgJc2NoX210eDsNCiAJVEFJTFFfSEVBRChzY2hfaGVhZCwg c3luY2FjaGUpCXNjaF9idWNrZXQ7DQogCXN0cnVjdCBjYWxsb3V0CXNjaF90 aW1lcjsNCi0JaW50CQlzY2hfbmV4dGM7DQogCXVfaW50CQlzY2hfbGVuZ3Ro Ow0KIAl1X2ludAkJc2NoX29kZGV2ZW47DQogCXVfaW50MzJfdAlzY2hfc2Vj Yml0c19vZGRbU1lOQ09PS0lFX1NFQ1JFVF9TSVpFXTsNCkBAIC0yNDAsMTYg KzI0MCwxMCBAQA0KIA0KICNkZWZpbmUgRU5EUFRTNl9FUShhLCBiKSAobWVt Y21wKGEsIGIsIHNpemVvZigqYSkpID09IDApDQogDQotI2RlZmluZSBTWU5D QUNIRV9USU1FT1VUKHNjLCBzY2gsIGNvKSBkbyB7CQkJCVwNCisjZGVmaW5l IFNZTkNBQ0hFX1RJTUVPVVQoc2MpIGRvIHsJCQkJCVwNCiAJKHNjKS0+c2Nf cnhtaXRzKys7CQkJCQkJXA0KIAkoc2MpLT5zY19yeHR0aW1lID0gdGlja3Mg KwkJCQkJXA0KIAkJVENQVFZfUlRPQkFTRSAqIHRjcF9iYWNrb2ZmWyhzYykt PnNjX3J4bWl0cyAtIDFdOwlcDQotCWlmICgoc2NoKS0+c2NoX25leHRjID4g KHNjKS0+c2Nfcnh0dGltZSkJCQlcDQotCQkoc2NoKS0+c2NoX25leHRjID0g KHNjKS0+c2Nfcnh0dGltZTsJCQlcDQotCWlmICghVEFJTFFfRU1QVFkoJihz Y2gpLT5zY2hfYnVja2V0KSAmJiAhKGNvKSkJCQlcDQotCQljYWxsb3V0X3Jl c2V0KCYoc2NoKS0+c2NoX3RpbWVyLAkJCVwNCi0JCQkoc2NoKS0+c2NoX25l eHRjIC0gdGlja3MsCQkJXA0KLQkJCXN5bmNhY2hlX3RpbWVyLCAodm9pZCAq KShzY2gpKTsJCQlcDQogfSB3aGlsZSAoMCkNCiANCiAjZGVmaW5lCVNDSF9M T0NLKHNjaCkJCW10eF9sb2NrKCYoc2NoKS0+c2NoX210eCkNCkBAIC0yNzUs NiArMjY5LDcgQEANCiBzeW5jYWNoZV9pbml0KHZvaWQpDQogew0KIAlpbnQg aTsNCisJc3RydWN0IHN5bmNhY2hlX2hlYWQgKnNjaDsNCiANCiAJdGNwX3N5 bmNhY2hlLmNhY2hlX2NvdW50ID0gMDsNCiAJdGNwX3N5bmNhY2hlLmhhc2hz aXplID0gVENQX1NZTkNBQ0hFX0hBU0hTSVpFOw0KQEAgLTMxNyw2ICszMTIs MTcgQEANCiAJdGNwX3N5bmNhY2hlLnpvbmUgPSB1bWFfemNyZWF0ZSgic3lu Y2FjaGUiLCBzaXplb2Yoc3RydWN0IHN5bmNhY2hlKSwNCiAJICAgIE5VTEws IE5VTEwsIE5VTEwsIE5VTEwsIFVNQV9BTElHTl9QVFIsIDApOw0KIAl1bWFf em9uZV9zZXRfbWF4KHRjcF9zeW5jYWNoZS56b25lLCB0Y3Bfc3luY2FjaGUu Y2FjaGVfbGltaXQpOw0KKw0KKwkvKg0KKwkgKiBTdGFydCB0aGUgc3luY2Fj aGUgaGVhZCB0aW1lcnMgcnVubmluZy4gIFRoZXkgZWFjaCBydW4gdGVuIHRp bWVzDQorCSAqIGEgc2Vjb25kLCBhbmQgYXJlIHNwcmVhZCBvdXQgc28gdGhh dCB0aGV5IGFyZSBub3QgYWxsIHJ1bm5pbmcgb24NCisJICogdGhlIHNhbWUg Y2xvY2sgdGljay4NCisJICovDQorCWZvciAoaSA9IDA7IGkgPCB0Y3Bfc3lu Y2FjaGUuaGFzaHNpemU7IGkrKykgew0KKwkJc2NoID0gJnRjcF9zeW5jYWNo ZS5oYXNoYmFzZVtpXTsNCisJCWNhbGxvdXRfcmVzZXQoJihzY2gpLT5zY2hf dGltZXIsIGkgKiAoaHogLyAxMCksICAgIA0KKyAgICAgICAgICAgICAgICBz eW5jYWNoZV90aW1lciwgKHZvaWQgKikoc2NoKSk7DQorCX0NCiB9DQogDQog LyoNCkBAIC0zNDYsOCArMzUyLDggQEANCiAJVEFJTFFfSU5TRVJUX0hFQUQo JnNjaC0+c2NoX2J1Y2tldCwgc2MsIHNjX2hhc2gpOw0KIAlzY2gtPnNjaF9s ZW5ndGgrKzsNCiANCi0JLyogUmVpbml0aWFsaXplIHRoZSBidWNrZXQgcm93 J3MgdGltZXIuICovDQotCVNZTkNBQ0hFX1RJTUVPVVQoc2MsIHNjaCwgMSk7 DQorCS8qIFNldCB0aGUgcmV0cmFuc21pdCB0aW1lciBmb3IgdGhpcyBzb2Nr ZXQuICovDQorCVNZTkNBQ0hFX1RJTUVPVVQoc2MpOw0KIA0KIAlTQ0hfVU5M T0NLKHNjaCk7DQogDQpAQCAtMzk4LDggKzQwNCw2IEBADQogCQkgKiBob3N0 IGRvZXMgdGhlIFNZTi9BQ0stPkFDSy4NCiAJCSAqLw0KIAkJaWYgKHNjLT5z Y19yeHR0aW1lID49IHRpY2spIHsNCi0JCQlpZiAoc2MtPnNjX3J4dHRpbWUg PCBzY2gtPnNjaF9uZXh0YykNCi0JCQkJc2NoLT5zY2hfbmV4dGMgPSBzYy0+ c2Nfcnh0dGltZTsNCiAJCQljb250aW51ZTsNCiAJCX0NCiANCkBAIC00MTYs MTEgKzQyMCwxMCBAQA0KIA0KIAkJKHZvaWQpIHN5bmNhY2hlX3Jlc3BvbmQo c2MpOw0KIAkJdGNwc3RhdC50Y3BzX3NjX3JldHJhbnNtaXR0ZWQrKzsNCi0J CVNZTkNBQ0hFX1RJTUVPVVQoc2MsIHNjaCwgMCk7DQorCQlTWU5DQUNIRV9U SU1FT1VUKHNjKTsNCiAJfQ0KLQlpZiAoIVRBSUxRX0VNUFRZKCYoc2NoKS0+ c2NoX2J1Y2tldCkpDQotCQljYWxsb3V0X3Jlc2V0KCYoc2NoKS0+c2NoX3Rp bWVyLCAoc2NoKS0+c2NoX25leHRjIC0gdGljaywNCi0JCQlzeW5jYWNoZV90 aW1lciwgKHZvaWQgKikoc2NoKSk7DQorCWNhbGxvdXRfcmVzZXQoJihzY2gp LT5zY2hfdGltZXIsIGh6IC8gMTAsDQorCQlzeW5jYWNoZV90aW1lciwgKHZv aWQgKikoc2NoKSk7DQogfQ0KIA0KIC8qDQpAQCAtMTAwNyw3ICsxMDEwLDcg QEANCiAJCSAgICAoIiVzOiBsYWJlbCBub3QgaW5pdGlhbGl6ZWQiLCBfX2Z1 bmNfXykpOw0KICNlbmRpZg0KIAkJaWYgKHN5bmNhY2hlX3Jlc3BvbmQoc2Mp ID09IDApIHsNCi0JCQlTWU5DQUNIRV9USU1FT1VUKHNjLCBzY2gsIDEpOw0K KwkJCVNZTkNBQ0hFX1RJTUVPVVQoc2MpOw0KIAkJCXRjcHN0YXQudGNwc19z bmRhY2tzKys7DQogCQkJdGNwc3RhdC50Y3BzX3NuZHRvdGFsKys7DQogCQl9 DQo= --0-1185318269-1184043945=:29353--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20070709234401.S29353>