Date: Fri, 17 Jul 2009 18:46:02 -0400 From: "Brian A. Seklecki" <seklecki@noc.cfi.pgh.pa.us> To: freebsd-net@freebsd.org Subject: Dropped/Duplicate SYN, Cisco PIX/ASA, and and random ISN w/ net.inet.ip.random_id=1 Message-ID: <1247870762.10382.14770.camel@soundwave.ws.pitbpa0.priv.collaborativefusion.com>
next in thread | raw e-mail | index | archive | help
All: We recently worked closely with some FreeBSD developers to track down an illusive bug in the stack. In a high performance environment, we observed dropped (or extreme delayed) SYN packets, but were unable to easily reproduce the problem using test case scenarios. Our environment: - FreeBSD 7.x Servers - FreeBSD 6.x clients - PIX/ASA 7.2.x stateful firewalls - pf(4) on the server with lots of jails - PHP clients and server with SOAP framework, so lots and lots of sockets, often thousands between any given client->server, in various TCP states. Getting to the heart of the matter, see: http://fxr.watson.org/fxr/source/netinet/tcp_timewait.c#L385 Here we drop SYN's from [client:source_ephemeral_socket] if: 1) We already have that exact combination in CLOSE_WAIT 2) The ISN of the new incoming SYN is lower than that of the existing socket in CLOSE_WAIT Those conditions are _highly_ unlikely, until you start hedging your bets. net.inet.ip.random_id=1 in sysctl.conf(5) is one way to exacerbate the problem. So are the magic scrubbing bubbles in pf.conf(5): scrub all random-id. Also, the PIX/ASA code randomizes IDs by default as well(*). net.inet.ip.portrange.randomized is another; since truly randomized numbers can involved duplicates. Additionally, the default random port ephemeral source port range is way too small for these HPC environments, leading to more likely collisions, so that can be increased: net.inet.ip.random_id=0 net.inet.ip.portrange.randomized=1 net.inet.ip.portrange.first=2048 Anyway, this discussion strictly for the benefit of the mailing list archives, in case, further down the road, someone else finds them self tcpdump(8)'ing duplicate SYNs and starting at netstat(8) -s output and beginning to doubt their own existence. ~Brian A. Seklecki (*) To disable port randomization on the Cisco PIX: tcp-map verify-chksum check-retransmission checksum-verification exceed-mss drop syn-data drop tcp-options selective-ack allow urgent-flag clear no ttl-evasion-protection ! icmp unreachable rate-limit 1 burst-size 1 timeout xlate 3:00:00 timeout conn 12:00:00 half-closed 0:10:00 udp 0:01:00 icmp 0:00:02 timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 ! policy-map global_policy class my_inspection_tcp set connection embryonic-conn-max 2048 per-client-max 1024\ per-client-embryonic-max 1024 random-sequence-number disable set connection timeout embryonic 0:02:00 tcp 1:30:00 dcd 24:00:00 5 set connection advanced-options verify-chksum ! service-policy global_policy interface [WhateverIF]
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1247870762.10382.14770.camel>