From owner-freebsd-net@FreeBSD.ORG Mon Nov 26 14:44:27 2007 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 90F9916A41A for ; Mon, 26 Nov 2007 14:44:27 +0000 (UTC) (envelope-from rpaulo@gmail.com) Received: from nf-out-0910.google.com (nf-out-0910.google.com [64.233.182.190]) by mx1.freebsd.org (Postfix) with ESMTP id C6F7D13C465 for ; Mon, 26 Nov 2007 14:44:26 +0000 (UTC) (envelope-from rpaulo@gmail.com) Received: by nf-out-0910.google.com with SMTP id b2so689087nfb for ; Mon, 26 Nov 2007 06:44:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:from:to:in-reply-to:content-type:content-transfer-encoding:mime-version:subject:date:references:x-mailer:sender; bh=SOdMqyzkrgimNXds0tfyhNU+2xYPY2vNj7K/ra8CCxQ=; b=SOGaCFpI9Pg7/tNx14ZLN5fSPyDZpbBKsznfGgzPhmm40eH8Dc054ZDw5BzrEw5bq3JEe1pAAhFU63fQJs8tGaSB6yy0zK1hxr/EvrRuITPqFVAsabv4b9/vIXuC9IVzWjU1Ge2zpmxZ4cQzNUBffnS+MIvrUD8ajcLeY5DhIqc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=received:message-id:from:to:in-reply-to:content-type:content-transfer-encoding:mime-version:subject:date:references:x-mailer:sender; b=qbvA5yCZ9oIvazGAqYUHJyMJQSIEifgoFUXQKWQ3oZMCErvHdGFQ3DxpnwiaC99cuVflSiO3UUNiK0VFw4iddprpd6x5FIQcsAEEuhMxGsZoA67Wk31v/Lak53IYVx3VqZmH2xMvtNS+Bkrl+BAIqzfLNFFsvwfF4RHUbe+OFyo= Received: by 10.86.98.18 with SMTP id v18mr2749877fgb.1196088259290; Mon, 26 Nov 2007 06:44:19 -0800 (PST) Received: from ?172.17.14.157? ( [193.136.24.128]) by mx.google.com with ESMTPS id 3sm2264429fge.2007.11.26.06.44.17 (version=SSLv3 cipher=OTHER); Mon, 26 Nov 2007 06:44:18 -0800 (PST) Message-Id: <26032689-AD73-4836-8AA1-65915F1D2857@FreeBSD.org> From: Rui Paulo To: freebsd-net@freebsd.org In-Reply-To: Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v915) Date: Mon, 26 Nov 2007 14:44:15 +0000 References: X-Mailer: Apple Mail (2.915) Sender: Rui Paulo Subject: Re: TCP ECN patch for review X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Nov 2007 14:44:27 -0000 Ok, after two changes, the implementation seems ready. You can grab the new diff at the same location: http://people.freebsd.org/~rpaulo/tcp_ecn.diff I tested this on the netperf cluster using pf/altq as the "congested" router. The following configuration was used: altq on em0 bandwidth 3Mb cbq queue { dflt } queue dflt bandwidth 100% cbq(default,ecn) pass in on em0 from any to any keep state queue dflt pass out on em0 from any to any keep state queue dflt In theory, this should give us max 384KB/s, but 360KB/s was observed both with ECN enabled and without ECN (which shows that basically our TCP stack is aggressive enough). The gain with ECN was not in throughput (as it was with NetBSD) but in the number of lost packets and retransmissions (both 0 when ECN was enabled.) If you like throughput graphs :-), http://people.freebsd.org/~rpaulo/throughput-withecn.png http://people.freebsd.org/~rpaulo/throughput-withoutecn.png (generated by wireshark) As you can see, with ECN, the connection throughput seems to stabilize much more earlier than without ECN. tcpdumps are available on request. Next follows the tcptrace -lrW output for both scenarios. Regards. WITH ECN: ------------------------------------------------------------------------- 1 arg remaining, starting with 'tcpdump-withecn' Ostermann's tcptrace -- version 6.6.7 -- Thu Nov 4, 2004 29646 packets seen, 29646 TCP packets traced elapsed wallclock time: 0:00:00.075908, 390551 pkts/sec analyzed trace file elapsed time: 0:00:56.917292 TCP connection info: 1 TCP connection traced: TCP connection 1: host a: 192.168.5.108:57760 host b: 192.168.5.109:22 complete conn: yes first packet: Mon Nov 26 09:46:29.787719 2007 last packet: Mon Nov 26 09:47:26.705011 2007 elapsed time: 0:00:56.917292 total packets: 29646 filename: tcpdump-withecn a->b: b->a: total packets: 18642 total packets: 11004 ack pkts sent: 18641 ack pkts sent: 11004 pure acks sent: 21 pure acks sent: 10661 sack pkts sent: 0 sack pkts sent: 0 dsack pkts sent: 0 dsack pkts sent: 0 max sack blks/ack: 0 max sack blks/ ack: 0 unique bytes sent: 19993839 unique bytes sent: 17903 actual data pkts: 18619 actual data pkts: 341 actual data bytes: 19993839 actual data bytes: 17903 rexmt data pkts: 0 rexmt data pkts: 0 rexmt data bytes: 0 rexmt data bytes: 0 zwnd probe pkts: 0 zwnd probe pkts: 0 zwnd probe bytes: 0 zwnd probe bytes: 0 outoforder pkts: 0 outoforder pkts: 0 pushed data pkts: 25 pushed data pkts: 341 SYN/FIN pkts sent: 1/1 SYN/FIN pkts sent: 1/1 req 1323 ws/ts: Y/Y req 1323 ws/ts: Y/Y adv wind scale: 3 adv wind scale: 3 req sack: Y req sack: N sacks sent: 0 sacks sent: 0 urgent data pkts: 0 pkts urgent data pkts: 0 pkts urgent data bytes: 0 bytes urgent data bytes: 0 bytes mss requested: 1460 bytes mss requested: 1460 bytes max segm size: 1448 bytes max segm size: 736 bytes min segm size: 1 bytes min segm size: 32 bytes avg segm size: 1073 bytes avg segm size: 52 bytes max win adv: 66608 bytes max win adv: 66608 bytes min win adv: 65872 bytes min win adv: 52128 bytes zero win adv: 0 times zero win adv: 0 times avg win adv: 66607 bytes avg win adv: 65010 bytes max owin: 2897 bytes max owin: 769 bytes min non-zero owin: 1 bytes min non-zero owin: 1 bytes avg owin: 1621 bytes avg owin: 15 bytes wavg owin: 1289 bytes wavg owin: 6 bytes initial window: 39 bytes initial window: 39 bytes initial window: 1 pkts initial window: 1 pkts ttl stream length: 19993839 bytes ttl stream length: 17903 bytes missed data: 0 bytes missed data: 0 bytes truncated data: 19440399 bytes truncated data: 7673 bytes truncated packets: 18247 pkts truncated packets: 341 pkts data xmit time: 56.875 secs data xmit time: 56.875 secs idletime max: 1673.5 ms idletime max: 1779.3 ms throughput: 351279 Bps throughput: 315 Bps RTT samples: 9559 RTT samples: 322 RTT min: 0.0 ms RTT min: 0.2 ms RTT max: 99.8 ms RTT max: 99.6 ms RTT avg: 0.0 ms RTT avg: 46.4 ms RTT stdev: 1.5 ms RTT stdev: 11.3 ms RTT from 3WHS: 0.0 ms RTT from 3WHS: 0.2 ms RTT full_sz smpls: 5140 RTT full_sz smpls: 1 RTT full_sz min: 0.0 ms RTT full_sz min: 0.3 ms RTT full_sz max: 2.3 ms RTT full_sz max: 0.3 ms RTT full_sz avg: 0.0 ms RTT full_sz avg: 0.3 ms RTT full_sz stdev: 0.0 ms RTT full_sz stdev: 0.0 ms post-loss acks: 0 post-loss acks: 0 segs cum acked: 9062 segs cum acked: 21 duplicate acks: 1 duplicate acks: 1 triple dupacks: 0 triple dupacks: 0 max # retrans: 0 max # retrans: 0 min retr time: 0.0 ms min retr time: 0.0 ms max retr time: 0.0 ms max retr time: 0.0 ms avg retr time: 0.0 ms avg retr time: 0.0 ms sdv retr time: 0.0 ms sdv retr time: 0.0 ms WITHOUT ECN: ------------------------------------------------------------------------- 1 arg remaining, starting with 'tcpdump-withoutecn' Ostermann's tcptrace -- version 6.6.7 -- Thu Nov 4, 2004 26429 packets seen, 26429 TCP packets traced elapsed wallclock time: 0:00:00.070504, 374858 pkts/sec analyzed trace file elapsed time: 0:00:56.651240 TCP connection info: 1 TCP connection traced: TCP connection 1: host a: 192.168.5.108:50080 host b: 192.168.5.109:22 complete conn: yes first packet: Mon Nov 26 09:55:28.595502 2007 last packet: Mon Nov 26 09:56:25.246743 2007 elapsed time: 0:00:56.651240 total packets: 26429 filename: tcpdump-withoutecn a->b: b->a: total packets: 14948 total packets: 11481 ack pkts sent: 14947 ack pkts sent: 11481 pure acks sent: 15 pure acks sent: 11168 sack pkts sent: 0 sack pkts sent: 0 dsack pkts sent: 0 dsack pkts sent: 0 max sack blks/ack: 0 max sack blks/ ack: 0 unique bytes sent: 19992591 unique bytes sent: 16367 actual data pkts: 14931 actual data pkts: 311 actual data bytes: 19992591 actual data bytes: 16367 rexmt data pkts: 0 rexmt data pkts: 0 rexmt data bytes: 0 rexmt data bytes: 0 zwnd probe pkts: 0 zwnd probe pkts: 0 zwnd probe bytes: 0 zwnd probe bytes: 0 outoforder pkts: 0 outoforder pkts: 0 pushed data pkts: 106 pushed data pkts: 311 SYN/FIN pkts sent: 1/1 SYN/FIN pkts sent: 1/1 req 1323 ws/ts: Y/Y req 1323 ws/ts: Y/Y adv wind scale: 3 adv wind scale: 3 req sack: Y req sack: N sacks sent: 0 sacks sent: 0 urgent data pkts: 0 pkts urgent data pkts: 0 pkts urgent data bytes: 0 bytes urgent data bytes: 0 bytes mss requested: 1460 bytes mss requested: 1460 bytes max segm size: 1448 bytes max segm size: 736 bytes min segm size: 1 bytes min segm size: 32 bytes avg segm size: 1338 bytes avg segm size: 52 bytes max win adv: 66608 bytes max win adv: 99376 bytes min win adv: 65872 bytes min win adv: 57920 bytes zero win adv: 0 times zero win adv: 0 times avg win adv: 66607 bytes avg win adv: 87736 bytes max owin: 9329 bytes max owin: 737 bytes min non-zero owin: 1 bytes min non-zero owin: 1 bytes avg owin: 2012 bytes avg owin: 25 bytes wavg owin: 1830 bytes wavg owin: 6 bytes initial window: 39 bytes initial window: 39 bytes initial window: 1 pkts initial window: 1 pkts ttl stream length: 19992591 bytes ttl stream length: 16367 bytes missed data: 0 bytes missed data: 0 bytes truncated data: 19545650 bytes truncated data: 7037 bytes truncated packets: 14851 pkts truncated packets: 311 pkts data xmit time: 56.608 secs data xmit time: 56.608 secs idletime max: 1885.2 ms idletime max: 1991.3 ms throughput: 352906 Bps throughput: 289 Bps RTT samples: 7980 RTT samples: 306 RTT min: 0.0 ms RTT min: 0.2 ms RTT max: 99.7 ms RTT max: 175.3 ms RTT avg: 0.1 ms RTT avg: 97.8 ms RTT stdev: 1.6 ms RTT stdev: 46.8 ms RTT from 3WHS: 0.0 ms RTT from 3WHS: 0.2 ms RTT full_sz smpls: 6983 RTT full_sz smpls: 1 RTT full_sz min: 0.0 ms RTT full_sz min: 0.2 ms RTT full_sz max: 3.0 ms RTT full_sz max: 0.2 ms RTT full_sz avg: 0.0 ms RTT full_sz avg: 0.2 ms RTT full_sz stdev: 0.1 ms RTT full_sz stdev: 0.0 ms post-loss acks: 0 post-loss acks: 0 segs cum acked: 6953 segs cum acked: 7 duplicate acks: 1 duplicate acks: 1 triple dupacks: 0 triple dupacks: 0 max # retrans: 0 max # retrans: 0 min retr time: 0.0 ms min retr time: 0.0 ms max retr time: 0.0 ms max retr time: 0.0 ms avg retr time: 0.0 ms avg retr time: 0.0 ms sdv retr time: 0.0 ms sdv retr time: 0.0 ms -- Rui Paulo