From owner-freebsd-current@FreeBSD.ORG Mon Dec 5 19:10:59 2011 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0C605106566C for ; Mon, 5 Dec 2011 19:10:59 +0000 (UTC) (envelope-from luigi@onelab2.iet.unipi.it) Received: from onelab2.iet.unipi.it (onelab2.iet.unipi.it [131.114.59.238]) by mx1.freebsd.org (Postfix) with ESMTP id A77D98FC13 for ; Mon, 5 Dec 2011 19:10:58 +0000 (UTC) Received: by onelab2.iet.unipi.it (Postfix, from userid 275) id 951627300A; Mon, 5 Dec 2011 20:27:03 +0100 (CET) Date: Mon, 5 Dec 2011 20:27:03 +0100 From: Luigi Rizzo To: current@freebsd.org Message-ID: <20111205192703.GA49118@onelab2.iet.unipi.it> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.2.3i Cc: Subject: datapoints on 10G throughput with TCP ? X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Dec 2011 19:10:59 -0000 Hi, I am trying to establish the baseline performance for 10G throughput over TCP, and would like to collect some data points. As a testing program i am using nuttcp from ports (as good as anything, i guess -- it is reasonably flexible, and if you use it in TCP with relatively large writes, the overhead for syscalls and gettimeofday() shouldn't kill you). I'd be very grateful if you could do the following test: - have two machines connected by a 10G link - on one run "nuttcp -S" - on the other one run "nuttcp -t -T 5 -w 128 -v the.other.ip" and send me a dump of the output, such as the one(s) at the end of the message. I am mostly interested in two configurations: - one over loopback, which should tell how fast is the CPU+memory As an example, one of my machines does about 15 Gbit/s, and one of the faster ones does about 44 Gbit/s - one over the wire using 1500 byte mss. Here it really matters how good is the handling of small MTUs. As a data point, on my machines i get 2..3.5 Gbit/s on the "slow" machine with a 1500 byte mtu and default card setting. Clearing the interrupt mitigation register (so no mitigation) brings the rate to 5-6 Gbit/s. Same hardware with linux does about 8 Gbit/s. HEAD seems 10-20% slower than RELENG_8 though i am not sure who is at fault. The receive side is particularly critical - on FreeBSD the receiver is woken up every two packets (do the math below, between the number of rx calls and throughput and mss), resulting in almost 200K activations per second, and despite the fact that interrupt mitigation is set to a much lower value (so incoming packets should be batched). On linux, i see much fewer reads, presumably the process is woken up only at the end of a burst. cheers luigi ------------ EXAMPLES OF OUTPUT ---------------------- > nuttcp -t -T 5 -w 128 -v 10.0.1.2 nuttcp-t: v6.1.2: socket nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> 10.0.1.2 nuttcp-t: time limit = 5.00 seconds nuttcp-t: connect to 10.0.1.2 with mss=1460, RTT=0.103 ms nuttcp-t: send window size = 131400, receive window size = 65700 nuttcp-t: 3095.0982 MB in 5.00 real seconds = 633785.85 KB/sec = 5191.9737 Mbps nuttcp-t: host-retrans = 0 nuttcp-t: 49522 I/O calls, msec/call = 0.10, calls/sec = 9902.99 nuttcp-t: 0.0user 2.7sys 0:05real 54% 100i+2639d 752maxrss 0+3pf 258876+6csw nuttcp-r: v6.1.2: socket nuttcp-r: buflen=65536, nstream=1, port=5001 tcp nuttcp-r: accept from 10.0.1.4 nuttcp-r: send window size = 33580, receive window size = 131400 nuttcp-r: 3095.0982 MB in 5.17 real seconds = 613526.42 KB/sec = 5026.0084 Mbps nuttcp-r: 1114794 I/O calls, msec/call = 0.00, calls/sec = 215801.03 nuttcp-r: 0.1user 3.5sys 0:05real 69% 112i+1104d 626maxrss 0+15pf 507653+188csw > > nuttcp -t -T 5 -w 128 -v localhost nuttcp-t: v6.1.2: socket nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> localhost nuttcp-t: time limit = 5.00 seconds nuttcp-t: connect to 127.0.0.1 with mss=14336, RTT=0.051 ms nuttcp-t: send window size = 143360, receive window size = 71680 nuttcp-t: 26963.4375 MB in 5.00 real seconds = 5521440.59 KB/sec = 45231.6413 Mbps nuttcp-t: host-retrans = 0 nuttcp-t: 431415 I/O calls, msec/call = 0.01, calls/sec = 86272.51 nuttcp-t: 0.0user 4.6sys 0:05real 93% 102i+2681d 774maxrss 0+3pf 2510+1csw nuttcp-r: v6.1.2: socket nuttcp-r: buflen=65536, nstream=1, port=5001 tcp nuttcp-r: accept from 127.0.0.1 nuttcp-r: send window size = 43008, receive window size = 143360 nuttcp-r: 26963.4375 MB in 5.20 real seconds = 5313135.74 KB/sec = 43525.2080 Mbps nuttcp-r: 767807 I/O calls, msec/call = 0.01, calls/sec = 147750.09 nuttcp-r: 0.1user 3.9sys 0:05real 79% 98i+2570d 772maxrss 0+16pf 311014+8csw on the server, run "