From owner-freebsd-performance@FreeBSD.ORG Wed Mar 11 15:10:51 2015 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BB6E334E for ; Wed, 11 Mar 2015 15:10:51 +0000 (UTC) Received: from mail-we0-x22a.google.com (mail-we0-x22a.google.com [IPv6:2a00:1450:400c:c03::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 55C52BF3 for ; Wed, 11 Mar 2015 15:10:51 +0000 (UTC) Received: by wesw55 with SMTP id w55so9940003wes.2 for ; Wed, 11 Mar 2015 08:10:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:to:subject:date:message-id:user-agent:mime-version :content-type; bh=vOSrtJExuOPAsuXlegQRhK8Zy09JU80PwptHvrQ4G4g=; b=VNKNGmZv/wicXk9XmLN7um2B/OvecRkxEsM5AIyhAexKF9led1HvUIT3csa1pSvY+4 333q6B146itQs5eKYsuP9WwK4Wq9yVYv9TTjzo9PzRN65jF32d5umMqEhRtj4UItaSDu w31oBXXNwSNZxcfdebaYno2YMRKvv8ZZve5FrMKUnWJNtLyKMeLhFLQdCTXZx5nbjatS LRZFQ06gI7JQ+gaNjOwEvcr3lQ21SCoI53LhzwL9/7ZdnoFKeRr+iW6slz1LFWXMgn8R B/ChgUJmHgmEtQH6uffd/+rRTg2qs27/mxUA0j7AqaNoQ2xtInyB59rz/iqGCYKmcxMK m20w== X-Received: by 10.194.171.136 with SMTP id au8mr80239842wjc.6.1426086649600; Wed, 11 Mar 2015 08:10:49 -0700 (PDT) Received: from localhost ([2a01:7e00::f03c:91ff:fe70:2696]) by mx.google.com with ESMTPSA id ga8sm6521861wib.11.2015.03.11.08.10.48 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 11 Mar 2015 08:10:48 -0700 (PDT) From: Malcolm Matalka To: freebsd-performance@freebsd.org Subject: NAT under performing direct connection by 10x Date: Wed, 11 Mar 2015 15:10:46 +0000 Message-ID: <87fv9bin7t.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Mar 2015 15:10:51 -0000 I am setting up my Raspberry Pi Model B+ as a router and am experience a throughput bottleneck that I cannot track down. I'd like to preface my question with two things: - I'm new to this so it is quite possible this is something obvious and I simply lack the knowledge to see it. If so, sorry for wasting anyones time and thank you very much to whoever points it out to me. - It is quite possible that my Raspberry Pi does not have the hardware to support the desired throughput, but I'm having trouble finding evidence to back up that hypothesis. The setup I have is: Laptop -> (wlan0 - RPi - ue0) -> Router I am using a USB dongle with a 'run0' interface running as a hostap. The RPi is then connected to a router which is connected to the internet. I have run the following command in different experiments: wget -O /dev/null ftp://ftp.freebsd.org/pub/FreeBSD/releases/VM-IMAGES/10.1-RELEASE/amd64/Latest/FreeBSD-10.1-RELEASE-amd64.raw.xz On the RPi: download speed of around 500 KB/s On the laptop: download speed of around 80 KB/s I have also scp'd a file from the laptop to another computer on the other side of the router: Laptop -> RPi -> Router -> Computer And I have a throughput of around 1 MB/s. This exercises both network interfaces similar to downloading from the internet. The only difference I am aware of is that NAT is happening in the Internet case. I have: - Checked CPU, in the NAT case the load average does not break 0.2. The CPU does not seem stressed. - Interrupt %, in top, doesn't go above a peak of 4% in either case. - netstat -i and netstat -s do not show any errors happening (that I can see). - While I have no idea if it is significant, I have noticed that in the case of a local download (on the RPi), in tcpdump the "win" goes down to around 1000. In the NAT case it stays at around 4096. Currently I am at a loss to explain the bottleneck. Is there a suggested next place to explore to track this down? As I said, it could be that the RPi is not capable of these throughputs, I'm using this as an excuse to learn how to investigate a performance bottleneck in FreeBSD though so I'm more interested in the meta-discussion of how to determine where the problem is. Thank you, /Malcolm From owner-freebsd-performance@FreeBSD.ORG Wed Mar 11 15:20:29 2015 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7D22B8F7 for ; Wed, 11 Mar 2015 15:20:29 +0000 (UTC) Received: from mail-ig0-x22b.google.com (mail-ig0-x22b.google.com [IPv6:2607:f8b0:4001:c05::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 43E97CBC for ; Wed, 11 Mar 2015 15:20:29 +0000 (UTC) Received: by igbhn18 with SMTP id hn18so40873965igb.2 for ; Wed, 11 Mar 2015 08:20:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=Y8OifBHAZ6osNLoQhUGnJ1lBlGlv2Js/MQL0p1jdhdM=; b=qUYT+B6sUJSfJRqeD9XxOzGt4u6JNq7Ov2b6cqXqnYrW5R+bCpKCn5tXDT7oYtJOlI XhxfC3nSRvJH3MnoYVpXKLdrKxOHqOPPB4a/xL2+t85CxOrKtmVUpkpZjqn+VNg+lumo E1tAJ4ZcUMtgaCzg8VedgcK1pTL2EWAaynyxsYF9v3p7OI68Jqr88KWxv0IceMVAgW7w rR76/calAIkWkql2b3QPE6C7cHswtz4c9aKRet3znQeNhwi5yDXoa5a3zmiIMu0SvI5r 474G6tV/nv7E8/qhPSiuvntXFkvXV06PoZebhNqtkdwIDqjfSOtCQ+AnfJqNr73CSSDZ UIug== MIME-Version: 1.0 X-Received: by 10.107.30.135 with SMTP id e129mr66753210ioe.26.1426087228672; Wed, 11 Mar 2015 08:20:28 -0700 (PDT) Received: by 10.107.151.206 with HTTP; Wed, 11 Mar 2015 08:20:28 -0700 (PDT) In-Reply-To: <87fv9bin7t.fsf@gmail.com> References: <87fv9bin7t.fsf@gmail.com> Date: Wed, 11 Mar 2015 08:20:28 -0700 Message-ID: Subject: Re: NAT under performing direct connection by 10x From: Atom Powers To: Malcolm Matalka Content-Type: text/plain; charset=UTF-8 Cc: freebsd-performance@freebsd.org X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Mar 2015 15:20:29 -0000 What are you using to implement NAT? ipf? iptables? Are you reassembling packets or passing them straight through? Are your memory buffers big enough? Who is your Internet provider and do you see those same rates for repeated tests? (Some provider will give you an early burst of speed and then throttle you back.) The window size could be significant. What happens if you manually set the window size to different values? On Wed, Mar 11, 2015 at 8:10 AM, Malcolm Matalka wrote: > I am setting up my Raspberry Pi Model B+ as a router and am experience > a throughput bottleneck that I cannot track down. > > I'd like to preface my question with two things: > > - I'm new to this so it is quite possible this is something obvious and > I simply lack the knowledge to see it. If so, sorry for wasting > anyones time and thank you very much to whoever points it out to me. > > - It is quite possible that my Raspberry Pi does not have the hardware > to support the desired throughput, but I'm having trouble finding > evidence to back up that hypothesis. > > The setup I have is: > > Laptop -> (wlan0 - RPi - ue0) -> Router > > I am using a USB dongle with a 'run0' interface running as a hostap. > The RPi is then connected to a router which is connected to the > internet. > > I have run the following command in different experiments: > > wget -O /dev/null > ftp://ftp.freebsd.org/pub/FreeBSD/releases/VM-IMAGES/10.1-RELEASE/amd64/Latest/FreeBSD-10.1-RELEASE-amd64.raw.xz > > On the RPi: download speed of around 500 KB/s > > On the laptop: download speed of around 80 KB/s > > I have also scp'd a file from the laptop to another computer on the other > side of the router: > > Laptop -> RPi -> Router -> Computer > > And I have a throughput of around 1 MB/s. This exercises both network > interfaces similar to downloading from the internet. > > The only difference I am aware of is that NAT is happening in the > Internet case. > > I have: > > - Checked CPU, in the NAT case the load average does not break 0.2. The > CPU does not seem stressed. > > - Interrupt %, in top, doesn't go above a peak of 4% in either case. > > - netstat -i and netstat -s do not show any errors happening (that I can > see). > > - While I have no idea if it is significant, I have noticed that in the > case of a local download (on the RPi), in tcpdump the "win" goes down > to around 1000. In the NAT case it stays at around 4096. > > Currently I am at a loss to explain the bottleneck. Is there a > suggested next place to explore to track this down? As I said, it could > be that the RPi is not capable of these throughputs, I'm using this as > an excuse to learn how to investigate a performance bottleneck in > FreeBSD though so I'm more interested in the meta-discussion of how to > determine where the problem is. > > Thank you, > /Malcolm > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" -- Perfection is just a word I use occasionally with mustard. --Atom Powers-- From owner-freebsd-performance@FreeBSD.ORG Wed Mar 11 15:32:12 2015 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3A963FBC for ; Wed, 11 Mar 2015 15:32:12 +0000 (UTC) Received: from mail-wi0-x229.google.com (mail-wi0-x229.google.com [IPv6:2a00:1450:400c:c05::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BD38FEC0 for ; Wed, 11 Mar 2015 15:32:11 +0000 (UTC) Received: by widex7 with SMTP id ex7so12795143wid.1 for ; Wed, 11 Mar 2015 08:32:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version:content-type; bh=jr6T+QDGj/fKlz1MzP8SyQ5n/79mvVpKPXshUWe1Bus=; b=yFYQL8GnDfUcbgleSaH12Cf6Ol1Ruf63G9kziL8XaJu934Y2UEWdpAjKgYnq500FDl Qp0ZExDB6kKqUACXTpv7xDAndMuWdB4r0GxIvI5j0xV0gwUVVZGK4q/73W3nE54N7xe8 Bu/4Okj3nBjx3UupXpaDX7GmNjABctS9tPHii2dALmQOsGVVxwH+cPyjFGp6Rp9bBmEp jCBhYL1MN6r468wVKg5WyYZtfk4xU0hfbQvgP4ZXtVYPgMB2i4Q4iqnDIc4RlT2zYgD0 MEZ5UMTsinMPT3m7xh2plYUZwNKKMSj0ABtyUcT407szjT2dVpIGqLjd+3q8tdwnfOav 1pFg== X-Received: by 10.180.105.40 with SMTP id gj8mr125501169wib.67.1426087930059; Wed, 11 Mar 2015 08:32:10 -0700 (PDT) Received: from localhost ([2a01:7e00::f03c:91ff:fe70:2696]) by mx.google.com with ESMTPSA id w4sm6576595wib.19.2015.03.11.08.32.08 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 11 Mar 2015 08:32:09 -0700 (PDT) From: Malcolm Matalka To: Atom Powers Subject: Re: NAT under performing direct connection by 10x References: <87fv9bin7t.fsf@gmail.com> Date: Wed, 11 Mar 2015 15:32:08 +0000 In-Reply-To: (Atom Powers's message of "Wed, 11 Mar 2015 08:20:28 -0700") Message-ID: <874mprim87.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain Cc: freebsd-performance@freebsd.org X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Mar 2015 15:32:12 -0000 Atom Powers writes: > What are you using to implement NAT? ipf? iptables? Are you Whoops, apologies, I knew I forgot something. I'm using ipfw. My rules are: # ipfw list 00100 nat 1 ip from any to any via ue0 09999 allow ip from any to any 65535 deny ip from any to any > reassembling packets or passing them straight through? Are your memory > buffers big enough? I have gateway_enable="YES". I'm not sure if that answers your question, if it doesn't please let me know what I should look at in order to answer your question. Here is the output of netstat -m while doing a transfer: $ netstat -m 12/498/510 mbufs in use (current/cache/total) 1/255/256/126174 mbuf clusters in use (current/cache/total/max) 1/252 mbuf+clusters out of packet secondary zone in use (current/cache) 1/47/48/4925 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/1459 9k jumbo clusters in use (current/cache/total/max) 0/0/0/820 16k jumbo clusters in use (current/cache/total/max) 9K/822K/831K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters delayed (4k/9k/16k) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/5/4272 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile While playing around, I also have add the following to /boot/loader.conf: kern.ipc.nmbclusters=126174 And to /etc/sysctl.conf: kern.ipc.nmbjumbop=63087 Neither had any effect. These values were taken from another FreeBSD machine to see if perhaps the defaults for the RPi were too low. > > Who is your Internet provider and do you see those same rates for > repeated tests? (Some provider will give you an early burst of speed > and then throttle you back.) I have done the tests multiple times over multiple files and jumping between going through the RPi and bypassing it, all with the result that my performance from my ISP is great except when I go through the RPi. > > The window size could be significant. What happens if you manually set > the window size to different values? I'm afraid I don't know how to do this, how do I? I have tired turning window scaling off with sysctl -w net.inet.tcp.rfc1323=0, however that had no effect. > > > > > On Wed, Mar 11, 2015 at 8:10 AM, Malcolm Matalka wrote: >> I am setting up my Raspberry Pi Model B+ as a router and am experience >> a throughput bottleneck that I cannot track down. >> >> I'd like to preface my question with two things: >> >> - I'm new to this so it is quite possible this is something obvious and >> I simply lack the knowledge to see it. If so, sorry for wasting >> anyones time and thank you very much to whoever points it out to me. >> >> - It is quite possible that my Raspberry Pi does not have the hardware >> to support the desired throughput, but I'm having trouble finding >> evidence to back up that hypothesis. >> >> The setup I have is: >> >> Laptop -> (wlan0 - RPi - ue0) -> Router >> >> I am using a USB dongle with a 'run0' interface running as a hostap. >> The RPi is then connected to a router which is connected to the >> internet. >> >> I have run the following command in different experiments: >> >> wget -O /dev/null >> ftp://ftp.freebsd.org/pub/FreeBSD/releases/VM-IMAGES/10.1-RELEASE/amd64/Latest/FreeBSD-10.1-RELEASE-amd64.raw.xz >> >> On the RPi: download speed of around 500 KB/s >> >> On the laptop: download speed of around 80 KB/s >> >> I have also scp'd a file from the laptop to another computer on the other >> side of the router: >> >> Laptop -> RPi -> Router -> Computer >> >> And I have a throughput of around 1 MB/s. This exercises both network >> interfaces similar to downloading from the internet. >> >> The only difference I am aware of is that NAT is happening in the >> Internet case. >> >> I have: >> >> - Checked CPU, in the NAT case the load average does not break 0.2. The >> CPU does not seem stressed. >> >> - Interrupt %, in top, doesn't go above a peak of 4% in either case. >> >> - netstat -i and netstat -s do not show any errors happening (that I can >> see). >> >> - While I have no idea if it is significant, I have noticed that in the >> case of a local download (on the RPi), in tcpdump the "win" goes down >> to around 1000. In the NAT case it stays at around 4096. >> >> Currently I am at a loss to explain the bottleneck. Is there a >> suggested next place to explore to track this down? As I said, it could >> be that the RPi is not capable of these throughputs, I'm using this as >> an excuse to learn how to investigate a performance bottleneck in >> FreeBSD though so I'm more interested in the meta-discussion of how to >> determine where the problem is. >> >> Thank you, >> /Malcolm >> _______________________________________________ >> freebsd-performance@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-performance >> To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org"