Date: Fri, 28 Aug 2015 09:35:41 -0700 From: Chris Stankevitz <chris@stankevitz.com> To: John-Mark Gurney <jmg@funkthat.com> Cc: freebsd-net@freebsd.org Subject: Re: ssh over WAN: TCP window too small Message-ID: <55E08DDD.1040804@stankevitz.com> In-Reply-To: <20150826223215.GS33167@funkthat.com> References: <55DCF080.7080208@stankevitz.com> <20150826010323.GN33167@funkthat.com> <55DD2A98.2010605@stankevitz.com> <20150826082457.GQ33167@funkthat.com> <55DE2FDF.5030707@stankevitz.com> <20150826223215.GS33167@funkthat.com>
next in thread | previous in thread | raw e-mail | index | archive | help
John-Mark, I'm going to rearrange the conversation a bit in the quotes below: On 8/26/15 3:32 PM, John-Mark Gurney wrote: > So, I looked at what getsockopt SO_RCVBUF returns, and it returns: > case SO_RCVBUF: > optval = so->so_rcv.sb_hiwat; > > Which is NOT S-BMAX, so it looks like OpenSSH only ever gets 66052 bytes > for the buffer... I believe that explains everything we are seeing. HPN developers thought SO_RCVBUF would return S-BMAX but it instead returns S-HIWA. > If it's decided to base it's waiting for ack on SO_RCVBUF, then this > is probably where the issue is... > > Try setting HPNBufferSize to something resonably large, like 1MB, or > above your bandwidth-delay product to see if this will make a difference.. > Per: > Conditions: HPNBufferSize SET, TCPRcvBufPoll enabled, TCPRcvBuf NOT Set > Result: HPN Buffer Size = grows to HPNBufferSize > The buffer will grow up to the maximum size specified here. Setting HPNBufferSize to 1 MB does not change S-BCNT which remains stuck at ~60KB. Although I might not necessarily expect a change. From README.hpn: HPNBufferSize: This is the default buffer size the HPN functionality uses when interacting with non-HPN SSH installations. > It would be interesting to know how long from the read of stdin (and is > it really reading stdin in 4k blocks? If so, that should be fixed) > to the write out the socket... Basicly, how long does it take to encrypt > the data... > >> Note in the above the blocking call to select at time 6.592065 that >> takes ~0.1 second. This is the reason my connection is slow. The >> content appears to be encrypted... I presume it's an application-level >> ssh ack. I posted the in-context kdump at: https://www.stankevitz.com/ssh-ktrace.txt stdin is read in 4k blocks... but each "cycle" reads 3 of these blocks. I define as "cycle" as the period beginning and ending at a 0.1 second select(2) sleep. Looks like encryption of a 4KB block takes 10 microseconds, which means: for every 0.1 seconds in select(2), .000030 seconds are encrypting. >> net.inet.tcp.sendspace: 32768 >> net.inet.tcp.recvspace: 65536 > > Try increasing these and reporting back... the buf_auto should override > those, but this may be limiting it... Okay. Now we're getting somewhere. Increase sendspace on "sending client": no change Increase recvspace on "sending client": no change Increase sendspace on "receiving server": no change Increase recvspace on "receiving server": sending client's S-BCNT increases proportionally! I'm going to try Kurts parameters now... Chris
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?55E08DDD.1040804>