From owner-freebsd-net@FreeBSD.ORG Tue Aug 2 11:35:55 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 87E481065686; Tue, 2 Aug 2011 11:35:55 +0000 (UTC) (envelope-from prvs=119502d6cc=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 6954E8FC14; Tue, 2 Aug 2011 11:35:54 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Tue, 02 Aug 2011 12:35:21 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Tue, 02 Aug 2011 12:35:21 +0100 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50014433252.msg; Tue, 02 Aug 2011 12:35:20 +0100 X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=119502d6cc=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk Message-ID: From: "Steven Hartland" To: "Steven Hartland" , "Andre Oppermann" References: <1F95A4C2D54E4F369830143CBDB5FF86@multiplay.co.uk><4E37C0F2.4080004@freebsd.org> <2B063B6D95AA4C27B004C50D96393F91@multiplay.co.uk> Date: Tue, 2 Aug 2011 12:35:52 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6109 Cc: freebsd-net@freebsd.org, lstewart@freebsd.org Subject: Re: tcp failing to recover from a packet loss under 8.2-RELEASE? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Aug 2011 11:35:55 -0000 ----- Original Message ----- From: "Steven Hartland" > ----- Original Message ----- > From: "Andre Oppermann" > ... >>> I believe this is tcps_rcvmemdrop in tcp_reass.c to which there's the >>> following comment:- >>> >>> * XXXLAS: Using sbspace(so->so_rcv) instead of so->so_rcv.sb_hiwat >>> * should work but causes packets to be dropped when they shouldn't. >>> >>> I notice this code is relatively new, so I'm wondering if this may be >>> something to do with what we're seeing, possibly still dropping packets >>> it shouldn't? >>> >>> @Lawrence apologies' for the direct mail, but I believe you where the >>> original author this particular change so wondered if you may be >>> able to shed any light on this? >> >> You could be onto something here. Please try this patch: >> http://people.freebsd.org/~andre/tcp_reass.c-logdebug-20110802.diff >> >> You can enable the log output with >> sysctl net.inet.tcp.log_debug=1 >> and report the log output (comes at LOG_DEBUG level). > > Thanks for the response Andre, I've applied the patch and I'm seeing > lots of the following during the test which is:- > 1. scp from local host (10.10.1.30) -> tcptest (10.10.1.20) reciever > which gets ~ 64MB/s > 2. scp from remote host (10.10.1.10) -> tcptest (10.10.1.20) reciever > which gets ~ 10MB/s (line has packet loss) > > Aug 2 11:08:50 tcptest kernel: TCP: [10.10.1.30]:60811 to [10.10.1.20]:22 tcpflags 0x10; tcp_reass: global zone limit > reached, segment dropped > Aug 2 11:08:50 tcptest kernel: TCP: [10.10.1.30]:60811 to [10.10.1.20]:22 tcpflags 0x10; tcp_reass: global zone limit > reached, segment dropped Hmm, based on this are we seeing something similar to the following? http://www.freebsd.org/cgi/query-pr.cgi?pr=155407 Other potentially useful info:- vmstat -z | head -1 ; vmstat -z | grep -i tcp ITEM SIZE LIMIT USED FREE REQUESTS FAILURES tcp_inpcb: 336, 25608, 115, 556, 707, 0 tcpcb: 880, 25600, 115, 405, 707, 0 tcptw: 72, 5150, 0, 600, 188, 0 tcpreass: 40, 1680, 106, 1574, 185926, 4414 sysctl net.inet.tcp.reass net.inet.tcp.reass.overflows: 0 net.inet.tcp.reass.cursegments: 106 net.inet.tcp.reass.maxsegments: 1680 netstat -s -f inet -p tcp | grep "discarded due" 4414 discarded due to memory problems net.inet.tcp.reass.maxsegments: 1680 sysctl kern.ipc.nmbclusters kern.ipc.nmbclusters: 25600 The default value of nmbclusters on the target machine explains the value of net.inet.tcp.reass.maxsegments which defaults to nmbclusters / 16 Setting net.inet.tcp.reass.maxsegments=8148 and rerunning the tests appears to result in a solid 14MB/s, its still running a full soak test but looking very promising :) So I suppose the question is should maxsegments be larger by default due to the recent changes e.g. - V_tcp_reass_maxseg = nmbclusters / 16; + V_tcp_reass_maxseg = nmbclusters / 8; or is the correct fix something more involved? Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk.