From owner-freebsd-net@FreeBSD.ORG Tue Aug 2 12:33:05 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 68606106564A; Tue, 2 Aug 2011 12:33:05 +0000 (UTC) (envelope-from prvs=119502d6cc=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 91DDB8FC08; Tue, 2 Aug 2011 12:33:04 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Tue, 02 Aug 2011 13:32:32 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Tue, 02 Aug 2011 13:32:32 +0100 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50014433717.msg; Tue, 02 Aug 2011 13:32:31 +0100 X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=119502d6cc=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk Message-ID: From: "Steven Hartland" To: "Andre Oppermann" References: <1F95A4C2D54E4F369830143CBDB5FF86@multiplay.co.uk><4E37C0F2.4080004@freebsd.org> <2B063B6D95AA4C27B004C50D96393F91@multiplay.co.uk> Date: Tue, 2 Aug 2011 13:32:53 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6109 Cc: freebsd-net@freebsd.org, lstewart@freebsd.org Subject: Re: tcp failing to recover from a packet loss under 8.2-RELEASE? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Aug 2011 12:33:05 -0000 ----- Original Message ----- From: "Steven Hartland" > Setting net.inet.tcp.reass.maxsegments=8148 and rerunning the > tests appears to result in a solid 14MB/s, its still running a > full soak test but looking very promising :) Ok so full test has completed with over 134GB of data transferred without a hitch. I haven't been able to complete this test without error on this target machine before the change to maxsegments; so packets being dropped in the tcp reassembly code due to tcp_reass global zone exhaustion is almost certainly the cause of the stalls we're seeing, which is good news :) I suspect there are a few contributing factors at play causing this. 1. There is known to be a dodgy fibre on the test routing, which is scheduled for cleaning, but is currently causing a small amount of packet loss between the sender and receiver. 2. The target machine has 24 cores and is running 8 queues on the igb interface which could result in the requirement to reorder packets even if they arrived in order on the wire. I say this as disabling msix on igb0, which results in just one queue, did reduce the occurrence rate of the problem. 3. Combining a low latency (<1ms) high throughput connection ~64MB/s with a lower throughput but still relatively high bandwidth ~14MB/s with a 10ms latency. I look forward to hearing peoples thoughts on what the actual fix should be: increased default nmbclusters, decreased nmbclusters => maxsegments divisor, or something else? Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk.