Date: Wed, 3 Aug 2011 00:57:34 +0100 From: "Steven Hartland" <killing@multiplay.co.uk> To: "Andre Oppermann" <andre@freebsd.org> Cc: freebsd-net@freebsd.org, lstewart@freebsd.org Subject: Re: tcp failing to recover from a packet loss under 8.2-RELEASE? Message-ID: <3E54CF756D4845298C690C570D80C032@multiplay.co.uk> References: <E18D678F05BB4F3B93ADEB304CCA8504@multiplay.co.uk><1F95A4C2D54E4F369830143CBDB5FF86@multiplay.co.uk><4E37C0F2.4080004@freebsd.org> <2B063B6D95AA4C27B004C50D96393F91@multiplay.co.uk> <C706DEE346684B8DB06CFC090F556E72@multiplay.co.uk> <BF7A6AD38BBC4BBFA85609E9560F52BC@multiplay.co.uk> <4E386B35.20303@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
----- Original Message ----- From: "Andre Oppermann" <andre@freebsd.org> To: "Steven Hartland" <killing@multiplay.co.uk> Cc: <freebsd-net@freebsd.org>; <lstewart@freebsd.org> Sent: Tuesday, August 02, 2011 10:25 PM Subject: Re: tcp failing to recover from a packet loss under 8.2-RELEASE? > On 02.08.2011 14:32, Steven Hartland wrote: >> ----- Original Message ----- From: "Steven Hartland" >>> Setting net.inet.tcp.reass.maxsegments=8148 and rerunning the >>> tests appears to result in a solid 14MB/s, its still running a >>> full soak test but looking very promising :) >> >> Ok so full test has completed with over 134GB of data transferred >> without a hitch. >> >> I haven't been able to complete this test without error on this >> target machine before the change to maxsegments; so packets being >> dropped in the tcp reassembly code due to tcp_reass global zone >> exhaustion is almost certainly the cause of the stalls we're >> seeing, which is good news :) > > The zone exhaustion is part of the problem but not the real cause > of the stall you're seeing. When the reassembly zone is exhausted > the retransmit that would fill the hole between the socket buffer > and the out-of-order segments in the reassembly queue can't be > processed anymore. This brings all TCP sessions with data in the > reassembly queue to a standstill. > >> I suspect there are a few contributing factors at play causing >> this. >> >> 1. There is known to be a dodgy fibre on the test routing, which >> is scheduled for cleaning, but is currently causing a small amount >> of packet loss between the sender and receiver. > > Packet loss and the use of the reassembly queue. > >> 2. The target machine has 24 cores and is running 8 queues on >> the igb interface which could result in the requirement to reorder >> packets even if they arrived in order on the wire. I say this as >> disabling msix on igb0, which results in just one queue, did reduce >> the occurrence rate of the problem. > > Maybe this is a compounding factor. > >> 3. Combining a low latency (<1ms) high throughput connection >> ~64MB/s with a lower throughput but still relatively high bandwidth >> ~14MB/s with a 10ms latency. > > What matters here is the socket buffer and receive window size. On the reciever the only none default sysclt's are:- vfs.read_max=32 net.inet.tcp.inflight.enable=0 net.inet.tcp.sendspace=65536 kern.ipc.maxsockbuf=524288 kern.maxfiles=50000 >> I look forward to hearing peoples thoughts on what the actual fix >> should be: increased default nmbclusters, decreased nmbclusters => >> maxsegments divisor, or something else? > > These items may or may not need some adjustments. Though it will > eventually wedge again. The reassembly queue must be able to process > the one missing segment despite having exhausted the global zone limit. > I had fixed that at one point in time. It seems that it got lost with > the recent changes. > > Please try this patch: > http://people.freebsd.org/~andre/tcp_reass.c-logdebug+missingsegment-20110802.diff Would it not make sense to always use stack for the missing segment, as this would avoid the allocation e.g. http://www.multiplaygameservers.com/dropzone/tcp_reass.c-logdebug+missingsegment-20110803.diff Also added a two additional cases to protect against using uma_zfree on the stack based tqs, not 100% clear if these could be triggered or not, but thought I'd check? Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3E54CF756D4845298C690C570D80C032>