From owner-freebsd-net@FreeBSD.ORG Sat May 3 17:43:49 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 199D91065670; Sat, 3 May 2008 17:43:49 +0000 (UTC) (envelope-from tim@gebbettco.com) Received: from outmail136022.authsmtp.co.uk (outmail136022.authsmtp.co.uk [62.13.136.22]) by mx1.freebsd.org (Postfix) with ESMTP id 964158FC16; Sat, 3 May 2008 17:43:48 +0000 (UTC) (envelope-from tim@gebbettco.com) Received: from mail-c188.authsmtp.com (mail-c188.authsmtp.com [62.13.128.25]) by punt3.authsmtp.com (8.13.8/8.13.8/Kp) with ESMTP id m43HSbsJ090891; Sat, 3 May 2008 18:28:37 +0100 (BST) Received: from [10.111.0.10] (host217-34-40-180.in-addr.btopenworld.com [217.34.40.180]) (authenticated bits=0) by mail.authsmtp.com (8.14.2/8.14.2/Kp) with ESMTP id m43HSaxB073790; Sat, 3 May 2008 18:28:36 +0100 (BST) Message-ID: <481CA0C4.2040001@gebbettco.com> Date: Sat, 03 May 2008 18:28:36 +0100 From: Tim Gebbett User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: Andre Oppermann References: <20080420025010.GJ73016@server.vk2pj.dyndns.org> <480BBD7E.8010700@freebsd.org> <480C9AC6.8090802@freebsd.org> <480E7901.5000804@freebsd.org> <481B7232.60608@freebsd.org> In-Reply-To: <481B7232.60608@freebsd.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Server-Quench: 5d01e6d6-1936-11dd-aecc-001871e930f4 X-AuthRoute: OCdxaQwcClZJTQEy BS4OBiJcVQ4iYBZL BAkGMA9GIUINWEQM c1ACdR12OUxbHwkB C3YKU15WWFdyXi13 ag1PbgFDZkpQXg1q T0pMQFdNFEs1BR95 ZU9WIBl1dAJPeDB1 Z09rEHUPXEUvcxJ7 X09dFmsbZGY0PH1O BRMLagNUcQtNf0pM aVApUD1vNG8XJC8x E09ydyo8IShFLj8d bz0sCH8+Zns3Vjk6 DwseEDwjDAU5bB17 JBsgLFMXAEcWNC05 X-Authentic-SMTP: 61633239343939.cat.dmpriest.net.uk:953/Kp X-Report-SPAM: If SPAM / abuse - report it at: http://www.authsmtp.com/abuse X-Virus-Status: No virus detected - but ensure you scan with your own anti-virus system! Cc: Peter Jeremy , rwatson@freebsd.org, Mark Hills , freebsd-net@freebsd.org Subject: Re: read() returns ETIMEDOUT on steady TCP connection X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 May 2008 17:43:49 -0000 Hi Andre, Just to introduce myself, I am now helping Mark Hills with testing. Thank you for your suggestion, here are the results from a similar system (RELENG-7) with increasing kern.ipc.nmbjumbop to 25600. at 1600 streams using approx 340mbit, netstat -m was reporting 12550/250/12800/12800 4k (page size) jumbo clusters in use After the read() returns ETIMEDOUT, 3857/10551/14408/25600 4k (page size) jumbo clusters in use sysctl kern.ipc.nmbjumbop=25600 > 51200 After the read() returns ETIMEDOUT, 200/25400/25600/51200 4k (page size) jumbo clusters in use (current/cache/total/max) netstat -m: 4140/26205/30345 mbufs in use (current/cache/total) 256/3482/3738/25600 mbuf clusters in use (current/cache/total/max) 256/3328 mbuf+clusters out of packet secondary zone in use (current/cache) 3882/21718/25600/51200 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) 17075K/100387K/117462K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/7/6656 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 0 calls to protocol drain routines Do you think we need to reel out further sysctls and should I apply the patch to see if tcp_output: error 55 is still occuring ? Thanks again, Tim Andre Oppermann wrote: > Mark Hills wrote: >> On Wed, 23 Apr 2008, Andre Oppermann wrote: >> >>> http://people.freebsd.org/~andre/tcp_output-error-log.diff >>> >>> Please apply this patch and enable the sysctl net.inet.tcp.log_debug=1 >>> and report any output. You likely get some (normal) noise from >>> syncache. >>> What we are looking for is reports from tcp_output. >> >> Hi Andre, I've applied the patch and tested. >> >> Aside from syncache noise, I get a constant stream of 'error 55' >> (ENOBUFS?), once the number of connection gets to around 150 at 192kbps. >> >> TCP: [192.168.5.43]:52153 to [192.168.5.40]:8080; tcp_output: error >> 55 while sending >> >> 192.168.5.40 is the IP address of this host, running the server. >> >> I tried to correlate the point of the application receiving ETIMEDOUT >> with these messages, but that is tricky as it seems to be outputting >> a lot of messages, and multiple messages over eachother (see below). >> >> Because of the mention of no buffer space available, I checked the >> values of net.inet.tcp.sendbuf* and recvbuf*, and increased the max >> values with no effect. >> >> When I get time I will modify the kernel to print errors which aren't >> ENOBUFS to see if there are any others. But in the meantime, this >> sounds like a problem to me. Is that correct? >> >> Mark >> >> >> :8080; tcp_output: error 55 while sending >> TCP: [192.168.5.42]:57384T CtPo: >> [[119922..116688..55..4402]]::85048400;1 ttoc p[_1o9u2t.p1u6t8:. >> 5e.r4r0o]r: 8080;5 5t cwp_hoiultep uste:n deirnrgor 55 while sending >> TCP: [192.168.5.42]:57382 to [192.168.5.40]:8080; tcp_output: error >> 55 while sending >> TCP: [192.168.5.42]:57381 to [192.168.5.40]:8080; tcp_output: error >> 55 while sending >> TCP: [192.168.5.42]:57380 to [192.168.5.40]:8080; tcp_output: error >> 55 while sending > > After tracing through the code it seems you are indeed memory limited. > Looking back at the netstat -m output: > > 12550/250/12800/12800 4k (page size) jumbo clusters in use > (current/cache/total/max) > 0/0/0 requests for jumbo clusters denied (4k/9k/16k) > > This shows that the supply of 4k jumbo clusters is pretty much exhausted. > The cache may be allocated to different CPUs and the one making the > request > at a given point may be depleted and can't get any from the global pool. > The big question is why the denied counter doesn't report anything. I've > looked at the code paths and don't see any obvious reason why it doesn't > get counted. Maybe Robert can give some insight here. > > Try doubling the amount of 4k page size jumbo mbufs. They are the > primary > workhorse in the kernel right now: > > sysctl kern.ipc.nmbjumbop=25600 > > This should get further. Still more may be necessary depending on > workloads. >