From owner-freebsd-stable@FreeBSD.ORG Tue Oct 30 16:09:14 2012 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 63FCEBD3; Tue, 30 Oct 2012 16:09:14 +0000 (UTC) (envelope-from adams-freebsd@ateamsystems.com) Received: from fss.sandiego.ateamservers.com (fss.sandiego.ateamservers.com [69.55.229.149]) by mx1.freebsd.org (Postfix) with ESMTP id 43D7D8FC0C; Tue, 30 Oct 2012 16:09:13 +0000 (UTC) Received: from [192.168.15.220] (gw.digitalspark.net [118.175.84.92]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by fss.sandiego.ateamservers.com (Postfix) with ESMTPSA id E1379BCEC5; Tue, 30 Oct 2012 12:09:09 -0400 (EDT) Message-ID: <508FFBA5.5000600@ateamsystems.com> Date: Tue, 30 Oct 2012 23:09:09 +0700 From: Adam Strohl User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20121010 Thunderbird/16.0.1 MIME-Version: 1.0 To: Adrian Chadd Subject: Re: No buffer space available / tcp_inpcb value References: <508FD46D.3040608@ateamsystems.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD-Stable ML X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 30 Oct 2012 16:09:14 -0000 On 10/30/2012 23:05, Adrian Chadd wrote: > Check the output of 'netstat -mb', maybe you're also running out of mbufs? There was nothing denied there that I can see: 35696/4039/39735 mbufs in use (current/cache/total) 2069/3797/5866/32768 mbuf clusters in use (current/cache/total/max) 2069/2077 mbuf+clusters out of packet secondary zone in use (current/cache) 4/3283/3287/16384 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/8192 9k jumbo clusters in use (current/cache/total/max) 0/0/0/4096 16k jumbo clusters in use (current/cache/total/max) 13078K/21735K/34813K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/0/0 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 0 calls to protocol drain routines > > Adrian > > > On 30 October 2012 06:21, Adam Strohl wrote: >> Hey -STABLE, >> >> I've got a client who we've setup a FreeBSD cluster for with about a dozens >> servers, all behind two front end proxies/LBs/firewalls which also act as >> NAT gateways for the internal servers. >> >> On the active front end proxy we've started seeing "fatal: socket: No buffer >> space available" errors during high-peak times. I can see in vmstat -z >> that this is what is getting denied: >> >> ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP >> tcp_inpcb: 392, 32770, 19398, 13372,1449734621,6312858, 0 >> >> We've got a lot of the other values bumped, and it appears to be this input >> limit that is getting hit. There are no other non-zero FAILed counters >> except 64 and 128 buckets which I believe are normal. >> >> I cannot seem to find the sysctl (or equiv) that controls this limit though, >> or even what it is. Anyone know? >> >> I'm obviously in need of this specific answer, but overall is there a codex >> of vmstat -z's items that explains this that I have just not found in my >> searches? This isn't the first time I've had to dig into a value like this >> to increase it's limit, but this time I'm not turning anything up. >> >> Any thoughts/ideas appreciated! >> >> -- >> Adam Strohl >> http://www.ateamsystems.com/ >> >> _______________________________________________ >> freebsd-stable@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-stable >> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" -- Adam Strohl http://www.ateamsystems.com/