From owner-freebsd-stable@FreeBSD.ORG  Tue Oct 30 17:05:33 2012
Return-Path: <owner-freebsd-stable@FreeBSD.ORG>
Delivered-To: freebsd-stable@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
 by hub.freebsd.org (Postfix) with ESMTP id B9B231CA
 for <freebsd-stable@freebsd.org>; Tue, 30 Oct 2012 17:05:33 +0000 (UTC)
 (envelope-from nparhar@gmail.com)
Received: from mail-pb0-f54.google.com (mail-pb0-f54.google.com
 [209.85.160.54])
 by mx1.freebsd.org (Postfix) with ESMTP id 8585D8FC12
 for <freebsd-stable@freebsd.org>; Tue, 30 Oct 2012 17:05:32 +0000 (UTC)
Received: by mail-pb0-f54.google.com with SMTP id rp8so357397pbb.13
 for <freebsd-stable@freebsd.org>; Tue, 30 Oct 2012 10:05:32 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
 h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject
 :references:in-reply-to:content-type:content-transfer-encoding;
 bh=4123lskdGGv5NMNNDqOqyfpDRBXQCae95+uP5Lf7EVI=;
 b=K0OnOpnitM19lSas49VX9Lte/60UgJm9JLpaDyNJjy/9TgQTb0ueC7RY1hkeDvXb0j
 bSIS+JP+LD53C23wermtI9zSx3Rm37udnXUqYh/jPsLH4GFajMo7EhyMwwNs350wsltk
 eXXTtjC3sFAsOLFI97sWd6hLt3r2oqYMN2LUUnSg28Y2oEFUyxs2PnNZkiUvKqZkyXC6
 cmVAryXG/uNYjRUeCKMC104gHdEDJziT/274b1uugQxXxO9Rw6PTpb9HpqW+DmPLDCwM
 18/IDDsNOIIYgxPaRuzYIqD6r152CP297/rNp+j7jc6YTMD2AlsY6tw1EMrZZM8cC4/P
 AgjA==
Received: by 10.68.221.225 with SMTP id qh1mr104650603pbc.50.1351616732169;
 Tue, 30 Oct 2012 10:05:32 -0700 (PDT)
Received: from [10.192.166.0] (stargate.chelsio.com. [67.207.112.58])
 by mx.google.com with ESMTPS id kv9sm812177pbc.34.2012.10.30.10.05.30
 (version=SSLv3 cipher=OTHER); Tue, 30 Oct 2012 10:05:30 -0700 (PDT)
Sender: Navdeep Parhar <nparhar@gmail.com>
Message-ID: <509008D9.2090006@FreeBSD.org>
Date: Tue, 30 Oct 2012 10:05:29 -0700
From: Navdeep Parhar <np@FreeBSD.org>
User-Agent: Mozilla/5.0 (X11; FreeBSD amd64;
 rv:16.0) Gecko/20121012 Thunderbird/16.0.1
MIME-Version: 1.0
To: Adam Strohl <adams-freebsd@ateamsystems.com>
Subject: Re: No buffer space available / tcp_inpcb value
References: <508FD46D.3040608@ateamsystems.com>
In-Reply-To: <508FD46D.3040608@ateamsystems.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Cc: FreeBSD-Stable ML <freebsd-stable@freebsd.org>
X-BeenThere: freebsd-stable@freebsd.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-stable>,
 <mailto:freebsd-stable-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-stable>
List-Post: <mailto:freebsd-stable@freebsd.org>
List-Help: <mailto:freebsd-stable-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-stable>,
 <mailto:freebsd-stable-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 30 Oct 2012 17:05:33 -0000

On 10/30/12 06:21, Adam Strohl wrote:
> Hey -STABLE,
>
> I've got a client who we've setup a FreeBSD cluster for with about a
> dozens servers, all behind two front end proxies/LBs/firewalls which
> also act as NAT gateways for the internal servers.
>
> On the active front end proxy we've started seeing "fatal: socket: No
> buffer space available" errors during high-peak times.   I can see in
> vmstat -z that this is what is getting denied:
>
> ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP
> tcp_inpcb:              392,  32770,   19398, 13372,1449734621,6312858,   0
>
> We've got a lot of the other values bumped, and it appears to be this
> input limit that is getting hit.  There are no other non-zero FAILed
> counters except 64 and 128 buckets which I believe are normal.
>
> I cannot seem to find the sysctl (or equiv) that controls this limit
> though, or even what it is.  Anyone know?

kern.ipc.maxsockets controls this limit.  See in_pcbinfo_init() for details.

Regards,
Navdeep