From owner-freebsd-stable@FreeBSD.ORG  Tue Nov 12 11:06:29 2013
Return-Path: <owner-freebsd-stable@FreeBSD.ORG>
Delivered-To: stable@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115])
 (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by hub.freebsd.org (Postfix) with ESMTPS id EEC30EFB
 for <stable@freebsd.org>; Tue, 12 Nov 2013 11:06:29 +0000 (UTC)
Received: from venus.codepro.be (venus.codepro.be [IPv6:2a01:4f8:162:1127::2])
 (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits))
 (No client certificate requested)
 by mx1.freebsd.org (Postfix) with ESMTPS id B41FB22C3
 for <stable@freebsd.org>; Tue, 12 Nov 2013 11:06:29 +0000 (UTC)
Received: from vega.codepro.be (unknown [172.16.1.3])
 by venus.codepro.be (Postfix) with ESMTP id 5D56394F2;
 Tue, 12 Nov 2013 12:06:27 +0100 (CET)
Received: by vega.codepro.be (Postfix, from userid 1001)
 id 59B793385; Tue, 12 Nov 2013 12:06:27 +0100 (CET)
Date: Tue, 12 Nov 2013 12:06:27 +0100
From: Kristof Provost <kristof@sigsegv.be>
To: Kai Gallasch <k@free.de>
Subject: Re: sonewconn: pcb 0xfffffe00c7223498: Listen queue overflow
Message-ID: <20131112110627.GO58987@vega.codepro.be>
References: <798639e66426509cd53d1f2a1c1d58e0@intertainservices.com>
 <20131001024308.89D557AEDA0@rock.dv.isc.org>
 <13DA2D84-38CE-4E70-B7E8-1A6780B8B0A4@free.de>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
In-Reply-To: <13DA2D84-38CE-4E70-B7E8-1A6780B8B0A4@free.de>
X-PGP-Fingerprint: E114 D9EA 909E D469 8F57  17A5 7D15 91C6 9EFA F286
User-Agent: Mutt/1.5.22 (2013-10-16)
Cc: Mike Jakubik <mike.jakubik@intertainservices.com>, stable@freebsd.org
X-BeenThere: freebsd-stable@freebsd.org
X-Mailman-Version: 2.1.16
Precedence: list
List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-stable>,
 <mailto:freebsd-stable-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-stable/>
List-Post: <mailto:freebsd-stable@freebsd.org>
List-Help: <mailto:freebsd-stable-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-stable>,
 <mailto:freebsd-stable-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 12 Nov 2013 11:06:30 -0000

On 2013-11-12 09:53:15 (+0100), Kai Gallasch <k@free.de> wrote:
> I also have quite a lot of this "Listen queue overflow" in the kernel
> messages on a 9.2-STABLE (r257053) and I tried to identify the
> listening processes with filled up listen queue with netstat. I tried
> both "netstat -nAa | grep $pcb" and "netstat -Lan" but found no match
> with the pcb.
> 
> Problem seems to be that there are server processes that dynamically
> fork child processes that do the listening and are only active for a
> short time.
> 
> Now I wonder if there is a nifty solution for this besides running a
> watchdog script every minute that scans the kernel.msg for "Listen
> queue overflow" and does the trick to find out the pid/process/jid of
> the connected process.
> 
I've also seen this happen on my box.

I'll test the following dtrace bit, to see if I get any useful feedback
out of it.

dtrace -n ::sonewconn:return'/args[1] == NULL/
{ 
print("sonewconn returned NULL"); 
printf("user stack %s (%d)", execname, pid);
ustack();
print("Kernel stack");
stack();
}'

Regards,
Kristof