From owner-freebsd-net@freebsd.org Wed Jan 10 12:44:33 2018 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3BBFBE7E54F for ; Wed, 10 Jan 2018 12:44:33 +0000 (UTC) (envelope-from thomas@gibfest.dk) Received: from mail.tyknet.dk (mail.tyknet.dk [IPv6:2a01:4f8:201:2327:144:76:253:226]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 058AC72CD0 for ; Wed, 10 Jan 2018 12:44:32 +0000 (UTC) (envelope-from thomas@gibfest.dk) Received: from [10.137.3.13] (nat2.hq.bornfiber.dk [185.96.91.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.tyknet.dk (Postfix) with ESMTPSA id 3553EBB145A for ; Wed, 10 Jan 2018 12:44:07 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.tyknet.dk 3553EBB145A DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=gibfest.dk; s=default; t=1515588247; bh=9S4pqCBu2CliNtCwUsYQAkICkeT2BVq5ppSMoaCkies=; h=To:From:Subject:Date; b=PPN9mRBaerGoc/Szpaf5ZJODWmpN9RcHoUyoaUlHLCMmnby3VkP5UEdzsOEWyP0gy JaN0acWBDwX9Mc8172LpG0mXn9ci/2Ke+BQCGkqxBcLAKrwUOA7EP7hq+SL8mvJ+Ek 13puT5iNKFO6JZdl8v3Hk92eD0QvIWAiEdBBQBVELWUwddOtaXUvYG4ex+LYaBlmze 7XeDabQMWYrwJsXxdKImy5KA0Ztn8CqGg1Z99SHNGZGquGY+ky+IVFmD+74vkPc0Ip zSQdDqqcUo/0mTjayr2aSETMK/2uKR1EteEUgIxbEfp9QMRy9YU2JruUtRin29E78l 5Y9voq1trzr4g== To: freebsd-net@freebsd.org From: Thomas Steen Rasmussen Subject: Help finding a pcb id Message-ID: <84c785a2-cf14-3921-797b-6e4157566f3b@gibfest.dk> Date: Wed, 10 Jan 2018 13:43:56 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.5.2 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Content-Language: en-US X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Jan 2018 12:44:33 -0000 Hello list, I have a jailhost with a busy TCP service which sometimes cannot keep up, triggering the usual "listen queue overflow" messages most of you will have seen at some point: kernel: sonewconn: pcb 0xfffff803f327a1d0: Listen queue overflow: 193 already in queue awaiting acceptance (1169 occurrences) kernel: sonewconn: pcb 0xfffff8031b131740: Listen queue overflow: 193 already in queue awaiting acceptance (1068 occurrences) These two pcb id's are repeating warnings: $ cat /var/log/messages | cut -w -f 8 | sort | uniq -c  142 0xfffff8031b131740:  483 0xfffff803f327a1d0: Through elimination I know which two listening TCP ports/services are causing the warnings. But I can't get the pcb id to match. Usually I would find the listening socket with netstat -Aan but the listening sockets causing these warnings have different pcb ids in the netstat output: fffff80319ff5000 tcp4       0      0 185.96.180.29.9091 *.*                LISTEN fffff80319e19410 tcp4       0      0 185.96.180.29.9090 *.*                LISTEN What am I missing? What is the correct way to match the pcb id from messages to a process/listening socket? Could this be because of these pf rdr rules? Does a pf rdr create a new pcb id which is "invisible" to netstat? rdr on $if inet proto tcp from any to $tor6v4 port 443 -> $tor6v4 port 9090 rdr on $if inet proto tcp from any to $tor6v4 port 80 -> $tor6v4 port 9091 On a more general note I am wondering is if it would be possible to log some more info from the socket around line 600 of sys/kern/uipc_socket.c? Local (and remote where relevant) address/port/or whatever relevant for the protocol, and perhaps also pid and jid? More info would make things a lot easier on busy jailhosts :) The cause of "listen queue overflow" messages can be almost impossible to find if the process/socket is short-lived. Thank you in advance for any input! Best regards, Thomas Steen Rasmussen