From owner-freebsd-hackers@FreeBSD.ORG Tue Apr 8 13:37:43 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 803362BF for ; Tue, 8 Apr 2014 13:37:43 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5A094123D for ; Tue, 8 Apr 2014 13:37:43 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 4FDA6B962; Tue, 8 Apr 2014 09:37:42 -0400 (EDT) From: John Baldwin To: Karl Pielorz Subject: Re: Stuck CLOSED sockets / sshd / zombies... Date: Tue, 8 Apr 2014 09:36:30 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.4-CBSD-20130906; KDE/4.5.5; amd64; ; ) References: <3FE645E9723756F22EF901AE@Mail-PC.tdx.co.uk> <201404071148.10157.jhb@freebsd.org> <9647C5438B5CD4A3058AB1A2@Mail-PC.tdx.co.uk> In-Reply-To: <9647C5438B5CD4A3058AB1A2@Mail-PC.tdx.co.uk> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201404080936.30651.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Tue, 08 Apr 2014 09:37:42 -0400 (EDT) Cc: freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Apr 2014 13:37:43 -0000 On Tuesday, April 08, 2014 5:04:24 am Karl Pielorz wrote: > > --On 07 April 2014 11:48 -0400 John Baldwin wrote: > > > Ok, do you have a matching /usr/src on the boxes in question? If so, > > please do this: > > > > cd /usr/src/lib/libc > > make DEBUG_FLAGS=-g all install > > That just installs stuff, not rebuild it right? (Looks to have installed > non-stripped versions?). Humm, it needs to build new ones with debug symbols. If it doesn't, you'll need to do 'make clean' before the other makes. I think you should be fine to do that (make clean then the command above) for those directories and restart gdb without having to restart your sshd. Please also add '/usr/src/libexec/rtld-elf' to the list of directories where you do this. Then do 'detach' in gdb, exit gdb and restart it. > > Then re-run the scan to get a stuck sshd. Once that happens, please > > attach to the top-most stock sshd (the one in "urdlck") with gdb > > (gdb /usr/sbin/sshd ) and run 'bt' and reply with the output. > > Ok, that gives: > > " > (gdb) bt > #0 0x00000008038ea89c in _umtx_op_err () from /lib/libthr.so.3 > #1 0x00000008038e104f in __thr_rwlock_rdlock () from /lib/libthr.so.3 > #2 0x00000008038e821c in _thr_rtld_init () from /lib/libthr.so.3 > #3 0x000000080064f9a2 in _rtld_get_stack_prot () from /libexec/ld-elf.so.1 > #4 0x00000008006498c9 in r_debug_state () from /libexec/ld-elf.so.1 > #5 0x00000008006470cd in .text () from /libexec/ld-elf.so.1 > #6 0x0000000000000246 in ?? () > #7 0x0000000000000000 in ?? () Hmmm, that is useful even though the debug symbols aren't there. Please do the rebuilds I asked for above and re-attach gdb and get 'bt' again. -- John Baldwin