From owner-freebsd-hackers@FreeBSD.ORG Fri Apr 11 16:31:46 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9B0991A1 for ; Fri, 11 Apr 2014 16:31:46 +0000 (UTC) Received: from udns.ultimateDNS.NET (ultimatedns.net [209.180.214.225]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3D22A10EF for ; Fri, 11 Apr 2014 16:31:44 +0000 (UTC) Received: from udns.ultimateDNS.NET (localhost [127.0.0.1]) by udns.ultimateDNS.NET (8.14.5/8.14.5) with ESMTP id s3BGZFV5097399; Fri, 11 Apr 2014 09:35:21 -0700 (PDT) (envelope-from bsd-lists@bsdforge.com) Received: (from www@localhost) by udns.ultimateDNS.NET (8.14.5/8.14.5/Submit) id s3BGZA11097396; Fri, 11 Apr 2014 09:35:10 -0700 (PDT) (envelope-from bsd-lists@bsdforge.com) Received: from unavailable02.ultimatedns.net ([209.180.214.228]) (UDNSMS authenticated user chrish) by ultimatedns.net with HTTP; Fri, 11 Apr 2014 09:35:10 -0700 (PDT) Message-ID: In-Reply-To: <20140411160628.GV21331@kib.kiev.ua> References: <20140409084951.GE21331@kib.kiev.ua> <2A722BB3B12E0D80CA9FF075@Mail-PC.tdx.co.uk> <20140409111917.GH21331@kib.kiev.ua> <851413886E3982D2CCFEA9D9@Mail-PC.tdx.co.uk> <20140410184855.GP21331@kib.kiev.ua> <211BD03C086DDB1A07FDF036@Mail-PC.tdx.co.uk> <20140411131649.GR21331@kib.kiev.ua> <652B8CA4866C0B9E4650430B@Mail-PC.tdx.co.uk> <20140411141526.GT21331@kib.kiev.ua> <464979E8F6FCBD7EA7DAA38B@Mail-PC.tdx.co.uk> <20140411160628.GV21331@kib.kiev.ua> Date: Fri, 11 Apr 2014 09:35:10 -0700 (PDT) Subject: Re: Stuck CLOSED sockets / sshd / zombies... From: "Chris H" To: "Konstantin Belousov" User-Agent: UDNSMS/2.0.3 MIME-Version: 1.0 Content-Type: text/plain;charset=utf-8 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal Cc: freebsd-hackers@freebsd.org, Karl Pielorz X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Apr 2014 16:31:46 -0000 > On Fri, Apr 11, 2014 at 03:57:00PM +0100, Karl Pielorz wrote: >> >> >> --On 11 April 2014 17:15 +0300 Konstantin Belousov >> wrote: >> >> > So my suspicious idea seems to be true. From the ldd output, libc >> > appears before libthr in the global order, so libc sigaction() symbol >> > is resolved before libthr interposer. The result is that libthr wrapper >> > thr_sighandler() for the signal handlers is not installed as the >> > recepient of the kernel signal, which prevents libthr locks for rtld >> > from working properly. >> >> Ok, I can just about follow that ;) >> >> > To confirm or deny my theory, please apply the patch below, in addition to >> > the previous patch, and rebuild sshd only, >> ># cd src/secure/usr.sbin/sshd && make clean all install >> > The patch tilts the order of initialization, for my build I got >> > sandy% ldd /usr/sbin/sshd >> > ... >> > libz.so.6 => /lib/libz.so.6 (0x802f0d000) >> > libthr.so.3 => /lib/libthr.so.3 (0x803123000) >> > libc.so.7 => /lib/libc.so.7 (0x803348000) >> > libldns.so.5 => /usr/lib/private/libldns.so.5 (0x8036d1000) >> > libmd.so.6 => /lib/libmd.so.6 (0x803926000) >> > which could be enough to prevent the bug. >> > >> > Please retest and report. >> >> Ok, patched the makefile - rebuilt / installed sshd restarted (which has >> the same initialisation order as yours above), it and ran the security scan >> against it. >> >> *This does indeed seem to fix the problem* >> >> The scan completes, and there are no stuck 'urdlck' sshd's - and no socket >> sitting around in CLOSE_WAIT or CLOSED - thanks! :) >> >> I re-ran the scan a couple of times more to be sure, with the same result - >> no zombies or anything. > Great. > >> >> Is this situation likely to be repeated anywhere else on the system? Or is >> it likely just to be specific to sshd? > Well. > > The issue with libthr so relying on interposition of libc has already bitten > us more than once. The biggest practical consequence of it is that libthr > cannot be dynamically loaded, it must be linked to the main binary for the > whole construct to work. This means that any program big enough to load > plugins at runtime must link to libthr if it might need to load plugin > linked to libthr. Recently, perl and other programs from ports started > doing just that. > > But this is first time I see interposing so broken due to wrong linking > order, esp. in the base system. > > The correct solution is to merge libthr into libc. Some neccessary > preparations were already done, but the main work did not started yet. > This is huge efforts, and it probably should be coordinated with some > other ABI changes planed for libthr to support process-shared locks. Sorry. Second time around. I used the wrong email address. :P This may be wishful thinking, or I might have even overlooked the obvious. Would it make any sense to create 2 versions of libthr, one with libc, and one without? At least that would overcome the /possible/ ordering issue associated with this sort of thing. Then it's simply a matter of choosing which one to load. --Chris > > Anyway, for now, this patch, possibly enhanced to only link with > libpthread when kerberos is used, should be good enough. >