Date: Fri, 11 Apr 2014 19:06:28 +0300 From: Konstantin Belousov <kostikbel@gmail.com> To: Karl Pielorz <kpielorz_lst@tdx.co.uk> Cc: freebsd-hackers@freebsd.org Subject: Re: Stuck CLOSED sockets / sshd / zombies... Message-ID: <20140411160628.GV21331@kib.kiev.ua> In-Reply-To: <464979E8F6FCBD7EA7DAA38B@Mail-PC.tdx.co.uk> References: <20140409084951.GE21331@kib.kiev.ua> <2A722BB3B12E0D80CA9FF075@Mail-PC.tdx.co.uk> <20140409111917.GH21331@kib.kiev.ua> <851413886E3982D2CCFEA9D9@Mail-PC.tdx.co.uk> <20140410184855.GP21331@kib.kiev.ua> <211BD03C086DDB1A07FDF036@Mail-PC.tdx.co.uk> <20140411131649.GR21331@kib.kiev.ua> <652B8CA4866C0B9E4650430B@Mail-PC.tdx.co.uk> <20140411141526.GT21331@kib.kiev.ua> <464979E8F6FCBD7EA7DAA38B@Mail-PC.tdx.co.uk>
next in thread | previous in thread | raw e-mail | index | archive | help
--kq9thXEmR7mnrMfl Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Apr 11, 2014 at 03:57:00PM +0100, Karl Pielorz wrote: >=20 >=20 > --On 11 April 2014 17:15 +0300 Konstantin Belousov <kostikbel@gmail.com>= =20 > wrote: >=20 > > So my suspicious idea seems to be true. From the ldd output, libc > > appears before libthr in the global order, so libc sigaction() symbol > > is resolved before libthr interposer. The result is that libthr wrapper > > thr_sighandler() for the signal handlers is not installed as the > > recepient of the kernel signal, which prevents libthr locks for rtld > > from working properly. >=20 > Ok, I can just about follow that ;) >=20 > > To confirm or deny my theory, please apply the patch below, in addition= to > > the previous patch, and rebuild sshd only, > ># cd src/secure/usr.sbin/sshd && make clean all install > > The patch tilts the order of initialization, for my build I got > > sandy% ldd /usr/sbin/sshd > > ... > > libz.so.6 =3D> /lib/libz.so.6 (0x802f0d000) > > libthr.so.3 =3D> /lib/libthr.so.3 (0x803123000) > > libc.so.7 =3D> /lib/libc.so.7 (0x803348000) > > libldns.so.5 =3D> /usr/lib/private/libldns.so.5 (0x8036d1000) > > libmd.so.6 =3D> /lib/libmd.so.6 (0x803926000) > > which could be enough to prevent the bug. > > > > Please retest and report. >=20 > Ok, patched the makefile - rebuilt / installed sshd restarted (which has= =20 > the same initialisation order as yours above), it and ran the security sc= an=20 > against it. >=20 > *This does indeed seem to fix the problem* >=20 > The scan completes, and there are no stuck 'urdlck' sshd's - and no socke= t=20 > sitting around in CLOSE_WAIT or CLOSED - thanks! :) >=20 > I re-ran the scan a couple of times more to be sure, with the same result= -=20 > no zombies or anything. Great. >=20 > Is this situation likely to be repeated anywhere else on the system? Or i= s=20 > it likely just to be specific to sshd? Well. The issue with libthr so relying on interposition of libc has already bitten us more than once. The biggest practical consequence of it is that libthr cannot be dynamically loaded, it must be linked to the main binary for the whole construct to work. This means that any program big enough to load plugins at runtime must link to libthr if it might need to load plugin linked to libthr. Recently, perl and other programs from ports started doing just that. But this is first time I see interposing so broken due to wrong linking order, esp. in the base system. The correct solution is to merge libthr into libc. Some neccessary preparations were already done, but the main work did not started yet. This is huge efforts, and it probably should be coordinated with some other ABI changes planed for libthr to support process-shared locks. Anyway, for now, this patch, possibly enhanced to only link with libpthread when kerberos is used, should be good enough. --kq9thXEmR7mnrMfl Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (FreeBSD) iQIcBAEBAgAGBQJTSBMDAAoJEJDCuSvBvK1BJw0P/A33MI9h6tlXhx6VxtKgxEpS v4P5WGQ1wyih8Rv/AVVaJFm5N9rMEf3R3KlUFgI3GcFmWVME6kHh94H6I3BjM14i YNaegwtfawgmlvenCFjXXbWCBr0lXWeY2OXusjECIzIhR42ZnJDNW0s3TRE8Z+1/ ZzLHyowCAAPm/7ANOLMQJmO50E/R7DEHXX85jqRIJWQTLjCOiqK+0okRKfm1X0cl anXYUUl8CPXiFiBgKypFyltyN8uScRH33QrqxiuKxj8r7nsqoOxLTmPOokRysYWG N8WcIBNjxmkM3PtBAYRZHX8kUPYa3a70I0pnbzezqRsZ1oE/jaWckoMdvVWgWsFr uMS+TXHEH8Aon+HDoMal7JuFC6sCyFUz8sOhNlnh2VH9A4FzVcMe/jFYkrwa2WAm SbqAGCNbz6UlHazdJzKDv9YFO4EC1LvwDplvy0MkBO341Lwv3l0eTWZ9WvF/3yBK OawsV88EYBwJpYlQDil60YQQhtf+itA/Nc3KHHd8KrpRmfc/N99GebqmHMv3liYg JF2eJ1jOxM/zmlEP2E/k9lV4TM4qiXdLjOrVXKL7wWc19MbdGvUy39ML4kynpkKn EH4e1o7GFVfEgsXmuknbVmGlUASeOdkhN7d0llT6QOuRQpSNab5qIexQRUbSOh+o zZ/DzIZyeHW+p21X/zGa =6/OA -----END PGP SIGNATURE----- --kq9thXEmR7mnrMfl--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20140411160628.GV21331>