From owner-freebsd-hackers@FreeBSD.ORG Fri Apr 11 13:16:59 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 914049E6; Fri, 11 Apr 2014 13:16:59 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 15AA31820; Fri, 11 Apr 2014 13:16:58 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.8/8.14.8) with ESMTP id s3BDGosW050727; Fri, 11 Apr 2014 16:16:50 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.8.3 kib.kiev.ua s3BDGosW050727 Received: (from kostik@localhost) by tom.home (8.14.8/8.14.8/Submit) id s3BDGofo050726; Fri, 11 Apr 2014 16:16:50 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 11 Apr 2014 16:16:49 +0300 From: Konstantin Belousov To: Karl Pielorz Subject: Re: Stuck CLOSED sockets / sshd / zombies... Message-ID: <20140411131649.GR21331@kib.kiev.ua> References: <201404081533.53990.jhb@freebsd.org> <92366925229B4C5B21B04D81@study64.tdx.co.uk> <20140408212319.GC21331@kib.kiev.ua> <20140409084951.GE21331@kib.kiev.ua> <2A722BB3B12E0D80CA9FF075@Mail-PC.tdx.co.uk> <20140409111917.GH21331@kib.kiev.ua> <851413886E3982D2CCFEA9D9@Mail-PC.tdx.co.uk> <20140410184855.GP21331@kib.kiev.ua> <211BD03C086DDB1A07FDF036@Mail-PC.tdx.co.uk> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="VdK9YnsVPSS4W/L5" Content-Disposition: inline In-Reply-To: <211BD03C086DDB1A07FDF036@Mail-PC.tdx.co.uk> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Apr 2014 13:16:59 -0000 --VdK9YnsVPSS4W/L5 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Apr 11, 2014 at 01:39:54PM +0100, Karl Pielorz wrote: >=20 > Ok, rebuilt a debug world (with your rtld-elf patch), installed it -=20 > reproduced the issue, and ran up gdb on a 'urdlck' stuck sshd, and got th= e=20 > trace below. The trace looks reasonable. I vaguelly remember that you already answered this, but I want to start investigating from the different angle. Please show me the output of 'ldd /usr/sbin/sshd' on your machine. This happens on stable/10, right ? I do not see any linking with libpthread in the sshd Makefile. Could it be that libthr is loaded as dependency of some pam module ? >=20 > Fingers crossed, >=20 > -Karl >=20 > ps. When running up gdb I get a number of these errors (having checked,= =20 > I've always got these - I just didn't notice before as they scroll past= =20 > right at the top of the output from gdb starting up): >=20 > " > Attaching to program: /usr/sbin/sshd, process 2220 >=20 > warning: current_sos: Can't read pathname for load map: Bad address >=20 >=20 > warning: current_sos: Can't read pathname for load map: Bad address > " >=20 > I'm presuming they can be ignored? Ignore this. >=20 > --- >=20 > " > [Switching to Thread 804006400 (LWP 100083/sshd)] > _umtx_op_err () at /usr/src/lib/libthr/arch/amd64/amd64/_umtx_op_err.S:37 > 37 RSYSCALL_ERR(_umtx_op) > (gdb) bt > #0 _umtx_op_err () at=20 > /usr/src/lib/libthr/arch/amd64/amd64/_umtx_op_err.S:37 > #1 0x00000008038e304f in __thr_rwlock_rdlock (rwlock=3D0x803afb480,=20 > flags=3D, tsp=3D) > at /usr/src/lib/libthr/thread/thr_umtx.c:277 > #2 0x00000008038ea22c in _thr_rtld_rlock_acquire (lock=3D0x803afb480) at= =20 > thr_umtx.h:196 > #3 0x000000080064f9a2 in rlock_acquire (lock=3D0x80085fe00,=20 > lockstate=3D0x7fffffffc058) at /usr/src/libexec/rtld-elf/rtld_lock.c:197 > #4 0x00000008006498c9 in _rtld_bind (obj=3D0x800662000, reloff=3D13008) = at=20 > /usr/src/libexec/rtld-elf/rtld.c:675 > #5 0x00000008006470cd in _rtld_bind_start () at=20 > /usr/src/libexec/rtld-elf/amd64/rtld_start.S:121 > #6 0x000000000041072c in grace_alarm_handler (sig=3D0) at=20 > /usr/src/secure/usr.sbin/sshd/../../../crypto/openssh/sshd.c:378 > #7 > #8 _umtx_op_err () at=20 > /usr/src/lib/libthr/arch/amd64/amd64/_umtx_op_err.S:37 > #9 0x00000008038e304f in __thr_rwlock_rdlock (rwlock=3D0x803afb480,=20 > flags=3D, tsp=3D) > at /usr/src/lib/libthr/thread/thr_umtx.c:277 > #10 0x00000008038ea22c in _thr_rtld_rlock_acquire (lock=3D0x803afb480) at= =20 > thr_umtx.h:196 > #11 0x000000080064f9a2 in rlock_acquire (lock=3D0x80085fe00,=20 > lockstate=3D0x7fffffffc668) at /usr/src/libexec/rtld-elf/rtld_lock.c:197 > #12 0x00000008006498c9 in _rtld_bind (obj=3D0x800662000, reloff=3D9240) a= t=20 > /usr/src/libexec/rtld-elf/rtld.c:675 > #13 0x00000008006470cd in _rtld_bind_start () at=20 > /usr/src/libexec/rtld-elf/amd64/rtld_start.S:121 > #14 0x000000000042f9dd in sshpam_sigchld_handler (sig=3D out>) at=20 > /usr/src/secure/usr.sbin/sshd/../../../crypto/openssh/auth-pam.c:152 > #15 > #16 0x000000080064a1c4 in dlclose (handle=3D0x800696c00) at=20 > /usr/src/libexec/rtld-elf/rtld.c:4136 > #17 0x0000000800ede121 in openpam_destroy_chain (chain=3D0x8040634e0) > at=20 > /usr/src/lib/libpam/libpam/../../../contrib/openpam/lib/libpam/openpam_lo= ad.c:92 > #18 0x0000000800ede0bc in openpam_destroy_chain (chain=3D0x804063460) > at=20 > /usr/src/lib/libpam/libpam/../../../contrib/openpam/lib/libpam/openpam_lo= ad.c:109 > #19 0x0000000800ede0bc in openpam_destroy_chain (chain=3D0x8040633e0) > at=20 > /usr/src/lib/libpam/libpam/../../../contrib/openpam/lib/libpam/openpam_lo= ad.c:109 > #20 0x0000000800ede051 in openpam_clear_chains (policy=3D0x80401a6c8) > at=20 > /usr/src/lib/libpam/libpam/../../../contrib/openpam/lib/libpam/openpam_lo= ad.c:128 > #21 0x0000000800eda9e7 in pam_end (pamh=3D0x80401a6c0, status=3D optimized out>) > at=20 > /usr/src/lib/libpam/libpam/../../../contrib/openpam/lib/libpam/pam_end.c:= 83 > #22 0x000000000042e15d in sshpam_cleanup () at=20 > /usr/src/secure/usr.sbin/sshd/../../../crypto/openssh/auth-pam.c:614 > #23 0x000000000041d58f in do_cleanup (authctxt=3D0x80401a600) at=20 > /usr/src/secure/usr.sbin/sshd/../../../crypto/openssh/session.c:2732 > #24 0x000000000041064f in ssh_cleanup_exit (i=3D255) at=20 > /usr/src/secure/usr.sbin/sshd/../../../crypto/openssh/sshd.c:2545 > #25 0x0000000000428f83 in mm_request_receive (sock=3D,=20 > m=3D) > at=20 > /usr/src/secure/usr.sbin/sshd/../../../crypto/openssh/monitor_wrap.c:153 > #26 0x0000000000427e26 in monitor_read (pmonitor=3D0x804022220, ent=3D0x6= 465a0,=20 > pent=3D0x7fffffffd240) > at /usr/src/secure/usr.sbin/sshd/../../../crypto/openssh/monitor.c:593 > #27 0x0000000000427b49 in monitor_child_preauth (_authctxt=3D out>, pmonitor=3D0x804022220) > at /usr/src/secure/usr.sbin/sshd/../../../crypto/openssh/monitor.c:387 > #28 0x000000000040fd15 in main (ac=3D, av=3D optimized out>) > at /usr/src/secure/usr.sbin/sshd/../../../crypto/openssh/sshd.c:679 > " --VdK9YnsVPSS4W/L5 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (FreeBSD) iQIcBAEBAgAGBQJTR+tBAAoJEJDCuSvBvK1BBsQP/ipVBTC9k7bkvYjv4pN8MslA XXOG8/ywlk3H/bkjSpUTBq/dU80qYY3JsfXHXKoAypOwgauFoDhjUiX00zgWbN4w TwAUe70k/8rbcYoVdXtLXpsWdbyg0mdY+VlnFExSHoptknS3U5KX8vSNnQdXaS/g Cq3AM2vEb5QCJSq0FNv+iHIsG84tJCtd6tny84JwZWpkEu1ukJhoIn3GBHlk46dl WqADwgkr3i+tRZ2BFQ2ogw27FIWNsCsloarLM8dPaw5NiDzaxVg38CUEbcRyK6h7 v9mqIufvEBfBKrPWCOWNbbiLuqFsf5FuT5Tlv7SzOOwK7UzwIp/jq5xauILtV4ZW Eely2JTN74xPg/WtvrzMQCG1edJoxJp2oO6Ja5niPwi5dcCroC764uo4jt8JzbqK JHiSaCG3lgwirQmjE2c9YcP3L3kBzWt9ABkBOKBIroqN6ReKFPstCEc5M/V23K+w 9DumVAX3Gv11G1WLBZXk3io7ZFuDEtrpQuFEUiJTrn+49wSmARFJgMgrVm6JkNG7 ncj2wMepO0WxApjUY6WGyiOIQpf1pTENyf4UDRdFjiKSOBEqiHelV4sqs1PrZcnH vYZNwxImDjS7vBVDAAc1SB7l9qnADJVaCBqhNKa+0ANLwW+0i/Yn8RPFXNIVpVv2 HNUabcPOodSwyuxbedV5 =P3DP -----END PGP SIGNATURE----- --VdK9YnsVPSS4W/L5--