Date: Fri, 11 Apr 2014 16:16:49 +0300 From: Konstantin Belousov <kostikbel@gmail.com> To: Karl Pielorz <kpielorz_lst@tdx.co.uk> Cc: freebsd-hackers@freebsd.org Subject: Re: Stuck CLOSED sockets / sshd / zombies... Message-ID: <20140411131649.GR21331@kib.kiev.ua> In-Reply-To: <211BD03C086DDB1A07FDF036@Mail-PC.tdx.co.uk> References: <201404081533.53990.jhb@freebsd.org> <92366925229B4C5B21B04D81@study64.tdx.co.uk> <20140408212319.GC21331@kib.kiev.ua> <D0B81EA30BF8126B37F98D18@study64.tdx.co.uk> <20140409084951.GE21331@kib.kiev.ua> <2A722BB3B12E0D80CA9FF075@Mail-PC.tdx.co.uk> <20140409111917.GH21331@kib.kiev.ua> <851413886E3982D2CCFEA9D9@Mail-PC.tdx.co.uk> <20140410184855.GP21331@kib.kiev.ua> <211BD03C086DDB1A07FDF036@Mail-PC.tdx.co.uk>
next in thread | previous in thread | raw e-mail | index | archive | help
--VdK9YnsVPSS4W/L5 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Apr 11, 2014 at 01:39:54PM +0100, Karl Pielorz wrote: >=20 > Ok, rebuilt a debug world (with your rtld-elf patch), installed it -=20 > reproduced the issue, and ran up gdb on a 'urdlck' stuck sshd, and got th= e=20 > trace below. The trace looks reasonable. I vaguelly remember that you already answered this, but I want to start investigating from the different angle. Please show me the output of 'ldd /usr/sbin/sshd' on your machine. This happens on stable/10, right ? I do not see any linking with libpthread in the sshd Makefile. Could it be that libthr is loaded as dependency of some pam module ? >=20 > Fingers crossed, >=20 > -Karl >=20 > ps. When running up gdb I get a number of these errors (having checked,= =20 > I've always got these - I just didn't notice before as they scroll past= =20 > right at the top of the output from gdb starting up): >=20 > " > Attaching to program: /usr/sbin/sshd, process 2220 >=20 > warning: current_sos: Can't read pathname for load map: Bad address >=20 >=20 > warning: current_sos: Can't read pathname for load map: Bad address > " >=20 > I'm presuming they can be ignored? Ignore this. >=20 > --- >=20 > " > [Switching to Thread 804006400 (LWP 100083/sshd)] > _umtx_op_err () at /usr/src/lib/libthr/arch/amd64/amd64/_umtx_op_err.S:37 > 37 RSYSCALL_ERR(_umtx_op) > (gdb) bt > #0 _umtx_op_err () at=20 > /usr/src/lib/libthr/arch/amd64/amd64/_umtx_op_err.S:37 > #1 0x00000008038e304f in __thr_rwlock_rdlock (rwlock=3D0x803afb480,=20 > flags=3D<value optimized out>, tsp=3D<value optimized out>) > at /usr/src/lib/libthr/thread/thr_umtx.c:277 > #2 0x00000008038ea22c in _thr_rtld_rlock_acquire (lock=3D0x803afb480) at= =20 > thr_umtx.h:196 > #3 0x000000080064f9a2 in rlock_acquire (lock=3D0x80085fe00,=20 > lockstate=3D0x7fffffffc058) at /usr/src/libexec/rtld-elf/rtld_lock.c:197 > #4 0x00000008006498c9 in _rtld_bind (obj=3D0x800662000, reloff=3D13008) = at=20 > /usr/src/libexec/rtld-elf/rtld.c:675 > #5 0x00000008006470cd in _rtld_bind_start () at=20 > /usr/src/libexec/rtld-elf/amd64/rtld_start.S:121 > #6 0x000000000041072c in grace_alarm_handler (sig=3D0) at=20 > /usr/src/secure/usr.sbin/sshd/../../../crypto/openssh/sshd.c:378 > #7 <signal handler called> > #8 _umtx_op_err () at=20 > /usr/src/lib/libthr/arch/amd64/amd64/_umtx_op_err.S:37 > #9 0x00000008038e304f in __thr_rwlock_rdlock (rwlock=3D0x803afb480,=20 > flags=3D<value optimized out>, tsp=3D<value optimized out>) > at /usr/src/lib/libthr/thread/thr_umtx.c:277 > #10 0x00000008038ea22c in _thr_rtld_rlock_acquire (lock=3D0x803afb480) at= =20 > thr_umtx.h:196 > #11 0x000000080064f9a2 in rlock_acquire (lock=3D0x80085fe00,=20 > lockstate=3D0x7fffffffc668) at /usr/src/libexec/rtld-elf/rtld_lock.c:197 > #12 0x00000008006498c9 in _rtld_bind (obj=3D0x800662000, reloff=3D9240) a= t=20 > /usr/src/libexec/rtld-elf/rtld.c:675 > #13 0x00000008006470cd in _rtld_bind_start () at=20 > /usr/src/libexec/rtld-elf/amd64/rtld_start.S:121 > #14 0x000000000042f9dd in sshpam_sigchld_handler (sig=3D<value optimized= =20 > out>) at=20 > /usr/src/secure/usr.sbin/sshd/../../../crypto/openssh/auth-pam.c:152 > #15 <signal handler called> > #16 0x000000080064a1c4 in dlclose (handle=3D0x800696c00) at=20 > /usr/src/libexec/rtld-elf/rtld.c:4136 > #17 0x0000000800ede121 in openpam_destroy_chain (chain=3D0x8040634e0) > at=20 > /usr/src/lib/libpam/libpam/../../../contrib/openpam/lib/libpam/openpam_lo= ad.c:92 > #18 0x0000000800ede0bc in openpam_destroy_chain (chain=3D0x804063460) > at=20 > /usr/src/lib/libpam/libpam/../../../contrib/openpam/lib/libpam/openpam_lo= ad.c:109 > #19 0x0000000800ede0bc in openpam_destroy_chain (chain=3D0x8040633e0) > at=20 > /usr/src/lib/libpam/libpam/../../../contrib/openpam/lib/libpam/openpam_lo= ad.c:109 > #20 0x0000000800ede051 in openpam_clear_chains (policy=3D0x80401a6c8) > at=20 > /usr/src/lib/libpam/libpam/../../../contrib/openpam/lib/libpam/openpam_lo= ad.c:128 > #21 0x0000000800eda9e7 in pam_end (pamh=3D0x80401a6c0, status=3D<value=20 > optimized out>) > at=20 > /usr/src/lib/libpam/libpam/../../../contrib/openpam/lib/libpam/pam_end.c:= 83 > #22 0x000000000042e15d in sshpam_cleanup () at=20 > /usr/src/secure/usr.sbin/sshd/../../../crypto/openssh/auth-pam.c:614 > #23 0x000000000041d58f in do_cleanup (authctxt=3D0x80401a600) at=20 > /usr/src/secure/usr.sbin/sshd/../../../crypto/openssh/session.c:2732 > #24 0x000000000041064f in ssh_cleanup_exit (i=3D255) at=20 > /usr/src/secure/usr.sbin/sshd/../../../crypto/openssh/sshd.c:2545 > #25 0x0000000000428f83 in mm_request_receive (sock=3D<value optimized out= >,=20 > m=3D<value optimized out>) > at=20 > /usr/src/secure/usr.sbin/sshd/../../../crypto/openssh/monitor_wrap.c:153 > #26 0x0000000000427e26 in monitor_read (pmonitor=3D0x804022220, ent=3D0x6= 465a0,=20 > pent=3D0x7fffffffd240) > at /usr/src/secure/usr.sbin/sshd/../../../crypto/openssh/monitor.c:593 > #27 0x0000000000427b49 in monitor_child_preauth (_authctxt=3D<value optim= ized=20 > out>, pmonitor=3D0x804022220) > at /usr/src/secure/usr.sbin/sshd/../../../crypto/openssh/monitor.c:387 > #28 0x000000000040fd15 in main (ac=3D<value optimized out>, av=3D<value= =20 > optimized out>) > at /usr/src/secure/usr.sbin/sshd/../../../crypto/openssh/sshd.c:679 > " --VdK9YnsVPSS4W/L5 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (FreeBSD) iQIcBAEBAgAGBQJTR+tBAAoJEJDCuSvBvK1BBsQP/ipVBTC9k7bkvYjv4pN8MslA XXOG8/ywlk3H/bkjSpUTBq/dU80qYY3JsfXHXKoAypOwgauFoDhjUiX00zgWbN4w TwAUe70k/8rbcYoVdXtLXpsWdbyg0mdY+VlnFExSHoptknS3U5KX8vSNnQdXaS/g Cq3AM2vEb5QCJSq0FNv+iHIsG84tJCtd6tny84JwZWpkEu1ukJhoIn3GBHlk46dl WqADwgkr3i+tRZ2BFQ2ogw27FIWNsCsloarLM8dPaw5NiDzaxVg38CUEbcRyK6h7 v9mqIufvEBfBKrPWCOWNbbiLuqFsf5FuT5Tlv7SzOOwK7UzwIp/jq5xauILtV4ZW Eely2JTN74xPg/WtvrzMQCG1edJoxJp2oO6Ja5niPwi5dcCroC764uo4jt8JzbqK JHiSaCG3lgwirQmjE2c9YcP3L3kBzWt9ABkBOKBIroqN6ReKFPstCEc5M/V23K+w 9DumVAX3Gv11G1WLBZXk3io7ZFuDEtrpQuFEUiJTrn+49wSmARFJgMgrVm6JkNG7 ncj2wMepO0WxApjUY6WGyiOIQpf1pTENyf4UDRdFjiKSOBEqiHelV4sqs1PrZcnH vYZNwxImDjS7vBVDAAc1SB7l9qnADJVaCBqhNKa+0ANLwW+0i/Yn8RPFXNIVpVv2 HNUabcPOodSwyuxbedV5 =P3DP -----END PGP SIGNATURE----- --VdK9YnsVPSS4W/L5--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20140411131649.GR21331>