From owner-svn-src-all@FreeBSD.ORG Sat Aug 30 19:37:05 2014 Return-Path: Delivered-To: svn-src-all@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A0A20B39; Sat, 30 Aug 2014 19:37:05 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D35391FBE; Sat, 30 Aug 2014 19:37:04 +0000 (UTC) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id s7UJavEs041696 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 30 Aug 2014 22:36:57 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua s7UJavEs041696 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id s7UJav0j041695; Sat, 30 Aug 2014 22:36:57 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 30 Aug 2014 22:36:57 +0300 From: Konstantin Belousov To: Bryan Drewery Subject: Re: svn commit: r270803 - head/libexec/rtld-elf Message-ID: <20140830193657.GR2737@kib.kiev.ua> References: <201408291044.s7TAiwmI077897@svn.freebsd.org> <540094DB.5050307@FreeBSD.org> <20140829172311.GP2737@kib.kiev.ua> <5400D873.70707@FreeBSD.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="JpBuuFAFSbNvDzdI" Content-Disposition: inline In-Reply-To: <5400D873.70707@FreeBSD.org> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: svn-src-head@freebsd.org, svn-src-all@freebsd.org, src-committers@freebsd.org X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 30 Aug 2014 19:37:05 -0000 --JpBuuFAFSbNvDzdI Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Aug 29, 2014 at 02:45:55PM -0500, Bryan Drewery wrote: > My only uses so far have been observing which file is loaded for a > library with dlopen(3) and where symbols are resolved from. Seeing where > symbols are resolved from is useful for both dlopen(3)/dlsym(3) usage > and LD_PRELOAD. Ok. Note that there are no traces of why the symbols where resolved from the specified object, and, why that symbol was selected (one object may contain several symbols with the same name, for several reasons). >=20 > I've written an application that "optionally" allows using some > libraries such as libtcl to be used without requiring the library at > build time or startup. At build time the library's headers are used to > generate wrapper functions for every symbol needed. The library is not > linked in. When the binary is ran it starts up fine if libtcl is > missing. At runtime if TCL is enabled then it will attempt to > dlopen(libtcl.so.VER) using the VER its symbols was compiled against. If > it cannot find it then the feature is disabled. It uses dlsym(3) to load > all symbols into a table and the wrapper functions use that table via > hash table lookups. Note that ELF PLT symbols (i.e. function call sites) resolution is lazy by default, i.e. the only tricky part to do for this with the standard tools is to avoid writing the DT_NEEDED tag into the binary you construct. Newer GNU ld has interesting options like --unresolved-symbols=3Dignore-all, or --warn-unresolved-symbols, which seemingly allow to link the binary even with unresolved symbols. If you use then dlopen(libname, RTLD_LAZY | RTLD_GLOBAL) before calling the functions from the library, you should get what you want without needing to laborously provide stubs and use dlsym(). And, this cannot work for non-PLT relocations, e.g. references to global variables. >=20 > This all pretty convoluted and only done to avoid requiring a library at > startup. The application is distributed pre-compiled for specific OS > releases and not intended to be compiled manually. >=20 > Anyway some examples from GNU's rtld: >=20 > > # env LD_DEBUG=3Dhelp ./app > > Valid options for the LD_DEBUG environment variable are: > >=20 > > libs display library search paths > > reloc display relocation processing > > files display progress for input file > > symbols display symbol table processing > > bindings display information about symbol binding > > versions display version dependencies > > all all previous options combined > > statistics display relocation statistics > > unused determined unused DSOs > > help display this help message and exit > >=20 > > To direct the debugging output into a file instead of standard output > > a filename can be specified using the LD_DEBUG_OUTPUT environment varia= ble. >=20 > With libs: >=20 > > # env LD_DEBUG=3Dlibs ./app > > ... > > 9328: find library=3Dlibtcl.so [0]; searching > > 9328: search cache=3D/etc/ld.so.cache > > 9328: search path=3D/lib64/tls/x86_64:/lib64/tls:/lib64/x86_= 64:/lib64:/usr/lib64/tls/x86_64:/usr/lib64/tls:/usr/lib64/x86_64:/usr/lib64= (system search path) > > 9328: trying file=3D/lib64/tls/x86_64/libtcl.so > > 9328: trying file=3D/lib64/tls/libtcl.so > > 9328: trying file=3D/lib64/x86_64/libtcl.so > > 9328: trying file=3D/lib64/libtcl.so > > 9328: trying file=3D/usr/lib64/tls/x86_64/libtcl.so > > 9328: trying file=3D/usr/lib64/tls/libtcl.so > > 9328: trying file=3D/usr/lib64/x86_64/libtcl.so > > 9328: trying file=3D/usr/lib64/libtcl.so > > 9328: > > 9328: find library=3Dlibpthread.so.0 [0]; searching > > 9328: search path=3D/usr/lib64 (system search path) > > 9328: trying file=3D/usr/lib64/libpthread.so.0 > > 9328: search cache=3D/etc/ld.so.cache > > 9328: trying file=3D/lib64/libpthread.so.0 > > 9328: > > 9328: > > 9328: calling init: /lib64/libpthread.so.0 > > 9328: > > 9328: > > 9328: calling init: /usr/lib64/libtcl.so > > 9328: > > 9328: find library=3Dlibnss_compat.so.2 [0]; searching > > 9328: search cache=3D/etc/ld.so.cache > > 9328: trying file=3D/lib64/libnss_compat.so.2 > > 9328: > > 9328: find library=3Dlibnsl.so.1 [0]; searching > > 9328: search cache=3D/etc/ld.so.cache > > 9328: trying file=3D/lib64/libnsl.so.1 > > 9328: > > 9328: > > 9328: calling init: /lib64/libnsl.so.1 > > 9328: > > 9328: > > 9328: calling init: /lib64/libnss_compat.so.2 > > 9328: > > 9328: find library=3Dlibnss_nis.so.2 [0]; searching > > 9328: search cache=3D/etc/ld.so.cache > > 9328: trying file=3D/lib64/libnss_nis.so.2 > > 9328: > > 9328: find library=3Dlibnss_files.so.2 [0]; searching > > 9328: search cache=3D/etc/ld.so.cache > > 9328: trying file=3D/lib64/libnss_files.so.2 > > 9328: > > 9328: > > 9328: calling init: /lib64/libnss_files.so.2 > > 9328: > > 9328: > > 9328: calling init: /lib64/libnss_nis.so.2 > > ^C > > 9328: calling fini: /usr/lib64/libtcl.so [0] > > 9328: > > 9328: > > 9328: calling fini: /lib64/libm.so.6 [0] > > 9328: > > 9328: > > 9328: calling fini: /lib64/libdl.so.2 [0] > > 9328: > > 9328: > > 9328: calling fini: /lib64/libpthread.so.0 [0] > > 9328: > > 9328: > > 9328: calling fini: /lib64/libnss_compat.so.2 [0] > > 9328: > > 9328: > > 9328: calling fini: /lib64/libnss_nis.so.2 [0] > > 9328: > > 9328: > > 9328: calling fini: /lib64/libnsl.so.1 [0] > > 9328: > > 9328: > > 9328: calling fini: /lib64/libnss_files.so.2 [0] > > 9328: > > 9328: > > 9328: calling fini: /lib64/libc.so.6 [0] >=20 > With symbols: >=20 > > # LD_DEBUG=3Dsymbols ./app > > ... > > 9355: symbol=3DTcl_SetObjResult; lookup in file=3D/lib64/lib= dl.so.2 [0] > > 9355: symbol=3DTcl_SetObjResult; lookup in file=3D/usr/lib/g= cc/x86_64-pc-linux-gnu/4.8.2/libstdc++.so.6 [0] > > 9355: symbol=3DTcl_SetObjResult; lookup in file=3D/lib64/lib= m.so.6 [0] > > 9355: symbol=3DTcl_SetObjResult; lookup in file=3D/usr/lib/g= cc/x86_64-pc-linux-gnu/4.8.2/libgcc_s.so.1 [0] > > 9355: symbol=3DTcl_SetObjResult; lookup in file=3D/lib64/lib= c.so.6 [0] > > 9355: symbol=3DTcl_SetObjResult; lookup in file=3D/lib64/ld-= linux-x86-64.so.2 [0] > > 9355: symbol=3DTcl_SetObjResult; lookup in file=3D/usr/lib64= /libcrypto.so.1.0.0 [0] > > 9355: symbol=3DTcl_SetObjResult; lookup in file=3D/lib64/lib= z.so.1 [0] > > 9355: symbol=3DTcl_SetObjResult; lookup in file=3D/usr/lib64= /libtcl.so [0] > > 9355: symbol=3DTcl_GetObjResult; lookup in file=3D./netplay = [0] > > 9355: symbol=3DTcl_GetObjResult; lookup in file=3D/lib64/lib= dl.so.2 [0] > > 9355: symbol=3DTcl_GetObjResult; lookup in file=3D/usr/lib/g= cc/x86_64-pc-linux-gnu/4.8.2/libstdc++.so.6 [0] > > 9355: symbol=3DTcl_GetObjResult; lookup in file=3D/lib64/lib= m.so.6 [0] > > 9355: symbol=3DTcl_GetObjResult; lookup in file=3D/usr/lib/g= cc/x86_64-pc-linux-gnu/4.8.2/libgcc_s.so.1 [0] > > 9355: symbol=3DTcl_GetObjResult; lookup in file=3D/lib64/lib= c.so.6 [0] > > 9355: symbol=3DTcl_GetObjResult; lookup in file=3D/lib64/ld-= linux-x86-64.so.2 [0] > > 9355: symbol=3DTcl_GetObjResult; lookup in file=3D/usr/lib64= /libcrypto.so.1.0.0 [0] > > 9355: symbol=3DTcl_GetObjResult; lookup in file=3D/lib64/lib= z.so.1 [0] > > 9355: symbol=3DTcl_GetObjResult; lookup in file=3D/usr/lib64= /libtcl.so [0] >=20 >=20 > I would love to work on adding such a feature to ours, especially if it > can assist with debugging rtld itself. I would think with some > predict_false() checks we could avoid much of the performance penalties. As I said, for debugging rtld, currently existing or, probably, any predefined set of debugging statements is mostly useless. A startup problems usually results in coredumps, which is compfortably debugged by loading core with debugging symbols (-g) in gdb. If you can determine useful set of the debugging output for application programmers to introspect the rtld activity, I am all for adding it and enabling LD_DEBUG unconditionally. >=20 > As a side note I think I may have found an rtld issue (on 8.4) with -32 > handling. Being a production machine I don't want to attempt installing > a DEBUG rtld to figure out the problem. Having LD_DEBUG would have been > useful on this as well. >=20 > The issue I ran into with -32 was from converting a i386 system to > amd64. I have hundreds of out-of-ports binaries that need to be manually > recompiled. To allow delaying this task for a bit I copied all of the > libraries from /usr/local/lib from the old i386 system to > /usr/local/lib32/compat/system and added all of the paths in there to > ldconfig: >=20 > > [~] # ldconfig -32 -r|head -n3 > > /var/run/ld-elf32.so.hints: > > search directories: /usr/lib32:/usr/local/lib32/compat/system:/= usr/local/lib32/compat/system/mysql:/usr/local/lib32/compat/system/gnutls3:= /usr/local/lib32/compat/system/pth:/usr/local/lib32/compat/system/qt4:/usr/= local/lib32/compat/system/tdom0.8.3:/usr/local/lib32/compat/system/gcc46:/u= sr/local/lib32/compat/system/gcc47:/usr/local/lib32/compat/system/gcc48:/us= r/local/lib32/compat/system/gcc49:/usr/local/lib32/compat/system/event2:/us= r/local/lib32/compat/pkg:/usr/local/lib32/compat >=20 > It worked for 90% of the system but then I ran into some libraries that > refused to find their own dependencies. The issue seemingly is that > dependencies for libraries that are also in the -32 path are not looked u= p. >=20 > For example: >=20 > > [~] # ldd -a /usr/local/lib32/compat/system/libssl.so > > /usr/local/lib32/compat/system/libssl.so: > > libcrypto.so.8 =3D> not found (0x0) > > libthr.so.3 =3D> /usr/lib32/libthr.so.3 (0x310fb000) > > libc.so.7 =3D> /usr/lib32/libc.so.7 (0x29165000) > > /usr/lib32/libthr.so.3: > > libc.so.7 =3D> /usr/lib32/libc.so.7 (0x29165000) > > [~] # ldconfig -32 -r|grep libcrypto.so.8 > > 105:-lcrypto.8 =3D> /usr/local/lib32/compat/system/libcrypto.so= =2E8 > > [~] # readelf -a /usr/local/lib32/compat/system/libssl.so|egrep "(path|= NEEDED)" > > 0x00000001 (NEEDED) Shared library: [libcrypto.so.= 8] > > 0x00000001 (NEEDED) Shared library: [libthr.so.3] > > 0x00000001 (NEEDED) Shared library: [libc.so.7] > > 0x0000000f (RPATH) Library rpath: [/usr/local/lib] >=20 > My guess is it is the RPATH. I am not sure without a simple debug ability. >=20 > --=20 > Regards, > Bryan Drewery >=20 --JpBuuFAFSbNvDzdI Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBAgAGBQJUAifZAAoJEJDCuSvBvK1BnEIP/jK7QVFtc8K3qj3q016N3ZNd ZWYGWbEOFi9MUKzvySqUwkAQaSOBsqjc9ezUGR+RyQJn1oO5pXBQ2LQDObZq2R5+ 1iyNFx3WzqaTf25OGh8bUQzhunWjwkcaKYtOR+ZzAVAH97298qNhDlxK9buA7FnI AxjQFKkVYd8wZUt5XP4GUzruXQSPU6C077L9MrppJpXi5HRoVPjHoM5Wqt8jdeEc KpVhyFp5uNp2naAJNsBULU7tRGa7Sso+wQHnMIAPZkdCMI9XNNb9Y7E2k3EkNIdI I2XMbDD+ZLC/DFmhZBIvygIX9jgpi1tnvulLjLqlL/Tzy5QChxsZ+vsiTFgCQwGU bwmeWuf6PcW27FN/ncT74GZ8mc197UI9vSXrqNfbLbanU0pvgWKIL2tyePWofItb bHSwj2Wv75IASeogIbfECqwi5MtzomhtebHVtaBH5Kdsrl1S8DivS8U7l7sJsd+1 vLG0gg95aLS3IvTY5F8zVpcWlZvRnjrCGUhjsR9VR67Sb8L2O4AS2AvBBDj8KqYH x/c+9v2LtSS5WTRuMMV54a86KWw9OhbWJwcWaJV67u0PlUvsMiBBBAW6cPIwKfTr dQrKnGsHryKZfcHMrPsxR8/Jl1++6JxvJz5TlHYqox3naF5hs7iOWkGZq628nEyl EkxVzbgfWn55NiBzRqba =Emte -----END PGP SIGNATURE----- --JpBuuFAFSbNvDzdI--