Date: Mon, 19 Mar 2012 16:59:34 -0400 From: John Baldwin <jhb@freebsd.org> To: freebsd-arch@freebsd.org Subject: Re: rtld enhancement to add osversion, version 2 Message-ID: <201203191659.34568.jhb@freebsd.org> In-Reply-To: <201203131729.q2DHTH3b066201@ambrisko.com> References: <201203131729.q2DHTH3b066201@ambrisko.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tuesday, March 13, 2012 1:29:17 pm Doug Ambrisko wrote: > This is round 2 of my rtld enhancements. The primary goal it to have > rtld look in different places for libraries depending on the legacy > binary that we want to run. This is especially a problem with binaries > linked to libraries from ports since the version of a library in > /usr/local/lib is the same whether it was FreeBSD 6, 7, 8 etc. until the > library itself changes version due to an interface change. At work we > need to run 3rd party binaries on our box of which we don't have source. > Having a FreeBSD 6 binaries load /usr/local/lib/libiconv.so.3 that was > built for FreeBSD 9 is not good. It is worse when libiconv.so.3 is > linked to libc.so.7 when the FreeBSD 6 binary needs libc.so.6. > > I solved this by having rtld extract the OSVERSION from the binary > and then use that to determine what to do. In my prior version, > I inserted that into the library directory. That meant to pull > in libc it would look at: > /lib/603000/libc.so.6 > /lib/6/libc.so.6 > /lib/libc.so.6 > to find libc.so.6. This meant a lot more look ups. Also I found it > had a problem in that if we had an ambiguous name say > /usr/local/lib/libcrypto.so.5 on the FreeBSD 6 machine and /lib/libcrypto.so.5 > on the new FreeBSD 8 system, just doing the search logic would get > a hit on /lib/libcrypto.so.5 before it got to /usr/local/lib/libcrypto.so.5. > So then it would mean we'd have to put all FreeBSD lib's in /lib/6 to > beat the search path. This wasn't a good solution. To solve the > performance and path issue, I now follow the 32 bit hints file name > model. Now it does a lookup of the hints file based on the osversion > and major. So now it looks for the hints file as: > /var/run/ld-elf-603000.so.hints > /var/run/ld-elf-6.so.hints > /var/run/ld-elf.so.hints > This is faster and has more unique paths for FreeBSD 6 libraries since > the FreeBSD 6 search paths would be in the hints file. I modified > ldconfig to accept an "-os=<version>" option to create this hints file. > I tweaked /etc/rc* to make this easy to setup like this: > ldconfig_os_versions="6" > ldconfig_6_path="/usr/local/lib/compat/6" I think this is a definite improvement from before, thanks! > This doesn't solve the LD_LIBRARY_PATH or LD_PRELOAD. Solving that > I still insert and iterate over the osverion, major and none into the > path to find the library. The reason for this is that a FreeBSD 8 > binary could exec a FreeBSD 6 binary that execs a FreeBSD 7 binary. > If for the FreeBSD 6 binary we needed to set a custom LD_LIBRARY_PATH > and the FreeBSD 7 binary find a library via the FreeBSD 6 search path > then the FreeBSD 7 binary would die. By adding in the osversion search > path I can put the FreeBSD 6 library into the search path + the directory > 6. Then only FreeBSD 6 binaries can find it. An example: > LD_LIBRARY_PATH=/usr/custom_software/lib > /usr/custom_software/lib/6/libfoo.so.6 > then only the FreeBSD 6 binary could load it. Since the searches > would be for the FreeBSD 6 binary: > /usr/custom_software/lib/603000/libfoo.so.6 > /usr/custom_software/lib/6/libfoo.so.6 > /usr/custom_software/lib/libfoo.so.6 > and for FreeBSD 7 binary: > /usr/custom_software/lib/702000/libfoo.so.6 > /usr/custom_software/lib/7/libfoo.so.6 > /usr/custom_software/lib/libfoo.so.6 > Only the FreeBSD 6 binary would load /usr/custom_software/lib/6/libfoo.so.6. > I do the same search with LD_PRELOAD. Hmm, I'm still not quite fan of this. Perhaps you could add an extension to ldconfig and the hints file to handle this case? That is, a way to store path mappings so you could do something like: ldconfig -os=6 -p /usr/local/lib /usr/local/lib/6 Or maybe you could make it an extension of how 'm' worked, so you could make directories accept an optional set of colon-separated paths that they serve as aliases for: ldconfig -os=6 -m /usr/local/lib/6:/usr/local/lib:/usr/lib (That would even fit into your existing rc.d script changes I believe). Then rtld would keep this internal directory mapping and be able to map the '/usr/local/lib' and '/usr/lib' directories in LD_PRELOAD and LD_LIBRARY_PATH to /usr/local/lib/6. The advantage of this is the same as with your previous change that all the mappings are configurable and not hard-coded into rtld itself. > Final, is that prior binaries built on FreeBSD i386 but run on FreeBSD amd64 > that set LD_* environemnt variables would fail on FreeBSD amd64 as is > since it didn't set the equivalent LD32_*. To address this for COMPAT_32BIT > I have rtld look for LD32_* first and then check for the LD_* second. > This way legacy applications work fine. Hmm, so Yahoo! had some patches to handle this as well. I think Yahoo's patches were different though. They actually patched the 32-bit libc to capture attempts to get or set LD_* and convert them to actually get/set LD32_* instead. I'm not sure which approach is best, but it might be worth asking Peter why Yahoo! did it that way and if there were reasons for that approach vs. doing it in rtld. I think the primary reason was so that you could set LD_LIBRARY_PATH or LD_PRELOAD to reference 64-bit libraries and not have it break 32-bit apps, but if a 32-bit app tried to set a variable before launching another app it would still DTRT. I do think this is definitely a problem worth solving. -- John Baldwin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201203191659.34568.jhb>