From owner-freebsd-arch@FreeBSD.ORG Tue Mar 13 17:29:18 2012 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C7E69106564A for ; Tue, 13 Mar 2012 17:29:18 +0000 (UTC) (envelope-from ambrisko@ambrisko.com) Received: from mail.ambrisko.com (mail.ambrisko.com [70.91.206.90]) by mx1.freebsd.org (Postfix) with ESMTP id A9E388FC19 for ; Tue, 13 Mar 2012 17:29:18 +0000 (UTC) X-Ambrisko-Me: Yes Received: from server2.ambrisko.com (HELO internal.ambrisko.com) ([192.168.1.2]) by ironport.ambrisko.com with ESMTP; 13 Mar 2012 10:29:23 -0700 Received: from ambrisko.com (localhost [127.0.0.1]) by internal.ambrisko.com (8.14.4/8.14.4) with ESMTP id q2DHTHCI066202 for ; Tue, 13 Mar 2012 10:29:17 -0700 (PDT) (envelope-from ambrisko@ambrisko.com) Received: (from ambrisko@localhost) by ambrisko.com (8.14.4/8.14.4/Submit) id q2DHTH3b066201 for arch@freebsd.org; Tue, 13 Mar 2012 10:29:17 -0700 (PDT) (envelope-from ambrisko) From: Doug Ambrisko Message-Id: <201203131729.q2DHTH3b066201@ambrisko.com> To: arch@freebsd.org Date: Tue, 13 Mar 2012 10:29:17 -0700 (PDT) X-Mailer: ELM [version 2.4ME+ PL124d (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="US-ASCII" Cc: Subject: rtld enhancement to add osversion, version 2 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Mar 2012 17:29:19 -0000 This is round 2 of my rtld enhancements. The primary goal it to have rtld look in different places for libraries depending on the legacy binary that we want to run. This is especially a problem with binaries linked to libraries from ports since the version of a library in /usr/local/lib is the same whether it was FreeBSD 6, 7, 8 etc. until the library itself changes version due to an interface change. At work we need to run 3rd party binaries on our box of which we don't have source. Having a FreeBSD 6 binaries load /usr/local/lib/libiconv.so.3 that was built for FreeBSD 9 is not good. It is worse when libiconv.so.3 is linked to libc.so.7 when the FreeBSD 6 binary needs libc.so.6. I solved this by having rtld extract the OSVERSION from the binary and then use that to determine what to do. In my prior version, I inserted that into the library directory. That meant to pull in libc it would look at: /lib/603000/libc.so.6 /lib/6/libc.so.6 /lib/libc.so.6 to find libc.so.6. This meant a lot more look ups. Also I found it had a problem in that if we had an ambiguous name say /usr/local/lib/libcrypto.so.5 on the FreeBSD 6 machine and /lib/libcrypto.so.5 on the new FreeBSD 8 system, just doing the search logic would get a hit on /lib/libcrypto.so.5 before it got to /usr/local/lib/libcrypto.so.5. So then it would mean we'd have to put all FreeBSD lib's in /lib/6 to beat the search path. This wasn't a good solution. To solve the performance and path issue, I now follow the 32 bit hints file name model. Now it does a lookup of the hints file based on the osversion and major. So now it looks for the hints file as: /var/run/ld-elf-603000.so.hints /var/run/ld-elf-6.so.hints /var/run/ld-elf.so.hints This is faster and has more unique paths for FreeBSD 6 libraries since the FreeBSD 6 search paths would be in the hints file. I modified ldconfig to accept an "-os=" option to create this hints file. I tweaked /etc/rc* to make this easy to setup like this: ldconfig_os_versions="6" ldconfig_6_path="/usr/local/lib/compat/6" This doesn't solve the LD_LIBRARY_PATH or LD_PRELOAD. Solving that I still insert and iterate over the osverion, major and none into the path to find the library. The reason for this is that a FreeBSD 8 binary could exec a FreeBSD 6 binary that execs a FreeBSD 7 binary. If for the FreeBSD 6 binary we needed to set a custom LD_LIBRARY_PATH and the FreeBSD 7 binary find a library via the FreeBSD 6 search path then the FreeBSD 7 binary would die. By adding in the osversion search path I can put the FreeBSD 6 library into the search path + the directory 6. Then only FreeBSD 6 binaries can find it. An example: LD_LIBRARY_PATH=/usr/custom_software/lib /usr/custom_software/lib/6/libfoo.so.6 then only the FreeBSD 6 binary could load it. Since the searches would be for the FreeBSD 6 binary: /usr/custom_software/lib/603000/libfoo.so.6 /usr/custom_software/lib/6/libfoo.so.6 /usr/custom_software/lib/libfoo.so.6 and for FreeBSD 7 binary: /usr/custom_software/lib/702000/libfoo.so.6 /usr/custom_software/lib/7/libfoo.so.6 /usr/custom_software/lib/libfoo.so.6 Only the FreeBSD 6 binary would load /usr/custom_software/lib/6/libfoo.so.6. I do the same search with LD_PRELOAD. Final, is that prior binaries built on FreeBSD i386 but run on FreeBSD amd64 that set LD_* environemnt variables would fail on FreeBSD amd64 as is since it didn't set the equivalent LD32_*. To address this for COMPAT_32BIT I have rtld look for LD32_* first and then check for the LD_* second. This way legacy applications work fine. The patch is at: http://people.freebsd.org/~ambrisko/rtld_hints_osversion.patch and has been tested a fair amount at work. It can make running a different version of a FreeBSD binary very easy without having any cross-linking. There is a bug in FreeBSD 6.0 that didn't record the osversion in the binary so then this doesn't work. Try to avoid those binaries! We have a local hack that says if there is no osversion then assume it is a 6 binary. Again, this is to solve running legacy binaries on a new version of FreeBSD of which we don't have source to recompile with new ld features of -z origin. Comments? Thanks, Doug A.