From owner-freebsd-hackers@FreeBSD.ORG Fri Oct 24 02:49:03 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B45F11065672 for ; Fri, 24 Oct 2008 02:49:03 +0000 (UTC) (envelope-from neldredge@math.ucsd.edu) Received: from euclid.ucsd.edu (euclid.ucsd.edu [132.239.145.52]) by mx1.freebsd.org (Postfix) with ESMTP id 819318FC12 for ; Fri, 24 Oct 2008 02:49:03 +0000 (UTC) (envelope-from neldredge@math.ucsd.edu) Received: from zeno.ucsd.edu (zeno.ucsd.edu [132.239.145.22]) by euclid.ucsd.edu (8.11.7p3+Sun/8.11.7) with ESMTP id m9O2n1611021; Thu, 23 Oct 2008 19:49:01 -0700 (PDT) Received: from localhost (neldredg@localhost) by zeno.ucsd.edu (8.11.7p3+Sun/8.11.7) with ESMTP id m9O2n1o12185; Thu, 23 Oct 2008 19:49:01 -0700 (PDT) X-Authentication-Warning: zeno.ucsd.edu: neldredg owned process doing -bs Date: Thu, 23 Oct 2008 19:49:01 -0700 (PDT) From: Nate Eldredge X-X-Sender: neldredg@zeno.ucsd.edu To: Alexander Sack In-Reply-To: <3c0b01820810231910u59f97cecg8b01f391a9b353f3@mail.gmail.com> Message-ID: References: <3c0b01820810231731s1b4d4659j7d1df8bf4abb229c@mail.gmail.com> <86hc72x0nx.fsf@ds4.des.no> <86d4hqwzur.fsf@ds4.des.no> <3c0b01820810231848r3e3e297cl3dc9bf1d0edcd588@mail.gmail.com> <3c0b01820810231910u59f97cecg8b01f391a9b353f3@mail.gmail.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: =?ISO-8859-1?Q?Dag-Erling_Sm=F8rgrav?= , freebsd-questions@freebsd.org, FreeBSD Hackers Subject: Re: Why does adding /usr/lib32 to LD_LIBRARY_PATH break 64-bit binaries? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Oct 2008 02:49:03 -0000 On Thu, 23 Oct 2008, Alexander Sack wrote: > Alright, well I found some weirdness: > > [root@hagen ~]# export LD_LIBRARY_PATH=/usr/bin:/usr/lib:/usr/lib32:/usr/lib64 > [root@hagen ~]# LD_DEBUG=1 ls > /libexec/ld-elf.so.1 is initialized, base address = 0x800506000 > RTLD dynamic = 0x80062ad78 > RTLD pltgot = 0x0 > processing main program's program header > Filling in DT_DEBUG entry > lm_init("(null)") > loading LD_PRELOAD libraries > loading needed objects > Searching for "libutil.so.5" > Trying "/usr/bin/libutil.so.5" > Trying "/usr/lib/libutil.so.5" > Trying "/usr/lib32/libutil.so.5" > loading "/usr/lib32/libutil.so.5" > /libexec/ld-elf.so.1: /usr/lib32/libutil.so.5: unsupported file layout > > That's because libutil.so.5 does not exist in /usr/lib only in /lib. > The /usr/lib directory has: > > [root@hagen ~]# ls -l /usr/lib/libutil* > -r--r--r-- 1 root wheel 100518 Aug 21 2007 /usr/lib/libutil.a > lrwxrwxrwx 1 root wheel 17 Sep 11 11:44 /usr/lib/libutil.so -> > /lib/libutil.so.5 > -r--r--r-- 1 root wheel 103846 Aug 21 2007 /usr/lib/libutil_p.a > > So rtld is looking for major number 5 of libutil, without the standard > /lib in my LD_LIBRARY_PATH it searches /usr/lib, doesn't find it but: > > [root@hagen ~]# ls -l /usr/lib32/libutil* > -r--r--r-- 1 root wheel 65274 Aug 21 2007 /usr/lib32/libutil.a > lrwxrwxrwx 1 root wheel 12 Sep 11 11:45 /usr/lib32/libutil.so -> > libutil.so.5 > -r--r--r-- 1 root wheel 46872 Aug 21 2007 /usr/lib32/libutil.so.5 > -r--r--r-- 1 root wheel 66918 Aug 21 2007 /usr/lib32/libutil_p.a > > And whalah, I'm broke since there is a libutil.so.5 in there. > > So my question to anyone out there, WHY does /usr/lib32 contain major > numbers but /usr/lib does not? This seems like a bug to me (FreeBSD > 7.0-RELEASE is the same) or at least a dubious design decision. I think the distinction is this. rtld is looking for libutil.so.5 (with version number). This file has to be in /lib, in the root filesystem, so that programs can run before /usr is mounted. libutil.so on the other hand is not searched for by rtld, but by ld (driven by cc), when the program is built. /usr/lib is the traditional place for it to search; I'm not sure if it searches /lib at all. In the case of static libraries, /usr/lib is certainly the right place for libutil.a to go, so having libutil.so there makes sense in my mind. I think your best bet is to dig into whatever is setting LD_LIBRARY_PATH and get it set correctly. Remove /usr/lib32 or at least ensure that /lib is searched first. Trying to change rtld's behavior is not the right approach, IMHO. -- Nate Eldredge neldredge@math.ucsd.edu