From owner-freebsd-toolchain@FreeBSD.ORG Wed Jul 17 20:13:42 2013 Return-Path: Delivered-To: freebsd-toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id C9AF040A for ; Wed, 17 Jul 2013 20:13:42 +0000 (UTC) (envelope-from lidl@pix.net) Received: from hydra.pix.net (hydra.pix.net [IPv6:2001:470:e254::3c]) by mx1.freebsd.org (Postfix) with ESMTP id 9DC8B8E3 for ; Wed, 17 Jul 2013 20:13:42 +0000 (UTC) Received: from torb.pix.net (torb.pix.net [IPv6:2001:470:e254:10:12dd:b1ff:febf:eca9]) (authenticated bits=0) by hydra.pix.net (8.14.5/8.14.5) with ESMTP id r6HKDfx8075871 for ; Wed, 17 Jul 2013 16:13:41 -0400 (EDT) (envelope-from lidl@pix.net) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.97.8 at mail.pix.net Message-ID: <51E6FAF5.3080802@pix.net> Date: Wed, 17 Jul 2013 16:13:41 -0400 From: Kurt Lidl User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:17.0) Gecko/20130620 Thunderbird/17.0.7 MIME-Version: 1.0 To: freebsd-toolchain@freebsd.org Subject: clang looking all over for linux libs [9.2 PRERELEASE] Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Jul 2013 20:13:42 -0000 I noticed this issue this morning, while looking at a unrelated compilation failure. I was using clang on an amd64 machine, running a system that corresponds closely to r253388. There are some local changes, but nothing with regards to the toolchain, except for removing GCC via the WITHOUT_GCC flag in src.conf. lidl@nine0-135: cat hello.c #include #include int main(int argc, char *argv[]) { printf("Hello world\n"); return 0; } lidl@nine0-136: ktrace -i clang -c hello.c lidl@nine0-137: kdump |less [...] 74004 clang CALL open(0x7fffffffbf00,0x120004,0x1d) 74004 clang NAMI "/usr/lib/gcc/x86_64-linux-gnu" 74004 clang RET open -1 errno 2 No such file or directory 74004 clang CALL open(0x7fffffffbf00,0x120004,0x2e) 74004 clang NAMI "/usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu" 74004 clang RET open -1 errno 2 No such file or directory My question is: Why is the system compiler looking for all these directories that do not exist? lidl@nine0-144: kdump |egrep -e NAMI -e /usr/lib | awk '{print $4}' [...] "/usr/lib/gcc/x86_64-linux-gnu" "/usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu" "/usr/lib/x86_64-linux-gnu" "/usr/lib/gcc/x86_64-unknown-linux-gnu" "/usr/lib/x86_64-unknown-linux-gnu/gcc/x86_64-unknown-linux-gnu" "/usr/lib/x86_64-unknown-linux-gnu" "/usr/lib/gcc/x86_64-pc-linux-gnu" "/usr/lib/x86_64-pc-linux-gnu/gcc/x86_64-pc-linux-gnu" "/usr/lib/x86_64-pc-linux-gnu" "/usr/lib/gcc/x86_64-redhat-linux6E" "/usr/lib/x86_64-redhat-linux6E/gcc/x86_64-redhat-linux6E" "/usr/lib/x86_64-redhat-linux6E" "/usr/lib/gcc/x86_64-redhat-linux" "/usr/lib/x86_64-redhat-linux/gcc/x86_64-redhat-linux" "/usr/lib/x86_64-redhat-linux" "/usr/lib/gcc/x86_64-suse-linux" "/usr/lib/x86_64-suse-linux/gcc/x86_64-suse-linux" "/usr/lib/x86_64-suse-linux" "/usr/lib/gcc/x86_64-manbo-linux-gnu" "/usr/lib/x86_64-manbo-linux-gnu/gcc/x86_64-manbo-linux-gnu" "/usr/lib/x86_64-manbo-linux-gnu" "/usr/lib/gcc/x86_64-linux-gnu" "/usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu" "/usr/lib/x86_64-linux-gnu" "/usr/lib/gcc/x86_64-slackware-linux" "/usr/lib/x86_64-slackware-linux/gcc/x86_64-slackware-linux" "/usr/lib/x86_64-slackware-linux" "/usr/lib/gcc/x86_64-unknown-freebsd9.2" "/usr/lib/x86_64-unknown-freebsd9.2/gcc/x86_64-unknown-freebsd9.2" "/usr/lib/x86_64-unknown-freebsd9.2" It's actually a lot worse than this -- it also looks in /usr/lib32 for a different set of directories, and then again in /usr/lib for that diffferent set of libraries, and then in /usr/bin/../lib for the original set of directories and then again(!) in /usr/bin/../lib32 and /usr/bin/../lib for the second set of directories. I would think, that as the configured system compiler, it wouldn't bother looking around for a bunch of stuff that it isn't going to find... Sure, the data is in the buffer cache after the first run, but how about just not making several hundred useless system calls for each invocation of the compiler? All this seems to come from: /usr/src/contrib/llvm/tools/clang/lib/Driver/ToolChains.cpp Is there something that can done to make this work better? -Kurt From owner-freebsd-toolchain@FreeBSD.ORG Wed Jul 17 21:33:50 2013 Return-Path: Delivered-To: freebsd-toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id B364BA5B for ; Wed, 17 Jul 2013 21:33:50 +0000 (UTC) (envelope-from rdivacky@vlakno.cz) Received: from vlakno.cz (mail.vlakno.cz [178.238.39.38]) by mx1.freebsd.org (Postfix) with ESMTP id 2D1FCCAF for ; Wed, 17 Jul 2013 21:33:49 +0000 (UTC) Received: by vlakno.cz (Postfix, from userid 1002) id 0983B1CC55C7; Wed, 17 Jul 2013 23:33:48 +0200 (CEST) Date: Wed, 17 Jul 2013 23:33:48 +0200 From: Roman Divacky To: Kurt Lidl Subject: Re: clang looking all over for linux libs [9.2 PRERELEASE] Message-ID: <20130717213347.GA23999@freebsd.org> References: <51E6FAF5.3080802@pix.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <51E6FAF5.3080802@pix.net> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-toolchain@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Jul 2013 21:33:50 -0000 Yes, this is because the FreeBSD driver toolchain shares the same infrastructure with Linux. So we stat ~200 linux dirs :( This rys.vlakno.cz/~rdivacky/freebsd-driver.patch proof of concept patch fixes it by unsharing FreeBSD from Linux but I am not convinced it's worth it. It should also be possible to only stat those libs when -m32/-m64 is specified. That would be the preferred solution. Roman On Wed, Jul 17, 2013 at 04:13:41PM -0400, Kurt Lidl wrote: > > I noticed this issue this morning, while looking at a unrelated > compilation failure. I was using clang on an amd64 machine, > running a system that corresponds closely to r253388. > There are some local changes, but nothing with regards to the > toolchain, except for removing GCC via the WITHOUT_GCC flag in src.conf. > > lidl@nine0-135: cat hello.c > #include > #include > > int main(int argc, char *argv[]) > { > printf("Hello world\n"); > return 0; > } > lidl@nine0-136: ktrace -i clang -c hello.c > lidl@nine0-137: kdump |less > [...] > 74004 clang CALL > open(0x7fffffffbf00,0x120004,0x1d) > 74004 clang NAMI "/usr/lib/gcc/x86_64-linux-gnu" > 74004 clang RET open -1 errno 2 No such file or directory > 74004 clang CALL > open(0x7fffffffbf00,0x120004,0x2e) > 74004 clang NAMI "/usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu" > 74004 clang RET open -1 errno 2 No such file or directory > > My question is: Why is the system compiler looking for all these > directories that do not exist? > > lidl@nine0-144: kdump |egrep -e NAMI -e /usr/lib | awk '{print $4}' > [...] > "/usr/lib/gcc/x86_64-linux-gnu" > "/usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu" > "/usr/lib/x86_64-linux-gnu" > "/usr/lib/gcc/x86_64-unknown-linux-gnu" > "/usr/lib/x86_64-unknown-linux-gnu/gcc/x86_64-unknown-linux-gnu" > "/usr/lib/x86_64-unknown-linux-gnu" > "/usr/lib/gcc/x86_64-pc-linux-gnu" > "/usr/lib/x86_64-pc-linux-gnu/gcc/x86_64-pc-linux-gnu" > "/usr/lib/x86_64-pc-linux-gnu" > "/usr/lib/gcc/x86_64-redhat-linux6E" > "/usr/lib/x86_64-redhat-linux6E/gcc/x86_64-redhat-linux6E" > "/usr/lib/x86_64-redhat-linux6E" > "/usr/lib/gcc/x86_64-redhat-linux" > "/usr/lib/x86_64-redhat-linux/gcc/x86_64-redhat-linux" > "/usr/lib/x86_64-redhat-linux" > "/usr/lib/gcc/x86_64-suse-linux" > "/usr/lib/x86_64-suse-linux/gcc/x86_64-suse-linux" > "/usr/lib/x86_64-suse-linux" > "/usr/lib/gcc/x86_64-manbo-linux-gnu" > "/usr/lib/x86_64-manbo-linux-gnu/gcc/x86_64-manbo-linux-gnu" > "/usr/lib/x86_64-manbo-linux-gnu" > "/usr/lib/gcc/x86_64-linux-gnu" > "/usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu" > "/usr/lib/x86_64-linux-gnu" > "/usr/lib/gcc/x86_64-slackware-linux" > "/usr/lib/x86_64-slackware-linux/gcc/x86_64-slackware-linux" > "/usr/lib/x86_64-slackware-linux" > "/usr/lib/gcc/x86_64-unknown-freebsd9.2" > "/usr/lib/x86_64-unknown-freebsd9.2/gcc/x86_64-unknown-freebsd9.2" > "/usr/lib/x86_64-unknown-freebsd9.2" > > It's actually a lot worse than this -- it also looks in /usr/lib32 for > a different set of directories, and then again in /usr/lib > for that diffferent set of libraries, and then in /usr/bin/../lib > for the original set of directories and then again(!) in > /usr/bin/../lib32 and /usr/bin/../lib for the second set of directories. > > I would think, that as the configured system compiler, it wouldn't > bother looking around for a bunch of stuff that it isn't going to > find... Sure, the data is in the buffer cache after the first run, > but how about just not making several hundred useless system calls > for each invocation of the compiler? > > All this seems to come from: > /usr/src/contrib/llvm/tools/clang/lib/Driver/ToolChains.cpp > > Is there something that can done to make this work better? > > -Kurt > > > > _______________________________________________ > freebsd-toolchain@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-toolchain > To unsubscribe, send any mail to "freebsd-toolchain-unsubscribe@freebsd.org" From owner-freebsd-toolchain@FreeBSD.ORG Thu Jul 18 12:48:30 2013 Return-Path: Delivered-To: freebsd-toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id AA7CF5E6; Thu, 18 Jul 2013 12:48:30 +0000 (UTC) (envelope-from lidl@pix.net) Received: from hydra.pix.net (hydra.pix.net [IPv6:2001:470:e254::3c]) by mx1.freebsd.org (Postfix) with ESMTP id 62982BDE; Thu, 18 Jul 2013 12:48:29 +0000 (UTC) Received: from torb.pix.net (torb.pix.net [IPv6:2001:470:e254:10:12dd:b1ff:febf:eca9]) (authenticated bits=0) by hydra.pix.net (8.14.5/8.14.5) with ESMTP id r6ICmRQS082828; Thu, 18 Jul 2013 08:48:27 -0400 (EDT) (envelope-from lidl@pix.net) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.97.8 at mail.pix.net Message-ID: <51E7E41B.3030504@pix.net> Date: Thu, 18 Jul 2013 08:48:27 -0400 From: Kurt Lidl User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:17.0) Gecko/20130620 Thunderbird/17.0.7 MIME-Version: 1.0 To: Roman Divacky Subject: Re: clang looking all over for linux libs [9.2 PRERELEASE] References: <51E6FAF5.3080802@pix.net> <20130717213347.GA23999@freebsd.org> In-Reply-To: <20130717213347.GA23999@freebsd.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-toolchain@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Jul 2013 12:48:30 -0000 On 7/17/13 5:33 PM, Roman Divacky wrote: > Yes, this is because the FreeBSD driver toolchain shares the same > infrastructure with Linux. So we stat ~200 linux dirs :( > > This rys.vlakno.cz/~rdivacky/freebsd-driver.patch proof of concept > patch fixes it by unsharing FreeBSD from Linux but I am not > convinced it's worth it. > > It should also be possible to only stat those libs when -m32/-m64 > is specified. That would be the preferred solution. > > Roman Thanks for that patch, but when applied to the 9.2-PRERELEASE clang tree, a buildworld fails like this: building static c library ranlib libc.a building shared library libc.so.7 /usr/bin/ld: this linker was not configured to use sysroots cc: error: linker command failed with exit code 1 (use -v to see invocation) *** [libc.so.7] Error code 1 1 error *** [lib/libc__L] Error code 2 1 error *** [libraries] Error code 2 1 error *** [_libraries] Error code 2 1 error *** [buildworld] Error code 2 1 error -Kurt > > On Wed, Jul 17, 2013 at 04:13:41PM -0400, Kurt Lidl wrote: >> >> I noticed this issue this morning, while looking at a unrelated >> compilation failure. I was using clang on an amd64 machine, >> running a system that corresponds closely to r253388. >> There are some local changes, but nothing with regards to the >> toolchain, except for removing GCC via the WITHOUT_GCC flag in src.conf. >> >> lidl@nine0-135: cat hello.c >> #include >> #include >> >> int main(int argc, char *argv[]) >> { >> printf("Hello world\n"); >> return 0; >> } >> lidl@nine0-136: ktrace -i clang -c hello.c >> lidl@nine0-137: kdump |less >> [...] >> 74004 clang CALL >> open(0x7fffffffbf00,0x120004,0x1d) >> 74004 clang NAMI "/usr/lib/gcc/x86_64-linux-gnu" >> 74004 clang RET open -1 errno 2 No such file or directory >> 74004 clang CALL >> open(0x7fffffffbf00,0x120004,0x2e) >> 74004 clang NAMI "/usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu" >> 74004 clang RET open -1 errno 2 No such file or directory >> >> My question is: Why is the system compiler looking for all these >> directories that do not exist? >> >> lidl@nine0-144: kdump |egrep -e NAMI -e /usr/lib | awk '{print $4}' >> [...] >> "/usr/lib/gcc/x86_64-linux-gnu" >> "/usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu" >> "/usr/lib/x86_64-linux-gnu" >> "/usr/lib/gcc/x86_64-unknown-linux-gnu" >> "/usr/lib/x86_64-unknown-linux-gnu/gcc/x86_64-unknown-linux-gnu" >> "/usr/lib/x86_64-unknown-linux-gnu" >> "/usr/lib/gcc/x86_64-pc-linux-gnu" >> "/usr/lib/x86_64-pc-linux-gnu/gcc/x86_64-pc-linux-gnu" >> "/usr/lib/x86_64-pc-linux-gnu" >> "/usr/lib/gcc/x86_64-redhat-linux6E" >> "/usr/lib/x86_64-redhat-linux6E/gcc/x86_64-redhat-linux6E" >> "/usr/lib/x86_64-redhat-linux6E" >> "/usr/lib/gcc/x86_64-redhat-linux" >> "/usr/lib/x86_64-redhat-linux/gcc/x86_64-redhat-linux" >> "/usr/lib/x86_64-redhat-linux" >> "/usr/lib/gcc/x86_64-suse-linux" >> "/usr/lib/x86_64-suse-linux/gcc/x86_64-suse-linux" >> "/usr/lib/x86_64-suse-linux" >> "/usr/lib/gcc/x86_64-manbo-linux-gnu" >> "/usr/lib/x86_64-manbo-linux-gnu/gcc/x86_64-manbo-linux-gnu" >> "/usr/lib/x86_64-manbo-linux-gnu" >> "/usr/lib/gcc/x86_64-linux-gnu" >> "/usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu" >> "/usr/lib/x86_64-linux-gnu" >> "/usr/lib/gcc/x86_64-slackware-linux" >> "/usr/lib/x86_64-slackware-linux/gcc/x86_64-slackware-linux" >> "/usr/lib/x86_64-slackware-linux" >> "/usr/lib/gcc/x86_64-unknown-freebsd9.2" >> "/usr/lib/x86_64-unknown-freebsd9.2/gcc/x86_64-unknown-freebsd9.2" >> "/usr/lib/x86_64-unknown-freebsd9.2" >> >> It's actually a lot worse than this -- it also looks in /usr/lib32 for >> a different set of directories, and then again in /usr/lib >> for that diffferent set of libraries, and then in /usr/bin/../lib >> for the original set of directories and then again(!) in >> /usr/bin/../lib32 and /usr/bin/../lib for the second set of directories. >> >> I would think, that as the configured system compiler, it wouldn't >> bother looking around for a bunch of stuff that it isn't going to >> find... Sure, the data is in the buffer cache after the first run, >> but how about just not making several hundred useless system calls >> for each invocation of the compiler? >> >> All this seems to come from: >> /usr/src/contrib/llvm/tools/clang/lib/Driver/ToolChains.cpp >> >> Is there something that can done to make this work better? >> >> -Kurt >> >> >> >> _______________________________________________ >> freebsd-toolchain@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-toolchain >> To unsubscribe, send any mail to "freebsd-toolchain-unsubscribe@freebsd.org" From owner-freebsd-toolchain@FreeBSD.ORG Thu Jul 18 13:16:18 2013 Return-Path: Delivered-To: freebsd-toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 25C22FCA; Thu, 18 Jul 2013 13:16:18 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from tensor.andric.com (tensor.andric.com [IPv6:2001:7b8:3a7:1:2d0:b7ff:fea0:8c26]) by mx1.freebsd.org (Postfix) with ESMTP id E2E7EDA2; Thu, 18 Jul 2013 13:16:17 +0000 (UTC) Received: from coleburn.avinity.tv (host-229-161-243.77.avinity.tv [77.243.161.229]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by tensor.andric.com (Postfix) with ESMTPSA id BDECA5C5A; Thu, 18 Jul 2013 15:16:13 +0200 (CEST) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.5 \(1508\)) Subject: Re: clang looking all over for linux libs [9.2 PRERELEASE] From: Dimitry Andric In-Reply-To: <51E7E41B.3030504@pix.net> Date: Thu, 18 Jul 2013 15:16:12 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <106B8A43-785E-48BC-83DE-DA0A0FC5604E@FreeBSD.org> References: <51E6FAF5.3080802@pix.net> <20130717213347.GA23999@freebsd.org> <51E7E41B.3030504@pix.net> To: Kurt Lidl X-Mailer: Apple Mail (2.1508) Cc: freebsd-toolchain@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Jul 2013 13:16:18 -0000 On Jul 18, 2013, at 14:48, Kurt Lidl wrote: > On 7/17/13 5:33 PM, Roman Divacky wrote: >> Yes, this is because the FreeBSD driver toolchain shares the same >> infrastructure with Linux. So we stat ~200 linux dirs :( >>=20 >> This rys.vlakno.cz/~rdivacky/freebsd-driver.patch proof of concept >> patch fixes it by unsharing FreeBSD from Linux but I am not >> convinced it's worth it. >>=20 >> It should also be possible to only stat those libs when -m32/-m64 >> is specified. That would be the preferred solution. >>=20 >> Roman >=20 > Thanks for that patch, but when applied to the 9.2-PRERELEASE > clang tree, a buildworld fails like this: >=20 > building static c library > ranlib libc.a > building shared library libc.so.7 > /usr/bin/ld: this linker was not configured to use sysroots > cc: error: linker command failed with exit code 1 (use -v to see = invocation) > *** [libc.so.7] Error code 1 > 1 error > *** [lib/libc__L] Error code 2 > 1 error > *** [libraries] Error code 2 > 1 error > *** [_libraries] Error code 2 > 1 error > *** [buildworld] Error code 2 > 1 error I'm not sure what goes wrong here, but please take the problem upstream, instead of using local patches. We want less of those, not more. :-) In any case, the probability of such a change making it into 9.2 is very small, since it is in no way critical. Apparently clang has been doing this for years, and you are the first one to notice... -Dimitry