Skip site navigation (1)Skip section navigation (2)
Date:      Thu, 12 Nov 2015 15:42:10 -0800
From:      "Simon J. Gerraty" <sjg@juniper.net>
To:        Bryan Drewery <bdrewery@FreeBSD.org>
Cc:        <brooks@FreeBSD.org>, <imp@FreeBSD.org>, <emaste@FreeBSD.org>, <toolchain@FreeBSD.org>, <sjg@juniper.net>
Subject:   Re: Meta mode toolchain bootstrapping [was Re: FreeBSD targets/ out-of-date]
Message-ID:  <13427.1447371730@chaos>
In-Reply-To: <56450AB8.90402@FreeBSD.org>
References:  <55E769EF.7090908@FreeBSD.org> <4924.1441306006@chaos> <56450AB8.90402@FreeBSD.org>

next in thread | previous in thread | raw e-mail | index | archive | help
Bryan Drewery <bdrewery@FreeBSD.org> wrote:

> On 9/3/2015 11:46 AM, Simon J. Gerraty wrote:
> > Anyway, bootstrap-toolchain leverages src/Makefile.inc1 to build an
> > initial set of tools.  It then attempts to use that to build toolchain
> > for MACHINE=host which is currently failing because in-tree libelf and
> > libdwarf are needed and libelf needs sys/sys/elf_common.h but but that
> > doesn't currently get staged for host (we try to minimize what we put in
> > host stage tree).
> 
> Using the special hack you came up with for lib/libelf, and a lot of
> other fixes for replacing binutils with elftoolchain, I've managed to
> get the targets/pseduo/bootstrap-tools build working again. I will
> commit this likely today or tomorrow.

Cool - thanks
 
> However...

> > It may be better to simply skip targets/pseudo/toolchain.host
> > that will work so long as we set TOOLSDIR to point to the stuff built by
> > Makefile.inc1 
> > 
> > Depending on where we are with external toolchian support that would
> > make sense - might be able to skip bootstrap-toolchain completely
> > which would be nice.
> 
> I think this is more right because there's many more people thinking
> about, testing daily, and maintaining the bootstrap logic in
> Makefile.inc1. The way targets/pseudo/bootstrap-tools works currently
> lead to bitrot and breakage with the elftoolchain replacement. It will
> very easily happen again.

I'd be favor of removing the need completely.

All that is required is *some* means of producing a viable toolchain in
such a way that you can point a variable at it.

> What I plan to do is make pseudo/targets/bootstrap-tools always build
> all of cross-tools,etc, from Makefile.inc1 respecting its own internal
> logic for when to bootstrap the compiler (without us telling it to skip
> the bootstrap compiler) and then add pseudo/targets/bootstrap-tools as a
> global DIRDEP. Given the use of cookies, this will only be done once per
> meta build, but it at least won't be a manual step anymore. For the

That can still add a lot of time to build, and IIRC building clang alone
poses problems for smaller machines (though using a mutex for linking
clang bits could at least serialize the huge linker steps so as to avoid
exhausting memory).

I quite like the way NetBSD allow separating the tools from the rest of
the build.  I do the equivalent of 'make tools' very rarely - and so
do not care how [in]efficient it is.
IIRC it will throw an error if your tools are incompatible - prompting
an update.

I would suggest if you want to be able to hook bootstrap-tools in
always, that you do it via a knob so it can be disabled.

> record, I do plan to extend .MAKE.MODE=meta to buildworld and its
> bootstrapping as well, which will give a lot of incremental build
> improvements to it.

WITH_META_FILES should give you improvements already in that regard.

> This means that the meta build will then default to bootstrapping the
> compiler, which we really must do to build the source tree reliably.

> There's really no reason the meta build should default to not
> bootstrapping the compiler. Setting WITHOUT_CROSS_COMPILER or
> WITHOUT_CLANG_BOOTSTRAP will avoid this and use just the host compiler
> as the meta build does now. So there's no real problem for those people
> who want to skip the clang bootstrap.

Ok, so long as there is a way to do so.
Otherwise we'd need to add it locally for our own builds - where we need
to use the toolchains provided by our compiler team.

> Then, I will work on the project of "building clang less" which will
> avoid building the bootstrap compiler for both the meta mode build and
> the buildworld build (and universe eventually) if /usr/bin/cc satisfies
> the needs (can cross compile the TARGET and is "new enough" compared to
> the version we want). This is not trivial but I think it is possible and
> it will make everyone happy!

Indeed.  As I say, NetBSD have this reasonably sorted.
But of course they have 2k line shell script driving a lot of it ;-)

> Related tidbit, using WITHOUT_CROSS_COMPILER (or WITHOUT_*_BOOTSTRAP)
> with buildworld leads to a broken build currently since it is the user
> basically asking to use their default external toolchain of /usr/bin/*,
> but the logic does not kick in to use --sysroot against WORLDTMP. I plan
> to fix this soon as well.

Assuming that you have previously built the correct toolchain
it should be valid to have something like -DWITH_TOOLSDIR
-DWITHOUT_BOOTSTRAP_TOOLSDIR (or -DWITHOUT_TOOLSDIR_BOOTSTRAP)
Such that the build logic is identical - all that is being skipped is
the expense of re-generating the toolchain



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?13427.1447371730>