Date: Thu, 12 Nov 2015 17:38:21 -0800 From: Bryan Drewery <bdrewery@FreeBSD.org> To: "Simon J. Gerraty" <sjg@juniper.net> Cc: brooks@FreeBSD.org, imp@FreeBSD.org, emaste@FreeBSD.org, toolchain@FreeBSD.org Subject: Re: Meta mode toolchain bootstrapping [was Re: FreeBSD targets/ out-of-date] Message-ID: <56453F0D.90206@FreeBSD.org> In-Reply-To: <13427.1447371730@chaos> References: <55E769EF.7090908@FreeBSD.org> <4924.1441306006@chaos> <56450AB8.90402@FreeBSD.org> <13427.1447371730@chaos>
next in thread | previous in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--0oqmDci2hpNGxNBp5Rtq0eMQ7TI1r758V
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable
On 11/12/2015 3:42 PM, Simon J. Gerraty wrote:
> Bryan Drewery <bdrewery@FreeBSD.org> wrote:
>=20
>> On 9/3/2015 11:46 AM, Simon J. Gerraty wrote:
>>> Anyway, bootstrap-toolchain leverages src/Makefile.inc1 to build an
>>> initial set of tools. It then attempts to use that to build toolchai=
n
>>> for MACHINE=3Dhost which is currently failing because in-tree libelf =
and
>>> libdwarf are needed and libelf needs sys/sys/elf_common.h but but tha=
t
>>> doesn't currently get staged for host (we try to minimize what we put=
in
>>> host stage tree).
>>
>> Using the special hack you came up with for lib/libelf, and a lot of
>> other fixes for replacing binutils with elftoolchain, I've managed to
>> get the targets/pseduo/bootstrap-tools build working again. I will
>> commit this likely today or tomorrow.
>=20
> Cool - thanks
> =20
>> However...
>=20
>>> It may be better to simply skip targets/pseudo/toolchain.host
>>> that will work so long as we set TOOLSDIR to point to the stuff built=
by
>>> Makefile.inc1=20
>>>
>>> Depending on where we are with external toolchian support that would
>>> make sense - might be able to skip bootstrap-toolchain completely
>>> which would be nice.
>>
>> I think this is more right because there's many more people thinking
>> about, testing daily, and maintaining the bootstrap logic in
>> Makefile.inc1. The way targets/pseudo/bootstrap-tools works currently
>> lead to bitrot and breakage with the elftoolchain replacement. It will=
>> very easily happen again.
>=20
> I'd be favor of removing the need completely.
>=20
> All that is required is *some* means of producing a viable toolchain in=
> such a way that you can point a variable at it.
>=20
>> What I plan to do is make pseudo/targets/bootstrap-tools always build
>> all of cross-tools,etc, from Makefile.inc1 respecting its own internal=
>> logic for when to bootstrap the compiler (without us telling it to ski=
p
>> the bootstrap compiler) and then add pseudo/targets/bootstrap-tools as=
a
>> global DIRDEP. Given the use of cookies, this will only be done once p=
er
>> meta build, but it at least won't be a manual step anymore. For the
>=20
> That can still add a lot of time to build, and IIRC building clang alon=
e
> poses problems for smaller machines (though using a mutex for linking
> clang bits could at least serialize the huge linker steps so as to avoi=
d
> exhausting memory).
>=20
> I quite like the way NetBSD allow separating the tools from the rest of=
> the build. I do the equivalent of 'make tools' very rarely - and so
> do not care how [in]efficient it is.
> IIRC it will throw an error if your tools are incompatible - prompting
> an update.
>=20
> I would suggest if you want to be able to hook bootstrap-tools in
> always, that you do it via a knob so it can be disabled.
I would just add some knob like WITH_BOOTSTRAP_TOOLCHAIN or
WITH_EXTERNAL_TOOLCHAIN to make this and the clang bootstrap just be
skipped and resort to the way the meta build works now.
>=20
>> record, I do plan to extend .MAKE.MODE=3Dmeta to buildworld and its
>> bootstrapping as well, which will give a lot of incremental build
>> improvements to it.
>=20
> WITH_META_FILES should give you improvements already in that regard.
Yes, it's a step. We'll need cookies in a lot of places too. I wish
WITH_META_MODE had been WITH_META_BUILD or WITH_DIRDEPS_BUILD so I could
check for "META_MODE" in the buildworld world and for discussion sake.
It seems I can use ${.MAKE.MODE:M*meta*} but that :U is needed in all
the uses. I'm not sure yet if :U really is needed. We have some
${MK_META_MODE} checks now around some cookies that would need to change
for what I'm planning.
>=20
>> This means that the meta build will then default to bootstrapping the
>> compiler, which we really must do to build the source tree reliably.
>=20
>> There's really no reason the meta build should default to not
>> bootstrapping the compiler. Setting WITHOUT_CROSS_COMPILER or
>> WITHOUT_CLANG_BOOTSTRAP will avoid this and use just the host compiler=
>> as the meta build does now. So there's no real problem for those peopl=
e
>> who want to skip the clang bootstrap.
>=20
> Ok, so long as there is a way to do so.
> Otherwise we'd need to add it locally for our own builds - where we nee=
d
> to use the toolchains provided by our compiler team.
Yes for sure. As noted above. At Isilon when a developer is building
they have no interest in building the toolchain 99% of the time. This is
why I am so committed to making this automatically skip the toolchain if
possible. As for having an external compiler from a team, that's kind of
already in the "external compiler" realm, so a WITH_EXTERNAL_COMPILER
would make sense to short-circuit all of what I'm wanting. Then you can
add it to your local make.conf or local.sys.mk and not have any change
in behavior.
>=20
>> Then, I will work on the project of "building clang less" which will
>> avoid building the bootstrap compiler for both the meta mode build and=
>> the buildworld build (and universe eventually) if /usr/bin/cc satisfie=
s
>> the needs (can cross compile the TARGET and is "new enough" compared t=
o
>> the version we want). This is not trivial but I think it is possible a=
nd
>> it will make everyone happy!
>=20
> Indeed. As I say, NetBSD have this reasonably sorted.
> But of course they have 2k line shell script driving a lot of it ;-)
Yes the NetBSD build, behavior wise, really impresses me.
>=20
>> Related tidbit, using WITHOUT_CROSS_COMPILER (or WITHOUT_*_BOOTSTRAP)
>> with buildworld leads to a broken build currently since it is the user=
>> basically asking to use their default external toolchain of /usr/bin/*=
,
>> but the logic does not kick in to use --sysroot against WORLDTMP. I pl=
an
>> to fix this soon as well.
>=20
> Assuming that you have previously built the correct toolchain
> it should be valid to have something like -DWITH_TOOLSDIR
> -DWITHOUT_BOOTSTRAP_TOOLSDIR (or -DWITHOUT_TOOLSDIR_BOOTSTRAP)
> Such that the build logic is identical - all that is being skipped is
> the expense of re-generating the toolchain
>=20
--=20
Regards,
Bryan Drewery
--0oqmDci2hpNGxNBp5Rtq0eMQ7TI1r758V
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQEcBAEBAgAGBQJWRT8OAAoJEDXXcbtuRpfP/xwH/A8moFzEflsJJvpXjEWxnCy+
jH0vXJsQTzAfLZv7LeqvN8GnPbI/hta/fwhXnLJXdePYWdmriHkQhYtk1IIccRz6
8Da/li4i+8Acn/Pf+P2Ka0Wlzf+upeNYlURiKt+p1oZqVkzlBQp32kpnPiy/r+ID
DXuAAg2fRbND13VWsptx07sCzVOtMw0eAaGu9Gbgyf4K+u9bJMMJ6sJa5KoPbZKl
df6zNhnwZsWY67kt4cL2JgW+whrCX+Mn8cVNLKahHgUNbPxL3qUTIJPbefu4JB35
jhp9hDz+2dU/JIOULTMK9ULaAH2+rjTn9K2a8I8xSL64YypIg+KdzRqrkY2BHt8=
=P1il
-----END PGP SIGNATURE-----
--0oqmDci2hpNGxNBp5Rtq0eMQ7TI1r758V--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?56453F0D.90206>