Skip site navigation (1)Skip section navigation (2)
Date:      Mon, 9 Oct 2017 21:57:04 -0700
From:      Nathan Whitehorn <nwhitehorn@freebsd.org>
To:        freebsd-arch@freebsd.org
Subject:   Re: Making C++11 a hard requirement for FreeBSD
Message-ID:  <577d3900-76f2-2c52-8ada-b8fb1fe881be@freebsd.org>
In-Reply-To: <CANCZdfrqoHh2_knYrmNZF5d=VFt5csuFu%2BkT5XpKKnH%2B0-Xbig@mail.gmail.com>
References:  <CANCZdfq5=KRp4NYKsc15gyS9C7CxrBFxcKQLPwnb_0oPb15vJw@mail.gmail.com> <2116882.XEKuxOb729@ralph.baldwin.cx> <20171006072010.ygq3k5ygwxykk4nb@ivaldir.net> <e1b1b84a-9f6f-d0d7-7e26-b3ef3cc35698@freebsd.org> <CANCZdfoTJG6=d9xtyiNJ8sTdsnm_SfHaMhPbqgALv75dKgb8Kg@mail.gmail.com> <ef8f368d-7e21-a5dd-5351-9cb6cc020798@freebsd.org> <CANCZdfrqoHh2_knYrmNZF5d=VFt5csuFu%2BkT5XpKKnH%2B0-Xbig@mail.gmail.com>

next in thread | previous in thread | raw e-mail | index | archive | help


On 10/09/17 11:32, Warner Losh wrote:
> On Oct 9, 2017 11:47 AM, "Nathan Whitehorn" <nwhitehorn@freebsd.org> wrote:
>
>
>
> On 10/08/17 22:26, Warner Losh wrote:
>
>
>
> On Sun, Oct 8, 2017 at 11:01 PM, Nathan Whitehorn <nwhitehorn@freebsd.org>
> wrote:
>
>>
>> On 10/06/17 00:20, Baptiste Daroussin wrote:
>>
>>> On Thu, Oct 05, 2017 at 11:47:57PM +0000, John Baldwin wrote:
>>>
>>>> On Thursday, October 05, 2017 04:28:44 PM Warner Losh wrote:
>>>>
>>>>> I'd like to start a conversation about the viability of making C++11 a
>>>>> hard
>>>>> requirement for bootstrapping FreeBSD and setting a specific deadline
>>>>> for
>>>>> doing so.
>>>>>
>>>>> This discussion is motivated by an ask from the jemalloc folks to use a
>>>>> limited subset of C++11 inside of malloc in such a way that is C safe
>>>>> (so
>>>>> the C programs wouldn't bloat with a C++ runtime). That's an ongoing
>>>>> discussion in another forum, and isn't appropriate for this thread
>>>>> because
>>>>> this has become a frequent request (but if you have opinions, please
>>>>> find
>>>>> the thread in current@ about it). I don't know the timeline of their
>>>>> plans
>>>>> to do this.
>>>>>
>>>>> I'd like to take the rather vague plans we've had "before 12" and
>>>>> create a
>>>>> timeline for removal of gcc 4.2 coupled with a requirement for support
>>>>> in
>>>>> clang or a working external toolchain. This requirement would be coupled
>>>>> with the requirement that the external toolchain support C++11
>>>>> constructs.
>>>>>
>>>>> I'd like to propose we do this 12/31/17. Any architectures that can't
>>>>> meet
>>>>> this timeline will be disconnected from universe at that time and
>>>>> deleted
>>>>> 3/31/18.
>>>>>
>>>>> It's my belief that i386, amd64, arm, aarch64, powerpc and powerpc64 are
>>>>> ready for this change and mips* would be ready for this change with an
>>>>> external clang toolchain. I'm unsure of riscv and sparc64, but suspect
>>>>> that
>>>>> a newer version of gcc as an external toolchain could work.
>>>>>
>>>> In-tree clang 5.0 for MIPS mostly works (modulo some small patches).
>>>> However,
>>>> it requires external ld.bfd.  I know that there ld.lld can link a working
>>>> mips64 world with some patches (notably the multigot patch).  mips works
>>>> fine
>>>> with external GCC.  riscv is already using external GCC that is
>>>> C++11-capable.
>>>>
>>>> The problem with external GCC is that you can cross-build a world +
>>>> kernel
>>>> just fine and get a working system via CROSS_TOOLCHAIN=foo-gcc.  However,
>>>> that system has no viable '/usr/bin/cc' once GCC 4.2 is removed.  bapt@
>>>> started on ports to cross-build binutils and gcc packages via the base/*
>>>> ports, but those are not yet finished / fully tested.  I don't think
>>>> anyone
>>>> has thought about how release builds will work either with only an
>>>> external
>>>> toolchain available.  (I'm pretty sure sparc64 has booted fine with
>>>> external GCC, it's just in the same boat as mips with regard to
>>>> /usr/bin/cc.)
>>>>
>>> Actually I did test those and they were working (tested in qemu) they were
>>> working fine. I have given up working on them due to the lack of
>>> interested by
>>> the community (by interest I mean people really testing, working on it,
>>> not just
>>> saying "hey nice sounds cool").
>>>
>>> As for the boot when I initially worked on external toolchain sparc64 was
>>> my
>>> guinea pig and so yes it worked an booted just fine.
>>>
>> So far as I know, we never solved any of the infrastructural problems
>> associated with this concept:
>> 1. Providing built releases with a /usr/bin/cc
>> 2. Coversioning base and in-ports toolchain, including ensuring commit
>> atomicity between toolchains and libc
>> 3. Adding a dependency on ports for src, including out-of-tree code that
>> has to be fetched from external servers
>> 4. Getting make universe to do the right thing
>>
>> We really need to solve those. If we go the external toolchain route,
>> which it is not clear to me is the best option, #2 and #1 are quite complex
>> problems. I think that, frankly, a deadline in two months to solve this set
>> of political problems we have had for two years is probably crazy, but
>> maybe making the status quo unsustainable will finally force progress.
>>
> External toolchains have been in flight for 5 or more years now. It's time
> they land. Though the requirements for them have never included
> cross-threading between /usr/src and /usr/ports like you suggest above, and
> those sorts of things won't be sorted by my deadlines (which are closer to
> 3 months). Nor, imho, should they.
>
>
> Well, sure. But the fact remains that we cannot build usable systems with
> external toolchains right now. Those are real problems that need to be
> solved somehow.
>
>
> Sure we can. I've built a bootable i386 system with gcc 6. It is a solved
> problem.

"Bootable" is not the same as "usable" here. External toolchain 
powerpc64 *boots* fine -- and has for years -- but there isn't much you 
can actual *do* with the system except admire dmesg without the ability 
to install ports.

> Let's focus on #1, the largest if not the only major problem. If I build,
> say, a ppc64 system with an external toolchain right now, it boots and runs
> fine. But the system is completely unusable since:
> - There are no binary packages built for PPC64, because of project policy
> preventing the use of native build systems
>
>
> System is still usable w/o packages. People can still fire up custom
> poudrier repos. We could also change project policy. This is is a specific
> wrinkle for powerpc64 it seems.

How would I do that? We can't run these natively, because the system 
ships without a compiler. And we can't cross-compile, because the ports 
tree does not support that.

>
> - You cannot cross-compile packages for PPC64, because of limitations in
> QEMU
>
>
> Then we should fix those, like we did for arm and MIPS.

That's a tremendous amount of work on a non-FreeBSD codebase. Are we now 
going to require QEMU patches from platform maintainers?

> - There is no compiler installed in the base system, so you cannot install
> any software from source code
>
>
> You can install it as a package.

Where do you get the package?

>
> - You cannot build the compiler from source, because you don't have one to
> bootstrap from and can't get one pre-built because of the no-packages
> problem.
>
>
> If you fix the other problems, this is solved.

Yes, absolutely. But how do we solve the others?

> We can't ship systems like this -- how is anyone expected to be able to use
> them?
>
>
> Most systems don't build software, so there are plenty of uses.

I think all systems *install* software. If we don't have packages, then 
it has to be buildable from source and we need a compiler. We just can't 
close *both* paths.

>
> These are easy to fix -- for example, here are three possibilities -- but
> solutions been held up for *years* by project policy:
> 1. Allow Tier-2 packages to be built on native hardware without as much
> centralized control, letting pkg install of the toolchain work. This fixes
> ppc64, but maybe not embedded systems.
>
>
> If we can't build in QEMU then sure. It doesn't matter where the binaries
> come from, so long as they work.

Yes, agreed. This has not been project policy thus far, however.

> 2. Have external-toolchain builds cross-build (or native-build) and
> pre-install a standard compiler package from ports during make buildworld,
> which keeps the user experience of current tier-2 and later tier-1 systems.
>
>
> No need for this to be during buildworld. I see no benefit from that. You
> can install it after installworld either native or to a destdir.

Fair enough. Read this as "included as part of the release media and, 
ideally, extracted and installed by default". I do not care -- at all -- 
how it comes to be part of the release media/hosted by freebsd.org, so 
long as it is and is updated at least as often as releases, ideally as 
part of the release process.

> 3. Include a newer compiler fully in the tree or in some nearby repository,
> which does the same.
>
>
> Yea, that's also possible. But then it isn't an external toolchain.

Absolutely -- this is included for completeness in the more general 
conversation of (finally) removing GCC 4.2.

> If we break any of these policy impasses by the deadline, I am 100% for
> dropping GCC 4.2. But we *have* to do something. Just axing code while
> preventing a meaningful replacement isn't a solution.
>
>
>
> We have sometthing that works today with a few warts. We shouldn't let the
> warts get in the way.

The situation right now is pretty wart-y: there is no way to install 
software, of any kind, on a number of platforms with external 
toolchains, powerpc64 among them.

> I'm happy to delay if there are specific items that have a definitive
> timeline, but wanting 100% full integration from external toolchain with
> full packages isn't a gating factor here.

Absolutely! I don't want to make the perfect the enemy of the good, 
especially when the good is finally freeing ourselves from GCC 4.2. But 
we have been blocked for years by the fact that we have not been able, 
as a result of basically purely policy issues, to find a way to 
bootstrap a GCC 4.2-less system so that ports can be installed. And, as 
much as I like running fortune, I think not being able to install 
3rd-party software is a blocker.

> Besides, I thought powerpc64 just worked with clang...

Unfortunately, no. There are a wide variety of subtle issues. You can 
boot a kernel built with -O0, but higher optimization levels or userland 
code (especially C++) break.
-Nathan

>
> Warner
>
> Also, are there any directions for how to build a toolchain from the base
>> ports? I haven't been able to figure it out -- the README starts with "pkg
>> install", but that doesn't get me very far on platforms without binary
>> packages, which are the ones this is meant to target.
>
> External toolchains are by definition external. Not part of the tree nor
> integrated into the tree. How you get them depends on the external
> toolchain and is unspecified. You get a lower level of integration than you
> do with in-tree toolchains, not the same level with reach-overs into some
> external repo.
>
>
> I think you missed the point. I was asking how to build the port to test it
> -- it doesn't build at present and I can't figure out how to make an
> external toolchain via the "base" ports (the other toolchains in ports work
> fine).
> -Nathan
>
> Warner
> _______________________________________________
> freebsd-arch@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-arch
> To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org"
>




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?577d3900-76f2-2c52-8ada-b8fb1fe881be>