Date: Mon, 26 May 2014 16:31:21 -0600 From: Warner Losh <imp@bsdimp.com> To: Tijl Coosemans <tijl@FreeBSD.org> Cc: Baptiste Daroussin <bapt@freebsd.org>, src-committers@freebsd.org, Ian Lepore <ian@freebsd.org>, svn-src-all@freebsd.org, Glen Barber <gjb@freebsd.org>, Nathan Whitehorn <nwhitehorn@freebsd.org>, svn-src-head@freebsd.org Subject: Re: svn commit: r266553 - head/release/scripts Message-ID: <05D1A11D-5985-42EA-84AD-209A8B51D391@bsdimp.com> In-Reply-To: <20140527001811.3e9d3e8d@kalimero.tijl.coosemans.org> References: <201405221922.s4MJM4Y9025265@svn.freebsd.org> <537F6706.6070509@freebsd.org> <20140523153619.GF72340@ivaldir.etoilebsd.net> <537F6EBC.3080008@freebsd.org> <20140523162020.GG72340@ivaldir.etoilebsd.net> <C5A59513-AF58-4749-BCD7-F54BB6F56E90@gmail.com> <20140524165940.3c687553@kalimero.tijl.coosemans.org> <5380C311.60201@freebsd.org> <20140524185345.263f230d@kalimero.tijl.coosemans.org> <1400955835.1152.323.camel@revolution.hippie.lan> <5380EBA8.1030200@freebsd.org> <20140525011307.142b41ab@kalimero.tijl.coosemans.org> <3CCAFAD3-FABE-40EF-ABF9-815FE5826349@bsdimp.com> <9FE34CE4-C71F-4806-9EF6-30CB1051C62F@bsdimp.com> <20140526113502.239db74d@kalimero.tijl.coosemans.org> <5383522F.30108@freebsd.org> <DAD3E386-6555-4C43-9BBA-F3BFD28CC19B@bsdimp.com> <20140527001811.3e9d3e8d@kalimero.tijl.coosemans.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--Apple-Mail=_A40E86D8-A260-43EC-A917-4D9DC2799B17 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1253 On May 26, 2014, at 4:18 PM, Tijl Coosemans <tijl@FreeBSD.org> wrote: > On Mon, 26 May 2014 09:53:57 -0600 Warner Losh wrote: >> On May 26, 2014, at 8:39 AM, Nathan Whitehorn = <nwhitehorn@freebsd.org> wrote: >>> On 05/26/14 02:35, Tijl Coosemans wrote: >>>> On Sat, 24 May 2014 19:00:18 -0600 Warner Losh wrote: >>>>> On May 24, 2014, at 5:53 PM, Warner Losh <imp@bsdimp.com> wrote: >>>>>> On May 24, 2014, at 5:13 PM, Tijl Coosemans <tijl@freebsd.org> = wrote: >>>>>>> There isn't necessarily any chroot environment. There's one = kernel, >>>>>>> two equally valid ABIs (ILP32 and LP64) and any binary like = uname might >>>>>>> use either of them. If uname -p returns a different result = depending on >>>>>>> which of these two ABIs it was compiled for that could be a = problem for >>>>>>> any script that uses it. >>>>>> Well, it depends on what you want to do with the script, eh? If = you want >>>>>> to know the ABI of the native binary uname, that=92s one thing. = But if you >>>>>> want to know the supported ABIs, you are doing it wrong by using = uname. >>>>>> You should be using sysctl kern.supported_abi. That will tell you = all the >>>>>> ABIs that you can install packages for on this machine, which is = what you >>>>>> really want to know. So I=92m having trouble connecting the dots = between >>>>>> this and what you are saying here. >>>>>>=20 >>>>>> I still am absolutely flabbergasted why the MACHINE_ARCH names = aren=92t >>>>>> necessary and sufficient for packaging. I=92ve yet to see any = coherent >>>>>> reason to not use them. >>>>> Why do I care that they match? Good question. When I was doing = FreeNAS, I >>>>> looked at integrating pkgng into nanobsd. At the time this was = quite >>>>> difficult because every single architecture name was different = between >>>>> pkgng and MACHINE_ARCH. This would mean I=92d have to drag around = a huge >>>>> table to know how to translate one to the other (there was no = simple regex >>>>> either, and things like mipsn32 wouldn=92t have fit into the = scheme at the >>>>> time). I would very much like us to see us keep these names in = sync and >>>>> avoid large translation tables that are difficult to maintain. >>>>>=20 >>>>> Now, do you need to get it from uname -p? No. If you want to parse = elf >>>>> files to get it, that=92s fine, so long as the names map directly = to the >>>>> MACHINE_ARCH names that we=92ve been using for years. They = completely >>>>> describe the universe of supported platforms. Are they perfect? = No, around >>>>> the edge there may be an odd-ball that=92s possible to build, but = is >>>>> unsupported and likely doesn=92t work at all. Have we learned from = these >>>>> mistakes? Yes. Anything that=92s actively supported has a proper = name. This >>>>> name is needed, btw, so that any machine can self-host, a nice = feature of >>>>> the /usr/src system. >>>> ABI consists of the following elements: >>>>=20 >>>> - OS >>>> - OS ABI version (major version number in FreeBSD) >>=20 >> These two are encoded in FreeBSD and major version. There=92s no = problem >> encoding these in the package architecture string. They are easily >> scriptable and totally obvious to FreeBSD users and pose no problems. >> Nobody is opposed to these, and actually they are rather a good idea. >>=20 >>>> - instruction set >>>> - programming model (ILP32 or LP64) >>>> - byte order (little/big endian) >>=20 >> These three are encoded in MACHINE_ARCH and have been for quite some >> time. And you forgot several things as well: register conventions, >> calling conventions, stack alignment, struct alignment, pointer >> conversion conventions, address space layout, page size constraints, >> etc. There are simply far too many to try to break down like you are >> trying to do. And that=92s even before we get into shared library >> conventions... >=20 > I didn't forget them, I just restricted it to the elements that came = up > so far. All these extra elements are like byte order: you use only = one > of each per combination of the first four fields so they can be = discarded. > Things like calling conventions and register use can be considered = part > of the programming model. The amd64 programming models that matter to > FreeBSD (both ILP32 and LP64) are documented in the System V = Application > Binary Interface AMD64 Architecture Processor Supplement. Well yes and no. n32 and n64 have vastly different register conventions = than o32. But we=92re talking about something that=92s off in the weeds. = It doesn=92t matter. It also doesn=92t address my basic thesis = =93MACHINE_ARCH is enough=94 which you=92ve not shown a coherent example = of where it isn=92t. >>>> These are almost orthogonal dimensions in the sense that almost any >>>> combination is possible. (A combination that isn't possible is a >>>> 32-bit instruction set with LP64.) >>=20 >> All of these items are encoded in MACHINE_ARACH and have been for at >> least a decade. There=92s no new argument here. If they were = actually >> orthogonal, then that would be one thing. But they aren=92t. They are = all >> closely interrelated and we only support a vanishingly small number = of >> possible conventions. Combinatorically, it can be hundreds. = Practically, >> it is usually only a handful. >>=20 >>>> What you are asking for now is to combine two dimensions into one = and >>>> combination in this case means multiplication so if you have 3 >>>> instruction sets and 2 programming models, the combined dimension = needs >>>> 6 different values. You need to make the case for why you think = this >>>> is a good idea. >>=20 >> Because uanme has to be 6 different things so the right binaries are >> built. It is really that simple. >=20 > Uname is a per system (or per jail) setting. Whether you then want a > 32-bit or 64-bit address space is a separate per program or per = package > setting. If you want to install a package you need to know the system > you're on and then you need to decide whether you'll use it with a = large > amount of data that requires a 64-bit address space or whether a = 32-bit > address space is enough and you want the performance benefit it gives > (smaller pointers means lower memory and cpu cache use and 32-bit = pointer > arithmetic may be a bit faster). I fail to see how this is relevant to the discussion. If you want to = install a package, just install what uname -p returns. Unless the user = says =93do FRED instead=94 via a command line argument. Then validate = that against the list of supported ABIs (or just allow it if you are = forcing). It really should be just that simple. I=92m running on arm, = and uname returns armv6, then install the armv6 packages and not the = armv6hf or the armeb packages. No need to parse elf headers to get that. >>>> For the past 20 years we got away with this because >>>> on every installation of FreeBSD we only used one programming model = at >>>> a time. This is still the case for byte order of course. >>=20 >> This isn=92t true. For the past 15 years we=92ve supported two = programming >> models on amd64 at the same time. For longer than that we=92ve = supported >> linux emulation on i386. The project has known about these things for = a >> long long time, and has settled on MACHINE_ARCH to represent all = possible >> builds. We=92ve had mixed MIPS for about a decade, though the support = has >> varied in quality and execution. We learned that TARGET_BIG_ENDIAN = was >> bad, really bad, and we had to have a separate name for each ABI we >> supported with no external info apart from that name. We could have >> easily picked the convention you are proposing here, but we didn=92t. = We >> picked another one. >>=20 >> Also, the =93for the past 20 years=94 argument cuts both ways. Look = at >> NetBSD. There, they have the same convention we have here of having a >> separate MACHINE_ARCH for each ABI. They have been even more = successful >> at it that we have, and have avoided the pitfalls of = TARGET_BIG_ENDIAN >> much better than we have. pkgsrc ties nicely into that. so for 20 = years >> people have successfully used the current model, not just in FreeBSD, >> but also elsewhere. >=20 > I'm talking about cases where the first three fields listed above are > not sufficient to distinguish between ABIs. The cases you listed are > already handled by those three, like linux !=3D freebsd for the OS = field > and i386 !=3D amd64 for the instruction set. You=92ve yet to provide an actual example where this is the case. >>>> What I'm saying is to keep the option open for installations with >>>> multiple programming models, where most binaries could use ILP32 = and >>>> only the ones that actually need a 64-bit address space use LP64. >>>> You query the instruction set using uname and the programming = models >>>> using getconf. >>=20 >> What I=92m saying is I don=92t see any benefit at all to our users to >> having an additional, arbitrary sting they have to deal with. There=92s= >> actually quite a few other details that you need to know before you = can >> even call getconf. >=20 > getconf _POSIX_V6_ILP32_OFFBIG > getconf _POSIX_V6_LP64_OFF64 >=20 > It'll print "1" when supported, "undefined" otherwise. Currently only > one is supported per system (or per jail) so MACHINE_ARCH is = sufficient > to describe the ABI. When both are supported on one system (or one = jail) > (i.e. both commands print "1"), the system (or jail) MACHINE_ARCH is = not > sufficient to distinguish between them. You have to specify something > extra in this case (the fourth field) to indicate which of the two > packages you want. None of this is really relevant to the discussion. MACHINE_ARCH is = totally sufficient. You completely misunderstand. Let me explain. MACHINE_ARCH uniquely defines the ABI. Now, there are some kernels that support running multiple MACHINE_ARCHs. = amd64 is one. It supports amd64 and i386 (and soon i386t64). In the jail = you can ask the sysctl what are the supported things. And if you are on amd64 and want to install an i3866 package, a simple = command line argument will take care of that, and it can even be = validated with the supported abi sysctl. >>>> I suppose you could replace the "x86" in the pkg scheme with = i386/amd64, >>>> but then you'd still be talking about i386:32, amd64:32 and = amd64:64 >>>> instead of x86:32, x86:x32 and x86:64. =20 >>=20 >> I suppose you could replace these by =93i386=94, =93x32=94 (or = =93amd64x32=94) and >> =93amd64=94 respectively. >=20 > So you're on an amd64 or mips64 system (as indicated by uname) but you > want to use the 32-bit package if possible. How does your script know > about the magic "x32", "amd64x32" or "mipsn32" strings? Wouldn't it = be > easier if you could just use "`uname -p`:32=94? Oh give me a break. You know it because you know you are building for = mipsn32 because that=92s what you=92ve set MACHINE_ARCH or TARGET_ARCH = to, which might be uname -p if that=92s left unspecified. No, you can=92t = just say =91uname -p=92:32. Sorry. That=92s lame and generally won=92t = work. Have you actually tried to write a script that turns a = MACHNIE_ARCH into one of these funky pkg names? It is a maze of special = cases that has to be updated each time a new MACHINE_ARCH is added to = FreeBSD. It would be so much more convenient for script writers and = users of our system to have only one thing to specify rather than two, = the second of which is just arbitrarily different without adding any = value. > I do realise it doesn't quite work like this right now because pkg = uses > "x86" instead of "amd64" or "i386" for the third field and uses the > fourth field to distinguish between them. I don't know if this is = unique > to the x86 family or if this is also the case for the others. This = may > need to be reconsidered, but the idea of a fourth field is solid as = far > as I can see. What possible benefit is there? You keep dodging this question. So far = you=92ve shown no benefit what so ever, and lots of hassle. It is cool = because it is different, and it is more descriptive, but it doesn=92t = add any value. Warner --Apple-Mail=_A40E86D8-A260-43EC-A917-4D9DC2799B17 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQIcBAEBCgAGBQJTg8C5AAoJEGwc0Sh9sBEAgzEQAMyPNionFzicCadXzhv5A98b ixm0qIGs7ui8ZT4L3486929cUXVnt/fjlObp5GilJBqLIZ/8CGHCUU1ItXWWm1ws EAswGOjWhrMh0fvDW+0nKkoOCP6gbsAzTZEMrD1iVKA7Ae24mFZif6AGC5HvFgt6 5RvWS52O6iqts20f/5TFU5mpc3ZA/S6QLlbjG1aSRvOh0YpKkw9oNY0zRJWi1QsD Eqrk0ot1yaoNW+bNWOFksVGaB7584c2k+4dZWp0LgYO9Q8aG9wrkHYGSZLnwBUV2 ja0bqNFSoVO2VgW5/gdB00Hd2uCJmSBo5SgSzyD4bLyv3L0CCDWqztpUlT4HpvCb +TrvTnNsSi2cjBf2kGRUiNeFc3rBVhLbwfU+9iijVO5YYosj3cX7gSpI7+ptDXZa p/vud/mMrCkv9r+LWKydWvL4wzJA7OQN0YapJSAlWSytjTOFLkgBbKCbNGcp8cEz e2PD5X4G0pYojcftzGMSZcdYtDTKOmQdMKcNMeA2R3XSoai3meBogTX582LDC8M2 qsgmxwNCWFDe1DEideUaNX5lhLSWo/PuCjHIb1AnbHxcAml7d7JVs/CD9jRN2WJq lMBr3w3hInyKy3D9WM+nwfZvFVCGjEZhs6D4b92vgnae7y3W+MQihEvnnaFGq4DS 43M6HoVImyhy/rBKZHFZ =NHQx -----END PGP SIGNATURE----- --Apple-Mail=_A40E86D8-A260-43EC-A917-4D9DC2799B17--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?05D1A11D-5985-42EA-84AD-209A8B51D391>