Date: Mon, 17 May 2021 19:49:16 -0700 From: Mark Millard <marklmi@yahoo.com> To: FreeBSD ports <freebsd-ports@freebsd.org> Subject: Re: Python 37/38 conflict, was Re: Trubles compiling lxqt on RPi4 Message-ID: <CE0B7882-C74A-4C74-A521-17A56E145B82@yahoo.com> References: <CE0B7882-C74A-4C74-A521-17A56E145B82.ref@yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
bob prohaska fbsd at www.zefox.net wrote on Mon May 17 23:46:38 UTC 2021 : > On Mon, May 17, 2021 at 12:28:24PM -0700, Mark Millard via = freebsd-ports wrote: > > bob prohaska fbsd at www.zefox.net wrote on > > Mon May 17 15:55:21 UTC 2021 : > >=20 > > > The existing conflict between versions of python strikes me as = more of a=20 > > > planning problem than a software bug. It may be naive, but I don't = see > > > why python37 and python38 can't use distinct names for files = placed in > > > system directories. > >=20 > > You seem to be under the impression that absent having > > any common file paths, things would just work. This > > seems unlikely. A simplified presentation of why > > follows. > >=20 >=20 > I'm under the impression that absence of common paths would help > reduce conflicts. True: possibly necessary, even if not sufficient. > In the case at hand it might be sufficient.=20 I do not know: I do not have a very complete understanding of the full status of your environment after the problem. (Nor of just what specifically lead to the problem.) I do know that I do not deal with the issue in my context --but that is because I use procedures that avoid the general type of issue (tolerating any other tradeoffs involved). (I have worked via portmaster and just plain make in the past before using poudriere as well.) > > Imagine two programs: > >=20 > > p37 that is set up for python37 > > p38 that is set up for python38 > >=20 > > imagine that both use a library plib that is > > not internal to each but external to both. > >=20 > > So should plib be built/present for python37? > > python38? Both? > >=20 > > If both: it suggests building a huge set of python > > software multiple times, once per supported version > > in the range of to-be-supported python versions. (I > > do assume python versions make for some degree of > > incompatibility distinctions. It might be only only > > some version changes have that property. But such > > would not change the basic point.) > >=20 >=20 > It suggests that p37 (installed first) would install > its preferred version of plib. When p38 is installed, it > would test for a compatible version of plib So now p38 is required to classify all prior combinations of versions of external libraries it might use (such plib), and to put in tests for handling all the combinations. And what if p38 is installed first and p37 second? p37 must do similarly --but has no way to well classify combinations involving library versions that did not exist when it was released. One alternative in the industry is having each package fully self contained (up to the interfacong with the OS or whatever the kind of context is). The package-build builds everything required and bundles what is needed at run time all together so the package does not depend on any other packages: its installer and installation results are self sufficient (up to the "OS"). This makes other tradeoffs. There are examples that are similar in ports, some tied to bootstrapping issues. For example, building ports-mgmt/pkg builds far more internally because pkg can not depend on other ports/packages that would need pkg to already be operational to get things setup. https://github.com/freebsd/pkg/tree/master/external/ lists: blake2/ libelf/ libfetch/ liblua/ libmachista/ libucl/ linenoise/ lua/ msgpuck/ picosat/ sqlite/ uthash/ yxml/ As I understand lang/rust contains and builds its own (subset of?) some llvm version instead of depending on one of llvm10/llvm11/llvm12. Its build time and resource use may be illustrations of some of the tradeoffs in this style: its build of an llvm does no good at saving time for any other port build that involves the same vintage of llvm. > and add a new=20 > one, with a different name, if the prior version isn't=20 > satisfactory. Libraries already seem to have a variety of > suffixes on their names, so it appears something of the sort > is already going on. Am I completely missing the point? See notes above. > > I know, for example, python39 invalidated code in > > something I've built in a non-FreeBSD context. The > > software had to be modified to be compatible with > > both older python's and python39 (if they wanted > > to support the older versions as well going > > forward). (It was not a context of wanting to take > > advantage of new things in the newer python. But > > that kind of context is not universal.) >=20 > Not sure I see a fundamental problem here. Python > 38 remaining useful/necessary after introduction of > 39 doesn't seem so bad. It seems the norm. How far back are the pre-supplied older versions supposed to go? On operating system A? B? C? (Likely differing choices will be made.) How many old versions continue to get bug fixes and security updates and the like? How much less effort is put into creating new versions (supposed improvements) as a consequence? To again use p37 and p38: how do they deal with the varying range of old versions on A vs. B vs. C? Some of this is or can be done, but the extent tends to be rather limited because the information requirements and effort to manage much of a variety is huge. And it does not indefinitely delay having to deal with updating to get things working --unless the history covered always goes back to the beginning. Otherwise, it just changes some about when the issue is faced. > > Most ports having a separate upstream to deal with > > and having a huge number of ports makes for > > port-developer and upstream-developer coordination > > based solutions having great difficulties overall. > > >=20 > Indeed, and they're getting worse over time. > =20 > > No technical-content has been presented to make these > > sorts of problems disappear. With the problems being > > present, it is a matter of working in a way that > > avoids running into the problems or with dealing with > > fixing things when the problems occur.=20 >=20 > I've wondered from time to time if it's possible, even > only in principle, to make the entire ports tree build > in one invocation. It is called "poudriere bulk -a . . ." --as used by some of the FreeBSD build server port builds if I understand right. Over 30,000 ports when no prior incomplete run's partial results are around. I once did a "poudriere bulk -a" when the ThreadRipper 1950X was new and I was testing it. (At the time FreeBSD had been having problems with the new type of AMD processors and I was looking to see what all would fail. Then I continued, only dealing with the failures.) The ThreadRipper 1950X is the only machine I've had access to that I'd ever try such a build with. Otherwise my time preferences would not allow waiting sufficiently long. (It is possible no other machine had sufficient resources to complete the process in a time frame I would tolerate, such as storage-space/RAM+swapspace limitations.) > At the moment, the answer seems to be=20 > "no". But is it "no" on first principles? The answer is yes: "poudriere bulk -a" is an example way of doing so. I'm not sure if anyone has tried a full build of all the ports tree without using poudriere to do it. At this point, poudriere, or something somewhat analogous to it, may be the only kind of port-build context that has a chance of completing the process reasonably. In that sense, poudriere is the most extensively tested of the ways of building the ports: fairly regular use to build the complete ports tree in one go. > > I've done both > > basic styles over the years and recognize tradeoffs. > > I'm happy to help someone explore one of the ways > > in which poudriere can be an alternative. It is not > > for me to declare how well it would end up fitting > > their goals, context, preferences, and so on vs. > > other alternatives overall. > > > =20 > I'll continue to explore poudriere. Your efforts are=20 > much appreciated, but also rather daunting. Any particular questions still pending? Non-obvious steps for your context that you wonder about? Have you been able to build some port via poudriere yet? > The learning > curve is steep I only managed my first poudriere experiments when someone did something similar for me, reporting what they changed from the basic install (and so implicitly: what they left alone). It allowed me to fairly well isolate what I needed to deal with explicitly to get started, given my goals and odd context (mostly tied to PowerMac support at the time). > and resource requirements are high. An example of: Using poudriere has tradeoffs involved. I do not know what using spinning rust would be like for how long things take. My contexts generally have had a USB3 SSD as the storage media --or better in some cases. I've never tried this with spinning rust or microsd cards or networked file systems. (The PowerMacs had SATA SSDs, but not modern SATA, so possibly not "better". But still not spinning rust nor microsd cards.) Ignoring old PowerMacs, possibly the slowest context was armv7 (Cortex-A7) with 1 or 2 GiBytes of RAM, 4 cores, and use of the USB3-capable SSD type of storage device on a USB2 bus instead of USB3 (in some cases with a powered hub needing to be involved). I tend to build for armv7 on aarch64 using an armv7 installworld that poudriere is configured to use for one of its jails. The USB3-capable SSDs are 240 GiByte ones. Only on old PowerMacs have I ever used anything with less capacity (120 GiByte) for this kind of activity. I do have access to some examples of higher capacity and faster media than SATA SSDs. After the 2-sockets/2-cores-each PowerMacs both had effectively died (over heating), I greatly restricted what I build (now on the 2 core PowerMacs). I still build for 32-bit powerpc on the powerpc64, much like the armv7 on aarch64 case. (Other issues have meant that the PowerMacs are not in regular use, however.) > If=20 > there's some larger point I'm missing please give a hint. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CE0B7882-C74A-4C74-A521-17A56E145B82>