From owner-freebsd-ports@freebsd.org Tue May 18 02:49:24 2021 Return-Path: Delivered-To: freebsd-ports@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id ED1DA63AFA4 for ; Tue, 18 May 2021 02:49:24 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic302-20.consmr.mail.gq1.yahoo.com (sonic302-20.consmr.mail.gq1.yahoo.com [98.137.68.146]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4FkgSg4VClz3sjS for ; Tue, 18 May 2021 02:49:23 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1621306161; bh=r7DnIm15/vkusCtNEvTNVXukiH28uTmYSh5RkymWJAp=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=bmKMX01v/wCBCbf1ssHGHw1E0tJR7uOgr/F1/FBNll8NRrS5l4gumq/6qesfhPPB94j6oabrlQVvPiT34UpkYdZaCm/Y5TtNXkssccofmQ26kED4LJT2hjZ8pjpKv5x65tlfRASLAWTUjREbOPYsQGI/qOm42IF5KAedggSd6EVP2hU0VeXgaFuTypJUaIKs+1LeikIGljmG7/T5yD9SOvCWhHZHPlypVHIwEwXFrm9vvFhkWZFKJC9SznsM8zl0kZRauHrnMlj0k9Fojt2IKf9vJNW/MMyDjx6qaMprgu8qo+6hc03lWnYFVLqEc7qgPH6gnqS919YRlytnnEWVjA== X-YMail-OSG: PvdB.k4VM1nxc4DFkoU3Na8jlgxuO9KwUjOFI2mhNkOU48u71BStUWRBoC8nyxv f4UfCiQ33H6pduN5TA3PqS8G8TA8PjcMz5ssoDNF2.04wtMsx4YsV02eplvdOTqMuQf44e49Vx2g l4Ip2ctiM_sV2fXwwQUJ71Oz77GnNZVR03DulV.hXla27B7VjI8sEnkl0nqzLCTAXuNgi5cDHtoV GoaICmXGBRnIT6MCr7QNFkkZm3g8kWvsZWisCuwO5W6giYvYz3xbgulG7ECXh95LvUdJenUYBgfj hLGIvX9dm9ae_JlRCvMFSaRu9W7jgvUuFS4A_M4C1g6uwkaC13WWCWOwJULKkACNrEGsER9MWKA7 OXl9XUdS302yU16cPN.ex4iyyMQHXnpWHKNwTLkzuYyuG7ckeI4drW4uqAJKZe5Q1ffot85DLceO HYkau8SvsagtIBAPR80wM9i7ZLYH2Gl7b9PiXXYmqCDE8Qnfv561F8VNM58EeUBuW1QAX5wqOgZ1 T4b.JAsrGafg8i.SoXPD1dOri430QmrIltdviEbQfad2I6NFlHwcCw1sC8nGkOO6h1HIsvisKSHM R0ovhe3Hm0jCbJ2pUQJO6j_Lz0Bq9ajEr8L9ao1Fzj0RGWaCxGYJx4d.q6dehK4C8bTl3ERtBF_U Ld4Mcrmqmb6z0PyNAE0lWk0mDrK7BQY6SeVEuVpKBvymJtUaJZ_u.6y9lU562MtMEO7Wabzurpuh q2w6rNv7U7yUkXHttsr4gRypQl35am2PJYIkzNVKf1Njjjzqdh3yJNP2oePLvBN9RoUdVTGVE5ps ukA75kgLTEoCyrMOraFKUbXVA6CyAMrpfj5WU45IW4sQcw8iAiV9WpOMXn1cxnrQMGL4b_lpstT1 j7ly06H_2ioWFR91QCpnRmJNXdmU2.NiTCGupnKSTXwAQ9a63TtED1f5vkUtTEVuJBluzrM3f6xZ cuZFWyY0lNUv4Ad5rRYILKWjatzGlsasy2oNzY3a2Yk3znFCHtRMDAxnWuPJ0IZkXLFCXTtrhPqH 4bJAMYSkhdU1_KNoXsUDu_tbxZc2PIEd.9aIMSlMWKDQYgSIhtsn3cZXdSWas0Q6wINU04tBbV7K 0quc5EkTiObim5G9n8bk_dg1fwsizSumhReJJnGbG9jw1obCR1t8YAeOdWMGya46Phc5mcns4Uoc F0WP8ovAJqr74nM2CQPcLYfX1bWpE24cS9GSdwH_an7aXnn9weHPi_oQGz1SwpxvLd9OQT9RfM4D 7iK1Myjq14QXoLU1QIYFS2LuOZPBQWKMXScN.zNfo3gWYrPf7VObMDTDmHvydrKPrHW_zBg2b6ry vwyo4Tss6hfLOjHwAXgs2Sq9Np0LNXFgHPaBQhTFUfkBoQ1JYeGmprteuH9mNwyW.W8Bn9H5Fga4 Vo15G14b3x7D2pZlBjOh1wJfTqGFzXpqXziQQ7.9Ibw2aj3Lj8LMh7YMzHmxwbrncNRTmE7mWDpI Cokl4mtQaW4lJ_OD72a_hq0Mw3.XqkDODyRM6RK3o40m3PiNMaf9kXfys07xvz47lBuV1Y9x.ufq aMjZaYXK7ilsjFVcOBjb6zM8Vsf679skcfVZ7VNDHEu1AUPW9RoWS4NZx5dF6aged9bVPdw1WHhD agXl3nd05COksKEy1vR_ke6hpZdDOdk7AKzhkPQS4Ja1xDeM5UpfS_0hML_s18EwK.ATXXSKVU05 1Y7foeMfH57zJC5dSBPsUxm3TH6WGDMYmxHMIEqmT46Wdrwtg9h8n9AjkmJBE3JRAoERA5E1Lx8n OXXXk0yHx5BwAXM0fq7OOghRWl1CPzb4lOsyV6seAw.PIIF4_mIqTZwDKdirFMSHHaWLmSQKNlLR M6cvhUh6Wb9cenVbf.Z0OpbzmnhKD8zvAAxWqttsy8cf5oc8KM_ATDfLuF.HV99hwWH12ZmWQYTQ yFTLQ5Iyl2fwR02cPdhSNsKsY9AYp_Zfcv5moXeqnLJW4fYoESDsK94ru48SzhxsBu4QK7s7qvcx fF4Zmhk1CHTky.WxxS99k92DfuvII9VjJ2NJl_lNa9Mfnbsp9nV_gtda3vwfTTbG45QoMBIkM6T. QT44uW6hToYNQerd5DgWs3AlI3xKAcrGeqTYRIINn30dMjjKsvCptMbRuwY5AhCQ1nr8wHzvO5jA tvys075euTsGL3ffsWWrEtfEAf3znLr8QiaC922iCuq7CniAEVaHKrSG5ZMMLMWKQ_LgrRXdnQzX m03t9j1jWFM3kc5y7hXb3Kzm0Z97ZvVTHtuMiisnA8KCQEIRRYmwuO8VImzj.VN28uEGnpOYM4ip 9tegVes2vgqI_RFT0TKdFKFanxg1X2.T7TEfv.v8JFXqM75n0R.ytPRJtLu0acHnpJ7EobraOKbe JiHNiHfa7eASzVFDbuxDA4zlwZD1a9mxmRpVZTcKf7A5GJhD.8eeSctqlufotUmt8HrzrzVVIto0 - X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic302.consmr.mail.gq1.yahoo.com with HTTP; Tue, 18 May 2021 02:49:21 +0000 Received: by kubenode559.mail-prod1.omega.gq1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID bf974e13c12c87ce9fa1ccdd1f923450; Tue, 18 May 2021 02:49:17 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.80.0.2.43\)) Subject: Re: Python 37/38 conflict, was Re: Trubles compiling lxqt on RPi4 Message-Id: Date: Mon, 17 May 2021 19:49:16 -0700 To: FreeBSD ports X-Mailer: Apple Mail (2.3654.80.0.2.43) References: X-Rspamd-Queue-Id: 4FkgSg4VClz3sjS X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.33 / 15.00]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_SHORT(-0.99)[-0.990]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RBL_DBL_DONT_QUERY_IPS(0.00)[98.137.68.146:from]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-0.84)[-0.842]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[98.137.68.146:from:127.0.2.255]; RCVD_IN_DNSWL_NONE(0.00)[98.137.68.146:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.68.146:from]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-ports] X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 May 2021 02:49:25 -0000 bob prohaska fbsd at www.zefox.net wrote on Mon May 17 23:46:38 UTC 2021 : > On Mon, May 17, 2021 at 12:28:24PM -0700, Mark Millard via = freebsd-ports wrote: > > bob prohaska fbsd at www.zefox.net wrote on > > Mon May 17 15:55:21 UTC 2021 : > >=20 > > > The existing conflict between versions of python strikes me as = more of a=20 > > > planning problem than a software bug. It may be naive, but I don't = see > > > why python37 and python38 can't use distinct names for files = placed in > > > system directories. > >=20 > > You seem to be under the impression that absent having > > any common file paths, things would just work. This > > seems unlikely. A simplified presentation of why > > follows. > >=20 >=20 > I'm under the impression that absence of common paths would help > reduce conflicts. True: possibly necessary, even if not sufficient. > In the case at hand it might be sufficient.=20 I do not know: I do not have a very complete understanding of the full status of your environment after the problem. (Nor of just what specifically lead to the problem.) I do know that I do not deal with the issue in my context --but that is because I use procedures that avoid the general type of issue (tolerating any other tradeoffs involved). (I have worked via portmaster and just plain make in the past before using poudriere as well.) > > Imagine two programs: > >=20 > > p37 that is set up for python37 > > p38 that is set up for python38 > >=20 > > imagine that both use a library plib that is > > not internal to each but external to both. > >=20 > > So should plib be built/present for python37? > > python38? Both? > >=20 > > If both: it suggests building a huge set of python > > software multiple times, once per supported version > > in the range of to-be-supported python versions. (I > > do assume python versions make for some degree of > > incompatibility distinctions. It might be only only > > some version changes have that property. But such > > would not change the basic point.) > >=20 >=20 > It suggests that p37 (installed first) would install > its preferred version of plib. When p38 is installed, it > would test for a compatible version of plib So now p38 is required to classify all prior combinations of versions of external libraries it might use (such plib), and to put in tests for handling all the combinations. And what if p38 is installed first and p37 second? p37 must do similarly --but has no way to well classify combinations involving library versions that did not exist when it was released. One alternative in the industry is having each package fully self contained (up to the interfacong with the OS or whatever the kind of context is). The package-build builds everything required and bundles what is needed at run time all together so the package does not depend on any other packages: its installer and installation results are self sufficient (up to the "OS"). This makes other tradeoffs. There are examples that are similar in ports, some tied to bootstrapping issues. For example, building ports-mgmt/pkg builds far more internally because pkg can not depend on other ports/packages that would need pkg to already be operational to get things setup. https://github.com/freebsd/pkg/tree/master/external/ lists: blake2/ libelf/ libfetch/ liblua/ libmachista/ libucl/ linenoise/ lua/ msgpuck/ picosat/ sqlite/ uthash/ yxml/ As I understand lang/rust contains and builds its own (subset of?) some llvm version instead of depending on one of llvm10/llvm11/llvm12. Its build time and resource use may be illustrations of some of the tradeoffs in this style: its build of an llvm does no good at saving time for any other port build that involves the same vintage of llvm. > and add a new=20 > one, with a different name, if the prior version isn't=20 > satisfactory. Libraries already seem to have a variety of > suffixes on their names, so it appears something of the sort > is already going on. Am I completely missing the point? See notes above. > > I know, for example, python39 invalidated code in > > something I've built in a non-FreeBSD context. The > > software had to be modified to be compatible with > > both older python's and python39 (if they wanted > > to support the older versions as well going > > forward). (It was not a context of wanting to take > > advantage of new things in the newer python. But > > that kind of context is not universal.) >=20 > Not sure I see a fundamental problem here. Python > 38 remaining useful/necessary after introduction of > 39 doesn't seem so bad. It seems the norm. How far back are the pre-supplied older versions supposed to go? On operating system A? B? C? (Likely differing choices will be made.) How many old versions continue to get bug fixes and security updates and the like? How much less effort is put into creating new versions (supposed improvements) as a consequence? To again use p37 and p38: how do they deal with the varying range of old versions on A vs. B vs. C? Some of this is or can be done, but the extent tends to be rather limited because the information requirements and effort to manage much of a variety is huge. And it does not indefinitely delay having to deal with updating to get things working --unless the history covered always goes back to the beginning. Otherwise, it just changes some about when the issue is faced. > > Most ports having a separate upstream to deal with > > and having a huge number of ports makes for > > port-developer and upstream-developer coordination > > based solutions having great difficulties overall. > > >=20 > Indeed, and they're getting worse over time. > =20 > > No technical-content has been presented to make these > > sorts of problems disappear. With the problems being > > present, it is a matter of working in a way that > > avoids running into the problems or with dealing with > > fixing things when the problems occur.=20 >=20 > I've wondered from time to time if it's possible, even > only in principle, to make the entire ports tree build > in one invocation. It is called "poudriere bulk -a . . ." --as used by some of the FreeBSD build server port builds if I understand right. Over 30,000 ports when no prior incomplete run's partial results are around. I once did a "poudriere bulk -a" when the ThreadRipper 1950X was new and I was testing it. (At the time FreeBSD had been having problems with the new type of AMD processors and I was looking to see what all would fail. Then I continued, only dealing with the failures.) The ThreadRipper 1950X is the only machine I've had access to that I'd ever try such a build with. Otherwise my time preferences would not allow waiting sufficiently long. (It is possible no other machine had sufficient resources to complete the process in a time frame I would tolerate, such as storage-space/RAM+swapspace limitations.) > At the moment, the answer seems to be=20 > "no". But is it "no" on first principles? The answer is yes: "poudriere bulk -a" is an example way of doing so. I'm not sure if anyone has tried a full build of all the ports tree without using poudriere to do it. At this point, poudriere, or something somewhat analogous to it, may be the only kind of port-build context that has a chance of completing the process reasonably. In that sense, poudriere is the most extensively tested of the ways of building the ports: fairly regular use to build the complete ports tree in one go. > > I've done both > > basic styles over the years and recognize tradeoffs. > > I'm happy to help someone explore one of the ways > > in which poudriere can be an alternative. It is not > > for me to declare how well it would end up fitting > > their goals, context, preferences, and so on vs. > > other alternatives overall. > > > =20 > I'll continue to explore poudriere. Your efforts are=20 > much appreciated, but also rather daunting. Any particular questions still pending? Non-obvious steps for your context that you wonder about? Have you been able to build some port via poudriere yet? > The learning > curve is steep I only managed my first poudriere experiments when someone did something similar for me, reporting what they changed from the basic install (and so implicitly: what they left alone). It allowed me to fairly well isolate what I needed to deal with explicitly to get started, given my goals and odd context (mostly tied to PowerMac support at the time). > and resource requirements are high. An example of: Using poudriere has tradeoffs involved. I do not know what using spinning rust would be like for how long things take. My contexts generally have had a USB3 SSD as the storage media --or better in some cases. I've never tried this with spinning rust or microsd cards or networked file systems. (The PowerMacs had SATA SSDs, but not modern SATA, so possibly not "better". But still not spinning rust nor microsd cards.) Ignoring old PowerMacs, possibly the slowest context was armv7 (Cortex-A7) with 1 or 2 GiBytes of RAM, 4 cores, and use of the USB3-capable SSD type of storage device on a USB2 bus instead of USB3 (in some cases with a powered hub needing to be involved). I tend to build for armv7 on aarch64 using an armv7 installworld that poudriere is configured to use for one of its jails. The USB3-capable SSDs are 240 GiByte ones. Only on old PowerMacs have I ever used anything with less capacity (120 GiByte) for this kind of activity. I do have access to some examples of higher capacity and faster media than SATA SSDs. After the 2-sockets/2-cores-each PowerMacs both had effectively died (over heating), I greatly restricted what I build (now on the 2 core PowerMacs). I still build for 32-bit powerpc on the powerpc64, much like the armv7 on aarch64 case. (Other issues have meant that the PowerMacs are not in regular use, however.) > If=20 > there's some larger point I'm missing please give a hint. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)