Date: Wed, 22 Feb 2017 12:03:54 -0800 From: Bryan Drewery <bdrewery@FreeBSD.org> To: Tijl Coosemans <tijl@FreeBSD.org>, "Mikhail T." <mi+thun@aldan.algebra.com> Cc: Mathieu Arnold <mat@FreeBSD.org>, Jan Beich <jbeich@freebsd.org>, ports-committers@freebsd.org, svn-ports-all@freebsd.org, svn-ports-head@freebsd.org Subject: Re: Misuse of PORTREVISION (Re: svn commit: r434379 - head/multimedia/x265) Message-ID: <055af097-60ca-67ba-7e52-a5a45c7c5af1@FreeBSD.org> In-Reply-To: <20170221191909.1639f628@kalimero.tijl.coosemans.org> References: <20170218210541.82AA915F6@freefall.freebsd.org> <f23cecfc-d669-e62b-1916-1e16e66fb3eb@aldan.algebra.com> <a9be8749-360f-15fb-aa75-93a378371c51@FreeBSD.org> <d5e81c35-41fe-4f5c-4f45-e069d93ce911@aldan.algebra.com> <fee7b990-45db-1e52-22d8-fb1b21e9c030@FreeBSD.org> <17b5afa1-c0b4-a79e-aea1-d0ae918469e8@aldan.algebra.com> <20170221191909.1639f628@kalimero.tijl.coosemans.org>
next in thread | previous in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --kAfmpFxH971jvbXSn02oH7X1lTvg6AanA Content-Type: multipart/mixed; boundary="t21XkPhsHUMJ0rkVbUcgRtHXaB6R5c71q"; protected-headers="v1" From: Bryan Drewery <bdrewery@FreeBSD.org> To: Tijl Coosemans <tijl@FreeBSD.org>, "Mikhail T." <mi+thun@aldan.algebra.com> Cc: Mathieu Arnold <mat@FreeBSD.org>, Jan Beich <jbeich@freebsd.org>, ports-committers@freebsd.org, svn-ports-all@freebsd.org, svn-ports-head@freebsd.org Message-ID: <055af097-60ca-67ba-7e52-a5a45c7c5af1@FreeBSD.org> Subject: Re: Misuse of PORTREVISION (Re: svn commit: r434379 - head/multimedia/x265) References: <20170218210541.82AA915F6@freefall.freebsd.org> <f23cecfc-d669-e62b-1916-1e16e66fb3eb@aldan.algebra.com> <a9be8749-360f-15fb-aa75-93a378371c51@FreeBSD.org> <d5e81c35-41fe-4f5c-4f45-e069d93ce911@aldan.algebra.com> <fee7b990-45db-1e52-22d8-fb1b21e9c030@FreeBSD.org> <17b5afa1-c0b4-a79e-aea1-d0ae918469e8@aldan.algebra.com> <20170221191909.1639f628@kalimero.tijl.coosemans.org> In-Reply-To: <20170221191909.1639f628@kalimero.tijl.coosemans.org> --t21XkPhsHUMJ0rkVbUcgRtHXaB6R5c71q Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 2/21/2017 10:19 AM, Tijl Coosemans wrote: > Poudriere does not rebuild everything that needs to be rebuilt by the w= ay. > It just always rebuilds everything, full stop. I have a lot of replies to this thread and picked this one at random. 1. First of all, this is not true. Poudriere *does not* rebuild everything every time. It only does that with the -c flag. Poudriere *by default* will do an incremental build. It only rebuilds things if they have, comparing to the previous package, a missing package, a new PORTVERSION, a new PORTREVISION, new LIB or RUN dependencies, changed OPTIONS, a changed PKGNAME, or lastly if a dependency rebuilds due to any of the previous. This last piece, of always rebuilding if a dependency is rebuilt is simply *wrong*. It is a crutch that was added to ensure that packages were working because people often *forgot* to bump PORTREVISION properly.= Before I go on I should point out that Pkg uses similar logic. It will not reinstall a package that Poudriere aggressively rebuilt. Doing so *requires a PORTREVISION* bump. 2. Shared libraries are not the only cause to need to rebuild a package. Any number of the *build dependencies* may have been changed to produce a new output that modifies the resulting package somehow. Be it runtime or documentation or other metadata files. What if Doxygen is updated and now has a drastically new output format (and generates the same exact file output names/paths so plists are not impacted), should all ports needing it be rebuilt automatically (which Poudriere would do) and then have the new documentation format? Pick one. If you think yes, then Poudriere is right and Pkg is wrong. What if Doxygen just had a buildtime fix for mips, so now we're rebuilding x86 packages and the Doxygen output is the same. It clearly is wrong to rebuild the packages in this case for x86. So now we're contradicting our "yes" from earlier. In fact we do this *very often* on the package builders, where we are rebuilding a ton of things that have 0% difference from the previous except some timestamps. 3. Blaming portmaster/portupgrade is not correct. Both of those tools use 95% logic from the Ports framework. In the case of rebuilds for dependencies, they would only do it if the DEPENDS line specified a version requirement that changed and a PORTREVISION bump occurred. I forget if either implements shared library auto rebuild logic, but regardless, the *ports framework* does not. 4. A reminder that not doing a PORTREVISION chase as-needed will make Pkg users have broken packages until they force reinstall some of them, unless it happens to be a shared library change which it will recognize. 5. What Poudriere does is emphatically wrong as it breaks reproducible packages. Even if we don't support that now, the idea that SOMEPACKAGE-1.0 built last week and built today may have Doxygen 1.0 docs and Doxygen Web2.0 docs respectively, is very wrong. In my example this is just documentation, but it could very well be that a package built last week worked but built today does not work, yet they are the same exact version and portrevision. Add in the Pkg problem of not reinstalling them and you get some random users having problems and others not, while *both* have the updated dependencies because Pkg did upgrade those, but those dependencies had *build time* differences that broke packages. 6. We could use tools like abi-compliance-checker to automate more of this, but they still don't track other side-effect changes from build dependencies, like the Doxygen example. 7. Using "Pkg with Ports as a backend" would be interesting, but it still doesn't fix the problem that any kind of static INDEX cannot magically know if things need a rebuild without things like PORTREVISION. 8. Lastly, I have always blamed Poudriere's aggressiveness on ports committers not doing their job correctly with chasing PORTREVISIONS. There likely are many technical fixes to the problem, but currently none of the tools (portmaster/portupgrade/poudriere/pkg) do 100% of the right thing. I left out synth as I am not familiar with its behavior here, but it too cannot be magical and know if a package *really* needs a rebuild or not, and still Pkg may ignore that new one. I found this thread via a PR to make portupgrade support checking shlib bumps automatically (which may be fine), and asserting that the consensus in this thread was that PORTREVISION bumps were wrong. Well, no, they are required. A committer may not like them, but they are still required. I wrote about all of these problems in a Journal article a few years ago. Nothing has really changed since then. --=20 Regards, Bryan Drewery --t21XkPhsHUMJ0rkVbUcgRtHXaB6R5c71q-- --kAfmpFxH971jvbXSn02oH7X1lTvg6AanA Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQEcBAEBAgAGBQJYre6uAAoJEDXXcbtuRpfPez4H/ioBkvLvQW0BbqLEVCiTYoF5 Vof1JPyDblNmXepu+kkehEafx1ucUWJZ8j2KZ1fc+o+wO3zv7pyIAbRtsXIChKTo XR79KamWGr131wafa5/f0HUdWgUXV42CkNm+8YjLKcgNcVrfEQ6WS2IWTSg7E2aU OBJuvONzJPTT1yhBdGA3QcrUoo+qeYYp3BpCiUsmkI+c/oqEepWLdqORnSB78aNt 892u5U8wY/SKojwW3PBhoDbd13yGnByKi4pPo3JDbN0/xmd4cRLxnO9iKHUvZ+2T G7fGu07gr6FJfnQ0PXdvU42uAXahQAGGGVgmHA40ZiQUhod0YzJ9cCLGNIeG1iY= =D6DR -----END PGP SIGNATURE----- --kAfmpFxH971jvbXSn02oH7X1lTvg6AanA--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?055af097-60ca-67ba-7e52-a5a45c7c5af1>