Date: Mon, 26 Jul 2021 09:18:44 +0200 From: Baptiste Daroussin <bapt@FreeBSD.org> To: ports-committers@FreeBSD.org, dev-commits-ports-all@FreeBSD.org, dev-commits-ports-main@FreeBSD.org Subject: Re: git: e497a16a2869 - main - ports-mgmt/pkg*: Release 1.17.0 Message-ID: <20210726071844.tjo3d2t3tqtuj6md@aniel.nours.eu> In-Reply-To: <CAK7dMtAwh=_MMmN%2B%2Bfp_Rj62mkm9ZVo-gDT4SxoCUKD-6pNV7g@mail.gmail.com> References: <202107220806.16M862GC052436@gitrepo.freebsd.org> <f5d0be1a-7f7c-7dd5-61d0-f88328847722@FreeBSD.org> <CAK7dMtAwh=_MMmN%2B%2Bfp_Rj62mkm9ZVo-gDT4SxoCUKD-6pNV7g@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
I just figured I only sent this email fo Kevin instead of sending it to everyone... see below. 25 juil. 2021 06:49:10 Kevin Bowling <kevin.bowling@kev009.com>: > On Sat, Jul 24, 2021 at 1:36 PM Bryan Drewery <bdrewery@freebsd.org> wrote: >> >> On 7/22/2021 1:06 AM, Baptiste Daroussin wrote: >>> The branch main has been updated by bapt: >>> >>> URL: https://cgit.FreeBSD.org/ports/commit/?id=e497a16a286972bfcab908209b11ee6a13d99dc9 >>> >>> commit e497a16a286972bfcab908209b11ee6a13d99dc9 >>> Author: Baptiste Daroussin <bapt@FreeBSD.org> >>> AuthorDate: 2021-07-22 07:54:38 +0000 >>> Commit: Baptiste Daroussin <bapt@FreeBSD.org> >>> CommitDate: 2021-07-22 08:05:59 +0000 >>> >>> ports-mgmt/pkg*: Release 1.17.0 >>> >>> TL;DR: new uniq package file extension: '.pkg' >>> >>> Full changelog for pkg: >>> - add a new "snap" prefix, to deal with pkg base snapshots >>> - pkg repo now accepts packages with different compression formats >>> - pkg now have a single extension: ".pkg" and a backward compatible >>> symlink is created when the package is created to help transitioning >>> - Default compression level for zstd is now set to 19 >>> - the default compression level is now a configuration option >>> - plenty of portability fixes >>> - fix plenty of typos >>> - expose the name of the package to lua and shell scripts >>> - plist: >>> remove internal support for @*exec (this is now in the ports tree) >>> remove support for @ignore >>> remove support for @dirrm/@dirrmtry >>> remove support for @pkgdep >>> remove stub support for @stopdaemon >>> remove stub support for @display >>> remove stub support for @mtree >>> remove stub support for @conflict >>> - VUXML: fetch .xz compressed version of the file by default >>> - triggers are deferred to later command (firstboot is planned) if run >>> with pkg -r >>> - pkg triggers commands has been added and can execute the deferred >>> triggers for example at firstboot >>> >>> Changes in the framework: >>> - Add a backward compatibility layer, so people are not forced to move >>> to pkg 1.17 yet (new quarter we will enforce the switch for triggers >>> anyway, but it gives time to people to switch) >>> - Introduce a new PKG_COMPRESSION_FORMAT for users to specify the >>> compression format they are willing to use when creating packages: >>> txz, tar, tgz, tbz, tzst are the valid ones (note that tzst is only >>> usable on FreeBSD 13 and 14) >>> - Add a backward compatibility for people who specified the compression >>> format already via PKG_SUFX and issue a warning to tell them about the >>> deprecation of PKG_SUFX >>> --- >>> Mk/bsd.port.mk | 58 +++++++++++++++++++++++-------------------- >>> ports-mgmt/pkg-devel/Makefile | 4 +-- >>> ports-mgmt/pkg-devel/distinfo | 6 ++--- >>> ports-mgmt/pkg/Makefile | 2 +- >>> ports-mgmt/pkg/distinfo | 6 ++--- >>> 5 files changed, 40 insertions(+), 36 deletions(-) >>> >>> diff --git a/Mk/bsd.port.mk b/Mk/bsd.port.mk >>> index 228644a1845f..5eba5d89da95 100644 >>> --- a/Mk/bsd.port.mk >>> +++ b/Mk/bsd.port.mk >>> @@ -1211,6 +1211,14 @@ _OSVERSION_MAJOR= ${OSVERSION:C/([0-9]?[0-9])([0-9][0-9])[0-9]{3}/\1/} >>> .if !defined(_PKG_VERSION) >>> _PKG_VERSION!= ${PKG_BIN} -v >>> .endif >>> +# XXX hack for smooth transition towards pkg 1.17 >>> +_PKG_BEFORE_PKGEXT!= ${PKG_BIN} version -t ${_PKG_VERSION:C/-.*//g} 1.17.0 >>> +.if ${_PKG_BEFORE_PKGEXT} == "<" >>> +_PKG_TRANSITIONING_TO_NEW_EXT= yes >>> +_EXPORTED_VARS+= _PKG_TRANSITIONING_TO_NEW_EXT >>> +.warning "It is strongly recommanded to upgrade to a newer vertion of pkg first" >>> +.endif >>> +# XXX End of hack >>> _PKG_STATUS!= ${PKG_BIN} version -t ${_PKG_VERSION:C/-.*//g} ${MINIMAL_PKG_VERSION} >>> .if ${_PKG_STATUS} == "<" >>> IGNORE= pkg(8) must be version ${MINIMAL_PKG_VERSION} or greater, but you have ${_PKG_VERSION}. You must upgrade the ${PKG_ORIGIN} port first >>> @@ -2227,23 +2235,29 @@ _PKGMESSAGES+= ${PKGMESSAGE} >>> >>> TMPPLIST?= ${WRKDIR}/.PLIST.mktmp >>> >>> -.if ${WITH_PKG} == devel >>> -PKG_SUFX?= .pkg >>> +# backward compatibility for users >>> +.if defined(_PKG_TRANSITIONING_TO_NEW_EXT) >>> .if defined(PKG_NOCOMPRESS) >>> -PKG_OLDSUFX?= .tar >>> +PKG_SUFX?= .tar >>> .else >>> -#.if ${OSVERSION} > 1400000 >>> -#PKG_OLDSUFX?= .tzst >>> -#.else >>> -PKG_OLDSUFX?= .txz >>> -#.endif >>> +PKG_SUFX?= .txz >>> .endif >>> +PKG_COMPRESSION_FORMAT?= ${PKG_SUFX:S/.//} >>> .else >>> +.if defined(PKG_SUFX) >>> +PKG_COMPRESSION_FORMAT?= ${PKG_SUFX:S/.//} >>> +.warning PKG_SUFX is defined, if should be replace with PKG_COMPRESSION_FORMAT >>> +.endif >>> +PKG_SUFX= .pkg >>> +.endif >> >> I love the idea here but can this be reverted for now please? Poudriere >> is not prepared for a default PKG_EXT change. >> >> 1. Now that it finds a new PKG_EXT of .pkg, it cannot find the existing >> bootstrap pkg.txz because it looks for Latest/pkg.pkg -> >> ../All/pkg-*.pkg even though the *existing* one is .txz. I'm not sure >> how this symlink is added before pkg is even built but it's not done in >> a compatible way. > > I haven't seen an issue on my end with the symlinks for pkg which > seems to bootstrap ok and > I don't fully understand the problem statement. I'd like to see what > bapt has to say for this one. The latest/pkg.pkg is done 100% like it was done before with .txz the new pkg.txz reuse kind of the same mechanism but point at the .pkg file, why is it a problem? It is used for 2 things: - bootstrapping pkg on 11.4 and 12.2 (others knows about .pkg) - get some abi information with poudriere. Both are satisfied by the symlink so what is the isuue here? > >> 2. The new PKG_EXT makes Poudriere now find no existing packages so >> everything is rebuilt. Not a deal breaker in itself but a problem for >> future builds... > > See below, maybe this is an acceptable transition hammer whenever > someone changes it. Yes that was expented, if one want to avoid this he/she can rename all the previous built packages imho this acceptable > >> 3. The rebuild won't remove all the .txz files. I assume once `pkg repo` >> runs it will find both .txz and .pkg files and cause duplicate version >> problems in `pkg repo` > > This one is poudriere specific, do you think an acceptable solution > would be to detect changes to the variable from the config versus the > repo by storing a canary in the repo and run the pkgclean module if it > changes? I can try and do that tomorrow with your design approval or > guidance. > > >> I see something creates a .txz -> .pkg symlink. What deles that symlink >> if Poudriere only deletes/rebuilds the .pkg? > > Unsure. > Right here we have an issue and we can have a simple fix.(proposal below) >> *** >> "all those .txz entries are actually _duplicates_ in packagesite.yaml" >> seen on IRC just now by someone. >> I can confirm that looking in a packagesite.yaml I just generated. Both >> the older .txz version of a port and a new revision-bumped one appear in >> the yaml. >> *** >> Yes for the one that are not symlinks (proposal below) >> 4. I've seen reports of packagesite.pkg which is concerning because >> Poudriere manages packagesite.txz with its own symlinks for >> hardlink-copied repositories. It does not use PKG_EXT there as they >> weren't packages, but I suspect we need special handling even with >> PKG_EXT added there. >> That I missed it, but does not sound too complicated to extend (I haven t looked) >> -- >> Regards, >> Bryan Drewery >> Before my proposal, let me explain my own frustration, the bootstrap have been modified in march to support .pkg so at the very least it is known since march where we were going! The exp-run have been done month ago! The bsd.port.mk has been patched with .pkg for WITH_PKG=devel While again I can be better at communication. An explicit mail to our mailing lists would have been nice. I consider everyone working on the ports tree to at least have a look at the changes in bsd.port.mk and read the various plg-devel. https://svnweb.freebsd.org/ports?view=revision&revision=568662 Even more the people working on tools that directly depends on pkg. Now my proposal. I may have overlooked the backward compatibility and we should maybe only do it for packagesite itself and not every single packages this way you don t have to clean dead symlinks in poudriere. We should also restrict pkg repo to only take in account .pkg files by default and add an option to accept previous valid extensions (off by default). With those 2 options everything would be seamless for end users The last part cleaning up all the old .txz files will become a one shot operation easily doable in poudriere. What do you think about this proposals? Bapt
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20210726071844.tjo3d2t3tqtuj6md>