Skip site navigation (1)Skip section navigation (2)
Date:      Sun, 25 Jul 2021 10:06:23 -0700
From:      Bryan Drewery <bdrewery@FreeBSD.org>
To:        Kevin Bowling <kevin.bowling@kev009.com>
Cc:        Baptiste Daroussin <bapt@freebsd.org>, ports-committers <ports-committers@freebsd.org>, dev-commits-ports-all@freebsd.org, dev-commits-ports-main@freebsd.org
Subject:   Re: git: e497a16a2869 - main - ports-mgmt/pkg*: Release 1.17.0
Message-ID:  <2a176f0b-3a52-346f-dd25-7cedf6ca3f38@FreeBSD.org>
In-Reply-To: <CAK7dMtAwh=_MMmN%2B%2Bfp_Rj62mkm9ZVo-gDT4SxoCUKD-6pNV7g@mail.gmail.com>
References:  <202107220806.16M862GC052436@gitrepo.freebsd.org> <f5d0be1a-7f7c-7dd5-61d0-f88328847722@FreeBSD.org> <CAK7dMtAwh=_MMmN%2B%2Bfp_Rj62mkm9ZVo-gDT4SxoCUKD-6pNV7g@mail.gmail.com>

next in thread | previous in thread | raw e-mail | index | archive | help
On 7/24/21 9:48 PM, Kevin Bowling wrote:
> On Sat, Jul 24, 2021 at 1:36 PM Bryan Drewery <bdrewery@freebsd.org> wrote:
>>
>> On 7/22/2021 1:06 AM, Baptiste Daroussin wrote:
>>> The branch main has been updated by bapt:
>>>
>>> URL: https://cgit.FreeBSD.org/ports/commit/?id=e497a16a286972bfcab908209b11ee6a13d99dc9
>>>
>>> commit e497a16a286972bfcab908209b11ee6a13d99dc9
>>> Author:     Baptiste Daroussin <bapt@FreeBSD.org>
>>> AuthorDate: 2021-07-22 07:54:38 +0000
>>> Commit:     Baptiste Daroussin <bapt@FreeBSD.org>
>>> CommitDate: 2021-07-22 08:05:59 +0000
>>>
>>>     ports-mgmt/pkg*: Release 1.17.0
>>>
>>>     TL;DR: new uniq package file extension: '.pkg'
>>>
>>>     Full changelog for pkg:
>>>     - add a new "snap" prefix, to deal with pkg base snapshots
>>>     - pkg repo now accepts packages with different compression formats
>>>     - pkg now have a single extension: ".pkg" and a backward compatible
>>>       symlink is created when the package is created to help transitioning
>>>     - Default compression level for zstd is now set to 19
>>>     - the default compression level is now a configuration option
>>>     - plenty of portability fixes
>>>     - fix plenty of typos
>>>     - expose the name of the package to lua and shell scripts
>>>     - plist:
>>>       remove internal support for @*exec (this is now in the ports tree)
>>>       remove support for @ignore
>>>       remove support for @dirrm/@dirrmtry
>>>       remove support for @pkgdep
>>>       remove stub support for @stopdaemon
>>>       remove stub support for @display
>>>       remove stub support for @mtree
>>>       remove stub support for @conflict
>>>     - VUXML: fetch .xz compressed version of the file by default
>>>     - triggers are deferred to later command (firstboot is planned) if run
>>>       with pkg -r
>>>     - pkg triggers commands has been added and can execute the deferred
>>>       triggers for example at firstboot
>>>
>>>     Changes in the framework:
>>>     - Add a backward compatibility layer, so people are not forced to move
>>>       to pkg 1.17 yet (new quarter we will enforce the switch for triggers
>>>       anyway, but it gives time to people to switch)
>>>     - Introduce a new PKG_COMPRESSION_FORMAT for users to specify the
>>>       compression format they are willing to use when creating packages:
>>>       txz, tar, tgz, tbz, tzst are the valid ones (note that tzst is only
>>>       usable on FreeBSD 13 and 14)
>>>     - Add a backward compatibility for people who specified the compression
>>>       format already via PKG_SUFX and issue a warning to tell them about the
>>>       deprecation of PKG_SUFX
>>> ---
>>>  Mk/bsd.port.mk                | 58 +++++++++++++++++++++++--------------------
>>>  ports-mgmt/pkg-devel/Makefile |  4 +--
>>>  ports-mgmt/pkg-devel/distinfo |  6 ++---
>>>  ports-mgmt/pkg/Makefile       |  2 +-
>>>  ports-mgmt/pkg/distinfo       |  6 ++---
>>>  5 files changed, 40 insertions(+), 36 deletions(-)
>>>
>>> diff --git a/Mk/bsd.port.mk b/Mk/bsd.port.mk
>>> index 228644a1845f..5eba5d89da95 100644
>>> --- a/Mk/bsd.port.mk
>>> +++ b/Mk/bsd.port.mk
>>> @@ -1211,6 +1211,14 @@ _OSVERSION_MAJOR=      ${OSVERSION:C/([0-9]?[0-9])([0-9][0-9])[0-9]{3}/\1/}
>>>  .if !defined(_PKG_VERSION)
>>>  _PKG_VERSION!=       ${PKG_BIN} -v
>>>  .endif
>>> +# XXX hack for smooth transition towards pkg 1.17
>>> +_PKG_BEFORE_PKGEXT!= ${PKG_BIN} version -t ${_PKG_VERSION:C/-.*//g} 1.17.0
>>> +.if ${_PKG_BEFORE_PKGEXT} == "<"
>>> +_PKG_TRANSITIONING_TO_NEW_EXT=       yes
>>> +_EXPORTED_VARS+=     _PKG_TRANSITIONING_TO_NEW_EXT
>>> +.warning "It is strongly recommanded to upgrade to a newer vertion of pkg first"
>>> +.endif
>>> +# XXX End of hack
>>>  _PKG_STATUS!=        ${PKG_BIN} version -t ${_PKG_VERSION:C/-.*//g} ${MINIMAL_PKG_VERSION}
>>>  .if ${_PKG_STATUS} == "<"
>>>  IGNORE=              pkg(8) must be version ${MINIMAL_PKG_VERSION} or greater, but you have ${_PKG_VERSION}. You must upgrade the ${PKG_ORIGIN} port first
>>> @@ -2227,23 +2235,29 @@ _PKGMESSAGES+=        ${PKGMESSAGE}
>>>
>>>  TMPPLIST?=   ${WRKDIR}/.PLIST.mktmp
>>>
>>> -.if ${WITH_PKG} == devel
>>> -PKG_SUFX?=   .pkg
>>> +# backward compatibility for users
>>> +.if defined(_PKG_TRANSITIONING_TO_NEW_EXT)
>>>  .if defined(PKG_NOCOMPRESS)
>>> -PKG_OLDSUFX?=        .tar
>>> +PKG_SUFX?=   .tar
>>>  .else
>>> -#.if ${OSVERSION} > 1400000
>>> -#PKG_OLDSUFX?=       .tzst
>>> -#.else
>>> -PKG_OLDSUFX?=        .txz
>>> -#.endif
>>> +PKG_SUFX?=   .txz
>>>  .endif
>>> +PKG_COMPRESSION_FORMAT?=     ${PKG_SUFX:S/.//}
>>>  .else
>>> +.if defined(PKG_SUFX)
>>> +PKG_COMPRESSION_FORMAT?=     ${PKG_SUFX:S/.//}
>>> +.warning PKG_SUFX is defined, if should be replace with PKG_COMPRESSION_FORMAT
>>> +.endif
>>> +PKG_SUFX=    .pkg
>>> +.endif
>>
>> I love the idea here but can this be reverted for now please? Poudriere
>> is not prepared for a default PKG_EXT change.
>>
>> 1. Now that it finds a new PKG_EXT of .pkg, it cannot find the existing
>> bootstrap pkg.txz because it looks for Latest/pkg.pkg ->
>> ../All/pkg-*.pkg even though the *existing* one is .txz. I'm not sure
>> how this symlink is added before pkg is even built but it's not done in
>> a compatible way.
> 
> I haven't seen an issue on my end with the symlinks for pkg which
> seems to bootstrap ok and
> I don't fully understand the problem statement.  I'd like to see what
> bapt has to say for this one.
> 
>> 2. The new PKG_EXT makes Poudriere now find no existing packages so
>> everything is rebuilt. Not a deal breaker in itself but a problem for
>> future builds...
> 
> See below, maybe this is an acceptable transition hammer whenever
> someone changes it.
> 
>> 3. The rebuild won't remove all the .txz files. I assume once `pkg repo`
>> runs it will find both .txz and .pkg files and cause duplicate version
>> problems in `pkg repo`
> 
> This one is poudriere specific, do you think an acceptable solution
> would be to detect changes to the variable from the config versus the
> repo by storing a canary in the repo and run the pkgclean module if it
> changes?  I can try and do that tomorrow with your design approval or
> guidance.
> 

Sorry for lashing out yesterday.

All of these issues are poudriere-specific. None are actually hard.
Considering the right fixes and testing is why I'm asking for a revert.
So we can do it right and give other tools time to adapt. Even today I
got a bug about portupgrade being broken. I don't care to fix it myself
but a notice in advance to users that a breaking change is coming would
be nice.

I do think the official package builds (<13) will be broken for the next
week or two. Looking at beefy6 right now (12.2 amd64) I do see both .txz
and .pkg files residing in the repository. Soon it will actually run
`pkg repo`, sync, and then be serving multiple versions of packages.
It's the first run since this commit.

pkgclean does indeed fix the problem. I've thought about, in the past,
having pkgclean basically always run. That is, when deleting packages
for new versions and new dependencies and such also delete files that
are unknown. I think in poudriere-devel all that is missing in
delete_old_pkg() that pkgclean does extra is delete files not matching
*.PKG_EXT. Sounds trivial to add that in. But the official builders have
an older version of Poudriere that would need to be updated or we do
testing of the new versions. Either way some special care is needed to
make sure it works right on the builders.

Ideally we shouldn't deleting them here. We should rename them and save
the user and world resources from needless rebuilds. A fix for
pre-commit and/or post-commit of PKG_EXT isn't as obvious to me and now
that it is committed I feel there is no time to consider it so we waste
a ton of global resources when we could have done some renames with more
planning.

Related to rebuilds, but not this thread, I have been testing not auto
deleting packages just because a dependency was updated (and addressing
the dependency version problem) [1]. I was going to release a prototype
this week but now will wait for PKG_EXT to settle down. It could lead to
very weird runtime dependency problems if not done right. We need to be
able to figure out what fallout is from what cause.

[1] https://github.com/freebsd/poudriere/issues/822

> 
>> I see something creates a .txz -> .pkg symlink. What deles that symlink
>> if Poudriere only deletes/rebuilds the .pkg?
> 
> Unsure.
> 
>> ***
>> "all those .txz entries are actually _duplicates_ in packagesite.yaml"
>> seen on IRC just now by someone.
>> I can confirm that looking in a packagesite.yaml I just generated. Both
>> the older .txz version of a port and a new revision-bumped one appear in
>> the yaml.
>> ***
>>
>> 4. I've seen reports of packagesite.pkg which is concerning because
>> Poudriere manages packagesite.txz with its own symlinks for
>> hardlink-copied repositories. It does not use PKG_EXT there as they
>> weren't packages, but I suspect we need special handling even with
>> PKG_EXT added there.
>>
>> --
>> Regards,
>> Bryan Drewery
>>


-- 
Regards,
Bryan Drewery



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?2a176f0b-3a52-346f-dd25-7cedf6ca3f38>