Date: Sun, 14 Jul 2013 10:12:19 +0000 From: "Teske, Devin" <Devin.Teske@fisglobal.com> To: Matthew Seaman <matthew@FreeBSD.org> Cc: Baptiste Daroussin <bapt@freebsd.org>, Devin Teske <dteske@freebsd.org>, "current@FreeBSD.org" <current@freebsd.org>, Peter Wemm <peter@wemm.org> Subject: Re: [HEADSUP] No more pkg_install on HEAD by default Message-ID: <13CA24D6AB415D428143D44749F57D7201FC547A@ltcfiswmsgmb21> In-Reply-To: <51E26FD3.4020208@FreeBSD.org> References: <20130712231637.GS85556@ithaqua.etoilebsd.net> <13CA24D6AB415D428143D44749F57D7201FC2DBD@ltcfiswmsgmb21> <20130713080732.GV85556@ithaqua.etoilebsd.net> <13CA24D6AB415D428143D44749F57D7201FC3AA2@ltcfiswmsgmb21> <13CA24D6AB415D428143D44749F57D7201FC3C92@ltcfiswmsgmb21> <CA%2B7WWSe7C2UXAsEX3OZH7nPJJbxotpQ_iYBxgake4R68sbUtLA@mail.gmail.com> <13CA24D6AB415D428143D44749F57D7201FC3FAA@ltcfiswmsgmb21> <CAGE5yCoH2auer_kKpUT_caFUZPpVM5TdAFH5tJcGgF4Ji12f0g@mail.gmail.com> <13CA24D6AB415D428143D44749F57D7201FC4E0D@ltcfiswmsgmb21> <51E26FD3.4020208@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Jul 14, 2013, at 2:30 AM, Matthew Seaman wrote: > On 14/07/2013 06:48, Teske, Devin wrote: >> Question: Where can I learn more about the actual format of what's in >> the new tarballs? This is going to be important not for bsdconfig, >> but $work (we have our own build platform; I'm going to have to >> rewrite it from mastering PLIST files to mastering YAML MANIFEST >> files and I want to know all the gritty details). >=20 > We do need a pkg-manifest(5) man page, which can double as a > pkg-tarball(5) page since the manifest file is where most of the > interesting bits are. >=20 > A pkg tarball is a compressed tar archive like so: >=20 > lucid-nonsense:...cache/pkg/All:% tar -tvf pkg-1.1.4.txz > -rw-r--r-- 0 root wheel 530 Jan 1 1970 +COMPACT_MANIFEST > -rw-r--r-- 0 root wheel 6385 Jan 1 1970 +MANIFEST > -rw-r--r-- 0 root wheel 17567 Jan 1 1970 +MTREE_DIRS > -r--r--r-- 0 root wheel 19453 Jul 7 12:26 > /usr/local/etc/bash_completion.d/_pkg.bash > -r-xr-xr-x 0 root wheel 629 Jul 7 12:26 > /usr/local/etc/periodic/daily/400.status-pkg > -r-xr-xr-x 0 root wheel 823 Jul 7 12:26 > /usr/local/etc/periodic/daily/411.pkg-backup > -r-xr-xr-x 0 root wheel 678 Jul 7 12:26 > /usr/local/etc/periodic/daily/490.status-pkg-changes > -r-xr-xr-x 0 root wheel 2558 Jul 7 12:26 > /usr/local/etc/periodic/security/410.pkg-audit > -r-xr-xr-x 0 root wheel 611 Jul 7 12:26 > /usr/local/etc/periodic/security/460.pkg-checksum > -r--r--r-- 0 root wheel 839 Jul 7 12:26 > /usr/local/etc/pkg.conf.sample > -r--r--r-- 0 root wheel 43432 Jul 7 12:26 /usr/local/include/pkg.h > -r--r--r-- 0 root wheel 727558 Jul 7 12:26 /usr/local/lib/libpkg.a > lrwxr-xr-x 0 root wheel 0 Jul 7 12:26 /usr/local/lib/libpkg.so > -> libpkg.so.1 > -r--r--r-- 0 root wheel 1227064 Jul 7 12:26 /usr/local/lib/libpkg.so.1 > -rw-r--r-- 0 root wheel 187 Jul 7 12:26 > /usr/local/libdata/pkgconfig/pkg.pc > [... etc ...] >=20 Interesting. I notice that (while looking ahead to see a prefix: of /usr/lo= cal in the +MANIFEST), the tarball itself has files that include /usr/local= in their path. Does pkg handle packages where the prefix (in +MANIFEST) is "/usr/local" _b= ut_ "tar tf" of the unxz'd tarball is _not_ "/usr/local"? (e.g. pkg_install= style) or is this the new way going forward? > There must at least be a +MANIFEST -- other meta data files are > optional. +COMPACT_MANIFEST is a subset of the full +MANIFEST. They're > both YAML documents. +COMPACT_MANIFEST looks like this, for example: >=20 > --- > name: pkg > version: 1.1.4 > origin: ports-mgmt/pkg > comment: New generation package manager > arch: freebsd:9:x86:64 > www: https://urldefense.proofpoint.com/v1/url?u=3Dhttp://wiki.freebsd.org= /pkgng&k=3D%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=3DMrjs6vR4%2Faj2Ns9%2FssHJ= jg%3D%3D%0A&m=3DLPoff3ddLDoHE30AU6R5vgBklqxsjVAosXGjJop6uO4%3D%0A&s=3D30b27= cf33d7e594144fc38f20bc75d84464788e715f7fc82b2495f813fd4be2d > maintainer: portmgr@FreeBSD.org > prefix: /usr/local > licenselogic: single > licenses: > - BSD > flatsize: 6311507 > desc: | > New Generation package management tool for FreeBSD >=20 > WWW: http://wiki.freebsd.org/pkgng > categories: > - ports-mgmt > shlibs_required: > - libpkg.so.1 > shlibs_provided: > - libpkg.so.1 > message: | > If you are upgrading from the old package format, first run: >=20 > # pkg2ng >=20 Nice. Very nice. > +MTREE_DIRS is a compatibility thing with the old pkg_tools. It's not > needed in general as +MANIFEST can provide all that meta data itself. > It isn't going to be deprecated for at least as long as the ports tree > continues to support pkg_tools though. >=20 > Beyond that, the rest of the pkg tarball just contains a tar archive of > all the files, directories, sym-links etc to be installed by the > package. Note that pkg(8) has no problem with creating an empty > directory for a package, unlike pkg_tools. >=20 Excellent! > Now, while you can grovel through the details of pkg tarballs and > internal data formats like this, be aware: the format of the manifest > and the details of the meta-data included in the pkg-tarballs is subject > to change without warning as we develop pkg(8) further. We only promise > API stability through the pkg(8) commands or for the libpkg.so library > functions; reports of breakage from usage outside those APIs will > receive little sympathy. >=20 Understood. So just to give you a better idea of how we manage packages here. We've realized that if you want to package for FreeBSD (in 9 and older), yo= u could get by alright if you knew how to create a +CONTENTS file from scra= tch. For RedHat, it's the SPECFILE. And now for FreeBSD 10, it looks an awf= ul lot like a SPECFILE (they are both in-fact YAML). So rather than teach the people here how to use tools, I taught them what I= think is more important... how to manage +CONTENTS, SPECFILE, and now up-a= nd-coming, +MANIFEST. However, I'd be lying if I said I taught them *all* the gory details. In re= ality, for +CONTENTS they edit a PLIST which is essentially +CONTENTS witho= ut the "@comment MD5:" entries. For SPECFILE, they manage the full thing ex= cept there's a small section that is the standard RPM bootstrap that is lab= eled as "do not touch". >From what you posted of +COMPACT_MANIFEST was nice, except it lacks the lis= t of files. The basic principle here is that if you have to maintain a list of files, a= nd that list ends up being compiled into a +MANIFEST, why not just teach yo= ur peers how to read/write/maintain +MANIFEST files (in version control of = course) so that there are never any mysteries as to how the package perform= s. I know this sounds screwy... but (as with our other YAML files), it's reall= y nice because (as is the case with SPECFILE), we version-control things as= -close to the end-product as possible (take for example the case of "pre-in= stall" or "post-install" et cetera scripts -- I'm sure a tool like "pkg cre= ate" or "pkg_create" or other could do a good job of assembling the pieces = into the +MANIFEST, but then it's not being version-controlled as closely. Before this workflow, mistakes were rampant and there wasn't much hope of w= rangling the "oops, forgot to touch the PLIST" or "oops, forgot to update t= he post-install script" mistakes. I see a nice bright future in re-working my "pkgbase" to be able to supplan= t metadata into a revision-controlled +MANIFEST. Ideally, I don't want them to have to be burdened with maintaining both a += MANIFEST and +COMPACT_MANIFEST, so it seems most logical to generate one fr= om the other (the latter being the revision-controlled copy sans-meta-data;= the meta-data is added at "make" time before then running validations and = generating the tarball). The problem of breakage to the system by API changes is less important, bec= ause we track security releng branches and use binaries so changes are very= slow to creep in. But since *can* use a different framework for each/every= major release branch, we can track API changes quite easily. I'm aware that only slow moving dinosaurs like us are interested in such ba= ckward-thinking mechanics... but this stuff really works for us, and I'm ce= rtain that digesting +MANIFEST will be very beneficial to our growth. --=20 Devin _____________ The information contained in this message is proprietary and/or confidentia= l. If you are not the intended recipient, please: (i) delete the message an= d all copies; (ii) do not disclose, distribute or use the message in any ma= nner; and (iii) notify the sender immediately. In addition, please be aware= that any message addressed to our domain is subject to archiving and revie= w by persons other than the intended recipient. Thank you.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?13CA24D6AB415D428143D44749F57D7201FC547A>