Date: Mon, 15 Jul 2013 16:06:48 +0000 From: "Teske, Devin" <Devin.Teske@fisglobal.com> To: Baptiste Daroussin <bapt@freebsd.org> Cc: Devin Teske <dteske@freebsd.org>, Matthew Seaman <matthew@FreeBSD.org>, "current@FreeBSD.org" <current@freebsd.org>, Peter Wemm <peter@wemm.org> Subject: Re: [HEADSUP] No more pkg_install on HEAD by default Message-ID: <13CA24D6AB415D428143D44749F57D7201FC7331@ltcfiswmsgmb21> In-Reply-To: <20130715062308.GE1516@ithaqua.etoilebsd.net> References: <13CA24D6AB415D428143D44749F57D7201FC2DBD@ltcfiswmsgmb21> <20130713080732.GV85556@ithaqua.etoilebsd.net> <13CA24D6AB415D428143D44749F57D7201FC3AA2@ltcfiswmsgmb21> <13CA24D6AB415D428143D44749F57D7201FC3C92@ltcfiswmsgmb21> <CA%2B7WWSe7C2UXAsEX3OZH7nPJJbxotpQ_iYBxgake4R68sbUtLA@mail.gmail.com> <13CA24D6AB415D428143D44749F57D7201FC3FAA@ltcfiswmsgmb21> <CAGE5yCoH2auer_kKpUT_caFUZPpVM5TdAFH5tJcGgF4Ji12f0g@mail.gmail.com> <13CA24D6AB415D428143D44749F57D7201FC4E0D@ltcfiswmsgmb21> <51E26FD3.4020208@FreeBSD.org> <13CA24D6AB415D428143D44749F57D7201FC547A@ltcfiswmsgmb21> <20130715062308.GE1516@ithaqua.etoilebsd.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On Jul 14, 2013, at 11:23 PM, Baptiste Daroussin wrote: On Sun, Jul 14, 2013 at 10:12:19AM +0000, Teske, Devin wrote: On Jul 14, 2013, at 2:30 AM, Matthew Seaman wrote: On 14/07/2013 06:48, Teske, Devin wrote: Question: Where can I learn more about the actual format of what's in the new tarballs? This is going to be important not for bsdconfig, but $work (we have our own build platform; I'm going to have to rewrite it from mastering PLIST files to mastering YAML MANIFEST files and I want to know all the gritty details). We do need a pkg-manifest(5) man page, which can double as a pkg-tarball(5) page since the manifest file is where most of the interesting bits are. A pkg tarball is a compressed tar archive like so: lucid-nonsense:...cache/pkg/All:% tar -tvf pkg-1.1.4.txz -rw-r--r-- 0 root wheel 530 Jan 1 1970 +COMPACT_MANIFEST -rw-r--r-- 0 root wheel 6385 Jan 1 1970 +MANIFEST -rw-r--r-- 0 root wheel 17567 Jan 1 1970 +MTREE_DIRS -r--r--r-- 0 root wheel 19453 Jul 7 12:26 /usr/local/etc/bash_completion.d/_pkg.bash -r-xr-xr-x 0 root wheel 629 Jul 7 12:26 /usr/local/etc/periodic/daily/400.status-pkg -r-xr-xr-x 0 root wheel 823 Jul 7 12:26 /usr/local/etc/periodic/daily/411.pkg-backup -r-xr-xr-x 0 root wheel 678 Jul 7 12:26 /usr/local/etc/periodic/daily/490.status-pkg-changes -r-xr-xr-x 0 root wheel 2558 Jul 7 12:26 /usr/local/etc/periodic/security/410.pkg-audit -r-xr-xr-x 0 root wheel 611 Jul 7 12:26 /usr/local/etc/periodic/security/460.pkg-checksum -r--r--r-- 0 root wheel 839 Jul 7 12:26 /usr/local/etc/pkg.conf.sample -r--r--r-- 0 root wheel 43432 Jul 7 12:26 /usr/local/include/pkg.h -r--r--r-- 0 root wheel 727558 Jul 7 12:26 /usr/local/lib/libpkg.a lrwxr-xr-x 0 root wheel 0 Jul 7 12:26 /usr/local/lib/libpkg.so -> libpkg.so.1 -r--r--r-- 0 root wheel 1227064 Jul 7 12:26 /usr/local/lib/libpkg.so.1 -rw-r--r-- 0 root wheel 187 Jul 7 12:26 /usr/local/libdata/pkgconfig/pkg.pc [... etc ...] Interesting. I notice that (while looking ahead to see a prefix: of /usr/lo= cal in the +MANIFEST), the tarball itself has files that include /usr/local= in their path. Does pkg handle packages where the prefix (in +MANIFEST) is "/usr/local" _b= ut_ "tar tf" of the unxz'd tarball is _not_ "/usr/local"? (e.g. pkg_install= style) or is this the new way going forward? There must at least be a +MANIFEST -- other meta data files are optional. +COMPACT_MANIFEST is a subset of the full +MANIFEST. They're both YAML documents. +COMPACT_MANIFEST looks like this, for example: --- name: pkg version: 1.1.4 origin: ports-mgmt/pkg comment: New generation package manager arch: freebsd:9:x86:64 www: https://urldefense.proofpoint.com/v1/url?u=3Dhttp://wiki.freebsd.org/p= kgng&k=3D%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=3DMrjs6vR4%2Faj2Ns9%2FssHJjg= %3D%3D%0A&m=3DLPoff3ddLDoHE30AU6R5vgBklqxsjVAosXGjJop6uO4%3D%0A&s=3D30b27cf= 33d7e594144fc38f20bc75d84464788e715f7fc82b2495f813fd4be2d maintainer: portmgr@FreeBSD.org<mailto:portmgr@FreeBSD.org> prefix: /usr/local licenselogic: single licenses: - BSD flatsize: 6311507 desc: | New Generation package management tool for FreeBSD WWW: https://urldefense.proofpoint.com/v1/url?u=3Dhttp://wiki.freebsd.org/p= kgng&k=3D%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=3DMrjs6vR4%2Faj2Ns9%2FssHJjg= %3D%3D%0A&m=3DcLKrok7Cl%2BjTLEB024lmmMXfxb1fKAOZvvxLV8RjU%2FU%3D%0A&s=3Dcef= 93a5fe98a3fa0fbc9df3e0d11c9bbeaee44290306834ddef30f2ba925446c categories: - ports-mgmt shlibs_required: - libpkg.so.1 shlibs_provided: - libpkg.so.1 message: | If you are upgrading from the old package format, first run: # pkg2ng Nice. Very nice. +MTREE_DIRS is a compatibility thing with the old pkg_tools. It's not needed in general as +MANIFEST can provide all that meta data itself. It isn't going to be deprecated for at least as long as the ports tree continues to support pkg_tools though. Beyond that, the rest of the pkg tarball just contains a tar archive of all the files, directories, sym-links etc to be installed by the package. Note that pkg(8) has no problem with creating an empty directory for a package, unlike pkg_tools. Excellent! Now, while you can grovel through the details of pkg tarballs and internal data formats like this, be aware: the format of the manifest and the details of the meta-data included in the pkg-tarballs is subject to change without warning as we develop pkg(8) further. We only promise API stability through the pkg(8) commands or for the libpkg.so library functions; reports of breakage from usage outside those APIs will receive little sympathy. Understood. So just to give you a better idea of how we manage packages here. We've realized that if you want to package for FreeBSD (in 9 and older), yo= u could get by alright if you knew how to create a +CONTENTS file from scra= tch. For RedHat, it's the SPECFILE. And now for FreeBSD 10, it looks an awf= ul lot like a SPECFILE (they are both in-fact YAML). The only thing you need if you really want to package is a small subset of = the manifest written in YAML: Have a look at what the ports tree do to create the minimalistic manifest a= nd what form it has and you will see how easy it is. It is so awful and complicated that there are already a load of custom scri= pts to create packages with pkg without using the ports tree. As an example: https://urldefense.proofpoint.com/v1/url?u=3Dhttps://github.com/z0nt/pkg&k= =3D%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=3DMrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D= %0A&m=3DcLKrok7Cl%2BjTLEB024lmmMXfxb1fKAOZvvxLV8RjU%2FU%3D%0A&s=3Dfa25d4b17= 9a2b815c1bffab49adb850f44a6437f175aa93dd159887652935258 , but there is way = more. I for example wrote in the past a plugin for pkg to be able to parse and USE RPM's specfiles as an input for pkg. Cool. Wonder if anyone will end up using that. It does sound rather neat fo= r (perhaps) people more RedHat centric that may want to re-use the same SPE= CFILE of their RPM to produce a pkgng file. So rather than teach the people here how to use tools, I taught them what I= think is more important... how to manage +CONTENTS, SPECFILE, and now up-a= nd-coming, +MANIFEST. The problem is that you compare 2 things that cannot be compared, In normal light, yes... but when actually, we've essentially wiped the exis= tence and knowledge of "source RPMs" from our minds as our build system imb= ues us with that luxury (the luxury of never needing a source RPM, never pr= oducing one, and never even thinking about one -- we go from SPECFILE direc= tly to binary RPM and when we need to modify a binary RPM, we take a templa= te SPECFILE and populate it with the metadata extracted from the rpm file i= tself (using "rpm" utility to extract said metadata). SPECFILE is the equivalent of pkg + the ports tree. In normal case, yes... in our case no. Every single SPECFILE in our build s= ystem uses the exact same assembly procedure to produce [directly] a binary= RPM. In fact specfiles are the equivalent of a given "port" !!! Are you writing = rpm files directly by hand? there is no SPECFILE inside RPM but rather a header container metadata in binary format and a body which is a cpio.(gz,bz2,xz). Correct, a binary RPM does not contain a SPECFILE. But, every piece of data= that goes into the SPECFILE is contained in metadata extractable with "rpm= -q -p RPMFILE --qf FORMAT" (iirc). This is how we turn a binary RPM into a build-directory with a SPECFILE tha= t -- when we say "make" -- produces the same binary RPM file with-or-withou= t any/all changes to the flat [unpacked] hierarchy: http://druidbsd.cvs.sf.net/viewvc/druidbsd/pkgbase/redhat/unpack.sh?revisio= n=3D1.1&content-type=3Dtext%2Fplain<http://druidbsd.cvs.sf.net/viewvc/druid= bsd/pkgbase/redhat/unpack.sh?revision=3D1.1&content-type=3Dtext/plain> You can take a look at an example SPECFILE that's been generated from a bin= ary RPM using the above script: http://druidbsd.cvs.sf.net/viewvc/druidbsd/pkgbase/redhat/rhel6-x86/System_= Environment/Daemons/sup/SPECFILE?revision=3D1.1&view=3Dmarkup NOTE: In the case of the above URL, it turned out that the "sup" package wa= sn't available for RedHat EL 6 -- so we slurped in some other simple packag= e and replaced its guts with that of a sup package compiled specifically fo= r RedHat EL 6 (with fixes). But the SPECFILE starts its life as a template for each/every unpack of a b= inary RPM (see below): http://druidbsd.cvs.sf.net/viewvc/druidbsd/pkgbase/redhat/Mk/template.spec?= revision=3D1.1&view=3Dmarkup You are not creating the RPM file directly but rather create a SPECFILE, the equivalent on FreeBSD is to create a port!!! Incorrect; as you can see from the above, we *do* treat SPECFILE as a PLIST= (aka +CONTENTS). We do build the RPM from the SPECFILE. You're absolutely right that the SPECFILE *normally* contains logic on how = to turn a vendor software X (in source form) into [first] a "source" RPM an= d then a "binary" RPM. But we've taken all that logic out of the SPECFILE a= nd turned it into a dumb thing that just goes direct to binary RPM. the +MANIFEST as used to be the +CONTENTS are the equivalent of the RPM metadata. Right -- we use "rpm"'s --queryformat parameter to extract the metadata. We wrote them in YAML so that they are easy to debug and that we can use third party well used and tested parsers instead of writting our own. I'm very thankful that the compiled pkgng package isn't as opaque as RPM. I= think it's a good thing for transparency. I also like that you're porting = your tools. pkgng has indeed a bright future and you've been making great c= hoices. (and there is no "but" -- just wanted to say "thank you") I hope you don't see my dissection attempt as a way to subvert what you're = doing. Quite the opposite, I'm dissecting so that I can learn in an effort = to integrate. However, I'd be lying if I said I taught them *all* the gory details. In re= ality, for +CONTENTS they edit a PLIST which is essentially +CONTENTS witho= ut the "@comment MD5:" entries. For SPECFILE, they manage the full thing ex= cept there's a small section that is the standard RPM bootstrap that is lab= eled as "do not touch". >From what you posted of +COMPACT_MANIFEST was nice, except it lacks the lis= t of files. The basic principle here is that if you have to maintain a list of files, a= nd that list ends up being compiled into a +MANIFEST, why not just teach yo= ur peers how to read/write/maintain +MANIFEST files (in version control of = course) so that there are never any mysteries as to how the package perform= s. Have a closer look at pkg create, you will see that we are still able to us= e the plist or +CONTENTS as a possible input for pkg create, it is even the main = way to create packages. Ooo, nice. That will smooth migration (however, we it will be a while befor= e we get onto 10.x+ so we've got some time -- meaning, I might favor a rout= e that is legacy free.. agree?) Once again, the main and most important requirement to build packages was t= o be compatible with the ports tree, and the ports tree is using plist/+CONTENTS. and pkg2ng does only works by using the +CONTENTS. I know this sounds screwy... but (as with our other YAML files), it's reall= y nice because (as is the case with SPECFILE), we version-control things as= -close to the end-product as possible (take for example the case of "pre-in= stall" or "post-install" et cetera scripts -- I'm sure a tool like "pkg cre= ate" or "pkg_create" or other could do a good job of assembling the pieces = into the +MANIFEST, but then it's not being version-controlled as closely. You can keep what you have already done. Once again pkg does not have the equivalent of the SPECFILE because we have= the ports tree! In Linux, rpmbuild will take a SPECFILE in the manner you mention (acting a= s the pre-cursor to the SRPM), but it will also take a SPECFILE to build a = binary RPM (and in that last manner, the SPECFILE is not like the ports-tre= e as it expects binaries, not source, to build the cpio-ball). In the manner we're using SPECFILE (the latter manner), it *is* like a PLIS= T or a MANIFEST. so we do not need to provide a custom package building system. We could, as it is really easy, but we do not need so it is not there yet, = as said earlier I already wrote a pkg plugin that is able to build packages fr= om a RPM specfile, I also modified archlinux's makepkg to be able to output pkgn= g files. I did the same on slackare, etc. Here the limitation is not technical at al= l. That's really nice and encouraging. As I do-say... it's remarkable and the = future of pkgng looks bright ;D -- Devin _____________ The information contained in this message is proprietary and/or confidentia= l. If you are not the intended recipient, please: (i) delete the message an= d all copies; (ii) do not disclose, distribute or use the message in any ma= nner; and (iii) notify the sender immediately. In addition, please be aware= that any message addressed to our domain is subject to archiving and revie= w by persons other than the intended recipient. Thank you.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?13CA24D6AB415D428143D44749F57D7201FC7331>