From owner-freebsd-arch@FreeBSD.ORG Wed Jul 11 03:17:56 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0F958106564A for ; Wed, 11 Jul 2012 03:17:56 +0000 (UTC) (envelope-from rysto32@gmail.com) Received: from mail-we0-f182.google.com (mail-we0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id 955C38FC12 for ; Wed, 11 Jul 2012 03:17:55 +0000 (UTC) Received: by weyx56 with SMTP id x56so541715wey.13 for ; Tue, 10 Jul 2012 20:17:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=LE81uihpirhweTW4cLbG0hdndBNy/F+8bjLaSHdDwOY=; b=CEZwtvt91QazCb0k0OaLJP983J936L1olNi5H01JXd4BYqNH//p3nGbN0c0BuPe+fu blMjRvgxzC/Pg17h4/X8ZDqMcky0QYJ9CWDy7wLBsFRwcVNrd3+R8MqS262Y4NV0A1Lo N3l3cp8iJZYX5AEAGTKUmbSL36YOKAo3S3wNu0Yx7kMqm+RJ83hQAKhTxbEFhowym1u1 XgDYW0neHTxQ8oANNeCAbco+I6zV1hIm5qyEkt60MFw9XQruoWQSfQSXaP09mar8oMa+ 1jEEjUji+jLmEiv+VTFgZyy03ZTMgNXehjnKi1U1JRmM0xq6/88kBvItkiaoXiQqUJSs m8UA== MIME-Version: 1.0 Received: by 10.180.80.134 with SMTP id r6mr43338727wix.1.1341976674747; Tue, 10 Jul 2012 20:17:54 -0700 (PDT) Received: by 10.180.103.137 with HTTP; Tue, 10 Jul 2012 20:17:54 -0700 (PDT) Date: Tue, 10 Jul 2012 23:17:54 -0400 Message-ID: From: Ryan Stone To: freebsd-arch@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Subject: Generating a tarball directly from make installworld X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Jul 2012 03:17:56 -0000 I was playing around with this a couple of months ago and a recent thread on -arch prompted me to pick it up again. The idea, for those who aren't familiar with it, is to have the base system build infrastructure generate tarballs directly. I believe that that the current approach is to do a make installworld installkernel distribution to an empty directory, and then tar that up. That's not an ideal system because in order to have file ownership, flags, etc be set correctly, installworld and friends must be run as root. The method that I was trying was to write a new tool based on libarchive that emulates the tools used by installworld and friends but acts directly on a tar file. I've made decent progress but I've hit some pretty big roadblocks and I think that I need to take a step back and solicit feedback before I go too much deeper down the rabbit hole. I'm hoping that people have already thought of the problems that I'm hitting and have ideas on how to get passed them. Currently, installworld and friends use the following tools to get the right bits on disk: - install - ln - mkdir - rm - zic (timezone data compiler) - install-info (install for GNU info pages) - makewhatis (generates manpage index) - cap_mkdb (generates pwd.db and spwd.db from /etc/passwd and master.passwd) - kldxref (generates linker.hints) I would say that makewhatis and kldxref are the knottiest problems. Both look at DESTDIR and generate metadata based on the files that have been installed there. As the files originate from many different locations in OBJDIR, it's not as simple as pointing the tools at OBJDIR and working out of there(by contrast, zic and cap_mkdb operate on a limited number of files in SRCDIR, so they could be worked around if we really had to). I suppose that the files could be all installed to a temporary location, and makewhatis/kldxref run against the temp directory (with the output from that being install'ed to the right place), but that's quite hacky. The other problem that I have is performance. When bsdtar appends to a tar file, it iterates over every entry in the tar to figure out where the end of it is. I gather that this is to get rid of padding but I'm not entirely sure. Even if this isn't necessary I still have to iterate over the entire file in most cases. The problem is in the sloppy semantics of ln and install: install foo bar means "install foo to path bar/foo" if bar is a directory, but "install foo to path bar" if bar is a regular file or it doesn't exist(symlinks add an extra layer of complexity). In order to implement this correctly, I have to iterate over the tar to figure out what type of file bar is, every time that install or ln is invoked. As you can probably image this gets quite slow quite quickly -- installing to a tar file on my system seems to be at least 10 times slower than installing to disk. I know that a lot of people have suggested generating an mtree file and then converting the mtree file into a tarball, but I admit that it's not at all clear to me how to generate the mtree file. It feels to me that it ends up being equivalent in complexity as what I've tried, but maybe I'm missing something about mtree's capabilities. Or maybe people are just proposing a more fundamental change to what tools we use to get bits on disk, and I missed that implication. I know that a lot of people have opinions on this, so kindly speak up. :) From owner-freebsd-arch@FreeBSD.ORG Wed Jul 11 04:47:15 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 841B8106564A for ; Wed, 11 Jul 2012 04:47:15 +0000 (UTC) (envelope-from tim@kientzle.com) Received: from monday.kientzle.com (99-115-135-74.uvs.sntcca.sbcglobal.net [99.115.135.74]) by mx1.freebsd.org (Postfix) with ESMTP id 5FF3A8FC08 for ; Wed, 11 Jul 2012 04:47:15 +0000 (UTC) Received: (from root@localhost) by monday.kientzle.com (8.14.4/8.14.4) id q6B4lD1W033032; Wed, 11 Jul 2012 04:47:13 GMT (envelope-from tim@kientzle.com) Received: from [192.168.2.143] (CiscoE3000 [192.168.1.65]) by kientzle.com with SMTP id 8f3qq7yj7ic4cq4hcby3rdra9e; Wed, 11 Jul 2012 04:47:13 +0000 (UTC) (envelope-from tim@kientzle.com) Mime-Version: 1.0 (Apple Message framework v1278) Content-Type: text/plain; charset=iso-8859-1 From: Tim Kientzle In-Reply-To: Date: Tue, 10 Jul 2012 21:47:13 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: <25149679-6B99-4FF0-AB8C-90D5A7880F00@kientzle.com> References: To: Ryan Stone X-Mailer: Apple Mail (2.1278) Cc: freebsd-arch@freebsd.org Subject: Re: Generating a tarball directly from make installworld X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Jul 2012 04:47:15 -0000 On Jul 10, 2012, at 8:17 PM, Ryan Stone wrote: >=20 > The other problem that I have is performance. When bsdtar appends to > a tar file, it iterates over every entry in the tar to figure out > where the end of it is. I gather that this is to get rid of padding > but I'm not entirely sure. It is unfortunately necessary if you want to append to an existing file. In short: The tar format wasn't designed for appending, and the append option of the standard tar command is a 30-year old hack. But, I wrote a better approach into libarchive and bsdtar a few years ago. See below. > Even if this isn't necessary I still have > to iterate over the entire file in most cases. The problem is in the > sloppy semantics of ln and install: install foo bar means "install foo > to path bar/foo" if bar is a directory, but "install foo to path bar" > if bar is a regular file or it doesn't exist(symlinks add an extra > layer of complexity). In order to implement this correctly, I have to > iterate over the tar to figure out what type of file bar is, every > time that install or ln is invoked.=20 The sloppy semantics are indeed a problem and I hadn't considered this before. I fear the only answer might be to fix the Makefiles so they don't rely on this (fortunately, most of the install and ln invocations are built from just a few places, so it might not be necessary to change very many places to fix it). > I know that a lot of people have suggested generating an mtree file > and then converting the mtree file into a tarball, but I admit that > it's not at all clear to me how to generate the mtree file. I can definitely help with this, since I had this exact use in mind when I originally built that part of libarchive. First, there are actually two different variants of mtree format. The one supported by FreeBSD's mtree is the older one. It's very pretty with all that indentation but not particularly amenable to this kind of task. The interesting one is a newer variant supported by NetBSD's mtree and also supported by libarchive. In the newer mtree variant, each line is completely self-contained, e.g., /bin/ls group=3Dwheel user=3Droot mode=3D0755 Such files can be easily combined (just append them together), can be appended to via "echo spec >>file", etc. Libarchive extends this further by adding a "contents" keyword, e.g., /bin/ls user=3Droot group=3Dwheel mode=3D0755 = contents=3D/usr/obj/usr/src/bin/ls/ls When libarchive reads this line, it returns a file description that has: * The specified name * The specified properties * The specified contents * (other properties --- including file size --- are taken from the = contents file) So, my idea was that 'install' or 'ln' could write a line like the above to /usr/obj/usr/src/bin/ls/ls.dist-mtree and at the end you could pull all those together and build a tar ball in a single fast pass: find . -name '*.dist-mtree' | xargs cat | bsdtar czf distfile.tgz @- I wrote "man 5 mtree" to attempt to develop a single consistent description of both mtree variants. Ignore the mention of a signature; that was misguided wishful thinking on my part as I wrestled with how to teach libarchive to automatically recognize mtree files. Michihiro fortunately figured out a better way to do that. The '@-' here is a bsdtar extension that reads an archive and appends the entries from that archive to the archive being created. For even more fun, you can install directly from the mtree descriptions: find . -name '*.dist-mtree' | xargs cat | bsdtar xf - Another nice trick with this extended mtree format: it's relatively easy to use tools like grep and see to filter the mtree description, so you could play with having makewhatis or kldxref read from an archive and then let libarchive unpack directly from mtree for you, e.g., find . -name '*.dist-mtree' | xargs cat | grep '/man/' | makewhatis = --read-from-stdin-archive Let me know if I can help. As I said, I had this exact application in mind when I built this support into libarchive and bsdtar. If there are additional tweaks that would help, I'll see what I can do. Tim From owner-freebsd-arch@FreeBSD.ORG Wed Jul 11 11:09:49 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 104161065673 for ; Wed, 11 Jul 2012 11:09:49 +0000 (UTC) (envelope-from des@des.no) Received: from smtp.des.no (smtp.des.no [194.63.250.102]) by mx1.freebsd.org (Postfix) with ESMTP id C414A8FC0C for ; Wed, 11 Jul 2012 11:09:48 +0000 (UTC) Received: from ds4.des.no (smtp.des.no [194.63.250.102]) by smtp.des.no (Postfix) with ESMTP id BF4146DEC; Wed, 11 Jul 2012 11:09:41 +0000 (UTC) Received: by ds4.des.no (Postfix, from userid 1001) id 5947B8BA2; Wed, 11 Jul 2012 13:09:41 +0200 (CEST) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: Tim Kientzle References: <25149679-6B99-4FF0-AB8C-90D5A7880F00@kientzle.com> Date: Wed, 11 Jul 2012 13:09:40 +0200 In-Reply-To: <25149679-6B99-4FF0-AB8C-90D5A7880F00@kientzle.com> (Tim Kientzle's message of "Tue, 10 Jul 2012 21:47:13 -0700") Message-ID: <861ukio8p7.fsf@ds4.des.no> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.4 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: Ryan Stone , freebsd-arch@freebsd.org Subject: Re: Generating a tarball directly from make installworld X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Jul 2012 11:09:49 -0000 Tim Kientzle writes: > /bin/ls user=3Droot group=3Dwheel mode=3D0755 contents=3D/usr/obj/usr/src= /bin/ls/ls Unfortunately, passing that line to tar is not quite equivalent to first installing ls then tarring it, because install normally strips ls. There may be other cases as well where the file that ends up being installed does not exist in that form - or at all - in the object tree, but is transformed or even created at install time. DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-arch@FreeBSD.ORG Wed Jul 11 16:49:19 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8EE3F106566C for ; Wed, 11 Jul 2012 16:49:19 +0000 (UTC) (envelope-from bright@elvis.mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 6808D8FC15 for ; Wed, 11 Jul 2012 16:49:19 +0000 (UTC) Received: by elvis.mu.org (Postfix, from userid 1192) id 85FE81A3C1B; Wed, 11 Jul 2012 09:41:11 -0700 (PDT) Date: Wed, 11 Jul 2012 09:41:11 -0700 From: Alfred Perlstein To: Ryan Stone Message-ID: <20120711164111.GJ88966@elvis.mu.org> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i Cc: freebsd-arch@freebsd.org Subject: Re: Generating a tarball directly from make installworld X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Jul 2012 16:49:19 -0000 * Ryan Stone [120710 20:18] wrote: > I was playing around with this a couple of months ago and a recent > thread on -arch prompted me to pick it up again. The idea, for those > who aren't familiar with it, is to have the base system build > infrastructure generate tarballs directly. I believe that that the > current approach is to do a make installworld installkernel > distribution to an empty directory, and then tar that up. That's not > an ideal system because in order to have file ownership, flags, etc be > set correctly, installworld and friends must be run as root. > This is pretty cool, I would suggest going along the route of mtree + tar as you mentioned. If you hacked install(1) to install without special permissions and then to also make a simple file that has "path/file perms owners", then later convert that to an mtree file you should be fine. It seems as if tar already can take an mtree file in order to "fixup" the permissions as you need. So all you really need to do is hack install(1) to log what permissions/owner it wants to use, then convert that to mtree, then input that mtree into tar. I could be missing something, but I think that should be a LOT easier. If you need help with mtree drop me a line, I'm not *that* familiar with it, but maybe together we can make something that works. -Alfred > The method that I was trying was to write a new tool based on > libarchive that emulates the tools used by installworld and friends > but acts directly on a tar file. I've made decent progress but I've > hit some pretty big roadblocks and I think that I need to take a step > back and solicit feedback before I go too much deeper down the rabbit > hole. I'm hoping that people have already thought of the problems > that I'm hitting and have ideas on how to get passed them. > > Currently, installworld and friends use the following tools to get the > right bits on disk: > - install > - ln > - mkdir > - rm > - zic (timezone data compiler) > - install-info (install for GNU info pages) > - makewhatis (generates manpage index) > - cap_mkdb (generates pwd.db and spwd.db from /etc/passwd and master.passwd) > - kldxref (generates linker.hints) > > I would say that makewhatis and kldxref are the knottiest problems. > Both look at DESTDIR and generate metadata based on the files that > have been installed there. As the files originate from many different > locations in OBJDIR, it's not as simple as pointing the tools at > OBJDIR and working out of there(by contrast, zic and cap_mkdb operate > on a limited number of files in SRCDIR, so they could be worked around > if we really had to). I suppose that the files could be all installed > to a temporary location, and makewhatis/kldxref run against the temp > directory (with the output from that being install'ed to the right > place), but that's quite hacky. > > The other problem that I have is performance. When bsdtar appends to > a tar file, it iterates over every entry in the tar to figure out > where the end of it is. I gather that this is to get rid of padding > but I'm not entirely sure. Even if this isn't necessary I still have > to iterate over the entire file in most cases. The problem is in the > sloppy semantics of ln and install: install foo bar means "install foo > to path bar/foo" if bar is a directory, but "install foo to path bar" > if bar is a regular file or it doesn't exist(symlinks add an extra > layer of complexity). In order to implement this correctly, I have to > iterate over the tar to figure out what type of file bar is, every > time that install or ln is invoked. As you can probably image this > gets quite slow quite quickly -- installing to a tar file on my system > seems to be at least 10 times slower than installing to disk. > > I know that a lot of people have suggested generating an mtree file > and then converting the mtree file into a tarball, but I admit that > it's not at all clear to me how to generate the mtree file. It feels > to me that it ends up being equivalent in complexity as what I've > tried, but maybe I'm missing something about mtree's capabilities. Or > maybe people are just proposing a more fundamental change to what > tools we use to get bits on disk, and I missed that implication. > > I know that a lot of people have opinions on this, so kindly speak up. :) > _______________________________________________ > freebsd-arch@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" -- - Alfred Perlstein .- VMOA #5191, 03 vmax, 92 gs500, 85 ch250, 07 zx10 .- FreeBSD committer From owner-freebsd-arch@FreeBSD.ORG Thu Jul 12 01:58:47 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 46C31106564A for ; Thu, 12 Jul 2012 01:58:47 +0000 (UTC) (envelope-from tim@kientzle.com) Received: from monday.kientzle.com (99-115-135-74.uvs.sntcca.sbcglobal.net [99.115.135.74]) by mx1.freebsd.org (Postfix) with ESMTP id 191F48FC16 for ; Thu, 12 Jul 2012 01:58:46 +0000 (UTC) Received: (from root@localhost) by monday.kientzle.com (8.14.4/8.14.4) id q6C1wZY3038023; Thu, 12 Jul 2012 01:58:35 GMT (envelope-from tim@kientzle.com) Received: from [192.168.2.143] (CiscoE3000 [192.168.1.65]) by kientzle.com with SMTP id f4dfrxnk5ryjg736w48389c8mi; Thu, 12 Jul 2012 01:58:35 +0000 (UTC) (envelope-from tim@kientzle.com) Mime-Version: 1.0 (Apple Message framework v1278) Content-Type: text/plain; charset=windows-1252 From: Tim Kientzle In-Reply-To: <861ukio8p7.fsf@ds4.des.no> Date: Wed, 11 Jul 2012 18:58:34 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: <8E4D9B54-C328-427D-89A1-0CB8E57CD064@kientzle.com> References: <25149679-6B99-4FF0-AB8C-90D5A7880F00@kientzle.com> <861ukio8p7.fsf@ds4.des.no> To: =?iso-8859-1?Q?Dag-Erling_Sm=F8rgrav?= X-Mailer: Apple Mail (2.1278) Cc: Ryan Stone , freebsd-arch@freebsd.org Subject: Re: Generating a tarball directly from make installworld X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jul 2012 01:58:47 -0000 On Jul 11, 2012, at 4:09 AM, Dag-Erling Sm=F8rgrav wrote: > Tim Kientzle writes: >> /bin/ls user=3Droot group=3Dwheel mode=3D0755 = contents=3D/usr/obj/usr/src/bin/ls/ls >=20 > Unfortunately, passing that line to tar is not quite equivalent to = first > installing ls then tarring it, because install normally strips ls. Yes, it does today. But we could easily produce the stripped executable at build time using something like this in bsd.prog.mk: ${PROG}: ${OBJS} ${CC} =85. -o ${.TARGET} ${OBJS} =85 cp ${.TARGET} ${.TARGET}.stripped strip ${.TARGET}.stripped Then ${.TARGET}.stripped is ready to be copied as-is into the tarball. While we're at it, I think the following points out how to address another of Ryan's issues: --- share/mk/bsd.prog.mk.original 2012-07-11 18:48:11.000000000 = -0700 +++ share/mk/bsd.prog.mk 2012-07-11 18:48:35.000000000 -0700 @@ -155,7 +155,7 @@ ${_INSTALLFLAGS} ${PROG} ${DESTDIR}${BINDIR}/${PROGNAME} .else ${INSTALL} ${STRIP} -o ${BINOWN} -g ${BINGRP} -m ${BINMODE} \ - ${_INSTALLFLAGS} ${PROG} ${DESTDIR}${BINDIR} + ${_INSTALLFLAGS} ${PROG} ${DESTDIR}${BINDIR}/${PROG} .endif .endif .endif # !target(realinstall) > There may be other cases as well where the file that ends up being > installed does not exist in that form - or at all - in the object = tree, > but is transformed or even created at install time. Yes. Ryan listed quite a few of these in his original message. I suspect they'll have to be handled very much on a case-by-case basis. There are several different strategies that might apply: Option #1. Move work from install time to build time. This is easy for single-file transforms such as stripping binaries, as above. It might be feasible for other cases: I think you could handle kldxref this way, though it would require a little bit of creativity to find all the modules. Option #2. Handle it using a "mini-install" For makewhatis, the easiest might be to install just the man files to a temporary dir and run the indexing against that. This would work well for makewhatis in particular since the ownership and permissions aren't critical for generating that file. Option #3. Integrate it into a special archiving tool. bsdtar demonstrates that mtree-to-tar.gz conversion is feasible, but that doesn't mean the tool that does this needs to be tar itself. A purpose-built tool could use a lot of the same code (much of which is already in libarchive) and add additional capabilities to generate various install files on-the-fly. That would be a lot of work for makewhatis or kldxref, of course, but might be a reasonable way to handle stripping binaries? Option #4. Defer to system boot. We already have systems that do initialization on the first boot after install (ssh host keys, for instance). A generic framework for this would be easy to add to our current boot scripts. I think this would be a very good fit for makewhatis in particular. I could see any of the above being useful, depending on the particular situation. Tim From owner-freebsd-arch@FreeBSD.ORG Thu Jul 12 03:53:58 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 26FD51065675 for ; Thu, 12 Jul 2012 03:53:58 +0000 (UTC) (envelope-from des@des.no) Received: from smtp.des.no (smtp.des.no [194.63.250.102]) by mx1.freebsd.org (Postfix) with ESMTP id D850C8FC0C for ; Thu, 12 Jul 2012 03:53:57 +0000 (UTC) Received: from ds4.des.no (smtp.des.no [194.63.250.102]) by smtp.des.no (Postfix) with ESMTP id BE6C96040; Thu, 12 Jul 2012 03:53:56 +0000 (UTC) Received: by ds4.des.no (Postfix, from userid 1001) id 5B6F48C4F; Thu, 12 Jul 2012 05:53:56 +0200 (CEST) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: Tim Kientzle References: <25149679-6B99-4FF0-AB8C-90D5A7880F00@kientzle.com> <861ukio8p7.fsf@ds4.des.no> <8E4D9B54-C328-427D-89A1-0CB8E57CD064@kientzle.com> Date: Thu, 12 Jul 2012 05:53:55 +0200 In-Reply-To: <8E4D9B54-C328-427D-89A1-0CB8E57CD064@kientzle.com> (Tim Kientzle's message of "Wed, 11 Jul 2012 18:58:34 -0700") Message-ID: <86r4shhbxo.fsf@ds4.des.no> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.4 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: Ryan Stone , freebsd-arch@freebsd.org Subject: Re: Generating a tarball directly from make installworld X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jul 2012 03:53:58 -0000 Tim Kientzle writes: > ${PROG}: ${OBJS} > ${CC} =E2=80=A6. -o ${.TARGET} ${OBJS} =E2=80=A6 > cp ${.TARGET} ${.TARGET}.stripped > strip ${.TARGET}.stripped objcopy -g This is turning out to be a lot harder than it seemed at first blush... DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-arch@FreeBSD.ORG Fri Jul 13 16:55:26 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 846FD1065670; Fri, 13 Jul 2012 16:55:26 +0000 (UTC) (envelope-from brooks@lor.one-eyed-alien.net) Received: from lor.one-eyed-alien.net (lor.one-eyed-alien.net [69.66.77.232]) by mx1.freebsd.org (Postfix) with ESMTP id D1F4C8FC16; Fri, 13 Jul 2012 16:55:25 +0000 (UTC) Received: from lor.one-eyed-alien.net (localhost [127.0.0.1]) by lor.one-eyed-alien.net (8.14.4/8.14.4) with ESMTP id q6DGs3aI013192; Fri, 13 Jul 2012 11:54:03 -0500 (CDT) (envelope-from brooks@lor.one-eyed-alien.net) Received: (from brooks@localhost) by lor.one-eyed-alien.net (8.14.4/8.14.4/Submit) id q6DGs38v013191; Fri, 13 Jul 2012 11:54:03 -0500 (CDT) (envelope-from brooks) Date: Fri, 13 Jul 2012 11:54:03 -0500 From: Brooks Davis To: Alfred Perlstein Message-ID: <20120713165403.GA89895@lor.one-eyed-alien.net> References: <20120711164111.GJ88966@elvis.mu.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="FL5UXtIhxfXey3p5" Content-Disposition: inline In-Reply-To: <20120711164111.GJ88966@elvis.mu.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Ryan Stone , freebsd-arch@freebsd.org Subject: Re: Generating a tarball directly from make installworld X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Jul 2012 16:55:26 -0000 --FL5UXtIhxfXey3p5 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Jul 11, 2012 at 09:41:11AM -0700, Alfred Perlstein wrote: > * Ryan Stone [120710 20:18] wrote: > > I was playing around with this a couple of months ago and a recent > > thread on -arch prompted me to pick it up again. The idea, for those > > who aren't familiar with it, is to have the base system build > > infrastructure generate tarballs directly. I believe that that the > > current approach is to do a make installworld installkernel > > distribution to an empty directory, and then tar that up. That's not > > an ideal system because in order to have file ownership, flags, etc be > > set correctly, installworld and friends must be run as root. > >=20 >=20 > This is pretty cool, I would suggest going along the route of mtree > + tar as you mentioned. >=20 > If you hacked install(1) to install without special permissions and > then to also make a simple file that has "path/file perms owners", > then later convert that to an mtree file you should be fine. NetBSD's install has a -M option to create said file. http://netbsd.gw.com/cgi-bin/man-cgi?install++NetBSD-current On thing that has been missing in this thread is that we also want to support building images with non-libarchive tools such as makefs. It already supports mtree files in place of directory crawls, but the use of specfiles to override permissions isn't usable without importing NetBSD's mtree or at least a lot of code from it. -- Brooks --FL5UXtIhxfXey3p5 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (FreeBSD) iD8DBQFQAFKrXY6L6fI4GtQRAnlnAJ0R5Ps5CjlD6gS8x+T5MeOoybOEXQCg5gJQ TODfl7cddzaLTS5+inGn5UI= =BSGl -----END PGP SIGNATURE----- --FL5UXtIhxfXey3p5-- From owner-freebsd-arch@FreeBSD.ORG Fri Jul 13 19:46:18 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 719D8106564A; Fri, 13 Jul 2012 19:46:18 +0000 (UTC) (envelope-from bright@elvis.mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 5864C8FC15; Fri, 13 Jul 2012 19:46:18 +0000 (UTC) Received: by elvis.mu.org (Postfix, from userid 1192) id 104A31A3C60; Fri, 13 Jul 2012 12:46:12 -0700 (PDT) Date: Fri, 13 Jul 2012 12:46:11 -0700 From: Alfred Perlstein To: Brooks Davis Message-ID: <20120713194611.GA33444@elvis.mu.org> References: <20120711164111.GJ88966@elvis.mu.org> <20120713165403.GA89895@lor.one-eyed-alien.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20120713165403.GA89895@lor.one-eyed-alien.net> User-Agent: Mutt/1.4.2.3i Cc: Ryan Stone , freebsd-arch@freebsd.org Subject: Re: Generating a tarball directly from make installworld X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Jul 2012 19:46:18 -0000 * Brooks Davis [120713 09:55] wrote: > On Wed, Jul 11, 2012 at 09:41:11AM -0700, Alfred Perlstein wrote: > > * Ryan Stone [120710 20:18] wrote: > > > I was playing around with this a couple of months ago and a recent > > > thread on -arch prompted me to pick it up again. The idea, for those > > > who aren't familiar with it, is to have the base system build > > > infrastructure generate tarballs directly. I believe that that the > > > current approach is to do a make installworld installkernel > > > distribution to an empty directory, and then tar that up. That's not > > > an ideal system because in order to have file ownership, flags, etc be > > > set correctly, installworld and friends must be run as root. > > > > > > > This is pretty cool, I would suggest going along the route of mtree > > + tar as you mentioned. > > > > If you hacked install(1) to install without special permissions and > > then to also make a simple file that has "path/file perms owners", > > then later convert that to an mtree file you should be fine. > > NetBSD's install has a -M option to create said file. > > http://netbsd.gw.com/cgi-bin/man-cgi?install++NetBSD-current > > On thing that has been missing in this thread is that we also want to > support building images with non-libarchive tools such as makefs. It > already supports mtree files in place of directory crawls, but the use > of specfiles to override permissions isn't usable without importing > NetBSD's mtree or at least a lot of code from it. Ok, so obviously we can't go in this direction. :) -- - Alfred Perlstein .- VMOA #5191, 03 vmax, 92 gs500, 85 ch250, 07 zx10 .- FreeBSD committer From owner-freebsd-arch@FreeBSD.ORG Sat Jul 14 18:57:11 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 749621065670 for ; Sat, 14 Jul 2012 18:57:11 +0000 (UTC) (envelope-from launspachontwerp@vhostlin3.jkit.nl) Received: from vhostlin3.jkit.nl (vhostlin3.jkit.nl [83.96.177.125]) by mx1.freebsd.org (Postfix) with ESMTP id 336938FC19 for ; Sat, 14 Jul 2012 18:57:11 +0000 (UTC) Received: by vhostlin3.jkit.nl (Postfix, from userid 10073) id F146E864D93; Sat, 14 Jul 2012 20:25:27 +0200 (CEST) To: freebsd-arch@freebsd.org X-PHP-Originating-Script: 7005:mailer2012.php From: Linda Joe MIME-Version: 1.0 Content-Type: text/plain Content-Transfer-Encoding: 8bit Message-Id: <20120714183437.F146E864D93@vhostlin3.jkit.nl> Date: Sat, 14 Jul 2012 20:25:27 +0200 (CEST) Subject: About Your Pending Funds X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: lindajoe00@gmail.com List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 14 Jul 2012 18:57:11 -0000 I am Engr Linda Joe. A computer scientist with central bank of Nigeria. I am 30 years old, just started work with C.B.N. I came across your file which was marked X and your released disk painted RED, I took time to study it and found out that you have paid VIRTUALLY all fees and certificate but the fund has not been release to you. The most annoying thing is that they cannot tell you the truth that on no account will they ever release the fund to you. Please this is like a Mafia setting in Nigeria; you may not understand it because you are not a Nigerian. The only thing I will need to release this fund is a special HARD DISK we call it HD120 GIG. I will buy two of it, recopy your information, destroy the previous one, and punch the computer to reflect in your bank within 24 bank Trus Plus@};- :x: banking hours. I will clean up the tracer and destroy your file, after which I will run away from Nigeria to meet with you. If you are interested. Do get in touch with me immediately, You should send to me your convenient tell/fax numbers for easy communications and also re confirm your banking details, so that there won't be any mistake, Get back to me as soon as possible. Regards, Engr:Ms Linda Joe