From owner-freebsd-arch@FreeBSD.ORG Sun Jan 11 20:52:10 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 02AFB16A4CE; Sun, 11 Jan 2004 20:52:10 -0800 (PST) Received: from mail.chesapeake.net (chesapeake.net [208.142.252.6]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3ED5F43D2F; Sun, 11 Jan 2004 20:52:06 -0800 (PST) (envelope-from jroberson@chesapeake.net) Received: from localhost (jroberson@localhost) by mail.chesapeake.net (8.11.6/8.11.6) with ESMTP id i0C4q4D14607; Sun, 11 Jan 2004 23:52:04 -0500 (EST) (envelope-from jroberson@chesapeake.net) Date: Sun, 11 Jan 2004 23:52:04 -0500 (EST) From: Jeff Roberson To: Scott Long In-Reply-To: <4000CD54.30801@freebsd.org> Message-ID: <20040111234854.L36463-100000@mail.chesapeake.net> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: arch@freebsd.org Subject: Re: Interrupt API change X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Jan 2004 04:52:10 -0000 On Sat, 10 Jan 2004, Scott Long wrote: > Bruce Evans wrote: > > > > That'a about all it can do. In the shared irq case, there is no > > alternative to calling all the handlers if one of the handlers did > > something, since activity by one handler is unrelated to activity by > > others (except as an optimization that only works in the edge triggered > > case -- devices usually rarely interrupt concurrently so assuming that > > thety never do is usually most efficient). In the non-shared case, > > individual handlers can better decide about interrupt storms in a > > device-specific way. The non-shared case will hopefully be almost all > > cases when APIC support becomes standard on i386's. > > > > This is way too overly optimistic. Interrupt routing is still limited > by things like the number of physical PCI INTx lines. The APIC can't > do anything about devices that share the same physical line. > > MSI will help this, but I suspect that MSI will only be supported on > the higher-end PCI/PCI-X cards, at least until PCI-Express is adopted. > I wouldn't expect the shared PCI INTx line problem to go away for > at least another 5-7 years. > > There is no reason to duplicate interrupt storm heuristics in every > single PCI driver. For now, the change will be essentially a no-op. > However, getting it in will allow us to experiment with it in the > future with ease. I'm not advocating that we break shared interrupt > semantics and use this to short-circuit handlers. > > > > > The first level interrupt handler should call (or schedule) other levels > > as necessary (as in RELENG_4, but not using inefficient scheduling as in > > -current). > > I understand, and that's why I haven't committed to doing it yet. There are still significant optimizations to be done in interrupt scheduling. It seems to me that the filter idea is not a good one if we can make ithread scheduling extremely cheap, which we can. Ideally, it should just be little more than a stack switch. I'd rather see us put effort into this than put effort into defeating the goal of a system which we have not even finished implementing yet. Cheers, Jeff > > > > > Bruce > > > > > _______________________________________________ > freebsd-arch@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" > From owner-freebsd-arch@FreeBSD.ORG Sun Jan 11 20:59:25 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3B57016A4CE for ; Sun, 11 Jan 2004 20:59:25 -0800 (PST) Received: from smtp.mho.com (smtp.mho.net [64.58.4.6]) by mx1.FreeBSD.org (Postfix) with SMTP id 05D9A43D39 for ; Sun, 11 Jan 2004 20:59:24 -0800 (PST) (envelope-from scottl@freebsd.org) Received: (qmail 71292 invoked by uid 1002); 12 Jan 2004 04:59:21 -0000 Received: from unknown (HELO freebsd.org) (64.58.1.252) by smtp.mho.net with SMTP; 12 Jan 2004 04:59:21 -0000 Message-ID: <40022941.6030703@freebsd.org> Date: Sun, 11 Jan 2004 21:57:37 -0700 From: Scott Long User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.5) Gecko/20031103 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Jeff Roberson References: <20040111234854.L36463-100000@mail.chesapeake.net> In-Reply-To: <20040111234854.L36463-100000@mail.chesapeake.net> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit cc: arch@freebsd.org Subject: Re: Interrupt API change X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Jan 2004 04:59:25 -0000 Jeff Roberson wrote: > > There are still significant optimizations to be done in interrupt > scheduling. It seems to me that the filter idea is not a good one if we > can make ithread scheduling extremely cheap, which we can. Ideally, it > should just be little more than a stack switch. I'd rather see us put > effort into this than put effort into defeating the goal of a system which > we have not even finished implementing yet. > > Cheers, > Jeff > I completely agree. However, I don't have the knowledge to do these optimizations, so I'm preparing this as a contingency plan. If you, Peter, Bosko, etc, can put your heads together and come up with a prototype in the next few weeks, that would be awesome. Scott From owner-freebsd-arch@FreeBSD.ORG Tue Jan 13 14:35:04 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 282B716A4CE for ; Tue, 13 Jan 2004 14:35:04 -0800 (PST) Received: from web14809.mail.yahoo.com (web14809.mail.yahoo.com [216.136.224.230]) by mx1.FreeBSD.org (Postfix) with SMTP id 1F18043D91 for ; Tue, 13 Jan 2004 14:34:46 -0800 (PST) (envelope-from rosti_bsd@yahoo.com) Message-ID: <20040113223445.26991.qmail@web14809.mail.yahoo.com> Received: from [192.117.108.59] by web14809.mail.yahoo.com via HTTP; Tue, 13 Jan 2004 14:34:45 PST Date: Tue, 13 Jan 2004 14:34:45 -0800 (PST) From: Rostislav Krasny To: freebsd-arch@freebsd.org In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Subject: Re: IRQ 2 problem X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Jan 2004 22:35:04 -0000 Thank you all for the discussion but recently released FreeBSD 5.2-RELEASE still have this IRQ 2 problem. Could you please tell me when this problem will be hopefully resolved? I think this IRQ 2 problem should be mentioned in the Errata of FreeBSD 5.2-RELEASE, shouldn't it? Thanks __________________________________ Do you Yahoo!? Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes http://hotjobs.sweepstakes.yahoo.com/signingbonus From owner-freebsd-arch@FreeBSD.ORG Tue Jan 13 15:33:08 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 54C1E16A4D5 for ; Tue, 13 Jan 2004 15:33:08 -0800 (PST) Received: from critter.freebsd.dk (critter.freebsd.dk [212.242.86.163]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0492E43D73 for ; Tue, 13 Jan 2004 15:32:40 -0800 (PST) (envelope-from phk@phk.freebsd.dk) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.12.10/8.12.10) with ESMTP id i0DNWbdN012417 for ; Wed, 14 Jan 2004 00:32:37 +0100 (CET) (envelope-from phk@phk.freebsd.dk) To: arch@freebsd.org From: Poul-Henning Kamp Date: Wed, 14 Jan 2004 00:32:37 +0100 Message-ID: <12416.1074036757@critter.freebsd.dk> Subject: About removable disks, mountroot and sw-raid X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Jan 2004 23:33:08 -0000 There has been some discussions about how we handle removable disks and mountroot, raid engines and such stuff. I think it would be a good idea if I dump my thinking to arch@ and we can discuss it here. First I will present the scenarios from which I have analyzed the situation. A: Normal boot. Machine boots kernel, in relatively short time all disks are found. B: Slow boot Machine boots kernel, disk dribble in at a rate of one every 20 seconds as the cabinet powers them up. C: Boot with failed root disk. Machine boots kernel, in relatively short time all disks are found, root disk is not one of them. (This is a strange scenario I know, but it is important for the analysis). D: Machine boots, all raid disks present. E: Machine boots, one raid disk missing. F: Machine running. Operator plugs complete raid-set in, one disk at a time. The solution: ------------- I want to add a counter (protected by a mutex) which the diskdrivers and GEOM will increment while they are configuring devices. That means that as soon as the ata-disk system notices that there _may_ be a disk on a cable, it will increment this counter. If it subsequently determines that there wasn't a disk after all, it decrements by one again. If it finds a disk, it hands it off to GEOM/disk_create(9) before decrementing the counter. GEOM will similarly hold reference counts until all tasting have settled down, so all geom classes have had their chance to do their thing. mount_root will stall on this counter being non-zero, and when it goes to zero, try to open the root dev and fail if it is not found. This solves scenario A. Scenario B is only solvable with outside knowledge. I propose to add a tunable which says either how long time in total or maybe more: useful how long time after the count went to zero before we give up looking for the root dev. This means that the system will "stick around for a while" hoping the missing disk appears, and after the timeout, it will fail. A default timeout of 40 seconds from the last disk appeared sounds like a good shot at a default to me. This solves scenario B. Provided what the user wants for scenario C is for mount_root to fail, we have also solved that. A magic timer configuration of -1 could mean "never", that caters for alternative desires. That solve scenario C. Now about sw-RAID (and mirror, and stripe and ...) In general these methods must collect tributaries until they are satisfied they can run. For non-redundant configs this is trivial: all bits must be present. For redundant methods, the administrator will have to set a policy and I can imagine the following policies: 1. Run when you have all tributaries. 2. Run when you have quorum (ie: one copy of mirror etc) 3. When you have quorum, run if no further tributaries have arrived in N seconds. Again a simple tunable integer can configure this (-1, 0, >0) and maybe for simplicity we should use the same as we use for mountroot. This solves scenario D, E and F I belive. And then the combination example: Boot from a raid root with a missing disk. (lots of intermediate steps left out for clarity) I have configured my system to wait for two minutes for disks. The kernel loads, and disks ad0, ad1, ad2 and ad3 are found in short order. GEOM does its thing and my RAID geom class gets each to taste in turn. It finds out that it has quorum, but is incomplete so the timer applies. It therefore is not finished autoconfiguring and grabs a count on the "block mountroot" counter. After two minutes, the disk has still not arrived, so the RAID geom class starts up in degraded mode, presents the provider to GEOM (which increments the counter since tasting is starting) and then drops its own count. The tasting of the provider happens in GEOM, possibly doing partitioning etc. Once it stops, the count goes to zero and mountroot gets to try to open the root device. If it cannot open the root device, it will retry for two minutes, and then give up and go to the root-device prompt. If I then go to scenario F, and then go take a pee after plugging four of my five disks in, then I may curse the two minute timer, but if I'm that advanced, I should have reset the counter. Or should we have two counter tunables ? one until root has been mounted and another afterwards ? No big deal, we can do that. Does all this make sense to people ? Poul-Henning -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-arch@FreeBSD.ORG Tue Jan 13 19:18:03 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 518B716A4CE for ; Tue, 13 Jan 2004 19:18:03 -0800 (PST) Received: from exchhz01.viatech.com.cn (ip-40-162-97-218.anlai.com [218.97.162.40]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7A22643D1D for ; Tue, 13 Jan 2004 19:17:59 -0800 (PST) (envelope-from davidxu@freebsd.org) Received: from freebsd.org (DAVIDWNT [10.4.1.99]) by exchhz01.viatech.com.cn with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id CY5M1L3V; Wed, 14 Jan 2004 10:55:42 +0800 Message-ID: <4004B4D0.907@freebsd.org> Date: Wed, 14 Jan 2004 11:17:36 +0800 From: David Xu User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.5b) Gecko/20030723 Thunderbird/0.1 X-Accept-Language: en-us, en MIME-Version: 1.0 To: arch@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: ptrace and thread X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Jan 2004 03:18:03 -0000 I am current working on debug support for KSE thread program, however I found ptrace interface is not thread-aware, in a threaded program, I need to get/set registers set for individual threads, current ptrace can not support that features, there are two ways to support these requirements: 1. keep current ptrace interface, add a command for example PT_SETDTHREAD to set current thread for debug, and subsequent request for example PT_SETREGS and PT_GETREGS will work on the thread, for single thread process, the default current thread is always the first thread in the process, this way we needn't change legacy debugger code. 2. introduce a second ptrace syscall, and accept a new parameter tid (thread id), the PT_SETREGS and PT_GETREGS will use the tid to operate on corresponding thread. For first method, I have a patch there: http://people.freebsd.org/~davidxu/kse/ptrace.diff The patch also includes some bits to support KSE debug, not just for pure 1:1 threading. David Xu From owner-freebsd-arch@FreeBSD.ORG Tue Jan 13 19:48:56 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 937A016A4CE for ; Tue, 13 Jan 2004 19:48:56 -0800 (PST) Received: from mtaw6.prodigy.net (mtaw6.prodigy.net [64.164.98.56]) by mx1.FreeBSD.org (Postfix) with ESMTP id 21F4643D54 for ; Tue, 13 Jan 2004 19:48:54 -0800 (PST) (envelope-from david.brinegar@acm.org) Received: from hush.corte.roble (adsl-64-161-24-215.dsl.sntc01.pacbell.net [64.161.24.215]) by mtaw6.prodigy.net (8.12.10/8.12.10) with ESMTP id i0E3m7FA000312; Tue, 13 Jan 2004 19:48:07 -0800 (PST) Received: by hush.corte.roble (Postfix, from userid 1000) id B5FF43FE; Tue, 13 Jan 2004 19:48:53 -0800 (PST) Date: Tue, 13 Jan 2004 19:48:53 -0800 From: David Brinegar To: arch@freebsd.org Message-ID: <20040114034853.GD2701@mail.brinegar-computing.com> References: <12416.1074036757@critter.freebsd.dk> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <12416.1074036757@critter.freebsd.dk> User-Agent: Mutt/1.4.1i cc: Poul-Henning Kamp Subject: Re: About removable disks, mountroot and sw-raid X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Jan 2004 03:48:56 -0000 Poul-Henning Kamp wrote: > B: Slow boot > Machine boots kernel, disk dribble in at a rate of one > every 20 seconds as the cabinet powers them up. Power dribble reminds me of usb disks that are slow to power up completely. The system detects the disk then outraces it, getting a failed attempt to query device size because the disk needs another second or two to come all the way up. It would be nice to be able to tune the stall in the disk driver before declaring the device not ready. > Scenario B is only solvable with outside knowledge. I propose to > add a tunable which says either how long time in total or maybe > more: useful how long time after the count went to zero before we > give up looking for the root dev. > > This means that the system will "stick around for a while" hoping > the missing disk appears, and after the timeout, it will fail. A tunable stall in the disk driver would do the same thing, without limiting it to boot. It would be nice if this were not limited to tuning boot. Say you plug in the cabinet after the system is booted, then you still need the disk driver to wait 20 seconds before declaring the first disk dead on arrival. That it is tunable means you could boot fast, zero stall so the disk driver quickly bails on the cabinet that isn't there, then tune it to 20 seconds when the cabinet is plugged in. The other issue is how to handle the operator removing the disk and plugging it back in during the stall. The stall should restart in any case, which may be easier to deal with in the driver than in a search for root dev. I guess your proposal handles that if the driver dings the count when the device is replugged, so there's not much difference there. > A default timeout of 40 seconds from the last disk appeared > sounds like a good shot at a default to me. And if it were a tunable stall in the driver it should default to zero, meaning don't add anything to the driver's default stall. -- David Brinegar http://brinegar-computing.com From owner-freebsd-arch@FreeBSD.ORG Tue Jan 13 21:31:53 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8CCA616A4CE for ; Tue, 13 Jan 2004 21:31:53 -0800 (PST) Received: from kientzle.com (h-66-166-149-50.SNVACAID.covad.net [66.166.149.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id AA61143D2D for ; Tue, 13 Jan 2004 21:31:51 -0800 (PST) (envelope-from kientzle@acm.org) Received: from acm.org ([66.166.149.54]) by kientzle.com (8.12.9/8.12.9) with ESMTP id i0E5VpkX071805 for ; Tue, 13 Jan 2004 21:31:51 -0800 (PST) (envelope-from kientzle@acm.org) Message-ID: <4004D445.7020205@acm.org> Date: Tue, 13 Jan 2004 21:31:49 -0800 From: Tim Kientzle User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.4) Gecko/20031006 X-Accept-Language: en-us, en MIME-Version: 1.0 To: freebsd-arch@freebsd.org Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Subject: Request for Comments: libarchive, bsdtar X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: kientzle@acm.org List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Jan 2004 05:31:53 -0000 Request for Comments: libarchive, bsdtar PROPOSAL Add "libarchive" to the tree, prepare to change the system tar command to "bsdtar" once it is sufficiently stable. DETAILS Step 1: Commit "libarchive" Step 2: Rename the current system "tar" command to "gnutar", also linked as "tar" Step 3: Once it is more complete (a couple of months?), commit "bsdtar" under that name. Provide a switch for linking "tar" to "bsdtar", get additional feedback and testing from the community. Step 4: After a suitable transition, have "tar" default to "bsdtar." Step 5: Leave "gnutar" in the tree for ~6 months before removing it. It will, of course, be available as a port indefinitely for those who require its unique features. Note that the first two steps will have no visible impact on end users. The third step will only impact those who choose to run bsdtar as the system tar. The fourth step will impact many 'tar' users, who will have the option of adapting to bsdtar, switching the system tar back to gnutar or continuing to use gnutar under a different name. Even after the entire process is complete, users will still have the option of using gnutar from ports. LIBARCHIVE BACKGROUND As many of you know, I've been working on a project to overhaul the pkg tools. Among many other things, this requires a library that can read/write tar archives. This avoids the significant overhead imposed from forking a separate tar program. The first fruit of my work is "libarchive," currently on display at http://people.freebsd.org/~kientzle/libarchive/ That page also includes links to the rough manpages for the library as well as a description of the history and goals of this work. I'm very pleased with the library at this point. I would like to commit it to the tree and get other people using it in order to shake out any remaining problems. Of course, to really use it requires some client programs. BSDTAR BACKGROUND The bsdtar program was originally written as a simple test harness for libarchive. It has grown a bit and is now very nearly SUSv2-compliant, plus a lot of additional features. (For reference, gnutar is not SUSv2 compliant and does not support the "pax interchange format," an extended tar format that appears in SUSv3 and is widely used on other systems.) To date, the bulk of the work on "bsdtar" has really involved fine-tuning the libarchive library, work that will eventually benefit the package tools and other libarchive clients. However, I've gotten to the point where there are a few significant features (multi-volume tape handling, rmt support) that are not required by the library. Before I invest more of my scarce time on refining bsdtar, I'd like to know whether there is real interest in replacing gnutar with bsdtar. Feedback and comments greatly appreciated, Tim Kientzle kientzle@ From owner-freebsd-arch@FreeBSD.ORG Wed Jan 14 04:33:22 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 038D716A4CE; Wed, 14 Jan 2004 04:33:22 -0800 (PST) Received: from mail.qubesoft.com (gate.qubesoft.com [217.169.36.34]) by mx1.FreeBSD.org (Postfix) with ESMTP id 74C5A43D53; Wed, 14 Jan 2004 04:33:18 -0800 (PST) (envelope-from dfr@nlsystems.com) Received: from bluebottle.qubesoft.com (bluebottle.qubesoft.com [192.168.1.2]) by mail.qubesoft.com (8.12.9/8.12.9) with ESMTP id i0ECXHHF003384; Wed, 14 Jan 2004 12:33:17 GMT (envelope-from dfr@nlsystems.com) Received: from builder02.qubesoft.com (builder02.qubesoft.com [192.168.1.8]) i0ECXHRb002943; Wed, 14 Jan 2004 12:33:17 GMT (envelope-from dfr@nlsystems.com) From: Doug Rabson To: David Xu In-Reply-To: <4004B4D0.907@freebsd.org> References: <4004B4D0.907@freebsd.org> Content-Type: text/plain Message-Id: <1074083597.5917.5.camel@builder02.qubesoft.com> Mime-Version: 1.0 X-Mailer: Ximian Evolution 1.4.0 Date: 14 Jan 2004 12:33:17 +0000 Content-Transfer-Encoding: 7bit cc: arch@freebsd.org Subject: Re: ptrace and thread X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Jan 2004 12:33:22 -0000 On Wed, 2004-01-14 at 03:17, David Xu wrote: > I am current working on debug support for KSE thread program, however I > found > ptrace interface is not thread-aware, in a threaded program, I need to > get/set registers > set for individual threads, current ptrace can not support that > features, there are two > ways to support these requirements: > > 1. keep current ptrace interface, add a command for example > PT_SETDTHREAD to > set current thread for debug, and subsequent request for example > PT_SETREGS and > PT_GETREGS will work on the thread, for single thread process, the > default current > thread is always the first thread in the process, this way we needn't > change legacy debugger > code. > > 2. introduce a second ptrace syscall, and accept a new parameter tid > (thread id), > the PT_SETREGS and PT_GETREGS will use the tid to operate on > corresponding > thread. As far as option 2 goes, I think the HP ttrace syscall might be a useful model to follow. You can read the manpage at: http://docs.hp.com/cgi-bin/onlinedocs.py?mpn=B2355-60103&service=hpux&path=00/13/1384&title=HP-UX%20Reference%20%2811i%20v2%29 From owner-freebsd-arch@FreeBSD.ORG Wed Jan 14 10:07:01 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5151816A4CE for ; Wed, 14 Jan 2004 10:07:01 -0800 (PST) Received: from arthur.nitro.dk (port324.ds1-khk.adsl.cybercity.dk [212.242.113.79]) by mx1.FreeBSD.org (Postfix) with ESMTP id 03C8543D7E for ; Wed, 14 Jan 2004 10:06:50 -0800 (PST) (envelope-from simon@arthur.nitro.dk) Received: by arthur.nitro.dk (Postfix, from userid 3000) id 950491142B; Wed, 14 Jan 2004 19:06:48 +0100 (CET) Date: Wed, 14 Jan 2004 19:06:48 +0100 From: "Simon L. Nielsen" To: Tim Kientzle Message-ID: <20040114180647.GA682@arthur.nitro.dk> References: <4004D445.7020205@acm.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="y0ulUmNC+osPPQO6" Content-Disposition: inline In-Reply-To: <4004D445.7020205@acm.org> User-Agent: Mutt/1.5.5.1i cc: freebsd-arch@freebsd.org Subject: Re: Request for Comments: libarchive, bsdtar X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Jan 2004 18:07:01 -0000 --y0ulUmNC+osPPQO6 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2004.01.13 21:31:49 -0800, Tim Kientzle wrote: > Step 2: Rename the current system "tar" command to > "gnutar", also linked as "tar" Isn't "gtar" the normal name for gnu tar ? (not that it matters much to me). > The first fruit of my work is "libarchive," currently > on display at > http://people.freebsd.org/~kientzle/libarchive/ I downloaded libarchive-2004-01-14.tgz but it seems to be damaged. It's only 10KB (the older version is 90KB), and fails on extract (with gnu tar) saying: tar: Unexpected EOF in archive > Before I invest more > of my scarce time on refining bsdtar, I'd like to know whether > there is real interest in replacing gnutar with bsdtar. >=20 > Feedback and comments greatly appreciated, While I have no cruse against GNU tools, I think it would really nice to replace GNU tar with a BSD licensed one. --=20 Simon L. Nielsen FreeBSD Documentation Team --y0ulUmNC+osPPQO6 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.3 (FreeBSD) iD8DBQFABYU2h9pcDSc1mlERAoyUAJwLG/wkNFe4kgdEKI+k42llUFE2hQCcCgLI G/yY5PAuUtGVcE9P83PthzE= =K3Up -----END PGP SIGNATURE----- --y0ulUmNC+osPPQO6-- From owner-freebsd-arch@FreeBSD.ORG Wed Jan 14 10:37:00 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2988016A4CE; Wed, 14 Jan 2004 10:37:00 -0800 (PST) Received: from tatiana.utanet.at (tatiana.utanet.at [213.90.36.46]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0C46443D69; Wed, 14 Jan 2004 10:36:57 -0800 (PST) (envelope-from josef@daemon.li) Received: from pam.utanet.at ([213.90.36.6]) by tatiana.utanet.at with esmtp (Exim 4.12) id 1Agpst-0000PM-00; Wed, 14 Jan 2004 19:36:55 +0100 Received: from dsl-244-254.utaonline.at ([212.152.244.254] helo=jenny.daemon.li) by pam.utanet.at with esmtp (Exim 4.12) id 1Agpss-0002Nm-00; Wed, 14 Jan 2004 19:36:54 +0100 Received: by jenny.daemon.li (Postfix, from userid 1005) id D4DBB10FF; Wed, 14 Jan 2004 19:37:10 +0100 (CET) Date: Wed, 14 Jan 2004 19:37:10 +0100 From: Josef El-Rayes To: "Simon L. Nielsen" Message-ID: <20040114183710.GA463@jenny.daemon.li> References: <4004D445.7020205@acm.org> <20040114180647.GA682@arthur.nitro.dk> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="3MwIy2ne0vdjdPXF" Content-Disposition: inline In-Reply-To: <20040114180647.GA682@arthur.nitro.dk> User-Agent: Mutt/1.4.1i Reply-Path: josef@daemon.li X-Operating-System: FreeBSD 4.9-STABLE cc: freebsd-arch@freebsd.org cc: Tim Kientzle Subject: Re: Request for Comments: libarchive, bsdtar X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: josef@daemon.li List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Jan 2004 18:37:00 -0000 --3MwIy2ne0vdjdPXF Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable "Simon L. Nielsen" wrote: > > Before I invest more > > of my scarce time on refining bsdtar, I'd like to know whether > > there is real interest in replacing gnutar with bsdtar. > >=20 > > Feedback and comments greatly appreciated, >=20 > While I have no cruse against GNU tools, I think it would really nice to > replace GNU tar with a BSD licensed one. i also think it is good to replace gnu tools with bsd equivalents. -josef --3MwIy2ne0vdjdPXF Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.2 (FreeBSD) iQEVAwUBQAWMVlnFItmnnbU8AQJc3ggAqlBSnbvAzy58F1R7jquRvP/JeYLFDhO5 uMYeycYJoAiGcoVV3QgDdBFisD/BUkvUwKKyBjxKLePbX1rx0L9L7LMTLUY9lQqx yAaQaskkNsyT2uNlUTNoIkAgnFGd3/S2uieohgyhF+bLsWyejHg11y7B9bCf9IW6 fjprtk87P0ubxfcOj5NS2J/QQvt4c2j5IApSaTerj8539hWXPpOmgYpb1VhcNWc9 omK5e/MKsIeV7r3HJZPb8dTObEZjzO+PyBibjd9PET61qA0N9+mRGUEIHLtv3Ld0 OnR/HSIVjzxfXjmJF6yVTuR+/Q50GFj64vHFSEWx7QVbelGkpxJbXw== =V75s -----END PGP SIGNATURE----- --3MwIy2ne0vdjdPXF-- From owner-freebsd-arch@FreeBSD.ORG Wed Jan 14 11:03:52 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4189516A4CE for ; Wed, 14 Jan 2004 11:03:52 -0800 (PST) Received: from carver.gumbysoft.com (carver.gumbysoft.com [66.220.23.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id EE22E43D48 for ; Wed, 14 Jan 2004 11:03:50 -0800 (PST) (envelope-from dwhite@gumbysoft.com) Received: by carver.gumbysoft.com (Postfix, from userid 1000) id DD97172DBF; Wed, 14 Jan 2004 11:03:50 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by carver.gumbysoft.com (Postfix) with ESMTP id DAE6472DB5; Wed, 14 Jan 2004 11:03:50 -0800 (PST) Date: Wed, 14 Jan 2004 11:03:50 -0800 (PST) From: Doug White To: David Brinegar In-Reply-To: <20040114034853.GD2701@mail.brinegar-computing.com> Message-ID: <20040114105700.H73725@carver.gumbysoft.com> References: <12416.1074036757@critter.freebsd.dk> <20040114034853.GD2701@mail.brinegar-computing.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: arch@freebsd.org cc: Poul-Henning Kamp Subject: Re: About removable disks, mountroot and sw-raid X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Jan 2004 19:03:52 -0000 On Tue, 13 Jan 2004, David Brinegar wrote: > Poul-Henning Kamp wrote: > > B: Slow boot > > Machine boots kernel, disk dribble in at a rate of one > > every 20 seconds as the cabinet powers them up. > > Power dribble reminds me of usb disks that are slow to power up > completely. The system detects the disk then outraces it, getting a > failed attempt to query device size because the disk needs another > second or two to come all the way up. It would be nice to be able > to tune the stall in the disk driver before declaring the device not > ready. Good point about USB. In the SCSI case, every cabinet I've ever seen sets the disks to start on START UNIT and the SCSI BIOS will send it and wait until all the disks have spun up (or failed to do so). If hardware RAID is involved and then this is totally irrelevant. It would be nice to make the wait tunable since most conditions won't require it and there's no sense in delaying everyone's boot for a few bizarre disk setups. -- Doug White | FreeBSD: The Power to Serve dwhite@gumbysoft.com | www.FreeBSD.org From owner-freebsd-arch@FreeBSD.ORG Wed Jan 14 11:05:47 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BE62E16A4CE for ; Wed, 14 Jan 2004 11:05:47 -0800 (PST) Received: from critter.freebsd.dk (critter.freebsd.dk [212.242.86.163]) by mx1.FreeBSD.org (Postfix) with ESMTP id 581E943D1F for ; Wed, 14 Jan 2004 11:05:46 -0800 (PST) (envelope-from phk@phk.freebsd.dk) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.12.10/8.12.10) with ESMTP id i0EJ5bdN027184; Wed, 14 Jan 2004 20:05:37 +0100 (CET) (envelope-from phk@phk.freebsd.dk) To: Doug White From: "Poul-Henning Kamp" In-Reply-To: Your message of "Wed, 14 Jan 2004 11:03:50 PST." <20040114105700.H73725@carver.gumbysoft.com> Date: Wed, 14 Jan 2004 20:05:37 +0100 Message-ID: <27183.1074107137@critter.freebsd.dk> cc: arch@freebsd.org cc: David Brinegar Subject: Re: About removable disks, mountroot and sw-raid X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Jan 2004 19:05:47 -0000 In message <20040114105700.H73725@carver.gumbysoft.com>, Doug White writes: >It would be nice to make the wait tunable since most conditions won't >require it and there's no sense in delaying everyone's boot for a few >bizarre disk setups. You boot would not be delayed in the normal case. The delays only apply to how long time we want for an expected disk to show up. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-arch@FreeBSD.ORG Wed Jan 14 11:23:12 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 683A416A4CE; Wed, 14 Jan 2004 11:23:12 -0800 (PST) Received: from kientzle.com (h-66-166-149-50.SNVACAID.covad.net [66.166.149.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6CA6243D54; Wed, 14 Jan 2004 11:23:10 -0800 (PST) (envelope-from kientzle@acm.org) Received: from acm.org ([66.166.149.54]) by kientzle.com (8.12.9/8.12.9) with ESMTP id i0EJN9kX074328; Wed, 14 Jan 2004 11:23:10 -0800 (PST) (envelope-from kientzle@acm.org) Message-ID: <4005971D.1060803@acm.org> Date: Wed, 14 Jan 2004 11:23:09 -0800 From: Tim Kientzle User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.4) Gecko/20031006 X-Accept-Language: en-us, en MIME-Version: 1.0 To: "Simon L. Nielsen" References: <4004D445.7020205@acm.org> <20040114180647.GA682@arthur.nitro.dk> In-Reply-To: <20040114180647.GA682@arthur.nitro.dk> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit cc: freebsd-arch@FreeBSD.org Subject: Re: Request for Comments: libarchive, bsdtar X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: kientzle@acm.org List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Jan 2004 19:23:12 -0000 Simon L. Nielsen wrote: > On 2004.01.13 21:31:49 -0800, Tim Kientzle wrote: > >> Step 2: Rename the current system "tar" command to >> "gnutar", also linked as "tar" > > Isn't "gtar" the normal name for gnu tar ? I've seen both. "gtar" seems to be the name used in the current port, so I suppose it would make sense to use that name instead of "gnutar". > I downloaded libarchive-2004-01-14.tgz but it seems to be damaged. Pointy hat to me. I *DID* mention that it needs more testing, didn't I? ;-) Should be fixed now. Tim From owner-freebsd-arch@FreeBSD.ORG Wed Jan 14 11:26:31 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 409E316A4CE for ; Wed, 14 Jan 2004 11:26:31 -0800 (PST) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id C978143D1D for ; Wed, 14 Jan 2004 11:26:29 -0800 (PST) (envelope-from robert@fledge.watson.org) Received: from fledge.watson.org (localhost [127.0.0.1]) by fledge.watson.org (8.12.10/8.12.10) with ESMTP id i0EJOlUd050812; Wed, 14 Jan 2004 14:24:47 -0500 (EST) (envelope-from robert@fledge.watson.org) Received: from localhost (robert@localhost)i0EJOlFT050809; Wed, 14 Jan 2004 14:24:47 -0500 (EST) (envelope-from robert@fledge.watson.org) Date: Wed, 14 Jan 2004 14:24:46 -0500 (EST) From: Robert Watson X-Sender: robert@fledge.watson.org To: Tim Kientzle In-Reply-To: <4004D445.7020205@acm.org> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-arch@freebsd.org Subject: Re: Request for Comments: libarchive, bsdtar X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Jan 2004 19:26:31 -0000 On Tue, 13 Jan 2004, Tim Kientzle wrote: ... All this generally sounds good. > LIBARCHIVE BACKGROUND > > As many of you know, I've been working on a project to overhaul the pkg > tools. Among many other things, this requires a library that can > read/write tar archives. This avoids the significant overhead imposed > from forking a separate tar program. If you become a bored person requiring entertainment, it might be quite interesting to create a read-only tarfs for use as a root file system loaded in an md device. While there's a lot more to it than this, one of the more irritating things about our current release build is that it requires privilege so that it can chroot(), but also so it can manage md devices and file system images. Just being able to use a tarball instead of a UFS image would go a long way, although presumably require changes to our loader as well. For work with diskless systems and network booting, I'd much rather stick a tarball on an NFS server than create UFS images. I know NetBSD has a neat tool to create file systems from userspace without privilege, but my understanding is that it has to pull in a lot of code from the kernel in fairly messy ways. Since tar files are a well supported portable format... :-) Robert N M Watson FreeBSD Core Team, TrustedBSD Projects robert@fledge.watson.org Senior Research Scientist, McAfee Research From owner-freebsd-arch@FreeBSD.ORG Wed Jan 14 11:52:38 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5CF2F16A4CF for ; Wed, 14 Jan 2004 11:52:38 -0800 (PST) Received: from f4.mail.ru (f4.mail.ru [194.67.57.34]) by mx1.FreeBSD.org (Postfix) with ESMTP id F294443D5C for ; Wed, 14 Jan 2004 11:52:36 -0800 (PST) (envelope-from vladimir-dozen@mail.ru) Received: from [213.170.97.162] (port=1367 helo=mail.ru) by f4.mail.ru with esmtp id 1Agr46-000D1A-00 for freebsd-arch@freebsd.org; Wed, 14 Jan 2004 22:52:35 +0300 Message-ID: <40059DFB.9000904@mail.ru> Date: Wed, 14 Jan 2004 14:52:27 -0500 From: Vladimir Dozen User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.4) Gecko/20030624 X-Accept-Language: ru, en-us, en MIME-Version: 1.0 Cc: freebsd-arch@freebsd.org References: In-Reply-To: Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected Subject: Re: Request for Comments: libarchive, bsdtar X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Jan 2004 19:52:38 -0000 ehlo. > If you become a bored person requiring entertainment, it might be quite > interesting to create a read-only tarfs for use as a root file system > loaded in an md device. I'm just curious: is it possible to write a GEOM class to mount a given tar as a FS in R/O mode (using libarchive, I mean)? dozen From owner-freebsd-arch@FreeBSD.ORG Wed Jan 14 11:52:54 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3CF0F16A4CE; Wed, 14 Jan 2004 11:52:54 -0800 (PST) Received: from kientzle.com (h-66-166-149-50.SNVACAID.covad.net [66.166.149.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id ADC6D43D39; Wed, 14 Jan 2004 11:52:52 -0800 (PST) (envelope-from kientzle@acm.org) Received: from acm.org ([66.166.149.54]) by kientzle.com (8.12.9/8.12.9) with ESMTP id i0EJqqkX074485; Wed, 14 Jan 2004 11:52:52 -0800 (PST) (envelope-from kientzle@acm.org) Message-ID: <40059E13.2020609@acm.org> Date: Wed, 14 Jan 2004 11:52:51 -0800 From: Tim Kientzle User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.4) Gecko/20031006 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Robert Watson References: In-Reply-To: Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit cc: freebsd-arch@freebsd.org Subject: Re: Request for Comments: libarchive, bsdtar X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: kientzle@acm.org List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Jan 2004 19:52:54 -0000 Robert Watson wrote: > On Tue, 13 Jan 2004, Tim Kientzle wrote: > >>I've been working on ... a library that can read/write tar archives. > > ... a read-only tarfs for use as a root file system ... > ... our current release build ... requires privilege so that it can ... manage md > devices and file system images. Just being able to use a tarball instead > of a UFS image would go a long way... > > I know NetBSD has a neat tool to create file systems from userspace > without privilege ... An interesting idea. A couple of relevant observations: 1) tar/gzip archives are streaming archives. There is no simple means for random access. Using a tar/gzip file as a filesystem would require reading through the archive to find each file. With careful design, this might not be a performance problem, but it does limit the applications somewhat. 2) Uncompressed tar archives can provide random access if you're willing to scan the archive and build an in-memory directory when you mount it. This is pretty easy to do, especially if you restrict yourself to basic 'ustar' archives. 3) An alternative would be to build a tool for creating/manipulating a UFS image, along the lines of mtools or mkisofs. Just the basics: an md image doesn't need UFS2 extended attributes or softupdates, so it shouldn't be that hard. To be honest, trying to share code with the kernel may be more work than it's worth. Such a tool may also be able to optimize the size of the resulting FS image (stat everything, calculate the FS layout, then have a second pass that actually writes the resulting FS). Sounds like a good project for an enthusiastic college student. I'm certain someone out there is taking a filesystems class and would be interested in taking a crack at this. Point them at some good UFS documentation and see what happens. 4) Of course, mkisofs/mkhybrid exists. Why not use cdroot? It may seem a bit peculiar to load a CDROM image into RAM from a series of floppy disks, but all of the necessary tools and support code already exist. Tim From owner-freebsd-arch@FreeBSD.ORG Wed Jan 14 12:01:22 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id EBAB816A4CE for ; Wed, 14 Jan 2004 12:01:22 -0800 (PST) Received: from mailhub.fokus.fraunhofer.de (mailhub.fokus.fraunhofer.de [193.174.154.14]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0B99943D68 for ; Wed, 14 Jan 2004 12:01:18 -0800 (PST) (envelope-from brandt@fokus.fraunhofer.de) Received: from beagle (beagle [193.175.132.100])i0EK1EL19206; Wed, 14 Jan 2004 21:01:14 +0100 (MET) Date: Wed, 14 Jan 2004 21:01:14 +0100 (CET) From: Harti Brandt To: Tim Kientzle In-Reply-To: <40059E13.2020609@acm.org> Message-ID: <20040114205807.U15194@beagle.fokus.fraunhofer.de> References: <40059E13.2020609@acm.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-arch@freebsd.org Subject: Re: Request for Comments: libarchive, bsdtar X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Jan 2004 20:01:23 -0000 Tim, you should probably co-ordinate the implementation of things like file flags and other stuff with Joerg Schilling so that we don't get non-interoperable tars. The same holds for multi-volume archives which seems to be a real challenge to get right. He has talked to me several hours explaining the problems and possible solutions. It make also make sense to co-ordinate rmt stuff with him (Sun's going to use his rmt as far as I know). harti -- harti brandt, http://www.fokus.fraunhofer.de/research/cc/cats/employees/hartmut.brandt/private brandt@fokus.fraunhofer.de, harti@freebsd.org From owner-freebsd-arch@FreeBSD.ORG Wed Jan 14 12:15:47 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 90C4F16A4CE for ; Wed, 14 Jan 2004 12:15:47 -0800 (PST) Received: from harmony.village.org (rover.bsdimp.com [204.144.255.66]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2260C43D45 for ; Wed, 14 Jan 2004 12:15:43 -0800 (PST) (envelope-from imp@bsdimp.com) Received: from localhost (warner@rover2.village.org [10.0.0.1]) by harmony.village.org (8.12.10/8.12.9) with ESMTP id i0EKFPET092274; Wed, 14 Jan 2004 13:15:28 -0700 (MST) (envelope-from imp@bsdimp.com) Date: Wed, 14 Jan 2004 13:15:23 -0700 (MST) Message-Id: <20040114.131523.03269738.imp@bsdimp.com> To: phk@phk.freebsd.dk From: "M. Warner Losh" In-Reply-To: <27183.1074107137@critter.freebsd.dk> References: <20040114105700.H73725@carver.gumbysoft.com> <27183.1074107137@critter.freebsd.dk> X-Mailer: Mew version 3.3 on Emacs 21.3 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit cc: arch@freebsd.org cc: david.brinegar@acm.org Subject: Re: About removable disks, mountroot and sw-raid X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Jan 2004 20:15:47 -0000 In message: <27183.1074107137@critter.freebsd.dk> "Poul-Henning Kamp" writes: : In message <20040114105700.H73725@carver.gumbysoft.com>, Doug White writes: : : >It would be nice to make the wait tunable since most conditions won't : >require it and there's no sense in delaying everyone's boot for a few : >bizarre disk setups. : : You boot would not be delayed in the normal case. The delays only apply : to how long time we want for an expected disk to show up. What are there plans to allow disks to arrive after we give up? Obviously if we give up, we'll be at the root> prompt, but if a disk is a little late, it would be nice to be able to say 'da0s1a' and have it find this device, rather than hard wiring the available devices at the time that the first root> prompt is given. Warner From owner-freebsd-arch@FreeBSD.ORG Wed Jan 14 12:17:43 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D270B16A4CE; Wed, 14 Jan 2004 12:17:43 -0800 (PST) Received: from harmony.village.org (rover.bsdimp.com [204.144.255.66]) by mx1.FreeBSD.org (Postfix) with ESMTP id DDDB343D5C; Wed, 14 Jan 2004 12:17:22 -0800 (PST) (envelope-from imp@bsdimp.com) Received: from localhost (warner@rover2.village.org [10.0.0.1]) by harmony.village.org (8.12.10/8.12.9) with ESMTP id i0EKH7ET092332; Wed, 14 Jan 2004 13:17:08 -0700 (MST) (envelope-from imp@bsdimp.com) Date: Wed, 14 Jan 2004 13:17:05 -0700 (MST) Message-Id: <20040114.131705.93652359.imp@bsdimp.com> To: rwatson@freebsd.org From: "M. Warner Losh" In-Reply-To: References: <4004D445.7020205@acm.org> X-Mailer: Mew version 3.3 on Emacs 21.3 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit cc: freebsd-arch@freebsd.org cc: kientzle@acm.org Subject: Re: Request for Comments: libarchive, bsdtar X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Jan 2004 20:17:44 -0000 In message: Robert Watson writes: : I know NetBSD has a neat tool to create file systems from userspace : without privilege, but my understanding is that it has to pull in a lot of : code from the kernel in fairly messy ways. Since tar files are a well : supported portable format... :-) The problem then reduces to needing to be able to create the tar file with ownership/permissions different than is in the unpriv'd build tree. Generation of this list, as well as getting tar/whatever to honor it are interesting problems. Warner From owner-freebsd-arch@FreeBSD.ORG Wed Jan 14 12:24:00 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E4E8416A4D4 for ; Wed, 14 Jan 2004 12:24:00 -0800 (PST) Received: from kientzle.com (h-66-166-149-50.SNVACAID.covad.net [66.166.149.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id DD97343DA4 for ; Wed, 14 Jan 2004 12:23:11 -0800 (PST) (envelope-from kientzle@acm.org) Received: from acm.org ([66.166.149.54]) by kientzle.com (8.12.9/8.12.9) with ESMTP id i0EKMLkX074625; Wed, 14 Jan 2004 12:22:21 -0800 (PST) (envelope-from kientzle@acm.org) Message-ID: <4005A4FC.4000103@acm.org> Date: Wed, 14 Jan 2004 12:22:20 -0800 From: Tim Kientzle User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.4) Gecko/20031006 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Harti Brandt References: <40059E13.2020609@acm.org> <20040114205807.U15194@beagle.fokus.fraunhofer.de> In-Reply-To: <20040114205807.U15194@beagle.fokus.fraunhofer.de> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit cc: freebsd-arch@freebsd.org Subject: Re: Request for Comments: libarchive, bsdtar X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: kientzle@acm.org List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Jan 2004 20:24:01 -0000 Harti Brandt wrote: > you should probably co-ordinate the implementation of things like > file flags and other stuff with Joerg Schilling ... Joerg and I have been trading notes on these issues. My intention is to have a high degree of interoperability between star and libarchive/bsdtar. Tim From owner-freebsd-arch@FreeBSD.ORG Wed Jan 14 12:31:00 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9FF7116A4CE for ; Wed, 14 Jan 2004 12:31:00 -0800 (PST) Received: from critter.freebsd.dk (critter.freebsd.dk [212.242.86.163]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1783D43D5C for ; Wed, 14 Jan 2004 12:30:17 -0800 (PST) (envelope-from phk@phk.freebsd.dk) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.12.10/8.12.10) with ESMTP id i0EKTKdN028662; Wed, 14 Jan 2004 21:29:20 +0100 (CET) (envelope-from phk@phk.freebsd.dk) To: "M. Warner Losh" From: "Poul-Henning Kamp" In-Reply-To: Your message of "Wed, 14 Jan 2004 13:15:23 MST." <20040114.131523.03269738.imp@bsdimp.com> Date: Wed, 14 Jan 2004 21:29:20 +0100 Message-ID: <28661.1074112160@critter.freebsd.dk> cc: arch@freebsd.org cc: david.brinegar@acm.org Subject: Re: About removable disks, mountroot and sw-raid X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Jan 2004 20:31:00 -0000 In message <20040114.131523.03269738.imp@bsdimp.com>, "M. Warner Losh" writes: >In message: <27183.1074107137@critter.freebsd.dk> > "Poul-Henning Kamp" writes: >: In message <20040114105700.H73725@carver.gumbysoft.com>, Doug White writes: >: >: >It would be nice to make the wait tunable since most conditions won't >: >require it and there's no sense in delaying everyone's boot for a few >: >bizarre disk setups. >: >: You boot would not be delayed in the normal case. The delays only apply >: to how long time we want for an expected disk to show up. > >What are there plans to allow disks to arrive after we give up? >Obviously if we give up, we'll be at the root> prompt, but if a disk >is a little late, it would be nice to be able to say 'da0s1a' and have >it find this device, rather than hard wiring the available devices at >the time that the first root> prompt is given. That's how it works today and how it should still work IMO. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-arch@FreeBSD.ORG Wed Jan 14 12:42:49 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 65A6F16A4CE; Wed, 14 Jan 2004 12:42:49 -0800 (PST) Received: from kientzle.com (h-66-166-149-50.SNVACAID.covad.net [66.166.149.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id F247A43D1D; Wed, 14 Jan 2004 12:42:29 -0800 (PST) (envelope-from kientzle@acm.org) Received: from acm.org ([66.166.149.54]) by kientzle.com (8.12.9/8.12.9) with ESMTP id i0EKgHkX074721; Wed, 14 Jan 2004 12:42:18 -0800 (PST) (envelope-from kientzle@acm.org) Message-ID: <4005A9A9.6090503@acm.org> Date: Wed, 14 Jan 2004 12:42:17 -0800 From: Tim Kientzle User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.4) Gecko/20031006 X-Accept-Language: en-us, en MIME-Version: 1.0 To: "M. Warner Losh" References: <4004D445.7020205@acm.org> <20040114.131705.93652359.imp@bsdimp.com> In-Reply-To: <20040114.131705.93652359.imp@bsdimp.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit cc: rwatson@freebsd.org cc: freebsd-arch@freebsd.org Subject: Re: Request for Comments: libarchive, bsdtar X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: kientzle@acm.org List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Jan 2004 20:42:49 -0000 M. Warner Losh wrote: > In message: > Robert Watson writes: > : I know NetBSD has a neat tool to create file systems from userspace > : without privilege, but my understanding is that it has to pull in a lot of > : code from the kernel in fairly messy ways. Since tar files are a well > : supported portable format... :-) > > The problem then reduces to needing to be able to create the tar file > with ownership/permissions different than is in the unpriv'd build > tree. Generation of this list, as well as getting tar/whatever to > honor it are interesting problems. Not all that interesting, really. ;-) For example, the following simple C program uses libarchive to read an archive from stdin, and write a new ustar/gzip archive to stdout with all entries owned by 'root'. You just create the archive as usual and then feed it through this filter to create a new archive with the proper ownership/permissions. The same technique can be used add entries to an archive, remove entries from an archive, rename archive entries, convert archive formats, etc. struct archive *in; struct archive *out; struct archive_entry *entry; /* Read an archive from stdin: format/compression will be auto-detected. */ in = archive_read_new(); archive_read_support_format_all(in); archive_read_support_compression_all(in); archive_read_open_file(in, NULL); /* NULL == stdin */ /* Write a ustar/gzip archive to stdout. */ out = archive_write_new(); archive_write_set_compression_gzip(out); archive_write_set_format_ustar(out); archive_write_open_file(out, NULL); /* NULL == stdout */ /* Copy entries over, doing any necessary editing */ while (archive_read_next_header(in, &entry) == ARCHIVE_READ_OK) { /* Edit the entry as appropriate */ archive_entry_set_uid(entry, 0); archive_entry_set_uname(entry, "root"); /* Write a header to the output archive */ archive_write_header(out, entry); /* TODO: XXX Copy data to output archive XXX */ } /* All done */ archive_write_finish(out); archive_read_finish(in); From owner-freebsd-arch@FreeBSD.ORG Wed Jan 14 13:22:07 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1CB7216A4DB for ; Wed, 14 Jan 2004 13:22:06 -0800 (PST) Received: from critter.freebsd.dk (critter.freebsd.dk [212.242.86.163]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8CF1643D58 for ; Wed, 14 Jan 2004 13:22:04 -0800 (PST) (envelope-from phk@phk.freebsd.dk) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.12.10/8.12.10) with ESMTP id i0ELM2dN029372; Wed, 14 Jan 2004 22:22:03 +0100 (CET) (envelope-from phk@phk.freebsd.dk) To: Vladimir Dozen From: "Poul-Henning Kamp" In-Reply-To: Your message of "Wed, 14 Jan 2004 14:52:27 EST." <40059DFB.9000904@mail.ru> Date: Wed, 14 Jan 2004 22:22:02 +0100 Message-ID: <29371.1074115322@critter.freebsd.dk> cc: freebsd-arch@freebsd.org Subject: Re: Request for Comments: libarchive, bsdtar X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Jan 2004 21:22:07 -0000 In message <40059DFB.9000904@mail.ru>, Vladimir Dozen writes: >ehlo. > >> If you become a bored person requiring entertainment, it might be quite >> interesting to create a read-only tarfs for use as a root file system >> loaded in an md device. > > I'm just curious: is it possible to write a GEOM class to mount > a given tar as a FS in R/O mode (using libarchive, I mean)? That would be a filesystem, not a GEOM class. And yes it would be. Part of the trouble is directory searches where you need to build some kind of index in order to not have to run through sequentially all the time. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-arch@FreeBSD.ORG Wed Jan 14 13:24:21 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0FF9116A4CE; Wed, 14 Jan 2004 13:24:21 -0800 (PST) Received: from harmony.village.org (rover.bsdimp.com [204.144.255.66]) by mx1.FreeBSD.org (Postfix) with ESMTP id C5E7F43D64; Wed, 14 Jan 2004 13:24:19 -0800 (PST) (envelope-from imp@bsdimp.com) Received: from localhost (warner@rover2.village.org [10.0.0.1]) by harmony.village.org (8.12.10/8.12.9) with ESMTP id i0ELOGET093261; Wed, 14 Jan 2004 14:24:17 -0700 (MST) (envelope-from imp@bsdimp.com) Date: Wed, 14 Jan 2004 14:24:14 -0700 (MST) Message-Id: <20040114.142414.97038855.imp@bsdimp.com> To: kientzle@acm.org From: "M. Warner Losh" In-Reply-To: <4005A9A9.6090503@acm.org> References: <20040114.131705.93652359.imp@bsdimp.com> <4005A9A9.6090503@acm.org> X-Mailer: Mew version 3.3 on Emacs 21.3 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit cc: rwatson@freebsd.org cc: freebsd-arch@freebsd.org Subject: Re: Request for Comments: libarchive, bsdtar X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Jan 2004 21:24:21 -0000 In message: <4005A9A9.6090503@acm.org> Tim Kientzle writes: : M. Warner Losh wrote: : > In message: : > Robert Watson writes: : > : I know NetBSD has a neat tool to create file systems from userspace : > : without privilege, but my understanding is that it has to pull in a lot of : > : code from the kernel in fairly messy ways. Since tar files are a well : > : supported portable format... :-) : > : > The problem then reduces to needing to be able to create the tar file : > with ownership/permissions different than is in the unpriv'd build : > tree. Generation of this list, as well as getting tar/whatever to : > honor it are interesting problems. : : Not all that interesting, really. ;-) A small matter of coding then to cause the meta data to be stored during the install and then written to the archive format. Warner From owner-freebsd-arch@FreeBSD.ORG Wed Jan 14 13:27:54 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 959FC16A4CE for ; Wed, 14 Jan 2004 13:27:54 -0800 (PST) Received: from odin.ac.hmc.edu (Odin.AC.HMC.Edu [134.173.32.75]) by mx1.FreeBSD.org (Postfix) with ESMTP id B3ABA43D58 for ; Wed, 14 Jan 2004 13:27:41 -0800 (PST) (envelope-from brdavis@odin.ac.hmc.edu) Received: from odin.ac.hmc.edu (IDENT:brdavis@localhost.localdomain [127.0.0.1]) by odin.ac.hmc.edu (8.12.10/8.12.3) with ESMTP id i0ELRcaT010245; Wed, 14 Jan 2004 13:27:38 -0800 Received: (from brdavis@localhost) by odin.ac.hmc.edu (8.12.10/8.12.3/Submit) id i0ELRcaD010244; Wed, 14 Jan 2004 13:27:38 -0800 Date: Wed, 14 Jan 2004 13:27:38 -0800 From: Brooks Davis To: Poul-Henning Kamp Message-ID: <20040114212736.GE4974@Odin.AC.HMC.Edu> References: <40059DFB.9000904@mail.ru> <29371.1074115322@critter.freebsd.dk> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="ZRyEpB+iJ+qUx0kp" Content-Disposition: inline In-Reply-To: <29371.1074115322@critter.freebsd.dk> User-Agent: Mutt/1.5.4i X-Virus-Scanned: by amavisd-milter (http://amavis.org/) on odin.ac.hmc.edu cc: freebsd-arch@freebsd.org cc: Vladimir Dozen Subject: Re: Request for Comments: libarchive, bsdtar X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Jan 2004 21:27:54 -0000 --ZRyEpB+iJ+qUx0kp Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Jan 14, 2004 at 10:22:02PM +0100, Poul-Henning Kamp wrote: > In message <40059DFB.9000904@mail.ru>, Vladimir Dozen writes: > >ehlo. > > > >> If you become a bored person requiring entertainment, it might be quite > >> interesting to create a read-only tarfs for use as a root file system > >> loaded in an md device. > > > > I'm just curious: is it possible to write a GEOM class to mount > > a given tar as a FS in R/O mode (using libarchive, I mean)? >=20 > That would be a filesystem, not a GEOM class. >=20 > And yes it would be. Part of the trouble is directory searches > where you need to build some kind of index in order to not have > to run through sequentially all the time. For cases where you know you're going to be using the archive as a file system, you might be able to cheat a bit by creating a special file containing an efficent index of the contents of the archive. That would let you avoid keeping everything in memory and rescanning the disk constantly. -- Brooks --=20 Any statement of the form "X is the one, true Y" is FALSE. PGP fingerprint 655D 519C 26A7 82E7 2529 9BF0 5D8E 8BE9 F238 1AD4 --ZRyEpB+iJ+qUx0kp Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.1 (GNU/Linux) iD8DBQFABbRHXY6L6fI4GtQRAvnLAKChcsMbh8lVpdetpzan8zMB4Z04JQCgr1Fw tKTw/3N6A7dSTWPR70QU2ps= =vSXp -----END PGP SIGNATURE----- --ZRyEpB+iJ+qUx0kp-- From owner-freebsd-arch@FreeBSD.ORG Wed Jan 14 13:34:19 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7B56716A4CE; Wed, 14 Jan 2004 13:34:19 -0800 (PST) Received: from rwcrmhc13.comcast.net (rwcrmhc13.comcast.net [204.127.198.39]) by mx1.FreeBSD.org (Postfix) with ESMTP id BA11E43D55; Wed, 14 Jan 2004 13:34:04 -0800 (PST) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([24.7.73.28]) by comcast.net (rwcrmhc13) with ESMTP id <2004011421340401500b7cjke>; Wed, 14 Jan 2004 21:34:04 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id NAA14710; Wed, 14 Jan 2004 13:34:03 -0800 (PST) Date: Wed, 14 Jan 2004 13:34:02 -0800 (PST) From: Julian Elischer To: David Xu In-Reply-To: <4004B4D0.907@freebsd.org> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: arch@freebsd.org Subject: Re: ptrace and thread X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Jan 2004 21:34:19 -0000 On Wed, 14 Jan 2004, David Xu wrote: > I am current working on debug support for KSE thread program, however I > found > ptrace interface is not thread-aware, in a threaded program, I need to > get/set registers > set for individual threads, current ptrace can not support that > features, there are two > ways to support these requirements: Yes I tried to address this a bit around the time when I added the single ttreading code. Ialso made several posts looking for advice from gdb/ptrace experts but got very little response.. As you noticed, the ptrace facility is almost completely useless WRT threads.. it is possible to imagine an extension where you select a single thread of interest but you would have to decide whether you want all the other threads to be left running or left suspended.. (you may need both possibilities to correctly debug a problem) The problem is that the thread becomes invisible to the kernel when it crosses over to userland so the UTS needs to take an active part, (unless the kernel can recognise when the thread has yielded and the UTS has been enterred. (possible I guess) at which time single stepping would be turned off allowing the UTS to run at full speed. The UTS would hav eto co-operate by using a method of re-enterring the thread that allows the kernel to re-start single stepping.. What the other threads are doing in teh meanwhile is unknown. > > 1. keep current ptrace interface, add a command for example > PT_SETDTHREAD to > set current thread for debug, and subsequent request for example > PT_SETREGS and > PT_GETREGS will work on the thread, for single thread process, the > default current > thread is always the first thread in the process, this way we needn't > change legacy debugger > code. yes.. > > 2. introduce a second ptrace syscall, and accept a new parameter tid > (thread id), > the PT_SETREGS and PT_GETREGS will use the tid to operate on > corresponding > thread. > as mentionned by others this is also what some other systems did. should this be in -arch or -threads? > For first method, I have a patch there: > http://people.freebsd.org/~davidxu/kse/ptrace.diff > The patch also includes some bits to support KSE debug, not just for > pure 1:1 threading. > > David Xu > > > _______________________________________________ > freebsd-arch@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" > From owner-freebsd-arch@FreeBSD.ORG Wed Jan 14 13:48:24 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E6FAA16A4CE; Wed, 14 Jan 2004 13:48:24 -0800 (PST) Received: from gw.catspoiler.org (217-ip-163.nccn.net [209.79.217.163]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0A98D43D49; Wed, 14 Jan 2004 13:48:20 -0800 (PST) (envelope-from truckman@FreeBSD.org) Received: from FreeBSD.org (mousie.catspoiler.org [192.168.101.2]) by gw.catspoiler.org (8.12.9p2/8.12.9) with ESMTP id i0ELm77E040545; Wed, 14 Jan 2004 13:48:11 -0800 (PST) (envelope-from truckman@FreeBSD.org) Message-Id: <200401142148.i0ELm77E040545@gw.catspoiler.org> Date: Wed, 14 Jan 2004 13:48:07 -0800 (PST) From: Don Lewis To: rwatson@FreeBSD.org In-Reply-To: MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii cc: freebsd-arch@FreeBSD.org cc: kientzle@acm.org Subject: Re: Request for Comments: libarchive, bsdtar X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Jan 2004 21:48:25 -0000 On 14 Jan, Robert Watson wrote: > On Tue, 13 Jan 2004, Tim Kientzle wrote: > > ... > > All this generally sounds good. > >> LIBARCHIVE BACKGROUND >> >> As many of you know, I've been working on a project to overhaul the pkg >> tools. Among many other things, this requires a library that can >> read/write tar archives. This avoids the significant overhead imposed >> from forking a separate tar program. > > If you become a bored person requiring entertainment, it might be quite > interesting to create a read-only tarfs for use as a root file system > loaded in an md device. While there's a lot more to it than this, one of > the more irritating things about our current release build is that it > requires privilege so that it can chroot(), but also so it can manage md > devices and file system images. Just being able to use a tarball instead > of a UFS image would go a long way, although presumably require changes to > our loader as well. For work with diskless systems and network booting, > I'd much rather stick a tarball on an NFS server than create UFS images. > > I know NetBSD has a neat tool to create file systems from userspace > without privilege, but my understanding is that it has to pull in a lot of > code from the kernel in fairly messy ways. Since tar files are a well > supported portable format... :-) Why not use iso9660? The userland code already exists to create it and the file system code already exists to read it. From owner-freebsd-arch@FreeBSD.ORG Wed Jan 14 13:59:35 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 55AAA16A4CE; Wed, 14 Jan 2004 13:59:35 -0800 (PST) Received: from smtp.des.no (flood.des.no [217.116.83.31]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2C6F743D2D; Wed, 14 Jan 2004 13:59:34 -0800 (PST) (envelope-from des@des.no) Received: by smtp.des.no (Pony Express, from userid 666) id 2F1BF530C; Wed, 14 Jan 2004 22:59:33 +0100 (CET) Received: from dwp.des.no (des.no [80.203.228.37]) by smtp.des.no (Pony Express) with ESMTP id C9FC75309; Wed, 14 Jan 2004 22:59:24 +0100 (CET) Received: by dwp.des.no (Postfix, from userid 2602) id B20F733C9A; Wed, 14 Jan 2004 22:59:24 +0100 (CET) To: Robert Watson References: From: des@des.no (Dag-Erling =?iso-8859-1?q?Sm=F8rgrav?=) Date: Wed, 14 Jan 2004 22:59:24 +0100 In-Reply-To: (Robert Watson's message of "Wed, 14 Jan 2004 14:24:46 -0500 (EST)") Message-ID: User-Agent: Gnus/5.090024 (Oort Gnus v0.24) Emacs/21.3 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-Spam-Checker-Version: SpamAssassin 2.61 (1.212.2.1-2003-12-09-exp) on flood.des.no X-Spam-Level: ss X-Spam-Status: No, hits=2.6 required=5.0 tests=RCVD_IN_DYNABLOCK, RCVD_IN_SORBS autolearn=no version=2.61 cc: freebsd-arch@freebsd.org cc: Tim Kientzle Subject: Re: Request for Comments: libarchive, bsdtar X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Jan 2004 21:59:35 -0000 Robert Watson writes: > If you become a bored person requiring entertainment, it might be quite > interesting to create a read-only tarfs for use as a root file system > loaded in an md device. What's wrong with iso9660 filesystems created by mkisofs? DES --=20 Dag-Erling Sm=F8rgrav - des@des.no From owner-freebsd-arch@FreeBSD.ORG Wed Jan 14 14:05:23 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9474E16A4CE; Wed, 14 Jan 2004 14:05:23 -0800 (PST) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 264CC43D66; Wed, 14 Jan 2004 14:05:17 -0800 (PST) (envelope-from robert@fledge.watson.org) Received: from fledge.watson.org (localhost [127.0.0.1]) by fledge.watson.org (8.12.10/8.12.10) with ESMTP id i0EM3TUd054250; Wed, 14 Jan 2004 17:03:29 -0500 (EST) (envelope-from robert@fledge.watson.org) Received: from localhost (robert@localhost)i0EM3TH8054247; Wed, 14 Jan 2004 17:03:29 -0500 (EST) (envelope-from robert@fledge.watson.org) Date: Wed, 14 Jan 2004 17:03:29 -0500 (EST) From: Robert Watson X-Sender: robert@fledge.watson.org To: Julian Elischer In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: arch@freebsd.org cc: David Xu Subject: Re: ptrace and thread X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Jan 2004 22:05:23 -0000 On Wed, 14 Jan 2004, Julian Elischer wrote: > On Wed, 14 Jan 2004, David Xu wrote: > > > I am current working on debug support for KSE thread program, however I > > found > > ptrace interface is not thread-aware, in a threaded program, I need to > > get/set registers > > set for individual threads, current ptrace can not support that > > features, there are two > > ways to support these requirements: > > Yes I tried to address this a bit around the time when I added the > single ttreading code. Ialso made several posts looking for advice from > gdb/ptrace experts but got very little response.. As you noticed, the > ptrace facility is almost completely useless WRT threads.. > > it is possible to imagine an extension where you select a single thread > of interest but you would have to decide whether you want all the other > threads to be left running or left suspended.. (you may need both > possibilities to correctly debug a problem) > > The problem is that the thread becomes invisible to the kernel when it > crosses over to userland so the UTS needs to take an active part, > (unless the kernel can recognise when the thread has yielded and the UTS > has been enterred. (possible I guess) at which time single stepping > would be turned off allowing the UTS to run at full speed. > > The UTS would hav eto co-operate by using a method of re-enterring the > thread that allows the kernel to re-start single stepping.. > > What the other threads are doing in teh meanwhile is unknown. Apparnelty one common model is to get the application's (or threading package's) help in the debugging process. Teach gdb to talk to libkse, and have libkse provide the information on threading to gdb, as well as do the scheduling/assert control. FWIW, I think ptrace/procfs are in a somewhat broken state following KSE -- stuff like gdb works, but strace is known to hang. Part of the problem appears to be slightly different models for stopping the process -- p_step vs p_stop(). I'm still getting my head around the details of the debugging facilities, which is complicated by the fact that procfs is a lot more capable than the man page claims :-). I hope to dig more into the problem next week. Robert N M Watson FreeBSD Core Team, TrustedBSD Projects robert@fledge.watson.org Senior Research Scientist, McAfee Research From owner-freebsd-arch@FreeBSD.ORG Wed Jan 14 14:10:18 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2BB2116A4CE for ; Wed, 14 Jan 2004 14:10:18 -0800 (PST) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 184DB43D6D for ; Wed, 14 Jan 2004 14:10:08 -0800 (PST) (envelope-from robert@fledge.watson.org) Received: from fledge.watson.org (localhost [127.0.0.1]) by fledge.watson.org (8.12.10/8.12.10) with ESMTP id i0EM77Ud054403; Wed, 14 Jan 2004 17:07:07 -0500 (EST) (envelope-from robert@fledge.watson.org) Received: from localhost (robert@localhost)i0EM77UC054400; Wed, 14 Jan 2004 17:07:07 -0500 (EST) (envelope-from robert@fledge.watson.org) Date: Wed, 14 Jan 2004 17:07:07 -0500 (EST) From: Robert Watson X-Sender: robert@fledge.watson.org To: Dag-Erling =?iso-8859-1?q?Sm=F8rgrav?= In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=ISO-8859-1 Content-Transfer-Encoding: QUOTED-PRINTABLE cc: freebsd-arch@freebsd.org cc: Tim Kientzle Subject: Re: Request for Comments: libarchive, bsdtar X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Jan 2004 22:10:18 -0000 On Wed, 14 Jan 2004, Dag-Erling Sm=F8rgrav wrote: > Robert Watson writes: > > If you become a bored person requiring entertainment, it might be quite > > interesting to create a read-only tarfs for use as a root file system > > loaded in an md device. >=20 > What's wrong with iso9660 filesystems created by mkisofs?=20 (picking one such reply seemingly at random) Good point. With the RR extensions, that should work just fine.=20 Robert N M Watson FreeBSD Core Team, TrustedBSD Projects robert@fledge.watson.org Senior Research Scientist, McAfee Research From owner-freebsd-arch@FreeBSD.ORG Wed Jan 14 15:48:13 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E266716A4CE for ; Wed, 14 Jan 2004 15:48:13 -0800 (PST) Received: from smtp02.syd.iprimus.net.au (smtp02.syd.iprimus.net.au [210.50.76.52]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5175343D49 for ; Wed, 14 Jan 2004 15:48:12 -0800 (PST) (envelope-from tim@robbins.dropbear.id.au) Received: from robbins.dropbear.id.au (210.50.44.40) by smtp02.syd.iprimus.net.au (7.0.020) id 3F8F522A01A12032; Thu, 15 Jan 2004 10:48:08 +1100 Received: by robbins.dropbear.id.au (Postfix, from userid 1000) id 824BF4155; Thu, 15 Jan 2004 10:48:29 +1100 (EST) Date: Thu, 15 Jan 2004 10:48:29 +1100 From: Tim Robbins To: Tim Kientzle Message-ID: <20040114234829.GA19067@cat.robbins.dropbear.id.au> References: <4004D445.7020205@acm.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4004D445.7020205@acm.org> User-Agent: Mutt/1.4.1i cc: freebsd-arch@freebsd.org Subject: Re: Request for Comments: libarchive, bsdtar X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Jan 2004 23:48:14 -0000 On Tue, Jan 13, 2004 at 09:31:49PM -0800, Tim Kientzle wrote: > Request for Comments: libarchive, bsdtar > > PROPOSAL > > Add "libarchive" to the tree, prepare to change the system > tar command to "bsdtar" once it is sufficiently stable. [...] > Feedback and comments greatly appreciated, Let me start by thanking you for working on replacing GNU utilities with higher quality and less restrictively licensed alternatives. I haven't had time to read over the code very thoroughly, but I have a few initial comments: - Padding gzip'd tar archives (with bsdtar czf) causes gzip to report "trailing garbage" and fail, and in turn this causes GNU tar to fail. BSD pax (-wzf) and GNU tar (czf) do not pad compressed archives. - I would prefer it if compression was done by opening a pipe to gzip/bzip2 instead of using libz/libbz2. This would make things simpler, and make it easier to support compress(1). - I don't think the URL/libfetch support belongs in a library that deals with archives. Perhaps the interface could be changed so that the caller could pass a FILE * or a file descriptor instead of a filename. - Filenames are too long :-) Tim From owner-freebsd-arch@FreeBSD.ORG Wed Jan 14 17:23:46 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4F4F116A4CF; Wed, 14 Jan 2004 17:23:46 -0800 (PST) Received: from kientzle.com (h-66-166-149-50.SNVACAID.covad.net [66.166.149.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0383943D70; Wed, 14 Jan 2004 17:23:43 -0800 (PST) (envelope-from kientzle@acm.org) Received: from acm.org ([66.166.149.54]) by kientzle.com (8.12.9/8.12.9) with ESMTP id i0F1NgkX076018; Wed, 14 Jan 2004 17:23:42 -0800 (PST) (envelope-from kientzle@acm.org) Message-ID: <4005EB9D.50506@acm.org> Date: Wed, 14 Jan 2004 17:23:41 -0800 From: Tim Kientzle User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.4) Gecko/20031006 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Tim Robbins References: <4004D445.7020205@acm.org> <20040114234829.GA19067@cat.robbins.dropbear.id.au> In-Reply-To: <20040114234829.GA19067@cat.robbins.dropbear.id.au> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit cc: freebsd-arch@freebsd.org Subject: Re: Request for Comments: libarchive, bsdtar X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: kientzle@acm.org List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Jan 2004 01:23:46 -0000 Tim Robbins wrote: > On Tue, Jan 13, 2004 at 09:31:49PM -0800, Tim Kientzle wrote: > >>Request for Comments: libarchive, bsdtar >> >>Add "libarchive" to the tree, prepare to change the system >>tar command to "bsdtar" once it is sufficiently stable. > > [...] > > Let me start by thanking you for working on replacing GNU utilities with > higher quality and less restrictively licensed alternatives. I haven't > had time to read over the code very thoroughly, but I have a few initial > comments: Thanks for the feedback. A lot of people rely on 'tar', so I want to make sure it's well-tested and does what people really need before it becomes the default. When you do have time to look over the code, please let me know what you think. > - Padding gzip'd tar archives (with bsdtar czf) causes gzip to report > "trailing garbage" and fail, and in turn this causes GNU tar to fail. Oddly, GNU tar does successfully and correctly extract the archive, and then exits with an error code. There's an easy one-line patch that fixes this bug in GNU tar, by the way. ;-) > BSD pax (-wzf) and GNU tar (czf) do not pad compressed archives. The issue here is correct blocking for devices that require it. (E.g., tape drives, floppies) libarchive correctly blocks all output, regardless of whether or not it is compressed. Neither GNU tar nor BSD pax gaurantee this. It goes a bit deeper in the case of libarchive. By design, libarchive knows nothing about the archive storage. This means there is no simple way for it to vary it's operation depending on whether it's writing to a file or character device, unlike monolithic programs such as GNU tar or BSD pax. I have some ideas about how to change this by generalizing the blocking calculations within libarchive and providing some client hooks for finer control over the blocking, but I haven't decided whether or not it's worth the effort. Somehow, though, I doubt you'll be the last person to complain about this ;-), so I'll start looking for a good way to change this behavior. > - I would prefer it if compression was done by opening a pipe to gzip/bzip2 > instead of using libz/libbz2. This would make things simpler, and make it > easier to support compress(1). Not really simpler for the library, and definitely not simpler for clients of the library. This is related to the blocking issue I mentioned just above. In order to correctly block the output, you need to collect the output of the compression program and reblock it. An early version of libarchive did exactly this, forking a three-stage pipeline with the compression/decompression program in the middle. Unfortunately, this created some odd problems, as the archive I/O then occurred in a separate process from the rest of the program. For example, this made it difficult for clients to monitor the I/O status from their mainline code, and hampered proper error reporting. It also seemed inappropriate for a library to be invoking client-provided callbacks in a different process. However, each compression type is handled in a cleanly-factored code module, and I do still have the code in my personal CVS repo to fork out the pipeline. I could resurrect this to fork compress(1) if there's real demand. > - I don't think the URL/libfetch support belongs in a library that deals > with archives. Perhaps the interface could be changed so that the > caller could pass a FILE * or a file descriptor instead of a filename. The libfetch tie-in (archive_read_open_url) is provided purely for the convenience of simple clients. If you don't like it, don't use it. It is completely optional. Generally, I've gone to a great deal of effort to minimize link pollution. For example, if you don't call the functions that handle gzip/bzip2 compression, they won't be linked in and neither will libz/libbz2. Similar comments apply to the various format support functions. I've even carefully separated archive reading and writing in case you only want to use one of them. As for I/O interfaces, the core archive_read_open and archive_write_open functions accept a collection of function pointers that the library will invoke for open/read/write/close operations on the archive. This is considerably more flexible than FILE * or file descriptors. Not to mention that passing file descriptors has some tricky implications if the library forks to run archive I/O in a separate process. FILE * is simply a bad idea because the stdio interface doesn't provide client control over blocking. (Yes, the libfetch convenience hooks do use FILE *, but blocking is unimportant for sockets, so that's okay.) > - Filenames are too long :-) Take a typing class. ;-) From owner-freebsd-arch@FreeBSD.ORG Wed Jan 14 18:28:14 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4331616A4CE for ; Wed, 14 Jan 2004 18:28:14 -0800 (PST) Received: from f4.mail.ru (f4.mail.ru [194.67.57.34]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1544343D1D for ; Wed, 14 Jan 2004 18:28:13 -0800 (PST) (envelope-from vladimir-dozen@mail.ru) Received: from [213.170.97.162] (port=1394 helo=mail.ru) by f4.mail.ru with esmtp id 1AgxEx-000H7z-00; Thu, 15 Jan 2004 05:28:11 +0300 Message-ID: <4005FAB4.7070304@mail.ru> Date: Wed, 14 Jan 2004 21:28:04 -0500 From: Vladimir Dozen User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.4) Gecko/20030624 X-Accept-Language: ru, en-us, en MIME-Version: 1.0 To: freebsd-arch@freebsd.org References: <4004D445.7020205@acm.org> <20040114234829.GA19067@cat.robbins.dropbear.id.au> <4005EB9D.50506@acm.org> In-Reply-To: <4005EB9D.50506@acm.org> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected cc: kientzle@acm.org Subject: Re: Request for Comments: libarchive, bsdtar X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Jan 2004 02:28:14 -0000 ehlo. > so I want to make sure it's well-tested BTW, how do you perform testing of the library/tar? This kind of code should (shall) contain almost 100% automated test coverage, in my opinion. It's due to it's complexity and extremely wide usage. >> - I would prefer it if compression was done by opening a pipe to >> gzip/bzip2 instead of using libz/libbz2. > Not really simpler for the library, and definitely not simpler > for clients of the library. Yeah. Spawning a process from inside the library is a bad idea, IMO. The right way to avoid code duplication between gzip and libarchive is to use common libgzip. The same applies to bzip and compress. dozen http://dozen.ru From owner-freebsd-arch@FreeBSD.ORG Wed Jan 14 19:39:56 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B77B916A4CE for ; Wed, 14 Jan 2004 19:39:56 -0800 (PST) Received: from kientzle.com (h-66-166-149-50.SNVACAID.covad.net [66.166.149.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6AF7543D55 for ; Wed, 14 Jan 2004 19:39:54 -0800 (PST) (envelope-from kientzle@acm.org) Received: from acm.org ([66.166.149.54]) by kientzle.com (8.12.9/8.12.9) with ESMTP id i0F3drkX076610; Wed, 14 Jan 2004 19:39:54 -0800 (PST) (envelope-from kientzle@acm.org) Message-ID: <40060B87.20906@acm.org> Date: Wed, 14 Jan 2004 19:39:51 -0800 From: Tim Kientzle User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.4) Gecko/20031006 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Vladimir Dozen References: <4004D445.7020205@acm.org> <20040114234829.GA19067@cat.robbins.dropbear.id.au> <4005EB9D.50506@acm.org> <4005FAB4.7070304@mail.ru> In-Reply-To: <4005FAB4.7070304@mail.ru> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit cc: freebsd-arch@freebsd.org Subject: Re: Request for Comments: libarchive, bsdtar X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: kientzle@acm.org List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Jan 2004 03:39:56 -0000 Vladimir Dozen wrote: > > so I want to make sure it's well-tested > > BTW, how do you perform testing of the library/tar? I've used automated tests to verify a few of the trickier routines, such as exercising boundary conditions in the formatting and parsing logic. There are a number of built-in logic tests in the code. Most notably, each public function starts with a call to "archive_check_magic" which verifies that the provided archive structure is in the correct state. I've also been collecting sample test archives to verify correct operation. Joerg Schilling's collection of test files has been very helpful. I haven't yet had a chance to automate the full-program tests, though. I have a few ideas about how to proceed and what needs testing, but haven't yet pieced anything together. I also plan to use dmalloc (or something similar) to test for memory leaks and invalid heap operations. I used some crude, home-grown routines early in development to verify memory usage but haven't had a chance to do more systematic testing. I have tested enough to know that: * libarchive correctly archives 64k pathnames * performance is comparable to gtar overall * When reading/writing compressed archives, zlib/bzlib are the performance bottlenecks. bzlib, in particular, seems very sensitive to the size of blocks you feed it; some work to compress/decompress larger blocks would be helpful * When writing non-compressed archives, getpwent/getgrent calls in bsdtar are the most obvious performance issue (about 10% of the CPU time for tar -cf /dev/null) I'm considering a simple LRU cache of uname/gname lookups to address this. > >> - I would prefer it if compression was done by opening a pipe to > >> gzip/bzip2 instead of using libz/libbz2. > ... > The right way to avoid code duplication between gzip and libarchive > is to use common libgzip. The same applies to bzip and compress. Yes, I use zlib and bzlib to handle compression. This handles essentially everything except for the relatively minor task of generating/verifying the gzip header (newer versions of zlib handle even that, but the version we currently have in the tree does not). I would love to see a library version of compress(1) with an API similar to that of zlib/bzlib. Tim From owner-freebsd-arch@FreeBSD.ORG Wed Jan 14 21:38:07 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 21A4616A4CE; Wed, 14 Jan 2004 21:38:07 -0800 (PST) Received: from gw.catspoiler.org (217-ip-163.nccn.net [209.79.217.163]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3300443D5E; Wed, 14 Jan 2004 21:38:02 -0800 (PST) (envelope-from truckman@FreeBSD.org) Received: from FreeBSD.org (mousie.catspoiler.org [192.168.101.2]) by gw.catspoiler.org (8.12.9p2/8.12.9) with ESMTP id i0F5bo7E041482; Wed, 14 Jan 2004 21:37:54 -0800 (PST) (envelope-from truckman@FreeBSD.org) Message-Id: <200401150537.i0F5bo7E041482@gw.catspoiler.org> Date: Wed, 14 Jan 2004 21:37:50 -0800 (PST) From: Don Lewis To: tjr@FreeBSD.org In-Reply-To: <20040114234829.GA19067@cat.robbins.dropbear.id.au> MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii cc: freebsd-arch@FreeBSD.org cc: kientzle@acm.org Subject: Re: Request for Comments: libarchive, bsdtar X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Jan 2004 05:38:07 -0000 On 15 Jan, Tim Robbins wrote: > - Padding gzip'd tar archives (with bsdtar czf) causes gzip to report > "trailing garbage" and fail, and in turn this causes GNU tar to fail. > BSD pax (-wzf) and GNU tar (czf) do not pad compressed archives. In the case of GNU tar, it depends on where it is writing. It pads when writing to stdout, but does not pad when writing to a file. I would hope that it pads when writing to a tape device, since many tapes require writing to some multiple of their native block size. % tar cfz /tmp/printcap.tgz /etc/printcap tar: Removing leading `/' from member names % ls -l /tmp/printcap.tgz -rw-r--r-- 1 dl wheel 1216 Jan 14 21:28 /tmp/printcap.tgz % tar cfz - /etc/printcap > /tmp/printcap.tgz tar: Removing leading `/' from member names % ls -l /tmp/printcap.tgz -rw-r--r-- 1 dl wheel 10240 Jan 14 21:28 /tmp/printcap.tgz I would prefer to have explicit control of this behavior. BTW, on a Unix variant that I used many years ago, there was an enhanced version of dd, called ddx, which had the useful option "mobs", which was used to specify a minimum output block size. You could run something like ddx obs=10k mobs=1k ... to cause all writes to be blocked to 10k except for the last partial write which would be padded out to at least 1k. I don't remember what it did if there was 1025 bytes left over for the last write ... From owner-freebsd-arch@FreeBSD.ORG Wed Jan 14 21:43:34 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E011F16A4CE for ; Wed, 14 Jan 2004 21:43:34 -0800 (PST) Received: from smtp.omnis.com (smtp.omnis.com [216.239.128.26]) by mx1.FreeBSD.org (Postfix) with ESMTP id D840443D67 for ; Wed, 14 Jan 2004 21:43:33 -0800 (PST) (envelope-from wes@softweyr.com) Received: from softweyr.homeunix.net (66-91-236-204.san.rr.com [66.91.236.204]) by smtp-relay.omnis.com (Postfix) with ESMTP id 04364188C438; Wed, 14 Jan 2004 21:43:33 -0800 (PST) From: Wes Peters Organization: Softweyr To: "M. Warner Losh" , phk@phk.freebsd.dk Date: Wed, 14 Jan 2004 21:43:32 -0800 User-Agent: KMail/1.5.4 References: <20040114105700.H73725@carver.gumbysoft.com> <27183.1074107137@critter.freebsd.dk> <20040114.131523.03269738.imp@bsdimp.com> In-Reply-To: <20040114.131523.03269738.imp@bsdimp.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200401142143.32886.wes@softweyr.com> cc: arch@freebsd.org Subject: Re: About removable disks, mountroot and sw-raid X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Jan 2004 05:43:35 -0000 On Wednesday 14 January 2004 12:15 pm, M. Warner Losh wrote: > In message: <27183.1074107137@critter.freebsd.dk> > > "Poul-Henning Kamp" writes: > : In message <20040114105700.H73725@carver.gumbysoft.com>, Doug White writes: > : >It would be nice to make the wait tunable since most conditions > : > won't require it and there's no sense in delaying everyone's boot > : > for a few bizarre disk setups. > : > : You boot would not be delayed in the normal case. The delays only > : apply to how long time we want for an expected disk to show up. > > What are there plans to allow disks to arrive after we give up? > Obviously if we give up, we'll be at the root> prompt, but if a disk > is a little late, it would be nice to be able to say 'da0s1a' and have > it find this device, rather than hard wiring the available devices at > the time that the first root> prompt is given. Or better yet, have it recognize that the new disk that just magically appeared is a candidate for disk boot and booting from it when it attaches. -- Where am I, and what am I doing in this handbasket? Wes Peters wes@softweyr.com From owner-freebsd-arch@FreeBSD.ORG Wed Jan 14 22:31:38 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6C81816A4CE for ; Wed, 14 Jan 2004 22:31:38 -0800 (PST) Received: from critter.freebsd.dk (critter.freebsd.dk [212.242.86.163]) by mx1.FreeBSD.org (Postfix) with ESMTP id EDE5343D48 for ; Wed, 14 Jan 2004 22:31:36 -0800 (PST) (envelope-from phk@phk.freebsd.dk) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.12.10/8.12.10) with ESMTP id i0F6VVdN035614; Thu, 15 Jan 2004 07:31:32 +0100 (CET) (envelope-from phk@phk.freebsd.dk) To: Wes Peters From: "Poul-Henning Kamp" In-Reply-To: Your message of "Wed, 14 Jan 2004 21:43:32 PST." <200401142143.32886.wes@softweyr.com> Date: Thu, 15 Jan 2004 07:31:31 +0100 Message-ID: <35613.1074148291@critter.freebsd.dk> cc: arch@freebsd.org Subject: Re: About removable disks, mountroot and sw-raid X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Jan 2004 06:31:38 -0000 In message <200401142143.32886.wes@softweyr.com>, Wes Peters writes: >> What are there plans to allow disks to arrive after we give up? >> Obviously if we give up, we'll be at the root> prompt, but if a disk >> is a little late, it would be nice to be able to say 'da0s1a' and have >> it find this device, rather than hard wiring the available devices at >> the time that the first root> prompt is given. > >Or better yet, have it recognize that the new disk that just magically >appeared is a candidate for disk boot and booting from it when it >attaches. No, once we hit the prompt we stay there. If people want a longer timeout than default, they set the tunable. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-arch@FreeBSD.ORG Thu Jan 15 10:25:50 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5F86516A4D3 for ; Thu, 15 Jan 2004 10:25:50 -0800 (PST) Received: from VARK.homeunix.com (adsl-68-124-78-95.dsl.pltn13.pacbell.net [68.124.78.95]) by mx1.FreeBSD.org (Postfix) with ESMTP id 610E043D58 for ; Thu, 15 Jan 2004 10:25:47 -0800 (PST) (envelope-from das@FreeBSD.ORG) Received: from VARK.homeunix.com (localhost [127.0.0.1]) by VARK.homeunix.com (8.12.10/8.12.10) with ESMTP id i0FIPWKu026508; Thu, 15 Jan 2004 10:25:32 -0800 (PST) (envelope-from das@FreeBSD.ORG) Received: (from das@localhost) by VARK.homeunix.com (8.12.10/8.12.10/Submit) id i0FIPWnS026507; Thu, 15 Jan 2004 10:25:32 -0800 (PST) (envelope-from das@FreeBSD.ORG) Date: Thu, 15 Jan 2004 10:25:32 -0800 From: David Schultz To: Tim Kientzle Message-ID: <20040115182532.GA26149@VARK.homeunix.com> Mail-Followup-To: Tim Kientzle , freebsd-arch@FreeBSD.ORG References: <4004D445.7020205@acm.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4004D445.7020205@acm.org> cc: freebsd-arch@FreeBSD.ORG Subject: Re: Request for Comments: libarchive, bsdtar X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Jan 2004 18:25:50 -0000 On Tue, Jan 13, 2004, Tim Kientzle wrote: > Request for Comments: libarchive, bsdtar > > PROPOSAL > > Add "libarchive" to the tree, prepare to change the system > tar command to "bsdtar" once it is sufficiently stable. Nice! I'm sure this will be immensely useful when finished. I have a few pseudorandom comments: - Have you considered extending the API such that it is able to efficiently support random access archive formats? This would also be useful for sequential access formats such as tar(5) because the library could be extended to generate an in-core hash index for the archive on the fly, speeding up multiple lookups. Java's ZIP interface does this, for instance. - The HTTP and FTP support in libarchive(3) seems superfluous. Applications that need this feature can use libfetch(3) directly. (I realize that similar libraries such as libcomprex(3) do this, but that doesn't necessarily make it right.) - When this is done, I'm wondering what potential impact it might have on sysinstall and the archive format it uses... (not that I really want to reopen that bikeshed) From owner-freebsd-arch@FreeBSD.ORG Thu Jan 15 11:00:15 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id F2B8F16A4CF; Thu, 15 Jan 2004 11:00:14 -0800 (PST) Received: from kientzle.com (h-66-166-149-50.SNVACAID.covad.net [66.166.149.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id AFA8643D5E; Thu, 15 Jan 2004 11:00:08 -0800 (PST) (envelope-from kientzle@acm.org) Received: from acm.org ([66.166.149.54]) by kientzle.com (8.12.9/8.12.9) with ESMTP id i0FJ08kX079534; Thu, 15 Jan 2004 11:00:08 -0800 (PST) (envelope-from kientzle@acm.org) Message-ID: <4006E337.7000404@acm.org> Date: Thu, 15 Jan 2004 11:00:07 -0800 From: Tim Kientzle User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.4) Gecko/20031006 X-Accept-Language: en-us, en MIME-Version: 1.0 To: David Schultz References: <4004D445.7020205@acm.org> <20040115182532.GA26149@VARK.homeunix.com> In-Reply-To: <20040115182532.GA26149@VARK.homeunix.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit cc: freebsd-arch@FreeBSD.ORG Subject: Re: Request for Comments: libarchive, bsdtar X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: kientzle@acm.org List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Jan 2004 19:00:15 -0000 David Schultz wrote: > On Tue, Jan 13, 2004, Tim Kientzle wrote: > > Nice! I'm sure [libarchive] will be immensely useful when finished. It should be useful now. > I have a few pseudorandom comments: > > - Have you considered extending the API such that it is able to > efficiently support random access archive formats? Yes, and I've chosen not to go that way. In short, random-access is a different problem: tar/gzip and tar/bzip2 do not support random-access at all; uncompressed tar archives do not cleanly support updates; tape drives/stdin/stdout/sockets do not support random access. Random access gains you some things, loses you others. In short, libarchive is for "streaming archive formats." On the other hand, many archive formats can be handled via streaming: it should be possible for libarchive to read zip archives, for example. (However, compressed zip archives can't be written in a pure streaming mode.) > - The HTTP and FTP support in libarchive(3) seems superfluous. Okay, it's gone. > - When this is done, I'm wondering what potential impact it might > have on sysinstall and the archive format it uses... Sysinstall uses tar/gzip format, libarchive reads tar/gzip format. This may potentially impact the implementation of sysinstall (which could unpack the base system itself rather than running a separate tar), but I see no potential impact on sysinstall's choice of archive format. Tim From owner-freebsd-arch@FreeBSD.ORG Thu Jan 15 12:45:21 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DA06216A4CF for ; Thu, 15 Jan 2004 12:45:21 -0800 (PST) Received: from VARK.homeunix.com (adsl-68-124-78-95.dsl.pltn13.pacbell.net [68.124.78.95]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1408943D58 for ; Thu, 15 Jan 2004 12:45:19 -0800 (PST) (envelope-from das@FreeBSD.ORG) Received: from VARK.homeunix.com (localhost [127.0.0.1]) by VARK.homeunix.com (8.12.10/8.12.10) with ESMTP id i0FKj3Ku027378; Thu, 15 Jan 2004 12:45:03 -0800 (PST) (envelope-from das@FreeBSD.ORG) Received: (from das@localhost) by VARK.homeunix.com (8.12.10/8.12.10/Submit) id i0FKj34J027374; Thu, 15 Jan 2004 12:45:03 -0800 (PST) (envelope-from das@FreeBSD.ORG) Date: Thu, 15 Jan 2004 12:44:58 -0800 From: David Schultz To: Tim Kientzle Message-ID: <20040115204458.GA27244@VARK.homeunix.com> Mail-Followup-To: Tim Kientzle , freebsd-arch@FreeBSD.ORG References: <4004D445.7020205@acm.org> <20040115182532.GA26149@VARK.homeunix.com> <4006E337.7000404@acm.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4006E337.7000404@acm.org> cc: freebsd-arch@FreeBSD.ORG Subject: Re: Request for Comments: libarchive, bsdtar X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Jan 2004 20:45:22 -0000 On Thu, Jan 15, 2004, Tim Kientzle wrote: > David Schultz wrote: > >On Tue, Jan 13, 2004, Tim Kientzle wrote: > > > >Nice! I'm sure [libarchive] will be immensely useful when finished. > > It should be useful now. I didn't mean to imply that it isn't; I just meant to say that it will be *really* useful when it becomes the default and has better compatibility with the gtar format. I'm really glad you went ahead and did this. > >- Have you considered extending the API such that it is able to > > efficiently support random access archive formats? > > Yes, and I've chosen not to go that way. In short, random-access > is a different problem: tar/gzip and tar/bzip2 do not support > random-access at all; uncompressed tar archives do not cleanly > support updates; tape drives/stdin/stdout/sockets do not support > random access. Random access gains you some things, loses you others. > > In short, libarchive is for "streaming archive formats." > > On the other hand, many archive formats can be handled via > streaming: it should be possible for libarchive to read > zip archives, for example. (However, compressed zip archives > can't be written in a pure streaming mode.) [...] > >- When this is done, I'm wondering what potential impact it might > > have on sysinstall and the archive format it uses... > > Sysinstall uses tar/gzip format, libarchive reads tar/gzip format. > This may potentially impact the implementation of sysinstall > (which could unpack the base system itself rather than running > a separate tar), but I see no potential impact on sysinstall's > choice of archive format. Okay, so random access is simply not in the scope of your project. That's fine. The problem I was hoping could be solved has to do with the fact that the package tools don't have an archive format that supports random access, so they wind up extracting a temporary copy of the whole archive, then processing the parts that they want. As I understand, this is one of the reasons that sysinstall makes a hard distinction between distributions and packages, a distinction that shouldn't exist. However, I don't (want to) understand these problems as well as others... From owner-freebsd-arch@FreeBSD.ORG Thu Jan 15 13:28:35 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5D2B216A4CE; Thu, 15 Jan 2004 13:28:35 -0800 (PST) Received: from kientzle.com (h-66-166-149-50.SNVACAID.covad.net [66.166.149.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7FF8643D53; Thu, 15 Jan 2004 13:28:32 -0800 (PST) (envelope-from kientzle@acm.org) Received: from acm.org ([66.166.149.54]) by kientzle.com (8.12.9/8.12.9) with ESMTP id i0FLSVkX080202; Thu, 15 Jan 2004 13:28:32 -0800 (PST) (envelope-from kientzle@acm.org) Message-ID: <400705FF.7090709@acm.org> Date: Thu, 15 Jan 2004 13:28:31 -0800 From: Tim Kientzle User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.4) Gecko/20031006 X-Accept-Language: en-us, en MIME-Version: 1.0 To: David Schultz References: <4004D445.7020205@acm.org> <20040115182532.GA26149@VARK.homeunix.com> <4006E337.7000404@acm.org> <20040115204458.GA27244@VARK.homeunix.com> In-Reply-To: <20040115204458.GA27244@VARK.homeunix.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit cc: freebsd-arch@FreeBSD.ORG Subject: Re: Request for Comments: libarchive, bsdtar X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: kientzle@acm.org List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Jan 2004 21:28:35 -0000 David Schultz wrote: > > ... the package tools don't have an archive format > that supports random access, so they wind up extracting a > temporary copy of the whole archive, then processing the parts > that they want. Funny you should mention this. As it turns out, random access is not necessary to avoid the temporary copy. To prove it, I've written a pkg_add replacement that extracts directly to the final location in a single pass with no temporary directory and no need for random access to the package file. In particular, it can extract and install a package directly from stdin or an FTP site with no disk overhead at all. How? Easy. Use libarchive to extract the packing list---which is always the first archive entry---directly into memory. Parse the packing list to find out where everything goes, then extract files from the package directly to their final location. Anything that needs to be ignored just gets skipped over. No random access necessary. My prototype pkg_add works and it's about three times faster than the pkg_add in the tree. However, it's also a bit of a mess. libarchive is a big step towards cleaning up that mess by separating out the tarfile reading code. My next project is to separate out some of the generic package management functions into a libpkg library. At that point, I should be able to overhaul my prototype to use these libraries and finally have something production-quality. It is certainly possible that cleaner handling of package files might allow us to eventually use package-management tools for the sysinstall distribution files, but that's certainly not a short-term concern. My goal right now is just to clean up the existing package tools. In particular, my new package tools will: * have the same names as the current tools * use the same command-line options (as far as possible ) * use the exact same file format as the current tools * be built from general-purpose modular library components This last item is key. This will make it easier to maintain and improve these tools over time with new capabilities (better conflict management, for example) and even new kinds of package tools. As I said, libarchive is just the first major product of this program. There's still a lot of work to do, but I have a roadmap and I'll keep chugging down it as my limited time allows. Tim From owner-freebsd-arch@FreeBSD.ORG Thu Jan 15 13:41:58 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 724AA16A4CE for ; Thu, 15 Jan 2004 13:41:58 -0800 (PST) Received: from VARK.homeunix.com (adsl-68-124-78-95.dsl.pltn13.pacbell.net [68.124.78.95]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8DA1643D53 for ; Thu, 15 Jan 2004 13:41:56 -0800 (PST) (envelope-from das@FreeBSD.ORG) Received: from VARK.homeunix.com (localhost [127.0.0.1]) by VARK.homeunix.com (8.12.10/8.12.10) with ESMTP id i0FLffKu027671; Thu, 15 Jan 2004 13:41:41 -0800 (PST) (envelope-from das@FreeBSD.ORG) Received: (from das@localhost) by VARK.homeunix.com (8.12.10/8.12.10/Submit) id i0FLffjL027670; Thu, 15 Jan 2004 13:41:41 -0800 (PST) (envelope-from das@FreeBSD.ORG) Date: Thu, 15 Jan 2004 13:41:41 -0800 From: David Schultz To: Tim Kientzle Message-ID: <20040115214141.GA27632@VARK.homeunix.com> Mail-Followup-To: Tim Kientzle , freebsd-arch@FreeBSD.ORG References: <4004D445.7020205@acm.org> <20040115182532.GA26149@VARK.homeunix.com> <4006E337.7000404@acm.org> <20040115204458.GA27244@VARK.homeunix.com> <400705FF.7090709@acm.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <400705FF.7090709@acm.org> cc: freebsd-arch@FreeBSD.ORG Subject: Re: Request for Comments: libarchive, bsdtar X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Jan 2004 21:41:58 -0000 On Thu, Jan 15, 2004, Tim Kientzle wrote: > David Schultz wrote: > > > >... the package tools don't have an archive format > >that supports random access, so they wind up extracting a > >temporary copy of the whole archive, then processing the parts > >that they want. > > Funny you should mention this. > > As it turns out, random access is not necessary to avoid the > temporary copy. To prove it, I've written a pkg_add replacement > that extracts directly to the final location in a single > pass with no temporary directory and no need for random > access to the package file. In particular, it can extract > and install a package directly from stdin or an FTP site > with no disk overhead at all. [...] > My goal right now is just to clean up the existing package tools. > In particular, my new package tools will: > * have the same names as the current tools > * use the same command-line options (as far as possible ) > * use the exact same file format as the current tools > * be built from general-purpose modular library components I have only one thing to say in response to this news: SWEEEEET! From owner-freebsd-arch@FreeBSD.ORG Fri Jan 16 03:16:18 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 40F1C16A4CE; Fri, 16 Jan 2004 03:16:18 -0800 (PST) Received: from smtp.des.no (flood.des.no [217.116.83.31]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1D53643D45; Fri, 16 Jan 2004 03:15:04 -0800 (PST) (envelope-from des@des.no) Received: by smtp.des.no (Pony Express, from userid 666) id 8CB20530A; Fri, 16 Jan 2004 12:12:33 +0100 (CET) Received: from dwp.des.no (des.no [80.203.228.37]) by smtp.des.no (Pony Express) with ESMTP id 33AA25308; Fri, 16 Jan 2004 12:12:25 +0100 (CET) Received: by dwp.des.no (Postfix, from userid 2602) id D311133C9A; Fri, 16 Jan 2004 12:12:24 +0100 (CET) To: kientzle@acm.org References: <4004D445.7020205@acm.org> <20040115182532.GA26149@VARK.homeunix.com> <4006E337.7000404@acm.org> <20040115204458.GA27244@VARK.homeunix.com> <400705FF.7090709@acm.org> From: des@des.no (Dag-Erling =?iso-8859-1?q?Sm=F8rgrav?=) Date: Fri, 16 Jan 2004 12:12:24 +0100 In-Reply-To: <400705FF.7090709@acm.org> (Tim Kientzle's message of "Thu, 15 Jan 2004 13:28:31 -0800") Message-ID: User-Agent: Gnus/5.090024 (Oort Gnus v0.24) Emacs/21.3 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-Spam-Checker-Version: SpamAssassin 2.61 (1.212.2.1-2003-12-09-exp) on flood.des.no X-Spam-Level: ss X-Spam-Status: No, hits=2.6 required=5.0 tests=RCVD_IN_DYNABLOCK, RCVD_IN_SORBS autolearn=no version=2.61 cc: David Schultz cc: freebsd-arch@FreeBSD.ORG Subject: Re: Request for Comments: libarchive, bsdtar X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Jan 2004 11:16:18 -0000 Tim Kientzle writes: > How? Easy. Use libarchive to extract the packing > list---which is always the first archive entry--- Ass, U, Me. DES --=20 Dag-Erling Sm=F8rgrav - des@des.no From owner-freebsd-arch@FreeBSD.ORG Fri Jan 16 09:52:41 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6C35916A4CE; Fri, 16 Jan 2004 09:52:41 -0800 (PST) Received: from kientzle.com (h-66-166-149-50.SNVACAID.covad.net [66.166.149.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id ED8CB43D5A; Fri, 16 Jan 2004 09:52:34 -0800 (PST) (envelope-from kientzle@acm.org) Received: from acm.org ([66.166.149.54]) by kientzle.com (8.12.9/8.12.9) with ESMTP id i0GHqVkX083957; Fri, 16 Jan 2004 09:52:31 -0800 (PST) (envelope-from kientzle@acm.org) Message-ID: <400824DF.6060604@acm.org> Date: Fri, 16 Jan 2004 09:52:31 -0800 From: Tim Kientzle User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.4) Gecko/20031006 X-Accept-Language: en-us, en MIME-Version: 1.0 To: =?ISO-8859-1?Q?Dag-Erling_Sm=F8rgrav?= References: <4004D445.7020205@acm.org> <20040115182532.GA26149@VARK.homeunix.com> <4006E337.7000404@acm.org> <20040115204458.GA27244@VARK.homeunix.com> <400705FF.7090709@acm.org> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit cc: David Schultz cc: freebsd-arch@FreeBSD.ORG Subject: Re: Request for Comments: libarchive, bsdtar X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: kientzle@acm.org List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Jan 2004 17:52:41 -0000 Dag-Erling Smørgrav wrote: > Tim Kientzle writes: > >>How? Easy. Use libarchive to extract the packing >>list---which is always the first archive entry--- > > Ass, U, Me. The current package tools do gaurantee that the packing list is always first in the archive. This was done a long time ago in an attempt to speed up pkg_add (which currently forks tar twice, once to extract just the packing list and then again to extract everything). Tim From owner-freebsd-arch@FreeBSD.ORG Fri Jan 16 11:03:13 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1384A16A4CE for ; Fri, 16 Jan 2004 11:03:13 -0800 (PST) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id A1A6C43D1D for ; Fri, 16 Jan 2004 11:02:43 -0800 (PST) (envelope-from robert@fledge.watson.org) Received: from fledge.watson.org (localhost [127.0.0.1]) by fledge.watson.org (8.12.10/8.12.10) with ESMTP id i0GJ0uUd095312 for ; Fri, 16 Jan 2004 14:00:56 -0500 (EST) (envelope-from robert@fledge.watson.org) Received: from localhost (robert@localhost)i0GJ0umE095309 for ; Fri, 16 Jan 2004 14:00:56 -0500 (EST) (envelope-from robert@fledge.watson.org) Date: Fri, 16 Jan 2004 14:00:56 -0500 (EST) From: Robert Watson X-Sender: robert@fledge.watson.org To: arch@FreeBSD.org Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Subject: Signal delivery to kernel threads/processes? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Jan 2004 19:03:13 -0000 Bill Paul raised an interesting question with me recently -- he observed that a userspace process running with root privileges could deliver a signal to a kthread, and that this might produce undesired behavior. I was sure that, at some point, we had a check disallowing this, but I don't see it in either RELENG_4 or HEAD. Do we rely on the ability to send/receive signals to interrupt kthreads, that we know of? While the signal delivery itself doesn't make sense, waking up a tsleep() with PCATCH could well be useful behavior. Should a kthread have to declare if it wants to receive interruptions? Given plans, at some point, to make kthreads be real threads, and be part of a kernel process, that would raise some challenges for code relying on the ability to be interrupted with a signal in kernel space, as signals generated by kill() are targetted at processes, not threads. Any thoughts? It's tempting simply to add the following to cr_cansignal() to at least disallow sourcing the signals in userspace: if (p->p_flag & P_SYSTEM) return (EPERM); But I don't have a broad enough view of what goes on in the kernel to reason about what disasters this might cause if signalling is relied on. Robert N M Watson FreeBSD Core Team, TrustedBSD Projects robert@fledge.watson.org Senior Research Scientist, McAfee Research From owner-freebsd-arch@FreeBSD.ORG Fri Jan 16 11:17:41 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: by hub.freebsd.org (Postfix, from userid 683) id 1DF4A16A4CF; Fri, 16 Jan 2004 11:17:41 -0800 (PST) Date: Fri, 16 Jan 2004 11:17:41 -0800 From: Eivind Eklund To: freebsd-arch@FreeBSD.org Message-ID: <20040116191741.GA60158@FreeBSD.org> Mime-Version: 1.0 Content-Type: text/plain; charset=unknown-8bit Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.4.1i Subject: Re: Request for Comments: libarchive, bsdtar X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Jan 2004 19:17:41 -0000 On Fri, Jan 16, 2004 at 12:12:24PM +0100, Dag-Erling Smørgrav wrote: > Tim Kientzle writes: > > How? Easy. Use libarchive to extract the packing > > list---which is always the first archive entry--- > > Ass, U, Me. I believe the packing list being first in the archive is part of the definition of the package format, and that it was done this way to make it should be possible to access the way Tim describes. If my memory fails me, I suggest we make it a required invariant. Eivind. From owner-freebsd-arch@FreeBSD.ORG Fri Jan 16 11:18:35 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5E83F16A4CE for ; Fri, 16 Jan 2004 11:18:35 -0800 (PST) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1F4A843D1F for ; Fri, 16 Jan 2004 11:18:34 -0800 (PST) (envelope-from robert@fledge.watson.org) Received: from fledge.watson.org (localhost [127.0.0.1]) by fledge.watson.org (8.12.10/8.12.10) with ESMTP id i0GJGlUd095866 for ; Fri, 16 Jan 2004 14:16:47 -0500 (EST) (envelope-from robert@fledge.watson.org) Received: from localhost (robert@localhost)i0GJGlmp095863 for ; Fri, 16 Jan 2004 14:16:47 -0500 (EST) (envelope-from robert@fledge.watson.org) Date: Fri, 16 Jan 2004 14:16:46 -0500 (EST) From: Robert Watson X-Sender: robert@fledge.watson.org To: arch@FreeBSD.org In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Subject: Re: Signal delivery to kernel threads/processes? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Jan 2004 19:18:35 -0000 On Fri, 16 Jan 2004, Robert Watson wrote: > if (p->p_flag & P_SYSTEM) > return (EPERM); Another possible interpretation, not quite the same, might be to use P_KTHREAD. Robert N M Watson FreeBSD Core Team, TrustedBSD Projects robert@fledge.watson.org Senior Research Scientist, McAfee Research From owner-freebsd-arch@FreeBSD.ORG Fri Jan 16 11:21:30 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8431016A4CE for ; Fri, 16 Jan 2004 11:21:30 -0800 (PST) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 11F7A43D64 for ; Fri, 16 Jan 2004 11:21:17 -0800 (PST) (envelope-from robert@fledge.watson.org) Received: from fledge.watson.org (localhost [127.0.0.1]) by fledge.watson.org (8.12.10/8.12.10) with ESMTP id i0GJJUUd096008 for ; Fri, 16 Jan 2004 14:19:30 -0500 (EST) (envelope-from robert@fledge.watson.org) Received: from localhost (robert@localhost)i0GJJUmA096005 for ; Fri, 16 Jan 2004 14:19:30 -0500 (EST) (envelope-from robert@fledge.watson.org) Date: Fri, 16 Jan 2004 14:19:30 -0500 (EST) From: Robert Watson X-Sender: robert@fledge.watson.org To: arch@FreeBSD.org In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Subject: Re: Signal delivery to kernel threads/processes? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Jan 2004 19:21:30 -0000 On Fri, 16 Jan 2004, Robert Watson wrote: > On Fri, 16 Jan 2004, Robert Watson wrote: > > > if (p->p_flag & P_SYSTEM) > > return (EPERM); > > Another possible interpretation, not quite the same, might be to use > P_KTHREAD. And, to add insult to injury by following up to my post yet another time: init has P_SYSTEM set, so P_KTHREAD is a much better choice. Either that or introduce a P_KERNEL flag. Robert N M Watson FreeBSD Core Team, TrustedBSD Projects robert@fledge.watson.org Senior Research Scientist, McAfee Research From owner-freebsd-arch@FreeBSD.ORG Fri Jan 16 11:59:43 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1963516A4CE for ; Fri, 16 Jan 2004 11:59:43 -0800 (PST) Received: from mail6.speakeasy.net (mail6.speakeasy.net [216.254.0.206]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4BBEE43D45 for ; Fri, 16 Jan 2004 11:59:28 -0800 (PST) (envelope-from john@baldwin.cx) Received: (qmail 9209 invoked from network); 16 Jan 2004 19:59:27 -0000 Received: from dsl027-160-063.atl1.dsl.speakeasy.net (HELO server.baldwin.cx) ([216.27.160.63]) (envelope-sender ) encrypted SMTP for ; 16 Jan 2004 19:59:27 -0000 Received: from 10.50.40.206 (gw1.twc.weather.com [216.133.140.1]) by server.baldwin.cx (8.12.10/8.12.10) with ESMTP id i0GJxMM0088588; Fri, 16 Jan 2004 14:59:23 -0500 (EST) (envelope-from john@baldwin.cx) From: John Baldwin To: Eivind Eklund , freebsd-arch@FreeBSD.org Date: Fri, 16 Jan 2004 14:26:53 -0500 User-Agent: KMail/1.5.4 References: <20040116191741.GA60158@FreeBSD.org> In-Reply-To: <20040116191741.GA60158@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Message-Id: <200401161426.53446.john@baldwin.cx> X-Spam-Checker-Version: SpamAssassin 2.55 (1.174.2.19-2003-05-19-exp) Subject: Re: Request for Comments: libarchive, bsdtar X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Jan 2004 19:59:43 -0000 On Friday 16 January 2004 02:17 pm, Eivind Eklund wrote: > On Fri, Jan 16, 2004 at 12:12:24PM +0100, Dag-Erling Sm=F8rgrav wrote: > > Tim Kientzle writes: > > > How? Easy. Use libarchive to extract the packing > > > list---which is always the first archive entry--- > > > > Ass, U, Me. > > I believe the packing list being first in the archive is part of > the definition of the package format, and that it was done this > way to make it should be possible to access the way Tim describes. > > If my memory fails me, I suggest we make it a required invariant. Yes, this seems to be a valid requirement and one that the existing=20 pkg_create(8) already fulfills. =2D-=20 John Baldwin <>< http://www.baldwin.cx/~john/ "Power Users Use the Power to Serve" =3D http://www.FreeBSD.org From owner-freebsd-arch@FreeBSD.ORG Fri Jan 16 13:34:37 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1A3E916A4CE; Fri, 16 Jan 2004 13:34:37 -0800 (PST) Received: from VARK.homeunix.com (adsl-68-124-78-95.dsl.pltn13.pacbell.net [68.124.78.95]) by mx1.FreeBSD.org (Postfix) with ESMTP id C819B43D2F; Fri, 16 Jan 2004 13:34:35 -0800 (PST) (envelope-from das@FreeBSD.ORG) Received: from VARK.homeunix.com (localhost [127.0.0.1]) by VARK.homeunix.com (8.12.10/8.12.10) with ESMTP id i0GLYIKu027678; Fri, 16 Jan 2004 13:34:18 -0800 (PST) (envelope-from das@FreeBSD.ORG) Received: (from das@localhost) by VARK.homeunix.com (8.12.10/8.12.10/Submit) id i0GLYIpI027677; Fri, 16 Jan 2004 13:34:18 -0800 (PST) (envelope-from das@FreeBSD.ORG) Date: Fri, 16 Jan 2004 13:34:18 -0800 From: David Schultz To: Robert Watson Message-ID: <20040116213418.GA27542@VARK.homeunix.com> Mail-Followup-To: Robert Watson , arch@FreeBSD.ORG References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: cc: arch@FreeBSD.ORG Subject: Re: Signal delivery to kernel threads/processes? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Jan 2004 21:34:37 -0000 On Fri, Jan 16, 2004, Robert Watson wrote: > Bill Paul raised an interesting question with me recently -- he observed > that a userspace process running with root privileges could deliver a > signal to a kthread, and that this might produce undesired behavior. I > was sure that, at some point, we had a check disallowing this, but I don't > see it in either RELENG_4 or HEAD. Do we rely on the ability to > send/receive signals to interrupt kthreads, that we know of? While the > signal delivery itself doesn't make sense, waking up a tsleep() with > PCATCH could well be useful behavior. Should a kthread have to declare if > it wants to receive interruptions? Given plans, at some point, to make > kthreads be real threads, and be part of a kernel process, that would > raise some challenges for code relying on the ability to be interrupted > with a signal in kernel space, as signals generated by kill() are > targetted at processes, not threads. > > Any thoughts? It's tempting simply to add the following to cr_cansignal() > to at least disallow sourcing the signals in userspace: > > if (p->p_flag & P_SYSTEM) > return (EPERM); I believe nfsd needs to accept signals so that it can be shut down. But now that I think about what you've said, I wonder what this implies for system processes such as aiod that switch their credentials to those of the request they are processing. It would seem that in the window during which they have the ordinary user's uid, they could be killed by an ordinary user. I hope there's already some safeguard against this that I don't know about... From owner-freebsd-arch@FreeBSD.ORG Fri Jan 16 14:49:19 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6697E16A4CE; Fri, 16 Jan 2004 14:49:19 -0800 (PST) Received: from mailman.zeta.org.au (mailman.zeta.org.au [203.26.10.16]) by mx1.FreeBSD.org (Postfix) with ESMTP id 55EAB43D80; Fri, 16 Jan 2004 14:49:00 -0800 (PST) (envelope-from bde@zeta.org.au) Received: from gamplex.bde.org (katana.zip.com.au [61.8.7.246]) by mailman.zeta.org.au (8.9.3p2/8.8.7) with ESMTP id JAA04973; Sat, 17 Jan 2004 09:48:58 +1100 Date: Sat, 17 Jan 2004 09:48:57 +1100 (EST) From: Bruce Evans X-X-Sender: bde@gamplex.bde.org To: Robert Watson In-Reply-To: Message-ID: <20040117093324.M6180@gamplex.bde.org> References: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: arch@FreeBSD.org Subject: Re: Signal delivery to kernel threads/processes? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Jan 2004 22:49:19 -0000 On Fri, 16 Jan 2004, Robert Watson wrote: > On Fri, 16 Jan 2004, Robert Watson wrote: > > > On Fri, 16 Jan 2004, Robert Watson wrote: > > > > > if (p->p_flag & P_SYSTEM) > > > return (EPERM); > > > > Another possible interpretation, not quite the same, might be to use > > P_KTHREAD. > > And, to add insult to injury by following up to my post yet another time: > init has P_SYSTEM set, so P_KTHREAD is a much better choice. Either that > or introduce a P_KERNEL flag. Setting P_SYSTEM for init is a bug. It also breaks debugging. Setting it for init seems to be just a bug in init_main.c 1.124. I have run without this setting for many years. I noticed the breakage mainly for "cat /proc/1/map". Bruce From owner-freebsd-arch@FreeBSD.ORG Fri Jan 16 15:16:54 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A984816A4CE; Fri, 16 Jan 2004 15:16:54 -0800 (PST) Received: from kientzle.com (h-66-166-149-50.SNVACAID.covad.net [66.166.149.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 24CFB43D3F; Fri, 16 Jan 2004 15:16:53 -0800 (PST) (envelope-from kientzle@acm.org) Received: from acm.org ([66.166.149.54]) by kientzle.com (8.12.9/8.12.9) with ESMTP id i0GNGqkX085545; Fri, 16 Jan 2004 15:16:52 -0800 (PST) (envelope-from kientzle@acm.org) Message-ID: <400870E3.8040508@acm.org> Date: Fri, 16 Jan 2004 15:16:51 -0800 From: Tim Kientzle User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.4) Gecko/20031006 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Don Lewis References: <200401150537.i0F5bo7E041482@gw.catspoiler.org> In-Reply-To: <200401150537.i0F5bo7E041482@gw.catspoiler.org> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit cc: tjr@FreeBSD.org cc: freebsd-arch@FreeBSD.org Subject: Re: Request for Comments: libarchive, bsdtar X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: kientzle@acm.org List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Jan 2004 23:16:54 -0000 On 15 Jan, Tim Robbins wrote: >- Padding gzip'd tar archives (with bsdtar czf) causes [various problems] I've updated libarchive so that: * last-block padding is set with a separate API call that can be invoked at anytime before the archive is closed (In particular, it can be invoked from within the client open callback.) * If it is not set manually, then the default behavior is: = uncompressed data within a gzip/bzip2 compressed stream is always padded = if archive_write_open_file is used, then the last block is padded if the output is stdout or a character or block device, otherwise the last block is not padded = if archive_write_open_file is not used, then the "default default" behavior is for the last block to not be padded. This may change. This appears to match the behavior of gtar. I've updated bsdtar to simply use the library defaults. Don Lewis wrote: > I would prefer to have explicit control of this behavior. > BTW, ... an enhanced version of dd ... had the useful option "mobs", which was > used to specify a minimum output block size. libarchive now has an API function: archive_write_set_bytes_in_last_block Unfortunately, the name is a bit misleading; suggestions appreciated. As a special case, if the argument to this function is zero, the last block will be padded to the full block size. Otherwise, the last block will be padded to a multiple of the indicated value. For example, if you specify 1024, and the block was 1025 bytes, it will get padded to 2048. If you specify 1, no padding will be added. However, in no case will the last block be padded to be larger than the archive block size (as set with archive_write_set_bytes_per_block). As described above, the archive_write_open_file function will set this for you only if you have not already invoked it manually. If you use the low-level archive_write_open, then you're on your own. Does this provide the "explicit control" you were looking for? Tim Kientzle From owner-freebsd-arch@FreeBSD.ORG Fri Jan 16 15:33:31 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 95A3B16A4CE; Fri, 16 Jan 2004 15:33:31 -0800 (PST) Received: from gw.catspoiler.org (217-ip-163.nccn.net [209.79.217.163]) by mx1.FreeBSD.org (Postfix) with ESMTP id 406DB43D62; Fri, 16 Jan 2004 15:33:20 -0800 (PST) (envelope-from truckman@FreeBSD.org) Received: from FreeBSD.org (mousie.catspoiler.org [192.168.101.2]) by gw.catspoiler.org (8.12.9p2/8.12.9) with ESMTP id i0GNXA7E047016; Fri, 16 Jan 2004 15:33:14 -0800 (PST) (envelope-from truckman@FreeBSD.org) Message-Id: <200401162333.i0GNXA7E047016@gw.catspoiler.org> Date: Fri, 16 Jan 2004 15:33:10 -0800 (PST) From: Don Lewis To: kientzle@acm.org In-Reply-To: <400870E3.8040508@acm.org> MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii cc: tjr@FreeBSD.org cc: freebsd-arch@FreeBSD.org Subject: Re: Request for Comments: libarchive, bsdtar X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Jan 2004 23:33:31 -0000 On 16 Jan, Tim Kientzle wrote: > On 15 Jan, Tim Robbins wrote: >>- Padding gzip'd tar archives (with bsdtar czf) causes [various problems] > > I've updated libarchive so that: > * last-block padding is set with a separate API call that can > be invoked at anytime before the archive is closed > (In particular, it can be invoked from within the client open > callback.) > * If it is not set manually, then the default behavior is: > = uncompressed data within a gzip/bzip2 compressed stream > is always padded > = if archive_write_open_file is used, then the last block > is padded if the output is stdout or a character or block > device, otherwise the last block is not padded > = if archive_write_open_file is not used, then the > "default default" behavior is for the last block > to not be padded. This may change. > > This appears to match the behavior of gtar. > I've updated bsdtar to simply use the library defaults. > > Don Lewis wrote: >> I would prefer to have explicit control of this behavior. >> BTW, ... an enhanced version of dd ... had the useful option "mobs", which was >> used to specify a minimum output block size. > > libarchive now has an API function: archive_write_set_bytes_in_last_block > > Unfortunately, the name is a bit misleading; suggestions appreciated. > > As a special case, if the argument to this function is zero, the > last block will be padded to the full block size. Otherwise, the > last block will be padded to a multiple of the indicated value. > For example, if you specify 1024, and the block was 1025 bytes, it will > get padded to 2048. If you specify 1, no padding will be added. > However, in no case will the last block be padded to be larger than the > archive block size (as set with archive_write_set_bytes_per_block). > > As described above, the archive_write_open_file function will > set this for you only if you have not already invoked it manually. > If you use the low-level archive_write_open, then you're on your own. > > Does this provide the "explicit control" you were looking for? Sounds good to me. From owner-freebsd-arch@FreeBSD.ORG Sat Jan 17 01:57:45 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DBA5116A4CE; Sat, 17 Jan 2004 01:57:45 -0800 (PST) Received: from gw.catspoiler.org (217-ip-163.nccn.net [209.79.217.163]) by mx1.FreeBSD.org (Postfix) with ESMTP id A405643D4C; Sat, 17 Jan 2004 01:57:44 -0800 (PST) (envelope-from truckman@FreeBSD.org) Received: from FreeBSD.org (mousie.catspoiler.org [192.168.101.2]) by gw.catspoiler.org (8.12.9p2/8.12.9) with ESMTP id i0H9va7E047876; Sat, 17 Jan 2004 01:57:40 -0800 (PST) (envelope-from truckman@FreeBSD.org) Message-Id: <200401170957.i0H9va7E047876@gw.catspoiler.org> Date: Sat, 17 Jan 2004 01:57:36 -0800 (PST) From: Don Lewis To: rwatson@FreeBSD.org In-Reply-To: MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii cc: arch@FreeBSD.org Subject: Re: Signal delivery to kernel threads/processes? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 Jan 2004 09:57:46 -0000 On 16 Jan, Robert Watson wrote: > > Bill Paul raised an interesting question with me recently -- he observed > that a userspace process running with root privileges could deliver a > signal to a kthread, and that this might produce undesired behavior. I > was sure that, at some point, we had a check disallowing this, but I don't > see it in either RELENG_4 or HEAD. Do we rely on the ability to > send/receive signals to interrupt kthreads, that we know of? While the > signal delivery itself doesn't make sense, waking up a tsleep() with > PCATCH could well be useful behavior. Should a kthread have to declare if > it wants to receive interruptions? Given plans, at some point, to make > kthreads be real threads, and be part of a kernel process, that would > raise some challenges for code relying on the ability to be interrupted > with a signal in kernel space, as signals generated by kill() are > targetted at processes, not threads. > > Any thoughts? It's tempting simply to add the following to cr_cansignal() > to at least disallow sourcing the signals in userspace: > > if (p->p_flag & P_SYSTEM) > return (EPERM); > > But I don't have a broad enough view of what goes on in the kernel to > reason about what disasters this might cause if signalling is relied on. The only thing I worried about is what happens to kthreads on shutdown. It looks like this is handled by kthread_suspend() which tells the thread that it has received a SIGSTOP, but this isn't done with the normal signal delivery mechanism.