From owner-freebsd-ports Wed Nov 22 08:35:45 1995 Return-Path: owner-ports Received: (from root@localhost) by freefall.freebsd.org (8.6.12/8.6.6) id IAA24061 for ports-outgoing; Wed, 22 Nov 1995 08:35:45 -0800 Received: from time.cdrom.com (time.cdrom.com [192.216.222.226]) by freefall.freebsd.org (8.6.12/8.6.6) with ESMTP id IAA24054 for ; Wed, 22 Nov 1995 08:35:36 -0800 Received: from localhost (localhost [127.0.0.1]) by time.cdrom.com (8.6.12/8.6.9) with SMTP id IAA03054; Wed, 22 Nov 1995 08:35:08 -0800 To: Michael Smith cc: asami@cs.berkeley.edu (Satoshi Asami), ports@FreeBSD.ORG Subject: Re: Proposal 3: Non-executable file in *_DEPEND In-reply-to: Your message of "Wed, 22 Nov 1995 06:15:35 GMT." <199511220615.GAA25100@genesis.atrad.adelaide.edu.au> Date: Wed, 22 Nov 1995 08:35:08 -0800 Message-ID: <3052.817058108@time.cdrom.com> From: "Jordan K. Hubbard" Sender: owner-ports@FreeBSD.ORG Precedence: bulk > Well, I think there are a number of people who might be willing to spend some > time writing a nice frontend; I'm currently reasonably enthused about doing > some work on a "pkg" tool, given that most of my other "great ideas" have > stonewalled 8( > > > This is already recommended, although not mandatory. I'm planning tho > > bring this up as "proposal 5" or something. :) > > Of curiosity, have you ever looked at "setld" under Ultrix? (OSF/1?) It's > one of the most horrific pieces of sh programming I've ever met, but it > really sets the basic standard for what we need. OK, here's my take on what needs to be done: 1. The existing package format loses and needs to die. More on that in a moment. 2. We need a very simple 3D filesystem library (yes, *library*). 3. We need a box of GUI tools. 4. We need applications to paper over all these clever mechanisms. --- 1. The existing package format needs to die for several reasons, the foremost being: o The "plist" syntax loses, lacks conditionals, is arcane, does not handle checksum verification, preserves insufficient information for proper deletion, sucks, is evil, gag, retch, puke. o Having the packages be gzip'd tarballs does not allow sufficient random access and requires hateful amounts of temporary storage. This is why there are "distributions" and there are "packages" and the twain have never met. That bites, because they're both different solutions for the same problem. o The idea of letting packages do their installation magic via shell scripts was one of the stupid ideas in the known universe. Yes, it's very convenient, but it's also a MASSIVELY LOSING SECURITY HOLE! One little package is capable of reducing your system to a smoking crater at its merest whim, or it can decide to be spiteful and trojan-horse you up the wazoo! Since packages aren't "signed" in any special way, you don't even know if the package you see on that archive is one of ours, or one of "theirs." That goes beyond evil - that is evil incarnate. o Too many other reasons to mention. Everything under /usr/src/usr.sbin/pkg_install needs to be toasted with a flamethrower, the remains of the disk it was on bombarded with hard gamma radiation, crushed into tiny fragments and dumped like confetti over the north sea. This code is *begging* to die. So what to put in its place? Well, I have some ideas. For one thing, the PLIST needs to go away and be replaced by a genuine interpreted language. I'm not saying forth, I'm not even saying TCL (though it sounds promising), maybe even my little C interpreter since I got it going again - it does a lot of things that TCL doesn't do, like compile and run on a virtual machine. In any case, the important thing is to make plists much more flexible, eliminate the idea of external require/install/deinstall scripts and use the language instead (so that you can do the equivalent of safe-tcl and declare certain operations off-limits for untrusted packages). We can also bolt in concepts like MD5 conflict detection and such a lot more easily with a system like this. Secondly, the package format itself needs to support the idea of extracting just the bootstrap segment or description info from the package (quickly) and allowing the bootstrap to unpack the package directly in-place (or through the 3DFS library) without going through an intermediate directory. Many of these component technologies could be worked on separately by different people (in fact, I almost recommend it) and then assembled into a complete system. 2. A very simple 3D filesystem library. What is he *talking* about? Don't worry - it's not as complicated as it sounds. Consider, for a moment, the life of your average /usr/local or /usr/src directory. Things come in and out, are updated, sometimes an update turns out to be a pain so we back off again, etc. The interdependencies that exist (e.g. /usr/X11R6/bin/fvwm depends on /usr/X11R6/lib/libXpm.so.3.4) also aren't really described in the filesystem, they're someplace else (either somebody's head, a Makefile or a package). This forces all tools to reinvent the wheel when it comes to keeping track of what they've done and who they depend on, and it leads to failure-prone implementations that only do crude first approximations of journaling, like the one in pkg_install. Since we're going to be right back at this point again the minute we start doing real patch maintainance on top of our releases, and we'd also really like to just shove all this stuff together so that the distinction between installing "distributions" and installing "packages" disappears anyway, what we really need is a unified solution for everything from /usr/X11R6 to /usr/src to /usr/bin (yes! binary updates!). Here's what I recommend: We implement a simple 3D filesystem model where it's possible to set the "pointer" on a filesystem (or some sub-section thereof) to any point in time using the 3DFS API. The pointer could be absolute, meaning that it caused the master copy to be changed, or it could be relative to a link tree created as a "shadow" of the master. Time stamp information would come out of a central database at the top of the tree, or some location specified by environment variable. The delta (history) information would be stored in the tree itself, with each directory containing the "undo" information required to roll back any file. I'll attempt to explain this in more detail: Let's say you have /usr/src under control of the 3DFS - you've called ``3dfs_init("/usr/src")'', or whatever API library call is provided for the purpose, and now there's a /usr/src/.3DFS file in place (note: I'm not saying it'd have to live there - I can see designing this to allow all of it to take place in a completely different dir as well). Now say you want to install the next version of the FreeBSD src distribution *splat* right on top, but you have the disk space to spare and you'd like to be able to get back to (or even just compare with) the previous version. Well, since the FreeBSD src dist would now be a number of "packages" in the newer scheme of things, the package adder would call a 3DFS API call for each prefix dir it was about to attack and essentially say "*knock* *knock* Can I come in? I have all these file updates for you under the heading `RELENG_2_1_1'." The 3DFS library would search for all the requisite database files, making sure it could lock and write to them, verify that the key was unique, and if all was cool and groovy it would say: "Sure! But be sure to use *my* API calls now for actually copying stuff into place!" The package installer would then assume a "hands off" approach to things under that prefix and hand 3DFS the file updates though a pipe created for that purpose by the API (anyone having better suggestions for avoiding temp files is welcome to contact me). The 3DFS code would look at the proposed file and calculate how best to store an "undo" record between the current version and the proposed version. If the change was really small, it might generate a context diff internally. If the change description was bigger than the file, it'd just save the old file entirely (like the old patch kit used to do). In any case, the 3DFS would handle all the sticky details of making sure you could move back to a previous state. It would also store dependency information for you, so you could choose to have things blip out of existance if something they depended upon was invalidated by the pointer position (or perform some other trickery). More on the sticky details? OK. The 3DFS's database would (at first ponder) consist of entries like this: +--------------------+ | textual description| +--------------------+ | date stamp | +--------------------+ | unique hash key | +--------------------+ | previous delta | +--------------------+ | misc info on entry | +--------------------+ | file spec 1 | +--------------------+ | ... | +--------------------+ | file spec n | +--------------------+ Each "event" description would be checked for uniqueness and the date verified (so we don't trash our database by somebody's screwed up clock :-) before being added to the locked database. We'd also use the date and/or event description to generate a unique 8 character base 36 "key" that describes the event uniquely. This key will be used for, among other things, generating the filename containing the change info between the "previous delta" and this one. If no previous delta exists, this change can be assumed to be a base delta. The delta files would exist under a shadow structure of the tree being managed by 3DFS, e.g. you might see something like: /usr/src/bin/ls/.3ddelta/ls.c/3A97qflz /usr/src/bin/ls/.3ddelta/ls.c/1291rmPd /usr/src/bin/ls/.3ddelta/ls.1/1291rmPd If ls.c and ls.1 had been changed in one revision (hashed by "1291rmPd") and then ls.c changed in a subsequent revision ("3A97qflz"). For tracing back this kind of information more easily, we'll have some sort of `3dlog' command that ferrets it out and looks it up for any given point in the tree. In any case, the important thing is that we be able to have the equivalent of CVS but for any file on the system, be it in /usr/local/bin or /usr/src/bin, and with more of an emphasis on *other* things changing your tree, not you the developer changing it. Implementing 3DFS will give us package removals that finally work and dependency records that actually don't require leaping around and trying to update multiple files (as pkg_add currently does). Best of all, none of these tools will have to sweat the details of roll-back - they'll just use the 3DFS API to make that code do all the work. Maybe we'll even get a new and improved patch kit out of this! :-) This is also actually just the vaguest description of 3DFS, and I've written it out in quite a bit more detail than this (I haven't even covered compaction, etc). But my fingers are already tired today, so anyone really interested in this is encouraged to contact me directly. There is still much work to be done on the *implementation* of this! :-) 3. We need a box of GUI tools. Yes, I'll say it until I'm blue. We need a replacement for libdialog and a bunch of other stuff besides! We need scrolling lists and pull down menus and frames and row-column layout managers and a whole host of things that the Windows weenies take for granted now. We need these so that we can construct the next-generation system installation and management system that the world has been crying out for, and so that we can properly front-end the packages we have. If you noticed any quick and obvious differences between pkg_manage and sysinstall's package menus, you probably noticed the fact that pkg_manage has horrible navigation but good description and extraction help, whereas sysinstall's package menus had *better* navigation (still not very good) but horrible description and extraction detail (read: none). In both cases, this is largely the result of poor GUI tools. We just don't have the objects available to do the kinds of things we need! Scrolling lists and pull-down menus would certainly do the fdisk and label screens in sysinstall a world of good, if nothing else! This could be done by any reasonably competent ncurses hacker. That would not, unfortunately, include me! :-( Would someone care to perhaps write an ncurses *abstraction* layer for someone like me, so I wouldn't have to figure out how to draw things out of the line drawing character set or properly refresh regions uncovered by pull-down menus? :-) If that were done, I could actually see doing the rest. Just so I don't need to know anything really deep about ncurses to actually draw the objects. This needs to be abstracted *anyway*, actually, so that we can do this under X too when the time comes. 4. We need applications to paper over all these clever mechanisms. Naturally, pkg_add, pkg_delete and friends will have to be re-written to take advantage of all these nice building blocks. So will sysinstall. I don't mind. :-) We will also want to have new commands for reporting on 3DFS information, reclaiming space, invalidating deltas, etc. I don't mind working on a lot of this, if I can get some help! We also need a specification format for the various GUI screens, since nobody wants to hack that stuff out in raw C if they can help it. Does this mean libforms? Something else? Comments? Jordan