Date: Wed, 22 Nov 1995 08:35:08 -0800 From: "Jordan K. Hubbard" <jkh@time.cdrom.com> To: Michael Smith <msmith@atrad.adelaide.edu.au> Cc: asami@cs.berkeley.edu (Satoshi Asami), ports@FreeBSD.ORG Subject: Re: Proposal 3: Non-executable file in *_DEPEND Message-ID: <3052.817058108@time.cdrom.com> In-Reply-To: Your message of "Wed, 22 Nov 1995 06:15:35 GMT." <199511220615.GAA25100@genesis.atrad.adelaide.edu.au>
next in thread | previous in thread | raw e-mail | index | archive | help
> Well, I think there are a number of people who might be willing to spend some
> time writing a nice frontend; I'm currently reasonably enthused about doing
> some work on a "pkg" tool, given that most of my other "great ideas" have
> stonewalled 8(
>
> > This is already recommended, although not mandatory. I'm planning tho
> > bring this up as "proposal 5" or something. :)
>
> Of curiosity, have you ever looked at "setld" under Ultrix? (OSF/1?) It's
> one of the most horrific pieces of sh programming I've ever met, but it
> really sets the basic standard for what we need.
OK, here's my take on what needs to be done:
1. The existing package format loses and needs to die. More on that in a
moment.
2. We need a very simple 3D filesystem library (yes, *library*).
3. We need a box of GUI tools.
4. We need applications to paper over all these clever mechanisms.
---
1. The existing package format needs to die for several reasons, the foremost
being:
o The "plist" syntax loses, lacks conditionals, is arcane,
does not handle checksum verification, preserves insufficient
information for proper deletion, sucks, is evil, gag, retch,
puke.
o Having the packages be gzip'd tarballs does not allow sufficient
random access and requires hateful amounts of temporary storage.
This is why there are "distributions" and there are "packages"
and the twain have never met. That bites, because they're both
different solutions for the same problem.
o The idea of letting packages do their installation magic via
shell scripts was one of the stupid ideas in the known universe.
Yes, it's very convenient, but it's also a MASSIVELY LOSING SECURITY
HOLE! One little package is capable of reducing your system to
a smoking crater at its merest whim, or it can decide to be
spiteful and trojan-horse you up the wazoo! Since packages aren't
"signed" in any special way, you don't even know if the package
you see on that archive is one of ours, or one of "theirs."
That goes beyond evil - that is evil incarnate.
o Too many other reasons to mention. Everything under
/usr/src/usr.sbin/pkg_install needs to be toasted with a
flamethrower, the remains of the disk it was on bombarded with
hard gamma radiation, crushed into tiny fragments and dumped like
confetti over the north sea. This code is *begging* to die.
So what to put in its place? Well, I have some ideas.
For one thing, the PLIST needs to go away and be replaced by a genuine
interpreted language. I'm not saying forth, I'm not even saying TCL
(though it sounds promising), maybe even my little C interpreter since
I got it going again - it does a lot of things that TCL doesn't do,
like compile and run on a virtual machine. In any case, the important
thing is to make plists much more flexible, eliminate the idea of
external require/install/deinstall scripts and use the language
instead (so that you can do the equivalent of safe-tcl and declare
certain operations off-limits for untrusted packages).
We can also bolt in concepts like MD5 conflict detection and such a
lot more easily with a system like this.
Secondly, the package format itself needs to support the idea of
extracting just the bootstrap segment or description info from the
package (quickly) and allowing the bootstrap to unpack the package
directly in-place (or through the 3DFS library) without going through
an intermediate directory.
Many of these component technologies could be worked on separately by
different people (in fact, I almost recommend it) and then assembled
into a complete system.
2. A very simple 3D filesystem library.
What is he *talking* about? Don't worry - it's not as complicated as
it sounds.
Consider, for a moment, the life of your average /usr/local or
/usr/src directory. Things come in and out, are updated, sometimes an
update turns out to be a pain so we back off again, etc. The
interdependencies that exist (e.g. /usr/X11R6/bin/fvwm depends on
/usr/X11R6/lib/libXpm.so.3.4) also aren't really described in the
filesystem, they're someplace else (either somebody's head, a Makefile
or a package).
This forces all tools to reinvent the wheel when it comes to keeping
track of what they've done and who they depend on, and it leads to
failure-prone implementations that only do crude first approximations
of journaling, like the one in pkg_install.
Since we're going to be right back at this point again the minute we
start doing real patch maintainance on top of our releases, and we'd
also really like to just shove all this stuff together so that the
distinction between installing "distributions" and installing
"packages" disappears anyway, what we really need is a unified
solution for everything from /usr/X11R6 to /usr/src to /usr/bin (yes!
binary updates!).
Here's what I recommend:
We implement a simple 3D filesystem model where it's possible to set
the "pointer" on a filesystem (or some sub-section thereof) to any
point in time using the 3DFS API. The pointer could be absolute,
meaning that it caused the master copy to be changed, or it could be
relative to a link tree created as a "shadow" of the master. Time
stamp information would come out of a central database at the top of
the tree, or some location specified by environment variable. The
delta (history) information would be stored in the tree itself, with
each directory containing the "undo" information required to roll back
any file. I'll attempt to explain this in more detail:
Let's say you have /usr/src under control of the 3DFS - you've called
``3dfs_init("/usr/src")'', or whatever API library call is provided
for the purpose, and now there's a /usr/src/.3DFS file in place (note:
I'm not saying it'd have to live there - I can see designing this to
allow all of it to take place in a completely different dir as well).
Now say you want to install the next version of the FreeBSD src
distribution *splat* right on top, but you have the disk space to
spare and you'd like to be able to get back to (or even just compare
with) the previous version.
Well, since the FreeBSD src dist would now be a number of "packages"
in the newer scheme of things, the package adder would call a 3DFS API
call for each prefix dir it was about to attack and essentially say
"*knock* *knock* Can I come in? I have all these file updates for you
under the heading `RELENG_2_1_1'."
The 3DFS library would search for all the requisite database files,
making sure it could lock and write to them, verify that the key was
unique, and if all was cool and groovy it would say: "Sure! But be
sure to use *my* API calls now for actually copying stuff into place!"
The package installer would then assume a "hands off" approach to
things under that prefix and hand 3DFS the file updates though a pipe
created for that purpose by the API (anyone having better suggestions
for avoiding temp files is welcome to contact me). The 3DFS code
would look at the proposed file and calculate how best to store an
"undo" record between the current version and the proposed version.
If the change was really small, it might generate a context diff
internally. If the change description was bigger than the file, it'd
just save the old file entirely (like the old patch kit used to do).
In any case, the 3DFS would handle all the sticky details of making
sure you could move back to a previous state. It would also store
dependency information for you, so you could choose to have things
blip out of existance if something they depended upon was invalidated
by the pointer position (or perform some other trickery).
More on the sticky details? OK. The 3DFS's database would (at first
ponder) consist of entries like this:
+--------------------+
| textual description|
+--------------------+
| date stamp |
+--------------------+
| unique hash key |
+--------------------+
| previous delta |
+--------------------+
| misc info on entry |
+--------------------+
| file spec 1 |
+--------------------+
| ... |
+--------------------+
| file spec n |
+--------------------+
Each "event" description would be checked for uniqueness and the date
verified (so we don't trash our database by somebody's screwed up
clock :-) before being added to the locked database. We'd also use
the date and/or event description to generate a unique 8 character
base 36 "key" that describes the event uniquely. This key will be
used for, among other things, generating the filename containing the
change info between the "previous delta" and this one. If no previous
delta exists, this change can be assumed to be a base delta.
The delta files would exist under a shadow structure of the tree being
managed by 3DFS, e.g. you might see something like:
/usr/src/bin/ls/.3ddelta/ls.c/3A97qflz
/usr/src/bin/ls/.3ddelta/ls.c/1291rmPd
/usr/src/bin/ls/.3ddelta/ls.1/1291rmPd
If ls.c and ls.1 had been changed in one revision (hashed by
"1291rmPd") and then ls.c changed in a subsequent revision
("3A97qflz"). For tracing back this kind of information more easily,
we'll have some sort of `3dlog' command that ferrets it out and looks
it up for any given point in the tree.
In any case, the important thing is that we be able to have the
equivalent of CVS but for any file on the system, be it in
/usr/local/bin or /usr/src/bin, and with more of an emphasis on
*other* things changing your tree, not you the developer changing it.
Implementing 3DFS will give us package removals that finally work and
dependency records that actually don't require leaping around and
trying to update multiple files (as pkg_add currently does). Best of
all, none of these tools will have to sweat the details of roll-back -
they'll just use the 3DFS API to make that code do all the work.
Maybe we'll even get a new and improved patch kit out of this! :-)
This is also actually just the vaguest description of 3DFS, and I've
written it out in quite a bit more detail than this (I haven't even
covered compaction, etc). But my fingers are already tired today, so
anyone really interested in this is encouraged to contact me directly.
There is still much work to be done on the *implementation* of this!
:-)
3. We need a box of GUI tools.
Yes, I'll say it until I'm blue. We need a replacement for libdialog
and a bunch of other stuff besides!
We need scrolling lists and pull down menus and frames and row-column
layout managers and a whole host of things that the Windows weenies
take for granted now. We need these so that we can construct the
next-generation system installation and management system that the
world has been crying out for, and so that we can properly front-end
the packages we have. If you noticed any quick and obvious
differences between pkg_manage and sysinstall's package menus, you
probably noticed the fact that pkg_manage has horrible navigation but
good description and extraction help, whereas sysinstall's package
menus had *better* navigation (still not very good) but horrible
description and extraction detail (read: none).
In both cases, this is largely the result of poor GUI tools. We just
don't have the objects available to do the kinds of things we need!
Scrolling lists and pull-down menus would certainly do the fdisk
and label screens in sysinstall a world of good, if nothing else!
This could be done by any reasonably competent ncurses hacker. That
would not, unfortunately, include me! :-(
Would someone care to perhaps write an ncurses *abstraction* layer for
someone like me, so I wouldn't have to figure out how to draw things
out of the line drawing character set or properly refresh regions
uncovered by pull-down menus? :-) If that were done, I could actually
see doing the rest. Just so I don't need to know anything really deep
about ncurses to actually draw the objects. This needs to be
abstracted *anyway*, actually, so that we can do this under X too
when the time comes.
4. We need applications to paper over all these clever mechanisms.
Naturally, pkg_add, pkg_delete and friends will have to be re-written
to take advantage of all these nice building blocks. So will
sysinstall. I don't mind. :-)
We will also want to have new commands for reporting on 3DFS
information, reclaiming space, invalidating deltas, etc. I don't mind
working on a lot of this, if I can get some help!
We also need a specification format for the various GUI screens, since
nobody wants to hack that stuff out in raw C if they can help it.
Does this mean libforms? Something else?
Comments?
Jordan
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3052.817058108>