From owner-freebsd-binup Sun Dec 22 6:53:16 2002 Delivered-To: freebsd-binup@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D1CF937B401 for ; Sun, 22 Dec 2002 06:53:11 -0800 (PST) Received: from arthur.nitro.dk (port324.ds1-khk.adsl.cybercity.dk [212.242.113.79]) by mx1.FreeBSD.org (Postfix) with ESMTP id 833F243EDC for ; Sun, 22 Dec 2002 06:53:10 -0800 (PST) (envelope-from simon@arthur.nitro.dk) Received: by arthur.nitro.dk (Postfix, from userid 1000) id F22BC10BF87; Sun, 22 Dec 2002 15:53:07 +0100 (CET) Date: Sun, 22 Dec 2002 15:53:07 +0100 From: "Simon L. Nielsen" To: freebsd-binup@freebsd.org Subject: Re: status? Message-ID: <20021222145307.GC375@nitro.dk> References: Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="kXdP64Ggrk/fb43R" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.1i Sender: owner-freebsd-binup@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG --kXdP64Ggrk/fb43R Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On 2002.12.21 17:02:05 +0000, Friedemann Becker wrote: > > Well, I'm interested in writing code at the moment. ;) > Are you in need of any assistance? > Maybe you could need a more or lessed skilled programemr? > > I'd like to help.... ;-) It appears that there are some people who want to get this project started again - so I think we should try to get this going... Somebody had started making a document about some kind of package abstraction layer (about a year ago) - does anyone know if anything came out of that? I have attached the doucmentation I have on the "old" code - I can't remeber if there is more, but I think there was homepage with some more documentation... Btw. The Anarcat just wrote to me that the binup CVS tree was saved from usw1 so and is now on the server the libh project uses. -- Simon L. Nielsen --kXdP64Ggrk/fb43R Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename=NOTES FreeBSD Binary Updates Project To setup your database to run the server : 1. Create a database for this application, 'updates' is the default used by the server if you don't specify another name. 2. Run the `setupdb' program to create 4 tables in your database that the server application will use to store information about the files. 3. Run the 'extract' program on a distribution to copy the dists from a CD set onto a directory tree - OR - 3. Realize that this has already been done and simply NFS mount mother's /a and look in /a/projects/updates 4. Run populate on the directory hierarchy created in step 3 and send the output to a temporary file. This file will contain a list of INSERT statements for every file in the hierarchy to setup your tables. 5. `cat sql.out | mysql -u updates updates' to actually populate the database 6. Build and execute the server process. An `ACTION' is a stream of data that is to be executed by the client. This stream is saved by the client as a file in a temporary directory so that a user can inspect scripts/binaries before they are executed. can be a shell script, perl script, binary, or any Miscellaneous design issues --------------------------- o Profiles What the user sees as "top level objects" in the upgrade system are canned profiles. A profile can represent a given user's system configuration or a generic system template (web server, mail server, etc) that we provide. Each profile consists of file entries and/or collection entries. A collection entry represents a grouped set of files like a package or what sysinstall calls a "distribution." Profiles exist on the server machine, though the client can also choose to cache copies for "tripwire" types of activities. Some typical profiles and their contents might look like this: [mysystem] [web-server] bin 4.2 bin 4.2 bash 2.02 manpages 4.2 src 4.2 apache 2.1 xblaster 1.0 A collection can also have a specific version number associated with it or have a "floating" version number, meaning that it tracks whatever's newest for that entity. A collection is represented by a base delta and n change deltas, new base deltas being created periodically as the size of the last base and n change deltas exceeds the cost of a new base delta. o Authentication Users will authenticate with the server via a username / password scheme which allows them to access their custom profiles as well as any system-defined ones. o Crypto Transport between the server and client will be done via a secure means with encryption of all data and proper sanity checking to prevent data corruption and/or man-in-the-middle attacks. Upgrade Client: --------------- The client supports connecting to an upgrade server, authenticating a user, browsing existing profiles or creating new ones and downloading file data and "actions" from the server. New file data will be created in such a way that partial updates do not cause corruption and whole transactions are committed in reasonably atomic fashion. The client will be implemented in a 3-stage process: o A set of libraries which implement the bulk of the client<->server functionality. o A command-line version of the client which supports all available library functions. o A GUI version of the client which either wraps around the client or calls the library routines directly, dependinng on whichever makes the most sense. Since a system can also be "upgraded" from a standing start, a special version of sysinstall will also be generated which basically just does the disklabelling and filesystem formatting part, selects a server, handles authentication and then brings up a menu of available profiles. From that point the upgrade system takes over and the system is "upgraded" into place rather than installed in the usual fashion. This version of sysinstall will also be a major consumer of the upgrade client library. Upgrade Server: --------------- The server supports connections by arbitrary numbers of clients and authenticating a user (or "anonymous" if the server is configured to support anonymous connections) for determining the available profiles. Once the server receives a manifest (e.g. a set of collections) from a client machine and a server-side profile name to match it with, it goes looking through each collection to see if a newer version of that collection exists on the server or if there are any change deltas pending against the collection which are greater than the corresponding patchlevel of the collection in the client manifest. Deltas and/or entire collections are sent to the client for unpacking along with any before/after actions for each delta or collection which should be executed on the client. Once the client has confirmed that all before/after actions and extraction of a given collection has completed successfully, it updates the stored profile and goes on to the next. If the transfer is interrupted at any point, the process can therefore pick up where it left off. The upgrade server provides local storage for a certain amount of profile data depending on disk space constraints and can also be used as a way of cloning machines. The user installs one machine entirely according to taste and then uploads its profile. Each subsequent machine is installed from this profile and voila, an army of clones. The server will probably not keep any truly client-side data like /etc/master.passwd or anything else it doesn't offer out of its own collections, but we can leave this decision open for later or make it a configuration option. Did I miss anything? - Jordan --[ Server Implementation ]--- The current implementation of the update server is a C program running on callisto.osd.bsdi.com port 31337 (no this will not be the permanent port number). The source code for this server is available from cvs /usr/local/cvsrep/updated. You can telnet to . Authentication / Encryption OpenSSL? --[ Dependencies ]--- In addition to requiring a database package to function, this software currently uses the GNU readline library to get its initial configuration information from a users terminal. This dependency will be removed soon. Murray --kXdP64Ggrk/fb43R-- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-binup" in the body of the message From owner-freebsd-binup Sun Dec 22 8: 8: 8 2002 Delivered-To: freebsd-binup@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E78D437B401 for ; Sun, 22 Dec 2002 08:08:07 -0800 (PST) Received: from arthur.nitro.dk (port324.ds1-khk.adsl.cybercity.dk [212.242.113.79]) by mx1.FreeBSD.org (Postfix) with ESMTP id 739C943ED8 for ; Sun, 22 Dec 2002 08:08:07 -0800 (PST) (envelope-from simon@arthur.nitro.dk) Received: by arthur.nitro.dk (Postfix, from userid 1000) id 1134710BF8F; Sun, 22 Dec 2002 17:08:06 +0100 (CET) Date: Sun, 22 Dec 2002 17:08:06 +0100 From: "Simon L. Nielsen" To: freebsd-binup@freebsd.org Subject: Re: status? Message-ID: <20021222160805.GF380@nitro.dk> References: <20021222145307.GC375@nitro.dk> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20021222145307.GC375@nitro.dk> User-Agent: Mutt/1.5.1i Sender: owner-freebsd-binup@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On 2002.12.22 15:53:07 +0000, Simon L. Nielsen wrote: > Btw. The Anarcat just wrote to me that the binup CVS tree was saved from > usw1 so and is now on the server the libh project uses. I just looked a bit in the old achieves and there actually is a project webpage on http://www.freebsd.org/projects/updater.html (not that is says much more than the NOTES document I attached in my previous mail) and the binup code is in the main FreeBSD cvs tree under projects/binup/. Is there any commiters that has the time to actually rewiew / commit code for this project... if not it might be a bit for a problem for us mere mortals :) to get the project really started again... -- Simon L. Nielsen To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-binup" in the body of the message From owner-freebsd-binup Sun Dec 22 9:14:28 2002 Delivered-To: freebsd-binup@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5756037B401 for ; Sun, 22 Dec 2002 09:14:27 -0800 (PST) Received: from arthur.nitro.dk (port324.ds1-khk.adsl.cybercity.dk [212.242.113.79]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8D4A743EDE for ; Sun, 22 Dec 2002 09:14:26 -0800 (PST) (envelope-from simon@arthur.nitro.dk) Received: by arthur.nitro.dk (Postfix, from userid 1000) id 5BBA510BF8F; Sun, 22 Dec 2002 18:14:24 +0100 (CET) Date: Sun, 22 Dec 2002 18:14:24 +0100 From: "Simon L. Nielsen" To: freebsd-binup@freebsd.org Subject: Minor build patch Message-ID: <20021222171423.GG380@nitro.dk> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="n8g4imXOkfNTN/H1" Content-Disposition: inline User-Agent: Mutt/1.5.1i Sender: owner-freebsd-binup@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG --n8g4imXOkfNTN/H1 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Hello To get things started here is a small patch to make binup actually build against recent postgresql. -- Simon L. Nielsen --n8g4imXOkfNTN/H1 Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="build_pgsql.patch" Index: server/Makefile =================================================================== RCS file: /home/mirror/freebsd/ncvs/projects/binup/server/Makefile,v retrieving revision 1.7 diff -u -d -r1.7 Makefile --- server/Makefile 19 Dec 2000 12:36:30 -0000 1.7 +++ server/Makefile 22 Dec 2002 17:07:12 -0000 @@ -3,6 +3,8 @@ # # Murray Stokely # +# $FreeBSD +# CC = gcc CFLAGS = -g -Wall -I../common/ @@ -27,8 +29,8 @@ MYSQLOBJS = mysql.o ## PostgreSQL Support -PGSQLLIBS = -L/usr/local/pgsql/lib -lpq -PGSQLINCS = -I/usr/local/pgsql/include +PGSQLLIBS = -L/usr/local/lib -lpq +PGSQLINCS = -I/usr/local/include PGSQLOBJS = pgsql.o .if defined(WITH_PGSQL) || exists(/usr/local/pgsql/) --n8g4imXOkfNTN/H1-- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-binup" in the body of the message From owner-freebsd-binup Sun Dec 22 12: 5:35 2002 Delivered-To: freebsd-binup@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4893237B401 for ; Sun, 22 Dec 2002 12:05:35 -0800 (PST) Received: from builder.freebsdmall.com (builder.freebsdmall.com [65.86.180.18]) by mx1.FreeBSD.org (Postfix) with ESMTP id EF0CA43EDA for ; Sun, 22 Dec 2002 12:05:34 -0800 (PST) (envelope-from murray@builder.freebsdmall.com) Received: by builder.freebsdmall.com (Postfix, from userid 2074) id 9FC36DC2; Sun, 22 Dec 2002 12:05:34 -0800 (PST) Date: Sun, 22 Dec 2002 12:05:34 -0800 From: murray@FreeBSD.org To: "Simon L. Nielsen" Cc: freebsd-binup@freebsd.org Subject: Re: Minor build patch Message-ID: <20021222200534.GD7639@freebsdmall.com> References: <20021222171423.GG380@nitro.dk> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20021222171423.GG380@nitro.dk> User-Agent: Mutt/1.4i X-GPG-Key-ID: 1024D/0E451F7D X-GPG-Key-Fingerprint: E2CA 411D DD44 53FD BB4B 3CB5 B4D7 10A2 0E45 1F7D Sender: owner-freebsd-binup@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Sun, Dec 22, 2002 at 06:14:24PM +0100, Simon L. Nielsen wrote: > To get things started here is a small patch to make binup actually build > against recent postgresql. Thanks, I've committed this! - Murray To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-binup" in the body of the message From owner-freebsd-binup Wed Dec 25 5:35:57 2002 Delivered-To: freebsd-binup@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DBBE637B401; Wed, 25 Dec 2002 05:35:55 -0800 (PST) Received: from mta05-svc.ntlworld.com (mta05-svc.ntlworld.com [62.253.162.45]) by mx1.FreeBSD.org (Postfix) with ESMTP id 191E243ED8; Wed, 25 Dec 2002 05:35:54 -0800 (PST) (envelope-from colin.percival@wadham.ox.ac.uk) Received: from piii600.wadham.ox.ac.uk ([62.254.149.90]) by mta05-svc.ntlworld.com (InterMail vM.4.01.03.27 201-229-121-127-20010626) with ESMTP id <20021225133551.MWCL900.mta05-svc.ntlworld.com@piii600.wadham.ox.ac.uk>; Wed, 25 Dec 2002 13:35:51 +0000 Message-Id: <5.0.2.1.1.20021225125238.037cd840@popserver.sfu.ca> X-Sender: cperciva@popserver.sfu.ca X-Mailer: QUALCOMM Windows Eudora Version 5.0.2 Date: Wed, 25 Dec 2002 13:35:50 +0000 To: freebsd-binup@freebsd.org, freebsd-stable@freebsd.org From: Colin Percival Subject: Binary security updates Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-freebsd-binup@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG I've put together a basic binary updates tool aimed at people who want to track a security branch without keeping a source tree and recompiling. I have tested this code to the best of my ability -- but since I only have one FreeBSD box (and it's on the other side of the world), that ability is rather limited. The modus operandi of this tool is simple: Build a -RELEASE world, build world(s) along the security branch, and compare. Unfortunately the details are a bit more complicated: Before comparing we have to strip out various time/date/user/host stamps -- stamps which we locate by building the same world twice. An RSA key is generated and used to sign the updates for security purposes. Once signed, however, the updates can be distributed via insecure means (eg, HTTP). At the client side, updates are fetched (and verified) into a staging directory; they are then installed on command (preferably in single-user mode, of course) and a rollback directory is created. In order for files to be updated, their original MD5 hashes must be correct, so any local modifications to the world will result in files not being updated. A few more incidental notes, in no specific order: 1. The practice of bumping newvers.sh for security updates leads to world-only updates causing the kernel to change. As a workaround, my code replaces the kernel stamp with a generic -SECURITY stamp. 2. /usr/lib/libobjc.a (and _p.a) is wierd; it is the only file in the world which builds completely differently every time. As a result, it gets ignored -- any security updates here will not be propagated. 3. A few other files (.../perllocal.pod, ...doc/psd/*, ...doc/usd/*, ...phantasia/void) are obviously not security-related, but have some messy variability; I'm ignoring those files as well. 4. I just realized that I'm snapping hard links if any such files need updating. Oops. 5. Anything else which doesn't update properly if it has another file mv'ed onto it won't be updated properly. I'm not sure if this applies to anything except /boot stuff. 6. In order to locate the time and date stamps, I play games with the time; the server code won't work properly if you have anything like ntp running. 7. The license is unusual. GPL fanatics will probably not be very happy. 8. This code should not be used to update anything other than a binary install of -RELEASE: If you use it to update a world you have built yourself, some files will not be updated (due to differing MD5 hashes). Thanks to: Chad David and Terry Lambert for helping me understand the build process; Nick Geyer and netwave.com for giving me some temporary webspace while I work out where I'm going to put this permanently. Now, for those of you who have read this far: The server code is at http://cperciva.netwave.com/freebsd-update-server.tgz with MD5 hash a9bbf3f514314b2e63ce54cc85246bd8 The client code is at http://cperciva.netwave.com/freebsd-update-client.tgz with MD5 hash 35da1fb837edf4fb86979d728c24d719 The client code includes a configuration file which will fetch updates (from 4.7-RELEASE to RELENG_4_7) which I have published -- I leave it up to you to decide if you want to trust me. Note that the above URLs are temporary; at some point I'll move these to a more permanent home. For that reason, I'd suggest not linking to them. Merry Christmas, Colin Percival To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-binup" in the body of the message From owner-freebsd-binup Thu Dec 26 16: 9: 0 2002 Delivered-To: freebsd-binup@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E5A3E37B401; Thu, 26 Dec 2002 16:08:58 -0800 (PST) Received: from mail.ubergeeks.com (lorax.ubergeeks.com [209.145.65.55]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1857743EDC; Thu, 26 Dec 2002 16:08:58 -0800 (PST) (envelope-from adrian+freebsd-binup@ubergeeks.com) Received: from mail.ubergeeks.com (localhost [127.0.0.1]) by mail.ubergeeks.com (8.12.5/8.12.5) with ESMTP id gBR08kIP068842; Thu, 26 Dec 2002 19:08:47 -0500 (EST) (envelope-from adrian+freebsd-binup@ubergeeks.com) Received: from localhost (adrian@localhost) by mail.ubergeeks.com (8.12.5/8.12.5/Submit) with ESMTP id gBR08kb3068839; Thu, 26 Dec 2002 19:08:46 -0500 (EST) (envelope-from adrian+freebsd-binup@ubergeeks.com) X-Authentication-Warning: lorax.ubergeeks.com: adrian owned process doing -bs Date: Thu, 26 Dec 2002 19:08:46 -0500 (EST) From: Adrian Filipi-Martin To: Colin Percival Cc: freebsd-binup@FreeBSD.ORG, Subject: Re: Binary security updates In-Reply-To: <5.0.2.1.1.20021225125238.037cd840@popserver.sfu.ca> Message-ID: <20021226190441.D68788-100000@lorax.ubergeeks.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-binup@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Wed, 25 Dec 2002, Colin Percival wrote: > I've put together a basic binary updates tool aimed at people who want > to track a security branch without keeping a source tree and > recompiling. I have tested this code to the best of my ability -- but > since I only have one FreeBSD box (and it's on the other side of the > world), that ability is rather limited. Interesting. I'm looking it over and perhaps you could answer one question for me quickly. How do you deal with .a-files? They may be identical excepting for their table of contents and md5's don't look into the archive. Also did you run into anything with respect to other archive/library file types? We built our own binary patch system for our appliances, but it's a bit specific to our appliance's needs. We were able to make certain simplifying statements such as no appliance includes compilation tools or libraries, so the .a-file problem went away. Adrian -- [ adrian@ubergeeks.com ] To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-binup" in the body of the message From owner-freebsd-binup Thu Dec 26 16:19:38 2002 Delivered-To: freebsd-binup@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id F295237B401; Thu, 26 Dec 2002 16:19:36 -0800 (PST) Received: from mta07-svc.ntlworld.com (mta07-svc.ntlworld.com [62.253.162.47]) by mx1.FreeBSD.org (Postfix) with ESMTP id 80CA943E4A; Thu, 26 Dec 2002 16:19:35 -0800 (PST) (envelope-from colin.percival@wadham.ox.ac.uk) Received: from piii600.wadham.ox.ac.uk ([62.254.149.90]) by mta07-svc.ntlworld.com (InterMail vM.4.01.03.27 201-229-121-127-20010626) with ESMTP id <20021227001933.PKYM9037.mta07-svc.ntlworld.com@piii600.wadham.ox.ac.uk>; Fri, 27 Dec 2002 00:19:33 +0000 Message-Id: <5.0.2.1.1.20021227001044.01c0fa40@popserver.sfu.ca> X-Sender: cperciva@popserver.sfu.ca X-Mailer: QUALCOMM Windows Eudora Version 5.0.2 Date: Fri, 27 Dec 2002 00:19:32 +0000 To: Adrian Filipi-Martin From: Colin Percival Subject: Re: Binary security updates Cc: freebsd-binup@FreeBSD.ORG, In-Reply-To: <20021226190441.D68788-100000@lorax.ubergeeks.com> References: <5.0.2.1.1.20021225125238.037cd840@popserver.sfu.ca> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-freebsd-binup@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG At 19:08 26/12/2002 -0500, Adrian Filipi-Martin wrote: >On Wed, 25 Dec 2002, Colin Percival wrote: > > I've put together a basic binary updates tool aimed at people who want > > to track a security branch without keeping a source tree and > > recompiling. I have tested this code to the best of my ability -- but > > since I only have one FreeBSD box (and it's on the other side of the > > world), that ability is rather limited. > > How do you deal with .a-files? They may be identical excepting for >their table of contents and md5's don't look into the archive. Assuming that the component object files are the same, .a files will be identical apart from the timestamps. There happens to be a timestamp for each object file, which (especially for libc) means an awful lot of timestamps; but my code happily finds all of them the same way as it deal with other timestamps. Files which do not contain any stamps are compared on the basis of their MD5 hashes; "polymorphic" files (those which contain stamps) are unstamped and then compared. > Also did you run into anything with respect to other >archive/library file types? Gzipped files need to be ungzipped before looking for / removing stamps, but those are the only files in the FreeBSD world which I needed to deal with specially; I can't say if other worlds would be so easily dealt with. I've been contacted by someone who is testing my code on OpenBSD and MicroBSD, but I haven't heard any results. Colin Percival To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-binup" in the body of the message From owner-freebsd-binup Thu Dec 26 21:18:42 2002 Delivered-To: freebsd-binup@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 31EF237B401; Thu, 26 Dec 2002 21:18:41 -0800 (PST) Received: from mail.microbsd.net (mail.microbsd.net [4.38.106.40]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8B78A43EA9; Thu, 26 Dec 2002 21:18:30 -0800 (PST) (envelope-from kerberus@microbsd.net) Received: from armada.microbsd.net (GNR-CAP-117.WYOMING.COM [216.67.137.133]) by mail.microbsd.net (Postfix) with ESMTP id 986EF6A1B; Fri, 27 Dec 2002 00:29:27 -0500 (EST) Content-Type: text/plain; charset="iso-8859-1" From: Kerberus Organization: MicroBSD To: Colin Percival Subject: Re: Binary security updates Date: Fri, 27 Dec 2002 00:15:49 -0500 User-Agent: KMail/1.4.3 Cc: freebsd-binup@FreeBSD.ORG, References: <5.0.2.1.1.20021225125238.037cd840@popserver.sfu.ca> <5.0.2.1.1.20021227001044.01c0fa40@popserver.sfu.ca> In-Reply-To: <5.0.2.1.1.20021227001044.01c0fa40@popserver.sfu.ca> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Message-Id: <200212270015.49683.kerberus@microbsd.net> Sender: owner-freebsd-binup@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG We will let you know tomorrow what we find on MicroBSD/OpenBSD, heck we m= ight=20 even try to tackle NetBSD : P On Thursday 26 December 2002 19:19, Colin Percival wrote: > At 19:08 26/12/2002 -0500, Adrian Filipi-Martin wrote: > >On Wed, 25 Dec 2002, Colin Percival wrote: > > > I've put together a basic binary updates tool aimed at people wh= o > > > want to track a security branch without keeping a source tree and > > > recompiling. I have tested this code to the best of my ability -- = but > > > since I only have one FreeBSD box (and it's on the other side of th= e > > > world), that ability is rather limited. > > > > How do you deal with .a-files? They may be identical excepti= ng > > for their table of contents and md5's don't look into the archive. > > Assuming that the component object files are the same, .a files will= be > identical apart from the timestamps. There happens to be a timestamp f= or > each object file, which (especially for libc) means an awful lot of > timestamps; but my code happily finds all of them the same way as it de= al > with other timestamps. > Files which do not contain any stamps are compared on the basis of t= heir > MD5 hashes; "polymorphic" files (those which contain stamps) are unstam= ped > and then compared. > > > Also did you run into anything with respect to other > >archive/library file types? > > Gzipped files need to be ungzipped before looking for / removing sta= mps, > but those are the only files in the FreeBSD world which I needed to dea= l > with specially; I can't say if other worlds would be so easily dealt wi= th. > I've been contacted by someone who is testing my code on OpenBSD and > MicroBSD, but I haven't heard any results. > > Colin Percival > > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-binup" in the body of the message To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-binup" in the body of the message From owner-freebsd-binup Fri Dec 27 2:43:34 2002 Delivered-To: freebsd-binup@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9461C37B401; Fri, 27 Dec 2002 02:43:33 -0800 (PST) Received: from melusine.cuivre.fr.eu.org (melusine.cuivre.fr.eu.org [62.212.105.185]) by mx1.FreeBSD.org (Postfix) with ESMTP id 054C043EB2; Fri, 27 Dec 2002 02:43:33 -0800 (PST) (envelope-from thomas@FreeBSD.ORG) Received: by melusine.cuivre.fr.eu.org (Postfix, from userid 1000) id BFF1C2C3D2; Fri, 27 Dec 2002 11:43:31 +0100 (CET) Date: Fri, 27 Dec 2002 11:43:31 +0100 From: Thomas Quinot To: Colin Percival Cc: freebsd-binup@freebsd.org, freebsd-stable@freebsd.org Subject: Re: Binary security updates Message-ID: <20021227104331.GA12196@melusine.cuivre.fr.eu.org> Reply-To: Thomas Quinot References: <5.0.2.1.1.20021225125238.037cd840@popserver.sfu.ca> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <5.0.2.1.1.20021225125238.037cd840@popserver.sfu.ca> User-Agent: Mutt/1.4i X-message-flag: WARNING! Using Outlook can damage your computer. Sender: owner-freebsd-binup@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Le 2002-12-25, Colin Percival écrivait : > I've put together a basic binary updates tool aimed at people who want This looks like a quite interesting proff-of-concept! > mode, of course) and a rollback directory is created. In order for files The rollback facility looks particularly attractive. Do you think it would be feasible to generalize it to a convenient way of rolling back an installworld? -- Thomas.Quinot@Cuivre.FR.EU.ORG To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-binup" in the body of the message From owner-freebsd-binup Fri Dec 27 3:15:31 2002 Delivered-To: freebsd-binup@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7E4E737B401; Fri, 27 Dec 2002 03:15:30 -0800 (PST) Received: from mta06-svc.ntlworld.com (mta06-svc.ntlworld.com [62.253.162.46]) by mx1.FreeBSD.org (Postfix) with ESMTP id ED7D543ED1; Fri, 27 Dec 2002 03:15:28 -0800 (PST) (envelope-from colin.percival@wadham.ox.ac.uk) Received: from piii600.wadham.ox.ac.uk ([62.254.149.90]) by mta06-svc.ntlworld.com (InterMail vM.4.01.03.27 201-229-121-127-20010626) with ESMTP id <20021227111524.VFEW20174.mta06-svc.ntlworld.com@piii600.wadham.ox.ac.uk>; Fri, 27 Dec 2002 11:15:24 +0000 Message-Id: <5.0.2.1.1.20021227110837.03824e90@popserver.sfu.ca> X-Sender: cperciva@popserver.sfu.ca X-Mailer: QUALCOMM Windows Eudora Version 5.0.2 Date: Fri, 27 Dec 2002 11:15:20 +0000 To: Thomas Quinot From: Colin Percival Subject: Re: Binary security updates Cc: freebsd-binup@FreeBSD.ORG, freebsd-stable@FreeBSD.ORG In-Reply-To: <20021227104331.GA12196@melusine.cuivre.fr.eu.org> References: <5.0.2.1.1.20021225125238.037cd840@popserver.sfu.ca> <5.0.2.1.1.20021225125238.037cd840@popserver.sfu.ca> Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1"; format=flowed Content-Transfer-Encoding: quoted-printable Sender: owner-freebsd-binup@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG At 11:43 27/12/2002 +0100, Thomas Quinot wrote: >Le 2002-12-25, Colin Percival =E9crivait : > > mode, of course) and a rollback directory is created. In order for= files > >The rollback facility looks particularly attractive. Do you think it >would be feasible to generalize it to a convenient way of rolling back >an installworld? All I do is copy files from / into ./rollback/ before replacing them; I= =20 can't see any reason why that wouldn't suffice for rolling back an=20 installworld, except that you'd need much more disk space. Colin Percival To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-binup" in the body of the message From owner-freebsd-binup Fri Dec 27 13:30:20 2002 Delivered-To: freebsd-binup@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9275637B401 for ; Fri, 27 Dec 2002 13:30:13 -0800 (PST) Received: from arthur.nitro.dk (port324.ds1-khk.adsl.cybercity.dk [212.242.113.79]) by mx1.FreeBSD.org (Postfix) with ESMTP id 45B4843EC2 for ; Fri, 27 Dec 2002 13:30:12 -0800 (PST) (envelope-from simon@arthur.nitro.dk) Received: by arthur.nitro.dk (Postfix, from userid 1000) id 2115010BF87; Fri, 27 Dec 2002 22:30:09 +0100 (CET) Date: Fri, 27 Dec 2002 22:30:09 +0100 From: "Simon L. Nielsen" To: freebsd-binup@freebsd.org Subject: Proposal for binup design document Message-ID: <20021227213008.GC415@nitro.dk> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="eAbsdosE1cNLO4uF" Content-Disposition: inline User-Agent: Mutt/1.5.1i Sender: owner-freebsd-binup@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG --eAbsdosE1cNLO4uF Content-Type: multipart/mixed; boundary="J/dobhs11T7y2rNN" Content-Disposition: inline --J/dobhs11T7y2rNN Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hello I really think binup could be useful and fill a gap for FreeBSD. Therefor to try to get things moving and to get some kind of structure on binup I have begun make a kind of design document containing some thought / ideas about binup. I think it is important to discuss these issues (especially the HTTP proposal) so we know what to make binup do and how it should be done. I have a attached a text only version of the document, but it has been written in HTML and can be found at http://simon.nitro.dk/binup/binup-2003.html . I plan to start looking at the server database backend today and adding/changing the binup design document along the way. --=20 Simon L. Nielsen --J/dobhs11T7y2rNN Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="binup-2003.txt" Binup 2003 proposal Contents * [1]Introduction * [2]General * [3]HTTP * [4]Main use * [5]Package abstraction * [6]Package the base system * [7]Client/server communication * [8]Tasks Updated : $Date: 2002/12/27 21:17:46 $ binup 2003 introduction This is a proposal by Simon L. Nielsen <[9]simon@nitro.dk> for a design document for the FreeBSD Binary Updater (binup). The document contains some ideas for changes to the current binup code/design and the beginning to a general design document about binup. The document is very much work in progress... and a bit of a mess at the moment. I have called this document binup 2003 to differentiate it from the current "old" binup code. The [10]"old" binup document should be read before this document since the main design and goals haven't changed. Please note that this document is meant as an early draft and most likely contains flaws, bad ideas and things that have not been thought fully through. There is also most likely a lot of spelling and grammatical errors - please ignore them for now. It contains comments/ideas on the subjects I think needs to be addressed before real coding on binup 2003 can begin. I would very much like comments on the document and the ideas. I would prefer that comments are posted to the [11]binup mailing-list for discussion but private comments directly to me are also very welcome. I plan to keep an updated version of this document on [12]http://simon.nitro.dk/binup/binup-2003.html. General I think it would be a good idea to change the binup system to use HTTP instead of doing a new TCP level protocol. Mainly because I think this could save a lot of coding. See the [13]HTTP section for more. As suggested on the binup mailing-list some time ago I think binup should be made independent of the packages actually transfered by binup. This would make it possible to use distribution tar-balls, .tgz's, libh, binary patches or any other packaging format without any major change to binup. See the [14]package abstraction section for more. There have been talk about making the binup project do a package API (/var/db/pkg API) and do the breaking up the base system into packages. While I think these are good ideas that would certainly benefit binup I don't think we should wait for this to be finished since that may take a long time and it should be possible to do a good binary updated without these tasks completed. HTTP The current binup code uses a special protocol on top of TCP. This makes the protocol more flexible but also require a lot of code to get working. Instead binup could be implemented on top of HTTP. This would make some things a lot simpler since a lot of existing programs/code can be used both server and client side. The use of HTTP will simplify implementation since a lot of functionality will come "for free" e.g. file transfer, SSL and connection management. Standard server/clients also gives the advantage that less low level C code has to be written and therefore minimizing the risk for buffer overflows and other common C bugs. Using HTTP it would also be quite simple for the binup server to point a client at a normal FreeBSD mirror for the actual downloading of updated packages (or perhaps the client can pick it's preferred mirror site). A checksum/crypto signature could still be send from the binup server and therefore increasing security a bit for trojan-ed files. This might not be a big increase - but it would help a bit. The main drawback of using HTTP is the lack of state between requests but I don't see this as a major drawback since most the needed state code can be kept client side instead. Advantages of using HTTP : * Less reinventing the wheel * Will be using standard Internet protocol (big advantage if client is behind a firewall) * Simpler prototyping (a lot of the code can be initially implemented in a scripting language) * Less low level C code Disadvantages of using HTTP : * No state * Will mean scraping most of the current binup code Main binup uses This section contains some thoughts on where/how binup could be useful. I think this is important to keep in mind to determine exactly whan binup is supposed to do. The main uses for binup as I see it is (in more or less prioritized order) : 1. Security updates (e.g. 4.7p1 -> 4.7p2) 2. Port updates 3. FreeBSD upgrades (e.g. 4.6 -> 4.7) 4. Fresh FreeBSD installs with profiles I see two main types of operation for a binup server - private and public servers. A private server can for example be used for a company to have a simple way to deploy updates to a server park. In this scenario updated will most likely have to be tested before being deployed widespread in the administrators must therefor have ability to add a package to be updated on a per package basis (package in a general sense both OS upgrade and port upgrade). The public server could be used as a kind of replacement for buildworld/installworld for people who just want to do a network upgrade. This could also be very useful for people who want to keep up to date with security fixes without having to recompile everything for a "simple" small patch. When clients do an update the binup client should allow this to be done either automatically or in an interactive mode. Package abstraction I belive it would be useful for binup only to have a minimal "knowledge" of the packages being transfered. That way binup is not being bound to a specific packing format which might change once libh is fully implemented or the base system is packaged in some special format. This would also alow update packages to be available in multiple packaging formats which might be useful in some cases. It would of course make the tripwire functionality harder to implement, but it could still be done. This could mean that binup could be used to transfer all kind of odd package formats even .rpm, .deb packages if that should be needed. binup mainly need three pieces of information for each package as far as I can see : 1. Package name - must be unique 2. Package format - so the client knows how to install the package 3. Package version - to find new versions The real tricky one is package version since third party programs use all kinds of odd version schemes. Client/server communication I think that when a client wants to upgrade the client should get a list of the most recent packages for a profile the client specifies and then the client decides which packages to download and install. This way partial upgrades should be handles automatically and the privacy of the client is preserved since the server does not know which packages the client does actually have installed. The client -> server can be done by a simple HTTP GET or POST with the needed variables. For the server -> client data there must be some kind of formatting/encoding so the client can parse it in a simple way. The main formatting/encoding styles I see are comma separated or XML. XML is very general and easy to extend later but it has a few major drawbacks. The main drawback is the requirement of an XML parser client side and I think the binup client should depend on as few external components as possible to minimize problems when blowing things away under the client and when doing from scratch installs. Another drawback for XML is the extra overhead both from larger network transfers and from the parsing. Comma separated would require special parsing code to be written and would most likely break if new fields are added to the lines. The advantage is that it requires no external library to parse and it is rather fast to parse. I lean mostly to comma separated format since it does not require a XML library on the client... binup tasks The tasks to get a complete binup system as I see it (not all may be required to get first version of binup working) * Find out what binup 2003 should really do * Specify client/server protocol * binup server + Create database back-end + Implement server in some scripting language (perl/PHP/???) + Implement server in C (perhaps apache module ?) * binup client * package up base system + Package lib API (API for /var/db/pkg) + Find out how to do versioning (Problem with source installs) Implementation notes Database back-end should be implemented in server so it can be replaced later e.g. with dbm files if needed. $Id: binup-2003.html,v 1.6 2002/12/27 21:17:46 simon Exp $ References 1. #intro 2. #general 3. #http 4. #main-use 5. #package-abstraction 6. #package-base 7. #client-server-comm 8. #tasks 9. mailto:simon@nitro.dk 10. http://www.freebsd.org/projects/updater.html 11. mailto:freebsd-binup@FreeBSD.org 12. http://simon.nitro.dk/binup/binup-2003.html 13. #http 14. #package-abstraction --J/dobhs11T7y2rNN-- --eAbsdosE1cNLO4uF Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.1 (FreeBSD) iD8DBQE+DMZg8kocFXgPTRwRAs2FAKCaIOn8QNEO1nJ0TlCjzqTQV2hVjACfTdAy 3QRJOpqSAKRV8Y7b5hyoMic= =blO5 -----END PGP SIGNATURE----- --eAbsdosE1cNLO4uF-- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-binup" in the body of the message From owner-freebsd-binup Sat Dec 28 6:29: 0 2002 Delivered-To: freebsd-binup@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5F7D437B401; Sat, 28 Dec 2002 06:28:59 -0800 (PST) Received: from tiamat.ipv6.hackerheaven.org (ipv6.hackerheaven.org [80.126.0.97]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2508A43ED8; Sat, 28 Dec 2002 06:28:58 -0800 (PST) (envelope-from coolvibe@hackerheaven.org) Received: from tiamat.ipv6.hackerheaven.org (localhost.ipv6.hackerheaven.org [127.0.0.1]) by tiamat.ipv6.hackerheaven.org (8.12.6/8.12.6) with ESMTP id gBSEHFUo010936; Sat, 28 Dec 2002 15:17:15 +0100 (CET) (envelope-from coolvibe@tiamat.ipv6.hackerheaven.org) Received: (from coolvibe@localhost) by tiamat.ipv6.hackerheaven.org (8.12.6/8.12.6/Submit) id gBSEHEKn010935; Sat, 28 Dec 2002 15:17:14 +0100 (CET) Date: Sat, 28 Dec 2002 15:17:14 +0100 From: Emiel Kollof To: Kerberus Cc: Colin Percival , freebsd-binup@FreeBSD.ORG, freebsd-stable@FreeBSD.ORG Subject: Re: Binary security updates Message-ID: <20021228141714.GC10549@hackerheaven.org> References: <5.0.2.1.1.20021225125238.037cd840@popserver.sfu.ca> <5.0.2.1.1.20021227001044.01c0fa40@popserver.sfu.ca> <200212270015.49683.kerberus@microbsd.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200212270015.49683.kerberus@microbsd.net> X-Mailer: Mutt 1.3.23i (2001-10-09) X-Editor: Vim http://www.vim.org/ X-Info: http://www.hackerheaven.org/ X-Info2: http://www.cmdline.org/ X-Info3: http://www.coolvibe.org/ X-message-flag: Out of cheese error! User-Agent: Mutt/1.5.1i Sender: owner-freebsd-binup@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG * Kerberus (kerberus@microbsd.net) wrote: > We will let you know tomorrow what we find on MicroBSD/OpenBSD, heck we might > even try to tackle NetBSD : P Ooooh, that would be NEAT! Especially since I have this absolute dog (as in slow) of a computer that's running the venerable NetBSD which takes about a week to build world and a good few hours to build kernels. This would speed things up nicely. Sure, I could cross-build on my FreeBSD box, but that's more hassle than it's worth. Cheers, Emiel -- The advertisement is the most truthful part of a newspaper -- Thomas Jefferson To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-binup" in the body of the message