From owner-freebsd-questions@FreeBSD.ORG Tue Jun 25 08:14:52 2013 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 284E8C2D for ; Tue, 25 Jun 2013 08:14:52 +0000 (UTC) (envelope-from mike@skew.org) Received: from chilled.skew.org (chilled.skew.org [70.90.116.205]) by mx1.freebsd.org (Postfix) with ESMTP id 064B919DD for ; Tue, 25 Jun 2013 08:14:51 +0000 (UTC) Received: from chilled.skew.org (localhost [127.0.0.1]) by chilled.skew.org (8.14.5/8.14.4) with ESMTP id r5P8EoSv077312 for ; Tue, 25 Jun 2013 02:14:51 -0600 (MDT) (envelope-from mike@chilled.skew.org) Received: (from mike@localhost) by chilled.skew.org (8.14.5/8.14.4/Submit) id r5P8Eo4Z077311 for freebsd-questions@freebsd.org; Tue, 25 Jun 2013 02:14:50 -0600 (MDT) (envelope-from mike) From: Mike Brown Message-Id: <201306250814.r5P8Eo4Z077311@chilled.skew.org> Subject: Re: Should I be able to use mergemaster with freebsd-update? In-Reply-To: <201306230420.r5N4K9Yr054286@chilled.skew.org> To: freebsd-questions@freebsd.org Date: Tue, 25 Jun 2013 02:14:50 -0600 (MDT) X-Whoa: whoa. X-Mailer: ELM [version 2.4ME+ PL125 (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Jun 2013 08:14:52 -0000 > I'm using freebsd-update to upgrade my system to the latest minor version > (from 8.3-RELEASE to 8.4-RELEASE). > > I'm surprised that the merge handling system isn't more robust. When upgrading > the old way, from source, I was used to using mergemaster to handle any > merges that couldn't be done automatically. > > But when using freebsd-update, it seems that any failed merges require that > you get dumped into an empty text editor for each file. > [...] As I continue with this process, doing all the mergemaster tasks manually, I'm finding that the situation is even worse than I first realized. First, the relatively painless part. As I mentioned, after running 'freebsd-update -r 8.4-RELEASE upgrade', I had to deal with the un-mergeable files. Although mergemaster apparently isn't an option, its interactive merge function is really just a front-end for sdiff, so I found that I could replicate that part of its functionality by doing this in a separate window (-w 100 because I use a 100-column terminal): cd /var/db/freebsd-update/merge/8.4-RELEASE find -X . -type f | xargs -n 1 -o -I % sh -c '{ echo Now processing %. left=current, right=new, help="?"; sdiff -d -w 100 -o ../new/% ../old/% %; }' This populated my 'new' directory with merged files, so that (in the first window) when I opened the text editor for each one, I only had to just give it a once-over and exit the editor. Among the diffs in this 8.3 to 8.4 upgrade were changes to /etc/master.passwd and /etc/passwd, to add the 'auditdistd' and 'hast' users. As reported in March 2012 [1] in relation to 8.x to 9.x upgrades, this won't work as expected, because freebsd-update doesn't run pwd_mkdb after the master.passwd update. Now the real hurt begins; in the 8.3 to 8.4 upgrade, it's even worse. Once I saved all the files in the editor, I was prompted to approve a diff for each one. I had to answer "y" or the entire process aborts. Among the changes I was asked to approve, besides visible diffs, were unspecified differences in /etc/pwd.db and /etc/spwd.db, the binary files that contain the password database. There's no choice but to answer "y" and approve them, and I don't get any opportunity to rebuild them properly. So apparently, freebsd-update wants to install new, stock password databases, which are out-of-sync with my customized, merged master.passwd & passwd files. (And because of the way the freebsd-update system works, what I actually approved were empty, 0-byte files, the result of the failed merges.) What would happen if I just let it do this? Surely I wouldn't be able to log in, after the reboot, right? After approving the files (again, I had no choice!), I was presented with lists of all the files that would be deleted, added, and modified. Sure enough, bad /etc/pwd.db and /etc/spwd.db files were in the list. At this point, the merge folders were now gone; I no longer had the new master.passwd in a recognizable place. So I thought, OK, I'll run 'freebsd-update install' and hope that the new files end up in /etc. Then I could just run 'pwd_mkdb -p /etc/master.passwd' to regenerate passwd, pwd.db and spwd.db before my reboot. But the 'freebsd-update install' didn't put them there yet; I guess that doesn't happen till after the reboot. So they're still sitting in a staging folder, now gzipped and with obfuscated names, indexed in a separate file. Averting this disaster-in-the-making is not at all straightforward: cd /var/db/freebsd-update mkdir -m 0700 /tmp/oldpwdfiles zcat files/`grep '^/etc/master\.passwd' install.LYQAJQ/INDEX-NEW | cut -d \| -f 7`.gz > /tmp/oldpwdfiles/master.passwd zcat files/`grep '^/etc/passwd' install.LYQAJQ/INDEX-NEW | cut -d \| -f 7`.gz > /tmp/oldpwdfiles/passwd zcat files/`grep '^/etc/pwd\.db' install.LYQAJQ/INDEX-NEW | cut -d \| -f 7`.gz > /tmp/oldpwdfiles/pwd.db zcat files/`grep '^/etc/spwd\.db' install.LYQAJQ/INDEX-NEW | cut -d \| -f 7`.gz > /tmp/oldpwdfiles/spwd.db mkdir -m 0700 /tmp/newpwdfiles pwd_mkdb -d /tmp/newpwdfiles -p /tmp/oldpwdfiles/master.passwd gzip /tmp/newpwdfiles/* mv /tmp/newpwdfiles/master.passwd.gz files/`grep '^/etc/master\.passwd' install.LYQAJQ/INDEX-NEW | cut -d \| -f 7`.gz mv /tmp/newpwdfiles/passwd.gz files/`grep '^/etc/passwd' install.LYQAJQ/INDEX-NEW | cut -d \| -f 7`.gz mv /tmp/newpwdfiles/pwd.db.gz files/`grep '^/etc/pwd\.db' install.LYQAJQ/INDEX-NEW | cut -d \| -f 7`.gz mv /tmp/newpwdfiles/spwd.db.gz files/`grep '^/etc/spwd\.db' install.LYQAJQ/INDEX-NEW | cut -d \| -f 7`.gz rm -fr /tmp/oldpwdfiles /tmp/newpwdfiles I'm really shocked that it came to this. Did I just overlook the "--no-surprises" option in freebsd-update? And now, before I reboot, I have to figure out how to handle the changes that got made in /etc/mail ... ordinarily I'd run 'make all install restart' in there. Not an option till after reboot, though. At least it's not crucial for the reboot to work. Again, this is something that mergemaster was really good for. But the freebsd-update system really makes it difficult to do that kind of pre-reboot merging and database-regeneration that's an essential part of the upgrade process. Well, thanks for reading this far. I'm scared to death to reboot now, since my server is in another city, but we'll see how it goes. [1]