From owner-svn-src-all@FreeBSD.ORG Tue Nov 12 19:44:19 2013 Return-Path: Delivered-To: svn-src-all@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9D8FF426; Tue, 12 Nov 2013 19:44:19 +0000 (UTC) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 8B4BF2FFE; Tue, 12 Nov 2013 19:44:19 +0000 (UTC) Received: from svn.freebsd.org ([127.0.1.70]) by svn.freebsd.org (8.14.7/8.14.7) with ESMTP id rACJiJk1079513; Tue, 12 Nov 2013 19:44:19 GMT (envelope-from jhb@svn.freebsd.org) Received: (from jhb@localhost) by svn.freebsd.org (8.14.7/8.14.5/Submit) id rACJiJnS079510; Tue, 12 Nov 2013 19:44:19 GMT (envelope-from jhb@svn.freebsd.org) Message-Id: <201311121944.rACJiJnS079510@svn.freebsd.org> From: John Baldwin Date: Tue, 12 Nov 2013 19:44:19 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r258066 - in head: tools/regression/usr.sbin/etcupdate usr.sbin/etcupdate X-SVN-Group: head MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Nov 2013 19:44:19 -0000 Author: jhb Date: Tue Nov 12 19:44:18 2013 New Revision: 258066 URL: http://svnweb.freebsd.org/changeset/base/258066 Log: Add a pre-world mode of updating similar to the -p option that can be passed to mergemaster. In this mode, only changes to /etc/master.passwd and /etc/group are merged to /etc. In addition, it uses a temporary tree to stage these changes rather than overwriting the existing 'current' and 'previous' trees so that a full update can be run after a normal installworld has completed. MFC after: 2 weeks Added: head/tools/regression/usr.sbin/etcupdate/preworld.sh (contents, props changed) Modified: head/usr.sbin/etcupdate/etcupdate.8 head/usr.sbin/etcupdate/etcupdate.sh Added: head/tools/regression/usr.sbin/etcupdate/preworld.sh ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/tools/regression/usr.sbin/etcupdate/preworld.sh Tue Nov 12 19:44:18 2013 (r258066) @@ -0,0 +1,238 @@ +#!/bin/sh +# +# Copyright (c) 2013 Advanced Computing Technologies LLC +# Written by: John H. Baldwin +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# 1. Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# 2. Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# +# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND +# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE +# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT +# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY +# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +# SUCH DAMAGE. +# +# $FreeBSD$ + +# Regression tests for the pre-world (-p) mode + +WORKDIR=work + +usage() +{ + echo "Usage: preworld.sh [-s script] [-w workdir]" + exit 1 +} + +# Allow the user to specify an alternate work directory or script. +COMMAND=etcupdate +while getopts "s:w:" option; do + case $option in + s) + COMMAND="sh $OPTARG" + ;; + w) + WORKDIR=$OPTARG + ;; + *) + echo + usage + ;; + esac +done +shift $((OPTIND - 1)) +if [ $# -ne 0 ]; then + usage +fi + +CONFLICTS=$WORKDIR/conflicts +SRC=$WORKDIR/src +OLD=$WORKDIR/current +TEST=$WORKDIR/test + +build_trees() +{ + + # Populate trees with pre-world files and an additional file + # that should not be touched. + + rm -rf $SRC $OLD $TEST $CONFLICTS + + # Create the "old" source tree as the starting point + mkdir -p $OLD/etc + cat >> $OLD/etc/master.passwd <> $OLD/etc/group <> $OLD/etc/inetd.conf <:/' $TEST/etc/master.passwd + cat >> $TEST/etc/master.passwd <:1001:1001::0:0:John Baldwin:/home/john:/bin/tcsh +messagebus:*:556:556::0:0:D-BUS Daemon User:/nonexistent:/usr/sbin/nologin +polkit:*:562:562::0:0:PolicyKit User:/nonexistent:/usr/sbin/nologin +haldaemon:*:560:560::0:0:HAL Daemon User:/nonexistent:/usr/sbin/nologin +EOF + awk '/wheel/ { printf "%s,john\n", $0; next } // { print }' \ + $OLD/etc/group > $TEST/etc/group + cat >> $TEST/etc/group <> $SRC/etc/inetd.conf < $WORKDIR/testn.out + +cat > $WORKDIR/correct.out < $WORKDIR/test.out + +echo "Differences for real:" +diff -u -L "correct" $WORKDIR/correct.out -L "test" $WORKDIR/test.out + +check_trees Modified: head/usr.sbin/etcupdate/etcupdate.8 ============================================================================== --- head/usr.sbin/etcupdate/etcupdate.8 Tue Nov 12 19:39:14 2013 (r258065) +++ head/usr.sbin/etcupdate/etcupdate.8 Tue Nov 12 19:44:18 2013 (r258066) @@ -25,7 +25,7 @@ .\" .\" $FreeBSD$ .\" -.Dd March 16, 2012 +.Dd November 12, 2013 .Dt ETCUPDATE 8 .Os .Sh NAME @@ -33,7 +33,7 @@ .Nd "manage updates to system files not updated by installworld" .Sh SYNOPSIS .Nm -.Op Fl nBF +.Op Fl npBF .Op Fl d Ar workdir .Op Fl r | Fl s Ar source | Fl t Ar tarball .Op Fl A Ar patterns @@ -64,6 +64,7 @@ .Op Fl M Ar options .Nm .Cm resolve +.Op Fl p .Op Fl d Ar workdir .Op Fl D Ar destdir .Op Fl L Ar logfile @@ -481,6 +482,33 @@ option is not specified, then a temporary .Dq current tree will be extracted to perform the comparison. +.It Fl p +Enable +.Dq pre-world +mode. +Only merge changes to files that are necessary to successfully run +.Sq make installworld +or +.Sq make installkernel . +When this flag is enabled, +the existing +.Dq current +and +.Dq previous +trees are left alone. +Instead, +a temporary tree is populated with the necessary files. +This temporary tree is compared against the +.Dq current +tree. +This allows a normal update to be run after +.Sq make installworld +has completed. +Any conflicts generated during a +.Dq pre-world +update should be resolved by a +.Dq pre-world +.Cm resolve . .It Fl r Do not update the .Dq current Modified: head/usr.sbin/etcupdate/etcupdate.sh ============================================================================== --- head/usr.sbin/etcupdate/etcupdate.sh Tue Nov 12 19:39:14 2013 (r258065) +++ head/usr.sbin/etcupdate/etcupdate.sh Tue Nov 12 19:44:18 2013 (r258066) @@ -61,14 +61,15 @@ usage() { cat < etcupdate diff [-d workdir] [-D destdir] [-I patterns] [-L logfile] etcupdate extract [-B] [-d workdir] [-s source | -t tarball] [-L logfile] [-M options] - etcupdate resolve [-d workdir] [-D destdir] [-L logfile] + etcupdate resolve [-p] [-d workdir] [-D destdir] [-L logfile] etcupdate status [-d workdir] [-D destdir] EOF exit 1 @@ -181,22 +182,31 @@ always_install() # $1 - directory to store new tree in build_tree() { - local make + local destdir dir file make make="make $MAKE_OPTIONS" log "Building tree at $1 with $make" mkdir -p $1/usr/obj >&3 2>&1 - (cd $SRCDIR; $make DESTDIR=$1 distrib-dirs) >&3 2>&1 || return 1 + destdir=`realpath $1` - if ! [ -n "$nobuild" ]; then - (cd $SRCDIR; \ - MAKEOBJDIRPREFIX=$1/usr/obj $make _obj SUBDIR_OVERRIDE=etc && - MAKEOBJDIRPREFIX=$1/usr/obj $make everything SUBDIR_OVERRIDE=etc && - MAKEOBJDIRPREFIX=$1/usr/obj $make DESTDIR=$1 distribution) \ + if [ -n "$preworld" ]; then + # Build a limited tree that only contains files that are + # crucial to installworld. + for file in $PREWORLD_FILES; do + dir=`dirname /$file` + mkdir -p $1/$dir >&3 2>&1 || return 1 + cp -p $SRCDIR/$file $1/$file || return 1 + done + elif ! [ -n "$nobuild" ]; then + (cd $SRCDIR; $make DESTDIR=$destdir distrib-dirs && + MAKEOBJDIRPREFIX=$destdir/usr/obj $make _obj SUBDIR_OVERRIDE=etc && + MAKEOBJDIRPREFIX=$destdir/usr/obj $make everything SUBDIR_OVERRIDE=etc && + MAKEOBJDIRPREFIX=$destdir/usr/obj $make DESTDIR=$destdir distribution) \ >&3 2>&1 || return 1 else - (cd $SRCDIR; $make DESTDIR=$1 distribution) >&3 2>&1 || return 1 + (cd $SRCDIR; $make DESTDIR=$destdir distrib-dirs && + $make DESTDIR=$destdir distribution) >&3 2>&1 || return 1 fi chflags -R noschg $1 >&3 2>&1 || return 1 rm -rf $1/usr/obj >&3 2>&1 || return 1 @@ -218,9 +228,15 @@ build_tree() # source tree. extract_tree() { + local files + # If we have a tarball, extract that into the new directory. if [ -n "$tarball" ]; then - if ! (mkdir -p $NEWTREE && tar xf $tarball -C $NEWTREE) \ + files= + if [ -n "$preworld" ]; then + files="$PREWORLD_FILES" + fi + if ! (mkdir -p $NEWTREE && tar xf $tarball -C $NEWTREE $files) \ >&3 2>&1; then echo "Failed to extract new tree." remove_tree $NEWTREE @@ -1298,6 +1314,11 @@ resolve_cmd() return fi + if ! [ -d $NEWTREE ]; then + echo "The current tree is not present to resolve conflicts." + exit 1 + fi + conflicts=`(cd $CONFLICTS; find . ! -type d) | sed -e 's/^\.//'` for file in $conflicts; do resolve_conflict $file @@ -1343,7 +1364,7 @@ update_cmd() usage fi - log "update command: rerun=$rerun tarball=$tarball" + log "update command: rerun=$rerun tarball=$tarball preworld=$preworld" if [ `id -u` -ne 0 ]; then echo "Must be root to update a tree." @@ -1376,9 +1397,22 @@ update_cmd() echo "Unable to create temporary directory." exit 1 fi - OLDTREE=$NEWTREE + + # A pre-world dryrun has already set OLDTREE to + # point to the current stock tree. + if [ -z "$preworld" ]; then + OLDTREE=$NEWTREE + fi NEWTREE=$dir + # For a pre-world update, blow away any pre-existing + # NEWTREE. + elif [ -n "$preworld" ]; then + if ! remove_tree $NEWTREE; then + echo "Unable to remove pre-world tree." + exit 1 + fi + # Rotate the existing stock tree to the old tree. elif [ -d $NEWTREE ]; then # First, delete the previous old tree if it exists. @@ -1421,6 +1455,12 @@ EOF # Initialize conflicts and warnings handling. rm -f $WARNINGS mkdir -p $CONFLICTS + + # Ignore removed files for the pre-world case. A pre-world + # update uses a stripped-down tree. + if [ -n "$preworld" ]; then + > $WORKDIR/removed.files + fi # The order for the following sections is important. In the # odd case that a directory is converted into a file, the @@ -1535,7 +1575,8 @@ always= dryrun= ignore= nobuild= -while getopts "d:nrs:t:A:BD:FI:L:M:" option; do +preworld= +while getopts "d:nprs:t:A:BD:FI:L:M:" option; do case "$option" in d) WORKDIR=$OPTARG @@ -1543,6 +1584,9 @@ while getopts "d:nrs:t:A:BD:FI:L:M:" opt n) dryrun=YES ;; + p) + preworld=YES + ;; r) rerun=YES ;; @@ -1633,6 +1677,9 @@ WARNINGS=$WORKDIR/warnings # Use $EDITOR for resolving conflicts. If it is not set, default to vi. EDITOR=${EDITOR:-/usr/bin/vi} +# Files that need to be updated before installworld. +PREWORLD_FILES="etc/master.passwd etc/group" + # Handle command-specific argument processing such as complaining # about unsupported options. Since the configuration file is always # included, do not complain about extra command line arguments that @@ -1644,19 +1691,39 @@ case $command in echo usage fi + if [ -n "$rerun" -a -n "$preworld" ]; then + echo "Only one of -p or -r can be specified." + echo + usage + fi + ;; + build|diff|status) + if [ -n "$dryrun" -o -n "$rerun" -o -n "$tarball" -o + -n "$preworld" ]; then + usage + fi ;; - build|diff|resolve|status) + resolve) if [ -n "$dryrun" -o -n "$rerun" -o -n "$tarball" ]; then usage fi ;; extract) - if [ -n "$dryrun" -o -n "$rerun" ]; then + if [ -n "$dryrun" -o -n "$rerun" -o -n "$preworld" ]; then usage fi ;; esac +# Pre-world mode uses a different set of trees. It leaves the current +# tree as-is so it is still present for a full etcupdate run after the +# world install is complete. Instead, it installs a few critical files +# into a separate tree. +if [ -n "$preworld" ]; then + OLDTREE=$NEWTREE + NEWTREE=$WORKDIR/preworld +fi + # Open the log file. Don't truncate it if doing a minor operation so # that a minor operation doesn't lose log info from a major operation. if ! mkdir -p $WORKDIR 2>/dev/null; then