Skip site navigation (1)Skip section navigation (2)
Date:      Thu, 20 Oct 2016 18:23:44 +0000 (UTC)
From:      Ryan Stone <rstone@FreeBSD.org>
To:        src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org
Subject:   svn commit: r307680 - in head/tools/tools: . git
Message-ID:  <201610201823.u9KINiwr031764@repo.freebsd.org>

next in thread | raw e-mail | index | archive | help
Author: rstone
Date: Thu Oct 20 18:23:44 2016
New Revision: 307680
URL: https://svnweb.freebsd.org/changeset/base/307680

Log:
  Add some tools to simplify the use of git
  
  Add some scripts that wraps some FreeBSD Project infrastructure
  and simplifies using them with git.  The scripts are:
  
   - arcgit, which creates a series of reviews in Differential
   - importgit, which applies a series of git commits to svn
  
  Differential Revision:      https://reviews.freebsd.org/D2071

Added:
  head/tools/tools/git/
  head/tools/tools/git/HOWTO   (contents, props changed)
  head/tools/tools/git/arcgit   (contents, props changed)
  head/tools/tools/git/importgit   (contents, props changed)
Modified:
  head/tools/tools/README

Modified: head/tools/tools/README
==============================================================================
--- head/tools/tools/README	Thu Oct 20 17:28:52 2016	(r307679)
+++ head/tools/tools/README	Thu Oct 20 18:23:44 2016	(r307680)
@@ -29,6 +29,7 @@ gdb_regofs	A simple tool that prints out
 		platforms only.
 genericize	Turn a kernel config into something that can more easily
 		be diffed against the appropriate GENERIC.
+git		Tools to simplify the use of git by committers.
 hcomp		Compress header files by removing comments and whitespace.
 html-mv         Rename HTML generated filenames to human readable filenames.
 ifinfo		Uses the interface MIB to print out all the information

Added: head/tools/tools/git/HOWTO
==============================================================================
--- /dev/null	00:00:00 1970	(empty, because file is newly added)
+++ head/tools/tools/git/HOWTO	Thu Oct 20 18:23:44 2016	(r307680)
@@ -0,0 +1,144 @@
+# $FreeBSD$
+
+This directory contains tools intended to help committers use git when
+interacting with standard FreeBSD project resources like Differential or svn.
+
+I. arcgit
+
+arcgit is a wrapper script around the arc command line tool that simplifies the
+automatic creation of a series of Differential reviews from a series of git
+commits.  The intended workflow is:
+
+1. Create a series of commits in git.  Each commit will be a separate review, so
+   try to make each commit as self-contained as possible.  The series of commits
+   should demonstrate a logical progression towards your end goal.  For example,
+   one commit might refactor existing code to expose a new API without changing
+   any current functionality.  A subsequent commit could then introduce your new
+   code that uses the new API.
+
+   It usually will not be helpful to present your code in the order in which it
+   was actually written and can actually be harmful.  For example, if you
+   introduced a bug early in your development process that you fixed in a
+   subsequent commit, it is a waste of your reviewer's time to have them review
+   old code with known bugs.  Instead, it would probably be best to squash the
+   bug fix into the commit that introduced it, so that the bug is never
+   presented to your reviewers in any review.
+
+   The commit headline and commit message will be imported verbatim into
+   Differential, so try to give each commit a meaningful commit message that
+   gives your reviewers the necessary context to understand your change.
+
+2. Create your reviews bu running this command in your git repo:
+     $ arcgit -r C1~..C2 -R reviewer -T testplan
+
+   C1 should be the first commit that you want reviewed, and C2 should be the
+   last commit that you want reviewed.  You may add multiple reviewers by
+   specifying the -R option multiple times.  You can CC (AKA subscribe) people
+   to a review with the -C option.  Note that if you subscribe a mailing list
+   to a review, the mailing list will be emailed for every comment or change
+   made to each review.  Please be judicious when subscibing mailing lists to
+   reviews.  It may be better to instead send a single email to the appropriate
+   list announcing all of the reviews and giving a short summary of the change
+   as a whole, along with a link to the individual reviews.
+
+3. When you need to make a change and upload it to a review, use the following
+   process.  First, check out the branch corresponding to the review (arcgit
+   automatically creates this branch for every review that it creates):
+
+     $ git checkout review_D1234
+
+   Next, make your change and perform whatever testing is necessary.  Commit it
+   to your repository with this command:
+
+     $ git commit --fixup HEAD
+
+   You can upload the change to the code review by running this command in your
+   repository while (ensure that you are still on the review_D1234 branch):
+
+     $ arc diff --update D1234 review_D1234_base
+
+   When you run arc, it will pull up your editor and give you a chance to
+   change the message that will be shown in Differential for this upload.  It's
+   recommended that you change it from the default "fixup! ...." as that does
+   not give your reviewers an indication of what changes were made.  It's not
+   recommended that you give the code review fixes meaningful commit messages
+   directly because we will be using git's autosquash feature in the next step,
+   which depends on the fixup! tag being present.
+
+   Do not merge in other branches, or rebase your review branches at this point.
+   Any changes that are made will show up in your reviews, and that will create
+   extra noise that your reviewers will have to ignore.  If a reviewer requests
+   a change that requires your commit to be based off of a later version of
+   head, I would suggest deferring the change from the review and creating a
+   new review with the change once you hit step 5.
+
+4. Once the reviews have been approved, you need to prepare your patch series
+   to be committed.  This involves squashing the fixes made in code review
+   back into the original commit that they applied to.  This gives you a clean
+   series of commits that are ready to be commited back to svn.
+
+   First, merge each of your review branches back into your main development
+   branch.  For example:
+
+     $ git checkout myfeature_dev
+     $ for branch in review_D1234 review_D1235 review_D1236; do \
+       git merge $branch || git mergetool -y || break; done
+
+  Next, do an interactive rebase to squash your code review fixes back into the
+  main commit:
+
+    $ git rebase -i -k review_D1234_base
+
+  review_D1234 should be the name of the *first* review that was created for
+  you in step 2.  For every commit, your editor will be pulled up and you will
+  be given one last chance to edit your commit message.  Make sure that you fill
+  in the "Reviewed by:" tag indicating who accepted the review.  This would
+  be a good point to add other tags like MFC after:, Sponsored by:, etc.
+
+  The rebase will not introduce any actual changes to your code if done
+  properly.  You can use this command to double-check that no changes were
+  inadvertently made here:
+
+    $ git diff myfeature_dev@{1}
+
+5. Finally, you should get up to date with the latest changes from head:
+
+    $ git pull --rebase origin master
+
+  It is not recommended that you combine this step with the rebase done in the
+  previous step.  The reason for this is that if you perform an interactive
+  rebase that changes the commit that you branch from, it becomes much more
+  difficult to use the reflog to confirm that the interactive rebase did not
+  introduce unwanted changes.
+
+  At this point, you are ready to commit your changes to head.  The importgit
+  script can be used to import your commits directly into git.
+
+II. importgit
+
+importgit is a script that can take a series of commits from git and commit them
+to a svn repository.  The script uses the git commit messages for the svn commit
+message, which allows importgit to be fully automated.  This does mean that once
+you start importgit, it will start commit things to svn without giving any
+further chance to sanity check what it's doing.
+
+importgit only supports importing commits that add or modify files.  It does not
+support importing commits that rename or delete files, to ensure that git's
+rename detection heuristics do not introduce an error in the import process.
+importgit also does not support importing merge commits.  Only linear history
+can be imported into svn.
+
+importgit must be run from a clean subversion checkout.  You should ensure that
+the working tree is up-to-date with "svn up" before running importgit.
+importgit will run svn directly, so make sure that your ssh-agent is running
+and has your ssh key loaded already.  Run importgit as follows:
+
+  $ importgit -r D1~..D2 -g /path/to/git/repo
+
+This will import every commit between D1 and D2, including both D1 and D2.  The
+invocation is very similar to the invocation given to arcgit but there is an
+important point to note.  When you rebased your commits as you followed steps 4
+and 5, the commit hashes of all of your commits changed, including C1 and C2.
+You must go back and find the new commit hashes of your commits to pass to
+importgit.  Passing -r C1~..C2 would import your commits as they were *before*
+your code review fixes were applied.

Added: head/tools/tools/git/arcgit
==============================================================================
--- /dev/null	00:00:00 1970	(empty, because file is newly added)
+++ head/tools/tools/git/arcgit	Thu Oct 20 18:23:44 2016	(r307680)
@@ -0,0 +1,214 @@
+#!/bin/sh
+#
+# Copyright (c) 2015 Ryan Stone. All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+# 1. Redistributions of source code must retain the above copyright
+#    notice, this list of conditions and the following disclaimer.
+# 2. Redistributions in binary form must reproduce the above copyright
+#    notice, this list of conditions and the following disclaimer in the
+#    documentation and/or other materials provided with the distribution.
+#
+# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+# ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+# SUCH DAMAGE.
+#
+# $FreeBSD$
+
+# This script is used to submit a series of git commits to Differential.  Each
+# commit is submitted as a separate review.  For each review, this script will
+# create a branch called review_DXXXX (e.g. review_D2185 for Differential
+# revision D2185).  When you need to make a change to a review, checkout the
+# review_D2185 branch, commit your change with "git commit --fixup HEAD".  To\
+# upload the change to Differential, use the command:
+#   $ arc diff --update D2185 review_D2185_base
+#
+# When your reviews are complete, merge all of the review_DXXXX branches
+# together, and then do a git rebase -ik to meld the code review fixes into the
+# commit that they fixed.  Now you have a clean series of patches to commit to
+# svn.
+
+usage()
+{
+	echo "Usage: arcgit <-c commit | -r commit1~..commit2> [-R reviewer] " >&2
+	echo "       [-C subscriber] [-T testplan] [-n]" >&2
+}
+
+error()
+{
+	echo "$@" >&2
+	usage
+	rm -f $phab_before $phab_after $arc_msg
+	exit 1
+}
+
+create_review()
+{
+	local commit phab_id arc_dir
+	unset phab_before phab_after arc_msg
+	commit=$1
+
+	phab_before=`mktemp -t arcoutput`
+	phab_after=`mktemp -t arcoutput`
+	echo "Create review for '`git show $commit -s --oneline`'"
+
+	if [ -n "$dry_run" ]
+	then
+		return
+	fi
+
+	git checkout $commit > /dev/null || error "Could not checkout $commit"
+
+	arc_dir="$(git rev-parse --show-toplevel)/.git/arc"
+	arc_msg="$arc_dir/create-message"
+	mkdir -p $arc_dir
+	git show -s --format='%B' HEAD > $arc_msg
+	echo >> $arc_msg
+	echo "Test Plan:" >> $arc_msg
+	cat $test_plan >> $arc_msg
+	echo >> $arc_msg
+	echo "Reviewers:" >> $arc_msg
+	echo "$reviewers" >> $arc_msg
+	echo >> $arc_msg
+	echo "Subscribers:" >> $arc_msg
+	echo "$cc_list" >> $arc_msg
+	echo >> $arc_msg
+
+	arc list > $phab_before
+	yes | env EDITOR=true arc diff --create --allow-untracked HEAD~
+	arc list > $phab_after
+
+	headline="$(git show $commit -s --format=%s)"
+	phab_id=`comm -13 "$phab_before" "$phab_after" | fgrep "$headline" \
+	    | egrep -o 'D[0-9]+:' | tr -d ':'`
+
+	if [ -z "$phab_id" ]
+	then
+		error "Could not get review ID"
+	fi
+
+	git branch review_${phab_id}_base HEAD~
+
+	git checkout -b review_$phab_id
+	cat - <<EOF | git commit --allow-empty -F -
+squash! $headline
+
+Differential Revision:	https://reviews.freebsd.org/$phab_id
+Reviewed by:
+EOF
+}
+
+unset range test_plan reviewers cc_list dry_run
+
+while getopts ":c:C:nr:R:T:" o
+do
+	case "$o" in
+	c)
+		range="${OPTARG}~..${OPTARG}"
+		;;
+	C)
+		if [ -z "$cc_list" ]
+		then
+			cc_list="$OPTARG"
+		else
+			cc_list="$cc_list, $OPTARG"
+		fi
+		;;
+	n)
+		dry_run=1
+		;;
+	r)
+		range=$OPTARG
+		;;
+	R)
+		if [ -z "$reviewers" ]
+		then
+			reviewers="$OPTARG"
+		else
+			reviewers="$reviewers, $OPTARG"
+		fi
+		;;
+	T)
+		test_plan=$OPTARG
+		;;
+	*)
+		error "Unrecognized argument '-$OPTARG'"
+	esac
+done
+
+shift $((OPTIND - 1))
+OPTIND=1
+
+if [ -n "$1" ]
+then
+        error "Unrecognized argument $1"
+fi
+
+if [ -z "$range" ]
+then
+	error "-c or -r argument is mandatory"
+fi
+
+if [ -n "$test_plan" -a ! -r "$test_plan" ]
+then
+	error "$test_plan is not readable"
+fi
+
+if ! type git > /dev/null 2> /dev/null
+then
+	error "Install devel/git first"
+fi
+
+if ! type arc > /dev/null 2> /dev/null
+then
+	error "Install devel/arcanist first"
+fi
+
+git update-index -q --refresh
+if ! git diff-index --quiet --cached HEAD
+then
+	error "index is unclean"
+fi
+
+if ! git diff-files --quiet
+then
+	error "Working directory is unclean"
+fi
+
+if git ls-files --other --error-unmatch . > /dev/null 2> /dev/null
+then
+	error "Working directory contains untracked files"
+fi
+
+# We have to do a git checkout in order to run arc, so save the original branch
+# so that we can check it out again once we're done.
+if ! orig_branch=$(git symbolic-ref --short -q HEAD)
+then
+	orig_branch=$(git show -s --pretty='%H' HEAD)
+fi
+
+git log --format=%H $range | tail -r | while read -r commit
+do
+	create_review $commit < /dev/null
+done
+
+# Note that due to the use of the pipeline above, the body of the while loop
+# above runs in a subshell.  If it exits with an error, execution resumes
+# here rather than exiting the script, so we have to cache the right exit code
+# and return it when we're done cleaning up.
+code=$?
+
+git checkout $orig_branch
+
+exit $code
+

Added: head/tools/tools/git/importgit
==============================================================================
--- /dev/null	00:00:00 1970	(empty, because file is newly added)
+++ head/tools/tools/git/importgit	Thu Oct 20 18:23:44 2016	(r307680)
@@ -0,0 +1,182 @@
+#!/bin/sh
+#
+# Copyright (c) 2015 Ryan Stone. All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+# 1. Redistributions of source code must retain the above copyright
+#    notice, this list of conditions and the following disclaimer.
+# 2. Redistributions in binary form must reproduce the above copyright
+#    notice, this list of conditions and the following disclaimer in the
+#    documentation and/or other materials provided with the distribution.
+#
+# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+# ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+# SUCH DAMAGE.
+#
+# $FreeBSD$
+
+usage()
+{
+	echo "Usage: importgit <-c commit | -r c1..c2> -g /path/to/git/repo [-n]" >&2
+}
+
+error()
+{
+	local print_usage
+
+	if [ "$1" = "-u" ]
+	then
+		shift
+		print_usage=1
+	else
+		print_usage=
+	fi
+
+	echo "$@" >&2
+	if [ -n "$print_usage" ]
+	then
+		usage
+	fi
+	exit 1
+}
+
+unset git range commit dry_run
+
+while getopts ":c:g:nr:" o
+do
+	case "$o" in
+	c)
+		range="${OPTARG}~..${OPTARG}"
+		;;
+	g)
+		git_repo=$OPTARG
+		;;
+	n)
+		dry_run=1
+		;;
+	r)
+		range=$OPTARG
+		;;
+	*)
+		error -u "Unrecognized argument '-$OPTARG'"
+        esac
+done
+
+shift $((OPTIND - 1))
+OPTIND=1
+
+if [ -n "$1" ]
+then
+	error -u "Unrecognized argument $1"
+fi
+
+if [ -z "$range" ]
+then
+	error -u "-c or -r argument is mandatory"
+fi
+
+if ! echo "$range" | egrep -qs '^[^.]+\.\.[^.]*$'
+then
+	error -u "$range is not a range of commits.  Did you mean '-c $range'?"
+fi
+
+if [ -z "$git_repo" ]
+then
+	error -u "-g <repo> argument is mandatory"
+fi
+
+git="$git_repo/.git"
+
+if [ ! -d "$git" ]
+then
+	error "$git_repo does not seem to be a git repo"
+fi
+
+if ! type git > /dev/null 2> /dev/null
+then
+	error "Install devel/git first"
+fi
+
+if ! type svn > /dev/null 2> /dev/null
+then
+	error "Install devel/subversion first"
+fi
+
+if [ -n "$(svn status)" ]
+then
+	error "Working tree is not clean"
+fi
+
+if ! svn --non-interactive ls > /dev/null
+then
+	error "Could not communicate with svn server.  Is your ssh key loaded?"
+fi
+
+git --git-dir=$git log --format=%H $range | tail -r | while read -r commit
+do
+	echo "Applying `git --git-dir=$git show -s --oneline $commit`"
+
+	if [ -n "$(git --git-dir=$git show --diff-filter=CDRTUXB $commit)" ]
+	then
+		error "Commit performed unsupported change (e.g. delete/rename)"
+	fi
+
+	if [ "$(git --git-dir=$git show -s --format=%P $commit | wc -w)" -ne 1 ]
+	then
+		error "Cannot import merge commits"
+	fi
+
+	git --git-dir=$git diff --diff-filter=A --name-only \
+	    ${commit}~..$commit | while read -r newfile
+	do
+		if [ -f "$newfile" ]
+		then
+			error "New file $newfile already exists in tree"
+		fi
+	done
+
+	# The previous while loop ran in a subshell, so we have to check if it
+	# exited with an error and bail out if so.
+	ret=$?
+	if [ "$ret" -ne 0 ]
+	then
+		exit $ret
+	fi
+
+	if [ -n "$dry_run" ]
+	then
+		continue
+	fi
+
+	git --git-dir=$git show $commit | patch -p 1 -s || \
+	    error "Failed to apply patch"
+
+	git --git-dir=$git diff --diff-filter=A --name-only \
+	    ${commit}~..$commit | while read -r newfile
+	do
+		svn add --parents --depth=infinity $newfile || \
+		    error "Failed to add new file"
+	done
+
+	# The previous while loop ran in a subshell, so we have to check if it
+	# exited with an error and bail out if so.
+	ret=$?
+	if [ "$ret" -ne 0 ]
+	then
+		exit $ret
+	fi
+
+	git --git-dir=$git show -s --format='%B' $commit | svn commit -F - || \
+	    error "Failed to commit"
+done
+



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201610201823.u9KINiwr031764>